Hello all, I am down to my last hurdle of my project, backups. I am thinking of three different ways to go and wanted to ask for input on what you think is the better choice. Not asking for 'how to' but more of 'what is best in your experience'
The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage.
What I want to do.. auto backups of the virtual machines to be stored on the virtual host's extra drives for later download to my home computer.
Many backup solutions and programs seem centered on a network of computers with file sharing. I do not have this and don't think I want to go that way on my host.
My three thoughts, not sure which one to pursue... (involves certain folders, /home/ (which includes maildir), /var/www/, /mysqlhotcopys and bin files, and maybe a few more. I can rebuild the comp pretty quick and then restore, or maybe just do one big backup of each server, then work on the folders as a solution)
1- Amanda. I do not know much about it or how it would deal with mysql databases, but it look promising. I do not have a NFS in place on any of the installs.
2- rsnyc - some kind of rsync going from the host to each machine, putting it on the host's backup drives. Adding a mysql hotcopy of some kind on the VMs, along with bin files, saved to a special folder that will then be part of the rsync. Once a week full of both rsync and mysqlcopy, then incremental daily.
3- Use kpartx ? and access the lvm the VM is on to rsync internally on the host, ditto above with the mysql copy/bin setup.
Number 3 seems like it is the securest way, but obviously not much info out there on it. Number 2 seems like the 'old way' and will require some real work to get it right number 1 looks good, but do not really know anything about it.
Which way would you go, or do you have a different way you like better?
On Sat, Mar 24, 2012 at 8:19 AM, Bob Hoffman bob@bobhoffman.com wrote:
The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage.
What I want to do.. auto backups of the virtual machines to be stored on the virtual host's extra drives for later download to my home computer.
Hi Bob, You want to backup the whole VM? You can set a script to periodically shutdown the VM and then just rsync the VM files to the other partition.
Snapshot the lvm and rsync it to your house. Then wipe out the snap. Seems easiest. Though it won't be the best for things like running databases.
Sent from my iPhone
On Mar 23, 2012, at 8:20 PM, Bob Hoffman bob@bobhoffman.com wrote:
Hello all, I am down to my last hurdle of my project, backups. I am thinking of three different ways to go and wanted to ask for input on what you think is the better choice. Not asking for 'how to' but more of 'what is best in your experience'
The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage.
What I want to do.. auto backups of the virtual machines to be stored on the virtual host's extra drives for later download to my home computer.
Many backup solutions and programs seem centered on a network of computers with file sharing. I do not have this and don't think I want to go that way on my host.
My three thoughts, not sure which one to pursue... (involves certain folders, /home/ (which includes maildir), /var/www/, /mysqlhotcopys and bin files, and maybe a few more. I can rebuild the comp pretty quick and then restore, or maybe just do one big backup of each server, then work on the folders as a solution)
1- Amanda. I do not know much about it or how it would deal with mysql databases, but it look promising. I do not have a NFS in place on any of the installs.
2- rsnyc - some kind of rsync going from the host to each machine, putting it on the host's backup drives. Adding a mysql hotcopy of some kind on the VMs, along with bin files, saved to a special folder that will then be part of the rsync. Once a week full of both rsync and mysqlcopy, then incremental daily.
3- Use kpartx ? and access the lvm the VM is on to rsync internally on the host, ditto above with the mysql copy/bin setup.
Number 3 seems like it is the securest way, but obviously not much info out there on it. Number 2 seems like the 'old way' and will require some real work to get it right number 1 looks good, but do not really know anything about it.
Which way would you go, or do you have a different way you like better? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sat, Mar 24, 2012 at 2:28 AM, Steven Crothers steven.crothers@gmail.comwrote:
Snapshot the lvm and rsync it to your house. Then wipe out the snap. Seems easiest. Though it won't be the best for things like running databases.
How about you lock all tables and flush buffers, then do the snapshot, and release the lock. Now backup then snapshot and remove it. Shouldn't take more than a few seconds for the lock-snapshot-release part, and you avoid the problem of inconsistent data in your db.
In other places, I've seen people do master-slave replication and using mysqldump on the slave (cuz mysqldump locks tables as well)
BR Bent
For MySQL backups, I think nobody mentioned doing nightly backups with mysqldump and then backing them up with rsync or other tools. That is what I do. No need to stop any services even for a second.
I have a two-tier backup solution, where I first backup things with rsync to "backup 1" and then use rdiff-backup to construe an incremental archive on "backup 2" (in other location).
- Jussi
On 24.3.2012 08:43, Jussi Hirvi wrote:
For MySQL backups, I think nobody mentioned doing nightly backups with mysqldump and then backing them up with rsync or other tools. That is what I do. No need to stop any services even for a second.
With MyISAM tables, to keep integrity, you might want to read lock the tables. Depending on the amount of data a full dump can take a long time. If you have to write while dumping, you cant, and this is a kind of a stopped service ;-)
On Fri, 23 Mar 2012 20:19:41 -0400, Bob Hoffman bob@bobhoffman.com said:
B> I am down to my last hurdle of my project, backups. Not asking for 'how B> to' but more of 'what is best in your experience'.
Some questions:
* What's the hardest stuff for you to recreate? I'd have that on both DVD and something network-accessible.
* What's your biggest PITA problem (for me, it would be bare-metal restore) vs. your most likely one (I'd assume loss of a MySQL table or a VM)? You mentioned being able to rebuild the host quickly, so if the bare-metal thing isn't a big problem, concentrate on the VMs instead.
* What are your priorities? If it's speed of the restore, and you have the IO/network bandwidth and room, then do like another poster said and rsync the VM files after shutting them down. If it's more like history where you want to go back in time to lots of versions, something finer-grained would be in order.
B> The scenario... centos server acting as a virtual host. Virtual B> machines are webservers and dns servers. All on one machine, all running B> centos 6. Virtual machines are kvm, sitting in lvm storage. What I B> want to do.. auto backups of the virtual machines to be stored on the B> virtual host's extra drives for later download to my home computer.
Your VMs sound like they start out identical, and then you add stuff to specialize each one. If so, I'd keep these backups:
a. one generic bare-bones VM that can be installed with as few commands as possible. b. each change-set you use to specialize for basic DNS, web, etc. c. smaller groups of individual files like DB schemas, web content, mailboxes, etc.
This way, any given restore breaks down to (a) plus (one or more b) plus (whatever's appropriate from c). When you get to the individual file backups within a VM, something like this might be all you need:
# cd / # find . -depth -type f -newer /etc/BKUP -print | pax -x cpio -wd | gzip -c > /path/to/$(date '+%Y/%m%d/%H%M').pax.gz # touch /etc/BKUP
B> 1- Amanda. I do not know much about it or how it would deal with mysql B> databases, but it look promising.
I set it up once, but it wasn't a close enough match to what we needed for me to craft an entire backup strategy around it. It's not a trivial thing to install or run, so you'll be spending time finding out how Amanda wants to do things and matching that to your goals.
B> 2- rsync - some kind of rsync going from the host to each machine, B> putting it on the host's backup drives.
That's what I use at work, but we're closer to the "networked fileservers with remote shares" setup. I use the find/pax/touch setup above to handle hourly backups for 800-1000 users, and they're happy little campers when they find out the spreadsheet they created at 6am and mangled around noon isn't completely gone.
On 3/23/2012 10:50 PM, Karl Vogel wrote:
On Fri, 23 Mar 2012 20:19:41 -0400, Bob Hoffmanbob@bobhoffman.com said:
B> I am down to my last hurdle of my project, backups. Not asking for 'how B> to' but more of 'what is best in your experience'.
Some questions: * What's the hardest stuff for you to recreate? I'd have that on both DVD and something network-accessible. * What's your biggest PITA problem (for me, it would be bare-metal restore) vs. your most likely one (I'd assume loss of a MySQL table or a VM)? You mentioned being able to rebuild the host quickly, so if the bare-metal thing isn't a big problem, concentrate on the VMs instead. * What are your priorities? If it's speed of the restore, and you have the IO/network bandwidth and room, then do like another poster said and rsync the VM files after shutting them down. If it's more like history where you want to go back in time to lots of versions, something finer-grained would be in order.B> The scenario... centos server acting as a virtual host. Virtual B> machines are webservers and dns servers. All on one machine, all running B> centos 6. Virtual machines are kvm, sitting in lvm storage. What I B> want to do.. auto backups of the virtual machines to be stored on the B> virtual host's extra drives for later download to my home computer.
Your VMs sound like they start out identical, and then you add stuff to specialize each one. If so, I'd keep these backups: a. one generic bare-bones VM that can be installed with as few commands as possible. b. each change-set you use to specialize for basic DNS, web, etc. c. smaller groups of individual files like DB schemas, web content, mailboxes, etc. This way, any given restore breaks down to (a) plus (one or more b) plus (whatever's appropriate from c). When you get to the individual file backups within a VM, something like this might be all you need: # cd / # find . -depth -type f -newer /etc/BKUP -print | pax -x cpio -wd | gzip -c> /path/to/$(date '+%Y/%m%d/%H%M').pax.gz # touch /etc/BKUPB> 1- Amanda. I do not know much about it or how it would deal with mysql B> databases, but it look promising.
I set it up once, but it wasn't a close enough match to what we needed for me to craft an entire backup strategy around it. It's not a trivial thing to install or run, so you'll be spending time finding out how Amanda wants to do things and matching that to your goals.B> 2- rsync - some kind of rsync going from the host to each machine, B> putting it on the host's backup drives.
That's what I use at work, but we're closer to the "networked fileservers with remote shares" setup. I use the find/pax/touch setup above to handle hourly backups for 800-1000 users, and they're happy little campers when they find out the spreadsheet they created at 6am and mangled around noon isn't completely gone.
I am not looking to back up the vms for a easy reinstall, I can do them in less than a 1/2 hour each. The back up is for the webservers so the database and html and some other folders are continually backed up incase of hack or whatever.
Still thinking amanda, or bacula as first choice, rsync second, kpartx somehow third.
On Sat, Mar 24, 2012 at 11:22 AM, Bob Hoffman bob@bobhoffman.com wrote:
I am not looking to back up the vms for a easy reinstall, I can do them in less than a 1/2 hour each. The back up is for the webservers so the database and html and some other folders are continually backed up incase of hack or whatever.
Still thinking amanda, or bacula as first choice, rsync second, kpartx somehow third.
If that's the case, you don't have to mention the VM at all. For database, there is automysqlbackup (and postrgre) too in sourceforge. For the html and folders, just use rsnapshot would be nice.
On 03/23/2012 05:19 PM, Bob Hoffman wrote:
Hello all, I am down to my last hurdle of my project, backups. I am thinking of three different ways to go and wanted to ask for input on what you think is the better choice. Not asking for 'how to' but more of 'what is best in your experience'
The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage.
What I want to do.. auto backups of the virtual machines to be stored on the virtual host's extra drives for later download to my home computer.
Many backup solutions and programs seem centered on a network of computers with file sharing. I do not have this and don't think I want to go that way on my host.
My three thoughts, not sure which one to pursue... (involves certain folders, /home/ (which includes maildir), /var/www/, /mysqlhotcopys and bin files, and maybe a few more. I can rebuild the comp pretty quick and then restore, or maybe just do one big backup of each server, then work on the folders as a solution)
1- Amanda. I do not know much about it or how it would deal with mysql databases, but it look promising. I do not have a NFS in place on any of the installs.
2- rsnyc - some kind of rsync going from the host to each machine, putting it on the host's backup drives. Adding a mysql hotcopy of some kind on the VMs, along with bin files, saved to a special folder that will then be part of the rsync. Once a week full of both rsync and mysqlcopy, then incremental daily.
I think backuppc, syncing over rsync directly to the VM makes sense here.
3- Use kpartx ? and access the lvm the VM is on to rsync internally on the host, ditto above with the mysql copy/bin setup.
You can't safely mount the VM's partitions without shutting down the VM. You might get away with it with an LVM snapshot taken inside the host, but that assumes that it's safe to access the LVM datastructure from outside the VM while the VM is running and it may not be.
Number 3 seems like it is the securest way, but obviously not much info out there on it. Number 2 seems like the 'old way' and will require some real work to get it right number 1 looks good, but do not really know anything about it.
For the database you have many options. I would suggest at least weekly do this: 1) shutdown the mysql database server 2) take a snapshot of the partition with the mysql database server 3) restart your mysql server (the whole shutdown/snapshot/restart take less than 15 seconds) 4) Do a full backup of that snapshot, using backuppc, tar, dump or whatever you choose
In order to do this, you must of course setup the database on an LV partition (MY LV partition is simply an ext4 or ext 3 partition, mounted on /var/lib/mysql). Make sure there is adequate space on your snapshot LV partition for any data that might get written to the database before the snapshot gets backuped and deleted.
If your database is small you can do daily backups of the entire database like this. Otherwise you can use tools like mysqldump and mysqlhotcopy, but you should make sure you have a pretty good understanding of these tools and that you are backing things up properly with them.
Some people run replication between two database servers and then shutdown the secondary server and back it up.
There are also backup tools like duplicity and storebackup which I have recently learned about that are supposed to be able to do incremental backups of binary files, where you could actually do a restore of the database from your full backup and then somehow apply an update to the database files from incrementals. I have not yet tested this method. There again, I would still shutdown/snapshot/restart the mysql server to ensure database integrity during the backup.
Whatever you choose, I suggest you perform test restores and if you choose database level backups, make sure you understand what you are doing and that you are saving the backup in a way that you are able to recover it.
If you data is important consider doing backing up in two different formats to 2 different media types.
I think simply doing at least a weekly full backup of the entire /var/lib/mysql directory (via snapshot or server shutdown) is the least prone to user error.
Be aware that things like changes in database format, etc can make old backups difficult or impossible to restore. Many people, myself included, ran into some problems when mysql changed the way it handled character set encodings and it was not so straightforward to restore mysqldump backups made with an older version of mysql, using the latest version of mysql. Many options had to be specified and some people still had to write scripts to manually go through and make some conversions. I ran into these problems while upgrading a mediawiki database that was running under a very old version of mysql for a client.
So if having long term access to old backups is important, you must check for compatibility between newer/older versions of mysql and either do conversion/refresh of your data, or make sure you still have functional copies of the tools necessary to restore older versions.
Nataraj
Which way would you go, or do you have a different way you like better? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hi Bob,
I just want to to throw in another alternative to make choice harder ... :-)
The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage.
My basic setup is quite similar to yours. CentOS 5 machine working as host, several virtual CentOS 5/6 boxes on that host, but additionally there are some Mac OS X and Ubuntu boxes around that also need to be backed up, some of them in remote locations.
I have a NAS drive set up in my small datacenter that works as an iSCSI host and serves a LUN for backups. The virtual host machine runs Bacula dir and sd, with the backup volumes on the LUN. All Clients run bacula fd and connect to the server, the ones connecting from the internet use SSL encryption and certificate authentication. I run full backups once a week, and daily incrementals.
For the last half year or so, the solution has proven rock-solid, not a single failure. I had to restore several files during that period, and there wasn't any problem at all with that as well.
Database backup is done using pre-scripts that perform a database dump and then back up that dump, which is a bit of a downside as other solutions often have database backup plugins, but nothing that could not be solved. After initial setup, configuration is done on the Bacular directory server alone, which saves some amount of maintenance. Backup strategies can be defined in a very flexible way, too.
There also is a feature that allows you to define a base system (e.g. a freshly installed CentOS box) and tell the server just to back up differences from that base system, saving large amounts of storage if you have many similar machines to be backed up.
Best regards,
Peter.
On 3/24/2012 4:35 AM, Peter Eckel wrote:
Hi Bob,
I just want to to throw in another alternative to make choice harder ... :-)
The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage.
My basic setup is quite similar to yours. CentOS 5 machine working as host, several virtual CentOS 5/6 boxes on that host, but additionally there are some Mac OS X and Ubuntu boxes around that also need to be backed up, some of them in remote locations.
I have a NAS drive set up in my small datacenter that works as an iSCSI host and serves a LUN for backups. The virtual host machine runs Bacula dir and sd, with the backup volumes on the LUN. All Clients run bacula fd and connect to the server, the ones connecting from the internet use SSL encryption and certificate authentication. I run full backups once a week, and daily incrementals.
For the last half year or so, the solution has proven rock-solid, not a single failure. I had to restore several files during that period, and there wasn't any problem at all with that as well.
Database backup is done using pre-scripts that perform a database dump and then back up that dump, which is a bit of a downside as other solutions often have database backup plugins, but nothing that could not be solved. After initial setup, configuration is done on the Bacular directory server alone, which saves some amount of maintenance. Backup strategies can be defined in a very flexible way, too.
There also is a feature that allows you to define a base system (e.g. a freshly installed CentOS box) and tell the server just to back up differences from that base system, saving large amounts of storage if you have many similar machines to be backed up.
Best regards,
Peter. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
thanks peter, that was where I was looking at going. I think amanda or bacula (or both) have a mysql backup program as an extra that will perform an incremental.
For mysql, it has a bin file system that can be rotated daily, hourly, whatever with a full dump only needed when you want (once a week?) so I would just grab the bin files I think.
When I figure this out, perfectly, I should post it all so others can have a backup solution. I have over 40 linux books and not one really goes into backups. They mention them, but no working examples of merit.
On Sat, 24 Mar 2012 09:48:15 -0400, Bob Hoffman bob@bobhoffman.com said:
B> When I figure this out, perfectly, I should post it all so others can B> have a backup solution. I have over 40 linux books and not one really B> goes into backups. They mention them, but no working examples of merit.
Backup & Recovery: Inexpensive Backup Solutions for Open Systems W. Curtis Preston Paperback: 768 pages Publisher: O'Reilly Media; 1 edition (January 10, 2007) ISBN-10: 0596102461 ISBN-13: 978-0596102463
Taken from the book description:
* Covers tar, dump, cpio, ntbackup, ditto, and rsync.
* Explains several open source backup products that automate backups using those utilities, including AMANDA, Bacula, BackupPC, rdiff-backup, and rsnapshot.
* How to perform bare metal recovery of AIX, HP-UX, Linux, Mac OS, Solaris, VMWare, & Windows systems using freely-available utilities.
* How to automate DB2, Exchange, MySQL, Oracle, PostgreSQL, SQL-Server, and Sybase backups without using a commercial backup product.
-
Barry Brimer -
Bent Terp -
Bob Hoffman -
Fajar Priyanto -
Jussi Hirvi -
Markus Falb -
Nataraj -
Peter Eckel -
Steven Crothers -
vogelke+centos＠pobox.com