What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
On 03/11/2012 08:12 PM, Scott Walker wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
What do you consider to be a "long time" to keep backups on hand?
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
I run my backups nightly to a hard drive using rsync. I use a directory named by the day of the week. I cycle through the seven daily directories until the 1st of the month when I run a complete backup to an monthly directory. Then for the next seven days I wipe the daily directories and start the cycle over again.
A couple of minor variations to this plan should work for you. I don't know what your network configuration looks like so this may not apply to you.
Here's a peek at the logic I use.
# BUILD DATE STAMP Date=`date +%Y%m%d` echo "Date= "$Date""
# Rev. 5.6 start Day=`date +%a` echo "Day= "$Day""
DayNum=`date +%d` # Rev. 7.0
# IF THIS IS A SUNDAY USE THE CALANDAR DATE if [ "$Day" == "Sun" ];then Day="$Date" else # IF THIS IS THE 1ST OF THE MONTH USE THE CALANDAR DATE if [ "$DayNum" == "01" ];then Day="$Date" fi fi
# USE THE DAY OF THE WEEK, EXCEPT FOR SUNDAY AND THE 1ST OF THE MONTH WHICH IS HANDLED ABOVE, AS THE DIRECTORY NAME Date="$Day"
# Rev. 5.6 end
# REMOVE PREVIOUS $Date DIRECTORY IF THIS IS THE FIRST USE THIS MONTH # Rev. 7.0 ENTIRE CASE STATEMENT ADDED case $DayNum in 02) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 03) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 04) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 05) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 06) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 07) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; *) echo "Old $Date directory not deleted" ;; esac
# TRANSER FILES
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Mark LaPierre Sent: Sunday, March 11, 2012 8:37 PM To: CentOS mailing list Subject: Re: [CentOS] CentOS Server Backup Options
On 03/11/2012 08:12 PM, Scott Walker wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I
like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
What do you consider to be a "long time" to keep backups on hand?
I like to have an archive copy of the data for each of the last twelve months.
I also like to have an archive copy of the data for each of the last 4 years.
That way if any files get accidently deleted, I still have a backup that is old enough to contain them.
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Indeed I've had lots of trouble with tape drives over the years. The DAT drives worked well when they worked but they always seemed to die after 4 or 5 years. With the small amount of data I have to worry about (in the range of 25 - 30gb) the time to backup to tape and verify in the middle of the night is not a factor.
I run my backups nightly to a hard drive using rsync. I use a directory named by the day of the week. I cycle through the seven daily directories until the 1st of the month when I run a complete backup to an monthly directory. Then for the next seven days I wipe the daily directories and start the cycle over again.
A couple of minor variations to this plan should work for you. I don't know what your network configuration looks like so this may not apply to you.
Here's a peek at the logic I use.
# BUILD DATE STAMP Date=`date +%Y%m%d` echo "Date= "$Date""
# Rev. 5.6 start Day=`date +%a` echo "Day= "$Day""
DayNum=`date +%d` # Rev. 7.0
# IF THIS IS A SUNDAY USE THE CALANDAR DATE if [ "$Day" == "Sun" ];then Day="$Date" else # IF THIS IS THE 1ST OF THE MONTH USE THE CALANDAR DATE if [ "$DayNum" == "01" ];then Day="$Date" fi fi
# USE THE DAY OF THE WEEK, EXCEPT FOR SUNDAY AND THE 1ST OF THE MONTH WHICH IS HANDLED ABOVE, AS THE DIRECTORY NAME Date="$Day"
# Rev. 5.6 end
# REMOVE PREVIOUS $Date DIRECTORY IF THIS IS THE FIRST USE THIS MONTH # Rev. 7.0 ENTIRE CASE STATEMENT ADDED case $DayNum in 02) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 03) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 04) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 05) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 06) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 07) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; *) echo "Old $Date directory not deleted" ;; esac
# TRANSER FILES
-- _ °v° /(_)\ ^ ^ Mark LaPierre Registerd Linux user No #267004 _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 03/12/2012 12:11 AM, Scott Walker wrote:
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Mark LaPierre Sent: Sunday, March 11, 2012 8:37 PM To: CentOS mailing list Subject: Re: [CentOS] CentOS Server Backup Options
On 03/11/2012 08:12 PM, Scott Walker wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I
like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
What do you consider to be a "long time" to keep backups on hand?
I like to have an archive copy of the data for each of the last twelve months.
I also like to have an archive copy of the data for each of the last 4 years.
That way if any files get accidently deleted, I still have a backup that is old enough to contain them.
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Indeed I've had lots of trouble with tape drives over the years. The DAT drives worked well when they worked but they always seemed to die after 4 or 5 years. With the small amount of data I have to worry about (in the range of 25 - 30gb) the time to backup to tape and verify in the middle of the night is not a factor.
I run my backups nightly to a hard drive using rsync. I use a directory named by the day of the week. I cycle through the seven daily directories until the 1st of the month when I run a complete backup to an monthly directory. Then for the next seven days I wipe the daily directories and start the cycle over again.
A couple of minor variations to this plan should work for you. I don't know what your network configuration looks like so this may not apply to you.
Here's a peek at the logic I use.
# BUILD DATE STAMP Date=`date +%Y%m%d` echo "Date= "$Date""
# Rev. 5.6 start Day=`date +%a` echo "Day= "$Day""
DayNum=`date +%d` # Rev. 7.0
# IF THIS IS A SUNDAY USE THE CALANDAR DATE if [ "$Day" == "Sun" ];then Day="$Date" else # IF THIS IS THE 1ST OF THE MONTH USE THE CALANDAR DATE if [ "$DayNum" == "01" ];then Day="$Date" fi fi
# USE THE DAY OF THE WEEK, EXCEPT FOR SUNDAY AND THE 1ST OF THE MONTH WHICH IS HANDLED ABOVE, AS THE DIRECTORY NAME Date="$Day"
# Rev. 5.6 end
# REMOVE PREVIOUS $Date DIRECTORY IF THIS IS THE FIRST USE THIS MONTH # Rev. 7.0 ENTIRE CASE STATEMENT ADDED case $DayNum in 02) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 03) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 04) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 05) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 06) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 07) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; *) echo "Old $Date directory not deleted" ;; esac
# TRANSER FILES
-- _ °v° /(_)\ ^ ^ Mark LaPierre Registerd Linux user No #267004 _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
If each of your backups contains the entire 25gb of data that you mention then a 1 tb drive will hold +/- 40 said backups. That should be enough for a years worth of backups. Just get a new 1 tb drive each year for four years and you're golden for a lot less than a single tape drive. Install a drive in a remote computer and have off site backups too.
On 12.3.2012 01:37, Mark LaPierre wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Anyway, I would not feel comfortable about backing up data residing on a harddisk to another harddisk. I believe that a backup media has to provide different characteristics than the original media. An incident that harms original should not harm the backup.
What about if a firmware bug destroys all data on day XXX on all harddisks ? Well, extra paranoid maybe and of course I have not thought of all possible things that *could* happen.
On Tue, Mar 13, 2012 at 1:26 PM, Markus Falb markus.falb@fasel.at wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Yes, but if they are online, in raid, with smart monitoring, you swap them (maybe every 5 years or so, getting better...) and let the replacement re-sync. Anything else and you don't find out that it is dead until you are doing a restore.
Anyway, I would not feel comfortable about backing up data residing on a harddisk to another harddisk. I believe that a backup media has to provide different characteristics than the original media. An incident that harms original should not harm the backup.
I thought the old saying was that if something was important you should have 3 copies, and don't let the person who destroyed the first 2 touch the 3rd.
What about if a firmware bug destroys all data on day XXX on all harddisks ? Well, extra paranoid maybe and of course I have not thought of all possible things that *could* happen.
If you've been replacing your raid drives as they die over the years, you probably won't be left with all the same model when this event happens.
Markus Falb wrote:
On 12.3.2012 01:37, Mark LaPierre wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Not that frequently.
Anyway, I would not feel comfortable about backing up data residing on a harddisk to another harddisk. I believe that a backup media has to provide different characteristics than the original media. An incident that harms original should not harm the backup.
Why?
What about if a firmware bug destroys all data on day XXX on all harddisks ? Well, extra paranoid maybe and of course I have not thought of all possible things that *could* happen.
Are you saying that you only buy one model from one maker? In the mid-nineties, every ISP in Chicago dumped *all* of their SCSI Seagate Barracudas for failures. The next year, at work, I had an external box for a Sun server that had *four* of them: in the next year, Sun replaced various of them *five* times. Meanwhile, the IBM drives inside the server were just fine.
mark
On 13.3.2012 19:46, m.roth@5-cent.us wrote:
Markus Falb wrote:
What about if a firmware bug destroys all data on day XXX on all harddisks ? Well, extra paranoid maybe and of course I have not thought of all possible things that *could* happen.
Are you saying that you only buy one model from one maker? In the mid-nineties, every ISP in Chicago dumped *all* of their SCSI Seagate Barracudas for failures.
Well, take it one step further. What if the bad bug were not in the firmware but in software (kernel?)
On Tue, Mar 13, 2012 at 5:12 PM, Markus Falb markus.falb@fasel.at wrote:
On 13.3.2012 19:46, m.roth@5-cent.us wrote:
Markus Falb wrote:
What about if a firmware bug destroys all data on day XXX on all harddisks ? Well, extra paranoid maybe and of course I have not thought of all possible things that *could* happen.
Are you saying that you only buy one model from one maker? In the mid-nineties, every ISP in Chicago dumped *all* of their SCSI Seagate Barracudas for failures.
Well, take it one step further. What if the bad bug were not in the firmware but in software (kernel?)
So a bug in linux that on some future date will eat all your disks? Something to think about, I guess... For a more generic bug that happens immediately, someone would notice before it makes it to CentOS. The more likely bug would be in the person setting up the backup software. But regular test testores can catch those.
Am 13.03.2012 19:46, schrieb m.roth@5-cent.us:
Markus Falb wrote:
On 12.3.2012 01:37, Mark LaPierre wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Not that frequently.
I beg to differ. Hard disk failures are by far the most frequent hardware problem I encounter at work. And those external USB drives people are so fond of for backup are certainly not better than typical server drives in that respect.
here's my tentative plans for a d2d backup in my lab.
2 identical servers, each with lots of SATA bays. each server configured with 2 raids, raid1 is this servers storage, and raid2 is a DRBD mirror of the other servers storage.
each server runs KVM and under KVM runs a CentOS virtual machine which runs BackupPC, that has half the backup jobs assigned to it.
if server2 fails, then server1 fires up the backup2 VM on its DRBD replica of server2's storage, and all continues as it was, with just less performance until the 2nd server can be restored, and the backup2 volume replicated back to it so the backup2 VM can be stopped on server1 and resumed on server2.
this doesn't address offsite archiving, and I haven't quite decided if I'm going to A) replicate the important backups to additional offsite storage, or B) periodically make archive backups onto a separate HD that is then dismounted and pulled and shipped to the data vault (the production IT at my site has weekly drops to a data security company, I could add a disk to their lockbox. I may end up doing a combination of these things.
On Tue, Mar 13, 2012 at 7:05 PM, Tilman Schmidt t.schmidt@phoenixsoftware.de wrote:
Am 13.03.2012 19:46, schrieb m.roth@5-cent.us:
Markus Falb wrote:
On 12.3.2012 01:37, Mark LaPierre wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Not that frequently.
I beg to differ. Hard disk failures are by far the most frequent hardware problem I encounter at work.
Don't forget to scale that by the number of hard disks you have per motherboard - they are probably also your most common component... And in my experience those failures are clustered within the first few months or out about 5 years.
On 03/13/2012 05:23 PM, Les Mikesell wrote:
On Tue, Mar 13, 2012 at 7:05 PM, Tilman Schmidt t.schmidt@phoenixsoftware.de wrote:
Am 13.03.2012 19:46, schrieb m.roth@5-cent.us:
Markus Falb wrote:
On 12.3.2012 01:37, Mark LaPierre wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Not that frequently.
I beg to differ. Hard disk failures are by far the most frequent hardware problem I encounter at work.
Don't forget to scale that by the number of hard disks you have per motherboard - they are probably also your most common component... And in my experience those failures are clustered within the first few months or out about 5 years.
I would have to dig up some references, but I have read some articles that claim that the reliability of a drive that is in full time operation in a server, running 24hrs/day and maybe even seeking under heavy load is way different than a drive that you run for a day or two and then it sits in an environmentally controlled storage, powered down for most of its lifetime. At least from what I read, the failure rate is much lower for the same drive used under the later conditions.
Even so, I still choose multiple different backup format. But if long term archival is important, I think I would be doing some data refreshing after a few years of service from backup drives.
Nataraj
On 03/13/12 7:05 PM, Nataraj wrote:
I would have to dig up some references, but I have read some articles that claim that the reliability of a drive that is in full time operation in a server, running 24hrs/day and maybe even seeking under heavy load is way different than a drive that you run for a day or two and then it sits in an environmentally controlled storage, powered down for most of its lifetime. At least from what I read, the failure rate is much lower for the same drive used under the later conditions.
on the other hand, the vibration and shock of transport is more likely to make a drive fail, so its all a tradeoff.
On 03/13/2012 09:17 PM, John R Pierce wrote:
On 03/13/12 7:05 PM, Nataraj wrote:
I would have to dig up some references, but I have read some articles that claim that the reliability of a drive that is in full time operation in a server, running 24hrs/day and maybe even seeking under heavy load is way different than a drive that you run for a day or two and then it sits in an environmentally controlled storage, powered down for most of its lifetime. At least from what I read, the failure rate is much lower for the same drive used under the later conditions.
on the other hand, the vibration and shock of transport is more likely to make a drive fail, so its all a tradeoff.
You could take your chances on the dyes with optical media. Some say that in a proper controlled environment, they will last much longer. The best media I think are the ones from Japan and singapore. There are several places in Japan that now ship to the US for reasonable rates. I just ordered from 1 on ebay. I think the reality is that nothing lasts forever. Optical media is probably much more likely to survive ICBM's, but then you may not have a drive to read them...
Nataraj
On Tue, Mar 13, 2012 at 11:26 PM, Nataraj incoming-centos@rjl.com wrote:
You could take your chances on the dyes with optical media. Some say that in a proper controlled environment, they will last much longer. The best media I think are the ones from Japan and singapore. There are several places in Japan that now ship to the US for reasonable rates. I just ordered from 1 on ebay. I think the reality is that nothing lasts forever. Optical media is probably much more likely to survive ICBM's, but then you may not have a drive to read them...
And then there is the problem of indexing and finding old stuff, assuming you still have the device to read it. If you keep archiving at the rate you are now, how much will you accumulate in a lifetime and how long will it take to find something if you ever care about retrieving it? I just keep buying bigger and bigger disks...
Am 14.03.2012 03:05, schrieb Nataraj:
I would have to dig up some references, but I have read some articles that claim that the reliability of a drive that is in full time operation in a server, running 24hrs/day and maybe even seeking under heavy load is way different than a drive that you run for a day or two and then it sits in an environmentally controlled storage, powered down for most of its lifetime. At least from what I read, the failure rate is much lower for the same drive used under the later conditions.
OTOH I remember reports about drives failing to start after having been powered off for extended time periods. Something about heads sticking to platters or somesuch. Though I don't know if that information still applies to current drive technologies.
On Wed, Mar 14, 2012 at 6:27 PM, Tilman Schmidt t.schmidt@phoenixsoftware.de wrote:
Am 14.03.2012 03:05, schrieb Nataraj:
I would have to dig up some references, but I have read some articles that claim that the reliability of a drive that is in full time operation in a server, running 24hrs/day and maybe even seeking under heavy load is way different than a drive that you run for a day or two and then it sits in an environmentally controlled storage, powered down for most of its lifetime. At least from what I read, the failure rate is much lower for the same drive used under the later conditions.
OTOH I remember reports about drives failing to start after having been powered off for extended time periods. Something about heads sticking to platters or somesuch. Though I don't know if that information still applies to current drive technologies.
Some high-density tapes will fail if you drop them on the floor. I think we can all agree that any media type has the potential to fail, which is why we use multiple copies on different physical media, so if one fails you still have another one. If you are storing all of your backups on a single tape/disk/cd/dvd/bd/holocube, you are doing it wrong.
Is this horse dead yet?
❧ Brian Mathis
On Tue, Mar 13, 2012 at 8:05 PM, Tilman Schmidt t.schmidt@phoenixsoftware.de wrote:
Am 13.03.2012 19:46, schrieb m.roth@5-cent.us:
Markus Falb wrote:
On 12.3.2012 01:37, Mark LaPierre wrote:
Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify.
Harddisks have a bad reputation too. They fail regulary.
Not that frequently.
I beg to differ. Hard disk failures are by far the most frequent hardware problem I encounter at work. And those external USB drives people are so fond of for backup are certainly not better than typical server drives in that respect.
When a disk fails, you still have the other copy. That's why they call it a backup. Otherwise, keep more than one disk as your backup media and rotate them. Now you have 3 copies.
❧ Brian Mathis
What (FOSS) backup apps can back up a system running at level 3/5?
On Wed, Mar 14, 2012 at 10:10 AM, ken gebser@mousecar.com wrote:
What (FOSS) backup apps can back up a system running at level 3/5?
Almost all backup methods except raw partition/disk images will work with the system running. You aren't guaranteed that files will be in a consistent state when restored, but the OS itself is fairly sure to work and databases and similar apps usually have their own ways to do live consistent snapshots.
On 03/14/2012 08:43 AM, Les Mikesell wrote:
On Wed, Mar 14, 2012 at 10:10 AM, ken gebser@mousecar.com wrote:
What (FOSS) backup apps can back up a system running at level 3/5?
Almost all backup methods except raw partition/disk images will work with the system running. You aren't guaranteed that files will be in a consistent state when restored, but the OS itself is fairly sure to work and databases and similar apps usually have their own ways to do live consistent snapshots.
I find that LVM snapshots are useful to insure data integrity. for example, I backup my mysql databases by stopping the mysql server, taking an LVM snapshot and restarting it. The whole snapshot process probably takes less then 15 seconds. Then I backup the snapshot LVM and it doesn't matter how long it takes. You must make sure that your snapshot volume is large enough that you won't run out of space before deleting the snapshot.
Nataraj
On 03/14/2012 02:59 PM Nataraj wrote:
On 03/14/2012 08:43 AM, Les Mikesell wrote:
On Wed, Mar 14, 2012 at 10:10 AM, ken gebser@mousecar.com wrote:
What (FOSS) backup apps can back up a system running at level 3/5?
Almost all backup methods except raw partition/disk images will work with the system running. You aren't guaranteed that files will be in a consistent state when restored, but the OS itself is fairly sure to work and databases and similar apps usually have their own ways to do live consistent snapshots.
I find that LVM snapshots are useful to insure data integrity. for example, I backup my mysql databases by stopping the mysql server, taking an LVM snapshot and restarting it. The whole snapshot process probably takes less then 15 seconds. Then I backup the snapshot LVM and it doesn't matter how long it takes. You must make sure that your snapshot volume is large enough that you won't run out of space before deleting the snapshot.
Nataraj
Thanks for the tip. The details of how a snapshot works has long evaded me. For example, when files are open but unsaved, what is written to the snapshot...? And is the entire contents of a file recorded by the snapshot... or something else?
tia.
On Fri, Mar 16, 2012 at 1:19 PM, ken gebser@mousecar.com wrote:
I find that LVM snapshots are useful to insure data integrity. for example, I backup my mysql databases by stopping the mysql server, taking an LVM snapshot and restarting it. The whole snapshot process probably takes less then 15 seconds. Then I backup the snapshot LVM and it doesn't matter how long it takes. You must make sure that your snapshot volume is large enough that you won't run out of space before deleting the snapshot.
Thanks for the tip. The details of how a snapshot works has long evaded me. For example, when files are open but unsaved, what is written to the snapshot...? And is the entire contents of a file recorded by the snapshot... or something else?
LVM snapshots just present a view of the disk as it was when the snapshot was taken. To do that, whenever you write to the disk with a snapshot active, it first copies the old data into the snapshot. Blocks that don't change don't need to be copied. This doesn't do anything by itself to make the files consistent, but you might stop your apps momentarily, make the snapshot, then let them run again while the snapshot is being backed up.
On 03/16/12 11:53 AM, Les Mikesell wrote:
LVM snapshots just present a view of the disk as it was when the snapshot was taken. To do that, whenever you write to the disk with a snapshot active, it first copies the old data into the snapshot. Blocks that don't change don't need to be copied. This doesn't do anything by itself to make the files consistent, but you might stop your apps momentarily, make the snapshot, then let them run again while the snapshot is being backed up.
one thing Microsoft did well in Windows NT... the built in 'volume snapshot service' support, VSS... when you go to create one, the volume manager sends messages to all applications that have registered themselves with the VSS subsystem to quiesce their disk state before it creates the snapshot, then tells them to resume as soon as the snapshot is stable. this is mainly of interest to database servers...
On 16.3.2012 19:53, Les Mikesell wrote:
On Fri, Mar 16, 2012 at 1:19 PM, ken gebser-Qz+AbkVSZgBWk0Htik3J/w@public.gmane.org wrote:
I find that LVM snapshots are useful to insure data integrity. for example, I backup my mysql databases by stopping the mysql server, taking an LVM snapshot and restarting it. The whole snapshot process probably takes less then 15 seconds. Then I backup the snapshot LVM and it doesn't matter how long it takes. You must make sure that your snapshot volume is large enough that you won't run out of space before deleting the snapshot.
Thanks for the tip. The details of how a snapshot works has long evaded me. For example, when files are open but unsaved, what is written to the snapshot...? And is the entire contents of a file recorded by the snapshot... or something else?
LVM snapshots just present a view of the disk as it was when the snapshot was taken. To do that, whenever you write to the disk with a snapshot active, it first copies the old data into the snapshot.
Note that this results in degraded write performance while the snapshot is active. Basically you have to "double write", at least.
On 03/16/12 12:44 PM, Markus Falb wrote:
Note that this results in degraded write performance while the snapshot is active. Basically you have to "double write", at least.
and with LVM, an entire PP block has to be copied, right? those are generally much larger than file system blocks, 4MB to 16MB, I believe? thats kind of brutal.
On Mar 16, 2012, at 4:29 PM, John R Pierce pierce@hogranch.com wrote:
and with LVM, an entire PP block has to be copied, right? those are generally much larger than file system blocks, 4MB to 16MB, I believe? thats kind of brutal.
Actually the larger the block the less of a load. It moves more data in a single operation and if another update on that same block occurs it no further COW operation is needed.
The alternative would mean lots of small IO operations which would hit the disk's max IOPS faster then 4-16MB blocks would hit a drive's max throughput.
-Ross
On 03/16/2012 02:53 PM Les Mikesell wrote:
On Fri, Mar 16, 2012 at 1:19 PM, ken gebser@mousecar.com wrote:
I find that LVM snapshots are useful to insure data integrity. for example, I backup my mysql databases by stopping the mysql server, taking an LVM snapshot and restarting it. The whole snapshot process probably takes less then 15 seconds. Then I backup the snapshot LVM and it doesn't matter how long it takes. You must make sure that your snapshot volume is large enough that you won't run out of space before deleting the snapshot.
Thanks for the tip. The details of how a snapshot works has long evaded me. For example, when files are open but unsaved, what is written to the snapshot...? And is the entire contents of a file recorded by the snapshot... or something else?
LVM snapshots just present a view of the disk as it was when the snapshot was taken. To do that, whenever you write to the disk with a snapshot active, it first copies the old data into the snapshot. Blocks that don't change don't need to be copied. This doesn't do anything by itself to make the files consistent, but you might stop your apps momentarily, make the snapshot, then let them run again while the snapshot is being backed up.
Thanks, Les! That fleshes the process out a little more for me. But what does it mean that a snapshot is "active"?
On 03/16/12 4:10 PM, ken wrote:
Thanks, Les! That fleshes the process out a little more for me. But what does it mean that a snapshot is "active"?
means one exists. LVM maps logical volumes (LV) onto a set of PP (physical partitions, also called extents) that are maybe 4-16MB each (this is configurable when you create the partition). when you create a snapshot, nothing actually happens until the file system is written, the snapshot is just another view of the PPs assembled into a new LV thats pointing to the same blocks... when the original LV is written, the existing data is copied to new PP's grabbed from the free pool, and the snapshot is pointed at these saved copies, then the file system is allowed to write onto the original PPs. when you delete the snapshot these new PPs containing the original data are lost. if you revert the file system to the snapshot, then the PPs that were updated are deleted and the snapshot becomes the LV.
the implementation of this in Linux came from IBM's AIX Unix, where the Logical Volume Manager is quite nicely integrated with their JVM/JVM2 file systems (grow the file system and the LV's are expanded automatically).
On Sun, Mar 11, 2012 at 8:12 PM, Scott Walker Scott_Walker@ramsystemscorp.com wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
The cost of disks is so low, it's very hard to justify tape. Don't forget you also need to have someone swapping the tapes every day or week, or spend more for a robot. For the amount you would spend on those tapes, you can get many TBs of disk space.
In general it works very well to spend your money on disks and backup to multiple locations. With disk, you get so many benefits, such as random-access recovery, and most disk-based systems support some level of data deduplication. If you use something like rsync backups with hard links, there's also never a need for a full backup after the first one.
I'm sure you will be able to come up with a few arguments against using disk, and in some situations tape is better, but almost never for some little server somewhere. Once you start talking about long-term archives and stuff like that, then yes, tapes are good. Disks also need a different type of maintenance, such as running a full read/refresh of the data every so often. In the SAN world they call this "scrubbing", though don't confuse it with the 'scrub' command that securely wipes all data from the disk...
Some common disk-to-disk backup tools: - BackupPC - rdiff-backup - dirvish - Duplicity - Duplicati
An overview of using rsync for backups: http://www.mikerubel.org/computers/rsync_snapshots/
❧ Brian Mathis
________________________________ From: Brian Mathis brian.mathis+centos@betteradmin.com To: CentOS mailing list centos@centos.org Sent: Sunday, March 11, 2012 5:38 PM Subject: Re: [CentOS] CentOS Server Backup Options
On Sun, Mar 11, 2012 at 8:12 PM, Scott Walker Scott_Walker@ramsystemscorp.com wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
A relatively inexpensive solution is to use a system with removable SATA disks (for the backup media) and use an open source backup application called Bacula ( http://bacula.org ) I have a SuperMicro with 8x1TB SATA disks. I keep one for the OS and application, and swap out the other 7 every week.
Scott Walker <Scott_Walker@...> writes:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night.
. . .
I stumbled on http://storebackup.org/ the other day. It looks pretty good for disc-disc backup.
Myself, I use a daily backup with rsync creating hard links to the backup directory - both remote (offsite) and local.
http://bhepple.freeshell.org/oddmuse/wiki.cgi/backup-copy
and supporting stuff at:
Hi,
On Monday, March 12, 2012 you wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night.
I stumbled on http://storebackup.org/ the other day. It looks pretty good for disc-disc backup.
I am using storeBackup for three years now and LOVE it. I am backing up three servers to two independent backup servers and get a new backup every four hours. storeBackup uses hard links on the target disk, so you do not need much physical space on the backup servers. A backup of a 250GB server takes less than 15 minutes.
This is certainly no solution if you need an insurance against hardware failures, but it is a perfect backup to store densely older versions of the server content.
Having the servers and the backup servers on RAID6 and having storeBackup makes me sleep well.
best regards --- Michael Schumacher PAMAS Partikelmess- und Analysesysteme GmbH Dieselstr.10, D-71277 Rutesheim Tel +49-7152-99630 Fax +49-7152-996333 Geschäftsführer: Gerhard Schreck Handelsregister B Stuttgart HRB 252024
On Sun, Mar 11, 2012 at 7:12 PM, Scott Walker Scott_Walker@ramsystemscorp.com wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
I'd look at backuppc first - running on a different machine which can be fairly low-powered or one that does something else in the daytime. It pools multiple copies of identical content with hardlinks so you can keep much more history online than you would expect, and it provides a nice web interface for browsing backups and doing restores. And it basically takes care of itself.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
Tapes are really inconvenient compared to online backups and disks are so inexpensive these days that tape only makes sense for archiving.
There are three reasons backups are made.
1: Protect from hardware failure
2: Protection from user deletion and/or corruption of files whether accidental or otherwise. (Yes, there are people who will intentionally damage a filesystem.)
3: Fire or other natural disasters.
From your description below, you seem reasonably well protected against hardware failure. I would not run RAID5 with three disks though. Instead, I would run RAID1 with the third disk as a hot-standby. For the amount of data you are managing, 300GB is plenty of space for 25GB and CentOS.
Protection from filesystem damage requires a rolling archive such as can be created the dump program. Think about dump in a tower-of-hanoi level sequence, or similar. This archive should be maintained on a separate device. Running a crontab scheduled dump every night will preserve all files as they exist at the close of business each day.
Protection of data from natural disasters requires that off-site backups be made.
The old school way to address all above was tape rotation. As disks have become more affordable various forms of RAID have taken #1 off the list of reasons to do tape rotation.
These days other choices are available.
An archive tree still needs to be maintained. My view is that the tree should be maintained on a local disk. Creating an off-site backup is simply taking a snapshot of the dump tree and transporting it to a safe place.
The off-site issue is something you need to address with your client. I always make the client assume the responsibility for setting the frequency of off-site backups and of actually doing them.
As to the backup device, I would suggest a USB hard-drive. I happen to have a 64GB USB flash drive I use to transport data. Such a device or smaller, if rotated weekly could be the backup device. My recommendation would be to attach a USB hard-drive to the system and maintain the dump tree on it. I would use the USB thumb drive for a transport device. 25GB of data isn't very much.
I don't know if you and perl have made friends. Personally, I do most of my system specific scripting in Perl rather than shell code, but that's just me.
I have a perl program I use to do exactly what I have described above for a client who has about 100GB of data to protect.
This program builds a hierarchy of weekly folders. On the first work-week of each quarter, it will write a level 0 dump file. All the daily files are incremental from that dump. The drive the dump-tree is on has more than sufficient space to keep a year's worth of backups. Once a quarter the client takes a snapshot of the most recently completed quarter's tree.
If you (or any others on the list) would like the program I will send if. If several ask, I will just post it. It is only about 350 lines with some comments.
Ray
On 03/11/2012 05:12 PM, Scott Walker wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 3/11/2012 6:12 PM, Scott Walker wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year.
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like.
I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time.
I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them.
Any input would be welcomed.
I believe in tape... it's just not a viable option with the large disk sizes we have today unless you have a lot of money for a fast, multi-drive solution. I can backup a bit over 500GB daily in 3 hours to external disk. Using a single tape drive that would (and did) take far too long.
So today I use TB size drives dropped into an external docking station. The docking station plugs into the server using eSATA. Then it's a relatively simple script run by cron to handle the daily backup. I'm happy to share the script if you're interested but it has long lines that don't do well in email. I'll send it offline if you'd like to use it as an example. Buying multiple drives allows us to do media rotation just like we did with tape.
The big difference with disks is that I just do full backups each time. In our situation there is time for that and it saves a lot of grief when trying to restore something in particular. None of this running back thru the incrementals to get at what you want. Of course, with incremental backups the typical daily time would be much much shorter.
For backing up multiple production servers I have a backup server and a private GB network to each system. Each server runs a backup script at night (via cron) to backup to the backup server. Then we backup the backup server to the external disk during the day. At least one external disk is off site at any given time.
I'm aware of the fancy tools to do the job for you but I like the simplicity of our home grown solution. And the only thing I absolutely need for a restore is tar. No databases, no extra applications, just tar. The catch is that I'm not sure it would scale up to a huge number of servers gracefully.
It's nice to know what works for the other guy but you gotta look for what will work for you. Good luck.
//Steve
Steve Lindemann wrote:
On 3/11/2012 6:12 PM, Scott Walker wrote:
What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be
<snip>
The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we
<snip>
over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the
<snip>
Any input would be welcomed.
I believe in tape... it's just not a viable option with the large disk sizes we have today unless you have a lot of money for a fast, multi-drive solution. I can backup a bit over 500GB daily in 3 hours to external disk. Using a single tape drive that would (and did) take far too long.
So today I use TB size drives dropped into an external docking station. The docking station plugs into the server using eSATA. Then it's a relatively simple script run by cron to handle the daily backup. I'm
Yup. Our home directories (NFS mounted) are on 2TB (or are being moved to them) drives; and we have online nightly b/u's that way. The semiweekly offline b/u's are to 3TB drives, dropped into an eSATA bay. The eSATA bay is about an order of magnitude cheaper than your vendor's recommending, and the eSATA uses bare drives, not even needing sleds. *Much* cheaper and easier.
For that matter, if you have to restore from it, assuming you don't need everything, it's much faster and easier. <snip>
The big difference with disks is that I just do full backups each time.
We use rsync w/ hard links. <snip> mark
On 03/12/2012 12:37 PM, m.roth@5-cent.us wrote:
So today I use TB size drives dropped into an external docking station. The docking station plugs into the server using eSATA. Then it's a relatively simple script run by cron to handle the daily backup. I'm
Yup. Our home directories (NFS mounted) are on 2TB (or are being moved to them) drives; and we have online nightly b/u's that way. The semiweekly offline b/u's are to 3TB drives, dropped into an eSATA bay. The eSATA bay is about an order of magnitude cheaper than your vendor's recommending, and the eSATA uses bare drives, not even needing sleds. *Much* cheaper and easier.
What hardware are you using for docking stations? Do you use multiple drives per ESATA port? What is your ESATA controller?
I've been using Thermaltake ST0014U's for some time now with USB interfaces and I recently tried plugging them into the ESATA port (using onboard Intel controller/AHCI driver) of a Dell R210 running CentOS 6. It doesn't seem to work with the port multiplier and I can only use one of the two drive slots. Even if there aren't two drives, only one of the slots work. Both slots work with USB. I get the following errors from the driver:
ar 5 16:06:33 myserver kernel: ata6.15: Port Multiplier 1.1, 0x197b:0x2352 r0, 2 ports, feat 0x0/0x0 Mar 5 16:06:33 myserver kernel: ata6.15: Asynchronous notification not supported, hotplug won't Mar 5 16:06:33 myserver kernel: work on fan-out ports. Use warm-plug instead. Mar 5 16:06:33 myserver kernel: ata6.00: hard resetting link
Mar 5 16:06:33 myserver kernel: ata6.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320) Mar 5 16:06:33 myserver kernel: ata6.01: hard resetting link Mar 5 16:06:33 myserver kernel: ata6.15: qc timeout (cmd 0xe4) Mar 5 16:06:33 myserver kernel: ata6.01: failed to read SCR 2 (Emask=0x4) Mar 5 16:06:33 myserver kernel: ata6.01: failed to read SCR 2 (Emask=0x40) Mar 5 16:06:33 myserver kernel: ata6.01: COMRESET failed (errno=-5) Mar 5 16:06:33 myserver kernel: ata6.01: failed to read SCR 0 (Emask=0x40) Mar 5 16:06:33 myserver kernel: ata6.01: reset failed, giving up Mar 5 16:06:33 myserver kernel: ata6.15: hard resetting link Mar 5 16:06:33 myserver kernel: ata6.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 5 16:06:33 myserver kernel: ata6.00: hard resetting link Mar 5 16:06:33 myserver kernel: ata6.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320) Mar 5 16:06:33 myserver kernel: ata6.01: hard resetting link Mar 5 16:06:33 myserver kernel: ata6.01: SATA link down (SStatus 0 SControl 320) Mar 5 16:06:33 myserver kernel: ata6.00: qc timeout (cmd 0xec) Mar 5 16:06:33 myserver kernel: ata6.00: failed to IDENTIFY (I/O error, err_mask=0x4) Mar 5 16:06:33 myserver kernel: ata6.15: hard resetting link
I would ideally like to get several drives on a single ESATA controller (At least 4 would be nice, though I know it won't have amazing performance if I access multiple drives at once).
Nataraj
On Mon, Mar 12, 2012 at 6:22 PM, Nataraj incoming-centos@rjl.com wrote:
I would ideally like to get several drives on a single ESATA controller (At least 4 would be nice, though I know it won't have amazing performance if I access multiple drives at once).
If you have internal space there are trayless hotswap SATA bays that work pretty well, including some for 2.5" drives that you should be able to get up to 1TB now (larger ones are thicker than the bays will take).
On 03/12/2012 04:28 PM, Les Mikesell wrote:
On Mon, Mar 12, 2012 at 6:22 PM, Nataraj incoming-centos@rjl.com wrote:
I would ideally like to get several drives on a single ESATA controller (At least 4 would be nice, though I know it won't have amazing performance if I access multiple drives at once).
If you have internal space there are trayless hotswap SATA bays that work pretty well, including some for 2.5" drives that you should be able to get up to 1TB now (larger ones are thicker than the bays will take).
Unfortunately in this case I don't. This is my economical home server that one of my clients gave to me. I'm thinking about putting an NVIDIA card in the single PCI Express slot and using it for my desktop, so the only remaining interfaces are ESATA and USB.
Nataraj
On Mon, Mar 12, 2012 at 2:27 PM, Steve Lindemann steve@marmot.org wrote:
I believe in tape... it's just not a viable option with the large disk sizes we have today unless you have a lot of money for a fast, multi-drive solution. I can backup a bit over 500GB daily in 3 hours to external disk. Using a single tape drive that would (and did) take far too long.
I think both amanda and backula would use disk as temporary holding space. Amanda will compute the best mix of incrementals and fulls to fit on the tape and run the backups to disk in parallel, starting the tape write when it has one complete file.
The big difference with disks is that I just do full backups each time. In our situation there is time for that and it saves a lot of grief when trying to restore something in particular. None of this running back thru the incrementals to get at what you want. Of course, with incremental backups the typical daily time would be much much shorter.
Backuppc will let you do normal fulls and incremental backups, but will transparently merge them when doing a restore.
For backing up multiple production servers I have a backup server and a private GB network to each system. Each server runs a backup script at night (via cron) to backup to the backup server. Then we backup the backup server to the external disk during the day. At least one external disk is off site at any given time.
The one down side of backuppc is that due to the extensive use of hardlinks to de-dup the content, it can be impractical to use normal approaches to copy a large archive for offsite rotation. Some people rotate the whole thing, letting the next incremental run catch up with the differences, some just run another instance over the network from a different location (often practical with rsync as the transfer method), and some use an image-copy scheme to copy the whole filesystem or device.
I'm aware of the fancy tools to do the job for you but I like the simplicity of our home grown solution. And the only thing I absolutely need for a restore is tar. No databases, no extra applications, just tar. The catch is that I'm not sure it would scale up to a huge number of servers gracefully.
Backuppc has a command line tool that will generate a tar image out of its storage format, and a way to do that from the web interface. You can make them periodically for archive copies that could be re-installed without backuppc. But, backuppc itself is just a perl script using files in a linux file system (with some compression/linking tricks to hold about 10x what you would expect).
Over the years I have run into several situations where for one reason or another a backup utility such as dump or tar couldn't read a particular backup. For that reason, I like to periodically do a backup using another backup format. So I might use backuppc for my main backup system, but once a month do a full backup using dump onto a completely separate media.
I have been sucessfully using 8GB dual layer DVDs for some of my backups/archiving and now that the price of Blu ray has come down I am about to experiment with that. I have been writing dump format files to the DVD's and then writing an SHA256 checksum for each dump file so it's very easy to verify the integrity of the dump.
I am also about to try daily emcrypted backups to http://rsync.net along with periodic archival to blu-ray disk for one of my backup needs.
I have noticed that two of the recently mentioned backup packages, duplicity and storebackup appear to support some kind of block level deduplication where you can backup a large file, database or possibly even a disk partition, incrementally over the network. I am interested in trying that for backup up of mysql databases.
Nataraj
Am 13.03.2012 00:48, schrieb Nataraj:
I have been sucessfully using 8GB dual layer DVDs for some of my backups/archiving and now that the price of Blu ray has come down I am about to experiment with that. I have been writing dump format files to the DVD's and then writing an SHA256 checksum for each dump file so it's very easy to verify the integrity of the dump.
I am also about to try daily emcrypted backups to http://rsync.net along with periodic archival to blu-ray disk for one of my backup needs.
In my experience, the long-term stability of DVDs is rather questionable. I've had quite a few nasty surprises with DVDs. Even single-layer ones regularly turn out to be unreadable after two or three years, and double-layer ones are worse. I don't know if Blueray is any better in that respect.
Tilman Schmidt wrote:
Am 13.03.2012 00:48, schrieb Nataraj:
I have been sucessfully using 8GB dual layer DVDs for some of my backups/archiving and now that the price of Blu ray has come down I am about to experiment with that. I have been writing dump format files to the DVD's and then writing an SHA256 checksum for each dump file so it's very easy to verify the integrity of the dump.
I am also about to try daily emcrypted backups to http://rsync.net along
1++
with periodic archival to blu-ray disk for one of my backup needs.
In my experience, the long-term stability of DVDs is rather questionable. I've had quite a few nasty surprises with DVDs. Even single-layer ones regularly turn out to be unreadable after two or three years, and double-layer ones are worse. I don't know if Blueray is any better in that respect.
Yup. I've been reading about that instability for several years now: the commercially-produced ones are ok, but not the ones you write; they will *not* last the same number of years.
mark mark
On 03/13/2012 08:09 AM, m.roth@5-cent.us wrote:
Tilman Schmidt wrote:
Am 13.03.2012 00:48, schrieb Nataraj:
I have been sucessfully using 8GB dual layer DVDs for some of my backups/archiving and now that the price of Blu ray has come down I am about to experiment with that. I have been writing dump format files to the DVD's and then writing an SHA256 checksum for each dump file so it's very easy to verify the integrity of the dump.
I am also about to try daily emcrypted backups to http://rsync.net along
1++
with periodic archival to blu-ray disk for one of my backup needs.
In my experience, the long-term stability of DVDs is rather questionable. I've had quite a few nasty surprises with DVDs. Even single-layer ones regularly turn out to be unreadable after two or three years, and double-layer ones are worse. I don't know if Blueray is any better in that respect.
Yup. I've been reading about that instability for several years now: the commercially-produced ones are ok, but not the ones you write; they will *not* last the same number of years.
mark mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Good point. I've been following the recommendations in articles such as this http://adterrasperaspera.com/blog/2006/10/30/how-to-choose-cddvd-archival-me... using mostly the Taiyo Yuden and verbatim media where I could identify the country of origin and the dyes and so far I've done ok. A good reminder for me to check some of my back archives. I also have this same data stored on hard drives, so there is redundancy.
As some have pointed out, if you really need long term archival of data I think a good plan would include periodic testing and refresh of media or rewrite to new media.
Nataraj
-
Bob Hepple -
Brian Mathis -
John R Pierce -
Joseph Spenner -
ken -
Les Mikesell -
m.roth＠5-cent.us -
Mark LaPierre -
Markus Falb -
Michael Schumacher -
Nataraj -
Raymond Lillard -
Ross Walker -
Scott Walker -
Steve Lindemann -
Tilman Schmidt