[CentOS] slow usb hard disk performance.

[Les Mikesell]

On Mon, 2005-12-05 at 13:27, Bryan J. Smith wrote:

> > My favorite for online disk backups:
> > http://backuppc.sourceforge.net/


> There is nothing more chronically stupid than direct backup
> to tape -- especially over a network.  I don't know how many
> times I go into companies and they are still trying to do
> end-node to tape-server backups in an 8 hour window and just
> can't do it.  I always just hit my head and say, "why don't
> you sync your filesystem changes to a centralized server, and
> then backup from that system locally to the tape -- which you
> can do in the middle of the day?!"
> 
> It's like an epiphany for them -- so obvious they missed it.
>
> You should at least buffer to disk first, if not just store
> backups whole on disk.

Amanda has done that for at least 15 years now.  It's clunky
to restore, but they got the backup side right from the beginning.
You can tell it how much bandwidth to use on your network(s)
and it will stream that much into the holding disk simultaneously
from many different hosts, writing to tape in sequence as they
are completely received.

Since it is almost full-auto - I still let amanda run tapes to
be held offsite but I really don't ever want to use them except
as a last resort, hence the offsite backuppc disk.

> > The downside has been firewire support on Linux the way I'm
> > trying to do it, which is to periodically sync a RAID1
> mirror
> > and break it for offsite storage.
> 
> I'd rather not backup in a filesystem format.  I'd rather
> backup to a streaming archive format which is far more
> recoverable when there are errors.  That way errors are
> localized and don't affect the rest of the backup. 
> Filesystems rely on meta-data to recover data -- streaming
> archive formats do not.

My backuppc archive probably has at least a million hardlinks
and conventional copy mechanisms take longer than practical.
It's about 2 hours to sync the 250 gig partiton to it's
raid mirror.  About 120 gigs are used but it would take
700 or more gigs to hold it any other way and equivalent
time to copy.  I do cycle through several of the offsite
drives so a single disk failure won't wipe out my whole
history.

> As far as off-line storage, I do not trust commodity fixed
> disk more than 3 months on the shelf (if even that long). 
> They are not designed to be stored that long and inactive
> after periodic activity.  The vendors have come up with
> removable rigid disk (RRD) to address the longevity issues,
> but then RRD loses the performance or cost of fixed disk.
> 
> > I suppose I could unmount the internal disk and use dd to
> > copy for the same effect.
> 
> Why not a streaming backup format like afio or ustar?  Far
> more recoverable and errors are localized than with a
> traditional filesystem.

Since the raid sync re-writes the whole disk, I don't accumulate
errors on the removable drive like I might with one that
is periodically connected for normal additions.  In practice I've
had more trouble with the on-line IDE side of the mirror and had
to run full-day fscks or even sync the external copy back to
a new drive after one failed.

-- 
  Les Mikesell
    lesmikesell at gmail.com