Hey, Les,
Thanks for changing the subject to OT.
Les Mikesell wrote:
On Tue, Nov 5, 2013 at 1:28 PM, m.roth@5-cent.us wrote:
As I noted, we make sure rsync uses hard links... but we have a good
number of individual people and projects with who *each* have a good number of terabytes of data and generated data. Some of our 2TB drives are
over 90% full, and then there's the honkin' huge RAID, and at least one
14TB partition is over 9TB full....
If you have database dumps or big text files that aren't compressed,
backuppc could be a big win. I think it is the only thing that can keep a compressed copy on the server side and work directly with a stock rsync and uncompressed files on the target hosts (and it can cache the block-checksums so it doesn't have to uncompress and
recompute them every run). While it is 'just a perl script' it's not
quite what you expect from simple scripting...
We have a *bunch* of d/bs. Oracle. MySQL. Postgresql. All with about a week's dumps from every night, and then backups of them to the b/u servers. I can't imagine how they'd be a win - don't remember just off the top of my head if they're compressed or not.
A *lot* of our data is not huge text files - lots and lots of pure datafiles, output from things like Matlab, R, and some local programs, like the one for modeling protein folding.
mark