On 04.12.2013 14:05, John Doe wrote:
From: Listslists@benjamindsmith.com
Our next big test is to try out ZFS filesystem send/receive in lieu of our current backup processes based on rsync. Rsync is a fabulous tool, but is beginning to show performance/scalability issues dealing with the many millions of files being backed up, and we're hoping that ZFS filesystem replication solves this.
Not sure if I already mentioned it but maybe have a look at: ?http://code.google.com/p/lsyncd/
I'm not so sure inotify works well with millions of files, not to mention it uses rsync. :D
-- Sent from the Delta quadrant using Borg technology! Nux!
I can attest to the usefulness of 'lsyncd' for large numbers of files (our file server has almost 2 million in active use, with a second backup server that's lsync'd to the first.
Things to note: - Yes, lsyncd does use rsync, but it issues an 'exclude *' followed by the list of only the file(s) that need updating at that moment.
- The inotify service can be jacked waaaay up (three kernel parameters) to handle millions of files if you wish. Just make sure you have lots of RAM. It's wise to tune ZFS to *not* use all available RAM.
- Updating is very quick and has never failed.
Regarding ZFS, our two ZFS-on-Linux servers have been in full production for several months, with zero problems so far. Updates to the latest version is quite painless.
Today I had to replace a failed 4TB drive ... it took just a few minutes and required only two commands to do the replacement and start the resilvering process. This was done while the server was in active use, with only a small performance hit. Sweet!
Chuck
On 05.12.2013 22:46, Chuck Munro wrote:
On 04.12.2013 14:05, John Doe wrote:
From: Listslists@benjamindsmith.com
Our next big test is to try out ZFS filesystem send/receive in lieu of our current backup processes based on rsync. Rsync is a fabulous tool, but is beginning to show performance/scalability issues dealing with the many millions of files being backed up, and we're hoping that ZFS filesystem replication solves this.
Not sure if I already mentioned it but maybe have a look at: ?http://code.google.com/p/lsyncd/
I'm not so sure inotify works well with millions of files, not to mention it uses rsync. :D
-- Sent from the Delta quadrant using Borg technology! Nux!
I can attest to the usefulness of 'lsyncd' for large numbers of files (our file server has almost 2 million in active use, with a second backup server that's lsync'd to the first.
Things to note:
- Yes, lsyncd does use rsync, but it issues an 'exclude *' followed by
the list of only the file(s) that need updating at that moment.
- The inotify service can be jacked waaaay up (three kernel
parameters) to handle millions of files if you wish. Just make sure you have lots of RAM.
Be careful with it. Sadly I found out that inotify would consistently fail on InnoDB files (ibd); I had to use stupid while loops and check mtimes to perform some stuff that inotify-cron would've done much more elegantly ...
-
Chuck Munro -
Nux!