> On 04.12.2013 14:05, John Doe wrote: >> >From: Lists<lists at benjamindsmith.com> >> > >>> >>Our next big test is to try out ZFS filesystem send/receive in lieu >>> >>of >>> >>our current backup processes based on rsync. Rsync is a fabulous >>> >>tool, >>> >>but is beginning to show performance/scalability issues dealing with >>> >>the >>> >>many millions of files being backed up, and we're hoping that ZFS >>> >>filesystem replication solves this. >> > >> >Not sure if I already mentioned it but maybe have a look at: >> >?http://code.google.com/p/lsyncd/ > I'm not so sure inotify works well with millions of files, not to > mention it uses rsync. :D > > -- Sent from the Delta quadrant using Borg technology! Nux! I can attest to the usefulness of 'lsyncd' for large numbers of files (our file server has almost 2 million in active use, with a second backup server that's lsync'd to the first. Things to note: - Yes, lsyncd does use rsync, but it issues an 'exclude *' followed by the list of only the file(s) that need updating at that moment. - The inotify service can be jacked waaaay up (three kernel parameters) to handle millions of files if you wish. Just make sure you have lots of RAM. It's wise to tune ZFS to *not* use all available RAM. - Updating is very quick and has never failed. Regarding ZFS, our two ZFS-on-Linux servers have been in full production for several months, with zero problems so far. Updates to the latest version is quite painless. Today I had to replace a failed 4TB drive ... it took just a few minutes and required only two commands to do the replacement and start the resilvering process. This was done while the server was in active use, with only a small performance hit. Sweet! Chuck