Mag Gam wrote:
> I need to copy over 100TB of data from one server to another via network.
> What is the best option to do this? I am planning to use rsync but is there
> a better tool or better way of doing this?
>
> For example, I plan on doing
> rsync -azv /largefs /targetfs
>
> /targetfs is a NFS mounted filesystem.
The only problem you are likely to have is that rsync reads the entire
directory contents into RAM before starting, then walks the list fixing
the differences. If you have a huge number of files and a small amount
of RAM, it may slow down due to swapping. 'cp -a ' can be faster if the
target doesn't already have any matching files. Also, the -v to display
the names can take longer than the file transfer on small files.
Running rsync over ssh instead of nfs has a tradeoff in that the remote
does part of the work but you lose some speed to ssh encryption. If the
filesystem is live, you might make an initial run copying the larger
directories with rsync or cp, then do whatever you can to stop the files
from changing and make another pass with 'rsync --av --delete' which
should go fairly quickly and and fix any remaining differences.
--
Les Mikesell
lesmikesell at gmail.com