Mag Gam wrote:
I need to copy over 100TB of data from one server to another via network. What is the best option to do this? I am planning to use rsync but is there a better tool or better way of doing this?
For example, I plan on doing rsync -azv /largefs /targetfs
/targetfs is a NFS mounted filesystem.
The only problem you are likely to have is that rsync reads the entire directory contents into RAM before starting, then walks the list fixing the differences. If you have a huge number of files and a small amount of RAM, it may slow down due to swapping. 'cp -a ' can be faster if the target doesn't already have any matching files. Also, the -v to display the names can take longer than the file transfer on small files. Running rsync over ssh instead of nfs has a tradeoff in that the remote does part of the work but you lose some speed to ssh encryption. If the filesystem is live, you might make an initial run copying the larger directories with rsync or cp, then do whatever you can to stop the files from changing and make another pass with 'rsync --av --delete' which should go fairly quickly and and fix any remaining differences.