[CentOS] recommendations for copying large filesystems

[Les Mikesell]

Mag Gam wrote:
> I need to copy over 100TB of data from one server to another via network.
> What is the best option to do this? I am planning to use rsync but is there
> a better tool or better way of doing this?
> 
> For example, I plan on doing
> rsync -azv /largefs /targetfs
> 
> /targetfs is a NFS mounted filesystem.

The only problem you are likely to have is that rsync reads the entire 
directory contents into RAM before starting, then walks the list fixing 
the differences.  If you have a huge number of files and a small amount 
of RAM, it may slow down due to swapping.  'cp -a ' can be faster if the 
target doesn't already have any matching files.  Also, the -v to display 
the names can take longer than the file transfer on small files. 
Running rsync over ssh instead of nfs has a tradeoff in that the remote 
does part of the work but you lose some speed to ssh encryption. If the 
filesystem is live, you might make an initial run copying the larger 
directories with rsync or cp, then do whatever you can to stop the files 
from changing and make another pass with 'rsync --av --delete' which 
should go fairly quickly and and fix any remaining differences.

-- 
   Les Mikesell
     lesmikesell at gmail.com