[CentOS] connection speeds between nodes

[John Hodrien]

On Mon, 7 Mar 2011, Ross Walker wrote:

> 1Gbe can do 115MB/s @ 64K+ IO size, but at 4k IO size (NFS) 55MB/s is about
> it.
>
> If you need each node to be able to read 90-100MB/s you would need to setup
> a cluster file system using iSCSI or FC and make sure the cluster file
> system can handle large block/cluster sizes like 64K or the application can
> handle large IOs and the scheduler does a good job of coalescing these (VFS
> layer breaks it into 4k chunks) into large IOs.
>
> It's the latency of each small IO that is killing you.

I'm not necessarily convinced it's quite that bad (here's some default NFSv3
mounts under CentOS 5.5, with Jumbo frames, rsize=32768,wsize=32768).

$ sync;time (dd if=/dev/zero of=testfile bs=1M count=10000;sync)
[I verified that it'd finished when it thought it had]
10485760000 bytes (10 GB) copied, 133.06 seconds, 78.8 MB/s

umount, mount (to clear any cache):

$ dd if=testfile of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 109.638 seconds, 95.6 MB/s

This machine only has a double-bonded gig interface so with four clients all
hammering at the same time, this gives:

$ dd if=/scratch/testfile of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 189.64 seconds, 55.3 MB/s

So with four clients (on single gig) and one server with two gig interfaces
you're getting an aggregate rate of 220Mbytes/sec.  Sounds pretty reasonable
to me!

If you want safe writes (sync), *then* latency kills you.

jh