On Mon, 7 Mar 2011, Ross Walker wrote:
1Gbe can do 115MB/s @ 64K+ IO size, but at 4k IO size (NFS) 55MB/s is about it.
If you need each node to be able to read 90-100MB/s you would need to setup a cluster file system using iSCSI or FC and make sure the cluster file system can handle large block/cluster sizes like 64K or the application can handle large IOs and the scheduler does a good job of coalescing these (VFS layer breaks it into 4k chunks) into large IOs.
It's the latency of each small IO that is killing you.
I'm not necessarily convinced it's quite that bad (here's some default NFSv3 mounts under CentOS 5.5, with Jumbo frames, rsize=32768,wsize=32768).
$ sync;time (dd if=/dev/zero of=testfile bs=1M count=10000;sync) [I verified that it'd finished when it thought it had] 10485760000 bytes (10 GB) copied, 133.06 seconds, 78.8 MB/s
umount, mount (to clear any cache):
$ dd if=testfile of=/dev/null bs=1M 10485760000 bytes (10 GB) copied, 109.638 seconds, 95.6 MB/s
This machine only has a double-bonded gig interface so with four clients all hammering at the same time, this gives:
$ dd if=/scratch/testfile of=/dev/null bs=1M 10485760000 bytes (10 GB) copied, 189.64 seconds, 55.3 MB/s
So with four clients (on single gig) and one server with two gig interfaces you're getting an aggregate rate of 220Mbytes/sec. Sounds pretty reasonable to me!
If you want safe writes (sync), *then* latency kills you.
jh