On Mon, 7 Mar 2011, Ross Walker wrote: > 1Gbe can do 115MB/s @ 64K+ IO size, but at 4k IO size (NFS) 55MB/s is about > it. > > If you need each node to be able to read 90-100MB/s you would need to setup > a cluster file system using iSCSI or FC and make sure the cluster file > system can handle large block/cluster sizes like 64K or the application can > handle large IOs and the scheduler does a good job of coalescing these (VFS > layer breaks it into 4k chunks) into large IOs. > > It's the latency of each small IO that is killing you. I'm not necessarily convinced it's quite that bad (here's some default NFSv3 mounts under CentOS 5.5, with Jumbo frames, rsize=32768,wsize=32768). $ sync;time (dd if=/dev/zero of=testfile bs=1M count=10000;sync) [I verified that it'd finished when it thought it had] 10485760000 bytes (10 GB) copied, 133.06 seconds, 78.8 MB/s umount, mount (to clear any cache): $ dd if=testfile of=/dev/null bs=1M 10485760000 bytes (10 GB) copied, 109.638 seconds, 95.6 MB/s This machine only has a double-bonded gig interface so with four clients all hammering at the same time, this gives: $ dd if=/scratch/testfile of=/dev/null bs=1M 10485760000 bytes (10 GB) copied, 189.64 seconds, 55.3 MB/s So with four clients (on single gig) and one server with two gig interfaces you're getting an aggregate rate of 220Mbytes/sec. Sounds pretty reasonable to me! If you want safe writes (sync), *then* latency kills you. jh