On Mar 7, 2011, at 9:55 AM, John Hodrien <J.H.Hodrien at leeds.ac.uk> wrote: > On Mon, 7 Mar 2011, Ross Walker wrote: > >> 1Gbe can do 115MB/s @ 64K+ IO size, but at 4k IO size (NFS) 55MB/s is about >> it. >> >> If you need each node to be able to read 90-100MB/s you would need to setup >> a cluster file system using iSCSI or FC and make sure the cluster file >> system can handle large block/cluster sizes like 64K or the application can >> handle large IOs and the scheduler does a good job of coalescing these (VFS >> layer breaks it into 4k chunks) into large IOs. >> >> It's the latency of each small IO that is killing you. > > I'm not necessarily convinced it's quite that bad (here's some default NFSv3 > mounts under CentOS 5.5, with Jumbo frames, rsize=32768,wsize=32768). > > $ sync;time (dd if=/dev/zero of=testfile bs=1M count=10000;sync) > [I verified that it'd finished when it thought it had] > 10485760000 bytes (10 GB) copied, 133.06 seconds, 78.8 MB/s > > umount, mount (to clear any cache): > > $ dd if=testfile of=/dev/null bs=1M > 10485760000 bytes (10 GB) copied, 109.638 seconds, 95.6 MB/s > > This machine only has a double-bonded gig interface so with four clients all > hammering at the same time, this gives: > > $ dd if=/scratch/testfile of=/dev/null bs=1M > 10485760000 bytes (10 GB) copied, 189.64 seconds, 55.3 MB/s > > So with four clients (on single gig) and one server with two gig interfaces > you're getting an aggregate rate of 220Mbytes/sec. Sounds pretty reasonable > to me! > > If you want safe writes (sync), *then* latency kills you. The OP wanted 90MB/s per node and we have no clue whether the application he is using is capable of driving 1MB block sizes. Why wouldn't you want safe writes? Is that like saying, and if you care for your data? -Ross