[CentOS] NFS performance - default rsize

Mon Jun 21 13:47:07 UTC 2010
Alexandre Lecuyer <alex.ranskis at gmail.com>

Hi list,

We've had an interesting NFS performance issue with the following setup.
Clients : CentOS 5.5, kernel 2.6.18-194.el5
Server : Solaris 10 kernel Generic_142901-08

Reading a file (not cached) with dd : dd if=file1 of=/dev/null
bs=1024k count=100
performance is around.. 700 kB/s !

/proc/mounts shows rsize has been negotiated to 1mB

If we force rsize to a smaller value, 32kB, the problem goes away and
we're back around 70/80 mB/s read performance.

Wiresharks show something interesting in the first (slow) case:
It seems that the NFS client is "waiting" between requests:


  1473 11.756852   client.net  nfs1.net    TCP      868 > nfs [ACK] Seq=9345
  1474 11.757888   nfs1.net    client.net  RPC      Continuation
  1475 11.797837   client.net  nfs1.net    TCP      868 > nfs [ACK] Seq=9345
  1476 11.798877   nfs1.net    client.net  RPC      Continuation
[....]

Notice how the server replies within about 0,001s, but the client
appears to be waiting 0,04s before asking for the next bit.
(we captured the traffic from the client). This pattern repeats for
the whole exchange

It's likely we missed something, can't figure it out.. has anyone seen
this before ?

Cheers,

Alex