[CentOS] NFS performance - default rsize

Mon Jun 21 14:38:14 UTC 2010
Nataraj <incoming-centos at rjl.com>

Alexandre Lecuyer wrote:
> Hi list,
>
> We've had an interesting NFS performance issue with the following setup.
> Clients : CentOS 5.5, kernel 2.6.18-194.el5
> Server : Solaris 10 kernel Generic_142901-08
>
> Reading a file (not cached) with dd : dd if=file1 of=/dev/null
> bs=1024k count=100
> performance is around.. 700 kB/s !
>
> /proc/mounts shows rsize has been negotiated to 1mB
>
> If we force rsize to a smaller value, 32kB, the problem goes away and
> we're back around 70/80 mB/s read performance.
>
> Wiresharks show something interesting in the first (slow) case:
> It seems that the NFS client is "waiting" between requests:
>
>
>   1473 11.756852   client.net  nfs1.net    TCP      868 > nfs [ACK] Seq=9345
>   1474 11.757888   nfs1.net    client.net  RPC      Continuation
>   1475 11.797837   client.net  nfs1.net    TCP      868 > nfs [ACK] Seq=9345
>   1476 11.798877   nfs1.net    client.net  RPC      Continuation
> [....]
>
> Notice how the server replies within about 0,001s, but the client
> appears to be waiting 0,04s before asking for the next bit.
> (we captured the traffic from the client). This pattern repeats for
> the whole exchange
>
> It's likely we missed something, can't figure it out.. has anyone seen
> this before ?
>
> Cheers,
>
> Alex
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>   
Well, it's been a long time since I've done troubleshooting on large NFS 
networks, but here's an idea...

Are you seeing any kind of packet loss/retransmissions?  Take a look at 
netstat  -s.  When I last did this work it was with NFS over udp, but I 
think retransmitted packets will cause more performance loss with large 
packet sizes.  I used to find machines with broken ethernet interfaces 
that would cause these kinds of problems.

Nataraj