Hi list,
We've had an interesting NFS performance issue with the following setup. Clients : CentOS 5.5, kernel 2.6.18-194.el5 Server : Solaris 10 kernel Generic_142901-08
Reading a file (not cached) with dd : dd if=file1 of=/dev/null bs=1024k count=100 performance is around.. 700 kB/s !
/proc/mounts shows rsize has been negotiated to 1mB
If we force rsize to a smaller value, 32kB, the problem goes away and we're back around 70/80 mB/s read performance.
Wiresharks show something interesting in the first (slow) case: It seems that the NFS client is "waiting" between requests:
1473 11.756852 client.net nfs1.net TCP 868 > nfs [ACK] Seq=9345 1474 11.757888 nfs1.net client.net RPC Continuation 1475 11.797837 client.net nfs1.net TCP 868 > nfs [ACK] Seq=9345 1476 11.798877 nfs1.net client.net RPC Continuation [....]
Notice how the server replies within about 0,001s, but the client appears to be waiting 0,04s before asking for the next bit. (we captured the traffic from the client). This pattern repeats for the whole exchange
It's likely we missed something, can't figure it out.. has anyone seen this before ?
Cheers,
Alex
From: Alexandre Lecuyer alex.ranskis@gmail.com
We've had an interesting NFS performance issue with the following setup. Clients : CentOS 5.5, kernel 2.6.18-194.el5 Server : Solaris 10 kernel Generic_142901-08 /proc/mounts shows rsize has been negotiated to 1mB If we force rsize to a smaller value, 32kB, the problem goes away and we're back around 70/80 mB/s read performance.
Not sure but the nfs faq says to look at /usr/include/linux/nfsd/const.h
/* * Maximum blocksize supported by daemon currently at 32K */ #define NFSSVC_MAXBLKSIZE (32*1024)
JD
Alexandre Lecuyer wrote:
Hi list,
We've had an interesting NFS performance issue with the following setup. Clients : CentOS 5.5, kernel 2.6.18-194.el5 Server : Solaris 10 kernel Generic_142901-08
Reading a file (not cached) with dd : dd if=file1 of=/dev/null bs=1024k count=100 performance is around.. 700 kB/s !
/proc/mounts shows rsize has been negotiated to 1mB
If we force rsize to a smaller value, 32kB, the problem goes away and we're back around 70/80 mB/s read performance.
Wiresharks show something interesting in the first (slow) case: It seems that the NFS client is "waiting" between requests:
1473 11.756852 client.net nfs1.net TCP 868 > nfs [ACK] Seq=9345 1474 11.757888 nfs1.net client.net RPC Continuation 1475 11.797837 client.net nfs1.net TCP 868 > nfs [ACK] Seq=9345 1476 11.798877 nfs1.net client.net RPC Continuation [....]
Notice how the server replies within about 0,001s, but the client appears to be waiting 0,04s before asking for the next bit. (we captured the traffic from the client). This pattern repeats for the whole exchange
It's likely we missed something, can't figure it out.. has anyone seen this before ?
Cheers,
Alex _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Well, it's been a long time since I've done troubleshooting on large NFS networks, but here's an idea...
Are you seeing any kind of packet loss/retransmissions? Take a look at netstat -s. When I last did this work it was with NFS over udp, but I think retransmitted packets will cause more performance loss with large packet sizes. I used to find machines with broken ethernet interfaces that would cause these kinds of problems.
Nataraj
On Mon, Jun 21, 2010 at 4:38 PM, Nataraj incoming-centos@rjl.com wrote:
[...]
Well, it's been a long time since I've done troubleshooting on large NFS networks, but here's an idea...
Are you seeing any kind of packet loss/retransmissions? Take a look at netstat -s. When I last did this work it was with NFS over udp, but I think retransmitted packets will cause more performance loss with large packet sizes. I used to find machines with broken ethernet interfaces that would cause these kinds of problems.
Nataraj
Thanks guys for the feedback. I've done more tests : There are very very few retransmits (less than 0,01%) so I don't think that's what happening. The client still seems to be "waiting" for something between requests, very strange.
On some servers this behavior returned despite rsize being set to 32k, I had to set it to 8k to get reasonnable throughput. So there's definitly something fishy going on. This has been reported on over 20 machines, so I don't think it's faulty hardware we're seeing.
Any thoughts, ideas on how to debug this ?
Best,
Alex
Alex Still wrote:
On Mon, Jun 21, 2010 at 4:38 PM, Nataraj incoming-centos@rjl.com wrote:
[...]
Well, it's been a long time since I've done troubleshooting on large NFS networks, but here's an idea...
Are you seeing any kind of packet loss/retransmissions? Take a look at netstat -s. When I last did this work it was with NFS over udp, but I think retransmitted packets will cause more performance loss with large packet sizes. I used to find machines with broken ethernet interfaces that would cause these kinds of problems.
Nataraj
Thanks guys for the feedback. I've done more tests : There are very very few retransmits (less than 0,01%) so I don't think that's what happening. The client still seems to be "waiting" for something between requests, very strange.
On some servers this behavior returned despite rsize being set to 32k,vbhcs.org I had to set it to 8k to get reasonnable throughput. So there's definitly something fishy going on. This has been reported on over 20 machines, so I don't think it's faulty hardware we're seeing.
Any thoughts, ideas on how to debug this ?
Best,
I would run tcpdump or other network sniffer. I don't know what your network topology is, i.e. weather you have routers, complex switch configurations etc in your network, but look for things like out of order packets, duplicate packets tcp window sizes etc. If you have a large or complex network topology, you may want to sniff at both the server and client end. Even if your not an expert here, one thing you can do is to setup a simple isolated client server example (could even be with a virtual machine on your notebook or desktop) and sniff a properly working connection and compare that with the situation where you have problems.
Also make sure all your forward/reverse dns entries/service are running correctly.
Nataraj
On Jun 21, 2010, at 12:26 PM, Alex Still alex.ranskis@gmail.com wrote:
On Mon, Jun 21, 2010 at 4:38 PM, Nataraj incoming-centos@rjl.com wrote:
[...]
Well, it's been a long time since I've done troubleshooting on large NFS networks, but here's an idea...
Are you seeing any kind of packet loss/retransmissions? Take a look at netstat -s. When I last did this work it was with NFS over udp, but I think retransmitted packets will cause more performance loss with large packet sizes. I used to find machines with broken ethernet interfaces that would cause these kinds of problems.
Nataraj
Thanks guys for the feedback. I've done more tests : There are very very few retransmits (less than 0,01%) so I don't think that's what happening. The client still seems to be "waiting" for something between requests, very strange.
On some servers this behavior returned despite rsize being set to 32k, I had to set it to 8k to get reasonnable throughput. So there's definitly something fishy going on. This has been reported on over 20 machines, so I don't think it's faulty hardware we're seeing.
Any thoughts, ideas on how to debug this ?
Can you explain the network environment and the connectivity between the client and server some more.
-Ross
[...]
On some servers this behavior returned despite rsize being set to 32k, I had to set it to 8k to get reasonnable throughput. So there's definitly something fishy going on. This has been reported on over 20 machines, so I don't think it's faulty hardware we're seeing.
Any thoughts, ideas on how to debug this ?
Can you explain the network environment and the connectivity between the client and server some more.
Clients are blade servers. The blade chassis have integrated cisco switches, which are plugged to a cisco 6509. The NFS server is on another site 40km away, directly connected to another 6509. These datacenters are linked via DWDM. Latency between a client and the NFS server is about half a millisecond. Jumbo frames are enabled.
Blades have 1 Gb link The NFS server has multiple 1Gb links, used for different shares Neither are close to full utilization, maybe 100Mb/s of traffic and 20 000 paquets/s at the server end
Alex
On Tue, 2010-06-22 at 17:44 +0200, Alex Still wrote:
Clients are blade servers. The blade chassis have integrated cisco switches, which are plugged to a cisco 6509. The NFS server is on another site 40km away, directly connected to another 6509. These datacenters are linked via DWDM. Latency between a client and the NFS server is about half a millisecond. Jumbo frames are enabled.
Blades have 1 Gb link The NFS server has multiple 1Gb links, used for different shares Neither are close to full utilization, maybe 100Mb/s of traffic and 20 000 paquets/s at the server end
--- http://www.google.com/search?hl=en&client=firefox-a&hs=qKR&rls=o...
Your not the only one with the prob..
John
On Tue, Jun 22, 2010 at 6:16 PM, JohnS jses27@gmail.com wrote:
On Tue, 2010-06-22 at 17:44 +0200, Alex Still wrote:
Clients are blade servers. The blade chassis have integrated cisco switches, which are plugged to a cisco 6509. The NFS server is on another site 40km away, directly connected to another 6509. These datacenters are linked via DWDM. Latency between a client and the NFS server is about half a millisecond. Jumbo frames are enabled.
Blades have 1 Gb link The NFS server has multiple 1Gb links, used for different shares Neither are close to full utilization, maybe 100Mb/s of traffic and 20 000 paquets/s at the server end
http://www.google.com/search?hl=en&client=firefox-a&hs=qKR&rls=o...
Your not the only one with the prob..
Not sure it has to do with the 6509, we're doing TCP, and the IOS is up to date. Also as mentionned an older distribution/kernel on the same hardware will not have the problem. If all else fails I'll see if we can run tests with a crossover cable.. :-)
On Jun 22, 2010, at 11:44 AM, Alex Still alex.ranskis@gmail.com wrote:
[...]
On some servers this behavior returned despite rsize being set to 32k, I had to set it to 8k to get reasonnable throughput. So there's definitly something fishy going on. This has been reported on over 20 machines, so I don't think it's faulty hardware we're seeing.
Any thoughts, ideas on how to debug this ?
Can you explain the network environment and the connectivity between the client and server some more.
Clients are blade servers. The blade chassis have integrated cisco switches, which are plugged to a cisco 6509. The NFS server is on another site 40km away, directly connected to another 6509. These datacenters are linked via DWDM. Latency between a client and the NFS server is about half a millisecond. Jumbo frames are enabled.
Blades have 1 Gb link The NFS server has multiple 1Gb links, used for different shares Neither are close to full utilization, maybe 100Mb/s of traffic and 20 000 paquets/s at the server end
I have seen non-standard jumbo frames cause problems in the past.
Can you try unmounting shares on one client, setting the MTU to 1500, re-mount the shares and see how it works?
TCP between server and client will negotiate down to client's MSS so no need to change server's MTU.
-Ross
On Tue, Jun 22, 2010 at 6:40 PM, Ross Walker rswwalker@gmail.com wrote:
On Jun 22, 2010, at 11:44 AM, Alex Still alex.ranskis@gmail.com wrote:
[...]
On some servers this behavior returned despite rsize being set to 32k, I had to set it to 8k to get reasonnable throughput. So there's definitly something fishy going on. This has been reported on over 20 machines, so I don't think it's faulty hardware we're seeing.
Any thoughts, ideas on how to debug this ?
Can you explain the network environment and the connectivity between the client and server some more.
Clients are blade servers. The blade chassis have integrated cisco switches, which are plugged to a cisco 6509. The NFS server is on another site 40km away, directly connected to another 6509. These datacenters are linked via DWDM. Latency between a client and the NFS server is about half a millisecond. Jumbo frames are enabled.
Blades have 1 Gb link The NFS server has multiple 1Gb links, used for different shares Neither are close to full utilization, maybe 100Mb/s of traffic and 20 000 paquets/s at the server end
I have seen non-standard jumbo frames cause problems in the past.
Can you try unmounting shares on one client, setting the MTU to 1500, re-mount the shares and see how it works?
TCP between server and client will negotiate down to client's MSS so no need to change server's MTU.
It doesn't seem to help. But we missed something during the first tests.. there are indeed some retransmits at the TCP level, on the server side. These go away when we set rsize to 32k. So ,probably a network issue we'll have to figure out.
Thanks for your feedback,
Alex
Alexandre Lecuyer wrote:
Hi list,
We've had an interesting NFS performance issue with the following setup. Clients : CentOS 5.5, kernel 2.6.18-194.el5 Server : Solaris 10 kernel Generic_142901-08
Reading a file (not cached) with dd : dd if=file1 of=/dev/null bs=1024k count=100 performance is around.. 700 kB/s !
/proc/mounts shows rsize has been negotiated to 1mB
Have you tested the same thing with a Linux NFS server?
The CentOS 5.x kernel has a maximum server [rw]size of 32Kb, so you would need to use something with a more recent kernel to get [rw]sizes to be 1Mb.
James Pearson
[..]
/proc/mounts shows rsize has been negotiated to 1mB
Have you tested the same thing with a Linux NFS server?
The CentOS 5.x kernel has a maximum server [rw]size of 32Kb, so you would need to use something with a more recent kernel to get [rw]sizes to be 1Mb.
Haven't tried with a linux server, will do. One thing tho, we are still mostly running old Fedora Core 6 installs (migrating to CentOS), which also negotiated rsize to 1mB with the same NFS server, and they don't have the problem :
$ dd if=toto of=/dev/null bs=1024k 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.44039 seconds, 72.8 MB/s
$ cat /etc/redhat-release Fedora Core release 6 (Zod)
tail -1 /proc/mounts nfs1:/local/users /users nfs rw,vers=3,rsize=1048576,wsize=1048576,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=nfs1 0 0
The setup and hardware is strictly identical to the CentOS environment. (we've installed both distros on the same servers, just to be sure).
Network traces don't show anything special, no fragmentation or delays. The only odd thing as I mentionned is that it's the client which is waiting a long time between requests. Wonder what's happening there. Well I'll try with a linux NFS server see if it changes something, will post here if we figure it out
Regards,
Alex
-
Alex Still -
Alexandre Lecuyer
-
James Pearson -
John Doe -
JohnS -
Nataraj -
Ross Walker