NFS problem in the latest kernel (Was: [CentOS] MySQL issues with kernel-2.6.18-53.1.6.el5.x86_64.rpm)
On Jan 30, 2008 8:25 AM, Johnny Hughes johnny@centos.org wrote:
Joshua Baker-LePain wrote:
On Wed, 30 Jan 2008 at 10:18am, Johnny Hughes wrote
Bent Terp wrote:
Has something changed with regard to the mount options? We use (rw,noatime,rsize=8192,wsize=8192,hard,udp,context="system_u:object_r:httpd_sys_content_t:s0)
which has worked fine until now.
I am trying to duplicate your options ... and noatime is not a valid option.
Could you please double check the /etc/export options again so I can try to duplicate the issue.
Using my standard /etc/exports on 2 i686 test platforms I have no problems at all.
Here are the options I used on my test:
(rw,insecure,sync,no_subtree_check)
Those are NFS export options. The OP's list is *mount* options (i.e. on the client side). He stated that his NFS server is actually an EMC Cellera.
AH ... now I see.
In any event, I can not duplicate the problem with an nfs export on c4 or c5 and connecting with a c5 client, regardless of the kernel using i686.
One other person has reported seemingly the same nfs problem in the Scientific Linux mail list:
http://listserv.fnal.gov/scripts/wa.exe?A2=ind0801&L=scientific-linux-de...
According to this post, "It only seems to affect x86_64 systems, or affects them much more noticeably than it does i386 ones."
Akemi
On Jan 30, 2008 4:01 PM, Akemi Yagi amyagi@gmail.com wrote:
On Jan 30, 2008 8:25 AM, Johnny Hughes johnny@centos.org wrote:
Joshua Baker-LePain wrote:
On Wed, 30 Jan 2008 at 10:18am, Johnny Hughes wrote
Bent Terp wrote:
Has something changed with regard to the mount options? We use (rw,noatime,rsize=8192,wsize=8192,hard,udp,context="system_u:object_r:httpd_sys_content_t:s0)
which has worked fine until now.
I am trying to duplicate your options ... and noatime is not a valid option.
Could you please double check the /etc/export options again so I can try to duplicate the issue.
Using my standard /etc/exports on 2 i686 test platforms I have no problems at all.
Here are the options I used on my test:
(rw,insecure,sync,no_subtree_check)
In any event, I can not duplicate the problem with an nfs export on c4 or c5 and connecting with a c5 client, regardless of the kernel using i686.
One other person has reported seemingly the same nfs problem in the Scientific Linux mail list:
http://listserv.fnal.gov/scripts/wa.exe?A2=ind0801&L=scientific-linux-de...
According to this post, "It only seems to affect x86_64 systems, or affects them much more noticeably than it does i386 ones."
This problem does not seem to be associated with specific hardware as implied in the original thread. The person on the SciLinux mail list has Dell machines, and I was able to confirm the issue using custom built boxes. I have added a note in the bug tracker at:
http://bugs.centos.org/view.php?id=2635
Akemi
Akemi Yagi wrote:
This problem does not seem to be associated with specific hardware as implied in the original thread. The person on the SciLinux mail list has Dell machines, and I was able to confirm the issue using custom built boxes. I have added a note in the bug tracker at:
Unrelated, but still kind of related, I'm only replying because of how crazy I went trying to track it down. But I had major NFS client issues on CentOS 4.5, specifically kernels after 2.6.9-55.ELsmp. Even under pretty light load, after seemingly random periods of time NFS write performance would go down 99.99999%. Read performance was fine. Downgrade the kernel and it worked fine again. Eventually for servers that ran our critical apps for NFS I just upgraded them to CentOS 5.1 and there is no problems there for us at least.
Spent several weeks researching the problem but never found a solution, since it took so long to reproduce(up to 24 hours typically), testing different configurations was very time consuming.
And no I haven't filed any bug reports or anything as I wanted to try to narrow it down further to find the source before that. But I ran out of time, and the kernel in CentOS 5.1 seems to work fine. (migrated production systems to it about 6 weeks ago)
The newer kernel didn't/does not affect the NFS servers at all, as they are all running CentOS 4.5 with 2.6.9-55.0.9.ELsmp, it's purely a client-side issue.
(sorry, end rant :) )
nate
On Jan 31, 2008 10:28 AM, nate centos@linuxpowered.net wrote:
Akemi Yagi wrote:
This problem does not seem to be associated with specific hardware as implied in the original thread. The person on the SciLinux mail list has Dell machines, and I was able to confirm the issue using custom built boxes. I have added a note in the bug tracker at:
And no I haven't filed any bug reports or anything as I wanted to try to narrow it down further to find the source before that. But I ran out of time, and the kernel in CentOS 5.1 seems to work fine. (migrated production systems to it about 6 weeks ago)
The newer kernel didn't/does not affect the NFS servers at all, as they are all running CentOS 4.5 with 2.6.9-55.0.9.ELsmp, it's purely a client-side issue.
Just a note to inform that Johnny Hughes filed a bug with the upstream bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=431092
Akemi
On Jan 31, 2008 6:21 PM, Akemi Yagi amyagi@gmail.com wrote:
This problem does not seem to be associated with specific hardware as implied in the original thread.
I did not intend to imply any such thing, at least not as far as the client is concerned, we've verified this on different hardware from different suppliers.
To me, this is an inter-op issue with the new kernel and EMC's celerra NFS server, and am currently working it together with EMC tech support - obviously it may affect other nfs servers as well; we can only test against what we have.
BR Bent
On Feb 1, 2008 12:42 AM, Bent Terp bent@nagstrup.dk wrote:
On Jan 31, 2008 6:21 PM, Akemi Yagi amyagi@gmail.com wrote:
This problem does not seem to be associated with specific hardware as implied in the original thread.
I did not intend to imply any such thing, at least not as far as the client is concerned, we've verified this on different hardware from different suppliers.
To me, this is an inter-op issue with the new kernel and EMC's celerra NFS server, and am currently working it together with EMC tech support
- obviously it may affect other nfs servers as well; we can only test
against what we have.
A test kernel with a patch that seemingly fixes the issue (at least in my test run) has been made available. The details are in the upstream bugzilla filed by Johnny:
https://bugzilla.redhat.com/show_bug.cgi?id=431092 (Comment #30)
It would be great if other people give it a try and report the result to that bugzilla.
Akemi
-
Akemi Yagi -
Bent Terp -
nate