Hi all, I'm migrating from Gentoo to CentOS... I'm experiencing a rather low performance in NFS r/w (as client). NFS server is solaris (which exports zfs volumes via nfs). The very same exports were mounted with the same parameters (auto,nosuid,exec) on gentoo and centos server (bot x86_64)... It happens that centos is 5-10 times slower either in read and write operations... Ok, I'll try to tune rsize and wsize, but does anybody have an hint on this low performance?
Thanks d
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
On Wed, Dec 10, 2008 at 11:07 AM, Davide Cittaro davide.cittaro@ifom-ieo-campus.it wrote:
Hi all, I'm migrating from Gentoo to CentOS... I'm experiencing a rather low performance in NFS r/w (as client). NFS server is solaris (which exports zfs volumes via nfs). The very same exports were mounted with the same parameters (auto,nosuid,exec) on gentoo and centos server (bot x86_64)... It happens that centos is 5-10 times slower either in read and write operations... Ok, I'll try to tune rsize and wsize, but does anybody have an hint on this low performance?
We've seen similar results, but only to specific types of servers. For example, we have an Onstor/XIV system that is showing 90MB/S throughput, but an acopia switch in front of it showing much degraded throughput.
Have you tried these measurements against a native Linux NFS server? It might be a good data point to find out if this only happens against Solaris.
However, it could be none of the above, and be degraded ZFS performance that the NFS protocol is exacerbating. Have you run network dumps?
--Russell
On Dec 10, 2008, at 8:14 PM, Russell Miller wrote:
Have you tried these measurements against a native Linux NFS server? It might be a good data point to find out if this only happens against Solaris.
Actually not, but I've tested other OS as client. Nothing compares to centos :-( I suspect there's something dealing with file permissions and owners... bah...
However, it could be none of the above, and be degraded ZFS performance that the NFS protocol is exacerbating. Have you run network dumps?
As far as other clients show better performances (close to network bandwidth limit) I guess Solaris is not the issue...
Thanks
d
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
Am Mittwoch, den 10.12.2008, 20:07 +0100 schrieb Davide Cittaro:
Hi all, I'm migrating from Gentoo to CentOS... I'm experiencing a rather low performance in NFS r/w (as client).
I am not the Linux Expert when it comes to the nfs implementation, but you should try two things:
1.) Nail Linux to use nfs v3 (imho mount option nfsvers=3) 2.) Use TCP, UDP Transmits are often slow these days. (mount option tcp)
NFS server is solaris (which exports zfs volumes via nfs). The very same exports were mounted with the same parameters (auto,nosuid,exec) on gentoo and centos server (bot x86_64)... It happens that centos is 5-10 times slower either in read and write operations... Ok, I'll try to tune rsize and wsize, but does anybody have an hint on this low performance?
Maybe gentoo uses v3 per default. We have made great improvements at our site with these mount options.
Thanks to all, I will double check either nfs server and client options...
d
On Dec 10, 2008, at 9:01 PM, John Kordash wrote:
2.) Use TCP, UDP Transmits are often slow these days. (mount option tcp)
Hmm, care to share any references for this? I'd be reaching for NFS over TCP in a long-haul type environment, but would run it over UDP otherwise.
-John _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
On Wed, Dec 10, 2008 at 12:06 PM, Davide Cittaro davide.cittaro@ifom-ieo-campus.it wrote:
Thanks to all, I will double check either nfs server and client options...
d
What is the kernel version of your CentOS? Some earlier version is known to have an issue with NFS (client). For example:
http://bugs.centos.org/view.php?id=2635
Akemi
Hi all, after some double checks and test I still have CentOS nfs client performance not exciting... Just for the record nfs server is Solaris 10. With CentOs client I reach at most ~ 80 MB/s on a gigabit ethernet... I should be satisfied, as it is close to real-world ethernet performance. Nevertheless I have ~100 MB/s with other linux clients (not CentOS). The mount options are the same on CentOS and on Gentoo...
rw ,nosuid ,vers = 3 ,rsize =1048576,wsize=1048576,hard,noacl,proto=tcp,timeo=600,retrans=2,sec=sys
Also, I've tried another NFS server based on FreeBSD
rw ,vers =3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys
(options are results of nfsstat -m). I have ~74 MB/s on Gentoo and 52 MB/s on CentOS... I've tried many wsize/rsize, a bunch of different mount options, but still I can't outperform those values...
d
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
On Tue, 2008-12-16 at 22:40 +0100, Davide Cittaro wrote:
Hi all, after some double checks and test I still have CentOS nfs client performance not exciting... Just for the record nfs server is Solaris 10. With CentOs client I reach at most ~ 80 MB/s on a gigabit ethernet... I should be satisfied, as it is close to real-world ethernet performance. Nevertheless I have ~100 MB/s with other linux clients (not CentOS). The mount options are the same on CentOS and on Gentoo...
rw ,nosuid ,vers = 3 ,rsize =1048576,wsize=1048576,hard,noacl,proto=tcp,timeo=600,retrans=2,sec=sys
Also, I've tried another NFS server based on FreeBSD
rw ,vers =3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys
(options are results of nfsstat -m). I have ~74 MB/s on Gentoo and 52 MB/s on CentOS... I've tried many wsize/rsize, a bunch of different mount options, but still I can't outperform those values...
---- larger rsize/wsize values do not necessarily translate to faster and I was of the opinion that going over 65536 wasn't going to help at all. With some NFS servers, a smaller rsize/wsize is better. (8192 or 16384) but that is the point of tuning I suppose.
hubs/switches, wires and the network cards themselves all can impact performance - it's not necessarily CentOS itself.
Craig
On Dec 16, 2008, at 11:49 PM, Craig White wrote:
larger rsize/wsize values do not necessarily translate to faster and I was of the opinion that going over 65536 wasn't going to help at all. With some NFS servers, a smaller rsize/wsize is better. (8192 or 16384) but that is the point of tuning I suppose.
I've tried r/w sizes from 8k to auto (that is 104k for Solaris server and 32k for FreeBSD server)... there are minor changes in max speed (let's say it ranges from 70 to 80 MB/s).
hubs/switches, wires and the network cards themselves all can impact performance - it's not necessarily CentOS itself.
Well, as long as Gentoo linux and FreeBSD clients show ~100 MB/s... BTW I'm using 2.6.18-92.1.18.el5.centos.plus kernel on x86_64 hardware... I need to use CentOS (I migrated from Gentoo for sake of simplicity) but this NFS issue really... well, it is not a good thing :-)
d
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
On Wed, 2008-12-17 at 00:00 +0100, Davide Cittaro wrote:
On Dec 16, 2008, at 11:49 PM, Craig White wrote:
larger rsize/wsize values do not necessarily translate to faster and I was of the opinion that going over 65536 wasn't going to help at all. With some NFS servers, a smaller rsize/wsize is better. (8192 or 16384) but that is the point of tuning I suppose.
I've tried r/w sizes from 8k to auto (that is 104k for Solaris server and 32k for FreeBSD server)... there are minor changes in max speed (let's say it ranges from 70 to 80 MB/s).
hubs/switches, wires and the network cards themselves all can impact performance - it's not necessarily CentOS itself.
Well, as long as Gentoo linux and FreeBSD clients show ~100 MB/s... BTW I'm using 2.6.18-92.1.18.el5.centos.plus kernel on x86_64 hardware... I need to use CentOS (I migrated from Gentoo for sake of simplicity) but this NFS issue really... well, it is not a good thing :-)
---- out of curiousity...why are you using the plus kernel? Do you need it to support particular hardware? Is it possible to boot/test with the standard kernel?
Craig
On Dec 17, 2008, at 12:05 AM, Craig White wrote:
out of curiousity...why are you using the plus kernel? Do you need it to support particular hardware? Is it possible to boot/test with the standard kernel?
:-) Good question! I've enabled plus repository and yum wanted to install plus kernel... As new CentOS user I've trusted yum... Tomorrow I will try the standard one (not now, I don't like rebooting servers from home ^__^)
d
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
On Tue, Dec 16, 2008 at 3:10 PM, Davide Cittaro davide.cittaro@ifom-ieo-campus.it wrote:
On Dec 17, 2008, at 12:05 AM, Craig White wrote:
out of curiousity...why are you using the plus kernel? Do you need it to support particular hardware? Is it possible to boot/test with the standard kernel?
:-) Good question! I've enabled plus repository and yum wanted to install plus kernel... As new CentOS user I've trusted yum... Tomorrow I will try the standard one (not now, I don't like rebooting servers from home ^__^) d
Well, you got what you asked for. There's lots of info on the centos.org web site, including info about the standard vs. plus repositories. You need to make an informed decision there, which it looks like you're moving towards, so keep going.
mhr
Well, you got what you asked for. There's lots of info on the centos.org web site, including info about the standard vs. plus repositories. You need to make an informed decision there, which it looks like you're moving towards, so keep going.
Interestingly, as a project for work I've been benchmarking different kernels against our configuration, using iozone. The latest redhat/centos 5.3 kernel against stock 2.6.27.8 (as close to the centos config as is feasible.)
I have some pretty graphs, but the long and short of it is, while the read times are comparable, the write times for the stock kernel.org kernel are over twice as fast. I mean it's pretty much blowing the RHEL/CentOS kernel out of the water.
--Russell
Russell Miller wrote:
I have some pretty graphs, but the long and short of it is, while the read times are comparable, the write times for the stock kernel.org kernel are over twice as fast. I mean it's pretty much blowing the RHEL/CentOS kernel out of the water.
What sort of I/O controller?
nate
On Tue, Dec 16, 2008 at 4:10 PM, nate centos@linuxpowered.net wrote:
Russell Miller wrote:
I have some pretty graphs, but the long and short of it is, while the read times are comparable, the write times for the stock kernel.org kernel are over twice as fast. I mean it's pretty much blowing the RHEL/CentOS kernel out of the water.
What sort of I/O controller?
00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) 00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 01:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 02:03.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 03:01.0 RAID bus controller: 3ware Inc 9xxx-series SATA-RAID
On Tue, 2008-12-16 at 16:15 -0800, Russell Miller wrote:
On Tue, Dec 16, 2008 at 4:10 PM, nate centos@linuxpowered.net wrote:
Russell Miller wrote:
I have some pretty graphs, but the long and short of it is, while the read times are comparable, the write times for the stock kernel.org kernel are over twice as fast. I mean it's pretty much blowing the RHEL/CentOS kernel out of the water.
What sort of I/O controller?
00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) 00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) 00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) 00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) 00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) 00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) 01:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 02:03.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10) 03:01.0 RAID bus controller: 3ware Inc 9xxx-series SATA-RAID
---- we're getting pretty far afield of the OP's original issue but ISTR that the 3ware 9xxx-series SATA RAID is especially slow in RAID-5 so if that's what you're doing, it would make sense that newer kernels than the ones RHEL/CentOS would ship out would get better performance as they probably have significantly upgraded the driver code for that controller.
But none of this relates to CentOS performance as NFS client to Solaris NFS server.
Craig
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Russell Miller Sent: Tuesday, December 16, 2008 7:03 PM To: CentOS mailing list Subject: Re: [CentOS] nfs slow?
Well, you got what you asked for. There's lots of info on the centos.org web site, including info about the standard vs. plus repositories. You need to make an informed decision there, which it looks like you're moving towards, so keep going.
Interestingly, as a project for work I've been benchmarking different kernels against our configuration, using iozone. The latest redhat/centos 5.3 kernel against stock 2.6.27.8 (as close to the centos config as is feasible.)
I have some pretty graphs, but the long and short of it is, while the read times are comparable, the write times for the stock kernel.org kernel are over twice as fast. I mean it's pretty much blowing the RHEL/CentOS kernel out of the water.
Hi Russel, Are you Bonding the two Broad Com NICs? Can you give us an idea speed wise of the performance for NFS? OT, are you doing any Samba testing?
JohnStanley
Hi Russel, Are you Bonding the two Broad Com NICs? Can you give us an idea speed wise of the performance for NFS? OT, are you doing any Samba testing?
No, we're not bonding the NICs. Roughly, writes on the centos kernel are about 600K/s on average, and on the stock kernel about 1.3M/s.
These are iozone tests, it's not line speed because of the way iozone does its tests.
--Russell
Am Mittwoch, den 10.12.2008, 15:01 -0500 schrieb John Kordash:
2.) Use TCP, UDP Transmits are often slow these days. (mount option tcp)
One Key Fact for UDP being much faster then TCP was
a.) The Lack of performant Hardware (CPU Time was expensive and should not be shared for IO/Interrupts)
b.) The Lack of a performant Networkhardware
(Modern Hardware does offload TCP Flow to the Adapter, uses Jumbo uses Jumbo Frames and so on, so the main argument for generating the TCP Overhead is gone.)
Nowadays you most often have Gigabit Ethernet Cards and can use Jumbo Frames with TCP. This often results in higher throughput rates.
On a High Performance Maschine NFS over TCP has the ability to have somethling like Flow Control which helps when the Server is 1Gbit+ and the Client only 100Mbit :) Or the other side around.
For us Jumboframes and TCP was the way to go.
Hmm, care to share any references for this? I'd be reaching for NFS over TCP in a long-haul type environment, but would run it over UDP otherwise.
-John _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
John Kordash wrote:
2.) Use TCP, UDP Transmits are often slow these days. (mount option tcp)
Hmm, care to share any references for this? I'd be reaching for NFS over TCP in a long-haul type environment, but would run it over UDP otherwise.
At least in my case, both of the vendors I am using for high performance NFS (BlueArc, and Exanet) have tcp as a best practice. Exanet actually runs on top of CentOS 4.4 though does not use the linux NFS stack.
These are the options I use for Exanet: rw,bg,hard,intr,tcp,nfsvers=3,timeo=2,retrans=10,rsize=32768,wsize=32768
These are the options I use for BlueArc: rw,bg,hard,intr,proto=tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768
Exanet is an active-active cluster, the BlueArcs are not configured in any sort of cluster. Planning on retiring the BlueArcs very soon as they are about to be end of life.
nate
On Wed, Dec 10, 2008, nate wrote:
John Kordash wrote:
2.) Use TCP, UDP Transmits are often slow these days. (mount option tcp)
Hmm, care to share any references for this? I'd be reaching for NFS over TCP in a long-haul type environment, but would run it over UDP otherwise.
At least in my case, both of the vendors I am using for high performance NFS (BlueArc, and Exanet) have tcp as a best practice. Exanet actually runs on top of CentOS 4.4 though does not use the linux NFS stack.
We started using tcp with nfs about five years ago, largely to cure a problem where a system running SuSE 9.0 Pro with multiple IP addresses on the NIC was responding to NFS UDP packets from one of the aliased IP addresses, not the primary. This caused NFS mounts by OS X clients to fail as they expected to get the UDP packets back from the same IP to which they sent. Using tcp naturally fixed this, and I never got around to figuring out why the replies were coming from the aliases IP address.
We use NFS mounted home Maildir directories on a system with about 10,000 e-mail accounts, and a cluster of 4 machines handling incoming e-mail, with most of the postfix configuration files NFS mounted as well. These handle about 100,000 incoming messages a day without problems (a fair number of which are dropped without delivery after checking with spamassassin). There are about 182,000 IMAP/POP3 daily logins to check mail. Load averages are fairly low on all the systems, and the incoming mail queues rarely get over five messages with most of the delivery time being spamassassin checking using a central bayesian database.
The central server that has all the home directories generally runs with a load average around 0.50 (a 4-year old SLES 9.2 system with a single Intel(R) Pentium(R) 4 CPU 3.00GHz, 2GB RAM with 7,200 RPM Seagate Barracuda SATA drives, hardly a high performance machine compared to what we're building today.
Bill
On Wed, Dec 10, 2008 at 12:07 PM, Davide Cittaro davide.cittaro@ifom-ieo-campus.it wrote:
Hi all, I'm migrating from Gentoo to CentOS... I'm experiencing a rather low performance in NFS r/w (as client). NFS server is solaris (which exports zfs volumes via nfs). The very same exports were mounted with the same parameters (auto,nosuid,exec) on gentoo and centos server (bot x86_64)... It happens that centos is 5-10 times slower either in read and write operations... Ok, I'll try to tune rsize and wsize, but does anybody have an hint on this low performance?
Which version of CentOS are you using? The upstream kernels have had issues at different points with NFS performance (the 2.6.18 had a bunch of problems that were fixed in 2.6.20-22 time frame and have had to be backported.. then there were changes and breakage somewhere after that.. ) I normally try a set of comparisons:
1) What does CentOS-4 do? What does CentOS-5 do? 4 may actually be faster than 5 in some cases due to switch/card issues. 2) Then there are issues with TCP, size of frames, etc...
On Dec 16, 2008, at 11:54 PM, Stephen John Smoogen wrote:
Which version of CentOS are you using? The upstream kernels have had issues at different points with NFS performance (the 2.6.18 had a bunch of problems that were fixed in 2.6.20-22 time frame and have had to be backported.. then there were changes and breakage somewhere after that.. ) I normally try a set of comparisons:
Great! I'm 2.6.18-92.1.18.el5.centos.plus! I don't se 2.6.20 in yum updates... is there any unstable repository I should enable?
d
Davide Cittaro davide.cittaro@ifom-ieo-campus.it
On Tue, Dec 16, 2008 at 3:04 PM, Davide Cittaro davide.cittaro@ifom-ieo-campus.it wrote:
Great! I'm 2.6.18-92.1.18.el5.centos.plus! I don't se 2.6.20 in yum updates... is there any unstable repository I should enable?
You're playing with fire now. Newer kernels are not necessarily backwards compatible, and major breaks can occur. You are free to do as you choose, of course, but this is CentOS - you break it, you own it.
Good luck.
mhr
MHR wrote:
On Tue, Dec 16, 2008 at 3:04 PM, Davide Cittaro davide.cittaro@ifom-ieo-campus.it wrote:
Great! I'm 2.6.18-92.1.18.el5.centos.plus! I don't se 2.6.20 in yum updates... is there any unstable repository I should enable?
You're playing with fire now. Newer kernels are not necessarily backwards compatible, and major breaks can occur. You are free to do as you choose, of course, but this is CentOS - you break it, you own it.
to clarify what MHR is saying for the benefit of the uninitiated... a given upstream EL major version (3, 4, 5) is released with a specific kernel, for instance, EL 5 uses 2.6.18. rather than releasing completely new kernels, upstream backpatches critical fixes into their 2.6.18 version, effectively creating a branch, the current version of which is 2.6.18-92.1.18.el5 (unless its been updated again since I last checked). This maintains near-total compatibility, so that a given release remains a stable platform for its entire lifespan.
-
Akemi Yagi -
Bill Campbell -
Craig White -
Davide Cittaro -
John -
John Kordash -
John R Pierce -
MHR -
nate -
Russell Miller -
Stefan Held -
Stephen John Smoogen