We upgraded from a 10/100Mbs to a 2 100/1000 bonding. We notice the speeds of NFS to be around 70-80Mb/sec. Which is slow, especially with bonding. I was wondering if we need to tune anything special with the Network and NFS. Does anyone have any experience with this?
TIA
Mag Gam wrote:
We upgraded from a 10/100Mbs to a 2 100/1000 bonding. We notice the speeds of NFS to be around 70-80Mb/sec. Which is slow, especially with bonding. I was wondering if we need to tune anything special with the Network and NFS. Does anyone have any experience with this?
my general experience is that Linux has a rather poor implementation of NFS server. Solaris works much better as a NFS server.
John R Pierce wrote:
Mag Gam wrote:
We upgraded from a 10/100Mbs to a 2 100/1000 bonding. We notice the speeds of NFS to be around 70-80Mb/sec. Which is slow, especially with bonding. I was wondering if we need to tune anything special with the Network and NFS. Does anyone have any experience with this?
my general experience is that Linux has a rather poor implementation of NFS server. Solaris works much better as a NFS server.
I wouldn't completely disagree with that but we have a number of linux NFS servers running that are working really well - once we tweaked the mount options. We are using these: -rw,bg,nosuid,timeo=50,retry=1
-Mark
Mag Gam wrote:
We upgraded from a 10/100Mbs to a 2 100/1000 bonding. We notice the speeds of NFS to be around 70-80Mb/sec. Which is slow, especially with bonding. I was wondering if we need to tune anything special with the Network and NFS. Does anyone have any experience with this?
What kind of test are you running? How many clients vs servers? If you have 1 client and 1 server you won't use more than 1 NIC with bonding. Because bonding operates based on source/destination MAC or IP addresses (some bonding can take into account layer 4 ports as well).
I just ran a really simple test on my home network between two debian systems on a gigE network(no bonding). No special tuning at all. Just a 'dd' write test. To local disk I got 198MB/sec, to the NFS volume (same controller, disks, OS, kernel on the other end) I got 29MB/sec(232Mbit). This was for 410MB of data, on oldish systems(new HDDs, 3 year old raid cards and 3year old systems). Running an iperf test to test raw network throughput I get 940Mbit. (Intel E1000 on each end, with a Linksys in between)
But as another poster noted, NFS under linux isn't that great. My company uses a BlueArc for their NFS needs, it works well (fastest NAS in the world I think). Though the model we have is pretty old and is going end of life soon.
I certainly would never use Linux for any *really* serious NFS use. Casual stuff, fine..
nate
On Wednesday 30 July 2008 05:20:10 Mag Gam wrote:
We upgraded from a 10/100Mbs to a 2 100/1000 bonding. We notice the speeds of NFS to be around 70-80Mb/sec. Which is slow, especially with bonding. I was wondering if we need to tune anything special with the Network and NFS. Does anyone have any experience with this?
TIA _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Did you configure the bonding like it should be? Cause configuring a loadbalanced bonding (which I suppose you're using) is not at all an easy task.
For example if you are using rr bonding the switch where the nfsserver is connected to has to channel the ports the nfs-server is connected to. And much more.
I would recheck bonding. Test how fast it is with only one nic in the bond and the like. But first of all read the bonding.txt which comes along with the kernel-docs or can be found here: http://www.mjmwired.net/kernel/Documentation/networking/bonding.txt
Mag Gam wrote on Tue, 29 Jul 2008 23:20:10 -0400:
70-80Mb/sec.
Mb or MB?
Kai
I am using mode 0, which is an aggregate.
I have 2 clients, which are bonded too.
70-80Mb/sec.
MB, sorry :-)
I was wondering if there were any tuning parameters I should look into. Would a tcp window help? Would increasing the number of server and client processes help (not sure how to do that BTW). Any others
TIA
On Wed, Jul 30, 2008 at 4:31 AM, Kai Schaetzl maillists@conactive.com wrote:
Mag Gam wrote on Tue, 29 Jul 2008 23:20:10 -0400:
70-80Mb/sec.
Mb or MB?
Kai
-- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Mag Gam wrote: ...
into. Would a tcp window help? Would increasing the number of server and client processes help (not sure how to do that BTW). Any others
To change the number of server processes edit /etc/sysconfig/nfs, remove the # from the line:
#RPCNFSDCOUNT=8
and increase the number 8.
Mogens
Mag Gam wrote:
I am using mode 0, which is an aggregate.
I have 2 clients, which are bonded too.
70-80Mb/sec.
MB, sorry :-)
That's totally different then, 80MBytes/second is very fast. What kind of storage is behind the NFS?
Run iostat -x 1 100 on the server while your doing the test. Look at the last column, what's the reading?
In my experience the typical I/O size for requests is in the 20kB range, in which case a 10k RPM disk will run at about 2.2-2.5MBytes/second(before latency spikes). Depends what your transferring though. Cache of course can mask this to some degree.
If your using gigE across the board you may be able to get more performance by using 9k jumbo frames. Though I would only use jumbo frames on secondary interfaces dedicated for storage as jumbo and non jumbo traffic doesn't mix well. Of course any network devices in between the systems must support/be configured for jumbo frames as well. In Linux at least as far as I can frame size is determined by the interface itself, so e.g. you can't run two VLANs on one NIC, one in jumbo and one not in jumbo. It's either on or it's off. (Unlike switches where you can do both on the same port)
If your going to be routinely pushing a gigabit worth of NFS traffic I certainly would ditch Linux and get a more robust NAS box.
nate
Mag Gam wrote on Wed, 30 Jul 2008 07:10:13 -0400:
70-80Mb/sec.
MB, sorry :-)
Then I don't think it's bad, it's actually quite good for a single NIC and in the normal range for two bonded NICs. I get a maximum of 30 MB/s between two 1000 NICs over a SOHO Switch from Netgear in my office, no matter if SSH or NFS. And I don't think it's Linux specific. I saw about the same (a little bit more, maybe half of the theoretical throughput) when I tested the same switch last year between Windows PCs.
Kai
Mag Gam wrote:
70-80Mb/sec.
MB, sorry :-)
thats on the order of 700-800Mbit/sec, which is quite good for a single session on GigE. as others have said, the sort of bonding you're doing doesn't speed up single transfers, instead it helps with multiple concurrent sessions.
On Wednesday 30 July 2008 20:52:07 John R Pierce wrote:
Mag Gam wrote:
70-80Mb/sec.
MB, sorry :-)
thats on the order of 700-800Mbit/sec, which is quite good for a single session on GigE. as others have said, the sort of bonding you're doing doesn't speed up single transfers, instead it helps with multiple concurrent sessions.
I would not agree that if you are using round-robin (mode 0) bonding that it does not scale with one session. I would say it should or must scale:
from bonding.txt: 1780 balance-rr: This mode is the only mode that will permit a single 1781 TCP/IP connection to stripe traffic across multiple 1782 interfaces. It is therefore the only mode that will allow a 1783 single TCP/IP stream to utilize more than one interface's 1784 worth of throughput. This comes at a cost, however: the 1785 striping generally results in peer systems receiving packets out 1786 of order, causing TCP/IP's congestion control system to kick 1787 in, often by retransmitting segments. That means it could if we forget about out of order delivery.
What I see in a project where we have bonded four nics together with rr is that the way out is "evenly" loaded over the four nics (although we are communicating with only two hosts). But the way back is still the problem. Cause all packets arrive at only one nic. Again this is explained by bonding.txt:
1818 This mode requires the switch to have the appropriate ports 1819 configured for "etherchannel" or "trunking."
And I also remember that I read that you can configure the etherchannel balancing. That means the way back. Some switches by default spread the packets going back by the MAC-Address (that's what we are seeing). But there should also be other balancing modes for a etherchannel. Got it here it is:
211 If ARP monitoring is used in an etherchannel compatible mode 212 (modes 0 and 2), the switch should be configured in a mode 213 that evenly distributes packets across all links. If the 214 switch is configured to distribute the packets in an XOR 215 fashion, all replies from the ARP targets will be received on 216 the same link which could cause the other team members to 217 fail.
What we have done is first measure the possible load over the network (with nc without any local i/o involved for example) and then you have a bottom line. Next see what nfs can do.
BTW. I didn't write the bonding.txt but it appears to be dealing with some topics discussed here.
BTW. from FS point of view it should not be a problem to get some 200MByte/sec out of a bunch of disks (depending on the speed and cache of the disks and the bus where the data goes through).
Marc Grimme wrote:
On Wednesday 30 July 2008 20:52:07 John R Pierce wrote:
Mag Gam wrote:
70-80Mb/sec.
MB, sorry :-)
thats on the order of 700-800Mbit/sec, which is quite good for a single session on GigE. as others have said, the sort of bonding you're doing doesn't speed up single transfers, instead it helps with multiple concurrent sessions.
I would not agree that if you are using round-robin (mode 0) bonding that it does not scale with one session. I would say it should or must scale:
from bonding.txt:
<snip>
That means it could if we forget about out of order delivery.
What I see in a project where we have bonded four nics together with rr is that the way out is "evenly" loaded over the four nics (although we are communicating with only two hosts). But the way back is still the problem. Cause all packets arrive at only one nic. Again this is explained by bonding.txt:
What you are forgetting is the two-way communication between client and server during a file transfer.
With NFS using 4k IOs over TCP (with NFSv3 NFSv4) there are acks that need to come back within the window (64K or every 18 IOs) and IO will pause for them to catch up, and even if things are running smoothly and IO doesn't need to pause for acks, there is the network latency of data travelling back and forth that needs to be taken into consideration.
With network IO it's not the bandwidth that kills you it's the latency.
Network bandwidth becomes important when the number of clients gets large, say 100 or 1000 clients.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
-
John R Pierce -
Kai Schaetzl -
Mag Gam -
Marc Grimme -
Mark Belanger -
Mogens Kjaer -
nate -
Ross S. W. Walker