[CentOS] slow NFS speed

Marc Grimme grimme at atix.de
Wed Jul 30 20:41:10 UTC 2008

On Wednesday 30 July 2008 20:52:07 John R Pierce wrote:
> Mag Gam wrote:
> >> 70-80Mb/sec.
> >
> > MB, sorry :-)
> thats on the order of 700-800Mbit/sec, which is quite good for a single
> session on GigE.   as others have said, the sort of bonding you're doing
> doesn't speed up single transfers, instead it helps with multiple
> concurrent sessions.
I would not agree that if you are using round-robin (mode 0) bonding that it 
does not scale with one session. I would say it should or must scale:

from bonding.txt:
1780    balance-rr: This mode is the only mode that will permit a single
1781            TCP/IP connection to stripe traffic across multiple
1782            interfaces. It is therefore the only mode that will allow a
1783            single TCP/IP stream to utilize more than one interface's
1784            worth of throughput.  This comes at a cost, however: the
1785            striping generally results in peer systems receiving packets 
1786            of order, causing TCP/IP's congestion control system to kick
1787            in, often by retransmitting segments.
That means it could if we forget about out of order delivery.

What I see in a project where we have bonded four nics together with rr is 
that the way out is "evenly" loaded over the four nics (although we are 
communicating with only two hosts). But the way back is still the problem. 
Cause all packets arrive at only one nic. Again this is explained by 

1818            This mode requires the switch to have the appropriate ports
1819            configured for "etherchannel" or "trunking."

And I also remember that I read that you can configure the etherchannel 
balancing. That means the way back. Some switches by default spread the 
packets going back by the MAC-Address (that's what we are seeing). But there 
should also be other balancing modes for a etherchannel. Got it here it is:

211             If ARP monitoring is used in an etherchannel compatible mode
212             (modes 0 and 2), the switch should be configured in a mode
213             that evenly distributes packets across all links. If the
214             switch is configured to distribute the packets in an XOR
215             fashion, all replies from the ARP targets will be received on
216             the same link which could cause the other team members to
217             fail.

What we have done is first measure the possible load over the network (with nc 
without any local i/o involved for example) and then you have a bottom line. 
Next see what nfs can do. 

BTW. I didn't write the bonding.txt but it appears to be dealing with some 
topics discussed here.

BTW. from FS point of view it should not be a problem to get some 200MByte/sec 
out of a bunch of disks (depending on the speed and cache of the disks and 
the bus where the data goes through).

Gruss / Regards,

Marc Grimme
http://www.atix.de/               http://www.open-sharedroot.org/

More information about the CentOS mailing list