[CentOS] Re: 10Gbit ethernet

[Jake Grimmett]

On Friday 14 March 2008 10:43, Kay Diederichs wrote:
> Jake Grimmett schrieb:
> > If I could ask question about 10Gbit ethernet....
> >
> > We have a 70 node cluster built on Centos 5.1, using NFS to mount user
> > home areas. At the moment the network is a bottleneck, and it's going to
> > get worse as we add another 112 CPUs in the form of two blade servers.
> >
> > To help things breath better, we are considering building three new NFS
> > servers, and connecting these and the blade servers directly via 10Gbit
> > to a new core switch, the older nodes will stay with 1Gbit ethernet, but
> > be given new switches uplinked via 10Gbit to the core switch.
> >
> > Before I spend a great deal of money, can I ask if anyone here has
> > experience of 10GBit? Dell, HP, Supermicro, IBM (etc..) seem to be
> > pushing the NetXen PCIe cards, so I guess these drivers work...? As to
> > media, although CX4 seems cheaper than optical, I hear the cabling is
> > nasty. And is the magical fairy going to fix 10Gbit over cat6A anytime
> > soon?
> >
> > any thoughts appreciated.
> >
> > Jake
>
> Jake,
>
> please note that the latest RedHat/CentOS-5.1 kernels have NFS
> performance issues - if these are (partly) responsible for the
> bottleneck then you may want to install a kernel which fixes the issue.
>
> Check out RedHat bugzilla 321111.
>
> HTH,
>
> Kay
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

Many thanks for your answers :) 

I will certainly talk to the beowulf crowd, and I think your right that our 
bottleneck's are *partly* caused by NFS issues. 

I'm probably not going to feed 10Gb to the individual blades, as I have few 
MPI sers, though it's an option with IBM and HP blades. However IBM, and Dell 
offer a 10Gbit XFP uplink to the blade servers internal switch, and this has 
to be worthwhile with 56 CPUs on the other side of it.

I'm most concerned about whether anyone has tried the Netxen or Chelsio 10Gbit 
NICs on Centos 5.1; I see drivers in /lib/modules for these...

Also - do people have good / bad experiences of CX4 cabling? As an economical 
short range solution (<15M) it seems ideal for a server room, but I have a 
sales rep who is trying to scare me off, but he is biased as the 10Gb SR XFP 
transceivers are very expensive (~£820)...

Many thanks

Jake