[CentOS] 10Gbit ethernet

[Ross S. W. Walker]

Jake Grimmett wrote:
> 
> If I could ask question about 10Gbit ethernet....
> 
> We have a 70 node cluster built on Centos 5.1, using NFS to mount user home 
> areas. At the moment the network is a bottleneck, and it's going to get worse 
> as we add another 112 CPUs in the form of two blade servers. 
> 
> To help things breath better, we are considering building three new NFS 
> servers, and connecting these and the blade servers directly via 10Gbit to a 
> new core switch, the older nodes will stay with 1Gbit ethernet, but be given 
> new switches uplinked via 10Gbit to the core switch.
> 
> Before I spend a great deal of money, can I ask if anyone here has experience 
> of 10GBit? Dell, HP, Supermicro, IBM (etc..) seem to be pushing the NetXen 
> PCIe cards, so I guess these drivers work...? As to media, although CX4 seems 
> cheaper than optical, I hear the cabling is nasty. And is the magical fairy 
> going to fix 10Gbit over cat6A anytime soon?
> 
> any thoughts appreciated.

I would seriously evaluate whether you need 10Gbe to the nodes or if
using it prodiguously within the switching fabric can give you the
performance you need.

10Gbe brings with it all kinds of other issues you may not have thought
about, compatibility of drivers and stability being just one. With the
increased speed comes increased CPU usage and interrupts as well as
PCI bus bandwidth.

With such a large cluster I don't think 10Gbe on the nodes is really
necessary as the bandwidth and load is spread out, but I agree you
will need some very good switching and VLANing to make it perform
at top speed.

I would post this question on one of the networking and switching
mailing lists and I am sure you can solicit some good responses
from someone that has a large HPCC.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.