Jake Grimmett wrote:
If I could ask question about 10Gbit ethernet....
We have a 70 node cluster built on Centos 5.1, using NFS to mount user home areas. At the moment the network is a bottleneck, and it's going to get worse as we add another 112 CPUs in the form of two blade servers.
To help things breath better, we are considering building three new NFS servers, and connecting these and the blade servers directly via 10Gbit to a new core switch, the older nodes will stay with 1Gbit ethernet, but be given new switches uplinked via 10Gbit to the core switch.
Before I spend a great deal of money, can I ask if anyone here has experience of 10GBit? Dell, HP, Supermicro, IBM (etc..) seem to be pushing the NetXen PCIe cards, so I guess these drivers work...? As to media, although CX4 seems cheaper than optical, I hear the cabling is nasty. And is the magical fairy going to fix 10Gbit over cat6A anytime soon?
any thoughts appreciated.
I would seriously evaluate whether you need 10Gbe to the nodes or if using it prodiguously within the switching fabric can give you the performance you need.
10Gbe brings with it all kinds of other issues you may not have thought about, compatibility of drivers and stability being just one. With the increased speed comes increased CPU usage and interrupts as well as PCI bus bandwidth.
With such a large cluster I don't think 10Gbe on the nodes is really necessary as the bandwidth and load is spread out, but I agree you will need some very good switching and VLANing to make it perform at top speed.
I would post this question on one of the networking and switching mailing lists and I am sure you can solicit some good responses from someone that has a large HPCC.
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.