On Fri, 2005-06-24 at 01:13 -0500, Bryan J. Smith wrote:
Whey people are willing to spend >>$1,000 and then skimp on a few hundred bucks more is beyond me. You'd be better off not going Opteron/Registered at all, and saving all that dough.
The other thing is the NIC. People will spend thousands of dollars, then put in a desktop NIC.
In the days of 100Mbps, it wasn't too bad. But in the days of GbE, it's unbelievable.
I remember back when NetGear first introduced the 533MBps PCI GA620 with only a 512KB SRAM cache for under $500. Man, I had never seen a GbE with less than 1MB (and typically 2MB+). Now that's back when it was glued (and not a single MAC+SRAM IC), but the wait state wasn't any worse.
But now we're in the age of MAC+SRAM in 1 IC.
Mainboards are coming with RealTek GbE MACs that have 2KB of SRAM -- yes _two_kilobytes_! You know what that means? It can store 1 Ethernet frame (and forget Jumbo frames) and that's all! ;-ppp Most of the Intel PCIe and AMD HyperTransport MACs are only 8-32KB as well. Okay for a client -- especially with 802.3x -- but detrimental for GbE on a server in _any_ capacity.
Even Intel's NICs are MACs with typically 16-64KB, with only 1 model has 256KB SRAM. The Broadcoms on Opterons are typically 64-96KB SRAM/port, although they do have a MAC for a NIC with 256KB in the single MAC+SRAM IC.
First thing I do when I walk into a client that says they're having performance issues with GbE is find out what their server NIC is. 9 times out of 10, it's a cheap, desktop NIC or embedded MAC -- let alone a pre-802.3x one.
802.3x doesn't solve the problem for servers (or will the newer sub- committees either), but at least it does let the system tell every transmitting node to shut-up until its ready again. Otherwise you get a compounding effect that makes good'ole broadcast storms look tame. At least the wire is the problem there, not the server itself.