[CentOS] Re: Opteron Mobo Suggestions -- ignoring the GbE NIC on a server

[Bryan J. Smith]

On Fri, 2005-06-24 at 01:13 -0500, Bryan J. Smith wrote:
> Whey people are willing to spend >>$1,000 and then skimp on a few
> hundred bucks more is beyond me.  You'd be better off not going
> Opteron/Registered at all, and saving all that dough.

The other thing is the NIC.
People will spend thousands of dollars, then put in a desktop NIC.

In the days of 100Mbps, it wasn't too bad.
But in the days of GbE, it's unbelievable.

I remember back when NetGear first introduced the 533MBps PCI GA620 with
only a 512KB SRAM cache for under $500.  Man, I had never seen a GbE
with less than 1MB (and typically 2MB+).  Now that's back when it was
glued (and not a single MAC+SRAM IC), but the wait state wasn't any
worse.

But now we're in the age of MAC+SRAM in 1 IC.

Mainboards are coming with RealTek GbE MACs that have 2KB of SRAM -- yes
_two_kilobytes_!  You know what that means?  It can store 1 Ethernet
frame (and forget Jumbo frames) and that's all!  ;-ppp  Most of the
Intel PCIe and AMD HyperTransport MACs are only 8-32KB as well.  Okay
for a client -- especially with 802.3x -- but detrimental for GbE on a
server in _any_ capacity.

Even Intel's NICs are MACs with typically 16-64KB, with only 1 model has
256KB SRAM.  The Broadcoms on Opterons are typically 64-96KB SRAM/port,
although they do have a MAC for a NIC with 256KB in the single MAC+SRAM
IC.

First thing I do when I walk into a client that says they're having
performance issues with GbE is find out what their server NIC is.  9
times out of 10, it's a cheap, desktop NIC or embedded MAC -- let alone
a pre-802.3x one.

802.3x doesn't solve the problem for servers (or will the newer sub-
committees either), but at least it does let the system tell every
transmitting node to shut-up until its ready again.  Otherwise you get a
compounding effect that makes good'ole broadcast storms look tame.  At
least the wire is the problem there, not the server itself.


-- 
Bryan J. Smith                                     b.j.smith at ieee.org 
--------------------------------------------------------------------- 
It is mathematically impossible for someone who makes more than you
to be anything but richer than you.  Any tax rate that penalizes them
will also penalize you similarly (to those below you, and then below
them).  Linear algebra, let alone differential calculus or even ele-
mentary concepts of limits, is mutually exclusive with US journalism.
So forget even attempting to explain how tax cuts work.  ;->