[CentOS] athlon64/opteron 8GB per CPU

[Joshua Baker-LePain]

On Mon, 10 Apr 2006 at 2:28pm, David Thompson wrote

> I'm putting together a spec for a big(ger) memory x86_64 host, and I hope some
> of you-all can help.  I'm looking for success stories of hardware that folks
> are using in production environments, preferably with CentOS 4, with more than
> 4GB of memory per CPU.
>
> I know there are lots of mobos out there that can do this, but I'm looking for
> what folks really have running.  Call me paranoid.

I have several Opteron/8GB RAM systems, all running centos-4/x86_64.  The 
older ones are on Arima HDAMA motherboards, and the newer ones are on 
Supermicro H8DAR-T boards.  They're very nice systems.

> The target is SATA + n*GigE + 1CPU + 8GB memory, and I can consider other
> configurations.  The incumbent system is a Xeon 8GB system running i686 CentOS
> 4.3 (what else?), with light CPU loading and very heavy IO loading (WD raptors
> are our friends).

One caveat with these Supermicro boards is the the onboard SATA (some 
sort of Adapted branded Marvell thingy) isn't supported.  They offer some 
sort of binary download, but we all know how much fun that is.  I just put 
2 port 3wares in 'em all (they're cluster compute nodes).  Sure, it's 
expensive for a simple SATA controller.  But it's also no fuss and rock 
solid.  The newer SM boards have HT2000/HT1000 based SATA which centos 4.3 
should support, but I've no experience with 'em.

> Folks with deep(er) knowledge of x86_64 architecture, I'd also be interested
> in the trade offs between 1CPU w/8GB and 2CPUs each with 4GB.  (E.g. what's
> the real cost of shipping data back and forth between the memory spaces of the
> two CPUs?  How does the 2.6 kernel deal with the split memory space?  etc...)
> The memory is all going to be used up in kernel buffers keeping bits of files
> in kernel memory.

I don't know how deep my knowledge is, but we run a lot of memory 
intensive code on these nodes and get a 2X speedup with dual CPUs.  So, 
IME, there's not much penalty with the extra hops to get to the other 
CPU's memory.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University