I'm putting together a spec for a big(ger) memory x86_64 host, and I hope some of you-all can help. I'm looking for success stories of hardware that folks are using in production environments, preferably with CentOS 4, with more than 4GB of memory per CPU.
I know there are lots of mobos out there that can do this, but I'm looking for what folks really have running. Call me paranoid.
The target is SATA + n*GigE + 1CPU + 8GB memory, and I can consider other configurations. The incumbent system is a Xeon 8GB system running i686 CentOS 4.3 (what else?), with light CPU loading and very heavy IO loading (WD raptors are our friends).
Folks with deep(er) knowledge of x86_64 architecture, I'd also be interested in the trade offs between 1CPU w/8GB and 2CPUs each with 4GB. (E.g. what's the real cost of shipping data back and forth between the memory spaces of the two CPUs? How does the 2.6 kernel deal with the split memory space? etc...) The memory is all going to be used up in kernel buffers keeping bits of files in kernel memory.
Also, trade offs between Opteron and Xeon architectures, i686 vs. x86_64, bounce buffers, etc., for an application like this would be helpful.
The application is enhancement to a mirror server (mirror.cs.wisc.edu) here at UW-Madison.
Many thanks to all in advance. Private or public replies welcome.
Dave Thompson UW-Madison
On Mon, 10 Apr 2006 at 2:28pm, David Thompson wrote
I'm putting together a spec for a big(ger) memory x86_64 host, and I hope some of you-all can help. I'm looking for success stories of hardware that folks are using in production environments, preferably with CentOS 4, with more than 4GB of memory per CPU.
I know there are lots of mobos out there that can do this, but I'm looking for what folks really have running. Call me paranoid.
I have several Opteron/8GB RAM systems, all running centos-4/x86_64. The older ones are on Arima HDAMA motherboards, and the newer ones are on Supermicro H8DAR-T boards. They're very nice systems.
The target is SATA + n*GigE + 1CPU + 8GB memory, and I can consider other configurations. The incumbent system is a Xeon 8GB system running i686 CentOS 4.3 (what else?), with light CPU loading and very heavy IO loading (WD raptors are our friends).
One caveat with these Supermicro boards is the the onboard SATA (some sort of Adapted branded Marvell thingy) isn't supported. They offer some sort of binary download, but we all know how much fun that is. I just put 2 port 3wares in 'em all (they're cluster compute nodes). Sure, it's expensive for a simple SATA controller. But it's also no fuss and rock solid. The newer SM boards have HT2000/HT1000 based SATA which centos 4.3 should support, but I've no experience with 'em.
Folks with deep(er) knowledge of x86_64 architecture, I'd also be interested in the trade offs between 1CPU w/8GB and 2CPUs each with 4GB. (E.g. what's the real cost of shipping data back and forth between the memory spaces of the two CPUs? How does the 2.6 kernel deal with the split memory space? etc...) The memory is all going to be used up in kernel buffers keeping bits of files in kernel memory.
I don't know how deep my knowledge is, but we run a lot of memory intensive code on these nodes and get a 2X speedup with dual CPUs. So, IME, there's not much penalty with the extra hops to get to the other CPU's memory.
David Thompson wrote:
I'm putting together a spec for a big(ger) memory x86_64 host, and I hope some of you-all can help. I'm looking for success stories of hardware that folks are using in production environments, preferably with CentOS 4, with more than 4GB of memory per CPU.
I have a dual opteron server with 6gb ram running RHEL4u3 (not CentOS but you understand). It has an Adaptec SCSI RAID controller and 5 70gb drives in a RAID5 array (mailserver).
I know there are lots of mobos out there that can do this, but I'm looking for what folks really have running. Call me paranoid.
Paranoid nothing, I fully understand the desire to get recommendations on a board, I've been burned in the past also. The board we're using is a Tyan S2882. They are not dual core Opteron's however we have a similar server running a newer Tyan board, 16gb ram and dual core opterons. It's been quite stable on the Operating System it runs (I'll give you a hint, it comes from Redmond, WA)
The target is SATA + n*GigE + 1CPU + 8GB memory, and I can consider other configurations. The incumbent system is a Xeon 8GB system running i686 CentOS 4.3 (what else?), with light CPU loading and very heavy IO loading (WD raptors are our friends).
Why SATA? Are you using the onboard controller? You may have issues here. I'd at least consider a SCSI RAID card if it's in the budget. If not, be sure your board's SATA controller is well supported.
Why i686? Sure the i386 version of CentOS 4 can access >4gb ram but it's quite ugly how it has to do it and there are still some limits. I'd strongly recommend using the x86_64 release if you can pull it off.
Folks with deep(er) knowledge of x86_64 architecture, I'd also be interested in the trade offs between 1CPU w/8GB and 2CPUs each with 4GB. (E.g. what's the real cost of shipping data back and forth between the memory spaces of the two CPUs? How does the 2.6 kernel deal with the split memory space? etc...) The memory is all going to be used up in kernel buffers keeping bits of files in kernel memory.
Here's where the Opteron with onboard memory controllers will kick Xeon's rear. Also, throwing more CPUs in the mix is not going to cut down on the amount of memory available, a system with 2-cpus and 8gb total ram will always be faster than a system with 1 cpu and 8gb ram (except in really strange workloads).
Also, trade offs between Opteron and Xeon architectures, i686 vs. x86_64, bounce buffers, etc., for an application like this would be helpful.
Opteron rules. Their x86_64 support is better and the onboard memory controller is much faster. Also, the Opterons have dedicated pathways between CPUs which means much faster process sharing and communication between CPUs. As I understand it if CPU1 has an area of memory cached and CPU2 needs that data, it can grab it from CPU1 directly. Xeons would need to go back to RAM to grab the data which is much slower.
The application is enhancement to a mirror server (mirror.cs.wisc.edu) here at UW-Madison.
Well, if all you're doing is rsync or similar than CPU really isn't going to be a bottleneck in the near term, bandwidth and I/O are the bigger bottlenecks. What speed will the network be running at?
Many thanks to all in advance. Private or public replies welcome.
Dave Thompson UW-Madison
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Jay Lee Sent: Monday, April 10, 2006 1:51 PM To: CentOS mailing list; thomas@cs.wisc.edu Subject: Re: [CentOS] athlon64/opteron 8GB per CPU
David Thompson wrote:
SNIP
Folks with deep(er) knowledge of x86_64 architecture, I'd
also be interested
in the trade offs between 1CPU w/8GB and 2CPUs each with
4GB. (E.g. what's
the real cost of shipping data back and forth between the
memory spaces of the
two CPUs? How does the 2.6 kernel deal with the split
memory space? etc...)
The memory is all going to be used up in kernel buffers
keeping bits of files
in kernel memory.
Here's where the Opteron with onboard memory controllers will kick Xeon's rear. Also, throwing more CPUs in the mix is not going to
cut
down on the amount of memory available, a system with 2-cpus and 8gb
total ram will always be faster than a system with 1 cpu and 8gb ram
(except in really strange workloads).
Also, trade offs between Opteron and Xeon architectures,
i686 vs. x86_64,
bounce buffers, etc., for an application like this would be
helpful.
Opteron rules.
To those interested in Opteron with big memory on Centos 4.3,
Yes Opteron rules by a large margin.
We have completed significant testing with Dual Core 280's on a 2882 and on a 2895 with Centos 4.3 X86_64 - kernel 2.6.9-37.Elsmp and 2.6.12.6 for a linux cluster.
Each machine had 4GB of DRAM which should present the same problems as 8 GB or 16 GB of RAM.
These problems can be summed up as:
A) Where is the PCI and I/O space going to be mapped? B) What about IOMMU? C) NUMA ? Do I need / want it? D) How do all of the above interact with I/O Cards? E) Which I/O cards will be used and how does this effect the drivers / kernel?
Machines (4) tested:
Tyan 2882 and 2895 MB Opteron 280 or 285's 4 x 1GB DDR-400 RECC Memory (Kingston or equivalent) PS = > 400 watts with > 20Amps of +12V Significant HSF and case cooling
Master node -> added a 3ware 7506-4 with 4 x 80 GB IDE disks This was added to allow for faster compiles of the kernel, the Silverstorm Infiniband source code, application code and MPI code. (kernel compiles were less than 2.5 minutes)
Slave node -> added 80 GB SATA drive for booting
All machines were loaded with Centos 4.3 X86_64 Complete install.
Problems encountered:
1) The MB BIOS had choices that effected functionality when the Silverstorm and 3ware cards were installed:
Choices were:
Software PCI hole remap -> enable or disable Hardware PCI hole remap -> enable or disable IOMMU -> enable or disable ACPI SRAT Table -> enable or disable (enable for NUMA) Node Interleave - Must Disable for NUMA to be enabled.
Wrong choices either ate up some of the below 4GB memory, caused the system to not function properly or cause a panic with numa=on.
Also tried PNP on and off and it only seemed to change which interrupt was used by a given I/O card at boot time but both worked.
2) NUMA was very important to us to keep the processes locked to a given NUMA processor / set of cores and a memory node -> boot command line addition -> numa=on
This increased performance by 5 to 20 percent depending on the application.
What this does is: At the start up of an application, processor affinity is soft but memory affinity is hard: ie: the processor running a given task can be switched but where that task's memory is assigned stays fixed once the task requests memory.
If numa is turned on the scheduler tries to keep a task located on the same memory and CPU node.
3) Compiling the kernel for the K8 as opposed to the generic X86_64 only provided a small increase in performance BUT using the 2.6.12.6 kernel and compiling it with the latest gcc 4.0.2 compiler and then compiling MPI and the application using the same compiler provided a 15 percent application performance increase.
Seems that gcc 4.0.2 received some significant changes that helped the K8.
BTW, 2.6.12.6 worked fine on top of Centos 4.3
Hope this helps answer some of the questions about Opteron / large memory / NUMA / system setup.
Seth Bardash
Integrated Solutions and Systems
719-495-5866
Failure can not cope with knowledge and perseverance!
On Mon, 2006-04-10 at 16:36 -0600, Seth Bardash wrote:
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Jay Lee Sent: Monday, April 10, 2006 1:51 PM To: CentOS mailing list; thomas@cs.wisc.edu Subject: Re: [CentOS] athlon64/opteron 8GB per CPU
David Thompson wrote:
SNIP
<snip>
*This* is a very useful and generous reply. I thank you for taking the time and effort. It would be great if we had more like this.
-
David Thompson -
Jay Lee -
Joshua Baker-LePain -
Seth Bardash -
William L. Maltby