On Mon, Mar 8, 2010 at 12:49 PM, Eduardo Grosclaude eduardo.grosclaude@gmail.com wrote:

I'm targeting E5520. I'll buy in Argentina, with a high stack of all sort of costs threw upon the product, so budget may not mean much to foreigners.

Eduardo,

Are you going to be writing your own HPC software, or using some pre-written cluster aware OS? Check around to see if Supermicro has a regional sales center for your local. They have a new line of servers setup for GPU based HPC applications... I have no need for such horsepower, but reading the specs makes me envious none the less:

http://www.supermicro.com/products/system/4U/7046/SYS-7046GT-TRF.cfm?GPU=TC4

Dual 5500 series cpu (four core each) plus four nVidia Tesla GPU based supercomputing engines (included).

If your application can't support GPU based processing, I think Peter's suggestion is most fitting. Load up a rack of dual socket 5520 servers from Dell or HP and then save some money by building your own shared-storage to feed the cluster. The big vendors crank out very inexpensive dual socket xeon servers, the only area they really seem to be price gouging in right now is storage.

Gordon

On Mon, Mar 08, 2010 at 08:09:57PM -0800, nate wrote:

Gordon McLellan wrote:

...

If your application can't support GPU based processing, I think Peter's suggestion is most fitting. Load up a rack of dual socket 5520 servers from Dell or HP and then save some money by building your own shared-storage to feed the cluster. The big vendors crank out very inexpensive dual socket xeon servers, the only area they really seem to be price gouging in right now is storage.

For me I have been working on spec'ing out a "HPC" cluster to run Hadoop on large amounts of data and fell in love with the SGI Cloud Rack C2.

I managed to come up with a configuration that had roughly 600 CPU cores, 1.2TB of memory and 300 1TB SATA disks in a single rack and consumes ~16,000 watts of power with 99% efficient rack level power supplies and N+1 power redundancy, rack level cooling as well. Very cost effective as well at least for larger scale deployments, assuming you have a data center that can support such density.

http://www.sgi.com/products/servers/cloudrack/cloudrackc2.html

My current data center does not support such density so I came up with a configuration of 320 CPU cores, 640GB memory, and 160x1TB disks that fit in a single 24U rack, and consumes roughly 8,000 watts(208V 30A 3-phase) and weighs in at just under 1,200 pounds (everything included).

Systems come fully racked, cabled & ready to plug in. Systems are built with commodity components wherever possible(MB/ram/CPU/HD), only custom stuff is the enclosure, cooling, and power distribution, which is how they achieve the extreme densities and power efficiency.

Wow, pretty cool system. Can you tell about the pricing?

-- Pasi

I work for Dell but I can't talk too much about the units you are referring to. The launch date is in a couple of weeks and then I can spill my guts :)

I can't talk about price since, to be honest, I don't really know pricing (I'm a tech person). But let me give some general hints. The unit you are speaking about has actually been selling for a couple of years to larger customers. There are more units of this in production use right now than all of Supermicro and HP combined :) One success I can mention since it's public is Wolfram's Alpha system is powered by these units.

What is new with the "launch" is that before you had to but them in quantities of 500-1,000. Now you can buy one of them.

So if Dell was doing these 2 years ago, imagine what is coming next :)

Jeff

________________________________ From: nate centos@linuxpowered.net To: centos@centos.org Sent: Tue, March 9, 2010 7:37:04 PM Subject: Re: [CentOS] Motherboards for HPC applications

Pasi Kärkkäinen wrote:

Wow, pretty cool system. Can you tell about the pricing?

I don't think I can, but it is competitive with Dell and HP as an example while the innovation put into the cloud rack is far beyond anything Dell or HP offer to mere mortals. Closest HP offers is the "SL" series of systems which are pretty decent, though offer roughly half the density as SGI for our particular application.

http://h10010.www1.hp.com/wwpc/us/en/sm/WF02a/15351-15351-3896136.html?jumpi...

Dell is coming out with something new soon

http://www.theregister.co.uk/2010/02/03/dell_cloudedge/

I've seen them, and honestly aren't all that creative, very similar to Supermicro Twin. They are decent for CPU and memory intensive stuff, but not as good for (local) I/O intensive. They seem pretty proud about these systems though considering Supermicro has had similar stuff on the market for quite some time now there isn't much to get excited about IMO.

SGI(formerly Rackable) has been pretty aggressive in patenting their designs, which is probably what lead to vendors like Supermicro building their "Twin" systems.

http://www.sgi.com/company_info/newsroom/press_releases/rs/2007/05082007.htm...

Dell has a custom design division which they can probably do some pretty crazy things but I'm told they have a ~1,500 server minimum to get anything from that group.

nate

_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Eduardo Grosclaude wrote:

On Tue, Mar 9, 2010 at 12:49 AM, Christopher Chan christopher.chan@bradbury.edu.hk wrote:

...

On Tuesday, March 09, 2010 12:34 AM, Eduardo Grosclaude wrote:

...

Hello, Can somebody recommend CentOS-OK, dual socket motherboards for compute elements? A quick look up at Intel pages suggests they are thinking of them as "server boards", but then they recommend them as "for SMB", I'm somewhat puzzled about it. It would be nice to know what MBs you are using, pros and cons. Thank you in advance

Could you give us a bit more information on the HPC part? Is this clustering or computing?

I'll be buying a single machine first, building a cluster some time later. As this second move may be delayed for an unpredictable amount of time, what I am really interested in is understanding the thought process a seasoned technician (sysadmin? clusadmin?) may follow when selecting hardware.

Do you have high i/o needs?

Well, perhaps this is my real problem... Don't have enough info about applications. There are several of them but I think I/O is not at premium, rather CPU computing is.

If you do not have enough information on the applications, I am afraid it is going to be rather hard to make a final decision. Maybe you want to overspec on the first box, find out what those apps really do and then spec accordingly.

Things to consider can include network bandwidth, disk bandwidth. 'bus' bandwidth, memory bandwidth and as John Pierce pointed out, what type of processing. Are the apps single threaded or multi threaded? Single threaded apps might call for the cpus with the highest possible frequencies while multi threaded ones not so much so but how many you can pack into whatever space you have.

If cpu processing power is the sole criteria, then why limit to dual-socket boards and not go for quad-socket boards?

On Wednesday, March 10, 2010 12:35 AM, Joshua Baker-LePain wrote:

On Tue, 9 Mar 2010 at 9:49pm, Chan Chung Hang Christopher wrote

...

If cpu processing power is the sole criteria, then why limit to dual-socket boards and not go for quad-socket boards?

In general, the price goes up non-linearly as you go above 2 sockets, making 2 sockets the sweet spot when it comes to price/performance.

Hmm, I see at most a 50% increase in motherboard pricing from a dual to a quad socket motherboard and that is with a difference in feature set too with the quad coming with an extra onboard LSI 8 port SAS controller. That is hardly going up non-linearly. (taking an extremely narrow angle ;-p)

On Wednesday, March 10, 2010 11:41 AM, John R Pierce wrote:

Christopher Chan wrote:

...

Hmm, I see at most a 50% increase in motherboard pricing from a dual to a quad socket motherboard and that is with a difference in feature set too with the quad coming with an extra onboard LSI 8 port SAS controller. That is hardly going up non-linearly. (taking an extremely narrow angle ;-p

did you price the quad socket CPUs ? they cost significantly more,

Guess why I said 'extremely narrow angle' :-D

too. the newest processor chips are usually the dual socket versions. Also, 4-socket servers may have performance issues caused by 4-way cache coherency chatter, non-uniform memory access, and so forth.

/me looking at AMD solutions where those issues do not exist. Performance is almost linear. On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that does not mean quad socket solutions should be discounted when an AMD solution will outperform the best Intel dual-socket setup available.

On Thursday, March 11, 2010 01:41 AM, Eduardo Grosclaude wrote:

On Wed, Mar 10, 2010 at 2:05 AM, Christopher Chan christopher.chan@bradbury.edu.hk wrote:

...

On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that

Wow, that's a pretty impressive statement, can you elaborate on that?

Intel does not have a NUMA architecture solution for quad socket boards yet and so bus contention kills scaling on quad socket Intel solutions. Right now, AMD owns the quad socket market when it comes to performance.

You mean, for every possible workload? Is it something you learned from direct experience, or have you read about it? If so, where? If from experience, what was the general setup, applications, etc?

Anandtech did some testing last quarter where they compared what appears to be the best quad socket Intel solution against the best dual socket Intel solution and the dual socket solution ran circles around the quad in some tests and pretty much matches it in other tests.

http://it.anandtech.com/IT/showdoc.aspx?i=3653&p=1

According to the article, Intel will be coming out with something that will scale at the quad socket level some time this year so things are probably going to change on this front.

On Thu, Mar 11, 2010 at 08:35:23AM +0800, Christopher Chan wrote:

On Thursday, March 11, 2010 01:41 AM, Eduardo Grosclaude wrote:

...

On Wed, Mar 10, 2010 at 2:05 AM, Christopher Chan christopher.chan@bradbury.edu.hk wrote:

...

On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that

Wow, that's a pretty impressive statement, can you elaborate on that?

Intel does not have a NUMA architecture solution for quad socket boards yet and so bus contention kills scaling on quad socket Intel solutions. Right now, AMD owns the quad socket market when it comes to performance.

...

You mean, for every possible workload? Is it something you learned from direct experience, or have you read about it? If so, where? If from experience, what was the general setup, applications, etc?

Anandtech did some testing last quarter where they compared what appears to be the best quad socket Intel solution against the best dual socket Intel solution and the dual socket solution ran circles around the quad in some tests and pretty much matches it in other tests.

http://it.anandtech.com/IT/showdoc.aspx?i=3653&p=1

According to the article, Intel will be coming out with something that will scale at the quad socket level some time this year so things are probably going to change on this front.

Nehalem-EX ?

I think all the big server vendors have been announcing new products based on those last week, this week, or next week.

-- Pasi

On Tue, Mar 9, 2010 at 1:08 AM, John R Pierce pierce@hogranch.com wrote:

for a high performance compute cluster, you'll probably want to use management software like Oscar, which integrates system management with MPI based distributed computing such that you can manage a cluster of 100s of servers like its a single big system

I've been using Kusu with much success. Sadly, you're pretty much on your own there as the project seems unsupported or sucked dry out by Platform.com. I hope it to fully reincarnate in Red Hat's HPC proposal and that it eventually makes its way into CentOS.