Hey, folks,
I'm working on finding some new compute nodes. What I'm looking for is a 64 core box, with room enough for a lot of RAM. I can get it from Dell, or HP (bleah! a 4U box), but I need to have three quotes, y'know. We've gotten a lot from Penguin in the past, but they're all Supermicro, and we've had a *lot* of problems with the 64 core boxes, so I'm looking for another vendor.
Suggestions? Recommendations? People to run the other way from? (Sun/Oracle is down there *under* "none of the above").
mark
On 29 May 2014 16:26, m.roth@5-cent.us wrote:
Hey, folks,
I'm working on finding some new compute nodes. What I'm looking for is a 64 core box, with room enough for a lot of RAM. I can get it from Dell, or HP (bleah! a 4U box), but I need to have three quotes, y'know. We've gotten a lot from Penguin in the past, but they're all Supermicro, and we've had a *lot* of problems with the 64 core boxes, so I'm looking for another vendor.
Whats wrong with the SuperMicro 64 core boxes? I have never seen any problems with them. I assume you mean AMD, in which case the IPMI sucks but other than that.....
Dell PowerEdge R815's are pretty ok.
Cheers,
Andrew
Andrew Holway wrote:
On 29 May 2014 16:26, m.roth@5-cent.us wrote:
I'm working on finding some new compute nodes. What I'm looking for is a 64 core box, with room enough for a lot of RAM. I can get it from Dell, or HP (bleah! a 4U box), but I need to have three quotes, y'know. We've gotten a lot from Penguin in the past, but they're all Supermicro, and we've had a *lot* of problems with the 64 core boxes, so I'm looking for another vendor.
Whats wrong with the SuperMicro 64 core boxes? I have never seen any problems with them. I assume you mean AMD, in which case the IPMI sucks but other than that.....
They're crap. We've had to send a number of them - we've got something like a couple dozen of them, and at least 4 of them had to be sent back, and got a m/b replacement after they verified, with our test, why they crashed, and one or two got sent back *twice*.
Dell PowerEdge R815's are pretty ok.
I agree. But I do need a third quote, though I suppose I could get a reseller along with Dell and the HP. I was sort of looking for anther vendor that's got 64 cores in 1U (the Dell's 2U).
mark
On Thu, May 29, 2014 at 10:46 AM, m.roth@5-cent.us wrote:
I agree. But I do need a third quote, though I suppose I could get a reseller along with Dell and the HP. I was sort of looking for anther vendor that's got 64 cores in 1U (the Dell's 2U).
If you are looking for density, shouldn't you be using blade systems? I think both HP and Dell have blades that can go to 64 cores.
On 05/29/2014 12:11 PM, Les Mikesell wrote:
On Thu, May 29, 2014 at 10:46 AM, m.roth@5-cent.us wrote:
I agree. But I do need a third quote, though I suppose I could get a reseller along with Dell and the HP. I was sort of looking for anther vendor that's got 64 cores in 1U (the Dell's 2U).
If you are looking for density, shouldn't you be using blade systems? I think both HP and Dell have blades that can go to 64 cores.
I suspect it's total cores per machine + density; I dunno if HP/Dell/anyone else can do that sort of density on cores with blades. I have some IBM 3850s that seem to get close (and I'd gladly give them to all sorts of people that I greatly dislike)
if blades are an option, Cisco UCS blades seems to work quite well for us, but I don't think they get much over the dual proc hexacore range.
zep wrote:
On 05/29/2014 12:11 PM, Les Mikesell wrote:
On Thu, May 29, 2014 at 10:46 AM, m.roth@5-cent.us wrote:
I agree. But I do need a third quote, though I suppose I could get a reseller along with Dell and the HP. I was sort of looking for anther vendor that's got 64 cores in 1U (the Dell's 2U).
If you are looking for density, shouldn't you be using blade systems? I think both HP and Dell have blades that can go to 64 cores.
I suspect it's total cores per machine + density; I dunno if HP/Dell/anyone else can do that sort of density on cores with blades. I have some IBM 3850s that seem to get close (and I'd gladly give them to all sorts of people that I greatly dislike)
if blades are an option, Cisco UCS blades seems to work quite well for us, but I don't think they get much over the dual proc hexacore range.
No, I'm not looking at blades. They're $$$, while the Dell & HP are in the $13k range.
mark
zep wrote:
On 05/29/2014 12:11 PM, Les Mikesell wrote:
On Thu, May 29, 2014 at 10:46 AM, m.roth@5-cent.us wrote:
I agree. But I do need a third quote, though I suppose I could get a reseller along with Dell and the HP. I was sort of looking for anther vendor that's got 64 cores in 1U (the Dell's 2U).
If you are looking for density, shouldn't you be using blade systems? I think both HP and Dell have blades that can go to 64 cores.
I suspect it's total cores per machine + density; I dunno if
<snip> Let me clarify a bit. In the past, we've gotten servers from Penguin, as I said, and they're all Supermicro boxes. I'm not just looking for three quotes - that, I could get easily - I'm looking for other OEMs, in a similar price range (or lower) that might offer us a replacment OEM for Penguin, that offers a) a good price; b) *reliable* systems that they've q/a'd, and c) decent technical support response.
On the last, Dell's hard to beat. Sun/Oracle ranks under "none of the above".... So....
mark
On the last, Dell's hard to beat. Sun/Oracle ranks under "none of the above".... So....
If you can find someone selling these puppies in the US Intel server platforms are rather nice:
http://www.intel.com/content/www/us/en/server-systems/server-board-s4600lh-l...
http://www.intelserveredge.com/ - Try here maybe
mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Andrew Holway wrote:
On the last, Dell's hard to beat. Sun/Oracle ranks under "none of the above".... So....
If you can find someone selling these puppies in the US Intel server platforms are rather nice:
http://www.intel.com/content/www/us/en/server-systems/server-board-s4600lh-l...
http://www.intelserveredge.com/ - Try here maybe
Look nice... but trying to google someone selling *servers* with the board isn't getting me anywhere. I happened across the Asus RS927, but that, after much looking, turns out to only do 32 cores max. Got a clue on models of *server*, from anyone, that uses the Intel board?
mark
Look nice... but trying to google someone selling *servers* with the board isn't getting me anywhere. I happened across the Asus RS927, but that, after much looking, turns out to only do 32 cores max. Got a clue on models of *server*, from anyone, that uses the Intel board?
Intel® Server System R2208LT2HKC4, Intel® Server System R2304LH2HKC
Andrew Holway wrote:
Look nice... but trying to google someone selling *servers* with the board isn't getting me anywhere. I happened across the Asus RS927, but
that,
after much looking, turns out to only do 32 cores max. Got a clue on models of *server*, from anyone, that uses the Intel board?
Intel® Server System R2208LT2HKC4, Intel® Server System R2304LH2HKC
Thanks. Looking around.
mark
On 5/29/2014 10:48 AM, Andrew Holway wrote:
On the last, Dell's hard to beat. Sun/Oracle ranks under "none of the above".... So....
If you can find someone selling these puppies in the US Intel server platforms are rather nice:
http://www.intel.com/content/www/us/en/server-systems/server-board-s4600lh-l...
http://www.intelserveredge.com/ - Try here maybe
those E5-46xx V2 cpus are only 10 or 12 cores, so thats only 40 or 48 cores... m.roth wanted 64, which pretty much means 8 socket and/or AMD land. oh, and the 12 core Intel cpu is $4400 each.
On 5/29/2014 11:20 AM, m.roth@5-cent.us wrote:
Look nice... but trying to google someone selling*servers* with the board isn't getting me anywhere. I happened across the Asus RS927, but that, after much looking, turns out to only do 32 cores max. Got a clue on models of*server*, from anyone, that uses the Intel board?
a bit confusing, but those are in fact 2U servers. http://ark.intel.com/products/61027
barebones, they are like $3200, so add CPU + RAM + Disk...
John R Pierce wrote:
On 5/29/2014 8:46 AM, m.roth@5-cent.us wrote:
I agree. But I do need a third quote,
Lenovo ?
The only rackmounts I see from their site are dual processor. I'll need four - mostly, I see 16-core CPUs in our systems, though some have 12-core (or the weird 10-core). The Asus I saw takes 4 processor, but only 8-core ones.
On 5/29/2014 11:34 AM, m.roth@5-cent.us wrote:
John R Pierce wrote:
On 5/29/2014 8:46 AM,m.roth@5-cent.us wrote:
I agree. But I do need a third quote,
Lenovo ?
The only rackmounts I see from their site are dual processor. I'll need four - mostly, I see 16-core CPUs in our systems, though some have 12-core (or the weird 10-core). The Asus I saw takes 4 processor, but only 8-core ones.
IBM has a 64 cpu (core) Power server ... if you see the price tag, I guarantee you'll freak out.
John R Pierce wrote:
On 5/29/2014 11:34 AM, m.roth@5-cent.us wrote:
John R Pierce wrote:
On 5/29/2014 8:46 AM,m.roth@5-cent.us wrote:
I agree. But I do need a third quote,
Lenovo ?
The only rackmounts I see from their site are dual processor. I'll need four - mostly, I see 16-core CPUs in our systems, though some have 12-core (or the weird 10-core). The Asus I saw takes 4 processor, but only 8-core ones.
IBM has a 64 cpu (core) Power server ... if you see the price tag, I guarantee you'll freak out.
As a comparison, the Penguins that we got a couple-three years ago were in the $11k range, and were 1U Supermicros. The Dell and HP I'm looking at are about $13k, so that's around where I'm looking.
Does anyone have any preferred *vendors* - as I say, we used to like Penguin. Silicon Mechanics? AVADirect? I'm just throwing out names here I've run into while googling.
mark
Andrew Holway wrote:
Does anyone have any preferred *vendors* - as I say, we used to like Penguin. Silicon Mechanics? AVADirect? I'm just throwing out names here I've run into while googling.
How many of these are you after?
The budget's limited - maybe one, maybe two, but next year, possibly more. As a comparison, I've got a cluster with a head node and 22 compute nodes: one 12 core, 10 or 11 48 core, and the rest 64 core.... This is for serious HPC.
mark
m.roth@5-cent.us wrote:
Andrew Holway wrote:
Does anyone have any preferred *vendors* - as I say, we used to like Penguin. Silicon Mechanics? AVADirect? I'm just throwing out names here I've run into while googling.
How many of these are you after?
The budget's limited - maybe one, maybe two, but next year, possibly more. As a comparison, I've got a cluster with a head node and 22 compute nodes: one 12 core, 10 or 11 48 core, and the rest 64 core.... This is for serious HPC.
Oh, and my users' jobs on those run for *days*, sometimes a week or more.
mark
Protein folding, among other things.
Doesn't that stuff parallelise? Quad socket boxes are pretty rare in HPC nowadays as the amount of memory available in a dual socket box is so high.
Infiniband?
mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Andrew Holway wrote:
Protein folding, among other things.
Doesn't that stuff parallelise? Quad socket boxes are pretty rare in HPC nowadays as the amount of memory available in a dual socket box is so high.
Infiniband?
Not on this. But the code's written with parallelization - it uses torque, a std. package, descended from beowolf clusters.
mark
On Thu, May 29, 2014 at 03:35:01PM -0400, m.roth@5-cent.us wrote:
Andrew Holway wrote:
serious HPC.
As opposed to silly HPC? Molecular dynamics huh. Those memory loving SOBs
Protein folding, among other things.
the FoldingAtHome project? with that much horsepower you must be kicking some serious booty!
On Thu, May 29, 2014 at 2:13 PM, m.roth@5-cent.us wrote:
As a comparison, the Penguins that we got a couple-three years ago were in the $11k range, and were 1U Supermicros. The Dell and HP I'm looking at are about $13k, so that's around where I'm looking.
HP blades pop up with a list price around $12k. If you need enough to fill a chassis (and can get a discount), I'd think that would come out in the same neighborhood.
On 5/29/2014 1:11 PM, Les Mikesell wrote:
On Thu, May 29, 2014 at 2:13 PM,m.roth@5-cent.us wrote:
As a comparison, the Penguins that we got a couple-three years ago were in the $11k range, and were 1U Supermicros. The Dell and HP I'm looking at are about $13k, so that's around where I'm looking.
HP blades pop up with a list price around $12k. If you need enough to fill a chassis (and can get a discount), I'd think that would come out in the same neighborhood.
the blade chassis infrastructure is expensive. for supercomputing, the cheap cloud tray computers are probably much more cost effective. bunches of folks, HP and Dell included, are making server trays that have 2 or more complete nodes per 2U chassis, much cheaper than traditional blades, like the Dell C microservers.
I *am* kind of surprised at the 64 core per node thing, single large nodes like that are more typically used as enterprise database servers where you have 100s/1000s of clients doing SQL queries concurrently... What the geophysicists at my son's U are using for seismic processing and such are racks of 2 socket machines with a pair of NVidia Cuda processors each to do the numeric heavy lifting.
if those 64 processor servers are like $14000 or whatever, I think I'd buy 18 of these instead :D http://www.ebay.com/itm/HP-ProLiant-DL180-G6-2U-2X-XEON-HC-X5650-2-66GHz-12x... (ok, that particular chassis is setup as a storage server, with 14 drive bays on a raid card, but you can find lots of similar things in various configurations)
We switched from HP to Fujitsu a couple of years ago, and couldn't be happier. Look into their RX line, I think the RX500 and RX900 (iirc) do 4 and 8 socket.
digimer
On 29/05/14 11:26 AM, m.roth@5-cent.us wrote:
Hey, folks,
I'm working on finding some new compute nodes. What I'm looking for isa 64 core box, with room enough for a lot of RAM. I can get it from Dell, or HP (bleah! a 4U box), but I need to have three quotes, y'know. We've gotten a lot from Penguin in the past, but they're all Supermicro, and we've had a *lot* of problems with the 64 core boxes, so I'm looking for another vendor.
Suggestions? Recommendations? People to run the other way from?(Sun/Oracle is down there *under* "none of the above").
mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I'd also recommend Fujitsu. If you're desperate there is the SGI H2106-G7 in 2U size. On May 29, 2014 8:52 PM, "Digimer" lists@alteeve.ca wrote:
We switched from HP to Fujitsu a couple of years ago, and couldn't be happier. Look into their RX line, I think the RX500 and RX900 (iirc) do 4 and 8 socket.
digimer
On 29/05/14 11:26 AM, m.roth@5-cent.us wrote:
Hey, folks,
I'm working on finding some new compute nodes. What I'm looking foris
a 64 core box, with room enough for a lot of RAM. I can get it from Dell, or HP (bleah! a 4U box), but I need to have three quotes, y'know. We've gotten a lot from Penguin in the past, but they're all Supermicro, and we've had a *lot* of problems with the 64 core boxes, so I'm looking for another vendor.
Suggestions? Recommendations? People to run the other way from?(Sun/Oracle is down there *under* "none of the above").
mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 05/29/14 21:13, John R Pierce wrote:
On 5/29/2014 6:00 PM, Evan Rowley wrote:
I'd also recommend Fujitsu. If you're desperate there is the SGI H2106-G7 in 2U size.
the SGI stuff I've seen has been rebranded Supermicro boxes/boards.
As I think about it, the control node for the UV 2000 looks an *awful* lot like a Penguin....
mark
On 5/30/2014 5:28 AM, Andrew Holway wrote:
As I think about it, the control node for the UV 2000 looks an*awful* lot like a Penguin....
SuperMicro gear is only as good as the server integration company selling it.
bingo. I think too many people buy 'whitebox' supermicro stuff direct and self-integrate, then are surprised when there are issues. Integration needs to include testing. All that integration and testing is why brands like HP are more expensive, you can usually assume its going to work.
Am 30.05.2014 um 19:34 schrieb John R Pierce pierce@hogranch.com:
On 5/30/2014 5:28 AM, Andrew Holway wrote:
As I think about it, the control node for the UV 2000 looks an*awful* lot like a Penguin....
SuperMicro gear is only as good as the server integration company selling it.
bingo. I think too many people buy 'whitebox' supermicro stuff direct and self-integrate, then are surprised when there are issues. Integration needs to include testing. All that integration and testing is why brands like HP are more expensive, you can usually assume its going to work.
True. The thing I hate about HP is that their SSD offerings are IMO a joke.
Not only are they several times as expensive as an equivalent Intel SSD (even taking into account that we don’t pay list-price) but in addition, they perform only half as well (in terms of IOP/s).
I suspect it’s because HP does not include a super cap and thus their SSDs don’t do write-caching (which the Intel does).
Also, due to the fact that they don’t offer a SAS-Controller that does JBOD, you have to setup each drive individually as a RAID0 - which is totally stupid, once you run something like FreeBSD where HPACUCLI is not available. Each failed drive necessitates a reboot then.
I could of course buy an LSI JBOD controller (which would also allow me to buy Intel SSDs) - but what’s the point of buying a HP server then?
IMO, HP does not want you to actually make good use of current-generation enterprise-SSDs - they’d prefer you buy a couple of dozens of P2000 arrays instead…
On 5/30/2014 11:14 AM, Rainer Duffner wrote:
Also, due to the fact that they don’t offer a SAS-Controller that does JBOD, you have to setup each drive individually as a RAID0
hmm?
The HP H220, H221, H220 are SAS2 HBA"s. also the S08e but thats older, and was only sold to support a specific P2000g3 array. AFAIK, the H22x are LSI 2008 based (9211-xx)
Am 30.05.2014 um 20:28 schrieb John R Pierce pierce@hogranch.com:
On 5/30/2014 11:14 AM, Rainer Duffner wrote:
Also, due to the fact that they don’t offer a SAS-Controller that does JBOD, you have to setup each drive individually as a RAID0
hmm?
The HP H220, H221, H220 are SAS2 HBA"s. also the S08e but thats older, and was only sold to support a specific P2000g3 array. AFAIK, the H22x are LSI 2008 based (9211-xx)
Interesting. Thanks a lot.
It’s sometime very difficult to find HP products that aren’t by default in their servers.
AFAIK, the 9211-series card don’t have the „right“ firmware for „IT-mode“ that would be required for an „ideal“ ZFS setup.
I’m not sure if one could flash those - they probably have an OEM firmware.
Maybe I can have one ordered to try it out.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of John R Pierce Subject: Re: [CentOS] OEM suggestions
On 5/30/2014 5:28 AM, Andrew Holway wrote:
As I think about it, the control node for the UV 2000 looks
an*awful* lot
like a Penguin....
SuperMicro gear is only as good as the server integration
company selling
it.
bingo. I think too many people buy 'whitebox' supermicro stuff
direct
and self-integrate, then are surprised when there are issues. Integration needs to include testing. All that integration and
testing
is why brands like HP are more expensive, you can usually assume
its
going to work.
This is exactly why we use SuperMicro for most of our system bases. They provide excellent integrator support but the complete systems MUST be tested and burned-in after assembly.
We build them up with the best components BUT we do a 168 hour burn-in and then test again afterwards. Once a unit has been burned-in for 168 hour with no failures, we have seen less than a 0.1% system failure rate over the first 3 years.
BTW, we see a >5% failure rate of disk drives during burn-in. Infant mortality is horrendous on disks. Especially on Seagate drives. We have a program we wrote in house just for disk / RAID array burn-in and performance testing under CentOS.
I agree with John: Any system is only as good as the integrator that builds, test AND burns in its products.
Seth Bardash
Integrated Solutions and Systems
719-495-5866 Shop 719-495-5870 Fax 719-337-4779 Cell
Failure can not cope with knowledge and perseverance!
On 05/29/14 21:00, Evan Rowley wrote:
On May 29, 2014 8:52 PM, "Digimer" lists@alteeve.ca wrote:
On 29/05/14 11:26 AM, m.roth@5-cent.us wrote:
I'm working on finding some new compute nodes. What I'm looking for is a 64 core box, with room enough for a lot of RAM. I can get it from Dell, or HP (bleah! a 4U box), but I need to have three quotes, y'know. We've gotten a lot from Penguin in the past, but they're all Supermicro, and we've had a *lot* of problems with the 64 core boxes, so I'm looking for another vendor.
Suggestions? Recommendations? People to run the other way from?(Sun/Oracle is down there *under* "none of the above").
We switched from HP to Fujitsu a couple of years ago, and couldn't be happier. Look into their RX line, I think the RX500 and RX900 (iirc) do 4 and 8 socket.
I'd also recommend Fujitsu. If you're desperate there is the SGI H2106-G7 in 2U size.
I'll check that out. We *do* have an SGI... a UV 2000.
mark
On 5/30/2014 5:19 AM, mark wrote:
I'll check that out. We*do* have an SGI... a UV 2000.
try this...
# dmidecode -t 1,2 ... Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: SGI.COM Product Name: ISS3500 Version: ISServer Serial Number: Yxxxxxx UUID: (big messy string) Wake-up Type: Power Switch SKU Number: To Be Filled By O.E.M. Family: 1234567890
Handle 0x0002, DMI type 2, 15 bytes Base Board Information Manufacturer: Supermicro Product Name: X8DTE-F Version: 1234567890 Serial Number: VM1ASxxxxxxxxx Asset Tag: 1234567890 Features: Board is a hosting board Board is replaceable Location In Chassis: To Be Filled By O.E.M. Chassis Handle: 0x0003 Type: Motherboard Contained Object Handles: 0
-
Andrew Holway -
Digimer -
Evan Rowley -
Fred Smith -
John R Pierce -
Les Mikesell -
m.roth＠5-cent.us -
mark
-
Rainer Duffner -
Seth Bardash -
zep