I'm spec-ing a development server for some Haskell hackers. In particular, they need to be able to build and run the Glasgow Haskell Compiler (ghc), the x86_64 port of which isn't quite there yet.
My current thought is to get an Opteron-based system, but load it (initially, at least) with the 32-bit i386 version of CentOS. That buys me near-term ghc compatibility. In the future, should ghc become 64-bit clean, I'll be able to upgrade to a full 64-bit OS.
The alternative is to build the server around high-end ia32 chips and leave the Opteron for another day.
Does anyone have experience running a 32-bit-only OS on Opterons? Are there any gotchas?
Paul,
The Opteron is completely 32bit AND 64bit compatible. You will be able to run a 32bit OS as flawlessly as you run a 64bit OS. However, one of the great things about the Opteron is that you can run 32bit binaries natively within the 64bit OS! ( and by the way the Opteron is the HIGHEST end ia32 CPU you can buy! ;) )
... thanks!
- doug
************************************************************************ Doug Zeman \ \ | __ \ ____ | Phone: 408 7747674 CAD Sys Eng II / \ | _/ | | | | Cell : 408 7187466 CA MicroProc Div ____ \ | | | | | | Fax : 408 7747811 Sunnyvale, CA _/ __| _| _____/ ____/ | doug.zeman@amd.com ************************************************************************
Paul Heinlein wrote:
I'm spec-ing a development server for some Haskell hackers. In particular, they need to be able to build and run the Glasgow Haskell Compiler (ghc), the x86_64 port of which isn't quite there yet.
My current thought is to get an Opteron-based system, but load it (initially, at least) with the 32-bit i386 version of CentOS. That buys me near-term ghc compatibility. In the future, should ghc become 64-bit clean, I'll be able to upgrade to a full 64-bit OS.
The alternative is to build the server around high-end ia32 chips and leave the Opteron for another day.
Does anyone have experience running a 32-bit-only OS on Opterons? Are there any gotchas?
Thanks Doug and Bryan! I really appreciate the feedback. You've confirmed my hunch, but the question was (to me, anyway) worth asking before laying out a few thousand dollars for a machine :-).
Paul Heinlein heinlein@madboa.com wrote:
I'm spec-ing a development server for some Haskell hackers. In particular, they need to be able to build and run the Glasgow Haskell Compiler (ghc), the x86_64 port of which isn't quite there yet. My current thought is to get an Opteron-based system, but load it (initially, at least) with the 32-bit i386 version of CentOS. That buys me near-term ghc compatibility. In the future, should ghc become 64-bit clean, I'll be able to upgrade to a full 64-bit OS.
AMD designed x86-64 to fully support running 32-bit applications _regardless_ what actual addressing model is being used by the kernel. In other words, the kernel can run the processor addressing in either legacy 36-bit mode (64GiB) or 52-bit mode (16EiB) mode, and still run 32-bit applications. The key in how AMD does this is by continuing to use 48-bit virtual addresses for both, which is how the Translation Lookahead Buffer (TLB) of the i486 onward works. The 52-bit addressing mode also uses the same Processor Address Extensions (PAE) approach as 36-bit mode.
[ NOTE: AMD has reserved a true "flat" 64-bit mode for future x86-64 implementations. Until then, the 48-bit virtual/52-bit physical register is how current Athlon64/Opterons are implemented. They are actually a 40-bit (1TiB) physical platform limitation, which is tied to its current interconnect logic (long story). ]
When you run a Linux/x86-64 kernel, you get the 52-bit (flat) mode. When you run a Linux/x86-64 kernel, you get the 36-bit mode (LOWMEM/HIGHMEM).
If you run 32-bit applications either, you will have 4GiB user-space limitations. You must also call only 32-bit libraries from 32-bit applications. That's about the only issue with x86-64 releases, the fact that you may require an i386 library, instead of a x86_64 library.
The alternative is to build the server around high-end ia32 chips and leave the Opteron for another day.
It is never recommended because of performance issues with 32-bit/4GiB models -- especially as you pass 1GiB of memory, and definitely when you exceed 4GiB.
Especially if you have a lot of I/O.
Does anyone have experience running a 32-bit-only OS on Opterons? Are there any gotchas?
None on Linux/x86-64 I'm aware of. GNU has been largely 64-bit clean since the mid-to-late '90s (thanx to Alpha, MIPS and other developments). This is unlike Windows (which was always released as 32-bit on Alpha, MIPS, etc..., long story).
Windows XP 64-bit Edition and Windows 2003 Server have many because the Win32 codebase is _not_ 64-bit clean like GNU has been for a long time. Microsoft's Windows 64 OSes use largely Win32 on Win64 (WoW) subsystem which is not very fast, instead of providing true Win64 services and programs.
The Opteron is currently a true 40-bit (1TiB) interconnect, 48-bit (256TiB) virtual, 52-bit (4TiB) register platform, including offering an I/O MMU for "safe" I/O transfers above 4GiB of memory (including any I/O card memory mapped to ranges above 4GiB, or using DMA to programs in memory above 4GiB).
In fact, for servers with 4+GiB of memory, Intel EM64T processors have the _same_ I/O limitations ("bounce buffers") as IA32 processors, whereas AMD Opterons do not.
On Mon, 1 Aug 2005, Paul Heinlein wrote:
Does anyone have experience running a 32-bit-only OS on Opterons? Are there any gotchas?
We've been running RHEL3 in 32 bit mode on Sun v40z's (rebadged Newisys boxes) for almost a year at work. No issues other than the usual goofiness about Broadcom network chips, etc. Note: I would NOT recommend the v40z or the straight Newisys box. They are poorly implemented platforms.
------------------------------------------------------------------------ Jim Wildman, CISSP, RHCE jim@rossberry.com http://www.rossberry.com "Society in every state is a blessing, but Government, even in its best state, is a necessary evil; in its worst state, an intolerable one." Thomas Paine
On Mon, 2005-08-01 at 21:28 -0400, Jim Wildman wrote:
Note: I would NOT recommend the v40z or the straight Newisys box. They are poorly implemented platforms.
Can you expand on this? They seem to be the most prolific 4-way Opteron design available. What am I missing? The quality of the design?
On Mon, 1 Aug 2005, Bryan J. Smith wrote:
On Mon, 2005-08-01 at 21:28 -0400, Jim Wildman wrote:
Note: I would NOT recommend the v40z or the straight Newisys box. They are poorly implemented platforms.
Can you expand on this? They seem to be the most prolific 4-way Opteron design available. What am I missing? The quality of the design?
Sure. The issues I am aware of are around the management processor implementation. The server has an embedded PPC based processor that has the ability to access either the serial port, or the console directly. However, it does neither automatically, out of the box or without some work on the OS side. There are no SNMP MIB's available (that I know of) for accessing the information it may have access to. It is particularly difficult to get Solaris to function with the serial port as 'console'.
This is in contrast to HP's RIB/iLO family of cards, which work out of the box for either direct or web based console access, and with minimal OS configuration can access the serial port. They come with a complete set of MIBs and easily installed OS modules. The latest versions support ssh as well as web, and both text and graphical access allow you to mount remote 'virtual' media for installation, etc.
Part of the issue may be that Sun did not/could not buy the complete package from Newisys. If you look at the Newisys web page, it looks like they have functionality that may come close to HP's, but it is locked up on the Sun boxes. The web server is running, but all roads (that I could find) lead no where. Additionally, the v40z is an orphan. Sun has announced plans to replace them (and their little brothers, v20z) with 'Galaxy' class boxes of their own design.
All of that said, once they are installed, they seem to be rock solid. If you are expecting a very Solaris looking platform for Linux, the v40z fills the bill. If you are expecting a Wintel type platform (ala HP), it doesn't even get in the game.
------------------------------------------------------------------------ Jim Wildman, CISSP, RHCE jim@rossberry.com http://www.rossberry.com "Society in every state is a blessing, but Government, even in its best state, is a necessary evil; in its worst state, an intolerable one." Thomas Paine
On Tue, 2005-08-02 at 00:09 -0400, Jim Wildman wrote:
Sure. The issues I am aware of are around the management processor implementation ... This is in contrast to HP's RIB/iLO family of cards,
So your complaints surround the management processor. Okay, thanx for that heads up.
I was just wondering if you had run into stability or other issues with the implementation itself, which seems to be the ultimate in I/O design.
Additionally, the v40z is an orphan. Sun has announced plans to replace them (and their little brothers, v20z) with 'Galaxy' class boxes of their own design.
Can't wait to see that design.
All of that said, once they are installed, they seem to be rock solid.
Okay, that was my concern because I've setup at least one SunFire V40z personally (although not it's management processor, it was for a small firm, long story), and I didn't have any issues.
If you are expecting a very Solaris looking platform for Linux, the v40z fills the bill. If you are expecting a Wintel type platform (ala HP), it doesn't even get in the game.
If I want Wintel I will go HP, not Sun/2nd-tier.
On Tuesday 02 August 2005 00:09, Jim Wildman wrote:
Sure. The issues I am aware of are around the management processor implementation. The server has an embedded PPC based processor that has the ability to access either the serial port, or the console directly. However, it does neither automatically, out of the box or without some work on the OS side. There are no SNMP MIB's available (that I know of) for accessing the information it may have access to. It is particularly difficult to get Solaris to function with the serial port as 'console'.
I have to agree with you there. Unless you're willing to set up the whole console setup (NSV and all) the box is a pain to deal with. If you set up everything properly, you get SNMP for monitoring at least (which is better than nothing)... If you only have a few of these servers NSV is a huge pain. If you have several racks full, you can actually see the benefits of this design and understand why they did what they did. Until then, you're left with either a useless SP or a setup that is stupid and "unusual" - just something else you need to learn.
The setup you have to do on the OS side is Intel's fault (http://www.intel.com/design/servers/ipmi/) - and not a design that newisys did themselves... if this spec lives on, some day you might have os support by default...
Compay btw, also required you to load software to have fully functional RIB functionality - its just part of their cd set...
This is in contrast to HP's RIB/iLO family of cards, which work out of the box for either direct or web based console access, and with minimal OS configuration can access the serial port. They come with a complete set of MIBs and easily installed OS modules. The latest versions support ssh as well as web, and both text and graphical access allow you to mount remote 'virtual' media for installation, etc.
Yes - that's the keyword... virtual media... That's the one functionality the v40z doesn't support that HP has... Galaxy gladly fixed that issue.
Part of the issue may be that Sun did not/could not buy the complete package from Newisys. If you look at the Newisys web page, it looks like they have functionality that may come close to HP's, but it is locked up on the Sun boxes. The web server is running, but all roads (that I could find) lead no where.
Yep :-( If you have a NSV, update to the latest firmware... fixes quite a bit but still doesn't give you all the functionality HP has... And it gets rid of the SP errors that cause the "press any key to continue" :-D
Additionally, the v40z is an orphan. Sun has announced plans to replace them (and their little brothers, v20z) with 'Galaxy' class boxes of their own design.
I don't think that the v40z being an older design should say much here. After all its been out there for a while and SUN will sell you the boxes even after Galaxy is out.
All of that said, once they are installed, they seem to be rock solid. If you are expecting a very Solaris looking platform for Linux, the v40z fills the bill. If you are expecting a Wintel type platform (ala HP), it doesn't even get in the game.
I can completely agree with this statement. Depending on what we do, we often run the v40z without SP at all - as a plain dumb box without management. The board, the layout of the box (hardware) and all is great... The SP is a huge pain... I get this complaint all the time and sun also knows about it (hence the move to galaxy). The huge difference there will be that the SP is completely redesigned. From what I can tell, they basicly reimplemented what Compaq had feature by feature...
Peter.
Peter Arremann loony@loonybin.org wrote:
I have to agree with you there. Unless you're willing to set up the whole console setup (NSV and all) the box is a pain to deal with. If you set up everything properly, you get SNMP for monitoring at least (which is better than nothing)...
Which is one of the reasons why when I went to setup one of these boxes for a small engineering firm, they weren't willing to pay me to do so.
If you only have a few of these servers NSV is a huge pain. If you have several racks full, you can actually see the benefits of this design and understand why they did what they did.
That's what I figured, based on what I read in the docs, but didn't have the time to try it out.
Until then, you're left with either a useless SP or a setup that is stupid and "unusual" - just something else you need to learn. The setup you have to do on the OS side is Intel's fault (http://www.intel.com/design/servers/ipmi/) - and not a design that newisys did themselves... if this spec lives on, some day you might have os support by default...
Ahhh, now that explains it. I'll have to read more on this (although it suprises me that Intel didn't try to force an XScale on everyone ;-).
Definitely thanx for the link! I figured a tier-2 OEM designer couldn't have come up with all this on their own. I didn't know it was Intel's IPMI, which is what I heard they came up for IA-64/Itanium2 IIRC.
Compay btw, also required you to load software to have fully functional RIB functionality - its just part of their cd set...
Yep. Done that too many times -- both Linux and Windows. I wouldn't call it "straight-forward" either, especially since some of HP's management software/merger with Compaq has required me to update a number of Proliant Service Packs (PSPs) to work correctly/more securely.
I don't think that the v40z being an older design should say much here. After all its been out there for a while and SUN will sell you the boxes even after Galaxy is out.
There's something to be said about three (3) AMD8131 HyperTransport bridges in a system, with four (4) of the slots being dedicated, full 133MHz PCI-X slots.
I can completely agree with this statement. Depending on what we do, we often run the v40z without SP at all - as a plain dumb box without management.
Which is the only time I set on up.
The board, the layout of the box (hardware) and all is great...
Okay, this is what I wanted to confirm. In the one unit I personally deployed, I felt the same.
Paul Heinlein wrote:
I'm spec-ing a development server for some Haskell hackers. In particular, they need to be able to build and run the Glasgow Haskell Compiler (ghc), the x86_64 port of which isn't quite there yet.
My current thought is to get an Opteron-based system, but load it (initially, at least) with the 32-bit i386 version of CentOS. That buys me near-term ghc compatibility. In the future, should ghc become 64-bit clean, I'll be able to upgrade to a full 64-bit OS.
The alternative is to build the server around high-end ia32 chips and leave the Opteron for another day.
Does anyone have experience running a 32-bit-only OS on Opterons? Are there any gotchas?
I planned on doing a similar thing until I found that for some reason, cool & quiet does not seem to work unless in 64 bit mode under linux at least. I decided to go 64 bit because the server was going to be running in my garage and I wanted it as cool as possible.
Has anyone managed to get CPU Throttling/CNQ working on a 32bit linux ?
Bards.
-
Bards1888 -
Bryan J. Smith -
Doug Zeman -
Jim Wildman -
Paul Heinlein -
Peter Arremann