On Tue, Jun 28, 2005 at 08:11:40PM -0400, Avtar Gill wrote:

Bryan J. Smith b.j.smith@ieee.org wrote:

...

People are building servers with CentOS. This stuff is pertinent.

I couldn't agree more. I'm not keeping tabs here but it seems to me that a good majority of Bryan's posts offer helpful insight into hardware issues that one might encounter while using CentOS.

There have been some threads that I didn't bother reading since they weren't relevant to me so I just ignored them. That's the key here; if you don't like what's on TV then change the channel. Initially I didn't want to say anything but some of these comments are just too harsh and unfair.

And a 'hear hear' from here. I've followed some of the recent threads with great interest, and skipped the bits that seemed less relevant to me. The threads have been labelled OT - just skip or filter them people. In the absence of an offtopic list I don't see the posters can really do too much more.

Off topic maybe, but -1 troll. :-)

Cheers, Gavin

On Tue, 2005-06-28 at 21:03 -0400, Peter Arremann wrote:

But at least for me, anyone who posts without even trying to find and shred of prove and we're just simply supposed to believe every word, sorry, that's a troll. If Byran had shown any form of prove - any docs that would show he is right, I would agree with you. But until then, I can just agree with Chris. Lots of hot air and no substance.

Dude, I haven't seen any proof from you. All we've gotten was 100% out-of-context assumption. You can look up board specs, that has *0* to do with this? Traces don't mean squat, we've been through this.

I've contacted the gentleman at AMD regarding the patch, because I can't find his post in the LKML archives (I already admitted off-the-bat, before anyone else, it was _not_ in the February 17, 2004 thread).

I'm also going through the kernel's VM right now. There are no less than 5 major models to x86/x86-64. And it _does_ require you to use a "BIOS hack" that is COMPLETELY INCOMPATIBLE with PAE36 OSes.

Ironic that I say that while I agree with Bryan so much on 3ware and board design, uch?

On Tue, 2005-06-28 at 21:26 -0400, Peter Arremann wrote:

They all have 0 to do with the problem? What kind of document would you accept?

One that you will _not_ find on developer.intel.com.

Intel will _not_ tell you how to hack the Athlon MP so address PAE36 linearly at the TLB, because it's processors can't do it prior to the Xeon MP with EM64T.

Alright - tell me what you want to see :-)

It's somewhere in the AMD system developer manuals, possibly ones not publicly available.

We're not talking board-level and we're not talking programmer either. We're talking about throwing the so-called "32-bit" Athlon [MP] in a mode that _breaks_ GTL, something that would _not_ normally be supported.

Thank you - 32bit only please... we all agree that AMD64 can address more than 4GB without issues.

This is a hack for so-called "32-bit" Athlon MP mainboards!

The so-called "32-bit" Athlon and so-called "64-bit" Athlon 64 / Opteron are of the _same_core_ design for the _same_ platform, EV6. I posted on the 3-generation "heritage" of Intel (i386-486, Pentium, PPro-P4) and AMD (386-486, Nx586-686/K5-K6, Athlon-Opteron). You don't design cores "on-a-dime," but for 5-7 years of lifespan (although the PPro is really old, largely because Itanium _was_ Intel's 4th gen design!).

Athlon 64 / Opteron just moves more of the "traditional northbridge" into the CPU, doubles the XMM registers and makes the ALU fully 64-bit. Most of the "northbridge" changes were already in the so-called "32-bit" Athlon, because EV6 is a 3-16 point _crossbar_ switch, not a "hub" like Intel. Because the CPUs in Athlon MP talk over _separate_, _switched_ interconnects, they must have some management units in the processors, not the "single point-of-contention chipset."

A64/Opteron merely turns this into a "partial mesh" instead of a "single switch." I.e., instead of "switching" in the "single chipset," you now "switch" in the individual CPUs. The CPUs _always_ acted "independent" -- even in Athlon MP, right into the EV6 switch.

The addressing is still 100% the same! Even the addressing registers -- 16-bit segment + 32-bit offset are the _same_! There is just now an official memory model called "Long Mode" -- the segment register becomes bits 32-47. In PAE36, the segment register is bits 4-36, which bits 4-31 being a "two's complement" with the offset register.

Now that's just the "programmer" level.

GTL was built for 32-bit. It made _no_sense_ for Intel to modify GTL until recently, because the underlying PAE36 model required paging in the OS anyway. I.e., why add all the logic to do linear addressing in GTL beyond 32-bit if there was no OS to do it?!?!?! Besides, IA-64 was the future, right?

[ Interconnects and memory addressing are _not_ things you can "do on a dime." It took Intel years to develop GTL, and Digital years to develop EV6. And it took years for AMD to adopt EV6 for GTL compatibility. ]

Athlon, including 32-bit Athlon, was AMD's first design that was _not_ GTL compatible at all! That means AMD had to add all sorts of GTL compatibility in to the chipset, CPU, etc... Since they already had a 40-bit interconnect anyway, they decided to support legacy PAE36 GTL as well as 32-bit GTL. That way it could use legacy OSes up to 64GiB. When these legacy PAE36 OSes run, they use Athlon MP in the same way Intel does above 4GiB, paging.

That was _until_ this "hack." It requires the BIOS to setup the EV6 interconnect in a way that _breaks_ GTL. That means the OS has got to know how to use it. Athlon MP mainboards with this hack are _rare_ (I'm still trying to find the e-mail which has this short-list).

Now that x86-64 is here, Intel was _finally_ given a reason to make GTL work above 4GiB. They have now done so in the new 40-bit implementation that Xeon MP uses. Linux/x86-64 takes advantage of this.

But on Linux/x86, when you break 4GiB, the paging must accommodate. What Intel doesn't have on its GTL/x86 that AMD/x86 does is a native, linear 40-bit TLB capability. Again, for Intel, it would have been a waste of transistors, because paging is how a 32-bit OS _must_ work for PAE36 -- or so it seemed.

On AMD, they already had >32-bit to support the EV6 interconnect. EV6 was _not_ designed for x86, but AXP. _All_ EV6 components are 40-bit compatible, they have to be for the specification, including even the so-called "32-bit" Athlon interconnect logic.

AMD had to add logic to support for GTL. They added PAE36 because they already had the address space to spare. _Every_other_ x86/PAE36 OS uses it with paging. This Linux hack is aware that the core TLB is designed for _linear_ >32-bit, when the hardware must be configured in such a way that is _completely_incompatible_ with GTL, including PAE36.

Again, I'm waiting on the technical information from a foremost Linux source at AMD. He'll understand it better than I.

-- Bryan

P.S. There is this farce out there that AMD64 allows 64-bit addressing. It does _not_. It allows PAE52/4PiB, PAE36/64GiB and 32-bit/4GiB programmatic-virtual, 48-bit/256TiB programmatic-physical and 40- bit/1TiB interconnect-linear. When running a PAE36 OS, it will linearly address up to 36-bit/64GiB, using the native, linear EV6 interconnect. That was (essentially) backported with this hack to so-called "32-bit" Athlon, and implemented on a handful of Athlon MP mainboards. Intel x86/GTL+ is _not_ capable of this, because it _violates_ how the CPU GTL + talks to the MCH in Intel's own specs -- _until_ EM64T came out (and even then there are still some issues prior to the new 40-bit Xeon MP).

P.S.S. The i486 TLB has _always_ been capable of 48-bit/256TiB "virtual addressing." It was just always "normalized" into 32-bit physical addresses. PAE36 just normalizes them into 36-bit physical addresses, although the PAE36 OSes still use a 32-bit offset register, which requires the "paging." What this "hack" does is take advantage of a non-GTL compatible mode of the Athlon, just like the A64/Opteron, and avoids paging at the TLB (IIRC).

On Tue, 2005-06-28 at 20:59 -0500, Bryan J. Smith wrote:

The addressing is still 100% the same! Even the addressing registers -- 16-bit segment + 32-bit offset are the _same_! There is just now an official memory model called "Long Mode" -- the segment register becomes bits 32-47. In PAE36, the segment register is bits 4-36, which bits 4-31 being a "two's complement" with the offset register.

Ugh, I've gotta correct this massive _error_.

In PAE36, segment register = bits 20-35 (not 4-36 -- doh!), two's complement with 20-31 (not 4-31 -- doh!), but bits 32-35 are _used_ for up to 64GiB.

Now that's just the "programmer" level.

I'm also going through the kernel's VM right now. There are no less than 5 major models to x86/x86-64. And it _does_ require you to use a "BIOS hack" that is COMPLETELY INCOMPATIBLE with PAE36 OSes.

...

Ironic that I say that while I agree with Bryan so much on 3ware and board design, uch?

I have a question here, is the performance issue to deal with PAE36 to do with the way memory is organized? Note, my knowledge of >32bit extensions is very limited - so I'm just guessing about how it could be...

ie. in normal Intel PAE36 you select and page in 8 512MB linear areas (from a total max of 36bit / 64GB physical RAM) into the normal 32bit address space, and these are the 4GB you can access, if you want to access some other memory then you have to replace a full 512MB block with a different 512MB block. This makes the top 4 (for 36bit) or 8 (for 40bit) address bus lines derived from the top 3 bits of a 32bit address by some very small nearly external to the CPU lookup table (ie. it's tacked on almost as an afterthought?)

while in this extended athlon mode, due to the processor being >32bit internally and not just externally you can somehow page in any 4KB page from anywhere in the 36bits of address space (or even 40bit???) into any spot into physical memory. So you still have a 32bit address space, but it's not limited to only accessing 8 512MB blocks, instead it accesses any 4KB pages from the whole of physical memory in any order?

Is that it? Because if it is this would make it obvious why the second is 'truely' 36 or 40bit while the first is a hack :) [although the second would be considered the hack since the first was first...]

Cheers, MaZe.

Johnny Hughes wrote:

And I have never seen Bryan make a harsh post, though he has received several.

If my post seemed harsh, I apologize to everyone (including Bryan). That was not my intent. But please have some pity on those of us that don't have the time/space budget to inhale some of these dry, repetititve (dare I say boring?) arguments. The S/N here used to be unusually high and I suppose I'm just grumpy to see it falter.

Cheers,

C

-}BTW, Bryan, every time you say "dude" all I can think about is the -}spokes person for Dell ;)) -}Best Regards, Jon McCauley

naw.... wasn't it the "Jeff Spicoli" character in Fast Times at Ridgemont High? ;->

- rh

lol, tlols thatsoo ....I remember ;))

Best Regards, Jon McCaulMcCauley>

_______________________________________________

CentOSCentOSng list CentOSCentOSscentos>

http:/httpts.centoscentosailman/listinlistinfoscentos Ok i kOKw my post Started this Second round of Beating the S&*t out of a topic, But it was not intended to point any fingers at anyone. Just wanted to vent my Frustration out about beating the crap out of a subject so bad that it starts appearing on the opposite side of the earth. this remark is not pointed at anyone but if you only take a few seconds, minutes before you decide to post your knee jerk responses then i think it will be a lot nicer and no one will be starting a flame war in this group. that has been happening lately in this group, and it is not one persons fault either. this is in my honest opinion.....So please dont tdon'tthis as an attack or a excuse to use it as an attack because it is NOT!!!!!!

Steven

"On the side of the software box, in the 'System Requirements' section, it said 'Requires Windows or better'. So I installed Linux."

Quoting Robert Hanson roberth@abbacomm.net:

-}BTW, Bryan, every time you say "dude" all I can think about is the -}spokes person for Dell ;)) -}Best Regards, Jon McCauley

naw.... wasn't it the "Jeff Spicoli" character in Fast Times at Ridgemont High? ;->

The image I have in my head is Jeff Lebowski (The Dude) character from the movie "The Big Lebowski". Sorry Bryan, but every time you say "dude", that's the face, personality and everything else that gets associated with your name ;-)

Note to myself. Do not post in OT threads, do not post in OT threads, do not post in OT threads... Darn, too late...

---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

On Tue, 2005-06-28 at 22:27 -0400, Peter Arremann wrote:

It wouldn't bother me as much if he didn't go with this all over the place.

Because it's _very_complex_ to understand.

Even you wanted to break it down to board layout. Of course there is 36-bit addressing, because you can't access up to 64GiB without it. ;->

It's similar to someone screaming "why doesn't this 256MB DIMM work!" when your mainboard supposedly supports 256MB DIMMs, even a higher SDRAM technology than the DIMM. It has nothing to do with the traces and support, but how it's actually implemented in the memory controller (or in our case, the TLB-page table).

Now that's an over-simplification, but as I've explained before,

Search google for PAE52 and filter out the non technical docs (or try PAE52 +athlon). Only Bryan's posts.

Dude, hit _both_ AMD _and_ Intel "programmer" guides. They discuss how PAE52 is a 7-part addressing model for 48-bit addressing that is also compatible with PAE36 model.

It's not only how a PAE52 OS can run PAE36 (and 32-bit) binaries, but also why PAE52 program's can't use PAE36 libraries or vice-versa without software-level translation.

PAE52 is the model _well_known_ among people building x86-64 software.

Search for Athlon64 and EV6 - and the only one that says it is the same bus and not hyperthreading (with a few legacy remainders of EV6 like the location of certain memory locations) - again only Bryan. People that say AthlonMP can go >4GB linear? Only Bryan.

I'm frantically trying to find the post from the LKML.

Note, AFAIK, the Athlon MP was _never_ implemented in a board that supported over 4GiB. But _all_ so-called "32-bit" Athlons _do_ support PAE36, including the flag being set in the processor as well so _all_ PAE36 OSes can see it. It goes back to its EV6 roots.

Posts that don't acknoledge that Athlon64 and 32bit athlon are indeed based on the same core but have major differences (i.e. 64bit extensions, ht bus rather than ev6, ...) ? you guessed it - only Bryan.

Athlon 64/Opteron _uses_ EV6! EV6 is tunneled over HyperTransport! It's how other bus logic, including GTL on Intel platforms, is tunneled.

People think Athlon 64/Opteron implements HyperTransport like a traditional MCH. Indeed, both the Apple G5 as well as HyperTransport between the "northbridge" and "southbridge" for Intel GTL+ processors on nVidia and SiS do this.

But in the Athlon 64/Opteron, the traditional "northbridge" is moved into the CPU. The CPU handles the EV6 logic internally, and has direct connections to both the DDR SDRAM channels as well as its own HyperTransport interconnect to other CPUs.

By your "simplification," the "narrow bus" design of HyperTransport would mean its _incapable_ of supporting 64-bit GTL or EV6 logic. Dude, you just have to dig into the design of the core to understand this. The info is out there man!

Anyhoo, the 40-bit physical addressing limitation of Ev6 comes into play in x86-64 as it is currently implemented in both AMD64 _and_ EM64T. Intel has extended GTL to support the same limitations as EV6.

I'm sure the 4th generation of both Intel x86 and AMD x86 will overcome this.

Sorry, but for me, that's a little weird... So many engineers in this world and only one person can see the truth?

Dude, get off the "truth" non-sense.

PAE52 is a reality. The foundations and limitations of GTL and EV6 are reality.

Most people don't understand how very different the Athlon is because Athlon only emulates most aspects of x86/GTL, even though it does not use it natively. And Athlon 64/Opteron is an evolution of EV6 into its current, multi- point, EV6-addressed NUMA/HyperTransport interconnect.

On Tue, 2005-06-28 at 23:12 -0400, Peter Arremann wrote:

Yes - what exactly was that page in what manual that says "PAE52" ?

AMD Architecture Programmer's Manual Volume 2: System Programming

http://www.amd.com/us- en/assets/content_type/white_papers_and_tech_docs/24593.pdf

Page 146:

"Currently, the x86-64 architecture defines a mechanism for translating 48-bit virtual addresses to 52-bit physical addresses. The mechanism used to translate a full 64-bit virtual address is reserved and will be described in a future x86-64 architectural specification."

Page 147:

"Setting CR4.PAE=1 enables virtual addresses to be translated into physical addresses up to 52 bits long. This is accomplished by doubling the size of paging data-structure entries from 32 bits to 64 bits to accommodate the larger physical base addresses for physical-pages. PAE must be enabled before activating long mode."

48-bit virtual addressing is i386 onward: 16-bit segment + 32-bit register

History ...

The 8086/8086 created a 20-bit "physical address" from a 32-bit "virtual address" by creating a two's complement from the 16-bit segment (bits 4-19) plus the 16-bit offset (bits 0-15). This process of getting 20- bit physical address from a 32-bit "virtual address" is known as "normalizing." If the two's complement is greater than bit 19, then an overflow, exception, etc... occurs.

[ SIDE NOTE: A small "hack" called the A20 line could be used to give another 16-bit 64KB "page" above 1MB, hence the common DOS location in XMS memory, not between 640-1024KB, but _at_ 1024KB. Long story short, it's when the two's complement overflows to A20, because all MSB 10-15 (A15-19) of the segment and MSB 15 (A15) of the register are set. This would result in the A20 line being set, and addresses of 1024KiB(-16) to 1088KiB(-16) being available. ]

The i386 onward normalizes the 32-bit "physical address" from a 48-bit "virtual address" by creating a two's complement from the 16-bit segment (bits 20-35) plus the 32-bit offset (bits 0-31). This 48-bit virtual -> 32-bit "normalizing" is radically speed up in the i486 TLB over the i386 (which does not have a TLB). If the two's complement is greater than bit 31 (32-bit), then an overflow, exception, etc... occurs. There is no A36 "hack" like there was for 8086/8088 though.

PAE is mode in the Pentium Pro onward, supported in GTL+, that uses the "4-bit overhang" (bits 32-35) of the 16-bit segment register to address beyond 4GiB. This would be, in 8086/8088 terminology, an A32- A35 hack. But can only do so in a way that pages into under 32-bit. This is known as PAE36 -- because it is a 36-bit processor address extensions (PAE) model, not a linear address access above 32-bit.

PAE in x86-64 still uses the 48-bit virtual addressing approach. The name "Long Mode" comes from the fact that instead of creating a two's complement of overlapping bits, the 16-bit segment and 32-bit offset are connected "Long" MSB of offset against the LSB of the register. So now 48-bit virtual addresses are actually 48-bit wide, not "normalized" to 32-bit or, using PAE (36-bit), 36-bit.

Where 52-bit comes into play is in the paging. The paging not only can do these 48-bit "Long" addresses, but also legacy 32-bit and PAE (36- bit) "Normalized" addresses. It's a bit complex, but the _full_ compatibility of running _both_ 48-bit "long" programs and 32/36-bit "normalized" programs in the same, compatible paging space is well detailed in the manual (and the choice of 52-bit wasn't arbitrary).

This 48/52-bit mode requires PAE, designed for simultaneous, paging- level compatibility with not only 32-bit, but also 36-bit (PAE36) "normalized" programs.

[ SIDE NOTE: A future x86-64 processor will offer a full 64-bit virtualized addressing. For now, it's 48-bit -- the "Long" version of a 32-bit offset register with the legacy 16-bit segment register in front of it -- _not_ overlapping/normalized. ]

[ SIDE NOTE: A future x86-64 processor will offer a full 52-bit physical addressing. For now, it's 40-bit -- i.e., the physical limitation of the EV6 interconnect. ;-> ]

That's why we refer to it as PAE 52-bit, or PAE52 for short. To differentiate from a processor that only does PAE 36-bit, PAE36 for short. Most of the time, AMD calls it x86-64 PAE.

-- Bryan

P.S. I also noted that the 4MiB Page Size actually allows up to 40-bit addressing with 32-bit offsets. I wonder if that's how the Xeon MP's page table directly and linearly addresses up to 40-bit/1TiB? As you can see from the table on page 146, AMD still uses 4KiB paging (although it _does_ offer a 2MiB paging mode -- which would definitely be GTL _incompatible_).

On Tue, 2005-06-28 at 22:50 -0500, Bryan J. Smith wrote:

AMD Architecture Programmer's Manual Volume 2: System Programming http://www.amd.com/us- en/assets/content_type/white_papers_and_tech_docs/24593.pdf ... [ SIDE NOTE: A future x86-64 processor will offer a full 52-bit physical addressing. For now, it's 40-bit -- i.e., the physical limitation of the EV6 interconnect. ;-> ]

Before someone screams I didn't back that last part up ...

Page 31 (and countless other parts of the manual):

"Implementations can support fewer than 52 physical-address bits. The first implementation of the x86-64 architecture, for example, supports 40-bit physical addressing in both long mode and legacy mode."

On Wed, 2005-06-29 at 00:14 -0400, Peter Arremann wrote:

I was simply wondering if you were referring to the long modes as defined by AMD

There are several "Long Modes" -- some are not yet implemented.

or if that was yet another unknown concept - maybe you should use the acronymes as the manufacturers do instead of inventing your own :-) keeps you from having to type long emails like this :-)

Dude, I'm _not_ the only person who refers to it as PAE52 or PAE 52-bit.

AMD and Intel _both_ refer to both PAE 52-bit and PAE 32-bit as just "PAE" for short, because the bit that enables it is the same.

Some just like to differentiate. I do. I'm not the only one either.

Let's assume that I have a machine with at least 4 CPUs. In case that one of these fails what are my options. I mean, could I take that cpu offline and maybe ( I say maybe) I could change it without taking down the server ? What is the suggested procedure? Is this the same procedure no matter what cpu architecture I have (Itanium, or PPc or whatever ) Thanks in advance for the attention

Zaharioudakis Nikos

Yep, I'm not aware of any "affordable by mere humans" implementations of hot swappable cpus either.

Cheers,

C

William Warren wrote:

AFAIK motherboards that support hot swapping are in the big iron range. CPU hotswaping IME is limited in the commodity space.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Btw, don't quote me on this one :) I'm only 90% sure of the hotswapping capabilities, and less than 50% sure about the price :)

Only used the machine for 1 week, and that was about 2 years ago.

[]s

On Wed, Jun 29, 2005 at 02:29:20PM -0300, Rodrigo Barbosa wrote:

I remember using an IBM rackmounted (4U ? 6U?) Xeon based machine with CPU Hotswaping.

On the US$35K range, if I recall.

On Wed, Jun 29, 2005 at 09:16:19AM -0400, Chris Mauritz wrote:

...

Yep, I'm not aware of any "affordable by mere humans" implementations of hot swappable cpus either.

Cheers,

C

William Warren wrote:

...

AFAIK motherboards that support hot swapping are in the big iron range. CPU hotswaping IME is limited in the commodity space.

-- Rodrigo Barbosa rodrigob@suespammers.org "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

- -- Rodrigo Barbosa rodrigob@suespammers.org "Quid quid Latine dictum sit, altum viditur" "Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)