New subject: [OT] Memory Models and Multi/Virtual-Cores -- WAS: 4.0 -> 4.1 update failing

25 Jun 2005


      From: Peter Arremann loony@loonybin.org
...
And that's exactly the part I don't get - if you have a 32bit address model 
then you have to use PAE of some sort (compatible to Intel PAE36 or not) to 
get to address more than that memory...
The BIOS hack basically says "don't be stupid" >4GB.  It tells the Athlon MP
to the PAE36 addresses linearlly, instead of paging in GTL compatible fashion.
With that said, do you mean _beyond_ 36-bit/64GiB?
First off, I don't think there ever was an Athlon MP mainboard with more than
64GiB (or more than 32GiB for that matter).
Secondly, programs are written to PAE36, so they can only address 64GiB
(if not 32-bit/4GiB).
*BUT* is it possible for the Athlon MP to still address more than 64GB
and "window" (not page) PAE36/64GiB programs?
That's a good question, and it's the #1 reason I'm trying to find that post
in the LKML.
In reality, if the hack uses Linux/x86-64, it could very well be that it puts
the Athlon MP in "48-bit/256TiB Long Mode" and translates addreses linearlly.
The Athlon MP wouldn't offer PAE52, no.  But it _could_ offer PAE36 windows
in a 48-bit (40-bit physical EV6) space.
Remember, the Athlon/MP and Athlon64/Opteron are the _exact_same_
core design, including the 40-bit/1TiB EV6 interconnect for addressing
outside the chip (even if the latter tunnels over HyperTransport between
CPUs and I/O to other memory, instead of using a "crossbar switch").
...
again - how do I generate a >32bit address when using a 32bit
address model without pages? :-)
Remember, in i386 (and, subsequently, the i486 TLB), the segment
register is offset at bit 20.  That means the most significant bit
(MSB) of the segment register actually has a value of 2^35 (32GiB)!
Segment:  1111 1111 1111 1111 ____ ____ ____ ____ ____
 Offset:  ____ 1111 1111 1111 1111 1111 1111 1111 1111
Before PAE36 in the i686, Intel would thrown an exception if you
set _any_ of the top 4-bits of the segment register, or if the
5th MSB of the segment was set and either the MSB of the offset,
or the previous bits "overflowed" when "normalized" into the physical
address.  That would result in a physical address >32-bit/>4GiB.
In i686 with PAE36, Intel now uses those >32-bit/>4GiB addresses
as "pages" into <32-bit/<4GiB addresses.
In EV6 with the incompatible "BIOS hack" and a supporting kernel,
the Athlon MP's TLB doesn't page, it allows _direct_, _linear_ access
above 32-bit/4GiB.
Digital EV6 is capable of physically addresssing 40-bit.
Intel GTL is only capable of 32-bit.
When Athlon is in GTL compatibility mode, it only does 32-bit for full
compatibility.  But you can tell it to use native, 40-bit EV6 with this
BIOS hack.  The BIOS hack sets up the crossbar registers which the
OS must then support.
I'll confirm how this is being done.  I believe it actually leverages
the Linux/x86-64 kernel.
...
Once I have that address, throwing it out on the bus is 
easy - but how do you generate that address without a PAE?
That's where the hack comes in.
Remember, _all_ kernels in _all_ OSes use the TLB, hence why
an i486 ISA compatible is virtually required these days (or they
do it in the kernel's software).
Normally the Athlon TLB will do the same thing as any Intel GTL,
so any such i686 kernel with PAE36 will normally do the same thing.
But the BIOS hack puts the Athlon TLB into its _true_ 40-bit EV6
addressing, and the kernel then knows of this and uses it.
...
So you're simply talking about being the ability to output an
address longer than 32bit on the bus?
It's more than just that.
It's the combination of presenting PAE36 to _user_ software,
while using _linear_ addressing physically.
Intel i686+GTL already does PAE36, and pages in addressings
above 32-bit.  Athlon+EV6 normally just emulates that, right
down to the TLB.
This hack doesn't, it unleashes the Athlon as it's meant to work
on a 40-bit interconnect designed for 64-bit Alpha processors.
And that includes having the kernel drive the TLB to just take
those >32-bit "normalized" addresses that would normally be
"paged" and just act like it's linear space.
As far as above 36-bit, I'm not sure.  I'll check on that.
...
on AMD64, yes, thats for sure... but you were referring to
32bit athlons in the statement I'm trying to understand.
And what I'm saying is that Athlon is Athlon.  It's the same
core, same 40-bit EV6 interconnect.  Everything else is
compatibility.
On Athlon 64, you can run a PAE52 kernel (on its 40-bit
physical platform) and run PAE36 applications, giving linear
address space to them all while they think paging is being used.
What this hack does is do the same thing for Athlon MP
-- marketed as "32-bit," but the _same_ 40-bit capable,
_physical_ platform.  EV6 is EV6, designed for 64-bit Alpha,
and AMD couldn't "cripple" it.  They use the same board logic
and cores, whether a 32-bit Athlon or 64-bit Alpha are used.
But doing anything but traditional, GTL 32-bit/PAE36 would
normally _break_ hardware and OSes.  Unless you have a
hack to the BIOS which enables this "Linux" memory mode,
and a kernel which supports it.
Again, I'll find the post, which will lead me to the tech info.
--
Bryan J. Smith   mailto:b.j.smith@ieee.org

Re: [CentOS] [OT] Memory Models and Multi/Virtual-Cores -- WAS: 4.0 -> 4.1 update failing