On Wed, 2005-06-29 at 00:26 -0400, Peter Arremann wrote:
RedHat and others don't want to have to support two separate kernels - so they limit IO to the lowest 4GB no matter if you're running an Opteron or EM64T.
On Wed, 2005-06-29 at 00:01 -0500, Bryan J. Smith wrote:
? I was unaware this is how they handled Opteron. I thought Red Hat _dynamically_ handled EM64T separately in their x86-64 kernels, and that was a major performance hit.
Looking again at the release notes ...
http://www.centos.org/docs/3/release-notes/as-amd64/RELEASE-NOTES-U2- x86_64-en.html#id3938207
From the looks of it, it's not just whether memory mapped I/O is above
4GiB, but _any_ direct memory access (DMA) by a device where either the source or destination is above 4GiB. I.e., the memory mapped I/O might be below 4GiB, but the device might be executing a DMA transfer to user memory above 4GiB.
That's where the "Software IOTLB" comes in, _only_enabled_ on EM64T.
If I remember back to the March 2004 onward threads on the LKML, that's how they dealt with it -- using pre-allocated kernel bounced buffers below 4GiB. A Linux/x86-64 kernel _always_ uses an I/O MMU -- it is just software for EM64T if either the source or destination address of a DMA transfer is above 4GiB.
I don't think it really matters where the memory mapped I/O is itself. Although it obviously is advantageous if it is setup under 4GiB on EM64T -- because it would only need the "bounce buffers" when a DMA transfer is to user memory above 4GiB, instead of _always_ if the memory mapped I/O was above 4GiB.