[CentOS] Higher memory requirements for 64 bit?

I have a dual Xeon 3.2GHz XDMCP server serving out about 50 Gnome
desktops to thin clients.  I currently am running 32 bit with 4GB of
memory.  Performance is fine, normally, and the machine runs with about
3GB in swap.  However, if a user app suddenly sucks up the maximum
allowed rss and virtual or 256MB/process as specified by ulimit, the
resulting swap drags the machine down to the point that people think it
is locked up.

I have another 4GB of memory on order.  But I suspect that part of the
problem may be running out of memory in zone-normal, which I know will
cause swap storms. (An oops in the log file suggests that is where
memory was tight.)  I had low memory protection set to 150MB, and have
increased that to 256MB. (I had to increase it to at least 135MB
previously, else dd if=/dev/zero of=/tempfile.tmp would start a swap
storm that would bring the machine to its knees.)

More users will be added to this machine, and I am worried about having
an ever higher percentage of this (16GB maximum) machine in zone high.

So I am considering moving it from 32 bit CentOS4 to 64 bit CentOS5 or
64 bit Fedora 7.  But a cursory test of the 32 and 64 bit versions of
firefox suggest to me that, just opening them up and going to
google.com, the 64 bit version uses about 50% more memory than the 32
bit version.

Could someone please comment on how much of an additional memory hit I
might expect?  The VM risks and benefits that I might be taking?
Really, *any* commentary about this scenario would be appreciated.

Thanks,
Steve Bergman