[CentOS] Increased memory usage between 4.8 and 5.5

I'm running into a memory problem when trying to switch some web servers
that are running centos4.8/x86_64 to centos5.5/x86_64.  The new servers
running centos5.5 are using a decent amount more memory than the
centos4.8 servers.

Here's an example from top:

--- centos 5.5 ---
$ uname -a
Linux ws51 2.6.18-194.17.1.el5 #1 SMP Wed Sep 29 12:50:31 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7799 nobody    16   0  348m 101m  43m S  0.0  2.6   1:10.91 httpd
20285 nobody    15   0  340m  94m  43m S  0.0  2.4   1:11.36 httpd
31734 nobody    15   0  340m  91m  41m S  0.0  2.3   1:13.52 httpd
 8904 nobody    15   0  341m  89m  39m S  0.0  2.3   0:35.28 httpd
 7353 nobody    15   0  336m  87m  42m S  0.0  2.2   1:21.17 httpd
26097 nobody    15   0  333m  87m  43m S  0.0  2.2   1:28.84 httpd
20765 nobody    15   0  335m  86m  42m S  0.0  2.2   0:48.50 httpd
23299 nobody    15   0  334m  86m  42m S  0.0  2.2   1:13.35 httpd


--- centos 4.8 ---
$ uname -a
Linux ws89 2.6.9-89.0.28.ELsmp #1 SMP Fri Aug 20 16:11:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25117 nobody    16   0  244m  57m  41m S  0.0  1.5   0:58.23 httpd
21318 nobody    15   0  240m  53m  41m S  0.0  1.3   0:53.25 httpd
18517 nobody    16   0  239m  51m  41m S  0.0  1.3   0:41.97 httpd
10383 nobody    15   0  238m  50m  40m S  0.0  1.3   0:31.07 httpd
29560 nobody    16   0  239m  49m  39m S  0.0  1.3   0:40.39 httpd
32459 nobody    16   0  238m  49m  40m S  0.0  1.3   0:23.07 httpd
 8441 nobody    15   0  238m  48m  39m S  0.0  1.2   0:22.87 httpd
 1350 nobody    15   0  238m  48m  39m S  0.0  1.2   0:29.83 httpd


The versions of both apache and php I have installed are the same
between the two releases, so I can't imagine that being the problem.
I've also tested newer versions and it doesn't seem to make a
difference.

I've looked into this quite a bit and below is what I've figured out so
far.

Looking at the output of running 'pmap -x <pid>' shows most of the
libraries in 5.5 are using 2044K or 2048K of memory, but in 4.8 they're
all using 1024K or less.  Example:

--- centos 5.5 ---
00002b37c3248000      16      16       0 r-x--  libnss_dns-2.5.so
00002b37c324c000    2044       0       0 -----  libnss_dns-2.5.so
00002b37c344b000       4       4       4 r----  libnss_dns-2.5.so
00002b37c344c000       4       4       4 rw---  libnss_dns-2.5.so

--- centos 4.8 ---
0000002aa4a5e000      16       -       -       - r-x-- libnss_dns-2.3.4.so
0000002aa4a62000    1020       -       -       - ----- libnss_dns-2.3.4.so
0000002aa4b61000       4       -       -       - r---- libnss_dns-2.3.4.so
0000002aa4b62000       4       -       -       - rw--- libnss_dns-2.3.4.so


This seems to be explained by an increase in ELF_MAXPAGESIZE between the
two releases according to this page:

http://www.greenend.org.uk/rjk/2009/dataseg.html


Another oddity I found that was only in centos 5.5 was a lot of heap
memory being marked as Private_Dirty according to /proc/<pid>/smaps:

2b37c9826000-2b37cc4ca000 rw-p 2b37c9826000 00:00 0 [heap]
Size:             45712 kB
Rss:              45424 kB
Shared_Clean:         0 kB
Shared_Dirty:      1580 kB
Private_Clean:        0 kB
Private_Dirty:    43844 kB
Swap:                 0 kB
Pss:              43913 kB

The highest amount of Private_Dirty showing up on 4.8 is just 1628 kB.


Does anyone have any ideas on why 5.5 is using so much memory per httpd
process than it does in 4.8?

I have 100+ machines I want to upgrade, but that'll have to wait until I
figure out why apache uses so much more memory in 5.5...


Thanks in advance,

Bryan