I'm running into a memory problem when trying to switch some web servers that are running centos4.8/x86_64 to centos5.5/x86_64. The new servers running centos5.5 are using a decent amount more memory than the centos4.8 servers.
Here's an example from top:
--- centos 5.5 --- $ uname -a Linux ws51 2.6.18-194.17.1.el5 #1 SMP Wed Sep 29 12:50:31 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7799 nobody 16 0 348m 101m 43m S 0.0 2.6 1:10.91 httpd 20285 nobody 15 0 340m 94m 43m S 0.0 2.4 1:11.36 httpd 31734 nobody 15 0 340m 91m 41m S 0.0 2.3 1:13.52 httpd 8904 nobody 15 0 341m 89m 39m S 0.0 2.3 0:35.28 httpd 7353 nobody 15 0 336m 87m 42m S 0.0 2.2 1:21.17 httpd 26097 nobody 15 0 333m 87m 43m S 0.0 2.2 1:28.84 httpd 20765 nobody 15 0 335m 86m 42m S 0.0 2.2 0:48.50 httpd 23299 nobody 15 0 334m 86m 42m S 0.0 2.2 1:13.35 httpd
--- centos 4.8 --- $ uname -a Linux ws89 2.6.9-89.0.28.ELsmp #1 SMP Fri Aug 20 16:11:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25117 nobody 16 0 244m 57m 41m S 0.0 1.5 0:58.23 httpd 21318 nobody 15 0 240m 53m 41m S 0.0 1.3 0:53.25 httpd 18517 nobody 16 0 239m 51m 41m S 0.0 1.3 0:41.97 httpd 10383 nobody 15 0 238m 50m 40m S 0.0 1.3 0:31.07 httpd 29560 nobody 16 0 239m 49m 39m S 0.0 1.3 0:40.39 httpd 32459 nobody 16 0 238m 49m 40m S 0.0 1.3 0:23.07 httpd 8441 nobody 15 0 238m 48m 39m S 0.0 1.2 0:22.87 httpd 1350 nobody 15 0 238m 48m 39m S 0.0 1.2 0:29.83 httpd
The versions of both apache and php I have installed are the same between the two releases, so I can't imagine that being the problem. I've also tested newer versions and it doesn't seem to make a difference.
I've looked into this quite a bit and below is what I've figured out so far.
Looking at the output of running 'pmap -x <pid>' shows most of the libraries in 5.5 are using 2044K or 2048K of memory, but in 4.8 they're all using 1024K or less. Example:
--- centos 5.5 --- 00002b37c3248000 16 16 0 r-x-- libnss_dns-2.5.so 00002b37c324c000 2044 0 0 ----- libnss_dns-2.5.so 00002b37c344b000 4 4 4 r---- libnss_dns-2.5.so 00002b37c344c000 4 4 4 rw--- libnss_dns-2.5.so
--- centos 4.8 --- 0000002aa4a5e000 16 - - - r-x-- libnss_dns-2.3.4.so 0000002aa4a62000 1020 - - - ----- libnss_dns-2.3.4.so 0000002aa4b61000 4 - - - r---- libnss_dns-2.3.4.so 0000002aa4b62000 4 - - - rw--- libnss_dns-2.3.4.so
This seems to be explained by an increase in ELF_MAXPAGESIZE between the two releases according to this page:
http://www.greenend.org.uk/rjk/2009/dataseg.html
Another oddity I found that was only in centos 5.5 was a lot of heap memory being marked as Private_Dirty according to /proc/<pid>/smaps:
2b37c9826000-2b37cc4ca000 rw-p 2b37c9826000 00:00 0 [heap] Size: 45712 kB Rss: 45424 kB Shared_Clean: 0 kB Shared_Dirty: 1580 kB Private_Clean: 0 kB Private_Dirty: 43844 kB Swap: 0 kB Pss: 43913 kB
The highest amount of Private_Dirty showing up on 4.8 is just 1628 kB.
Does anyone have any ideas on why 5.5 is using so much memory per httpd process than it does in 4.8?
I have 100+ machines I want to upgrade, but that'll have to wait until I figure out why apache uses so much more memory in 5.5...
Thanks in advance,
Bryan