[CentOS] High memory needs [SOLVED]

I finally found a solution to our problem. I think some people running
like us a combination CentOS 64 bits\Sun Grid Engine,
could encounter the same situation.
Here is a detailed explanation, hope it can be useful to someone!

The file /usr/lib/locale/locale-archive is a memory-mapped file used
by glibc (the gnu C library).  This file contains the languages used
over the system (for instance, man pages). This memory-mapping allows
to read in the file as if it were in memory, avoiding system calls
used to perform disk-read operations, therefore it allows much faster
access. Memory-mapped files (as well as shared libraries) are counted
as part of the virtual memory of processes (see "top" command, "VIRT"
field). Therefore, the part of the locale-archive file mapped to
memory adds up to the virtual memory of every processes that makes use
of glibc (basically everything), while this is actually only once in
memory. In other words, for every processes, the virtual memory
overestimates the real memory of the process by, at least, the part of
the locale-archive file which is memory-mapped.

Apparently, on 32bits distros, and on most 64bits distros, only a part
of the locale-archive file is mmapped: for instance on CentOS 32 bits:
 $ pmap $$
 b7689000   2048K r----  /usr/lib/locale/locale-archive
Only 2MB of the file are mmapped, while the file is actually ~54MB.
Some distros only install a small subsets of languages, for instance
on my Ubuntu 12.04, this file is only ~3MB.

I still don't get the reason, but CentOS x86_64 6.2 (and 6.3) mmappes
the *entire* file:
 $ pmap $$
 00007f217fae4000  96836K r----  /usr/lib/locale/locale-archive
Consequence: every processes that use glibc is overestimated by ~100MB
(according to the virtual memory).

That, alone, is not a problem at all. It is known that the virtual
memory is a bad estimation of the actual memory used, the fact that it
is largely overestimated does not matter much in most cases.

The problem, however, is that if you deal with a computing cluster
that uses Sun Grid Engine (we have version 6.2u5) it uses this value
to check whether jobs exceed their memory limit or not. The
consequence is a false impression that jobs require a huge amount of
memory, SGE being miss-leaded, and thus killing them.

A solution could be to restrict the number of languages when
installing glibc-common (which provides the locale-archive file), to
have a smaller locale-archive file.  This should be possible with:
 $ echo "%_install_langs <lang1>:<lang2>:<lang3>" >> /etc/rpm/macros.lang
 $ rpm -e glibc-common --nodeps
 $ rm /usr/lib/locale/locale-archive
 $ rpm -i <glibc-common.rpm>
I don't know why but we couldn't have glibc-common (2.12-1.80) taking
it into account, though. All languages were always installed.  I read
somewhere that it is done on purpose, not sure...

The solution that worked for us is a post-install fix that consists of
removing all languages that we don't use.  We checked with "locale"
which languages are used on the system. Then we remove all the others:
 $ localedef --list-archive | grep -v -e "en_US" -e "de_DE" -e "en_GB"
| xargs localedef --delete-from-archive
 $ mv /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl
 $ build-locale-archive
After this, the size of the locale-archive file is ~4MB, and running a
single Bash instance does not show "107MB" for SGE anymore :-) Et voilà!


   Jérémie


2012/9/27 Les Mikesell <lesmikesell at gmail.com>:
> On Thu, Sep 27, 2012 at 10:46 AM, Gordon Messmer <yinyang at eburg.com> wrote:
>>
>>> I understand it may not be very precise, however I still don't
>>> understant the difference compared to other x64 ditributions,
>>> under CentOS the value is 7 times higher!
>
> This might explain it:
> https://bugzilla.redhat.com/show_bug.cgi?id=156477
> The mmapped local-archive contains all languages even though only the
> ones you use are accessed from it.  Other distros split them and the
> installers only install what you want.
>
>> 64 bit system:
>> 00007f8ae84b7000 102580K r----  /usr/lib/locale/locale-archive
>>
>> 32 bit:
>> b7689000   2048K r----  /usr/lib/locale/locale-archive
>
> That's an interesting difference on its own, since the underlying
> files are about 95M and 54M respectively.  Does the 32 bit kernel use
> some tricks to sparsely map files where the 64 bit one does it
> directly with page tables?
>
> --
>   Les Mikesell
>      lesmikesell at gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos