[CentOS] Setting overcommit_memory=2 kills system

Fri May 18 17:03:28 UTC 2012
Michael Coffman <michael.coffman at avagotech.com>

On Fri, May 18, 2012 at 10:39 AM, Les Mikesell <lesmikesell at gmail.com>wrote:

> On Fri, May 18, 2012 at 11:24 AM, Michael Coffman
> <michael.coffman at avagotech.com> wrote:
> >
> >> If you are already overcommitted, what do you expect to happen when
> >> you say not to allow that?   The kernel doesn't have a really good way
> >> to handle that situation (or any other OOM condition for that
> >> matter...).
> >>
> >>
> > OK.  So I was confused because free shows I have plenty of memory and I
> am
> > only now noticing that Committed_AS shows a very large value.   This is
> > largely an idle system:
> >
> >             total       used       free     shared    buffers     cached
> > Mem:       2052176     951648    1100528          0     147580     626096
> > -/+ buffers/cache:     177972    1874204
> > Swap:      2052088          0    2052088
> >
> > So my real question should have been why would the Committed_AS value be
> so
> > large?
> > Committed_AS:   137197248820 kB
> >
> > On the exact same hardware with fresh build of centos5u4 and overcommit
> > turned on:
> > Committed_AS:   125716 kB
>
> I think that means an application has malloc()'d but not actually used
> a huge amount of memory.   It used to be a common practice to test the
> amount of memory available by requesting more until the request
> failed, so it could be meaningless.  If overcommit is turned off the
> app gets an error when it expects and backs off.  The Committed_AS is
> an estimate of how much RAM/swap you would be likely to need if the
> requested storage is actually used.
>
>
I get how overcommit works and it's affect on malloc.  I intentionally turn
it off so malloc will return a correct value when a process can't get
enough memory.   What I currently don't understand is the large value of
Committed_AS on my centos6/rhel6 systems.

I just did a fresh build of centos6u2 and there is very little running and
the value is:
Committed_AS:   136382202700 kB

There is almost nothing running....   Filtering out kernel processes here
is the list of running processes

 $ps -e -o uid,pid,ppid,rsz,vsz,cmd |grep -v \\[                    UID
PID  PPID   RSZ    VSZ CMD
    0     1     0  1608  23636 /sbin/init
    0   493     1  1156  11112 /sbin/udevd -d
    0  1072   493  1168  11108 /sbin/udevd -d
    0  1073   493  1164  11108 /sbin/udevd -d
    0  1147     1   828  27688 auditd
    0  1172     1  1472 252980 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
   32  1215     1  1004  19084 rpcbind
   29  1233     1  1212  25292 rpc.statd
   81  1269     1   940  25676 dbus-daemon --system
   70  1281  1280   524  31952 avahi-daemon: chroot helper
    0  1291     1  3420  92432 cupsd -C /etc/cups/cupsd.conf
    0  1316     1   624   4132 /usr/sbin/acpid
   68  1325     1  4272  29572 hald
    0  1326  1325  1336  18156 hald-runner
    0  1365  1326  1268  20272 hald-addon-input: Listening on
/dev/input/event3 /dev/input/event0 /dev/input/event1
   68  1369  1326  1164  17856 hald-addon-acpi: listening on acpid socket
/var/run/acpid.socket
    0  1389     1  1428 120820 /usr/sbin/ypbind
    0  1439     1   280  10524 rpc.rquotad
    0  1455     1   808  21464 rpc.mountd
    0  1472     1   520   6300 /usr/sbin/mcelog --daemon
    0  1492     1  1216  64068 /usr/sbin/sshd
    0  1500     1  1088  24236 xinetd -stayalive -pidfile
/var/run/xinetd.pid
   38  1508     1  1740  34448 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    0  1516     1   368   8356 rpc.rstatd
    0  1525     1  6220  57364 /usr/sbin/amd -F /etc/amd.conf
    0  1608     1  3304  80768 /usr/libexec/postfix/master
    0  1629     1   884   9216 abrt-dump-oops -d /var/spool/abrt -rwx
/var/log/messages
    0  1638     1  1304  22000 /usr/sbin/abrtd
    0  1646     1  1236  20416 crond
    0  2032     1   468  21492 /usr/sbin/atd
    0  2292     1  4564 100224 /usr/sbin/snmpd -LS 4 d -Lf /dev/null -p
/var/run/snmpd.pid -a
   89  2305  1608  3260  80848 pickup -l -t fifo -u
   89  2306  1608  3304  80916 qmgr -l -t fifo -u
   99  2425     1  4052 136382123488
/opt/ictools/64bit/synopsys-aserver/bin/AServer
    0  2433     1   580   4116 /sbin/mingetty /dev/tty1
    0  2435     1   584   4116 /sbin/mingetty /dev/tty2
    0  2437     1   580   4116 /sbin/mingetty /dev/tty3
    0  2439     1   580   4116 /sbin/mingetty /dev/tty4
    0  2441     1   580   4116 /sbin/mingetty /dev/tty5
    0  2443     1   580   4116 /sbin/mingetty /dev/tty6
    0  3001  1492  3888  99900 sshd: rootm at pts/0
    0  3004  3001  3692  13320 -bash
    0  3277  3004   920  11316 ps -e -o uid,pid,ppid,rsz,vsz,cmd



> --
>    Les Mikesell
>      lesmikesell at gmail.com
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



-- 
-MichaelC