Hello,
In general I am in the habit of turning off memory overcommit because I believe it's a bad thing in a multi-user environment. This was never a problem on rhel5 systems, but on rhel6, I am having issues. When I try to set overcommit_memory=2, my system locks up. It basically behaves as if the memory is all used up... I see the same behavior on centos6 or rhel6. Following is some output from each platform.
# -- RHEL6 # uname -a Linux joker 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
#lsb_release -a LSB Version: :core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: RedHatEnterpriseWorkstation Description: Red Hat Enterprise Linux Workstation release 6.2 (Santiago) Release: 6.2 Codename: Santiago
# free total used free shared buffers cached Mem: 2052176 234828 1817348 0 15352 112852 -/+ buffers/cache: 106624 1945552 Swap: 2052088 0 2052088
# sysctl -a |grep commit vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.nr_overcommit_hugepages = 0 # sysctl -w vm.overcommit_memory=2 vm.overcommit_memory = 2 # ls -bash: fork: Cannot allocate memory #
#--- CENTOS6 -------------------------- # uname -a Linux joker 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
# lsb_release -a LSB Version: :core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: CentOS Description: CentOS release 6.2 (Final) Release: 6.2 Codename: Final
# free total used free shared buffers cached Mem: 2052176 378668 1673508 0 19472 252576 -/+ buffers/cache: 106620 1945556 Swap: 2052088 0 2052088
# sysctl -a |grep commit vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.nr_overcommit_hugepages = 0 # sysctl -w vm.overcommit_memory=2 vm.overcommit_memory = 2 [root@joker ~]# ls -bash: fork: Cannot allocate memory
One last point. If I set the overcommit values in /etc/sysctl.conf and then reboot, the values get set correctly on boot and everything seems fine. In addition I can then change the value of overcommit_memory to 0 and back to 2 with out any ill affects.
Searches for issues with setting overcommit_memory=2 haven't turned up anything useful..
Thanks.
On Thu, 2012-05-17 at 13:08 -0600, Michael Coffman wrote:
vm.overcommit_ratio = 50
^^
Try changing the commit ratio to a greater value (yes you can go over 100). Then sysctl -w vm.overcommit_memory=2
Just because one machine fails gracefully does not mean the next will. But really buy some ram for real. OR solve the real problem.
John
On Fri, May 18, 2012 at 6:15 AM, John Stanley john.stanley@elslc.comwrote:
On Thu, 2012-05-17 at 13:08 -0600, Michael Coffman wrote:
vm.overcommit_ratio = 50
^^Try changing the commit ratio to a greater value (yes you can go over 100). Then sysctl -w vm.overcommit_memory=2
OK.
Just because one machine fails gracefully does not mean the next will.
I don't even know what the above means.
But really buy some ram for real. OR solve the real problem.
Well that's just ridiculous. 'Real problem'?. If I run a command to change a kernel value, I assume that my system will not be rendered useless because it now believes all the memory is consumed. It seems like a problem to me no matter how much memory the machine has.
John
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, May 18, 2012 at 8:46 AM, Michael Coffman michael.coffman@avagotech.com wrote:
Just because one machine fails gracefully does not mean the next will.
I don't even know what the above means.
But really buy some ram for real. OR solve the real problem.
Well that's just ridiculous. 'Real problem'?. If I run a command to change a kernel value, I assume that my system will not be rendered useless because it now believes all the memory is consumed. It seems like a problem to me no matter how much memory the machine has.
If you are already overcommitted, what do you expect to happen when you say not to allow that? The kernel doesn't have a really good way to handle that situation (or any other OOM condition for that matter...).
On Fri, May 18, 2012 at 7:50 AM, Les Mikesell lesmikesell@gmail.com wrote:
On Fri, May 18, 2012 at 8:46 AM, Michael Coffman michael.coffman@avagotech.com wrote:
Just because one machine fails gracefully does not mean the next will.
I don't even know what the above means.
But really buy some ram for real. OR solve the real problem.
Well that's just ridiculous. 'Real problem'?. If I run a command to change a kernel value, I assume that my system will not be rendered
useless
because it now believes all the memory is consumed. It seems like a problem to me no matter how much memory the machine has.
If you are already overcommitted, what do you expect to happen when you say not to allow that? The kernel doesn't have a really good way to handle that situation (or any other OOM condition for that matter...).
OK. So I was confused because free shows I have plenty of memory and I am only now noticing that Committed_AS shows a very large value. This is largely an idle system:
total used free shared buffers cached Mem: 2052176 951648 1100528 0 147580 626096 -/+ buffers/cache: 177972 1874204 Swap: 2052088 0 2052088
So my real question should have been why would the Committed_AS value be so large? Committed_AS: 137197248820 kB
On the exact same hardware with fresh build of centos5u4 and overcommit turned on: Committed_AS: 125716 kB
--
Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, May 18, 2012 at 11:24 AM, Michael Coffman michael.coffman@avagotech.com wrote:
If you are already overcommitted, what do you expect to happen when you say not to allow that? The kernel doesn't have a really good way to handle that situation (or any other OOM condition for that matter...).
OK. So I was confused because free shows I have plenty of memory and I am only now noticing that Committed_AS shows a very large value. This is largely an idle system:
total used free shared buffers cached Mem: 2052176 951648 1100528 0 147580 626096 -/+ buffers/cache: 177972 1874204 Swap: 2052088 0 2052088
So my real question should have been why would the Committed_AS value be so large? Committed_AS: 137197248820 kB
On the exact same hardware with fresh build of centos5u4 and overcommit turned on: Committed_AS: 125716 kB
I think that means an application has malloc()'d but not actually used a huge amount of memory. It used to be a common practice to test the amount of memory available by requesting more until the request failed, so it could be meaningless. If overcommit is turned off the app gets an error when it expects and backs off. The Committed_AS is an estimate of how much RAM/swap you would be likely to need if the requested storage is actually used.
On Fri, May 18, 2012 at 10:39 AM, Les Mikesell lesmikesell@gmail.comwrote:
On Fri, May 18, 2012 at 11:24 AM, Michael Coffman michael.coffman@avagotech.com wrote:
If you are already overcommitted, what do you expect to happen when you say not to allow that? The kernel doesn't have a really good way to handle that situation (or any other OOM condition for that matter...).
OK. So I was confused because free shows I have plenty of memory and I
am
only now noticing that Committed_AS shows a very large value. This is largely an idle system:
total used free shared buffers cachedMem: 2052176 951648 1100528 0 147580 626096 -/+ buffers/cache: 177972 1874204 Swap: 2052088 0 2052088
So my real question should have been why would the Committed_AS value be
so
large? Committed_AS: 137197248820 kB
On the exact same hardware with fresh build of centos5u4 and overcommit turned on: Committed_AS: 125716 kB
I think that means an application has malloc()'d but not actually used a huge amount of memory. It used to be a common practice to test the amount of memory available by requesting more until the request failed, so it could be meaningless. If overcommit is turned off the app gets an error when it expects and backs off. The Committed_AS is an estimate of how much RAM/swap you would be likely to need if the requested storage is actually used.
I get how overcommit works and it's affect on malloc. I intentionally turn it off so malloc will return a correct value when a process can't get enough memory. What I currently don't understand is the large value of Committed_AS on my centos6/rhel6 systems.
I just did a fresh build of centos6u2 and there is very little running and the value is: Committed_AS: 136382202700 kB
There is almost nothing running.... Filtering out kernel processes here is the list of running processes
$ps -e -o uid,pid,ppid,rsz,vsz,cmd |grep -v \[ UID PID PPID RSZ VSZ CMD 0 1 0 1608 23636 /sbin/init 0 493 1 1156 11112 /sbin/udevd -d 0 1072 493 1168 11108 /sbin/udevd -d 0 1073 493 1164 11108 /sbin/udevd -d 0 1147 1 828 27688 auditd 0 1172 1 1472 252980 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4 32 1215 1 1004 19084 rpcbind 29 1233 1 1212 25292 rpc.statd 81 1269 1 940 25676 dbus-daemon --system 70 1281 1280 524 31952 avahi-daemon: chroot helper 0 1291 1 3420 92432 cupsd -C /etc/cups/cupsd.conf 0 1316 1 624 4132 /usr/sbin/acpid 68 1325 1 4272 29572 hald 0 1326 1325 1336 18156 hald-runner 0 1365 1326 1268 20272 hald-addon-input: Listening on /dev/input/event3 /dev/input/event0 /dev/input/event1 68 1369 1326 1164 17856 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket 0 1389 1 1428 120820 /usr/sbin/ypbind 0 1439 1 280 10524 rpc.rquotad 0 1455 1 808 21464 rpc.mountd 0 1472 1 520 6300 /usr/sbin/mcelog --daemon 0 1492 1 1216 64068 /usr/sbin/sshd 0 1500 1 1088 24236 xinetd -stayalive -pidfile /var/run/xinetd.pid 38 1508 1 1740 34448 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g 0 1516 1 368 8356 rpc.rstatd 0 1525 1 6220 57364 /usr/sbin/amd -F /etc/amd.conf 0 1608 1 3304 80768 /usr/libexec/postfix/master 0 1629 1 884 9216 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages 0 1638 1 1304 22000 /usr/sbin/abrtd 0 1646 1 1236 20416 crond 0 2032 1 468 21492 /usr/sbin/atd 0 2292 1 4564 100224 /usr/sbin/snmpd -LS 4 d -Lf /dev/null -p /var/run/snmpd.pid -a 89 2305 1608 3260 80848 pickup -l -t fifo -u 89 2306 1608 3304 80916 qmgr -l -t fifo -u 99 2425 1 4052 136382123488 /opt/ictools/64bit/synopsys-aserver/bin/AServer 0 2433 1 580 4116 /sbin/mingetty /dev/tty1 0 2435 1 584 4116 /sbin/mingetty /dev/tty2 0 2437 1 580 4116 /sbin/mingetty /dev/tty3 0 2439 1 580 4116 /sbin/mingetty /dev/tty4 0 2441 1 580 4116 /sbin/mingetty /dev/tty5 0 2443 1 580 4116 /sbin/mingetty /dev/tty6 0 3001 1492 3888 99900 sshd: rootm@pts/0 0 3004 3001 3692 13320 -bash 0 3277 3004 920 11316 ps -e -o uid,pid,ppid,rsz,vsz,cmd
-- Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, May 18, 2012 at 10:39 AM, Les Mikesell lesmikesell@gmail.comwrote:
On Fri, May 18, 2012 at 11:24 AM, Michael Coffman michael.coffman@avagotech.com wrote:
If you are already overcommitted, what do you expect to happen when you say not to allow that? The kernel doesn't have a really good way to handle that situation (or any other OOM condition for that matter...).
OK. So I was confused because free shows I have plenty of memory and I
am
only now noticing that Committed_AS shows a very large value. This is largely an idle system:
total used free shared buffers cachedMem: 2052176 951648 1100528 0 147580 626096 -/+ buffers/cache: 177972 1874204 Swap: 2052088 0 2052088
So my real question should have been why would the Committed_AS value be
so
large? Committed_AS: 137197248820 kB
On the exact same hardware with fresh build of centos5u4 and overcommit turned on: Committed_AS: 125716 kB
I think that means an application has malloc()'d but not actually used a huge amount of memory. It used to be a common practice to test the amount of memory available by requesting more until the request failed, so it could be meaningless. If overcommit is turned off the app gets an error when it expects and backs off. The Committed_AS is an estimate of how much RAM/swap you would be likely to need if the requested storage is actually used.
Hmm. I replied and included a PS listing and go the following error:
Symantec Mail Security detected prohibited content in a message sent from your address (SYM:13316346581192243434)
My reply -
I get how overcommit works and it's affect on malloc. I intentionally turn it off so malloc will return a correct value when a process can't get enough memory. What I currently don't understand is the large value of Committed_AS on my centos6/rhel6 systems.
I just did a fresh build of centos6u2 and there is very little running and the value is: Committed_AS: 136382202700 kB
There is almost nothing running.... Filtering out kernel processes here is the list of running processes
See attached ps listing if interested...
--
Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, May 18, 2012 at 11:23 AM, John R Pierce pierce@hogranch.com wrote:
On 05/18/12 10:09 AM, Michael Coffman wrote:
99 2425 1 4052 136382123488/opt/ictools/64bit/synopsys-aserver/bin/AServer ^^^^^^^^^^
well, I dunno what that process is, but thats a HUGE vsize.
Wow. I feel like an idiot :( I was so focused on rsz, I didn't even see that. Thanks for taking the time to reply. So as often seems to be case, I'm causing my own problems :)
Well, at least I understand the overcommit a little better :)
Thanks for the replies
-- john r pierce N 37, W 122 santa cruz ca mid-left coast
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, May 18, 2012 at 10:39 AM, Les Mikesell lesmikesell@gmail.comwrote:
On Fri, May 18, 2012 at 11:24 AM, Michael Coffman michael.coffman@avagotech.com wrote:
If you are already overcommitted, what do you expect to happen when you say not to allow that? The kernel doesn't have a really good way to handle that situation (or any other OOM condition for that matter...).
OK. So I was confused because free shows I have plenty of memory and I
am
only now noticing that Committed_AS shows a very large value. This is largely an idle system:
total used free shared buffers cachedMem: 2052176 951648 1100528 0 147580 626096 -/+ buffers/cache: 177972 1874204 Swap: 2052088 0 2052088
So my real question should have been why would the Committed_AS value be
so
large? Committed_AS: 137197248820 kB
On the exact same hardware with fresh build of centos5u4 and overcommit turned on: Committed_AS: 125716 kB
I think that means an application has malloc()'d but not actually used a huge amount of memory. It used to be a common practice to test the amount of memory available by requesting more until the request failed, so it could be meaningless. If overcommit is turned off the app gets an error when it expects and backs off. The Committed_AS is an estimate of how much RAM/swap you would be likely to need if the requested storage is actually used.
I get how overcommit works and it's affect on malloc. I intentionally
turn it off so malloc will return a correct value when a process can't get enough memory. What I currently don't understand is the large value of Committed_AS on my centos6/rhel6 systems.
I just did a fresh build of centos6u2 and there is very little running and the value is: Committed_AS: 136382202700 kB
There is almost nothing running.... Filtering out kernel processes here is the list of running processes
I tried to post the ps listing, both as inline text and as an attached ps.txt file and the message was rejected by Symantec :(
--
Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
John R Pierce -
John Stanley -
Les Mikesell -
Michael Coffman