[CentOS-devel] Malloc behavior

Sun Mar 11 05:06:04 UTC 2007
Leonardo Vilela Pinheiro <leopinheiro at gmail.com>

Hi,

I'm sorry for a so big email, but this might be a bug.

As some of you may know, there is a characteristic of malloc called
overcommit, which lets a program "allocate" more virtual memory than is
available on the system. I've set overcommit_memory=2 and turned off swap,
but I believe the behaviour of malloc is not the expected.

Let me introduce the overcommit, then show the test code, then the results.

//-- Starting introduction of overcommit

According to http://www.die.net/doc/linux/man/man3/malloc.3.html:

"By default, Linux follows an optimistic memory allocation strategy. This
means that when *malloc*() returns non-NULL there is no guarantee that the
memory really is available. This is a really bad bug. In case it turns out
that the system is out of memory, one or more processes will be killed by
the infamous OOM killer. In case Linux is employed under circumstances where
it would be less desirable to suddenly lose some randomly picked processes,
and moreover the kernel version is sufficiently recent, one can switch off
this overcommitting behavior using a command like # echo 2 >
/proc/sys/vm/overcommit_memorySee also the kernel Documentation directory,
files *vm/overcommit-accounting* and *sysctl/vm.txt*."


Looking at vm/overcommit-accounting:

"The Linux kernel supports the following overcommit handling modes

0       -       Heuristic overcommit handling. Obvious overcommits of
                address space are refused. Used for a typical system. It
                ensures a seriously wild allocation fails while allowing
                overcommit to reduce swap usage.  root is allowed to
                allocate slightly more memory in this mode. This is the
                default.

1       -       Always overcommit. Appropriate for some scientific
                applications.

2       -       Don't overcommit. The total address space commit
                for the system is not permitted to exceed swap + a
                configurable percentage (default is 50) of physical RAM.
                Depending on the percentage you use, in most situations
                this means a process will not be killed while accessing
                pages but will receive errors on memory allocation as
                appropriate.

The overcommit policy is set via the sysctl `vm.overcommit_memory'.

The overcommit percentage is set via `vm.overcommit_ratio'."

//-- Ending introduction of overcommit

Then I started doing some tests. Below is my the test code:

//---------------------------- Top of test code
#include <stdio.h>
#include <malloc.h>
#include <limits.h>
#include <string.h>
#include <unistd.h>

int
main(void)
{
    int i;
    char *p;

4    char *r = malloc(1048576);

    // Filling some useless space to copy later
    for (i = 0; i <= 1048576; i++)
    {
        *(char*)(r+i) = i;
    }


    for (i = 1; ; i++)
    {
        p = malloc(1048576);

        if (p == NULL)
        {
            printf("malloc %d MB - %p\n", i, p);
            printf("Returned null\n");
            return 0;
        }
        else
        {
//            printf("malloc %d MB - %p\n", i, p);
            if (memcpy(p, r, 1048576) == 0)
            {
                printf("Error in memcpy\n");
            }
            usleep(5);
        }

    }

  printf("Ok, no NULL\n");
  return 0;
}
//---------------------------- End of test code

Conditions:
Kernel: 2.6.9-42.0.10.ELsmp
RAM memory: 2GB
Swap: OFF
overcommit_memory=2 (changed by me)
overcommit_ratio=50 (default)

' free '  yields:
                     total       used       free     shared    buffers
cached
Mem:       2065252     602872    1462380          0      12372     283248
-/+ buffers/cache:     307252    1758000
Swap:                0          0          0

There is about 1.4GB free RAM memory.

Then I've executed the test code got:
malloc 178 MB - (nil)
Returned null

According to the kernel document above:
"The total address space commit for the system is not permitted to exceed
swap + a
configurable percentage (default is 50) of physical RAM. Depending on the
percentage
you use, in most situations this means a process will not be killed while
accessing pages
but will receive errors on memory allocation as appropriate."

Shouldn't I have got something about 1000 MB instead of 178 MB, because 1000
MB is 50% of  physical memory? As long as I have 1.4GB free, I should be
able to allocate those 1000 MB. Are there other (configurable or not) limits
for memory allocation? (ulimit is not being used)

Then I repeated the same test, but with swap on. Swap size is 1GB.

Conditions:
Kernel: 2.6.9-42.0.10.ELsmp
RAM memory: 2GB
Swap: 1GB
overcommit_memory=2 (changed by me)
overcommit_ratio=50 (default)

' free '  yields:
                     total       used       free     shared    buffers
cached
Mem:       2065252     605480    1459772          0      12628     283512
-/+ buffers/cache:     309340    1755912
Swap:      1048568          0    1048568

Then I've executde the test code and got:
malloc 1198 MB - (nil)
Returned null

It means this time I've allocated successfuly 1198 MB. Shouldn't I have got
something about 2GB?


Thanks
Leonardo Pinheiro
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20070311/30c9c741/attachment-0006.html>