Hi,
I'm sorry for a so big email, but this might be a bug.
As some of you may know, there is a characteristic of malloc called overcommit, which lets a program "allocate" more virtual memory than is available on the system. I've set overcommit_memory=2 and turned off swap, but I believe the behaviour of malloc is not the expected.
Let me introduce the overcommit, then show the test code, then the results.
//-- Starting introduction of overcommit
According to http://www.die.net/doc/linux/man/man3/malloc.3.html:
"By default, Linux follows an optimistic memory allocation strategy. This means that when *malloc*() returns non-NULL there is no guarantee that the memory really is available. This is a really bad bug. In case it turns out that the system is out of memory, one or more processes will be killed by the infamous OOM killer. In case Linux is employed under circumstances where it would be less desirable to suddenly lose some randomly picked processes, and moreover the kernel version is sufficiently recent, one can switch off this overcommitting behavior using a command like # echo 2 > /proc/sys/vm/overcommit_memorySee also the kernel Documentation directory, files *vm/overcommit-accounting* and *sysctl/vm.txt*."
Looking at vm/overcommit-accounting:
"The Linux kernel supports the following overcommit handling modes
0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.
1 - Always overcommit. Appropriate for some scientific applications.
2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.
The overcommit policy is set via the sysctl `vm.overcommit_memory'.
The overcommit percentage is set via `vm.overcommit_ratio'."
//-- Ending introduction of overcommit
Then I started doing some tests. Below is my the test code:
//---------------------------- Top of test code #include <stdio.h> #include <malloc.h> #include <limits.h> #include <string.h> #include <unistd.h>
int main(void) { int i; char *p;
4 char *r = malloc(1048576);
// Filling some useless space to copy later for (i = 0; i <= 1048576; i++) { *(char*)(r+i) = i; }
for (i = 1; ; i++) { p = malloc(1048576);
if (p == NULL) { printf("malloc %d MB - %p\n", i, p); printf("Returned null\n"); return 0; } else { // printf("malloc %d MB - %p\n", i, p); if (memcpy(p, r, 1048576) == 0) { printf("Error in memcpy\n"); } usleep(5); }
}
printf("Ok, no NULL\n"); return 0; } //---------------------------- End of test code
Conditions: Kernel: 2.6.9-42.0.10.ELsmp RAM memory: 2GB Swap: OFF overcommit_memory=2 (changed by me) overcommit_ratio=50 (default)
' free ' yields: total used free shared buffers cached Mem: 2065252 602872 1462380 0 12372 283248 -/+ buffers/cache: 307252 1758000 Swap: 0 0 0
There is about 1.4GB free RAM memory.
Then I've executed the test code got: malloc 178 MB - (nil) Returned null
According to the kernel document above: "The total address space commit for the system is not permitted to exceed swap + a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate."
Shouldn't I have got something about 1000 MB instead of 178 MB, because 1000 MB is 50% of physical memory? As long as I have 1.4GB free, I should be able to allocate those 1000 MB. Are there other (configurable or not) limits for memory allocation? (ulimit is not being used)
Then I repeated the same test, but with swap on. Swap size is 1GB.
Conditions: Kernel: 2.6.9-42.0.10.ELsmp RAM memory: 2GB Swap: 1GB overcommit_memory=2 (changed by me) overcommit_ratio=50 (default)
' free ' yields: total used free shared buffers cached Mem: 2065252 605480 1459772 0 12628 283512 -/+ buffers/cache: 309340 1755912 Swap: 1048568 0 1048568
Then I've executde the test code and got: malloc 1198 MB - (nil) Returned null
It means this time I've allocated successfuly 1198 MB. Shouldn't I have got something about 2GB?
Thanks Leonardo Pinheiro