Hi all, today we've encountered quite strange issues with memory allocation on one of our VPS running CentOS 6.2. So far I've been unable to determine what's causing it - hopefully someone here will know what's up. The VPS is a "small" machine - just 512MB of RAM, 1 CPU, running 6.2. with current kernel (2.6.32-220.4.2.el6.x86_64, but I've tried the 2.6.32-71.el6.x86_64 too). There's a quite common stack installed, i.e. Nainstalován je na něm celkem standardní stack - apache, php, postgresql, postfix, dovecot, memcached and ssh. Basically nothing exotic, everything from official repos (except from postfix and postgresql). The machine is not heavily used. More detailed logs (than posted here) are available here: http://pastebin.com/vYxRUyUX We've been hitting some I/O utilization issues (cause by other VPS instances on the same hw) so we've migrated to a different physical hw. After the migration, the VPS started failing because of memory alloc issues - the services fail either at startup time or when processing the requests - although there's enough free mem: [root at vps audit]# free total used free shared buffers cached Mem: 502728 294224 208504 0 18604 163608 -/+ buffers/cache: 112012 390716 Swap: 0 0 0 i.e. about 200 MB of free memory, but apache fails because of segfaults when forking a child process: [16:49:51 2012] [error] (12)Cannot allocate memory: fork: Unable to fork new process [16:51:17 2012] [notice] child pid 2577 exit signal Segmentation fault (11) or when processing requests: [26 16:30:16 2012] [error] [client 66.249.72.1] PHP Fatal error: Out of memory (allocated 262144) (tried to allocate 523800 bytes) in Unknown on line 0 The memory_limit in PHP is set to 32MB, so it's not the case. Similar issues happen to PostgreSQL: 16:42:01 CET pid=2504 db=xxxxxx-drupal user=xxxxxx FATAL: out of memory 16:42:01 CET pid=2504 db=xxxxxx-drupal user=xxxxxx DETAIL: Failed on request of size 2488. 16:42:01 CET pid=2438 db= user= LOG: could not fork new process for connection: Nelze alokovat paměť 16:42:01 CET pid=2438 db= user= 4f4a5247.986:21 LOG: could not fork new process for connection: cannot allocate memory I have absolutely no clue what's causing this / how to fix it. According to free/vmstat there's about 200MB of free RAM all the time, so I have no idea why the alloc calls fail. What makes is even more puzzling is that after adding a swapfile, all the issues suddenly disappear, although the swapfile is not used at all ... and it's not possible to disable it because of memory alloc. # dd if=/dev/zero of=swap.img bs=1024 count=409600 # mkswap swap.img # swapon swap.img ... now the services are starting fine ... # swapon -s Filename Type Size Used Priority /root/swap.img file 399992 0 -1 # free total used free shared buffers cached Mem: 503412 294192 209220 0 11740 99980 -/+ buffers/cache: 182472 320940 Swap: 399992 0 399992 # swapoff swap.img swapoff: swap.img: swapoff selhal: Nelze alokovat paměť Any ideas what might cause this? The fact that I haven't noticed these issues before the migration are probably caused by a swap file - I've manually added it during a maintenance and forgot to remove it after that, but it disappeared when the machine was rebooted during migration. There's a SELinux enable, but I doubt it's causing the issues - there's nothing in audit logs except for an information that there was a segfault. Nothing suspicious. Otherwise it's just a standard CentOS install, the only thing I had to tune a bit were kernel limits (in sysctl.conf) related to shared memory (because of the database). Currently there's kernel.shmmax = 68719476736 kernel.shmall = 134217728 vm.swappiness = 0 vm.overcommit_memory = 2 which should be fine IMHO ... any ideas? regards Tomáš