[CentOS] Manual OOM killing?

[Daniel De Marco]

* Alan McKay <alan.mckay at gmail.com> [07/25/2012 09:36]:
> Yesterday I had one of my scientists kill one of my servers when his
> program ran amok and gobbled up all the memory, or forked too many
> processes, or I'm just not exactly sure what to be honest.
> 
> Is there something I can run manually in cron to look for rampant
> programs and kill them?   I know that may be hard to discern but I
> could also include a list if "known good" programs not to kill, as
> well as a list of "known suspect" user IDs
> 
> Anyone ever done this?  Searching the list on "OOM" does not bring up much.

I would look into disabling memory overcommit. 

On our compute cluster nodes we use the following sysctls:
# disable overcommit (allocation_limit=swap + overcommit_ratio/100 * phys_mem)
vm.overcommit_ratio=70
vm.overcommit_memory=2

This will prevent programs to allocate more than allocation_limit of
memory. In this way the program that tries to allocate the memory that
would put the memory usage over the limit will be prevented to do so.
The malloc will return NULL.

For the forking too many processes you can look into man limits.conf
where you can limit the maximum number of processes per user. 

Daniel.