Christophe Caron wrote:

Hello,

I run CentOS 5.6 on a Dell PowerEdge R815 with 256 GB of RAM. We use Sun Grid Engine to schedule jobs on this node.

I want to limit the memory usage about 150 GB per process.

<snip>

But, at least one process (oases - a bioinformatics tool) bypass this limitation and use always 240 GB of memory (the last run) !!

<snip> Is there some way to limit the number of threads the job can have? We had a problem like that - a user on a 48-core system that proceded, as the final step of the job, to want half again as much memory as the system had (256G!!!). After discussions, he limited what he submitted, so that's why I wondered if you could control that administratively.

mark

--On Friday, June 03, 2011 07:10:44 PM +0200 Christophe Caron christophe.caron@sb-roscoff.fr wrote:

I want to limit the memory usage about 150 GB per process. So i use the /etc/security/limits.conf configuration file. I test this configuration with some tools with a lower GB limit (about 2 or 4 GB), and it works !

But, at least one process (oases - a bioinformatics tool) bypass this limitation and use always 240 GB of memory (the last run) !!

A few of things come to mind:

1. Is your bioinformatics tool running as a privileged process (perhaps because it is being started as a daemon and is not dropping privs?) I don't think the hard limit applies in this case. If this is happening, you may want to take a big stick to your application programmers :)

2. /etc/security/limits.conf is used by pam_limits. Have you verified that that module is configured and required by pam?

3. How is the bioinformatics tool being started, and did you start your other test tools (where you checked with a lower GB limit) the same way. I don't know one way or the other whether pam restrictions apply to programs started as daemons.

4. Are you able to have the bioinformatics tool log its hard and soft limits, and real & effective uid/gids upon startup? (If you can't, then maybe write a C wrapper for it that will do this.) That may be the quickest way to tell you what your OS config is actually doing.

Devin