Running across some curious stuff with ulimit on CentOS 5.10.
We have a non CentOS packaged version of Asterisk (using their packages) that we start at boot time with a typical RC script.
Recently it started whining that it couldn't open enough file handles.
As we dug further into this, it appears that at boot time, it inherits ulimit from init, which is pretty low: 1024.
We've set /etc/security/limits.conf and sysctl significantly higher both for root/system wide and also for the user Asterisk runs under, to no avail.
If we log in via ssh as root (or sudo) the correct root/system-wide ulimit of 8192 is set.
Looking in /proc/[PID]/limits shows the lower ulimit only if the package is started from init (standard rc3.d type sysV stuff...). If we restart it via console, remote ssh, anything else, the limit bumps to 8196.
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
Right now, we're not totally against just taking it out of the startup and starting it manually anyway, since we really don't want the Asterisk platform coming up after a crash/reboot anyway, and any other reboots there will always be a human involved, but the way init is handling ulimit seems utterly retarded and broken.
Some indication (different engineer found it, I haven't seen the RHEL case number) appears to indicate that folks wanted init ulimited heavily in case of startup DDoS type stuff, but we haven't figured out a semi-sane unix-conventional type way AROUND this when it's needed that if we were hit by the proverbial bus, a "normal" unix guy would find.
Perhaps we're missing something.
Thoughts?
-- Nate Duehr denverpilot@me.com
On Wed, Apr 23, 2014 at 5:19 PM, Nathan Duehr denverpilot@me.com wrote:
Running across some curious stuff with ulimit on CentOS 5.10.
We have a non CentOS packaged version of Asterisk (using their packages) that we start at boot time with a typical RC script.
Recently it started whining that it couldn't open enough file handles.
As we dug further into this, it appears that at boot time, it inherits ulimit from init, which is pretty low: 1024.
Yep, it's rather low.
We've set /etc/security/limits.conf and sysctl significantly higher both for root/system wide and also for the user Asterisk runs under, to no avail.
I've had success making changes in limits.conf (for MySQL in my case). In my case I increased both the system wide and per user (mysql user).
Example: <my_user> hard nofile 2048 <my_user> soft nofile 2048
SSH to that box (as <my_user>) ... run ulimit -Hn and -Sn ... bingo 2048
Are you able to switch users to the user your Asterisk user to run `ulimit -Hn` and `ulimit -Sn`? Once you've verified new sessions get the proper limits, stop and then start your Asterisk daemon. Hopefully from there it will stop whining about file handles.
If we log in via ssh as root (or sudo) the correct root/system-wide ulimit of 8192 is set.
Looking in /proc/[PID]/limits shows the lower ulimit only if the package is started from init (standard rc3.d type sysV stuff...). If we restart it via console, remote ssh, anything else, the limit bumps to 8196.
Interesting - I've not seen that odd behavior. It worked across the board (initscript and all) when I made changes via limits.conf.
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
Right now, we're not totally against just taking it out of the startup and starting it manually anyway, since we really don't want the Asterisk platform coming up after a crash/reboot anyway, and any other reboots there will always be a human involved, but the way init is handling ulimit seems utterly retarded and broken.
Some indication (different engineer found it, I haven't seen the RHEL case number) appears to indicate that folks wanted init ulimited heavily in case of startup DDoS type stuff, but we haven't figured out a semi-sane unix-conventional type way AROUND this when it's needed that if we were hit by the proverbial bus, a "normal" unix guy would find.
Perhaps we're missing something.
Thoughts?
-- Nate Duehr denverpilot@me.com
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Nathan Duehr wrote:
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
init.d scripts run as root so you should be able to set a hard/soft limit in the init.d script before the package is started
James Pearson
Am 24.04.2014 um 19:44 schrieb James Pearson james-p@moving-picture.com:
Nathan Duehr wrote:
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
init.d scripts run as root so you should be able to set a hard/soft limit in the init.d script before the package is started
to survive updates, the init script should include one config file under /etc/sysconfig/ - there should such a entry be safe.
-- LF
On Apr 25, 2014, at 4:46 AM, Leon Fauster leonfauster@googlemail.com wrote:
Am 24.04.2014 um 19:44 schrieb James Pearson james-p@moving-picture.com:
Nathan Duehr wrote:
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
init.d scripts run as root so you should be able to set a hard/soft limit in the init.d script before the package is started
to survive updates, the init script should include one config file under /etc/sysconfig/ - there should such a entry be safe.
This is definitely true. The init script came from the Asterisk folks (we're not using a package modified to behave specifically with RHEL/CentOS) and would get over-written on upgrade.
Seems like the brokenness is the behavior of init ignoring /etc/security/limits.conf, to my way of thinking anyway.
-- Nate Duehr denverpilot@me.com
On Mon, Apr 28, 2014 at 04:20:25PM -0600, Nathan Duehr wrote:
Seems like the brokenness is the behavior of init ignoring /etc/security/limits.conf, to my way of thinking anyway.
Umm, no. That's you not understanding what limits.conf is.
Limits are hard to grok. I had to write a massive document at work explaining it. And people still don't get it.
Basically: init scripts inherit from init (pid 1), which gets defaults from the kernel
Processes initiated by a user will inherit limits from the the user's environment. For most users that will have involved a PAM session, and most PAM configs call pam_limits and _that_ reads limits.conf.
Doing a 'su' will involve PAM and that may cause pam_limits (and thus limits.conf) to be read.
Remember that init processes started at boot time will run as root and so can increase limits. You need to increase hard limits before you increase soft limits.
Processes started as a user can _not_ increase hard limits. You need to "su" to root, or "su" to a user defined in limits.conf to change those values.
Bottom line: limits.conf is a PAM config setting for pam_limits. It's not in the general path. Other process _may_ use the file but they need to have root level privs to obey it properly.
On Apr 28, 2014, at 6:40 PM, Stephen Harris lists@spuddy.org wrote:
On Mon, Apr 28, 2014 at 04:20:25PM -0600, Nathan Duehr wrote:
Seems like the brokenness is the behavior of init ignoring /etc/security/limits.conf, to my way of thinking anyway.
Umm, no. That's you not understanding what limits.conf is.
Limits are hard to grok. I had to write a massive document at work explaining it. And people still don't get it.
Actually, I completely understand, but I still had to step through it in various forms and write myself a flowchart. It's beyond non-intuitive at this point in Linux "design" history.
For starters, how Linux is handling it is *really* FUBAR compared to some of the commericial *nix variants, but that's kinda normal... Linux loves to be obtuse and overly-complex at times. It's an endearing quality. :-) Note that if you had to "write a massive document" it's probably overly-complex for no good design reason.
Basically: init scripts inherit from init (pid 1), which gets defaults from the kernel
Processes initiated by a user will inherit limits from the the user's environment. For most users that will have involved a PAM session, and most PAM configs call pam_limits and _that_ reads limits.conf.
Not processes started that change to a non-root user from a root/init/rc script. No session. At least not from what I was seeing in 5.10. Intended or not, it wasn't behaving like PAM was ever involved. :-)
Doing a 'su' will involve PAM and that may cause pam_limits (and thus limits.conf) to be read.
Remember that init processes started at boot time will run as root and so can increase limits. You need to increase hard limits before you increase soft limits.
Processes started as a user can _not_ increase hard limits. You need to "su" to root, or "su" to a user defined in limits.conf to change those values.
Bottom line: limits.conf is a PAM config setting for pam_limits. It's not in the general path. Other process _may_ use the file but they need to have root level privs to obey it properly.
This is all true, but doesn't apply to the scenario presented.
The PAM stuff is entertaining... since most new admins would think to themselves, "I'll just become that user and see what that user's ulimit is..." do an "su - [user]" and see something and assume it's what would happen to the user's limits inside that start script.
Of course, the best place to look in the end, is /proc/[PID] ...
Here's the real fun... the asterisk guys (at least back in this old version), punted... LOL... from their safe_asterisk secondary start script that gets called from their main init.d script...
# since we're going to change priority and open files limits, we need to be # root. if running asterisk as other users, pass that to asterisk on the command # line. # if we're not root, fall back to standard everything. if test `id -u` != 0 ; then echo "Oops. I'm not root. Falling back to standard prio and file max." >&2 echo "This is NOT suitable for large systems." >&2 PRIORITY=0
Even more fun... if your non-Production environment is KVM virtuals, and your Production systems are on bare-metal... don't get me started on that one... but that's a pretty common scenario these days, too. (Limits are exceedingly horked on KVM VMs... haha...)
Anyway, we're all sorted here, and I had a hell week doing a lot of other stuff in-between, but it was a fun reminder of the whole ulimit mess. I suspect this stuff will be even more entertaining as init goes away in lieu of upstart and similar. Heh.
Appreciate the thought put into the reply. Perhaps someone struggling with this down the road will find the thread via Google or similar and realize they have to think a lot harder than really should be necessary to get it set where they want it. Most "basics" type docs out there just point newbies to limits.conf and do all examples from a user's shell that they got to from su... which isn't how much really starts/runs on Production servers. :-)
-- Nate Duehr denverpilot@me.com
On Mon, May 05, 2014 at 12:44:01PM -0600, Nathan Duehr wrote:
Not processes started that change to a non-root user from a root/init/rc script. No session. At least not from what I was seeing in 5.10. Intended or not, it wasn't behaving like PAM was ever involved. :-)
If you're doing it as "su user" then pam.d/su is called which calls system-auth which calls pam_limits. If you're doing it as "runuser" then pam.d/runuser is called which directly calls pam_limits
If your program just does setreuid() calls (which it can do if started as root, or is setuid) then it's not going near PAM and so will inherit the kernel defaults (if started by init) or the user current values (if started by a user).
On May 5, 2014, at 1:14 PM, Stephen Harris lists@spuddy.org wrote:
On Mon, May 05, 2014 at 12:44:01PM -0600, Nathan Duehr wrote:
Not processes started that change to a non-root user from a root/init/rc script. No session. At least not from what I was seeing in 5.10. Intended or not, it wasn't behaving like PAM was ever involved. :-)
If you're doing it as "su user" then pam.d/su is called which calls system-auth which calls pam_limits. If you're doing it as "runuser" then pam.d/runuser is called which directly calls pam_limits
If your program just does setreuid() calls (which it can do if started as root, or is setuid) then it's not going near PAM and so will inherit the kernel defaults (if started by init) or the user current values (if started by a user).
Yup... guess which one Asterisk did back in 1.4 ? :-)
(Yeah it's ancient... [insert usual operational excuses here].)
-- Nate Duehr denverpilot@me.com
On Apr 24, 2014, at 11:44 AM, James Pearson james-p@moving-picture.com wrote:
Nathan Duehr wrote:
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
init.d scripts run as root so you should be able to set a hard/soft limit in the init.d script before the package is started
Tried that. Didn't work. Very interesting.
Also attempted /etc/security/limits.conf for both "*" and the asterisk user, to no avail. It's not honored by init-started stuff, apparently on CentOS 5.10.
-- Nate Duehr denverpilot@me.com
Am 29.04.2014 um 00:18 schrieb Nathan Duehr denverpilot@me.com:
On Apr 24, 2014, at 11:44 AM, James Pearson james-p@moving-picture.com wrote:
Nathan Duehr wrote:
Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.
init.d scripts run as root so you should be able to set a hard/soft limit in the init.d script before the package is started
Tried that. Didn't work. Very interesting.
Could you show us how do you tried that? Break your task down please.
__ LF
-
James Pearson -
Leon Fauster -
Nathan Duehr -
SilverTip257 -
Stephen Harris