Hello fellow CentOS sysadmins,
I run a small multiplayer card game with around 500 users at peak times.
The client is in Flash and the server is in Perl.
The Perl server binds to port 8080, i.e. only 1 instance of it can be started (important detail).
The Perl server poll()s TCP-sockets and forks only once - at the startup by calling this method:
sub daemonize { die "Can not fork: $!\n" unless defined (my $child = fork()); # the parent should die exit 0 if $child;
setsid(); open(STDIN, '</dev/null'); open(STDOUT, '>/tmp/pref.txt'); open(STDERR, '>&STDOUT'); chdir('/'); umask(0); }
It runs on CentOS 5.6 Linux / 64 bit, PostgreSQL 8.4.8 and Perl 5.8.8.
Because my budget is small and I had enough troubles already, I want to use as little additional software as possible - so that I can change hosters or reinstall my cheapo server quickly. That is why I for example just log to /tmp/pref.txt instead of installing syslog-ng. And that is why I'd like to use /etc/inittab for restarting my Perl daemon.
My Perl daemon runs mostly stable, but approx. once a week it can crash with a
May 29 11:06:46 myhost kernel: pref.pl[3113]: segfault at 00007fffa21e6fd8 rip 0000003cce274460 rsp 00007fffa21e6fd0 error 6
Since I'm tired of restarting the server manually, I've tried to add it to the /etc/inittab:
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl' nobody
(and I've added a nightly cronjob to "pkill pref.pl" in the hope to refresh perl this way).
Unfortunately this does not work as expected - in the /var/log/messages I see that the script is being started again and again every 5 mins:
Jun 2 18:55:56 myhost init: Id "pref" respawning too fast: disabled for 5 minutes
What am I doing wrong here? I was hoping to being able to use /etc/inittab here, because I remember using it for a similar situation at work few years ago (also with a Perl daemon) and it worked well then...
Thank you! Alex
P.S. I've also posted my question at http://serverfault.com/questions/276428/restarting-a-perl-script-socket-daem...
Alexander Farber wrote:
Hello fellow CentOS sysadmins,
I run a small multiplayer card game with around 500 users at peak times.
The client is in Flash and the server is in Perl.
<snip>
My Perl daemon runs mostly stable, but approx. once a week it can crash with a
May 29 11:06:46 myhost kernel: pref.pl[3113]: segfault at 00007fffa21e6fd8 rip 0000003cce274460 rsp 00007fffa21e6fd0 error 6
<snip> Ok, here's the more significant question: why's it SEGV'ing? What's at line 3113 that could segv?
mark, was a programmer longer than I've been a sysadmin
On 6/2/2011 2:46 PM, Alexander Farber wrote:
The Perl server poll()s TCP-sockets and forks only once - at the startup by calling this method:
sub daemonize { die "Can not fork: $!\n" unless defined (my $child = fork()); # the parent should die exit 0 if $child;
[....]
Since I'm tired of restarting the server manually, I've tried to add it to the /etc/inittab:
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl' nobody
(and I've added a nightly cronjob to "pkill pref.pl" in the hope to refresh perl this way).
Unfortunately this does not work as expected - in the /var/log/messages I see that the script is being started again and again every 5 mins:
Jun 2 18:55:56 myhost init: Id "pref" respawning too fast: disabled for 5 minutes
What am I doing wrong here?
It needs to not fork/exit on its own if you want init to respawn when it exits.
On Thu, Jun 2, 2011 at 10:10 PM, Les Mikesell lesmikesell@gmail.com wrote:
On 6/2/2011 2:46 PM, Alexander Farber wrote:
The Perl server poll()s TCP-sockets and forks only once - at the startup by calling this method:
sub daemonize { die "Can not fork: $!\n" unless defined (my $child = fork()); # the parent should die exit 0 if $child;
[....]
Since I'm tired of restarting the server manually, I've tried to add it to the /etc/inittab:
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl' nobody
(and I've added a nightly cronjob to "pkill pref.pl" in the hope to refresh perl this way).
Unfortunately this does not work as expected - in the /var/log/messages I see that the script is being started again and again every 5 mins:
Jun 2 18:55:56 myhost init: Id "pref" respawning too fast: disabled for 5 minutes
What am I doing wrong here?
It needs to not fork/exit on its own if you want init to respawn when it exits.
Thank you Les, so init does the forking for me?
I'll try it in few hours, when I have less users online.
And I wonder how often does init try to run a program, before it stops for 5 minutes...
Mark, my pref.pl is 80 lines long, the rest is in few modules.
Yes, perl interpreter 5.8.8 crashes for me once a week, but I don't really have a chance to solve it - I'm not a perl interpreter developer myself, I don't have an easy test case for them to try, I don't have a 2nd server or time to reproduce it myself. I need to solve this problem now (going to vacation on Sunday).
Greetings from Germany Alex
I'll omit fork() and run my script from /etc/inittab as
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl' nobody 2>&1 > /tmp/pref.txt
Do you think I still need setsid(); chdir("/"); and umask(0); ?
Regards Alex
On 6/2/2011 3:36 PM, Alexander Farber wrote:
I'll omit fork() and run my script from /etc/inittab as
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl' nobody 2>&1> /tmp/pref.txt
Do you think I still need setsid(); chdir("/"); and umask(0); ?
Not sure about the setsid() - I think you want to stay in the same process group so init catches the sigchld and knows to respawn. Then again, init inherits orphan children to clean up their exit status anyway so maybe it doesn't matter. The rest affect the program behavior and should stay the same.
Thank you, now my perl daemon works with /etc/inittab
I've removed fork() and used this line:
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl >/tmp/pref.txt 2>&1' nobody
Regards Alex
2011/6/2 Alexander Farber alexander.farber@gmail.com:
On Thu, Jun 2, 2011 at 10:10 PM, Les Mikesell lesmikesell@gmail.com wrote:
On 6/2/2011 2:46 PM, Alexander Farber wrote:
The Perl server poll()s TCP-sockets and forks only once - at the startup by calling this method:
sub daemonize { die "Can not fork: $!\n" unless defined (my $child = fork()); # the parent should die exit 0 if $child;
[....]
Since I'm tired of restarting the server manually, I've tried to add it to the /etc/inittab:
pref:3:respawn:/bin/su -c '/usr/local/pref/pref.pl' nobody
(and I've added a nightly cronjob to "pkill pref.pl" in the hope to refresh perl this way).
Unfortunately this does not work as expected - in the /var/log/messages I see that the script is being started again and again every 5 mins:
Jun 2 18:55:56 myhost init: Id "pref" respawning too fast: disabled for 5 minutes
What am I doing wrong here?
It needs to not fork/exit on its own if you want init to respawn when it exits.
Thank you Les, so init does the forking for me?
I'll try it in few hours, when I have less users online.
And I wonder how often does init try to run a program, before it stops for 5 minutes...
Mark, my pref.pl is 80 lines long, the rest is in few modules.
Yes, perl interpreter 5.8.8 crashes for me once a week, but I don't really have a chance to solve it - I'm not a perl interpreter developer myself, I don't have an easy test case for them to try, I don't have a 2nd server or time to reproduce it myself. I need to solve this problem now (going to vacation on Sunday).
how about using monit to monitor/restart your perl server?
-- Eero