We have some third party software running on a CentOS 4.5 virtual machine. The software is delivered as compiled python and I wrote an init script for it myself (/etc/init.d/gk). Because the software lacks the usual robustness of CentOS services, I have a bash script (/etc/cron.daily/gk-restart) which simply calls "/etc/init.d/gk restart". So, as expected, root gets an email every day when cron runs the script.
Here's the puzzling part: If I need to manually restart the service, I will use the command "/etc/init.d/gk restart". But then I get the very same email message from the cron daemon as if the daily cron job had been run automatically. The email is timestamped for the time at which I manually restarted the service. How on earth is the manual restart being monitored by the cron daemon?
The init script is full featured and maintains pid and lock files in /var/run and /var/lock/subsys respectively. Is that the connection?
Jeff
On Wed, Dec 19, 2007 at 10:52:43AM -0600, Jeff Larsen alleged:
We have some third party software running on a CentOS 4.5 virtual machine. The software is delivered as compiled python and I wrote an init script for it myself (/etc/init.d/gk). Because the software lacks the usual robustness of CentOS services, I have a bash script (/etc/cron.daily/gk-restart) which simply calls "/etc/init.d/gk restart". So, as expected, root gets an email every day when cron runs the script.
Here's the puzzling part: If I need to manually restart the service, I will use the command "/etc/init.d/gk restart". But then I get the very same email message from the cron daemon as if the daily cron job had been run automatically. The email is timestamped for the time at which I manually restarted the service. How on earth is the manual restart being monitored by the cron daemon?
The init script is full featured and maintains pid and lock files in /var/run and /var/lock/subsys respectively. Is that the connection?
Could a previous cronjob be hanging, waiting for the initscript to finish?
I bet the daemon doesn't die as expected sometimes.
If, under some condition, the initscript could hang on the daemon, then the cronjob would be sitting around forever. Then you come along, restart the daemon, and presto! the cronjob finishes and sends off the email.
Could a previous cronjob be hanging, waiting for the initscript to finish?
I bet the daemon doesn't die as expected sometimes.
Aha! looking at 'ps aux' we have:
crond /bin/bash /usr/bin/run-parts /etc/cron.daily awk -v progname=/etc/cron.daily/gk-restart ... <lots more junk>
All at 4:02 AM which is when cron.daily is processed.
The awk process is from the run-parts script. So even though my init script works perfectly from the command line, it seems to be somehow incompatible with run-parts. I guess that's something to go on. Looks like I'll need to disect run-parts to see what's happening.
Thanks,
Jeff
On Wed, Dec 19, 2007 at 11:24:46AM -0600, Jeff Larsen alleged:
Could a previous cronjob be hanging, waiting for the initscript to finish?
I bet the daemon doesn't die as expected sometimes.
Aha! looking at 'ps aux' we have:
crond /bin/bash /usr/bin/run-parts /etc/cron.daily awk -v progname=/etc/cron.daily/gk-restart ... <lots more junk>
All at 4:02 AM which is when cron.daily is processed.
The awk process is from the run-parts script. So even though my init script works perfectly from the command line, it seems to be somehow incompatible with run-parts. I guess that's something to go on. Looks like I'll need to disect run-parts to see what's happening.
I doubt it has anything to do with run-parts; it just doesn't do much.
Have you tried the 'service' command? It is the more "proper" way to run initscripts: service gk restart
Could a previous cronjob be hanging, waiting for the initscript to finish?
I bet the daemon doesn't die as expected sometimes.
Aha! looking at 'ps aux' we have:
crond /bin/bash /usr/bin/run-parts /etc/cron.daily awk -v progname=/etc/cron.daily/gk-restart ... <lots more junk>
All at 4:02 AM which is when cron.daily is processed.
The awk process is from the run-parts script. So even though my init script works perfectly from the command line, it seems to be somehow incompatible with run-parts. I guess that's something to go on. Looks like I'll need to disect run-parts to see what's happening.
I doubt it has anything to do with run-parts; it just doesn't do much.
The problem was a failure to redirect stderr in my home-grown init script. I was sending stdout to /dev/null, but not stderr. Both run-parts, and cron in general try to grab both stderr and stdout and pipe anything they get to email. But since my script wasn't closing stderr, cron was hanging on and waiting for input. Adding 2>&1 to the end of the python command fixed it. Dumb mistake on my part. But in my defense, interpreted languages make for lousy daemons.
Jeff
-
Garrick Staples -
Jeff Larsen