Hiya, have a bit of an odd problem so looking for thoughts before proceeding any more. Problem is on centos3 (also runs fine on another but fails on another, so not sure this is an issue or not), dual xeon setup. Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to problem). Crontab -l shows the following... 00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh file basically sources with full path a config with some vars, then runs the perl script). There are a couple of other scripts in the crontab which run absolutely fine, so cron itself seems to be working, its just the combination. If I run this (at that time or anytime) via cron, the whole server totally dies no warning, nothing in logs instantly. If I watch in top waiting for the cron you don't even see it appear, machine just totally hangs, can't ping or anything, needs a cold boot. Nothing in logs or anything apart from bootup info after. If I run the perl script outside of cron from any folder at all, it works fine, never crashes which is where I'm confused, its only as soon as its run from cron. To test I've put it on a test machine, and to confuse it further runs absolutely fine under cron (old redhat install though) no matter what I do, it also seems to run fine on other machines (except one with identical problem) and runs fine from cron on other linux dists/machines, but can't do as much testing on them. I'm really not sure where the problem lies, so may be offtopic, I'm just confused why it runs on some and not others (although maybe running the same version, its possible they were configured slightly different on install), and pretty much clutching at straws. Worse thing is machines a production one, and want to keep downtime to a minimum, so trying to get some angle on what may be wrong before testing more. Any thoughts much appreciated. Ian
Hi, Try to monitor your server performance via top while cron is running with run_stats.sh script. It will give you an idea about what is being done? Does this process takes a lot of CPU or spawns other processes that consumes a lot of CPU of memory
I got such behavior with Apache+PHP (running as CGI)+MySQL on CentOS 3.5 and i found that MySQL was taking about 90% CPU before the server dies with nothing could be done except cold booting.
If you have a similar problem, Let us determine which process exactly is doing this biss-behaviour and try to find alternatives.
Hameed ----- Original Message ----- From: Ian mu To: centos@centos.org Sent: Monday, October 03, 2005 11:30 AM Subject: [CentOS] Cron script crashing server...
Hiya, have a bit of an odd problem so looking for thoughts before proceeding any more.
Problem is on centos3 (also runs fine on another but fails on another, so not sure this is an issue or not), dual xeon setup.
Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to problem). Crontab -l shows the following...
00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh file basically sources with full path a config with some vars, then runs the perl script).
There are a couple of other scripts in the crontab which run absolutely fine, so cron itself seems to be working, its just the combination.
If I run this (at that time or anytime) via cron, the whole server totally dies no warning, nothing in logs instantly. If I watch in top waiting for the cron you don't even see it appear, machine just totally hangs, can't ping or anything, needs a cold boot. Nothing in logs or anything apart from bootup info after.
If I run the perl script outside of cron from any folder at all, it works fine, never crashes which is where I'm confused, its only as soon as its run from cron. To test I've put it on a test machine, and to confuse it further runs absolutely fine under cron (old redhat install though) no matter what I do, it also seems to run fine on other machines (except one with identical problem) and runs fine from cron on other linux dists/machines, but can't do as much testing on them.
I'm really not sure where the problem lies, so may be offtopic, I'm just confused why it runs on some and not others (although maybe running the same version, its possible they were configured slightly different on install), and pretty much clutching at straws. Worse thing is machines a production one, and want to keep downtime to a minimum, so trying to get some angle on what may be wrong before testing more.
Any thoughts much appreciated.
Ian
------------------------------------------------------------------------------
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hiya, thanks for the reply, it would normally take a lot of cpu, but it runs totally fine when its not under cron. What I also tried doing was removing the log files it parses (so it basically had no work to do), and it still crashed the machine under cron, but fine when ran from the console (from any folder, i.e it has full paths etc in it). I tried running top with constant refresh to try and watch it appear, but for example suppose I'm waiting for 16:00 as soon as it gets to 16:00 (displays 15:59:59 only) whole server is instantly dead when it would display 16:00:00 otherwise, there's no "build up". It's possible I may be able to find another similar script, but I'm firstly worried that the whole server crashes under a perl script when its introduced to cron as there may be other issues with other scripts tried later, its not having enough time for load to be an issue or anything, and when the logs removed there would be no load either. Ian
On 10/3/05, Abd El-Hameed Ayad hamid@use-trade.com wrote:
Hi, Try to monitor your server performance via top while cron is running with run_stats.sh script. It will give you an idea about what is being done? Does this process takes a lot of CPU or spawns other processes that consumes a lot of CPU of memory I got such behavior with Apache+PHP (running as CGI)+MySQL on CentOS 3.5and i found that MySQL was taking about 90% CPU before the server dies with nothing could be done except cold booting. If you have a similar problem, Let us determine which process exactly is doing this biss-behaviour and try to find alternatives. Hameed
----- Original Message ----- *From:* Ian mu mu.llamas@gmail.com *To:* centos@centos.org *Sent:* Monday, October 03, 2005 11:30 AM *Subject:* [CentOS] Cron script crashing server...
Hiya, have a bit of an odd problem so looking for thoughts before proceeding any more. Problem is on centos3 (also runs fine on another but fails on another, so not sure this is an issue or not), dual xeon setup. Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to problem). Crontab -l shows the following... 00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh file basically sources with full path a config with some vars, then runs the perl script). There are a couple of other scripts in the crontab which run absolutely fine, so cron itself seems to be working, its just the combination. If I run this (at that time or anytime) via cron, the whole server totally dies no warning, nothing in logs instantly. If I watch in top waiting for the cron you don't even see it appear, machine just totally hangs, can't ping or anything, needs a cold boot. Nothing in logs or anything apart from bootup info after. If I run the perl script outside of cron from any folder at all, it works fine, never crashes which is where I'm confused, its only as soon as its run from cron. To test I've put it on a test machine, and to confuse it further runs absolutely fine under cron (old redhat install though) no matter what I do, it also seems to run fine on other machines (except one with identical problem) and runs fine from cron on other linux dists/machines, but can't do as much testing on them. I'm really not sure where the problem lies, so may be offtopic, I'm just confused why it runs on some and not others (although maybe running the same version, its possible they were configured slightly different on install), and pretty much clutching at straws. Worse thing is machines a production one, and want to keep downtime to a minimum, so trying to get some angle on what may be wrong before testing more. Any thoughts much appreciated. Ian
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ian mu wrote:
Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to problem). Crontab -l shows the following...
00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh file basically sources with full path a config with some vars, then runs the perl script).
The first thing I would do to put some echos in run_stats.sh and change > /dev/null to > /path/somefile
If you're getting as far as the perl script, put some print statements in it.
You might try doing an strace from cron:
00 6 * * * strace -f -oOutputFile /home/statsman/stats/run_stats.sh >/path/somefile 2>&1
-Mark
You could also check what environment variables the perl and shell scripts depend upon. Perhaps they are missing in the cron environment (Cron jobs don't get the whole shell environment, which might explain why it does work from the shell)
On Mon, Oct 03, 2005 at 07:39:48AM -0400, Mark Belanger wrote:
Ian mu wrote:
Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to problem). Crontab -l shows the following...
00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh file basically sources with full path a config with some vars, then runs the perl script).
The first thing I would do to put some echos in run_stats.sh and change > /dev/null to > /path/somefile
If you're getting as far as the perl script, put some print statements in it.
You might try doing an strace from cron:
00 6 * * * strace -f -oOutputFile /home/statsman/stats/run_stats.sh
/path/somefile 2>&1
-Mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Bestu kveðjur,
Ingimar
Ok, weirder, thanks for the advice by the way, I guess its isolated it somewhat and not where I thought it would be....added the strace, but looks like not needed as never gets that far... Piped the cron commandline to a file crondebug as mark suggested, its created at cron run time, but empty. 36 13 * * * /home/statsman/stats/run_stats.sh
/home/statsman/stats/crondebug
For the first line of the run_stats.sh (tried also with #!/bin/bash) file it calls I added echo "stage 1" >> /home/statsman/stats/gendebug Run it manually from console it echos stage 1, (also echo stage 2 I added, then after that the perl script is invoked) fine. Run from cron it never even does the echo "stage 1" to the file (or at least doesn't create it). So it looks like its either dying on execing the file and not getting any further as it never writes the first line of the simple bash script or not even trying to call the file? (file gendebug isn't even created). More puzzled at this moment than I thought (nothing in root mail from cron either btw, and rest of crontab seems ok, I leave the line in just comment out the line with a single # and it doesn't crash ). Baffled :), thanks again, Ian
On 10/3/05, Ingimar Robertsson iar@skyrr.is wrote:
You could also check what environment variables the perl and shell scripts depend upon. Perhaps they are missing in the cron environment (Cron jobs don't get the whole shell environment, which might explain why it does work from the shell)
On Mon, Oct 03, 2005 at 07:39:48AM -0400, Mark Belanger wrote:
Ian mu wrote:
Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to problem). Crontab -l shows the following...
00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh file basically sources with full path a config with some vars, then
runs
the perl script).
The first thing I would do to put some echos in run_stats.sh and change > /dev/null to > /path/somefile
If you're getting as far as the perl script, put some print statements in it.
You might try doing an strace from cron:
00 6 * * * strace -f -oOutputFile /home/statsman/stats/run_stats.sh
/path/somefile 2>&1
-Mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Bestu kveðjur,
Ingimar
-- Ingimar Róbertsson, kerfisstjóri, RHCE EMAIL: iar@skyrr.is Skýrr hf TEL: +354-5695100 Ármúli 2, 108 Reykjavík FAX: +354-5695251 http://www.skyrr.is/legal/disclaimer.txt _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ian mu wrote:
Ok, weirder, thanks for the advice by the way, I guess its isolated it somewhat and not where I thought it would be....added the strace, but looks like not needed as never gets that far...
The next thing I would do is to attach strace to cron just before it kicks off. Make sure to capture the output - there will be a lot of it. strace -p PIDofCron -f -o/path/somefile
You might also look at the cron file with an octal editor to make sure there are no funny characters in it. Or just create a brand new one.
I would also look carefully each directory in the path and the contents of them as well i.e.
ls -ld /home ls -la /home ls -ld /home/statsman ls -la /home/statsman ls -ld /home/statsman/stats ls -la /home/statsman/stats
-Mark
Piped the cron commandline to a file crondebug as mark suggested, its created at cron run time, but empty. 36 13 * * *
/home/statsman/stats/crondebug
For the first line of the run_stats.sh (tried also with #!/bin/bash) file it calls I added
echo "stage 1" >> /home/statsman/stats/gendebug
Run it manually from console it echos stage 1, (also echo stage 2 I added, then after that the perl script is invoked) fine.
Run from cron it never even does the echo "stage 1" to the file (or at least doesn't create it).
So it looks like its either dying on execing the file and not getting any further as it never writes the first line of the simple bash script or not even trying to call the file? (file gendebug isn't even created).
More puzzled at this moment than I thought (nothing in root mail from cron either btw, and rest of crontab seems ok, I leave the line in just comment out the line with a single # and it doesn't crash ).
Baffled :), thanks again, Ian
On 10/3/05, *Ingimar Robertsson* <iar@skyrr.is mailto:iar@skyrr.is> wrote:
You could also check what environment variables the perl and shell scripts depend upon. Perhaps they are missing in the cron environment (Cron jobs don't get the whole shell environment, which might explain why it does work from the shell) On Mon, Oct 03, 2005 at 07:39:48AM -0400, Mark Belanger wrote: > Ian mu wrote: > > Basically I have a perl script that runs once, first thing every > > morning, gathering some stats (currently turned off due to problem). > > Crontab -l shows the following... > > > > 00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null 2>&1 (.sh > > file basically sources with full path a config with some vars, then runs > > the perl script). > > The first thing I would do to put some echos in run_stats.sh > and change > /dev/null to > /path/somefile > > If you're getting as far as the perl script, put some print > statements in it. > > You might try doing an strace from cron: > > 00 6 * * * strace -f -oOutputFile /home/statsman/stats/run_stats.sh > >/path/somefile 2>&1 > > -Mark > > _______________________________________________ > CentOS mailing list > CentOS@centos.org <mailto:CentOS@centos.org> > http://lists.centos.org/mailman/listinfo/centos Bestu kveðjur, Ingimar -- Ingimar Róbertsson, kerfisstjóri, RHCE EMAIL: iar@skyrr.is <mailto:iar@skyrr.is> Skýrr hf TEL: +354-5695100 Ármúli 2, 108 Reykjavík FAX: +354-5695251 http://www.skyrr.is/legal/disclaimer.txt _______________________________________________ CentOS mailing list CentOS@centos.org <mailto:CentOS@centos.org> http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hiya, never thought of stracing cron itself, will try that tomorrow (is a bit too busy to risk it going down atm). Thanks, Ian
On 10/3/05, Mark Belanger mark_belanger@ltx.com wrote:
Ian mu wrote:
Ok, weirder, thanks for the advice by the way, I guess its isolated it somewhat and not where I thought it would be....added the strace, but looks like not needed as never gets that far...
The next thing I would do is to attach strace to cron just before it kicks off. Make sure to capture the output - there will be a lot of it. strace -p PIDofCron -f -o/path/somefile
You might also look at the cron file with an octal editor to make sure there are no funny characters in it. Or just create a brand new one.
I would also look carefully each directory in the path and the contents of them as well i.e.
ls -ld /home ls -la /home ls -ld /home/statsman ls -la /home/statsman ls -ld /home/statsman/stats ls -la /home/statsman/stats
-Mark
Piped the cron commandline to a file crondebug as mark suggested, its created at cron run time, but empty. 36 13 * * *
/home/statsman/stats/crondebug
For the first line of the run_stats.sh (tried also with #!/bin/bash) file it calls I added
echo "stage 1" >> /home/statsman/stats/gendebug
Run it manually from console it echos stage 1, (also echo stage 2 I added, then after that the perl script is invoked) fine.
Run from cron it never even does the echo "stage 1" to the file (or at least doesn't create it).
So it looks like its either dying on execing the file and not getting any further as it never writes the first line of the simple bash script or not even trying to call the file? (file gendebug isn't even created).
More puzzled at this moment than I thought (nothing in root mail from cron either btw, and rest of crontab seems ok, I leave the line in just comment out the line with a single # and it doesn't crash ).
Baffled :), thanks again, Ian
On 10/3/05, *Ingimar Robertsson* <iar@skyrr.is mailto:iar@skyrr.is> wrote:
You could also check what environment variables the perl and shell scripts depend upon. Perhaps they are missing in the cron environment (Cron jobs don't get the whole shell environment, which might explain why it does work from the shell)
On Mon, Oct 03, 2005 at 07:39:48AM -0400, Mark Belanger wrote:
Ian mu wrote:
Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to
problem).
Crontab -l shows the following...
00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null
2>&1 (.sh
file basically sources with full path a config with some vars,
then runs
the perl script).
The first thing I would do to put some echos in run_stats.sh and change > /dev/null to > /path/somefile
If you're getting as far as the perl script, put some print statements in it.
You might try doing an strace from cron:
00 6 * * * strace -f -oOutputFile /home/statsman/stats/run_stats.sh
/path/somefile 2>&1
-Mark
CentOS mailing list CentOS@centos.org mailto:CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Bestu kveðjur,
Ingimar
-- Ingimar Róbertsson, kerfisstjóri, RHCE EMAIL: iar@skyrr.is mailto:iar@skyrr.is Skýrr hf TEL: +354-5695100 Ármúli 2, 108 Reykjavík FAX: +354-5695251 http://www.skyrr.is/legal/disclaimer.txt _______________________________________________ CentOS mailing list CentOS@centos.org mailto:CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- Mark Belanger LTX Corporation _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Hiya, redid the crontab file and still crashed, had the strace running on it and compared the run without cron to the run from cron. It does actually run the stats script so was misleading before, so I'm guessing previously just hadn't got as far as outputting it to file or something. There's loads of stuff in there, so just highlighting the end where there seems to be a difference in case anything springs to mind... I don't want anyone to go through the pain of spending time to debug everything naturally (not my script and thousands of lines in it anyway), it was really just in case something popped out at them obvious. Thanks for the replies, Ian. Working none cron strace for script up to the point the cron equivalent dies. 3045 read(4, ",\n '*' => 'ae181a',\n "..., 4096) = 4096 3045 brk(0) = 0x9fe9000 3045 brk(0xa00a000) = 0xa00a000 3045 read(4, ">[0] eq 'COMT' and $r >= $floor;"..., 4096) = 4096 3045 read(4, "\n}\n\n### MODS ###################"..., 4096) = 3988 3045 brk(0) = 0xa00a000 3045 brk(0xa02b000) = 0xa02b000 3045 read(4, "", 4096) = 0 3045 close(4) = 0 3045 open("", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
Via cron never reaches here
3045 read(3, "T:ALL]";\n$snipe_head{ 'GARAND_SC"..., 4096) = 4096 Crashing server from cron strace, 10297 is the script, 10296 is a mrtg running from a different account which runs fine whatever... 10297 read(4, ",\n '*' => 'ae181a',\n "..., 4096) = 4096 10296 read(4, " '0' x 6\n# creates deep recursio"..., 4096) = 4096 10297 read(4, ">[0] eq 'COMT' and $r >= $floor;"..., 4096) = 4096 10297 brk(0) = 0x870a000 10297 brk(0x872b000) = 0x872b000 10296 read(4, "|| $x->is_zero();\n\n my $sign = "..., 4096) = 4096 10296 stat64("/usr/bin/../lib/mrtg2/integer.pmc", 0xbfff9ba0) = -1 ENOENT (No such file or directory) 10296 open("/usr/bin/../lib/mrtg2/integer.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) 10296 stat64("/usr/bin/integer.pmc", 0xbfff9ba0) = -1 ENOENT (No such file or directory) 10296 open("/usr/bin/integer.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) 10296 stat64("/usr/lib/perl5/5.8.0/i386-linux-thread-multi/integer.pmc", 0xbfff9ba0) = -1 ENOENT (No such file or directory) 10296 open("/usr/lib/perl5/5.8.0/i386-linux-thread-multi/integer.pm", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) 10296 stat64("/usr/lib/perl5/5.8.0/integer.pmc", 0xbfff9ba0) = -1 ENOENT (No such file or directory) 10296 open("/usr/lib/perl5/5.8.0/integer.pm", O_RDONLY|O_LARGEFILE) = 5 10296 ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfff99a8) = -1 ENOTTY (Inappropriate ioctl for device) 10296 _llseek(5, 0, [0], SEEK_CUR) = 0 10297 read(4, <unfinished ...> 10296 read(5, <unfinished ...> 10297 <... read resumed> "\n}\n\n### MODS ###################"..., 4096) = 3988 10296 <... read resumed> "package integer;\n\nour $VERSION ="..., 4096) = 3266 10296 read(5, "", 4096) = 0 10296 close(5) = 0 10296 brk(0) = 0x9229000 10296 brk(0x924a000) = 0x924a000 10297 read(4, "", 4096) = 0 10297 close(4) = 0 10296 read(4, "the nth decimal digit, negative "..., 4096) = 4096 10297 open("", O_RDONLY|O_LARGEFILE <unfinished ...> 10296 read(4, "norm(), but $x\n\n my $x = shift;"..., 4096) = 4096 <strace stopped here>
On 10/3/05, Ian mu mu.llamas@gmail.com wrote:
Hiya, never thought of stracing cron itself, will try that tomorrow (is a bit too busy to risk it going down atm). Thanks, Ian
On 10/3/05, Mark Belanger mark_belanger@ltx.com wrote:
Ian mu wrote:
Ok, weirder, thanks for the advice by the way, I guess its isolated it somewhat and not where I thought it would be....added the strace, but looks like not needed as never gets that far...
The next thing I would do is to attach strace to cron just before it kicks off. Make sure to capture the output - there will be a lot of it. strace -p PIDofCron -f -o/path/somefile
You might also look at the cron file with an octal editor to make sure there are no funny characters in it. Or just create a brand new one.
I would also look carefully each directory in the path and the contents of them as well i.e.
ls -ld /home ls -la /home ls -ld /home/statsman ls -la /home/statsman ls -ld /home/statsman/stats ls -la /home/statsman/stats
-Mark
Piped the cron commandline to a file crondebug as mark suggested, its created at cron run time, but empty. 36 13 * * *
/home/statsman/stats/crondebug
For the first line of the run_stats.sh (tried also with #!/bin/bash) file it calls I added
echo "stage 1" >> /home/statsman/stats/gendebug
Run it manually from console it echos stage 1, (also echo stage 2 I added, then after that the perl script is invoked) fine.
Run from cron it never even does the echo "stage 1" to the file (or at least doesn't create it).
So it looks like its either dying on execing the file and not getting any further as it never writes the first line of the simple bash
script
or not even trying to call the file? (file gendebug isn't even
created).
More puzzled at this moment than I thought (nothing in root mail from cron either btw, and rest of crontab seems ok, I leave the line in
just
comment out the line with a single # and it doesn't crash ).
Baffled :), thanks again, Ian
On 10/3/05, *Ingimar Robertsson* <iar@skyrr.is mailto:iar@skyrr.is> wrote:
You could also check what environment variables the perl and shell scripts depend upon. Perhaps they are missing in the cron environment (Cron jobs don't get the whole shell environment, which might explain why it does work from the shell)
On Mon, Oct 03, 2005 at 07:39:48AM -0400, Mark Belanger wrote:
Ian mu wrote:
Basically I have a perl script that runs once, first thing every morning, gathering some stats (currently turned off due to
problem).
Crontab -l shows the following...
00 6 * * * nice /home/statsman/stats/run_stats.sh >/dev/null
2>&1 (.sh
file basically sources with full path a config with some vars,
then runs
the perl script).
The first thing I would do to put some echos in run_stats.sh and change > /dev/null to > /path/somefile
If you're getting as far as the perl script, put some print statements in it.
You might try doing an strace from cron:
00 6 * * * strace -f -oOutputFile /home/statsman/stats/run_stats.sh
/path/somefile 2>&1
-Mark
CentOS mailing list CentOS@centos.org mailto:CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Bestu kveðjur,
Ingimar
-- Ingimar Róbertsson, kerfisstjóri, RHCE EMAIL: iar@skyrr.is <mailto:iar@skyrr.is > Skýrr hf TEL: +354-5695100 Ármúli 2, 108 Reykjavík FAX: +354-5695251 http://www.skyrr.is/legal/disclaimer.txt _______________________________________________ CentOS mailing list CentOS@centos.org <mailto: CentOS@centos.org> http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- Mark Belanger LTX Corporation _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ian mu wrote:
Hiya, redid the crontab file and still crashed, had the strace running on it and compared the run without cron to the run from cron. It does actually run the stats script so was misleading before, so I'm guessing previously just hadn't got as far as outputting it to file or something.
Hmm,,, if you're echoing in the stats script it should show up some place.
Crashing server from cron strace, 10297 is the script, 10296 is a mrtg running from a different account which runs fine whatever...
First off, I'd remove mrtg from the equation(cron) and strip it down to just your program.
I guess then I'd focus on the portion of your script that follows "### MODS ##################".
You might try different strace flags - make sure you are following all forks.
One trick I've used before is to replace the call to the perl program in the stats script to open up the program inside the perl debugger. For instance, if the stats script is running /usr/local/bin/myscript.pl You could change it to: typeset -x DISPLAY=mydisplay:0 xterm -e 'perl -d /usr/local/bin/myscript.pl'
Once it comes up, you can just continue to see if the crash happens and if so you could step through the code until you find the offending lines. If you're a "debug with print" kind of guy you could still do the xterm trick but just insert sleeps and prints into the perl program to narrow down the exact line that is causing the problem. After realize how much time was wasted with print statements, you can spend a couple of hours learning the perl debugger :)
I'm afraid I'm running out of helpful(and intelligent) feedback on this problem. One thing I did notice, the mrtg script seemed to go through a fair amount of gyrations to find integer.pm - though it's probably not related to the problem.
10297 read(4, ",\n '*' => 'ae181a',\n "..., 4096) = 4096 10297 read(4, ">[0] eq 'COMT' and $r >= $floor;"..., 4096) = 4096 10297 brk(0) = 0x870a000 10297 brk(0x872b000) = 0x872b000 ENOENT (No such file or directory) -1 ENOENT (No such file or directory) file or directory) such file or directory) 0xbfff9ba0) = -1 ENOENT (No such file or directory) O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) (No such file or directory) (Inappropriate ioctl for device) 10297 read(4, <unfinished ...> 10297 <... read resumed> "\n}\n\n### MODS ###################"..., 4096) = 3988 = 3266 10297 read(4, "", 4096) = 0 10297 close(4) = 0 10297 open("", O_RDONLY|O_LARGEFILE <unfinished ...>
<strace stopped here>
-Mark
Hiya, just wanted to say thanks, will leave it at that and do some debugging I think :). Cheers all, Ian
On 10/4/05, Mark Belanger mark_belanger@ltx.com wrote:
Ian mu wrote:
Hiya, redid the crontab file and still crashed, had the strace running on it and compared the run without cron to the run from cron. It does actually run the stats script so was misleading before, so I'm guessing previously just hadn't got as far as outputting it to file or something.
Hmm,,, if you're echoing in the stats script it should show up some place.
Crashing server from cron strace, 10297 is the script, 10296 is a mrtg running from a different account which runs fine whatever...
First off, I'd remove mrtg from the equation(cron) and strip it down to just your program.
I guess then I'd focus on the portion of your script that follows "### MODS ##################".
You might try different strace flags - make sure you are following all forks.
One trick I've used before is to replace the call to the perl program in the stats script to open up the program inside the perl debugger. For instance, if the stats script is running /usr/local/bin/myscript.pl You could change it to: typeset -x DISPLAY=mydisplay:0 xterm -e 'perl -d /usr/local/bin/myscript.pl'
Once it comes up, you can just continue to see if the crash happens and if so you could step through the code until you find the offending lines. If you're a "debug with print" kind of guy you could still do the xterm trick but just insert sleeps and prints into the perl program to narrow down the exact line that is causing the problem. After realize how much time was wasted with print statements, you can spend a couple of hours learning the perl debugger :)
I'm afraid I'm running out of helpful(and intelligent) feedback on this problem. One thing I did notice, the mrtg script seemed to go through a fair amount of gyrations to find integer.pm http://integer.pm - though it's probably not related to the problem.
10297 read(4, ",\n '*' => 'ae181a',\n "..., 4096) = 4096 10297 read(4, ">[0] eq 'COMT' and $r >= $floor;"..., 4096) = 4096 10297 brk(0) = 0x870a000 10297 brk(0x872b000) = 0x872b000 ENOENT (No such file or directory) -1 ENOENT (No such file or directory) file or directory) such file or directory) 0xbfff9ba0) = -1 ENOENT (No such file or directory) O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) (No such file or directory) (Inappropriate ioctl for device) 10297 read(4, <unfinished ...> 10297 <... read resumed> "\n}\n\n### MODS ###################"..., 4096) = 3988 = 3266 10297 read(4, "", 4096) = 0 10297 close(4) = 0 10297 open("", O_RDONLY|O_LARGEFILE <unfinished ...>
<strace stopped here>
-Mark
-- Mark Belanger LTX Corporation _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Abd El-Hameed Ayad -
Ian mu -
Ingimar Robertsson -
Mark Belanger