[CentOS] strange system outage

Wed May 10 20:35:07 UTC 2017
Larry Martell <larry.martell at gmail.com>

On Wed, May 10, 2017 at 4:18 PM, Jonathan Billings <billings at negate.org> wrote:
> On Wed, May 10, 2017 at 03:19:05PM -0400, Larry Martell wrote:
>> > How are you starting this daemon?
>>
>> I am using code something like this: https://gist.github.com/slor/5946334.
>
> Oh, I was assuming that since you called it a daemon, it was actually
> something started automatically on boot, instead of something you
> manually started and it daemonized.

So on all the other systems that this is deployed on it is started
automatically at boot and it's controlled with service or systemctl.
On this system because I do not have remote access and I do not have a
very competent set of hands over there it was never set up to run as a
service. So a user manually starts it by running the daemon script.

This had been running on the machine like that for 8 months. The
nightly crash just started on April 21 and has happened every day
since.

> If it is dying, are you logged in when its running?

I am not 100% sure, but I think the user sus, starts the daemon
process and then exits the su.

> does it require you to be connected when its running?

No. The underlying script polls dirs looking for files, and loads data
into a db.

> Maybe you should run it in a
> screen/tmux instead of daemonizing, so you can see stderr/stdout?

stdout and stderr are written to a file and when the daemon gets
killed or dies or whatever happens to it, the output in the file
abruptly stops, just like what I showed in the messages file.