[CentOS] strange system outage

[Fred Smith]

On Wed, May 10, 2017 at 04:35:07PM -0400, Larry Martell wrote:
> On Wed, May 10, 2017 at 4:18 PM, Jonathan Billings <billings at negate.org> wrote:
> > On Wed, May 10, 2017 at 03:19:05PM -0400, Larry Martell wrote:
> >> > How are you starting this daemon?
> >>
> >> I am using code something like this: https://gist.github.com/slor/5946334.
> >
> > Oh, I was assuming that since you called it a daemon, it was actually
> > something started automatically on boot, instead of something you
> > manually started and it daemonized.
> 
> So on all the other systems that this is deployed on it is started
> automatically at boot and it's controlled with service or systemctl.
> On this system because I do not have remote access and I do not have a
> very competent set of hands over there it was never set up to run as a
> service. So a user manually starts it by running the daemon script.
> 
> This had been running on the machine like that for 8 months. The
> nightly crash just started on April 21 and has happened every day
> since.
> 
> > If it is dying, are you logged in when its running?
> 
> I am not 100% sure, but I think the user sus, starts the daemon
> process and then exits the su.
> 
> > does it require you to be connected when its running?
> 
> No. The underlying script polls dirs looking for files, and loads data
> into a db.
> 
> > Maybe you should run it in a
> > screen/tmux instead of daemonizing, so you can see stderr/stdout?
> 
> stdout and stderr are written to a file and when the daemon gets
> killed or dies or whatever happens to it, the output in the file
> abruptly stops, just like what I showed in the messages file.

Could you enable core dumps (ulimit -c unlimited) then after it
has died, check for core files? 

-- 
---- Fred Smith -- fredex at fcshome.stoneham.ma.us -----------------------------
                      The eyes of the Lord are everywhere, 
                    keeping watch on the wicked and the good.
----------------------------- Proverbs 15:3 (niv) -----------------------------