Hi all,
I'm running CentOS 5.5 Final, Java version "1.6.0_17" OpenJDK Runtime Environment (IcedTea6 1.7.5) (rhel-1.16.b17.el5-x86_64) OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode) installed via Yum.
We have a java application, packaged as a jar, running on our servers which, periodically, crawls RSS feeds and writes the articles to a database.
Randomly, and seemingly without cause, these processes will die, not through the application exiting, or due to my killing it, but due to something that seems to kill without leaving a trace.
My first step in diagnosing this was to log all output from the application, as well as sending stderr and stdout to a logfile, but none of these output logs contain anything that would indicate why these processes have died.
My next instinct was the kernel-level out of memory killer, but the system is never low on memory (8GB installed, routinely showing 6.5GB in free cache) and usually has somewhere between 1GB and 3GB of memory free at any given point in time.
I next thought the system could be hitting bad memory, segfaulting, and killing the process because of that, but I've mirrored the system on an identically configured server in a different datacentre, and the processes are still being killed.
The java virtual machine does trigger JVM core dumps on exit, so the process is being killed by something, but the JVM dumps don't have any useful information.
My question is: does anyone know what might be causing it, and where I should start looking to diagnose the cause?
Thanks.