In our environment we have many legacy application servers running apache/jserv. There is a web server front end, then a couple of load-balanced java servers on the backside. One of the problems we are faced with is hung or stuck jvms. I have looked at the java process with the ps command, and there are many times when URL(s) do not respond, yet the java looks healthy, at least from the OS point of view. The usual cure for this situation is to restart the JVM, then the URLs come right back up.
Are any of you aware of tools for monitoring apache jserv, either from localhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone.
Sean Carolan wrote:
Are any of you aware of tools for monitoring apache jserv, either from localhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone.
How about setting up a cron to monitor it and auto restart if it's not responding?
wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to restart jserv)
I'm not familiar with jserv but it looks like it serves a purpose similar to tomcat, just a much older code base.
I've used similar scripts for tomcat in the past, such as this cron */5 * * * * /bin/grep -q "java.lang.OutOfMemoryError: Java heap space" /logs/tomcat/web/1/run.latest.log 1>/dev/null 2>&1 && export MYHOST=`hostname -s` && echo "Restarting Tomcat on $MYHOST - OutOfMemoryError" | mail -s "[$MYHOST] Restarting Tomcat due to OutOfMemoryError" my@email.address && /etc/init.d/appname restart 1>/dev/null 2>&1
When the app had a memory leak it helped me sleep better at night, as the apps were restarting about 10x a day until the developers found the source of the leak. Fortunately the app restarted reliably 99.9% of the time. Another place I worked at before that had an app that was so poor that you had to babysit it all the time, the number of things that could go wrong was really too much work for us to automate it in a reasonable amount of time.
Ideally the "load balancer" should be performing "health checks" against the back end nodes and stop sending traffic to them if they are no longer healthy. I'm not familiar with the load balancing method your using, only with bigger load balancers like F5 BigIP.
nate
How about setting up a cron to monitor it and auto restart if it's not responding?
wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to restart jserv)
I tried pulling up port 8008 in a web browser, but it doesn't work quite like that. Apache is configured with mod_jserv like this:
<IfModule mod_jserv.c> ApJServMount /servlets ajpv12://hostname.web.domain.com:8008/root ApjServAction .html /servlets/gnujsp </IfModule>
Simply pulling up the URL in a web browser always returns an error:
Status: 400 Bad Request Servlet-Error: Malformed data sent to JServ
I'm not familiar with jserv but it looks like it serves a purpose similar to tomcat, just a much older code base.
Yes, it is very similar to tomcat. Both have a java process, with a minimum and maximum heap size, etc. Unfortunately I don't know of an easy way to test the java process to make sure it is healthy, short of pulling up the front-end URL in a web browser. The problem here is, it is possible that one URL could be down, but the other ones still functioning correctly. We don't want to restart java just because of a false positive on one URL.
When the app had a memory leak it helped me sleep better at night, as the apps were restarting about 10x a day until the developers found the source of the leak. Fortunately the app restarted reliably 99.9% of the time.
We are not quite having to restart ten times a day, but there actually is no chance of a code fix in my situation since this is deprecated code, and the customer is not willing/able to upgrade. This is why we want to simply auto-restart the JVM when the server gets into this state.
Ideally the "load balancer" should be performing "health checks" against the back end nodes and stop sending traffic to them if they are no longer healthy. I'm not familiar with the load balancing method your using, only with bigger load balancers like F5 BigIP.
We have such a load-balancing device but it sits in front of the web server, not behind (the web servers are load balanced too). There is an apache plugin that handles the jserv load balancing.
Sean Carolan wrote:
How about setting up a cron to monitor it and auto restart if it's not responding?
wget -q --timeout=30 http://localhost:8008/ -O /dev/null || (command to restart jserv)
I tried pulling up port 8008 in a web browser, but it doesn't work quite like that. Apache is configured with mod_jserv like this:
<IfModule mod_jserv.c> ApJServMount /servlets ajpv12://hostname.web.domain.com:8008/root ApjServAction .html /servlets/gnujsp </IfModule>Simply pulling up the URL in a web browser always returns an error:
Status: 400 Bad Request Servlet-Error: Malformed data sent to JServ
Sounds similar to the mod_jk connector in apache to connect to tomcat. When I had to deal with this I setup a dedicated apache instance on each system running tomcat whose sole purpose for existence was for testing that connector.
So say setup an apache instance on another port, and have it direct all traffic back to mod_jserv, then hit the apache instance with http. It's not perfect but at least for me apache was a lot more stable than tomcat especially in such a basic configuration, so it worked well as a way to test the health of the mod_jk connector.
nate
Sounds similar to the mod_jk connector in apache to connect to tomcat. When I had to deal with this I setup a dedicated apache instance on each system running tomcat whose sole purpose for existence was for testing that connector.
So say setup an apache instance on another port, and have it direct all traffic back to mod_jserv, then hit the apache instance with http. It's not perfect but at least for me apache was a lot more stable than tomcat especially in such a basic configuration, so it worked well as a way to test the health of the mod_jk connector.
Excellent idea, Nate. I already have apache installed on all these servers, so it should be fairly trivial to set up a local test site just for this purpose. I'm going to test this out in our lab environment today and see how it performs.
Sean Carolan wrote:
Sounds similar to the mod_jk connector in apache to connect to tomcat. When I had to deal with this I setup a dedicated apache instance on each system running tomcat whose sole purpose for existence was for testing that connector.
So say setup an apache instance on another port, and have it direct all traffic back to mod_jserv, then hit the apache instance with http. It's not perfect but at least for me apache was a lot more stable than tomcat especially in such a basic configuration, so it worked well as a way to test the health of the mod_jk connector.
Excellent idea, Nate. I already have apache installed on all these servers, so it should be fairly trivial to set up a local test site just for this purpose. I'm going to test this out in our lab environment today and see how it performs.
If this is old enough to be running a Sun 1.4.x JVM and the web server generates images, there was a bug that caused the JVM itself to leak memory under certain circumstances. I've forgotten the details but it was something like an unreleased file reference per image.
Sounds similar to the mod_jk connector in apache to connect to tomcat. When I had to deal with this I setup a dedicated apache instance on each system running tomcat whose sole purpose for existence was for testing that connector.
We have decided to take this tactic and set up a dedicated apache instance on each server simply for monitoring the local java process. I have run into a snag, however and I'm not sure what to do next. At first I attempted to install the mod_jserv rpm but that didn't work:
[scarolan@pstest-app13:~]$ sudo rpm -ivh ApacheJServ-1.1.2-1.i386.rpm error: Failed dependencies: apache >= 1.3.6 is needed by ApacheJServ-1.1.2-1
Ok, so next thing I tried was compiling the module from source code:
[scarolan@pstest-app13:~/ApacheJServ-1.1.2]$ ./configure --with-jdk-home=/usr/local/mercury/Sun/jdk1.5.0_01 --with-JSDK=/usr/local/mercury/Sun/JSDK2.0/lib/jsdk.jar --with-apache-src=/usr/include/httpd/
[bunch of automake stuff removed] checking for Apache source directory (assume static build)... configure: error: Directory is not a valid Apache source distribution
I have installed both the httpd-devel rpm and the httpd src rpm, and tried directing --with-apache-src at each directory but no dice. What am I doing wrong here?
[scarolan@pstest-app13:~/ApacheJServ-1.1.2]$ ./configure --with-jdk-home=/usr/local/mercury/Sun/jdk1.5.0_01 --with-JSDK=/usr/local/mercury/Sun/JSDK2.0/lib/jsdk.jar --with-apache-src=/usr/include/httpd/
If I run the configure command without --with-apache-src here is what I get:
configure: error: Could not locate one or more of the necessary Apache header files. Please check that you have passed the correct location for the Apache directory. Note that you must run either Apache's 'Configure' or 'configure' program at least once if you have never compiled Apache before; this generates certain necessary headers files based on your platform.
This seems to indicate that it wants the apache header files, which are installed in /usr/include/httpd. Anyway if someone has an idea how I can get a working mod_jserv module for CentOS3 let me know.
This seems to indicate that it wants the apache header files, which are installed in /usr/include/httpd. Anyway if someone has an idea how I can get a working mod_jserv module for CentOS3 let me know.
Ok, so after doing some more reading it appears that you can simply build the mod_jserv.so module with a command like this:
apxs -c mod_jserv.c -o mod_jserv.so
However when I run the command I get another error:
[root@pstest-app13 c]# /usr/sbin/apxs -c mod_jserv.c -o mod_jserv.so /usr/bin/libtool --silent --mode=compile gcc -prefer-pic -O2 -g -pipe -march=i386 -mcpu=i686 -DSSL_EXPERIMENTAL_ENGINE -I/usr/kerberos/include -DAP_HAVE_DESIGNATED_INITIALIZER -DLINUX=2 -D_REENTRANT -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE -pthread -I/usr/include/httpd -c -o mod_jserv.lo mod_jserv.c && touch mod_jserv.slo mod_jserv.c:63:19: jserv.h: No such file or directory
I think I'm getting closer to building mod_jserv.so, but this really is feeling like a wild goose chase. We try as much as possible to stick with RPM/yum for our package management but unfortunately the previous sysadmin liked to build a lot from source. Can someone help? Where do I find this jserv.h, and how do I tell apxs where it lives?
On Tue, 2008-06-17 at 11:33 -0500, Sean Carolan wrote:
I think I'm getting closer to building mod_jserv.so, but this really is feeling like a wild goose chase. We try as much as possible to stick with RPM/yum for our package management but unfortunately the previous sysadmin liked to build a lot from source. Can someone help? Where do I find this jserv.h, and how do I tell apxs where it lives?
mod_jserv is really old, are you sure it can be compiled against apache 2? If you need a jk connector, use mod_jk. You can find the source rpm in the RHWAS repository (I didn't check if CentOS has a binary version somewhere).
ciao ad
mod_jserv is really old, are you sure it can be compiled against apache 2? If you need a jk connector, use mod_jk. You can find the source rpm in the RHWAS repository (I didn't check if CentOS has a binary version somewhere).
ciao ad
Hi Andrea, thanks for your reply. I know mod_jserv is ancient, but we have to support it because it's still being used on production machines. Will mod_jk connect in the same way that mod_jserv does?
Hi Andrea, thanks for your reply. I know mod_jserv is ancient, but we have to support it because it's still being used on production machines. Will mod_jk connect in the same way that mod_jserv does?
I have mod_jk module properly loaded now, how would I duplicate this function of jserv with mod_jk?
<IfModule mod_jserv.c> ApJServMount /servlets ajpv12://servername.com:8008/root ApjServAction .html /servlets/gnujsp </IfModule>
I have mod_jk module properly loaded now, how would I duplicate this function of jserv with mod_jk?
<IfModule mod_jserv.c> ApJServMount /servlets ajpv12://servername.com:8008/root ApjServAction .html /servlets/gnujsp </IfModule>
I should add that "servername.com" is localhost, so this could certainly be a local file path rather than a URL.
I found this on the mod_jk howto from the apache site:
********************* For example the following directives will send all requests ending in .jsp or beginning with /servlet to the "ajp13" worker, but jsp requests to files located in /otherworker will go to "remoteworker".
JkMount /*.jsp ajp13 JkMount /servlet/* ajp13 JkMount /otherworker/*.jsp remoteworker
You can use the JkMount directive at the top level or inside <VirtualHost> sections of your httpd.conf file. *********************
I just need to figure out how to get the equivalent JkMount command to replace the old ApJServMount from before. If anyone has an idea how this is done please let me know. Most of the documentation I have read is tomcat-specific, whereas I am not actually using this to connect to a modern Tomcat server.
On Tue, 2008-06-17 at 13:09 -0500, Sean Carolan wrote:
I found this on the mod_jk howto from the apache site:
For example the following directives will send all requests ending in .jsp or beginning with /servlet to the "ajp13" worker, but jsp requests to files located in /otherworker will go to "remoteworker".
JkMount /*.jsp ajp13 JkMount /servlet/* ajp13 JkMount /otherworker/*.jsp remoteworkerYou can use the JkMount directive at the top level or inside <VirtualHost> sections of your httpd.conf file.
I just need to figure out how to get the equivalent JkMount command to replace the old ApJServMount from before. If anyone has an idea how this is done please let me know. Most of the documentation I have read is tomcat-specific, whereas I am not actually using this to connect to a modern Tomcat server.
It doesn't matter. You can use the ajp12 connector if your application server doesn't support ajp13.
The server URL (remoteworker in the above example) has to be configured in a different file. You'll find two example files attached. mod_jk.conf has to be included by httpd.conf, workers.properties is referred by mod_jk.
ciao andrea
Andrea thank you again for your help. I think I have almost got this set up right. I copied your workers.properties file and the appropriate entries from mod_jk.conf and now I can connect, but get a 400 error. I only have the default Apache site configured on this box, and my mod_jk.conf file inside of /etc/httpd/conf.d/. Any idea how to correct the "No body with status=400"? All i really need is a simple page that we can check to see whether it's up or down. Sorry for this long post, I want to make sure you have all the info. Here are my config files:
mod_jk.conf:
************ <IfModule !mod_jk.c> LoadModule jk_module modules/mod_jk.so </IfModule>
<IfModule mod_jk.c> JkWorkersFile "/etc/httpd/conf/workers.properties" JkLogFile "/var/log/httpd/mod_jk.log" JkLogLevel debug JkLogStampFormat "[%a %b %d %H:%M:%S %Y]" JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories </IfModule>
JkMount /* ajp12 JkMount /servlets ajp12 *************
workers.properties - exactly the same as yours except that our java runs on port 8008
Main apache config file - unchanged from the way it comes out-of-the-box. It was working fine as the default CentOS page until I added mod_jk to the mix.
Debug log output:
[Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_request::jk_ajp12_worker.c (367): Into ajpv12_handle_request [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_request::jk_ajp12_worker.c (371): ajpv12_handle_request, sending the ajp12 start sequence [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_request::jk_ajp12_worker.c (429): ajpv12_handle_request, sending the headers [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_request::jk_ajp12_worker.c (447): ajpv12_handle_request, sending the terminating mark [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_request::jk_ajp12_worker.c (497): ajpv12_handle_request done [Tue Jun 17 13:55:43 2008][7336:47264] [debug] service::jk_ajp12_worker.c (123): In jk_endpoint_t::service, sent request [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (512): Into ajpv12_handle_response [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (533): ajpv12_handle_response, read Status: 400 Bad Request [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (565): ajpv12_handle_response, read Status=400 Bad Request [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (533): ajpv12_handle_response, read Servlet-Error: Received empty servlet name [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (565): ajpv12_handle_response, read Servlet-Error=Received empty servlet name [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (588): ajpv12_handle_response, allocating header arrays [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (533): ajpv12_handle_response, read [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (535): ajpv12_handle_response, headers are done [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (615): ajpv12_handle_response, starting response [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (626): ajpv12_handle_response, reading response body [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (644): ajpv12_handle_response, response body is done [Tue Jun 17 13:55:43 2008][7336:47264] [debug] ajpv12_handle_response::jk_ajp12_worker.c (658): ajpv12_handle_response done [Tue Jun 17 13:55:43 2008][7336:47264] [debug] done::jk_ajp12_worker.c (139): Into jk_endpoint_t::done [Tue Jun 17 13:55:43 2008][7336:47264] [info] jk_handler::mod_jk.c (2040): No body with status=400 for worker=ajp12
Sean Carolan wrote:
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
I'm not too famillar with those JkOptions but looking at my old mod_jk configs I have no JkOptions defined, try removing them and see if anything changes? My old configs were ajp13, so perhaps they might be needed with ajp12, not sure.
nate
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
I'm not too famillar with those JkOptions but looking at my old mod_jk configs I have no JkOptions defined, try removing them and see if anything changes? My old configs were ajp13, so perhaps they might be needed with ajp12, not sure.
Yea, these options were in the mod_jk template that I started with. It was left here by a previous administrator. I commented this out and restarted apache, yet I get the same error.
I guess what I'm not clear on is how you replace mod_jserv's configuration:
ApJServMount /servlets ajpv12://host.domain.com:8008/root
with the equivalent version using JkMount.
I guess what I'm not clear on is how you replace mod_jserv's configuration:
ApJServMount /servlets ajpv12://host.domain.com:8008/rootwith the equivalent version using JkMount.
On the old server running mod_jserv our configuration looks like this:
<IfModule mod_jserv.c> ApJServMount /servlets ajpv12://hostname.web.domain.com:8008/root ApjServAction .html /servlets/gnujsp </IfModule>
How do we duplicate this line with mod_jk?
ApjServAction .html /servlets/gnujsp
Sean Carolan wrote:
I guess what I'm not clear on is how you replace mod_jserv's configuration:
ApJServMount /servlets ajpv12://host.domain.com:8008/rootwith the equivalent version using JkMount.
On the old server running mod_jserv our configuration looks like this:
<IfModule mod_jserv.c> ApJServMount /servlets ajpv12://hostname.web.domain.com:8008/root ApjServAction .html /servlets/gnujsp </IfModule>
How do we duplicate this line with mod_jk?
ApjServAction .html /servlets/gnujsp
What does ApjServAction do ?
Might it be
JkMount /*.html ajp12
assuming ajp12 is the name of your worker in worker.properties
nate
Might it be
JkMount /*.html ajp12
assuming ajp12 is the name of your worker in worker.properties
Yea, I tried that and even just a simple wildcard like this:
JkMount /* ajp12
but no dice. If I can't solve this then I may have to just install apache 1.3 everywhere to get it working.
In our environment we have many legacy application servers running
apache/jserv. There is a web server front end, then a couple of load-balanced java servers on the backside. One of the problems we are faced with is hung or stuck jvms. I have looked at the java process with the ps command, and there are many times when URL(s) do not respond, yet the java looks healthy, at least from the OS point of view. The usual cure for this situation is to restart the JVM, then the URLs come right back up.
Are any of you aware of tools for monitoring apache jserv, either from
localhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone.
Hello,
I don't know jserv or about eventual specific jserv support, but Hyperic might be part of the answer. I know it can provide metrics about Tomcat and JVMs, and application/server specific plugins can be written without too much effort. It's by far the most powerful/complete monitoring solution I've seen, providing support out of the box or with plugins for a bunch of services and applications. Check http://support.hyperic.com/confluence/display/hypcomm/HyperForge/#HyperFORGE... for existing plugins. Perhaps what you want can be done with a JMX plugin ?
Really worth a try anyway...
Check http://support.hyperic.com/confluence/display/hypcomm/HyperForge/#HyperFORGE... for existing plugins. Perhaps what you want can be done with a JMX plugin ?
Hyperic looks interesting, but anytime someone claims "Zero-Touch Systems Management" I have to raise a skeptical eyebrow. I will take a look at it and install the demo, but our company has already invested in a different monitoring system. I was really hoping for something simple, perhaps a status page, jserv management port, shell script perhaps...
On Tue, Jun 10, 2008 at 5:07 PM, Sean Carolan scarolan@gmail.com wrote: <snip>
Are any of you aware of tools for monitoring apache jserv, either from localhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone.
The Sun JDK provides tools for pulling info from a running JVM.
http://java.sun.com/javase/6/webnotes/trouble/other/tools6-Unix.html
Not jserv (time for an upgrade?) specific, but attaches directly to the underlying JVM. The one I've played with is jmap. Of course, you need to be running a recent JVM. I do not proclaim to be an expert with these tools, but it may be of some use in your situation.
Sean Carolan wrote:
In our environment we have many legacy application servers running apache/jserv. There is a web server front end, then a couple of load-balanced java servers on the backside. One of the problems we are faced with is hung or stuck jvms. I have looked at the java process with the ps command, and there are many times when URL(s) do not respond, yet the java looks healthy, at least from the OS point of view. The usual cure for this situation is to restart the JVM, then the URLs come right back up.
Are any of you aware of tools for monitoring apache jserv, either from localhost or by connecting to port 8008 over the network? I really want to find out if there is a way to detect a "sick" JVM other than getting a bunch of down URL alerts on my phone.
Being far from an expert, if restarting a service or script is what's needed, you might find SIM[1] helpful
YMMV, -R
-
Andrea Dell'Amico -
Jeff -
Les Mikesell -
nate -
Ray Leventhal -
Sean Carolan -
Vincent Knecht