Hey all,
I'm having a little bit of a weird issue on my web host. I was just wondering if anyone's seen anything like this before.
The problem is that you can't seem to restart apache via the stock init script that comes with apache:
[root@beta:~] #service httpd restart Stopping httpd: [FAILED] Starting httpd: [Mon Mar 03 11:21:14 2014] [warn] module php5_module is already loaded, skipping (98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80http://0.0.0.0/ no listening sockets available, shutting down Unable to open logs [FAILED] [root@beta:~] #
IT complains that apache can't restart because apache is already listening.
So what you have to do in this case is kill it like so:
[root@beta:~] #pgrep httpd 12322 22405 22505 22506 22507 22508 22509 22510 22511 22512 [root@beta:~] #
[root@beta:~] #pkill httpd
And repeat as necessary until you can achieve this:
[root@beta:~] #pgrep httpd [root@beta:~] #
It's a real pain with it being the way it is, because this makes it impossible for puppet to manage this service. I'm mainly just tired of seeing errors in the reports about this! :)
So whaddya think? Have you ever seen this problem before and do you know how to solve this issue?
Thanks, Tim
On Mon, Mar 3, 2014 at 3:28 PM, Tim Dunphy bluethundr@gmail.com wrote:
Hey all,
I'm having a little bit of a weird issue on my web host. I was just wondering if anyone's seen anything like this before.
The problem is that you can't seem to restart apache via the stock init script that comes with apache:
[root@beta:~] #service httpd restart Stopping httpd: [FAILED]
That's not normal. It should say [OK] there.
So what you have to do in this case is kill it like so:
It is supposed to be killing the pid found in /var/run/httpd/httpd.pid. Can you tell why that doesn't work? You might get a hint from: sh -x /etc/init.d/httpd restart
On 03/03/2014 03:28 PM, Tim Dunphy wrote:
Hey all,
I'm having a little bit of a weird issue on my web host. I was just wondering if anyone's seen anything like this before.
The problem is that you can't seem to restart apache via the stock init script that comes with apache:
[root@beta:~] #service httpd restart Stopping httpd: [FAILED] Starting httpd: [Mon Mar 03 11:21:14 2014] [warn] module php5_module is already loaded, skipping (98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80http://0.0.0.0/ no listening sockets available, shutting down Unable to open logs [FAILED] [root@beta:~] #
IT complains that apache can't restart because apache is already listening.
So what you have to do in this case is kill it like so:
[root@beta:~] #pgrep httpd 12322 22405 22505 22506 22507 22508 22509 22510 22511 22512 [root@beta:~] #
[root@beta:~] #pkill httpd
And repeat as necessary until you can achieve this:
[root@beta:~] #pgrep httpd [root@beta:~] #
It's a real pain with it being the way it is, because this makes it impossible for puppet to manage this service. I'm mainly just tired of seeing errors in the reports about this! :)
So whaddya think? Have you ever seen this problem before and do you know how to solve this issue?
Thanks, Tim
Are you sure all the instances were started by the stock initscript initially ... it sounds like something is killing or creating apache instances that are in /var/run/httpd/
That normally happens if httpd is started manually by another script or user process rather than the one in /etc/init.d/
On 03/03/2014 05:01 PM, Johnny Hughes wrote:
On 03/03/2014 03:28 PM, Tim Dunphy wrote:
Hey all,
I'm having a little bit of a weird issue on my web host. I was just wondering if anyone's seen anything like this before.
The problem is that you can't seem to restart apache via the stock init script that comes with apache:
[root@beta:~] #service httpd restart Stopping httpd: [FAILED] Starting httpd: [Mon Mar 03 11:21:14 2014] [warn] module php5_module is already loaded, skipping (98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80http://0.0.0.0/ no listening sockets available, shutting down Unable to open logs [FAILED] [root@beta:~] #
IT complains that apache can't restart because apache is already listening.
So what you have to do in this case is kill it like so:
[root@beta:~] #pgrep httpd 12322 22405 22505 22506 22507 22508 22509 22510 22511 22512 [root@beta:~] #
[root@beta:~] #pkill httpd
And repeat as necessary until you can achieve this:
[root@beta:~] #pgrep httpd [root@beta:~] #
It's a real pain with it being the way it is, because this makes it impossible for puppet to manage this service. I'm mainly just tired of seeing errors in the reports about this! :)
So whaddya think? Have you ever seen this problem before and do you know how to solve this issue?
Thanks, Tim
Are you sure all the instances were started by the stock initscript initially ... it sounds like something is killing or creating apache instances that are in /var/run/httpd/
I meant something else is starting httpd and NOT putting the info in /var/run/httpd/
That normally happens if httpd is started manually by another script or user process rather than the one in /etc/init.d/
Hey thanks for the input!
Are you sure all the instances were started by the stock initscript
initially ... it sounds like something is killing or creating apache instances that are in /var/run/httpd/ That normally happens if httpd is started manually by another script or user process rather than the one in /etc/init.d/
I meant something else is starting httpd and NOT putting the info in
Absolutely positive that apache is not being started any other way than the init script in /etc/init.d.
Thanks Tim
On Mon, Mar 3, 2014 at 6:03 PM, Johnny Hughes johnny@centos.org wrote:
On 03/03/2014 05:01 PM, Johnny Hughes wrote:
On 03/03/2014 03:28 PM, Tim Dunphy wrote:
Hey all,
I'm having a little bit of a weird issue on my web host. I was just wondering if anyone's seen anything like this before.
The problem is that you can't seem to restart apache via the stock init script that comes with apache:
[root@beta:~] #service httpd restart Stopping httpd: [FAILED] Starting httpd: [Mon Mar 03 11:21:14 2014] [warn] module php5_module is already loaded, skipping (98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80http://0.0.0.0/ no listening sockets available, shutting down Unable to open logs [FAILED] [root@beta:~] #
IT complains that apache can't restart because apache is already
listening.
So what you have to do in this case is kill it like so:
[root@beta:~] #pgrep httpd 12322 22405 22505 22506 22507 22508 22509 22510 22511 22512 [root@beta:~] #
[root@beta:~] #pkill httpd
And repeat as necessary until you can achieve this:
[root@beta:~] #pgrep httpd [root@beta:~] #
It's a real pain with it being the way it is, because this makes it impossible for puppet to manage this service. I'm mainly just tired of seeing errors in the reports about this! :)
So whaddya think? Have you ever seen this problem before and do you know how to solve this issue?
Thanks, Tim
Are you sure all the instances were started by the stock initscript initially ... it sounds like something is killing or creating apache instances that are in /var/run/httpd/
I meant something else is starting httpd and NOT putting the info in /var/run/httpd/
That normally happens if httpd is started manually by another script or user process rather than the one in /etc/init.d/
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Tue, Mar 4, 2014 at 3:42 PM, Tim Dunphy bluethundr@gmail.com wrote:
Hey thanks for the input!
Are you sure all the instances were started by the stock initscript
initially ... it sounds like something is killing or creating apache instances that are in /var/run/httpd/ That normally happens if httpd is started manually by another script or user process rather than the one in /etc/init.d/
I meant something else is starting httpd and NOT putting the info in
Absolutely positive that apache is not being started any other way than the init script in /etc/init.d.
So does the process id in /var/run/httpd/httpd.pid belong to the master apache instance, and does killing it kill all of them?
Hey guys,
Well it took a little while for me to be able to reproduce this. It seems that this problem is intermittent and sporadic.
But I tried running a sh -x /etc/init.d/httpd restart command once I reallized I had another incident of this and this is what I saw as the output:
+ . /etc/rc.d/init.d/functions ++ TEXTDOMAIN=initscripts ++ umask 022 ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin ++ export PATH ++ '[' -z '' ']' ++ COLUMNS=80 ++ '[' -z '' ']' +++ /sbin/consoletype ++ CONSOLETYPE=pty ++ '[' -f /etc/sysconfig/i18n -a -z '' ']' ++ . /etc/profile.d/lang.sh +++ sourced=0 +++ '[' -z '' -a -n en_US.UTF-8 ']' +++ sourced=1 +++ '[' -n '' ']' +++ '[' 1 = 1 ']' +++ '[' -n en_US.UTF-8 ']' +++ export LANG +++ '[' -n '' ']' +++ unset LC_ADDRESS +++ '[' -n '' ']' +++ unset LC_CTYPE +++ '[' -n '' ']' +++ unset LC_COLLATE +++ '[' -n '' ']' +++ unset LC_IDENTIFICATION +++ '[' -n '' ']' +++ unset LC_MEASUREMENT +++ '[' -n '' ']' +++ unset LC_MESSAGES +++ '[' -n '' ']' +++ unset LC_MONETARY +++ '[' -n '' ']' +++ unset LC_NAME +++ '[' -n '' ']' +++ unset LC_NUMERIC +++ '[' -n '' ']' +++ unset LC_PAPER +++ '[' -n '' ']' +++ unset LC_TELEPHONE +++ '[' -n '' ']' +++ unset LC_TIME +++ '[' -n '' ']' +++ unset LC_ALL +++ '[' -n '' ']' +++ unset LANGUAGE +++ '[' -n '' ']' +++ unset LINGUAS +++ '[' -n '' ']' +++ unset _XKB_CHARSET +++ consoletype=pty +++ '[' -z pty ']' +++ '[' -n '' ']' +++ '[' -n '' ']' +++ '[' -n en_US.UTF-8 ']' +++ case $LANG in +++ '[' screen = linux ']' +++ unset SYSFONTACM SYSFONT +++ unset sourced +++ unset langfile ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/init ']' ++ . /etc/sysconfig/init +++ BOOTUP=color +++ GRAPHICAL=yes +++ RES_COL=60 +++ MOVE_TO_COL='echo -en \033[60G' +++ SETCOLOR_SUCCESS='echo -en \033[0;32m' +++ SETCOLOR_FAILURE='echo -en \033[0;31m' +++ SETCOLOR_WARNING='echo -en \033[0;33m' +++ SETCOLOR_NORMAL='echo -en \033[0;39m' +++ LOGLEVEL=3 +++ PROMPT=yes +++ AUTOSWAP=no ++ '[' pty = serial ']' ++ '[' color '!=' verbose ']' ++ INITLOG_ARGS=-q ++ __sed_discard_ignored_files='/(~|.bak|.orig|.rpmnew|.rpmorig|.rpmsave)$/d' + '[' -f /etc/sysconfig/httpd ']' + . /etc/sysconfig/httpd + HTTPD_LANG=C + INITLOG_ARGS= + apachectl=/usr/sbin/apachectl + httpd=/usr/sbin/httpd + prog=httpd + pidfile=/var/run/httpd/httpd.pid + lockfile=/var/lock/subsys/httpd + RETVAL=0 + case "$1" in + stop + echo -n 'Stopping httpd: ' Stopping httpd: + killproc -p /var/run/httpd/httpd.pid -d 10 /usr/sbin/httpd + local RC killlevel= base pid pid_file= delay + RC=0 + delay=3 + '[' 5 -eq 0 ']' + '[' -p = -p ']' + pid_file=/var/run/httpd/httpd.pid + shift 2 + '[' -d = -d ']' + delay=10 + shift 2 + '[' -n '' ']' + base=httpd + __pids_var_run /usr/sbin/httpd /var/run/httpd/httpd.pid + local base=httpd + local pid_file=/var/run/httpd/httpd.pid + pid= + '[' -f /var/run/httpd/httpd.pid ']' + return 3 + '[' -z /var/run/httpd/httpd.pid -a -z '' ']' + '[' -n '' ']' + '[' -n '' -a -n '' ']' + failure 'httpd shutdown' + local rc=1 + '[' color '!=' verbose -a -z '' ']' + echo_failure + '[' color = color ']' + echo -en '\033[60G' [60G+ echo -n '[' [+ '[' color = color ']' + echo -en '\033[0;31m' [0;31m+ echo -n FAILED FAILED+ '[' color = color ']' + echo -en '\033[0;39m' [0;39m+ echo -n ']' ]+ echo -ne '\r' + return 1 + '[' -x /usr/bin/rhgb-client ']' + return 1 + RC=0 + '[' -z '' ']' + rm -f /var/run/httpd/httpd.pid + return 0 + RETVAL=0 + echo
+ '[' 0 = 0 ']' + rm -f /var/lock/subsys/httpd /var/run/httpd/httpd.pid + start + echo -n 'Starting httpd: ' Starting httpd: + LANG=C + daemon --pidfile=/var/run/httpd/httpd.pid /usr/sbin/httpd + local gotbase= force= nicelevel corelimit + local pid base= user= nice= bg= pid_file= + nicelevel=0 + '[' --pidfile=/var/run/httpd/httpd.pid '!=' -pidfile=/var/run/httpd/httpd.pid ']' + case $1 in + pid_file=/var/run/httpd/httpd.pid + shift + '[' /usr/sbin/httpd '!=' /usr/sbin/httpd ']' + '[' -z '' ']' + base=httpd + __pids_var_run httpd /var/run/httpd/httpd.pid + local base=httpd + local pid_file=/var/run/httpd/httpd.pid + pid= + '[' -f /var/run/httpd/httpd.pid ']' + return 3 + '[' -n '' -a -z '' ']' + corelimit='ulimit -S -c 0' + '[' -n '' ']' + '[' color = verbose -a -z '' ']' + '[' -z '' ']' + /bin/bash -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/httpd' (98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs + '[' 1 -eq 0 ']' + failure 'httpd startup' + local rc=1 + '[' color '!=' verbose -a -z '' ']' + echo_failure + '[' color = color ']' + echo -en '\033[60G' [60G+ echo -n '[' [+ '[' color = color ']' + echo -en '\033[0;31m' [0;31m+ echo -n FAILED FAILED+ '[' color = color ']' + echo -en '\033[0;39m' [0;39m+ echo -n ']' ]+ echo -ne '\r' + return 1 + '[' -x /usr/bin/rhgb-client ']' + return 1 + RETVAL=1 + echo
+ '[' 1 = 0 ']' + return 1 + exit 1
Not really sure how to interpret that, unfortunately.
However looked for the pid file for apache and noticed that it DOESN'T EXIST!
[root@beta:~] #ls -l /var/run/httpd/ total 0
Well, that would explain why the init script isn';t able to kill the process. Maybe puppet is doing something weird with that pid file? I don't really know offhand, but I guess I will have to investigate that.
Thanks for all your input.
Tim
On Tue, Mar 4, 2014 at 5:07 PM, Les Mikesell lesmikesell@gmail.com wrote:
On Tue, Mar 4, 2014 at 3:42 PM, Tim Dunphy bluethundr@gmail.com wrote:
Hey thanks for the input!
Are you sure all the instances were started by the stock initscript
initially ... it sounds like something is killing or creating apache instances that are in /var/run/httpd/ That normally happens if httpd is started manually by another script or user process rather than the one in /etc/init.d/
I meant something else is starting httpd and NOT putting the info in
Absolutely positive that apache is not being started any other way than
the
init script in /etc/init.d.
So does the process id in /var/run/httpd/httpd.pid belong to the master apache instance, and does killing it kill all of them?
-- Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
In article CAOZy0en0x_wrBZkVjZUpaTYMOD7Z_VTBOMorMukEDknrWNfQMA@mail.gmail.com, Tim Dunphy bluethundr@gmail.com wrote:
Hey guys,
Well it took a little while for me to be able to reproduce this. It seems that this problem is intermittent and sporadic.
But I tried running a sh -x /etc/init.d/httpd restart command once I reallized I had another incident of this and this is what I saw as the output:
- /bin/bash -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/httpd'
(98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down
Not really sure how to interpret that, unfortunately.
However looked for the pid file for apache and noticed that it DOESN'T EXIST!
[root@beta:~] #ls -l /var/run/httpd/ total 0
Well, that would explain why the init script isn';t able to kill the process. Maybe puppet is doing something weird with that pid file? I don't really know offhand, but I guess I will have to investigate that.
Thanks for all your input.
Have a look to see what process is actually doing the listening on port 80:
# netstat -natp
Look for a local address with a port of 80 and a state of LISTEN.
The final column shows you the PID and program name.
Cheers Tony
ok thanks for the tip!
So I did a netstat as you suggested and this is what I found:
[root@beta:~] #netstat -natp | grep 80 tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN 2354/python2.6 tcp 0 0 0.0.0.0:8010 0.0.0.0:* LISTEN 8198/python2.6 tcp 0 0 0.0.0.0:8013 0.0.0.0:* LISTEN 8198/python2.6 tcp 0 0 166.78.8.98:8081 0.0.0.0:* LISTEN 10950/java tcp 0 0 0.0.0.0:28017 0.0.0.0:* LISTEN 2289/mongod tcp 0 1 166.78.8.98:33612 72.52.4.74:80 SYN_SENT 17471/wget tcp 0 672 166.78.8.98:22 24.38.100.4:35265 ESTABLISHED 5680/sshd tcp 0 0 :::995 :::* LISTEN 1806/couriertcpd tcp 0 0 :::110 :::* LISTEN 1800/couriertcpd tcp 0 0 :::80 :::* LISTEN 31589/httpd
And it does look as if it's apache that's taking up port 80 and nothing else.
I also checked /var/run/httpd and saw that it was EMPTY! No pid file to be found. I had a look at the puppet manifests and couldn't see ANYTHING that could be causing the pid file to go missing.
Does anyone have any suggestions on how I can track down why the pid file keeps disappearing?
Thanks! Tim
On Fri, Mar 7, 2014 at 7:32 AM, Tony Mountifield tony@softins.co.uk wrote:
In article < CAOZy0en0x_wrBZkVjZUpaTYMOD7Z_VTBOMorMukEDknrWNfQMA@mail.gmail.com>, Tim Dunphy bluethundr@gmail.com wrote:
Hey guys,
Well it took a little while for me to be able to reproduce this. It
seems
that this problem is intermittent and sporadic.
But I tried running a sh -x /etc/init.d/httpd restart command once I reallized I had another incident of this and this is what I saw as the output:
- /bin/bash -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/sbin/httpd'
(98)Address already in use: make_sock: could not bind to address [::]:80 (98)Address already in use: make_sock: could not bind to address
0.0.0.0:80
no listening sockets available, shutting down
Not really sure how to interpret that, unfortunately.
However looked for the pid file for apache and noticed that it DOESN'T EXIST!
[root@beta:~] #ls -l /var/run/httpd/ total 0
Well, that would explain why the init script isn';t able to kill the process. Maybe puppet is doing something weird with that pid file? I
don't
really know offhand, but I guess I will have to investigate that.
Thanks for all your input.
Have a look to see what process is actually doing the listening on port 80:
# netstat -natp
Look for a local address with a port of 80 and a state of LISTEN.
The final column shows you the PID and program name.
Cheers Tony -- Tony Mountifield Work: tony@softins.co.uk - http://www.softins.co.uk Play: tony@mountifield.org - http://tony.mountifield.org _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
In article CAOZy0enCsMUGAdp631bhuARosUo6TUQJp=zym4L3ccSN6-9ddQ@mail.gmail.com, Tim Dunphy bluethundr@gmail.com wrote:
ok thanks for the tip!
So I did a netstat as you suggested and this is what I found:
[root@beta:~] #netstat -natp | grep 80 tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN 2354/python2.6 tcp 0 0 0.0.0.0:8010 0.0.0.0:* LISTEN 8198/python2.6 tcp 0 0 0.0.0.0:8013 0.0.0.0:* LISTEN 8198/python2.6 tcp 0 0 166.78.8.98:8081 0.0.0.0:* LISTEN 10950/java tcp 0 0 0.0.0.0:28017 0.0.0.0:* LISTEN 2289/mongod tcp 0 1 166.78.8.98:33612 72.52.4.74:80 SYN_SENT 17471/wget tcp 0 672 166.78.8.98:22 24.38.100.4:35265 ESTABLISHED 5680/sshd tcp 0 0 :::995 :::* LISTEN 1806/couriertcpd tcp 0 0 :::110 :::* LISTEN 1800/couriertcpd tcp 0 0 :::80 :::* LISTEN 31589/httpd
And it does look as if it's apache that's taking up port 80 and nothing else.
I also checked /var/run/httpd and saw that it was EMPTY! No pid file to be found. I had a look at the puppet manifests and couldn't see ANYTHING that could be causing the pid file to go missing.
Does anyone have any suggestions on how I can track down why the pid file keeps disappearing?
It's probably a case of piecing together bits of evidence, e.g.
- "ps -fp 31589" to see when the process started.
- Compare that with /var/log/httpd/error_log* - apache logs a message there when it starts up.
- Do "ls -ld /var/run/httpd" to see when /var/sun/httpd was last changed (due to the deletion of httpd.pid)
- Look through logfiles in /var/log and /var/log/httpd for anything that happened just at that time.
- Kill off the httpd process manually using "kill 31589" (or whatever) and check with "ps -ef" that all instances of httpd disappear.
- Start it up again with "service httpd start" and then watch more closely.
Hope you manage to find an explanation!
Cheers Tony
On Fri, Mar 7, 2014 at 5:37 AM, Tim Dunphy bluethundr@gmail.com wrote:
Not really sure how to interpret that, unfortunately.
However looked for the pid file for apache and noticed that it DOESN'T EXIST!
[root@beta:~] #ls -l /var/run/httpd/ total 0
Well, that would explain why the init script isn';t able to kill the process. Maybe puppet is doing something weird with that pid file? I don't really know offhand, but I guess I will have to investigate that.
Is one created at a successful startup? And how is puppet involved?
Does the same issue arise if the restart is split into a stop and start? My thinking is that the stop IS working, but is taking longer than the script expects, so the stop step fails when the program checks the PID to see if it has shutdown properly. Then when the start happens the Apache has not completely shutdown and is still holding the port. After the failure the stop completes and the PID file is deleted. The PID file could be empty because the startup creates the PID file, but can't write the PID to it because the startup failed.
I'm just guessing though, as I don't have a CENTOS/RHEL system to look at. I'm not even sure if my suggested scenario makes sense!
Cheers,
Cliff
On Sat, Mar 8, 2014 at 4:28 AM, Les Mikesell lesmikesell@gmail.com wrote:
On Fri, Mar 7, 2014 at 5:37 AM, Tim Dunphy bluethundr@gmail.com wrote:
Not really sure how to interpret that, unfortunately.
However looked for the pid file for apache and noticed that it DOESN'T EXIST!
[root@beta:~] #ls -l /var/run/httpd/ total 0
Well, that would explain why the init script isn';t able to kill the process. Maybe puppet is doing something weird with that pid file? I
don't
really know offhand, but I guess I will have to investigate that.
Is one created at a successful startup? And how is puppet involved?
-- Les Mikesell lesmikesell@gmail.com _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Cliff Pratt -
Johnny Hughes -
Les Mikesell -
Tim Dunphy -
tony＠softins.co.uk