One of the company Debian Lenny 5.0 Webservers, where I'm working as sys admin sometimes stops to properly server HTTP requests.
Whenever this oddity happens, the Apache server seems to be running okay but it is not failing to return requested content
I can see the webserver listens on port 80 and establishing connections to remote hosts – the apache processes show normally as I can see in netstat …:
apache:~# netstat -enp 80
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 xxx.xxx.xxx.xx:80 46.253.9.36:5665 SYN_RECV 0 0 -
tcp 0 0 xxx.xxx.xxx.xx:80 78.157.26.24:5933 SYN_RECV 0 0 -
...
Also the apache forked child processes show normally in process list:
apache:~# ps axuwwf|grep -i apache
root 46748 0.0 0.0 112300 872 pts/1 S+ 18:07 0:00 _ grep -i apache
root 42530 0.0 0.1 217392 6636 ? Ss Mar14 0:39 /usr/sbin/apache2 -k start
www-data 42535 0.0 0.0 147876 1488 ? S Mar14 0:01 _ /usr/sbin/apache2 -k start
root 28747 0.0 0.1 218180 4792 ? Sl Mar14 0:00 _ /usr/sbin/apache2 -k start
www-data 31787 0.0 0.1 219156 5832 ? S Mar14 0:00 | _ /usr/sbin/apache2 -k start
In spite of that, in any client browser to any of the Apache (Virtual hosts) websites, there is no HTML content returned…
This weird problem continues until the Apache webserver is retarted.
Once webserver is restarted everything is back to normal.
I use Apache Check Apache shell script set on few remote hosts to regularly check with nmap if port 80 (www) of my server is open and responding, anyways this script just checks if the open and reachable and thus using it was unable to detect Apache wasn't able to return back HTML content.
To work around the malfunctions I wrote tiny script – retart_apache_if_empty_content_is_returned.sh
The scripts idea is very simple;
A request is made a remote defined host with lynx text browser, then the output of lines is counted, if the output returned by lynx -dump http://someurl.com is less than the number returned whether normally invoked, then the script triggers an apache init script restart.
I've set the script to periodically run in a cron job, every 5 minutes each hour.
# check if apache returns empty content with lynx and if yes restart and log it
*/5 * * * * /usr/sbin/restart_apache_if_empty_content.sh >/dev/null 2>&1
This is not perfect as sometimes still, there will be few minutes downtime, but at least the downside will not be few hours until I am informed ssh to the server and restart Apache manually …
A quick way to download and set from cron execution my script every 5 minutes use:
apache:~# cd /usr/sbin
apache:/usr/sbin# wget -q https://www.pc-freak.net/bscscr/restart_apache_if_empty_content.sh
apache:/usr/sbin# chmod +x restart_apache_if_empty_content.sh
apache:/usr/sbin# crontab -l > /tmp/file; echo '*/5 * * * *' /usr/sbin/restart_apache_if_empty_content.sh 2>&1 >/dev/null
More helpful Articles
Tags: address state, apache processes, apache server, apache webserver, apache2, apacheroot, Auto, checks, child processes, client, client browser, content, cron, Draft, enp, everything, grep, HTML, HTTP, internet connections, lenny, Lynx, nmap, oddity, program, proto, Restart, restart apache, root, script, scripts, Shell, shell script, show, spite, ss, SYN, User, Virtual, virtual hosts, webservers, weird problem
Mozilla/5.0 (iPod; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B176 Safari/7534.48.3
Have you seen “monit”? It is my tool of choice for these type of problems.
View CommentView CommentMozilla/5.0 (Windows NT 6.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Yes, I know monit, but I think monit will not restart Apache if the apache server is running but not returning any content to clients?
Did you have a case like mine and used monit to solve it?
best
View CommentView CommentMozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Have you determined the root cause to the issue? Just wondering how often this happens and I wonder if running a watch job on say, lsof to see where the last return on content is left.
View CommentView CommentMozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+ Debian/squeeze (2.30.6-1) Epiphany/2.30.6
Hi Justin,
No unfrotunately, yet I didn’t got it. I will further investigate and watch it closely. If I got the issue I will drop a comment here.
Best
View CommentView CommentGeorgi
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0
Not exactly but close. I have a webserver that proxy’s out to some backend servers which are crash prone. Monit can alert when a non sucessful return code(i.e. return code fo 500). This is how I have it configured
check host site1 with address 1.2.3.4
alert admin@example.com
stop program = “/usr/local/bin/sudo /usr/local/stop_site_JVMs.sh -site site1”
start program = “/usr/local/bin/sudo /usr/local/start_site_JVMs.sh -site site1”
if 10 restarts within 100 cycles then timeout
if failed port 80 protocol http and request “/site1” then restart
mode active
group RATES
In the monit documentation it states monit can also alert if a checksum or regular expression don’t match though I have not use these myself. See here for more info
http://mmonit.com/monit/documentation/monit.html#connection_testing
and serach for “checksum” or “exspect “
As to why I am not monitoring the backend processes, if I monitor apache I monitor more closely what the customer would see and thus catch apache problems as well.
View CommentView CommentMozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+ Debian/squeeze (2.30.6-1) Epiphany/2.30.6
Hi,
Thx a lot for the info, maybe I will give the monit with return code.
By the way, the same server, where I’ve placed this apache check and restart script has been overloaded today. The reason was munin was doing some things in the background completely stoning the server. Now I’ve disabled the munin-node. Will monitor now to see if the strange Apache issue re-occur.
Will be seeing you around.
Best
View CommentView CommentGeorgi
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Thank you for this very useful script. There is definitely a need for good shell scripts like these to help beginning sysadmins!
View CommentView Comment