Check and Restart Apache if it is malfunctioning (not returning HTML content) shell script

Check and Restart Apache Webserver on Malfunction, Apache feather logo

One of the company Debian Lenny 5.0 Webservers, where I'm working as sys admin sometimes stops to properly server HTTP requests.
Whenever this oddity happens, the Apache server seems to be running okay but it is not failing to return requested content

I can see the webserver listens on port 80 and establishing connections to remote hosts – the apache processes show normally as I can see in netstat …:

apache:~# netstat -enp 80
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 xxx.xxx.xxx.xx:80 46.253.9.36:5665 SYN_RECV 0 0 -
tcp 0 0 xxx.xxx.xxx.xx:80 78.157.26.24:5933 SYN_RECV 0 0 -
...

Also the apache forked child processes show normally in process list:

apache:~# ps axuwwf|grep -i apache
root 46748 0.0 0.0 112300 872 pts/1 S+ 18:07 0:00 _ grep -i apache
root 42530 0.0 0.1 217392 6636 ? Ss Mar14 0:39 /usr/sbin/apache2 -k start
www-data 42535 0.0 0.0 147876 1488 ? S Mar14 0:01 _ /usr/sbin/apache2 -k start
root 28747 0.0 0.1 218180 4792 ? Sl Mar14 0:00 _ /usr/sbin/apache2 -k start
www-data 31787 0.0 0.1 219156 5832 ? S Mar14 0:00 | _ /usr/sbin/apache2 -k start

In spite of that, in any client browser to any of the Apache (Virtual hosts) websites, there is no HTML content returned…
This weird problem continues until the Apache webserver is retarted.
Once webserver is restarted everything is back to normal.
I use Apache Check Apache shell script set on few remote hosts to regularly check with nmap if port 80 (www) of my server is open and responding, anyways this script just checks if the open and reachable and thus using it was unable to detect Apache wasn't able to return back HTML content.
To work around the malfunctions I wrote tiny script – retart_apache_if_empty_content_is_returned.sh

The scripts idea is very simple;
A request is made a remote defined host with lynx text browser, then the output of lines is counted, if the output returned by lynx -dump http://someurl.com is less than the number returned whether normally invoked, then the script triggers an apache init script restart.

I've set the script to periodically run in a cron job, every 5 minutes each hour.
# check if apache returns empty content with lynx and if yes restart and log it
*/5 * * * * /usr/sbin/restart_apache_if_empty_content.sh >/dev/null 2>&1

This is not perfect as sometimes still, there will be few minutes downtime, but at least the downside will not be few hours until I am informed ssh to the server and restart Apache manually

A quick way to download and set from cron execution my script every 5 minutes use:

apache:~# cd /usr/sbin
apache:/usr/sbin# wget -q http://www.pc-freak.net/bscscr/restart_apache_if_empty_content.sh
apache:/usr/sbin# chmod +x restart_apache_if_empty_content.sh
apache:/usr/sbin# crontab -l > /tmp/file; echo '*/5 * * * *' /usr/sbin/restart_apache_if_empty_content.sh 2>&1 >/dev/null

 

Share this on

Download PDFDownload PDF

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

7 Responses to “Check and Restart Apache if it is malfunctioning (not returning HTML content) shell script”

  1. Timdaman says:

    Have you seen “monit”? It is my tool of choice for these type of problems.

    • admin says:

      Yes, I know monit, but I think monit will not restart Apache if the apache server is running but not returning any content to clients?
      Did you have a case like mine and used monit to solve it?

      best

  2. Justin Andrusk says:

    Have you determined the root cause to the issue? Just wondering how often this happens and I wonder if running a watch job on say, lsof to see where the last return on content is left.

    • admin says:

      Hi Justin,

      No unfrotunately, yet I didn’t got it. I will further investigate and watch it closely. If I got the issue I will drop a comment here.

      Best
      Georgi

  3. Timdaman says:

    Not exactly but close. I have a webserver that proxy’s out to some backend servers which are crash prone. Monit can alert when a non sucessful return code(i.e. return code fo 500). This is how I have it configured

    check host site1 with address 1.2.3.4
    alert admin@example.com
    stop program = “/usr/local/bin/sudo /usr/local/stop_site_JVMs.sh -site site1″
    start program = “/usr/local/bin/sudo /usr/local/start_site_JVMs.sh -site site1″
    if 10 restarts within 100 cycles then timeout
    if failed port 80 protocol http and request “/site1″ then restart
    mode active
    group RATES

    In the monit documentation it states monit can also alert if a checksum or regular expression don’t match though I have not use these myself. See here for more info
    http://mmonit.com/monit/documentation/monit.html#connection_testing
    and serach for “checksum” or “exspect “

    As to why I am not monitoring the backend processes, if I monitor apache I monitor more closely what the customer would see and thus catch apache problems as well.

    • admin says:

      Hi,

      Thx a lot for the info, maybe I will give the monit with return code.
      By the way, the same server, where I’ve placed this apache check and restart script has been overloaded today. The reason was munin was doing some things in the background completely stoning the server. Now I’ve disabled the munin-node. Will monitor now to see if the strange Apache issue re-occur.

      Will be seeing you around.

      Best
      Georgi

  4. Ted Best says:

    Thank you for this very useful script. There is definitely a need for good shell scripts like these to help beginning sysadmins!

Leave a Reply