Posts Tagged ‘apache semaphores break’

Apache Webserver: No space left on device: Couldn’t create accept lock /var/lock/apache2/accept.lock – Fix

Wednesday, April 8th, 2015

Apache-http-server-no-space-left-on-device-semaphores-quotes-hard-disk-space-resolve-fix-howto
If out of a sudden your Apache webserver crashes and is refusing to start up by manually trying to restart it through its init script on Debian Linux servers – /etc/init.d/apache2 and RPM based ones: /etc/init.d/httpd

Checking in php_error.log there was no shown errors related to loading PHP modules, however apache's error.log show following errors:

[Wed Apr 08 14:20:14 2015] [error] [client 180.76.5.149] client denied by server configuration: /var/www/sploits/info/trojans_info/tr_data/y3190.html
[Wed Apr 08 14:20:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:20:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.15974) (5)
[Wed Apr 08 14:25:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:25:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16790) (5)
[Wed Apr 08 14:27:03 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:03 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16826) (5)
[Wed Apr 08 14:27:53 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:53 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16852) (5)
[Wed Apr 08 14:30:48 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:30:48 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17710) (5)
[Wed Apr 08 14:31:21 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:31:21 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17727) (5)
[Wed Apr 08 14:32:40 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:32:40 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17780) (5)
[Wed Apr 08 14:38:32 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?

As you can read the most likely reason behind above errors preventing for apache to start is /var/run/apache2.pid  unable to be properly written due to lack of disk space or due to disk quota set for users including for userID with which Apache is running.

First thing I did is of course to see how much free space is on the server:

df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg00-rootvol       4.0G  1.7G  2.2G  44% /
udev                           7.8G  204K  7.8G   1% /dev
tmpfs                           24G     0   24G   0% /dev/shm
/dev/sda1                      486M   40M  422M   9% /boot
/dev/mapper/vg00-lv_crashdump 1008M   34M  924M   4% /crashdump
/dev/mapper/vg00-homevol       496M   26M  445M   6% /home
/dev/mapper/vg00-lv_opt         12G  1.4G  9.9G  13% /opt
/dev/mapper/vg00-tmpvol        2.0G   68M  1.9G   4% /tmp
/dev/mapper/vg00-varvol        7.9G  609M  6.9G   8% /var
/dev/mapper/vg00-crashvol      1.9G   35M  1.8G   2% /var/crash
/dev/mapper/vg00-auditvol      124M  5.6M  113M   5% /var/log/audit
/dev/mapper/vg00-webdienste     60G   12G   48G  19% /webservice

 

As visible from above df command output , there is enough disk on HDD, so this is definitely not the issue:

Then I Checked whether there is Quota enabled on the Linux server with repquota command shows there are no quotas enabled:

# repquota / var/
repquota: Mountpoint (or device) / not found or has no quota enabled.
repquota: Mountpoint (or device) /var not found or has no quota enabled.
repquota: Not all specified mountpoints are using quota.

 

So obviously the only few left possible reason for Apache failing to start after invoked via init script is  either due to left tainted semaphores or due to some server hardware  RAM problem / or a dying  hard disk with bad blocks.

So what are Semaphores? Generally speaking Semaphores are apparatus for conveying information by means of visual signals between applications (something like sockets).They're used for communicating between the active processes of a certain application. In the case of Apache, they’re used to communicate between the parent and child processes, hence if Apache can’t properly write and coordinate these things down, then it can’t communicate properly with all of the processes it starts and hence the Main HTTPD process can't spawn probably its childs preventing Webserver to enter "started mode" and write its PID file.

To check general information about system semaphore arrays there is the ipcs -s command, however my experience is that ipcs -a is more useful (because it lists generally all kind of semaphores) including Semaphore Shared Memory Signals which are the most likely to cause you the problem.

ipcs -a

—— Shared Memory Segments ——–
key        shmid      owner      perms      bytes      nattch     status

—— Semaphore Arrays ——–
key        semid      owner      perms      nsems
0x00000000 22970368   www-data   600        1

—— Message Queues ——–
key        msqid      owner      perms      used-bytes   messages

As you see in my case there is a Semaphore Arrays which had to be cleaned to make Apache2 be able to start again.
 

To clean all left semaphores (arrays) preventing Apache from start properly, use below for one liner bash loop:
 

for i in `ipcs -s | awk '/www-data/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


Note that above for loop is specific to Debian on CentOS / Fedora / RHEL and other Linuxes the username with which stucked semaphores might stay will be apache or httpd

Depending on the user with which the Apache Webserver is running, run above loop like so:

For RPM based distros (CentOS / RHEL):

 

for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


For other distros such as Slackware or FreeBSD or any custom compiled Apache webserver:

for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


If there is also Shared Memory Segments you can remove them with ipcrm i.e.:

ipcrm -m 0x63637069


An alternative way to get rid of left uncleaned semaphores is with xargs:
 

ipcs -s | grep nobody | awk ‘ { print $2 } ‘ | xargs ipcrm


Even though this fixes the issue I understood my problems were due to exceeding semaphores, to check default number of set semaphores on Linux Kernel level as well as few Semaphore related values run below sysctl:

sysctl -a | egrep kernel.sem\|kernel.msgmni
kernel.msgmni = 15904

kernel.sem = 250        32000   32      128


As you can see the number of maximum semaphores is quite large so in my case the failure because of left semaphores was most likely due to some kind of Cracker / Automated bot scanner attack or someone trying malicious against the webserver or simply because of some kind of Apache bug or enormous high load the server faced.