If out of a sudden your Apache webserver crashes and is refusing to start up by manually trying to restart it through its init script on Debian Linux servers – /etc/init.d/apache2 and RPM based ones: /etc/init.d/httpd
Checking in php_error.log there was no shown errors related to loading PHP modules, however apache's error.log show following errors:
[Wed Apr 08 14:20:14 2015] [error] [client 180.76.5.149] client denied by server configuration: /var/www/sploits/info/trojans_info/tr_data/y3190.html
[Wed Apr 08 14:20:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:20:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.15974) (5)
[Wed Apr 08 14:25:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:25:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16790) (5)
[Wed Apr 08 14:27:03 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:03 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16826) (5)
[Wed Apr 08 14:27:53 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:53 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16852) (5)
[Wed Apr 08 14:30:48 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:30:48 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17710) (5)
[Wed Apr 08 14:31:21 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:31:21 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17727) (5)
[Wed Apr 08 14:32:40 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:32:40 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17780) (5)
[Wed Apr 08 14:38:32 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
As you can read the most likely reason behind above errors preventing for apache to start is /var/run/apache2.pid unable to be properly written due to lack of disk space or due to disk quota set for users including for userID with which Apache is running.
First thing I did is of course to see how much free space is on the server:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-rootvol 4.0G 1.7G 2.2G 44% /
udev 7.8G 204K 7.8G 1% /dev
tmpfs 24G 0 24G 0% /dev/shm
/dev/sda1 486M 40M 422M 9% /boot
/dev/mapper/vg00-lv_crashdump 1008M 34M 924M 4% /crashdump
/dev/mapper/vg00-homevol 496M 26M 445M 6% /home
/dev/mapper/vg00-lv_opt 12G 1.4G 9.9G 13% /opt
/dev/mapper/vg00-tmpvol 2.0G 68M 1.9G 4% /tmp
/dev/mapper/vg00-varvol 7.9G 609M 6.9G 8% /var
/dev/mapper/vg00-crashvol 1.9G 35M 1.8G 2% /var/crash
/dev/mapper/vg00-auditvol 124M 5.6M 113M 5% /var/log/audit
/dev/mapper/vg00-webdienste 60G 12G 48G 19% /webservice
As visible from above df command output , there is enough disk on HDD, so this is definitely not the issue:
Then I Checked whether there is Quota enabled on the Linux server with repquota command shows there are no quotas enabled:
# repquota / var/
repquota: Mountpoint (or device) / not found or has no quota enabled.
repquota: Mountpoint (or device) /var not found or has no quota enabled.
repquota: Not all specified mountpoints are using quota.
So obviously the only few left possible reason for Apache failing to start after invoked via init script is either due to left tainted semaphores or due to some server hardware RAM problem / or a dying hard disk with bad blocks.
So what are Semaphores? Generally speaking Semaphores are apparatus for conveying information by means of visual signals between applications (something like sockets).They're used for communicating between the active processes of a certain application. In the case of Apache, they’re used to communicate between the parent and child processes, hence if Apache can’t properly write and coordinate these things down, then it can’t communicate properly with all of the processes it starts and hence the Main HTTPD process can't spawn probably its childs preventing Webserver to enter "started mode" and write its PID file.
To check general information about system semaphore arrays there is the ipcs -s command, however my experience is that ipcs -a is more useful (because it lists generally all kind of semaphores) including Semaphore Shared Memory Signals which are the most likely to cause you the problem.
ipcs -a
—— Shared Memory Segments ——–
key shmid owner perms bytes nattch status—— Semaphore Arrays ——–
key semid owner perms nsems
0x00000000 22970368 www-data 600 1—— Message Queues ——–
key msqid owner perms used-bytes messages
As you see in my case there is a Semaphore Arrays which had to be cleaned to make Apache2 be able to start again.
To clean all left semaphores (arrays) preventing Apache from start properly, use below for one liner bash loop:
for i in `ipcs -s | awk '/www-data/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069
Note that above for loop is specific to Debian on CentOS / Fedora / RHEL and other Linuxes the username with which stucked semaphores might stay will be apache or httpd
Depending on the user with which the Apache Webserver is running, run above loop like so:
For RPM based distros (CentOS / RHEL):
for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069
For other distros such as Slackware or FreeBSD or any custom compiled Apache webserver:
for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069
If there is also Shared Memory Segments you can remove them with ipcrm i.e.:
ipcrm -m 0x63637069
An alternative way to get rid of left uncleaned semaphores is with xargs:
ipcs -s | grep nobody | awk ‘ { print $2 } ‘ | xargs ipcrm
Even though this fixes the issue I understood my problems were due to exceeding semaphores, to check default number of set semaphores on Linux Kernel level as well as few Semaphore related values run below sysctl:
sysctl -a | egrep kernel.sem\|kernel.msgmni
kernel.msgmni = 15904kernel.sem = 250 32000 32 128
As you can see the number of maximum semaphores is quite large so in my case the failure because of left semaphores was most likely due to some kind of Cracker / Automated bot scanner attack or someone trying malicious against the webserver or simply because of some kind of Apache bug or enormous high load the server faced.
