If out of a sudden your Apache webserver crashes and is refusing to start up by manually trying to restart it through its init script on Debian Linux servers – /etc/init.d/apache2 and RPM based ones: /etc/init.d/httpd
Checking in php_error.log there was no shown errors related to loading PHP modules, however apache's error.log show following errors:
[Wed Apr 08 14:20:14 2015] [error] [client 180.76.5.149] client denied by server configuration: /var/www/sploits/info/trojans_info/tr_data/y3190.html
[Wed Apr 08 14:20:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:20:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.15974) (5)
[Wed Apr 08 14:25:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:25:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16790) (5)
[Wed Apr 08 14:27:03 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:03 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16826) (5)
[Wed Apr 08 14:27:53 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:53 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16852) (5)
[Wed Apr 08 14:30:48 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:30:48 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17710) (5)
[Wed Apr 08 14:31:21 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:31:21 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17727) (5)
[Wed Apr 08 14:32:40 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:32:40 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17780) (5)
[Wed Apr 08 14:38:32 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
As you can read the most likely reason behind above errors preventing for apache to start is /var/run/apache2.pid unable to be properly written due to lack of disk space or due to disk quota set for users including for userID with which Apache is running.
First thing I did is of course to see how much free space is on the server:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-rootvol 4.0G 1.7G 2.2G 44% /
udev 7.8G 204K 7.8G 1% /dev
tmpfs 24G 0 24G 0% /dev/shm
/dev/sda1 486M 40M 422M 9% /boot
/dev/mapper/vg00-lv_crashdump 1008M 34M 924M 4% /crashdump
/dev/mapper/vg00-homevol 496M 26M 445M 6% /home
/dev/mapper/vg00-lv_opt 12G 1.4G 9.9G 13% /opt
/dev/mapper/vg00-tmpvol 2.0G 68M 1.9G 4% /tmp
/dev/mapper/vg00-varvol 7.9G 609M 6.9G 8% /var
/dev/mapper/vg00-crashvol 1.9G 35M 1.8G 2% /var/crash
/dev/mapper/vg00-auditvol 124M 5.6M 113M 5% /var/log/audit
/dev/mapper/vg00-webdienste 60G 12G 48G 19% /webservice
As visible from above df command output , there is enough disk on HDD, so this is definitely not the issue:
Then I Checked whether there is Quota enabled on the Linux server with repquota command shows there are no quotas enabled:
# repquota / var/
repquota: Mountpoint (or device) / not found or has no quota enabled.
repquota: Mountpoint (or device) /var not found or has no quota enabled.
repquota: Not all specified mountpoints are using quota.
So obviously the only few left possible reason for Apache failing to start after invoked via init script is either due to left tainted semaphores or due to some server hardware RAM problem / or a dying hard disk with bad blocks.
So what are Semaphores? Generally speaking Semaphores are apparatus for conveying information by means of visual signals between applications (something like sockets).They're used for communicating between the active processes of a certain application. In the case of Apache, they’re used to communicate between the parent and child processes, hence if Apache can’t properly write and coordinate these things down, then it can’t communicate properly with all of the processes it starts and hence the Main HTTPD process can't spawn probably its childs preventing Webserver to enter "started mode" and write its PID file.
To check general information about system semaphore arrays there is the ipcs -s command, however my experience is that ipcs -a is more useful (because it lists generally all kind of semaphores) including Semaphore Shared Memory Signals which are the most likely to cause you the problem.
ipcs -a
—— Shared Memory Segments ——–
key shmid owner perms bytes nattch status—— Semaphore Arrays ——–
key semid owner perms nsems
0x00000000 22970368 www-data 600 1—— Message Queues ——–
key msqid owner perms used-bytes messages
As you see in my case there is a Semaphore Arrays which had to be cleaned to make Apache2 be able to start again.
To clean all left semaphores (arrays) preventing Apache from start properly, use below for one liner bash loop:
for i in `ipcs -s | awk '/www-data/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069
Note that above for loop is specific to Debian on CentOS / Fedora / RHEL and other Linuxes the username with which stucked semaphores might stay will be apache or httpd
Depending on the user with which the Apache Webserver is running, run above loop like so:
For RPM based distros (CentOS / RHEL):
for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069
For other distros such as Slackware or FreeBSD or any custom compiled Apache webserver:
for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069
If there is also Shared Memory Segments you can remove them with ipcrm i.e.:
ipcrm -m 0x63637069
An alternative way to get rid of left uncleaned semaphores is with xargs:
ipcs -s | grep nobody | awk ‘ { print $2 } ‘ | xargs ipcrm
Even though this fixes the issue I understood my problems were due to exceeding semaphores, to check default number of set semaphores on Linux Kernel level as well as few Semaphore related values run below sysctl:
sysctl -a | egrep kernel.sem\|kernel.msgmni
kernel.msgmni = 15904kernel.sem = 250 32000 32 128
As you can see the number of maximum semaphores is quite large so in my case the failure because of left semaphores was most likely due to some kind of Cracker / Automated bot scanner attack or someone trying malicious against the webserver or simply because of some kind of Apache bug or enormous high load the server faced.
How to enable Ctrl+Alt+Backspace in Ubuntu 11.10 (Oneiric Ocelot) Linux
Monday, January 16th, 2012My sister, experience some programs running with wine (Windows Emulator) to crash on her Ubuntu 11.10.
As she is quite new with Linux, she has no idea about the existence of CTRL ALT BACKSPACE key combination to restart a hanged GNOME, KDE by directly killing the Xorg server.
I felt obliged to explain her it is better to use CTRL ALT BACKSPACE X kill switch instead of restarting the whole Linux kernel (which basiclly is working) and that it is just the display keeping blacnk.
Pressing the would kill Xorg and therefore all applicatins previously running on top of it will die. In Ubuntu Xorg is configured to run via gdm, so once killed it will automatically reload the GDM (Gnome Display Manager).
I was about to explain her that its better she use CTRL+ALT+BACKSPACE instead of restarting the whole system but suddenly I realized this is not working.
In UBUNTU 11.10 and I guess in all UBUNTU's after 9.04 CTRL ALT BACKSPACE is substituted with the key switch combination ALT PRINTSCREEN K, I've explained her about that.
This change is actually a change implied by most Linux distributions nowdas and is some kind of change in Xorg newer versions…
To enable back the CTRL + ALT + BACKSPACE , I've issued cmd:
stanimira@ubuntu~:$ echo' setxkbmap -option terminate:ctrl_alt_bksp' >> ~/.xinitrc
An alternative way to set setxkbmap -option terminate:ctrl_alt_bksp to run on Ubuntu user login is by setting it as a startup application using;
stanimira@ubuntu:~$ gnome-session-properties
Press the Add button and type in the box to appear;
Name: setxkbmap
Command: setxkbmap -option terminate:ctrl_alt_bksp
Comment: setxkbmap
Reverting the Xserver kill switch back to the classical Ctrl+Alt+Backspace should also be running fine on older Ubuntu Linuces – 11.04, 10.10, 10.04 etc.
Tags: add button, Alt, Auto, backspace, Button, change, Comment, Ctrl, Display, Draft, existence, GDM, Gnome, kde, key switch, kill, kill switch, Linux, linux kernel, login, most linux distributions, ocelot, option, Press, printscreen, Reverting, session properties, setxkbmapCommand, startup, startup application, switch combination, type, Ubuntu, wine, wine windows, xinitrcAn, Xorg, xserver
Posted in Linux, Linux and FreeBSD Desktop | No Comments »