Posts Tagged ‘apache’

Where are Apache log files on my server – Apache log file locations on Debian / Ubuntu / CentOS / Fedora and FreeBSD ?

Tuesday, November 7th, 2017

apache-where-are-httpd-access-log-files

Where are Apache log files on my server?

1. Finding Linux / FreeBSD operating system distribtion and version

Before finding location of Apache log files it is useful to check what is the remote / local Linux operating system version, hence

First thing to do when you login to your remote Linux server is to check what kind of GNU / Linux you're dealing with:

cat /etc/issue
cat /etc/issue.net


In most GNU / Linux distributions should give you enough information about the exact Linux distribution and version remote server is running.

You will get outputs like

# cat /etc/issue
SUSE LINUX Enterprise Server 10.2 Kernel \r (\m), \l

or

# cat /etc/issue
Debian GNU/Linux 8 \n \l

If remote Linux is Fedora look for fedora-release file:

cat /etc/fedora-release Fedora release 7 (Moonshine)

The proposed freedesktop.org standard with the introduction of systemd across all Linux distributions is

/etc/os-release

 

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"


Once we know what kind of Linux distribution we're dealing with, we can proceed with looking up for standard location of Apache config:

2. Apache config file location for Fedora / CentOS / RHEL and other RPM based distributions

RHEL / Red Hat / CentOS / Fedora Linux Apache access file location
 

/var/log/httpd/access_log


3. Apache config file location for Debian / Ubuntu and other deb based Linux distributions

Debian / Ubuntu Linux Apache access log file location

/var/log/apache2/access.log


4. Apache config file location for FreeBSD

FreeBSD Apache access log file location –

/var/log/httpd-access.log


5. Finding custom Apache access log locations
 

If for some reason the system administrator on the remote server changed default path for each of distributions, you can find custom configured log files through:

a) On Debian / Ubuntu / deb distros:

debian:~# grep CustomLog /etc/apache2/apache2.conf


b) On CentOS / RHEL / Fedora Linux RPM based ones:

[root@centos:  ~]# grep CustomLog /etc/httpd/conf/httpd.conf


c) On FreeBSD OS

 

freebsd# grep CustomLog /etc/httpd/conf/httpd.conf
 # a CustomLog directive (see below).
    #CustomLog "/var/log/httpd-access.log" common
    CustomLog "/var/log/httpd-access.log" combined

Fix “FAIL – Application at context path /application-name could not be started” deployment error in Tomcat manager

Thursday, October 1st, 2015

tomcat-manager-FAIL-Application-at-context-path-application-name-could-not-be-started-fix-solution-error

While deploying an environment called "Interim" which is pretty much like a testing Java application deployed from a Java EAR (Enterprise Archive) file from within a Tomcat Manager GUI web interface after stopping the application and trying to start it, the developers come across the error:

 

FAIL – Application at context path /application-name could not be started


The error puzzled me for a while until I checked the catalina.out I've seen a number of thrown Java Eceptions errors like:

Okt 01, 2015 10:48:46 AM org.springframework.web.context.ContextLoader initWebApplicationContext

Schwerwiegend: Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.sun.xml.ws.transport.http.servlet.SpringBinding#2' defined in ServletContex

t resource [/WEB-INF/pp-server-beans.xml]: Cannot create inner bean ‘(inner bean)’ of type [org.jvnet.jax_ws_commons.spring.SpringService] while setting bean property

'service'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#33': FactoryBean threw exception on

 object creation; nested exception is java.lang.OutOfMemoryError: PermGen space

I've googled a bit about the error:

"FAIL – Application at context path /application-name could not be started"

and come across this Stackoverflow thread and followed suggested solution to fix web.xml tag closing error but it seems there was no such error in my case, I then also tried solution suggested by this thread (e.g. adding in logging.properties) file:
 

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

unfortunately this helped neither to solve the error when it is tried to be started from tomcat manager.

After asking for help a colleague Kostadin, he pointed me to take a closer look in the error (which is a clear indication) that the reserved space is not enough (see below err):
 

java.lang.OutOfMemoryError: PermGen space

And he pointed me then to Solution (which is to modify the present tomcat setenv.sh) settings which looked like this:

# Heap size settings

export JAVA_OPTS="-Xms2048M -Xmx2048M"

 

# SSCO test page parameter

export JAVA_OPTS="$JAVA_OPTS -DTS1A_TESTSEITE_CONFIG_PATH=test-myapplication.com"

# Default garbage collector settings

export JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=128M"

 

# Aggressive garbage collector settings.

# Do not use! For testing purposes only!

#export JAVA_OPTS="$JAVA_OPTS -Xss128k -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts -XX:MaxPermSize=128M"

 

####### DO NOT CHANGE BELOW HERE #######

# Disable X11 usage

unset DISPLAY

export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

 

# Garbage collection log settings

export JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/web/tomcat/current/logs/gc.log -XX:-TraceClassUnloading"

 

# Enable JMX console

export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"

 

 

 

 

 

 

The solution is to to add following two options to export JAVA_OPTS (above options):

-XX:PermSize=512m -XX:MaxPermSize=512m


After modifications, my new line within setenv.sh looked like so:

 

JAVA_OPTS="-Xms2048M -Xmx2048M -XX:PermSize=512m -XX:MaxPermSize=512m"


Finally to make new JAVA_OPTS settings, I've restarted Tomcat with:

 

cd /web/tomcat/bin/
./shutdown.sh
sleep 5; ./startup.sh


And hooray it works fine thanks God! 🙂

Remove \r (Carriage Return) from string with standard bash shell / sed / tr / vim or awk – Replace \r hidden messy characters from files

Tuesday, February 10th, 2015

remove_r_carriage_return_from_string_with-standard-bash_shell_sed_tr_or_awk_replace_annoying_hidden_messy_characters_from_files

I've been recently writting this Apache webserver / Tomcat / JBoss / Java decomissioning bash script. Part of the script includes extraction from httpd.conf of DocumentRoot variable configured for Apache host.
I was using following one liner to grep and store DocumentRoot set directory into new variable:

documentroot=$(grep -i documentroot /usr/local/apache/conf/httpd.conf | awk '{ print $2 }' |sed -e 's#"##g');

Above line greps for documentroot prints 2nd column of the matchi (which is the Apache server set docroot and then removes any " chars).

However I faced the issue that parsed string contained in $documentroot variable there was mysteriously containing r – return carriage – this is usually Carriage Return (CR) sent by Mac OS and Apple computers. For those who don't know the End of Line of files in UNIX / Linux OS-es is LF – often abreviated as n – often translated as return new line), while Windows PCs use for EOF CR + LF – known as the infamous  rn. I was running the script from the server which is running SuSE SLES 11 Linux, meaning the CR + LF end of file is standardly used, however it seem someone has editted the httpd.conf earlier with a text editor from Mac OS X (Terminal). Thus I needed a way to remove the r from CR character out of the variable, because otherwise I couldn't use it to properly exec tar to archive the documentroot set directory, cause the documentroot directory was showing unexistent.

Opening the httpd.conf in standard editor didn't show the r at the end of
"directory", e.g. I could see in the file when opened with vim

DocumentRoot "/usr/local/apache/htdocs/site/www"

However obviously the r character was there to visualize it I had to use cat command -v option (–show-nonprinting):

cat -v /usr/local/apache/conf/httpd.conf

DocumentRoot "/usr/local/apache/htdocs/site/wwwr"


1. Remove the r CR with bash

To solve that with bash, I had to use another quick bash parsing that scans through $directory and removes r, here is how:

documentroot=${documentroot%$'r'}

It is also possible to use same example to remove "broken" Windows rn Carriage Returns after file is migrated from Windows to Liunx /  FreeBSD host:

documentroot=${documentroot%$'rn'}

 

2. Remove r Carriage Return character with sed

Other way to do remove (del) Windows / Mac OS Carriage Returns in case if Migrating to UNIX is with sed (stream editor).

sed -i s/r// filename >> filename_out.txt


3. Remove r CR character with tr

There is a third way also to do it with (tr) – translate or delete characters old shool *nix command:

tr -d 'r' < file_with_carriagereturns > file_without_carriage_returns

 

4. Remove r CRs with awk (pattern scanning and processing language)

 awk 'sub("$", "r")' inputf_with_crs.txt > outputf_without_crs.txt


5. Delete r CR with VIM editor

:%s/r//g


6. Converting  file DOS / UNIX OSes with dos2unix and unix2dos command line tools

For sysadmins who don't want to bother with writting code to convert CR when moving files between Windows and UNIX hosts there are dos2unix and unix2dos installable commands.

All done Cheers ! 🙂

Apache SSLCertificateChainFile adding SSL with Certificate Chain / What is Certificate Chain

Friday, January 31st, 2014

configure-apache-ssl-certificate-chain-ssl-certificate-keychain-each-signing-each-other

If you work in a big company with large network infrastructure who has to deal with SSL Certificates you will sooner or later will have to learn about existence of SSL Certificate Chains.
Its worthy thus to know what is SSL Certificate Chains and how such a chain is configured in Apache?

Personal SSL certificates (certificates issued to an individual or a company) can be used by clients to uniquely identify themselves when they are involved in starting an SSL connection.
SSL Certificate file contains X.509 certificate, which, in turn, contains a public key used for encryption.
Each personal certificate has zero or more certificate chains of certification authority certificates that extend back to the root certification authority.
 

Certificate R (Root Certification Authority)
    |
    | represents issuer of
    V
Certificate I1 (Intermediate Certification Authority)
    |
    | represents issuer of
    V
Certificate I2 (A subsidiary Intermediate Certification Authority)
    |
    | represents issuer of
    V
Certificate I3 (A further subsidiary Intermediate Certification Authority)
    |
    | represents issuer of
    V
Certificate P (A personal certificate that is used to identify its owner 
               on an SSL handshake)

Certificate chains are used to verify the authenticity of each certificate in that chain, including the personal certificate. Each certificate in the chain is validated using its 'parent' certificate, which in turn is validated using the next certificate up the chain, and so on, from the personal certificate up to the root certification authority certificate.

Now after explaining thoroughfully what is SSL Certificate Chain, here is how to configure a SSL Certificate in Apache Webserver.

Open apache2.conf or httpd.conf (depending on GNU / Linux distribution) and add to it;

  SSLEngine On
   SSLCertificateFile conf/cert/webserver-host.crt
   SSLCertificateKeyFile conf/cert/webserver-host.key
   SSLCertificateChainFile conf/cert/internet-v4.crt
   # SSLCertificateChainFile conf/cert/intranet-v3.crt
   SSLOptions +StdEnvVars +OptRenegotiate +ExportCertData

SSLCertificateChainFile conf/cert/chain-cert.crt
loads a chain of separate Personal SSL certificates each signing each other on different levels, chain is leading to top ROOT CA (Certificate Authority).

Fix Apache [error] [client xx.xxx.xxx.xx] PHP Warning: Unknown: Input variables exceeded 1000. To increase the limit change max_input_vars in php.ini. in Unknown on line 0

Wednesday, August 14th, 2013

I have a busy Linux server with 24 cores each running on ..4 Ghz. Server is configured to server  Apache and MySQL queries and a dozen of high traffic websites are hosted on it. Until just recently server worked fine and since about few days I started getting SMS notifications that server is inaccessible few times a day. To check what's wrong I checked in /var/log/apache2/error.log  and there I found  following error:

[error] [client 95.107.233.2] PHP Warning:  Unknown: Input variables exceeded 1000.

To increase the limit change max_input_vars in php.ini. in Unknown on line 0 referer: http://www.site-domain-name.com/predict/2013-08-10
 

Before I check Apache error.log, I had a guess that ServerLimit of 256 (spawned servers max) is reached so solution would be raise of ServerLimit to more than MaxClients setting defined in /etc/apache2/apache2.conf. After checking /var/log/apache2/error.log I've realized problem is because the many websites hosted on server exceed maximum defined variables possible to assign by libphp in php.ini. maximum possible defined variables before PHP stops servering is set through max_input_vars variable

As I'm running a Debian Squeeze Linux server, here is what is set as default for max_input_vars in /etc/php5/apache2/php.ini:

; How many GET/POST/COOKIE input variables may be accepted
; max_input_vars = 1000

So to fix it in php.ini just raised a bit setting to 1500, i.e.:

max_input_vars = 1500

Though I hit the error on Debian I assume same error occurs on Redhat RPM based (Fedora, CentOS, RHEL Linux) servers.
Hence I assume

max_input_vars = 1500

or higher should fix on those servers too. Looking forward to hear if same error is hit on RedHats.

Enjoy 🙂
 

Linux PHP Disable chmod() and chown() functions for better Apache server security

Monday, July 15th, 2013

php_tighten_security_by_enabling_safe_mode-php-ini-function-prevent-crackers-break-in-your-server
I have to administer few inherited Linux servers with Ubuntu and Debian Linux. The servers hosts mainly websites with regularly un-updated Joomlas and some custom developed websites which were developed pretty unsecure. To mitigate hacked websites I already disabled some of most insecure functions like system(); eval etc. – I followed literally my previous tutorial PHP Webhosting security disable exec();, system();, open(); and eval();
Still in logs I see shits like:
 

[error] [client 66.249.72.100] PHP Warning:  mkdir(): No such file or directory in /var/www/site/plugins/system/jfdatabase/intercept.jdatabasemysql.php on line 161

Hence to prevent PHP mkdir(); and chown(); functiosn being active, I had to turn on in /etc/php5/apache2/php.ini – safe_mode . For some reason whoever configured Apache leave it off.

safe_mode = on

Hopefully by disabling this functions will keep cracker bot scripts to not create some weird directory structures on HDD or use it as mean to DoS overflow servers filesystem.

Hope this help others stabilize their servers too. Enjoy ! 🙂

Debian Linux: Installing and monitoring servers with Icanga (Nagios fork soft)

Monday, June 3rd, 2013

icinga-monitoring-processes-and-servers-linux-logo

There is plenty of software for monitoring how server performs and whether servers are correctly up and running. There is probably no Debian Linux admin who didn't already worked or at least tried Nagios and Mointor to monitor and notify whether server is unreachable or how server services operate. Nagios and Munin are play well together to prevent possible upcoming problems with Web / Db / E-mail services or get notify whether they are completely inaccessible. One similar "next-generation" and less known software is Icanga.
The reason, why to use Icinga  instead of Nagios is  more features a list of what does Icinga supports more than Nagios is on its site here
I recently heard of it and decided to try it myself. To try Icanga I followed Icanga's install tutorial on Wiki.Icanga.Org here
In Debian Wheezy, Icinga is already part of official repositories so installing it like in Squeeze and Lenny does not require use of external Debian BackPorts repositories.

1. Install Icinga pre-requirement packages

debian:# apt-get --yes install php5 php5-cli php-pear php5-xmlrpc php5-xsl php5-gd php5-ldap php5-mysql

2. Install Icanga-web package

debian:~# apt-get --yes install icinga-web

Here you will be prompted a number of times to answer few dialog questions important for security, as well as fill in MySQL server root user / password as well as SQL password that will icinga_web mySQL user use.

icinga-choosing-database-type

configuring-icinga-web-debian-linux-configuring-database-shot

debian-config-screenshot-configuring-icinga-idoutils

icinga-password-confirmation-debian-linux
….

Setting up icinga-idoutils (1.7.1-6) …
dbconfig-common: writing config to /etc/dbconfig-common/icinga-idoutils.conf
granting access to database icinga for icinga-idoutils@localhost: success.
verifying access for icinga-idoutils@localhost: success.
creating database icinga: success.
verifying database icinga exists: success.
populating database via sql…  done.
dbconfig-common: flushing administrative password
Setting up icinga-web (1.7.1+dfsg2-6) …
dbconfig-common: writing config to /etc/dbconfig-common/icinga-web.conf

Creating config file /etc/dbconfig-common/icinga-web.conf with new version
granting access to database icinga_web for icinga_web@localhost: success.
verifying access for icinga_web@localhost: success.
creating database icinga_web: success.
verifying database icinga_web exists: success.
populating database via sql…  done.
dbconfig-common: flushing administrative password

Creating config file /etc/icinga-web/conf.d/database-web.xml with new version
database config successful: /etc/icinga-web/conf.d/database-web.xml

Creating config file /etc/icinga-web/conf.d/database-ido.xml with new version
database config successful: /etc/icinga-web/conf.d/database-ido.xml
enabling config for webserver apache2…
Enabling module rewrite.
To activate the new configuration, you need to run:
  service apache2 restart
`/etc/apache2/conf.d/icinga-web.conf' -> `../../icinga-web/apache2.conf'
[ ok ] Reloading web server config: apache2 not running.
root password updates successfully!
Basedir: /usr Cachedir: /var/cache/icinga-web
Cache already purged!

3. Enable Apache mod_rewrite
 

 

debian:~# a2enmod rewrite
debian:~# /etc/init.d/apache2 restart


4. Icinga documentation files

Some key hints on Enabling some more nice Icinga features are mentioned in Icinga README files, check out, all docs files included with Icinga separate packs are into:
 

debian:~# ls -ld *icinga*/
drwxr-xr-x 3 root root 4096 Jun  3 10:48 icinga-common/
drwxr-xr-x 3 root root 4096 Jun  3 10:48 icinga-core/
drwxr-xr-x 3 root root 4096 Jun  3 10:48 icinga-idoutils/
drwxr-xr-x 2 root root 4096 Jun  3 10:48 icinga-web/

debian:~# less /usr/share/doc/icinga-web/README.Debian debian:~# less /usr/share/doc/icinga-idoutils/README.Debian

5. Configuring Icinga

Icinga configurations are separated in two directories:

debian:~# ls -ld *icinga*

drwxr-xr-x 4 root root 4096 Jun  3 10:50 icinga
drwxr-xr-x 3 root root 4096 Jun  3 11:07 icinga-web

>

etc/icinga/ – (contains configurations files for on exact icinga backend server behavior)

 

/etc/icinga-web – (contains all kind of Icinga Apache configurations)
Main configuration worthy to look in after install is /etc/icinga/icinga.cfg.

6. Accessing newly installed Icinga via web

To access just installed Icinga, open in browser URL – htp://localhost/icinga-web

icinga web login screen in browser debian gnu linux

logged in inside Icinga / Icinga web view and control frontend

 

7. Monitoring host services with Icinga (NRPE)

As fork of Nagios. Icinga has similar modular architecture and uses number of external plugins to Monitor external host services list of existing plugins is on Icinga's wiki here.
Just like Nagios Icinga supports NRPE protocol (Nagios Remote Plugin Executor). To setup NRPE, nrpe plugin from nagios is used (nagios-nrpe-server). 

To install NRPE on any of the nodes to be tracked;
debian: ~# apt-get install –yes nagios-nrpe-server

 Then to configure NRPE edit /etc/nagios/nrpe_local.cfg


 

Once NRPE is supported in Icinga, you can install on Windows or Linux hosts NRPE clients like in Nagios to report on server processes state and easily monitor if server disk space / load or service is in critical state.

screen -d Fix “Must run suid root for multiuser support.” su user detach error

Thursday, March 28th, 2013

I had to run a shell script to run automatically in detached screen during Linux system boot up via /etc/rc.local. This is needed because the server uses the tiny shell script to fetch data from remote host database and fill information into local MySQL server.

My idea was to su from root to www-data (Apache) user – the script has requirements to run with Apache user, then it has to run detached using GNU screen (multi terminal emulator. The tiny one line script I imagined would do the trick is like so:

# tty=$(tty); su www-data -c 'cd /home/user/www/enetpulse; screen -d /home/user/www/enetpulse/while_true.sh'; chmod 0720 $tty

I run this as root user to test whether it will work or not before I put it in /etc/rc.local but for my surprise got an error:
 

Must run suid root for multiuser support.

After a quick investigation on what is causing it I came across the solution which is to include screen arguments (-m -S shared). The working variant that gets around the error – i.e. successfully changes user privileges to Debian Apache user (www-data) and then detach with screen is:

# tty=$(tty); chmod a+rw $tty; su www-data -c 'cd /home/user/www/enetpulse; screen -d -m -S shared /home/user/www/enetpulse/while_true.sh'; chmod 0720 $tty;

That's all now script works out as planned on next server reboot

How to set repository to install binary packages on amd64 FreeBSD 9.1

Friday, January 11th, 2013

Though, it is always good idea to build from source for better performance of Apache + MySQL + PHP, its not worthy the time on installing minor things like; trafshow, tcpdump or deco (MC – midnight commander like native freebsd BSD program).

If you're on a 64 bit version of FreeBSD ( amd64) 9.1 and you try to install a binary package with;

freebsd# pkg_add -vr vim

Ending up with an error;

Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9.1-release/Latest/vim.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9.1-release/Latest/vim.tbz' by URL
pkg_add: 1 package addition(s) failed

The error is caused by lack of special packages-9.1-release directory existing on FreeBSD.org servers. I've realized this after doing a quick manual check opening ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64. The existing URL containing working fbsd 9.1 binaries is:

ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/
h

You will have to set a repository for FreeBSD 9.1 amd64 packages manually with cmd:
freebsd# echo $SHELL
/bin/csh
freebsd# setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/

If you're on bash shell use export instead:

freebsd# export PACKAGESITE="ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"

To make ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/ as a permanent binary repository:

echo 'setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/' >> /root/.cshrc

or

echo 'export PACKAGESITE="ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"' >> /root/.bashrc

Now, pkg_add as much as you like 😉

Fixing 127.0.0.1 – – “OPTIONS * HTTP/1.0” 200 136 “-” “Apache (internal dummy connection)” / ::1 – – [-.. :- .. +0200] “OPTIONS * HTTP/1.0” 200 Apache access.log junk records

Saturday, December 1st, 2012

If you're on Debian Linux and you played with mpm_prefork_module MinSpareServers and MaxSpareServers directives, it is very likely your access.log apache log ends up with a plenty of junk messages like:

127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"

It was quite unexplainable to me what is causing all this errors. I've seen plenty of posts on the Internet discussing on that but most are somehow outdated and suggested solutions to the weird logged  internal dummy connection messages did not work well for me.

I would not care so much about the message, only if it was not creating a lot of bulk records in my logs which when later are compressed just take up useless disk space and besides that it makes following the Apache log with:

# tail -f  /var/log/apache2/access.log

hardly readable.

  • One of the many solutions and posts suggested a solution with mod_rewrite rules. It claims adding the rules to .htaccess or to apache config files (vhost confs whether multiple vhosts domains):

RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
RewriteRule .* – [F,L]

The full article you read the whole here.
I've tested this rules, and thought I might be doing something wrong this proved unworking for me. Besides that even if it worked I would not imply such fix, as it will be creating a useless extra load on each incoming Apache connection.

 

As a second solution as I found on stackoverflow's website is to add in apache / vhost configs:

<Limit OPTIONS Order allow,deny Deny from all </Limit> I tested this as well but it does not work either. I've seen a bunch of other posts and none seemed to be working, until I finally came across Linux Guru's blog which was discussing a similar issue suggesting a fix. The post is discussing on Apache access.log being filled with messages like: ::1 - - [13/Mar/2008:09:05:13 +0200] "OPTIONS * HTTP/1.0" 200 Which are almost the same except, the 127.0.0.1 is the IPv6's equivalent ::1. The blog provided solution is to use: SetEnvIf Remote_Addr "::1" dontlog CustomLog /var/log/apache2/access.log combined env=!dontlog What this makes is to completely clear up all occurances of ::1 in /var/log/apache2/access.log. Once it uses Apache Internal directive SetEnvIf Remote_Addr "::1" dontlog to "bind" ::1 to dontlog variable and then after the usual Log location definition – e.g. – CustomLog /var/log/apache2/access.log combined it instructs the environment not to log dontlog variable matches, i.e. env=!dontlog

Following he same logic to get rid of the so annoying:

127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)"

I used as a solution adding:

SetEnvIf Remote_Addr "127.0.0.1" dontlog
CustomLog /var/log/apache2/access.log combined env=!dontlog

to /etc/apache2/sites-available/000-default (the default virtualhost), with the CustomLog directive, for more domains and more CustomLog VirtualHost definitions it might be necessary to add it to all Vhosts too.

This solution to Request of the Server to itself is also found on Apache's wiki  check what httpd wiki here.

As I've read further it appeared the same Internal Dummy Connection error is experienced on CentOS Linux too and the SetEnvIf method works there too well you can read post here.

Another possible solution though this didn't work for me is to just play with the settings of MinSpareServers and MaxSpareServers in apache2.conf (or httpd.conf on RedHats and BSD).

There is plenty of things written on the problem and it is really confusing to read about it, as most of the people writing about it were looking for the quick fix and thus just dropped few lines on what worked for them without much details on exact OS en Apache version.

The reason why:
127.0.0.1 – – [25/Nov/2012:06:27:21 +0200] "OPTIONS * HTTP/1.0" 200 136 "-" "Apache (internal dummy connection)" appear in log is due to the fact in Apache 2.x series Apache developers change the the Parent Apache controlling process to send periodic requests to its waiting idling childs, just to make the childs are still alive, this is done somehow in the very inefficient method IMHO by sending those dummy connection requests.

Maybe better and more thoroughful explanation on What is the Dummy Internal Connection and what causes it is on another Bulgarian Fellow Valery Dachev you can read his explan.

On a couple of occasions, I've experienced a very high server loads like load avarage of 180etc. , I have some suspicion that this super high loads are caused somehow by the Internal Dummy Connection thing too, though I'm not sure if my assumptions are correct. It could be I have messed up something with MaxSpareServers / MinSpareServers too, or just the hardware on the host is unable to process a sudden traffic peaks. I've red online other people who complain of similar overloads and complaininng about the Internal Dummy Connection too. But as long as my little research go, I couldn't find noone knowing anything on that. If some of the readers of this post has an idea on that please drop a comment !

Well that's it hope my little blog post sheds some more light on the topic, and lets hope in future Apache versions developers will come with less resource hungry method to do internal dummy checks for exmpl. by sending a SIGUSR signal.