Posts Tagged ‘mechanism’

Recommended logrorate practices on heavy loaded (busy) Apache Linux servers

Wednesday, March 7th, 2012

Apache logrotate Debian good configuration for heavy loaded servers

If you are sys admin of Apache Webserver running on Debian Linux relying on logrorate to rorate logs, you might want to change the default way logroration is done.

Little changes in the way Apache log files are served on busy servers can have positive outcomes on the overall way the server CPU units burden. A good logrotation strategy can also prevent your server from occasional extra overheads or downtimes.

The way Debian GNU / Linux process logs is well planned for small servers, however the default logroration Apache routine doesn't fit well for servers which process millions of client requests each day.

I happen to administrate, few servers which are constantly under a heavy load and have occasionally overload troubles because of Debian's logrorate default mechanism.

To cope with the situation I have made few modifications to /etc/logrorate.d/apache2 and decided to share it here hoping, this might help you too.

1. Rotate Apache acccess.log log file daily instead of weekly

On Debian Apache's logrorate script is in /etc/logrotate.d/apache2

The default file content will be like so like so:

debian:~# cat /etc/logrotate.d/apache2
/var/log/apache2/*.log {
weekly
missingok
rotate 52
size 1G
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
/etc/init.d/apache2 reload > /dev/null
fi
endscript
}

To change the rotation from weekly to daily change:

weekly

to

#weekly

2. Disable access.log log file gzip compression

By default apache2 logrotate script is tuned ot make compression of rotated file (exmpl: copy access.log to access.log.1 and gzip it, copy access.log to access.log.2 and gzip it etc.). On servers where logs are many gigabytes, once logrotate initiates its scheduled work it will have to compress an enormous log record of apache requests. On very busy Apache servers from my experience, just for a day the log could grow up to approximately 8 / 10 Gigabytes.
I'm sure there are more busy servers out there, which log files are growing to over 100GB for just a single day.
Gzipping a 100GB file piece takes an enormous load on the CPU, as well as often takes long time. When this logrotation gzipping occurs at a moment where the servers CPU cores are already heavy loaded from Apache serving HTTP requests, Apache server becomes inaccessible to most of the clients.
Then for end clients various oddities are experienced, for example Apache dropped connection errors, webserver returning empty pages, or simply inability to respond to the client browser.
Sometimes as a result of the overload, even secure shell connection to SSHD to the server is impossible …

To prevent your server from this roration overloads remove logrorate's default access.log gzipping by commenting:

compress

to

#comment

3. Change maximum log roration by logrorate to be up to 30

By default logrorate is configured to create and keep up to 52 rotated and gzipped access.log files, changing this to a lower number is a good practice (in my view), in cases where log files grow daily to 10 or more GBs. Doing so will save a lot of disk space and reduce the chance the hard disk gets filled in because of the multiple rorated ungzipped enormous access.log files.

To tune the default keep max rorated logs to 30, change:

rotate 52

torotate 30

The way logrorate's apache log processing on RHEL / CentOS Linux is working better on high load servers, by default on CentOS logrorate is not configured to do log gzipping at all.

Here is the default /etc/logrorate.d/httpd script for
CentOS release 5.6 (Final)

[hipo@centos httpd]$ cat /etc/logrotate.d/httpd /var/log/httpd/*log {
missingok
notifempty
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
endscript
}

 

The creator of C and UNIX Dennis Ritchie passed away R.I.P. Dennis

Thursday, October 13th, 2011

Dennis Ritchie old young picture

I just read the lwn.net – Linux Weekly news ‘s website the very sad news that one of the greatest modern day computer heroes Dennis MacAlistair Ritchie after a long illness has passed away in his home.

The original notification for this grieving news are on Rob Pike’s Google Plus wall , this is the original message:

Rob Pike - 1:02 AM - Public
I just heard that, after a long illness, Dennis Ritchie (dmr) died at home this weekend. I have no more information.
I trust there are people here who will appreciate the reach of his contributions and mourn his passing appropriately.
He was a quiet and mostly private man, but he was also my friend, colleague, and collaborator, and the world has lost a truly great mind.

For all those who haven’t heard about Dennis Ritchie , he was a computer scientist who developed the C Programming language and had an immeasurable influence on all kind of Modern programming.

C Programming Language cover Dennis Ritchie

Dennis worked on the development of Unix’s predecessor Multics as well as with Ken Thompson worked together in Bell Labs and are practically the fathers of UNIX.
Unix the Seventh Edition source code has later become the basis for the early UNIX BSD distributions. Among the most important technical contributions Dennis has done is the introduction of a Streams mechanism – pipes – (as called today in GNU/Linux and BSD and other unices).
Ritchie’s C Language creation on top of Ken Thompson’s B Programming language has been standartized and become the de-facto standard for almost every modern existing OS around.
Moreover dmr has been among the co-creators of Plan 9 Operating system (which is currently open-source distributed) as well as coded a few bits for the Inferno OS which today is known under the code name Vita Nuova

Unix Live Free or die Bell labs early UNIX logo

dmr (the hacker nickname of Dennis) lines up across the most notable computer hackers of all times. He received U.S. national Medal of Technology in 1999 from president Bill Clinton for his contributions to co-inventing the UNIX operating system and the creation of C Language

Denis Ritchie receives national prize in 1999 for Technology from president Bill Clinton
To sum it up DMR is just an “icon” in the computer geek world and his memory will surely live forever in the hacker undeground and computer geek culture.

Dennis Ritche near a personal computer picture

A few quotes dmr is so famous with:

"I am not now, nor have I ever been, a member of the demigodic party."
"Usenet is a strange place."
"UNIX is very simple, it just needs a genius to understand its simplicity."
"C is quirky, flawed, and an enormous success."
"We really didn't buy it thinking we'd have this enormous investment."

Here is also a short video telling a few words of UNIX history and showing Dennis Ritchie in his UNIX development years:

Farewell Denis! See you in Hacker’s paradise 😉