Archive for December 12th, 2012

Some Apache performance optimizations to do on brand new installed Linux servers – Apache performance tuning tips

Wednesday, December 12th, 2012

good tips to optimize Apache webserver on Debian CentOS and RHEL Linux for better performance and faster website openings

It is a good idea, on any productive server which is supposed to run Apache + PHP on Linux to do some initial Apache configurations which will guarantee a better WebServer performance and improved Apache client thoroughput. On every each and other new configured Linux server planned to server as an Apache + some database backend, I routinely make this tune ups even without thinking. The reason I do it is time and experience proofed this optimizations works like a charm and almost in 100% of cases they can only improve situation with the server, decrease the general expected load and thus save costs for potential hardware. Besides that the few config options which I'm about to suggest in this article guarantee improved WebPage opening times and most of times overall Apache response times. The consequence of embedding the optimizations has a straight influence on Google / Yahoo PageRanking as it is not a secret most (if not all) Search Engines, rank with a Higher PageRank webpages which load up for lower opening times.

 

1. Change values for KeepAlive, Timeout and KeepAliveTimeout

First thing to change in Apache default config is reduce the default value set for KeepAliveTimeout and KeepAlive and TimeOut

a. Reducing  KeepAliveTimeout

  a.In Debian, Ubuntu servers this value has to be changed in /etc/apache2/apache2.conf

b. in RHEL, Fedora and other RPM based distros check in /etc/httpd/conf/httpd.conf

By default KeepAliveTimeout is set to 15 – KeepAliveTimeout 15. 15 Seconds is a long delay and on a by Apache servers it is very likely you will have hundreds if not thousands of Apache forks or internal threads, keeping still open for clients which already navigated off from the website or websites hosted and served by Apache.

Taking this in consideration, most of the times I prefer setting the KeepAliveTimeout value to 7 secs – i.e.;

KeepAliveTimeout 7

even to some hosts, where you have a well tested PHP Code or just serving static files it is a good idea to decrease it to 5 secs (this is much more risky and likely to create problems, I set it to 5 secs in a vary rare occasions, anyhow you might want to experiment)

Bear in mind that in some cases, where page execution (lets say a PHP script) takes longer to execute than 7 seconds clients might end up with empty pages as Apache will drop off the opened TCP / IP connection to remote client. Thus for some people who run badly written websites with PHP scripts which take long time to execute lowering default KeepAliveTimeout might have negative results. Therefore as a rule of thumb if you reduce the KeepAliveTimeout, be sure to monitor closely with the website testers team or via some website feedback form if the website continues to perform okay for end clients, if not just tune up KeepAliveTimeout to a value with which the website works fine. Other reason why KeepAliveTimeout is so good in almost all cases to reduce is by simply closing quicker opened network connections, less Apache childs keeps loaded in memory and therefore more memory is available for eventual new clients  connecting.

Here is also KeepAliveTimeout explained as pasted from a Debian apache2.conf:

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
#KeepAliveTimeout 15
KeepAliveTimeout 5

b. Turn on KeepAlive

By default most Linux distros came with KeepAlive setting turned off, switch it on;

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
# KeepAlive Off
KeepAlive On

c. Reduce the amount for TimeOut of client inactivity

Default TimeOut setting is set to 300 seconds!
A good value to reduce it to is 40 or 80. 80 value is less likely to create content serving unexpected interrupts. On most servers I just set to 40 as so far this value works well for me.
 

#
# Timeout: The number of seconds before receives and sends time out.
#
#Timeout 300
Timeout 40
 

2. Enable Apache mod-expires – WebServer content caching

debian:~# ln -sf /etc/apache2/mods-available/expires.load /etc/apache2/mods-enabled/expires.load

Depending on Deb or RPM based Linux distro in Apache config (apache2.conf or httpd.conf), add following mod_expires directives;

<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault A86400
ExpiresByType image/x-icon A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType text/css A2592000
ExpiresByType image/gif A604800
ExpiresByType image/png A604800
ExpiresByType image/jpeg A604800
</IfModule>

One note to make, here that on some websites based on Smarty, Zend PHP Framework etc. PHP frameworks mod_expires might cause some troubles, however in 70-80% of the cases just enabling it causes no harm to the overall website functionality. Be sure to test it well if you enable it and don't blame me if it cause you issues.


3. Set ServerRoot and  Raise-up ServerLimit and MaxKeepAliveRequests  directives

By default the value set for ServerLimit is too low for productive servers (256 mpm_prefork Apache childs maximum), thus for servers which are expected to get in parallel few hundreds of unique IP clients I usually set it along with ServerRoot like so;

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at <URL:http://httpd.apache.org/docs/2.2/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot "/etc/apache2"
ServerRoot "/etc/apache2"
ServerLimit 10600

Another good practice is to set MaxKeepAliveRequests which will be handled by Apache forked child to a high value but not to 0 (which will make once forked Apache childs to never die – making them likely to mess up assigned memory due to memory leaks or Apache bugs). On a productive servers I set values from  5000 to 50000.

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 50000

4. Enable mod_rewrite Apache support

This step is not optimizing Apache performance but it is useful to enable mod_rewrite, as there is almost no website today which doesn't use mod_rewrite via .htaccess passed directives.

debian:~# ln -sf  /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/rewrite.load

5. Adjusting default values of StartServers, MinSpareServers, MaxSpareServers, MaxClients and MaxRequestsPerChild  for mpm_prefork

Default config values set for mpm_prefork, are for a tiny home server, depending on  the server amount of memory and CPU power – StartServers, MinSpareServers, MaxSpareServers, MaxClients and MaxRequestsPerChild – should be carefully tailored and tested with Apache Benchmark little tool and Siege or any other benchmarking tool before WebServer is made publicly accessible.

Default values from apache2.conf are like so:

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

A good configuration for a productive server with 24GB of Memory and 8 CPUs x 2.13 Ghz (about 17Ghz Computing Power) would be for exmpl.:

<IfModule mpm_prefork_module>
    StartServers          2000
    MinSpareServers       600
    MaxSpareServers      800
    MaxClients          3600
    MaxRequestsPerChild   10000
</IfModule>

5. Intall and Enable Eaccelerator

On almost all servers I install I install immediately after basic Apache + PHP + MySQL packages, Eaccelerator. Eaccelerator helps utilizing better server free memory and significantly accelerates Apache pages serve time

I've earlier blogged on How to install Eaccelerator to decrease server CPU load and increase page serving performance here

6. Disable Server Side Includes ( SSI ) support

I've earlier blogged how to disable Apache SSI on Debian Linux – you can read here. The change SSI will make whether off is not so big so even leaving it on  is not a big deal.

7. Remove and Purge Suhosin apache module

suhosin is useful module that tightens Apache security, however for me it has earlier create a lot of issues and it is my personal view that life is better without suhosin. I've earlier stumbled on a weird issue causing Apache to mysteriously crash – removing suhosin solved it all. I'm not sure if suhosin is installed by default on Debian, but it is often installed a a package dependency to some php-devel packages, so I find it wise always to check if it is present on the system and remove it if it is.
 

8. Enable Apache mod_deflate (gzip) compression to speed up delivery of CSS and Javascripts

Archiving with gzip and de-archiving CSS, JS and HTML is very useful, as it reduces the size of transferred content. This however might impose a bit of higher CPU load, so I only enable this one whether I target increase in network thoroughput, however for people concerned of CPU load it is better to keep it off as it is by default.

For a bit more on how mod_deflate is enabled on Debian check my previous article – Speeding Apache hosted websites with mod_deflate gzip compression

CentOS and RHEL users who need to enable mod_deflate – check here

9. Change the way logrotate handles log rotation (disable log gzip compession) or disable Apache logging completely

On Linux servers with Apache where 30000 to 50000 of unique IP visitors requests are served, the access.log becomes enormous. Things become even worser as by default Apache logs are configured to be rotated once a week instead of daily. Thus once logrotation takes place, a huge log has to be processed – for instance 20 GB. This puts extra load on the server and often makes the normal Apache operation bloated. To get rid of this problem I suggest you check my previous article – Recommended access.log logrotate practices on heavy loaded servers

Alternatively it is sometimes, better to completely disable Apache access.log logging to reduce a bit the Apache load – though from security and statistical point of view it is bad practice. I've disabled it however, as on some servers logging is implemented on PHP scripts level instead. I've earlier blogged how disabling access.log and error.log is done here

10. Disable Apache version reporting

This is more of a security than performance optimization, but also has neglectful effect, as on requests one line less is reported by Apache 🙂
To disable Apache version reporting check my previous article here

11. Switch from mpm_prefork to  mpm_worker Apache (threaded) engine

For some new Apache configurations, which doesn't need exec(); or system(); or any other PHP embedded external code execution functions, from performance point of view it is much better to just switch to the much more sophisticated performance efficient and less memory hungry Apache2  mpm-worker engine – the downside of it is you will have to configure PHP to be executed via php5-cgi apache module.

12. Tune up (increase) PHP memory_limit variable

This is not Apache optimization, but most servers need it as they run Apache and PHP in a line. Default PHP memory_limit is set to the low 16 Mb it is good to raise it to 64 or 128MB (but be careful as this might make Apache easier to DoS or DDoS)
I've blogged on the topic of memory_limit and timezone issues I experienced earlier here

13. Make sure you have a good quick DNS set in /etc/resolv.conf

An usual /etc/resolv.conf which I use for new servers with Apache looks like so:

debian:~# cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 208.67.222.222
nameserver 208.67.220.220
 

The first line is set to use 127.0.0.1, as I find it very useful and to improve overall system efficiency and make it much fail proof, if the server is configured to run a custom DJBDNS server on localhost.

As you see further DNS set in my usual resolv.conf's are Google's Public DNS 8.8.8.8 and 8.8.4.4 and OpenDNS's 208.67.222.222 and 208.67.220.220

I highly recommend you follow my practice and install DJBDNS local caching DNS to speed up resolving efficiency and hence speed up Apache client interactions (of course this is useful only if Apache or some PHP scripts use DNS requests, but as most do it is a  good practice)

After all changes, to take affect I do the usual Apache restart with;

debian:~# apache2ctl -k restart
.....

That's it, if you know of other optimization tips, Please drop a comment 🙂

Get Hardware System info on Debian Linux / How to detecting hardware and servers model on GNU / Linux

Wednesday, December 12th, 2012

hardware-info-getting-on-linux-howto

Users who are novice to Linux should be probably interested on how to get a decent Hardware System Information. Getting system info on Windows is quite straight forward, however on Linux and especially on Linux servers it is a bit confusing at first and even for people who spend years administrating Linux servers, or even have a Linux desktop it is very likely after a period of time to forget how exactly last time got the hardware system information. I'm administrating Linux servers and running a linux desktop for already almost 11 years and often it happened I'm away from configuring a new server for a year, or even when configuring a new server I don't need to get exact system information from command line, as I know it already from the server hardware manual. However whether managing a bunch of dedicated servers or purchasing new systems which are physically away and someone pre-configured the server with some basis Linux install, often a very raw info is provided by the Dedicated Provider on exact server metrics. Other situation, where it is good idea to have a precise system hardware vendor information on a server, is if you just joined a company with a bunch of existing dedicated servers, whose exact hardware configuration is no documented anywhere and suddenly some RAID or piece of hardware located on 1 of the 100 dedicated servers starts misbehaving causing hour down-times and client important data loss.

In any of those cases it always takes me few times of research to find out what exact methodology I used to get the hardware info last time. To make my life for future times easier and not loose the few minutes of research and reading on how to get Linux server system information I decided to write this short article, which might hopefully be useful to others out there who face similar periodic questioning on what was the command to get hardware system info.

Of course the general commands to get some general overview on a Linux server as anyone knows are:

a. dmesg
b. cat /proc/cpuinfo
c. lspci
d. lsusb
c. free -m

A note to make here is that in order to have lsusb and lspci commands present you will have to have installed the deb packs lsusb and pciutils.

However as I prior said, this tools output is not enough or the output is not enough systematic and hard to read and understand especially for lazy or short memory admins like me. Thus it is worthy to mention few others which can be installed as a separate packages and gives more structured and very precised information on what kind of machine hardware you're accessing through ssh.

Here is the list of all of profiled hardware detection progs and scripts:

1. dmidecode

2. lshw

3. x86info

4. hwinfo

5. hardinfo

6. biosdecode

 

To install all of them in a raw with apt-get do:

debian:~# apt-get install --yes dmidecode lshw x86info hwinfo hardinfo superiotool
Reading package lists... Done
Building dependency tree      
Reading state information... Done
dmidecode is already the newest version.
hardinfo is already the newest version.
lshw is already the newest version.
The following extra packages will be installed:
  libhd16
The following NEW packages will be installed:
  hwinfo libhd16 superiotool x86info
0 upgraded, 4 newly installed, 0 to remove and 9 not upgraded.
Need to get 827 kB of archives.
After this operation, 4,506 kB of additional disk space will be used.
Get:1 http://ftp.uk.debian.org/debian/ squeeze/main libhd16 amd64 16.0-2 [696 kB]
Get:2 http://ftp.uk.debian.org/debian/ squeeze/main hwinfo amd64 16.0-2 [46.6 kB]
Get:3 http://ftp.uk.debian.org/debian/ squeeze/main superiotool amd64 0.0+r5050-1 [43.0 kB]
Get:4 http://ftp.uk.debian.org/debian/ squeeze/main x86info amd64 1.25-1 [40.9 kB]
Fetched 827 kB in 2s (378 kB/s)  
Selecting previously deselected package libhd16.
(Reading database ... 85783 files and directories currently installed.)
Unpacking libhd16 (from .../libhd16_16.0-2_amd64.deb) ...
Selecting previously deselected package hwinfo.
Unpacking hwinfo (from .../hwinfo_16.0-2_amd64.deb) ...
Selecting previously deselected package superiotool.
Unpacking superiotool (from .../superiotool_0.0+r5050-1_amd64.deb) ...
Selecting previously deselected package x86info.
Unpacking x86info (from .../x86info_1.25-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up libhd16 (16.0-2) ...
Setting up hwinfo (16.0-2) ...
Setting up superiotool (0.0+r5050-1) ...
Setting up x86info (1.25-1) ...

Next just try to launch the tools one by one and check the content of the output, in my view  the most useful one and maybe also the most popular is dmidecode, the rest however might be useful to get specific hardware debug info.

1.  hwinfo

debian:~# hwinfo |tee -a server-hardware-info.txt
....

hwinfo will provide you a very long list of very thoroughful information on hardware. A lot of the info it shows however is not so useful for regular admins, but will be of high value to people who need to develop a new Linux driver for respective hardware.

2. lswh

debian:~# lshw > linux-hw-info.txt

lshw provides long list of debug information and if the output is not redirected to a file the screen gets flooded, if not piped to less. For that reason I will not paste output here.

3. x86info

debian:~# x86info

x86info v1.25.  Dave Jones 2001-2009
Feedback to <davej@redhat.com>.

Found 2 CPUs
————————————————————————–
CPU #1

EFamily: 0 EModel: 2 Family: 6 Model: 42 Stepping: 7
CPU Model: Unknown model.
Processor name string: Intel(R) Pentium(R) CPU G630 @ 2.70GHz
Type: 0 (Original OEM)    Brand: 0 (Unsupported)
Number of cores per physical package=8
Number of logical processors per socket=16
Number of logical processors per core=2
APIC ID: 0x0    Package: 0  Core: 0   SMT ID 0
————————————————————————–
CPU #2
EFamily: 0 EModel: 2 Family: 6 Model: 42 Stepping: 7
CPU Model: Unknown model.
Processor name string: Intel(R) Pentium(R) CPU G630 @ 2.70GHz
Type: 0 (Original OEM)    Brand: 0 (Unsupported)
Number of cores per physical package=8
Number of logical processors per socket=16
Number of logical processors per core=2
APIC ID: 0x2    Package: 0  Core: 0   SMT ID 2
————————————————————————–
WARNING: Detected SMP, but unable to access cpuid driver.
Used Uniprocessor CPU routines. Results inaccurate.

As you see x86info, mainly provides information on CPU Cache, exact model, family AND APIC (don't mix it with ACPI – advanced power management interface)
APIC is a chip that remaps IOs and IRQs of your computer to the CPU(s), thus in most cases it is more of not so needed debug information.

4. biosdecode

debian:~#  biosdecode
# biosdecode 2.9
ACPI 2.0 present.
    OEM Identifier: LENOVO
    RSD Table 32-bit Address: 0xBCD9C028
    XSD Table 64-bit Address: 0x00000000BCD9C068
SMBIOS 2.6 present.
    Structure Table Length: 2233 bytes
    Structure Table Address: 0x000EBB70
    Number Of Structures: 59
    Maximum Structure Size: 184 bytes
PNP BIOS 1.0 present.
    Event Notification: Not Supported
    Real Mode 16-bit Code Address: F000:BC66
    Real Mode 16-bit Data Address: F000:0000
    16-bit Protected Mode Code Address: 0x000FBC8E
    16-bit Protected Mode Data Address: 0x000F0000
PCI Interrupt Routing 1.0 present.
    Router ID: 00:1f.0
    Exclusive IRQs: None
    Compatible Router: 8086:27b8
    Slot Entry 1: ID 00:1f, on-board
    Slot Entry 2: ID 00:1b, on-board
    Slot Entry 3: ID 00:16, on-board
    Slot Entry 4: ID 00:1c, on-board
    Slot Entry 5: ID 02:00, slot number 21
    Slot Entry 6: ID 00:01, on-board
    Slot Entry 7: ID 00:06, on-board
    Slot Entry 8: ID 00:1d, on-board
    Slot Entry 9: ID 00:1a, on-board
    Slot Entry 10: ID 03:00, on-board
    Slot Entry 11: ID 00:02, on-board
    Slot Entry 12: ID 00:00, on-board

As you see biosdecode, also provides a lot of hex addresses, also reports on the exact CPU architecture on the system.

The line   XSD Table 64-bit Address: 0x00000000BCD9C068, indicated the host is running a 64 bit CPU, most of the rest info like Slot entries IDs etc. is not so useful.

The most useful info that biosdecode provides is the exact type of BIOS (Basic Input Output System) bundled with the system in my case the BIOS is running on a Lenovo host and is vendored by Lenovo, thus it shows in the cmd output:

OEM Identifier: LENOVO

5. hardinfo

debian:~# hardinfo | tee -a hardware-info.txt


hardinfo gnome screenshot debian gnu / linux

HardInfo is the GNOME GTK+ program which displays robust and thouroughful info in same was as Windows System Info does on  GNOME Desktop. If however you run it under console or via ssh it does display what it detects as: 

Computer hardware, operating system, kernel modules, supported system languages, existing filesystems, Display, set environment variables, Existing system users, Processor type, Memory, PCI and USB devices, Printers (if attached), Battery type (if run on laptop), Storage, Other Input devices

hardinfo, does a few benchmarking tests using CPU stress test algorithms to do Blowfish encryption, CryptoHash, Fibonacci, N-Queens, FPU FFT and FPU raytracing. This benchmark values, if run on a couple of hosts can be used to compare different hardware performances.

6. dmidecode

debian: # dmidecode > system-hware-info.txt

The output from dmidecode is very very detailed and verbose. Though along with the useful info there is plenty of debug information, the debug information it provides is much user friendly / user comprehensible than the rest of tools, thus I guess dmidecode is nowadays preferred by me and probably most of the Linux sys admins.

debian:~# dmidecode |head -n 34
# dmidecode 2.9
SMBIOS 2.6 present.
59 structures occupying 2233 bytes.
Table at 0x000EBB70.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
    Vendor: LENOVO
    Version: 9QKT37AUS
    Release Date: 02/14/2012
    Address: 0xF0000
    Runtime Size: 64 kB
    ROM Size: 2560 kB
    Characteristics:
        PCI is supported
        BIOS is upgradeable
        BIOS shadowing is allowed
        Boot from CD is supported
        Selectable boot is supported
        BIOS ROM is socketed
        EDD is supported
        5.25"/1.2 MB floppy services are supported (int 13h)
        3.5"/720 KB floppy services are supported (int 13h)
        3.5"/2.88 MB floppy services are supported (int 13h)
        Print screen service is supported (int 5h)
        8042 keyboard services are supported (int 9h)
        Serial services are supported (int 14h)
        Printer services are supported (int 17h)
        ACPI is supported
        USB legacy is supported
        BIOS boot specification is supported
        Targeted content distribution is supported
    BIOS Revision: 0.37
 

Though it is the most useful tool on some hardware configurations it might not display any data because the BIOS is lacking a DMI implementation.

In almost all cases dmidecode is enough to check what kind of hardware you have ssh-ed to. dmidecode is available also not only on Debian but on Fedora and almost all (if not all Linux distros), through default repositories.