Posts Tagged ‘CentOS’

yum add proxy on CentOS, RHEL, Fedora Linux howto

Thursday, June 5th, 2014

yum-via-proxy-yum-package-management-mascot
Whether you had to install a CentOS server in a DMZ-ed network with paranoic system firewall rules or simply you want to use your own created RPM local repository to run RPM installs and CentOS system updates via monitored Proxy you will have to configure yum to use a proxy.

There is a standard way to do it by adding a proxy directive to /etc/yum.conf as explained in CentOS official documetnation.
However for some reason:

proxy=http://your-proxy-url.com:8080
proxy_username=yum-user
proxy_password=qwerty

proxy vars adding to /etc/yum.conf [main] section is not working on CentOS 6.5?
However there is a dirty patch by using the OS environment standard variable http_proxy
To make yum work via proxy in gnome-terminal run first:

export http_proxy=http://your-proxy-server.com:8080

or if proxy is protected by username / password run instead:

export username='yum-user'
export password='qwerty'
export http_proxy="http://$username:$password@your-proxy-server:8080/

Afterwards yum will work via the proxy, i.e.:

yum update && yum upgrade

To make http_proxy exported system wide check my previous post – Set Proxy System-Wide

Hope this helps someone.

Installing Virtualbox Guest Additions Vboxadditions on CentOS 7, Fedora 19 / 20 and RHEL 6.5 / 5.10 on Windows host

Thursday, July 17th, 2014

Virtualbox_guest_additions-on-centos-7-howto-virtualbox-logo
If you decided to use Redhat based Linux distribution inside Virtualbox Virtual Machine on Windows 7 / 8, first thing to do right after installing the Linux guest OS is to install Virtualbox Guest Additions. Earlier I've blogged How to enable virtualbox VM Windows XP fullscreen mode on Virtualbox running on Ubuntu now I need it vice versa a Linux OS living inside Virtual Machine running on top of Windows 7 Enterprise. need to enable Full-screen mode. Another reason why Vboxadditions should be installed right after Linux install is complete is for performance reasons and better integration between host OS and guest OS. Virtualbox Guest Additions enables mouse's copy / paste functions to work between Win and Virtualized Linux as well as enables arbitrary screen resolutions (resizing VM guest window) etc. On Windows versions of Virtualbox to install Vbox Guest Additions it is no longer necessery to download and mount a separate vbox-guest-additions ISO file, this is handled by Virtualbox itself.

The steps described here for installation of VirtualBox Guest Additions are tested and prooved as working on current latest CentOS 6.5, however they should be working (with or without minor modifications) in rest of RPM based Linux distributions

Fedora 20, 19, 18, 17, 16, 15, 14, 13, 12, CentOS 6.5, 6.4, 6.3, 6.2, 6.1, 6.0, 5.10 and Redhat Enterprise Linux (RHEL) 6.5, 6.4, 6.3, 6.2, 6.1, 6, 5.10 VirtualBox version installed on Windows 7 host is 4.3.14 build.

 

Devices -> Drag'n'Drop (Bidirectional)

Devices -> Share Clipboard (Bidirectional)


1. Installing VirtualBox GuestAdditions on CentOS 7, Redhat, Fedora

a) become superuser

su root
[root@centos~:]#

b) Mount Virtualbox provided Guest additions into virtual cdrom /dev/cdrom

From VirtualBox click on
 

Devices -> Install Guest Additions

centos7-install-guest-additions-cd-screenshot-microsoft-windows-virtualbox

mkdir /media/VirtualBoxGuestAdditions
mount -r /dev/cdrom /media/VirtualBoxGuestAdditions

c) Update to the latest Linux kernel with yum

yum -y update kernel*

Then to boot up into the updated kernel – restart the system

shutdown -r now


d) Install Kernel headers bzip dkms bzip2

yum install -y gcc kernel-devel kernel-headers dkms make bzip2 perl

If you're installing VirtualBox Guest Additions on CentOS 6 or RHEL 5, you will have to install also below 2 rpms:
 

## CentOS 6 and Red Hat (RHEL) 5 ##
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
 
## CentOS 5 and Red Hat (RHEL) 5 ##
rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

 


e) export KERN_DIR to just installed kernel source

 

For CentOS 7 export following KERN_DIR

 

## Current running kernel on Fedora, CentOS 7 and Red Hat (RHEL) 7 ##
KERN_DIR=/usr/src/kernels/`uname -r`
export KERN_DIR

For older CentOS and RHEL 5 export
## Current running kernel on CentOS 5 and Red Hat (RHEL) 5 ##

KERN_DIR=/usr/src/kernels/`uname -r`-`uname -m`
export KERN_DIR

f) Download VBoxAdditions 4.3.14 iso from Virtualbox website

I've made mirror of VBoxGuestAdditions_4.3.14.iso for download here:
 

mkdir /usr/share/virtualbox
cd /usr/share/virtualbox

wget http://dlc.sun.com.edgesuite.net/virtualbox/4.3.14/VBoxGuestAdditions_4.3.14.iso

g) Mount the iso
 

mount -t iso9660 VBoxGuestAdditions_4.3.14.iso /media/VirtualBoxGuestAdditions/


h) Compile and Install Virtualbox GuestAdditions kernel modules
For both 32-bit and 64-bit systems run one and the same shell script:

cd /media/VirtualBoxGuestAdditions
./VBoxLinuxAdditions.run

Verifying archive integrity… All good.
Uncompressing VirtualBox 4.3.14_RC1 Guest Additions for Linux…………
VirtualBox Guest Additions installer
Removing installed version 4.3.14_RC1 of VirtualBox Guest Additions…
Copying additional installer modules …
Installing additional modules …
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module                   [  OK  ]
Building the shared folder support module                  [  OK  ]
Building the OpenGL support module                         [  OK  ]
Doing non-kernel setup of the Guest Additions              [  OK  ]
You should restart your guest to make sure the new modules are actually used

Installing the Window System drivers
Installing X.Org Server 1.15 modules                       [  OK  ]
Setting up the Window System to use the Guest Additions    [  OK  ]
You may need to restart the hal service and the Window System (or just restart
the guest system) to enable the Guest Additions.

Installing graphics libraries and desktop services componen[  OK  ]

l) Restart the CentOS Virtual Machine
 

reboot

Now get a beer and celebrate the full-screen / USB / Webcamera enabled support ! 🙂

This article is possible thanks to earlier article called Virtualbox guest additions on Fedora CentOS and Redhat.

http://www.if-not-true-then-false.com/2010/install-virtualbox-guest-additions-on-fedora-centos-red-hat-rhel/

 

 

Disable php notice logging / stop variable warnings in error.log on Apache / Nginx / Lighttpd

Monday, July 28th, 2014

disable_php_notice_warnings_logging_in-apache-nginx-lighttpd
At one of companies where I administrate few servers, we are in process of optimizing the server performance to stretch out the maximum out of server hardware and save money from unnecessery hardware costs and thus looking for ways to make server performance better.

On couple of web-sites hosted on few of the production servers, administrating, I've noticed dozens of PHP Notice errors, making the error.log quickly grow to Gigabytes and putting useless hard drive I/O overhead. Most of the php notice warnings are caused by unitialized php variables.

I'm aware having an unitialized values is a horrible security hole, however the websites are running fine even though the notice warnings and currently the company doesn't have the necessery programmers resource to further debug and fix all this undefined php vars, thus what happens is monthly a couple of hundreds megabytes of useless same php notice warnings are written in error.log.

That  error.log errors puts an extra hardship for awstats which is later generating server access statistics while generating the 404 errors statistics and thus awstats script has to read and analyze huge files with plenty of records which doesn't have nothing to do with 404 error

We found this PHP Notice warnings logged is one of the things we can optimize had to be disabled.

Here is how this is done:
On the servers running Debian Wheezy stable to disable php notices.

I had to change in /etc/php5/apache2/php.ini error_reporting variable.

Setting was to log everything (including PHP critical errors, warning and notices) like so:
 

vi /etc/php5/apache2/php.ini

error_reporting = E_ALL & ~E_DEPRECATED

to

error_reporting = E_COMPILE_ERROR|E_ERROR|E_CORE_ERROR


On CentOS, RHEL, SuSE based servers, edit instead /etc/php.ini.

This setting makes Apache to only log in error.log critical errors, php core dump (thread) errors and php code compilation (interpretation errors)

To make settings take affect on Debian host Apache webserver:

/etc/init.d/apache2 restart

On CentOS, RHEL Linux, had to restart Apache with:

/etc/init.d/httpd restart

For other servers running Nginx and Lighttpd webservers, after changing php.ini:

service nginx reload
service lighttpd restart

To disable php notices errors only on some websites, where .htaccess enabled, you can use also place in website DocumentRoot .htaccess:
 

php_value error_reporting 2039


Other way to disable via .htaccess is by adding to it code:
 

php_flag display_errors off


Also for hosted websites on some of the servers, where .htaccess is disabled, enabling / disabling php notices can be easily triggered by adding following php code to index.php

define('DEBUG', true);

if(DEBUG == true)
{
    ini_set('display_errors', 'On');
    error_reporting(E_ALL);
}
else
{
    ini_set('display_errors', 'Off');
    error_reporting(0);
}

 

How to query LDAP (Windows Domain Controller) directory entries from Linux – ldapsearch common searche examples

Tuesday, November 18th, 2014

active-directory-logo
If you have a hybrid network of Windows servers and computers in Active Directory (AD) Domain Names and Linux hosts hosting various Java / PHP / Python applications like many of the middle and big companies (organization) have, sooner or later you will have to deploy an application which uses some some user authentication from the Linux host to Windows Domain Controller, you will end up in need to be able to query the AD, which is using LDAP (Lightweight Directory Access Protocol) to store the AD user credentials and tons of other information important for proper Active Directory operations.

LDAP is a key industry standard for storing and accessing distributed directory information services over Internet Protocol (IP). LDAP is great for sharing of information about users, systems, networks, services, and applications throughout the network. The corporate world nowadays would have been impossible without LDAP.
As of time of writting latest RFC  (Resource for Comment) 4511 document describes industrial specification of LDAP version 3.0 and therefore this is the most often used and implemented version.

LDAP protocol supports generally following operations:

Adding, Delete, Bind (Authenticate to LDAP server), Delete Search and Compare, Modify and Modify DN (Distringuished Name)
Deleting recordsh

On Linux to retrieve / locate AD entries, there is ldapsearch  command which opens connection to LDAP host server port, with set username and password. ldapsearch tool makes its search based on a filter.

To have make and modify queries in LDAP from GNU / Linux you will have to have installed ldap-utils on Debian, i.e.:

apt-get –yes install ldap-utils


to have ldapseach, ldapmodify, ldapsearch ldappasswd on CentOS / Redhat Linux, you need openldap-clients.x86_64

yum -y install openldap-clients.x86_64

Returned result from ldapsearch clients will be returned in LDIF format (LDAP Data Interchange format).

ldapsearch basic format is like thsi:

ldapsearch [optional_options] [optional_search_filter] [optional_list_of_attributes]

ldapsearch could query (LDAP – ADs) in unencrypted form simple LDAP, encrypted form with SSL certificate (LDAPs) or through LDAP with STARTTLS.
Logically most organizations nowadays are using LDAPs, as it offers the highest level of security. Unencrypted LDAP servers listen usually on
port 389, LDAPs communicates on port 636 once an SSL handshake is made between client and server and LDAP with STARTTLS communicates on standard port 389.

Here is 3 examples of common  ldapsearch queries

1. Return all entries in LDAP server
 

ldapsearch -D "cn=directory manager" -w secret -p 389 -h ldap.your-organization.org -b "dc=your-organization,dc=com" -s sub "(objectclass=*)"


"objectclass=*" is a serch filter matching all entries in the directory (time and size limits on output limit set for the server will take affect)

2. Searching the Root DSE Entry

root DSE is special entry containing list of all suffixes supported by local Directory Server. Getting root DSE is done with  base of "", a search scope of base, and a filter of "objectclass=*"

ldapsearch -D "cn=directory manager" -w secret_pass -p 389 -h ldaps.your-organization.org  -b "dc=your-organization,dc=com" -s sub "cn=babs jensen"

 

3. Searching Directory Server Schema Entry

LDAP server stores all directory server schema in special entry cn=schema.
schema entry contains information on every object class and attribute defined for the Directory Server. Command to searches  contents of the cn=schema entry is:
 

ldapsearch -D "cn=directory manager" -w secret_pass -p 389 -h ldaps.your-organization.org -b "cn=schema" -s base "objectclass=*"


4. Check whether cn=My-Account1 account is working and enabled

ldapsearch -H ldaps://ldaps.your-organization.org -b o=my-org,c=bg -s sub -D cn=My-Account1,ou=users,ou=ABC,o=my-org,c=ABC -W '(&(cn=My-Acount1)(objectclass=my-org-Account))'


5. check all members of cn=MY_ADMINISTRATION

 

ldapsearch -H ldaps://ldaps.your-organization.org -b o=my-org,c=bg -s sub -D cn=My-Account1,ou=users,ou=ABC,o=my-org,c=ABC -W '(&(cn=MY_ADMINISTRATION)(member=*))'

 

6. check all members of all groups belonging to user
 

ldapsearch -H ldaps://ldaps.your-organization.org -b ou=ABC,ou=ABC1,ou=ABC2,ou=groups,ou=ABC,o=my-org,c=ABC -s sub -D cn=My-Account1,ou=users,ou=ABC,o=my-org,c=ABC -W '(cn=*)'
 

Whether ldapsearch queries are to be common and scripted or just for simplification of readability of query to LDAP it is useful to use LDAP_BASEDN – a query search base. By setting search base you can further omit in query -b

export LDAP_BASEDN="dc=your-organization,dc=com"
ldapsearch -D "cn=directory manager" -w secret_pass -p 389 -h ldap.your-organization.org "cn=labs jordan"

In Linux LDAP's open-source implementation is called OpenLDAP.
On Linux LDAP protocol can be easily integrated / used in combination with FTP servers (such as proftpd), DNS servers, Mail Servers (Courier), Samba servers, Radius (IP Telephony), sudo, as well as most programming languages such as PHP, Python etc.

How to disable IPv6 on Debian / Ubuntu / CentOS and RHEL Linux

Friday, December 9th, 2011

I have few servers, which have automatically enabled IPv6 protocols (IPv6 gets automatically enabled on Debian), as well as on most latest Linux distribituions nowdays.

Disabling IPv6 network protocol on Linux if not used has 2 reasons:

1. Security (It’s well known security practice to disable anything not used on a server)
Besides that IPv6 has been known for few criticil security vulnerabilities, which has historically affected the Linux kernel.
2. Performance (Sometimes disabling IPv6 could have positive impact on IPv4 especially on heavy traffic network servers).
I’ve red people claiming disabling IPv6 improves the DNS performance, however since this is not rumors and did not check it personally I cannot positively confirm this.

Disabling IPv6 on all GNU / Linuces can be achieved by changing the kernel sysctl settings net.ipv6.conf.all.disable_ipv6 by default net.ipv6.conf.all.disable_ipv6 equals 1 which means IPv6 is enabled, hence to disable IPv6 I issued:

server:~# sysctl net.ipv6.conf.all.disable_ipv6=0

To set it permanently on system boot I put the setting also in /etc/sysctl.conf :

server:~# echo 'net.ipv6.conf.all.disable = 1 >> /etc/sysctl.conf

The aforedescribed methods should be working on most Linux kernels version > 2.6.27 in that number it should work 100% on recent versions of Fedora, CentOS, Debian and Ubuntu.

To disable IPv6 protocol on Debian Lenny its necessery to blackist the ipv6 module in /etc/modprobe.d/blacklist by issuing:

echo 'blacklist ipv6' >> /etc/modprobe.d/blacklist

On Fedora / CentOS there is a another universal “Redhat” way disable IPv6.

On them disabling IPv6 is done by editting /etc/sysconfig/network and adding:

NETWORKING_IPV6=no
IPV6INIT=no

I would be happy to hear how people achieved disabling the IPv6, since on earlier and (various by distro) Linuxes the way to disable the IPv6 is probably different.
 

Alto to stop Iptables IPV6 on CentOS / Fedora and RHEL issue:

# service ip6tables stop

# service ip6tables off

Text Monitoring of connection server (traffic RX / TX) business in ASCII graphs with speedometer / Easy Monitor network traffic performance

Friday, May 4th, 2012

While reading some posts online related to MS-Windows TcpViewnetwork traffic analyzing tool. I've came across very nice tool for tracking connection speed for Linux (Speedometer). If I have to compare it, speedometer is somehow similar to nethogs and iftop bandwidth network measuring utilities .

What differentiates speedometer from iftop / nethogs / iptraf is it is more suitable for visualizing a network file or data transfers.
The graphs speedometer draws are way easier to understand, than iftop graphs.

Even complete newbies can understand it with no need for extraordinary knowledge in networking. This makes Speedometer, a top tool to visually see the amount of traffic flowing through server network interface (eth0) … (eth1) etc.

What speedometer shows is similar to the Midnight Commander's (mc) file transfer status bar, except the statistics are not only for a certain file transfer but can show overall statistics over server passing network traffic amount (though according to its manual it can be used to also track individual file transfers).

The simplicity for basic use makes speedometer nice tool to track for network congestion issues on Linux. Therefore it is a  must have outfit for every server admin. Below you see a screenshot of my terminal running speedometer on a remote server.

Speedometer ascii traffic track server network business screenshot in byobu screen like virtual terminal emulator

1. Installing speedometer on Debian / Ubuntu and Debian derivatives

For Debian and Ubuntu server administrators speedometer is already packaged as a deb so its installation is as simple as:

debian:~# apt-get --yes install speedometer
....

2. Installing speedometer from source for other Linux distributions CentOS, Fedora, SuSE etc.

Speedometer is written in python programming language, so in order to install and use on other OS Linux platforms, it is necessery to have installed (preferably) an up2date python programming language interpreter (python ver. 2.6 or higher)..
Besides that it is necessary to have installed the urwid -( console user interface library for Python) available for download via excess.org/urwid/

 

Hence to install speedometer on RedHat based Linux distributions one has to follow these steps:

a) Download & Install python urwid library

[root@centos ~]# cd /usr/local/src
[root@centos src]# wget -q http://excess.org/urwid/urwid-1.0.1.tar.gz
[root@centos src]# tar -zxvvf urwid-1.0.1.tar.gz
....
[root@centos src]# cd urwid-1.0.1
[root@centos urwid-1.0.1]# python setup.py install
running install
running build
running build_py
creating build
creating build/lib.linux-i686-2.4
creating build/lib.linux-i686-2.4/urwid
copying urwid/tests.py -> build/lib.linux-i686-2.4/urwid
copying urwid/command_map.py -> build/lib.linux-i686-2.4/urwid
copying urwid/graphics.py -> build/lib.linux-i686-2.4/urwid
copying urwid/vterm_test.py -> build/lib.linux-i686-2.4/urwid
copying urwid/curses_display.py -> build/lib.linux-i686-2.4/urwid
copying urwid/display_common.py -> build/lib.linux-i686-2.4/urwid
....

b) Download and install python-setuptools

python-setuptools is one other requirement of speedometer, happily on CentOS and Fedora the rpm package is already there and installable with yum:

[root@centos ~]# yum -y install python-setuptools
....

c) Download and install Speedometer

[root@centos urwid-1.0.1]# cd /usr/local/src/
[root@centos src]# wget -q http://excess.org/speedometer/speedometer-2.8.tar.gz
[root@centos src]# tar -zxvvf speedometer-2.8.tar.gz
.....
[root@centos src]# cd speedometer-2.8
[root@centos speedometer-2.8]# python setup.py install
Traceback (most recent call last):
File "setup.py", line 26, in ?
import speedometer
File "/usr/local/src/speedometer-2.8/speedometer.py", line 112
n = n * granularity + (granularity if r else 0)
^

While running the CentOS 5.6 installation of speedometer-2.8, I hit the
"n = n * granularity + (granularity if r else 0)
error.

After consultation with some people in #python (irc.freenode.net), I've figured out this error is caused due the outdated version of python interpreter installed by default on CentOS Linux 5.6. On CentOS 5.6 the python version is:

[root@centos ~]# python -V
Python 2.4.3

As I priorly said speedometer 2.8's minimum requirement for a python to be at v. 2.6. Happily there is quick way to update python 2.4 to python 2.6 on CentOS 5.6, as there is an RPM repository maintained by Chris Lea which contains RPM binary of python 2.6.

To update python 2.4 to python 2.6:

[root@centos speedometer-2.8]# rpm -Uvh http://yum.chrislea.com/centos/5/i386/chl-release-5-3.noarch.rpm[root@centos speedometer-2.8]# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CHL[root@centos speedometer-2.8]# yum install python26

Now the newly installed python 2.6 is executable under the binary name python26, hence to install speedometer:

[root@centos speedometer-2.8]# python26 setup.py install
[root@centos speedometer-2.8]# chown root:root /usr/local/bin/speedometer
[root@centos speedometer-2.8]# chmod +x /usr/local/bin/speedometer

[root@centos speedometer-2.8]# python26 speedometer -i 1 -tx eth0

The -i will instruct speedometer to refresh the screen graphs once a second.

3. Using speedometer to keep an eye on send / received traffic network congestion

To observe, the amount of only sent traffic via a network interface eth0 with speedometer use:

debian:~# speedometer -tx eth0

To only keep an eye on received traffic through eth0 use:

debian:~# speedometer -rx eth0

To watch over both TX and RX (Transmitted and Received) network traffic:

debian:~# speedometer -tx eth0 -rx eth0

If you want to watch in separate windows TX and RX traffic while  running speedometer you can run in separate xterm windows speedometer -tx eth0 and speedometer -rx eth0, like in below screenshot:

Monitor Received and Transmitted server Network traffic in two separate xterm windows with speedometer ascii graphs

4. Using speedometer to test network maximum possible transfer speed between server (host A) and server (host B)

The speedometer manual suggests few examples one of which is:

How fast is this LAN?

host-a$ cat /dev/zero | nc -l -p 12345
host-b$ nc host-a 12345 > /dev/null
host-b$ speedometer -rx eth0

When I red this example in speedometer's manual, it wasn't completely clear to me what the author really meant, but a bit after when I thought over the example I got his point.

The idea behind this example is that a constant stream of zeros taken from /dev/zero will be streamed over via a pipe (|) to nc which will bind a port number 12345, anyone connecting from another host machine, lets say a server with host host-b to port 12345 on machine host-a will start receiving the /dev/zero streamed content.

Then to finally measure the streamed traffic between host-a and host-b machines a speedometer is started to visualize the received traffic on network interface eth0, thus measuring the amount of traffic flowing from host-a to host-b

I give a try to the exmpls, using for 2 test nodes my home Desktop PC, Linux running  arcane version of Ubuntu and my Debian Linux notebook.

First on the Ubuntu PC I issued
 

hipo@hip0-desktop:~$ cat /dev/zero | nc -l -p 12345
 

Note that I have previously had installed the netcat, as nc is not installed by default on Ubuntu and Debian. If you, don't have nc installed yet, install it with:

apt-get –yes install netcat

"cat /dev/zero | nc -l -p 12345" will not produce any output, but will display just a blank line.

Then on my notebook I ran the second command example, given in the speedometer manual:
 

hipo@noah:~$ nc 192.168.0.2 12345 > /dev/null

Here the 192.168.0.2 is actually the local network IP address of my Desktop PC. My Desktop PC is connected via a normal 100Mbit switch to my routing machine and receives its internet via  NAT. The second test machine (my laptop), gets its internet through a WI-FI connection received by a Wireless Router connected via a UTP cable to the same switch to which my Desktop PC is connected.

Finally to test / get my network maximum thoroughput I had to use:

hipo@noah:~$ speedometer -rx wlan0

Here, I  monitor my wlan0 interface, as this is my (laptop) wireless card interface over which I have connectivity to my local network and via which through the the WI-FI router I get connected to the internet.

Below is a snapshot captured showing approximately what is the max network thoroughput from:

Desktop PC -> to my Thinkpad R61 laptop

Using Speedometer to test network thorougput between two network server hosts screenshot Debian Squeeze Linux

As you can see in the shot approximately the maximum network thoroughput is in between:
2.55MB/s min and 2.59MB/S max, the speed is quite low for a 100 MBit local network, but this is normal as most laptop wireless adapters hardly transfer traffic in more than 10 to 20 MBits per sec.

If the same nework thoroughput test is conducted between two machines both connected to a same 100 M/bit switch, the traffic should be at least a 8 MB/sec.

There is something, else to take in consideration that probably makes the provided example network thoroughput measuring a bit inaccurate. The fact that the /dev/zero content is stremed over is slowing down the zeroes sent over network because of the  pipe ( | ) use slows down the stream.

5. Using speedometer to visualize maximum writting speed to a local hard drive on Linux

In the speedometer manual, I've noticed another interesting application of this nifty tool.

speedometer can be used to track and visualize the maximum writing speed a hard disk drive or hard drive partition can support on Linux OS:

A copy paster from the manual text is as follows:

How fast can I write data to my filesystem? (with at least 1GB free)
dd bs=1000000 count=1000 if=/dev/zero of=bigfile &
speedometer bigfile

However, when I tried copy/pasting the example in terminal, to test the maximum writing speed to an external USB hard drive, only dd command was started and speedometer failed to initialize and display graphs of the file creation speed.

I've found a little "hack" that makes the man example work by adding a 3 secs sleep like so:

debian:/media/Expansion Drive# dd bs=1000000 count=1000 if=/dev/zero of=bigfile & sleep 3; speedometer bigfile

Here is a screenshot of the bigfile created by dd and tracked "in real time" by speedometer:

How fast is writting data to local USB expandable hard disk Debian Linux speedometer screenshot

Actually the returned results from this external USB drive are, quite high, the possible reason for that is it is connected to my laptop over an USB protocol verion 3.

6. Using Speedometer to keep an eye on file download in progress

This application of speedometer is mostly useless especially on Linux where it is used as a Desktop.

However in some occasions if files are transferred over ssh or in non interactive FTP / Samba file transfers between Linux servers it can come handy.

To visualize the download and writing speed of lets say FTP transferred .AVI movie (during the actual file transfer) on the download host issue:

# speedometer Download-Folder/What-goes-around-comes-around.avi

7. Estimating approximate time for file transfer

There is another section in the speedometer manual pointing of the program use to calculate the time remaining for a file transfer.

The (man speedometer) provided example text is:

How long it will take for my 38MB transfer to finish?
speedometer favorite_episode.rm $((38*1024*1024))

At first glimpse it hard to understand (like the other manual example). A bit of reasoning and I comprehend what the man author meant by the obscure calculation:

$((38*1024*1024))

This is a formula used in which 38 has to be substituted with the exact file size amount of the transferred file. The author manual used a 38MB file so this is why he put $((38* … in the formula.

I give it a try – (just for the sake to see how it works) with a file with a size of 2500MB, in below two screenshot pictures I show my preparation to copy the file and the actual copying / "real time" transfer tracking with speedometer's status percentage completion bar.

xterm terminal copy file and estimate file copying operation speed on linux with speedometer preparation

Two xterm terminals one is copying a file the other one uses speedometer to estimate the time remaining to complete the file transfer from expansion USB hard drive to my laptop harddrive

 

Monitoring MySQL server queries and debunning performance (slow query) issues with native MySQL commands and with mtop, mytop

Thursday, May 10th, 2012

If you're a Linux server administrator running MySQL server, you need to troubleshoot performance and bottleneck issues with the SQL database every now and then. In this article, I will pinpoint few methods to debug basic issues with MySQL database servers.

1. Troubleshooting MySQL database queries with native SQL commands

a)One way to debug errors and get general statistics is by logging in with mysql cli and check the mysql server status:

# mysql -u root -p
mysql> SHOW STATUS;
+-----------------------------------+------------+
| Variable_name | Value |
+-----------------------------------+------------+
| Aborted_clients | 1132 |
| Aborted_connects | 58 |
| Binlog_cache_disk_use | 185 |
| Binlog_cache_use | 2542 |
| Bytes_received | 115 |
.....
.....
| Com_xa_start | 0 |
| Compression | OFF |
| Connections | 150000 |
| Created_tmp_disk_tables | 0 |
| Created_tmp_files | 221 |
| Created_tmp_tables | 1 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
.....
.....
| Handler_write | 132 |
| Innodb_page_size | 16384 |
| Innodb_pages_created | 6204 |
| Innodb_pages_read | 8859 |
| Innodb_pages_written | 21931 |
.....
.....
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 0 |
| Sort_merge_passes | 0 |
| Sort_range | 0 |
| Sort_rows | 0 |
| Sort_scan | 0 |
| Table_locks_immediate | 4065218 |
| Table_locks_waited | 196 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 0 |
| Threads_cached | 51 |
| Threads_connected | 1 |
| Threads_created | 52 |
| Threads_running | 1 |
| Uptime | 334856 |
+-----------------------------------+------------+
225 rows in set (0.00 sec)

SHOW STATUS; command gives plenty of useful info, however it is not showing the exact list of queries currently processed by the SQL server. Therefore sometimes it is exactly a stucked (slow queries) execution, you need to debug in order to fix a lagging SQL. One way to track this slow queries is via enabling mysql slow-query.log. Anyways enabling the slow-query requires a MySQL server restart and some critical productive database servers are not so easy to restart and the SQL slow queries have to be tracked "on the fly" so to say.
Therefore, to check the exact (slow) queries processed by the SQL server (without restarting it), do
 

mysql> SHOW processlist;
+——+——+—————+——+———+——+————–+——————————————————————————————————+
| Id | User | Host | db | Command | Time | State | Info |
+——+——+—————+——+———+——+————–+——————————————————————————————————+
| 609 | root | localhost | blog | Sleep | 5 | | NULL |
| 1258 | root | localhost | NULL | Sleep | 85 | | NULL |
| 1308 | root | localhost | NULL | Query | 0 | NULL | show processlist |
| 1310 | blog | pcfreak:64033 | blog | Query | 0 | Sending data | SELECT comment_author, comment_author_url, comment_content, comment_post_ID, comment_ID, comment_aut |
+——+——+—————+——+———+——+————–+——————————————————————————————————+
4 rows in set (0.00 sec)
mysql>

SHOW processlist gives a good view on what is happening inside the SQL.

To get more complete information on SQL query threads use the full extra option:

mysql> SHOW full processlist;

This gives pretty full info on running threads, but unfortunately it is annoying to re-run the command again and again – constantly to press UP Arrow + Enter keys.

Hence it is useful to get the same command output, refresh periodically every few seconds. This is possible by running it through the watch command:

debian:~# watch "'show processlist' | mysql -u root -p'secret_password'"

watch will run SHOW processlist every 2 secs (this is default watch refresh time, for other timing use watch -n 1, watch -n 10 etc. etc.

The produced output will be similar to:

Every 2.0s: echo 'show processlist' | mysql -u root -p'secret_password' Thu May 10 17:24:19 2012

Id User Host db Command Time State Info
609 root localhost blog Sleep 3 NULL1258 root localhost NULL Sleep 649 NULL1542 blog pcfreak:64981 blog Query 0 Copying to tmp table \
SELECT p.ID, p.post_title, p.post_content,p.post_excerpt, p.pos
t_date, p.comment_count, count(t_r.o
1543 root localhost NULL Query 0 NULL show processlist

Though this "hack" is one of the possible ways to get some interactivity on what is happening inside SQL server databases and tables table. for administering hundred or thousand SQL servers running dozens of queries per second – monitor their behaviour few times aday using mytop or mtop is times easier.

Though, the names of the two tools are quite similar and I used to think both tools are one and the same, actually they're not but both are suitable for monitoring sql database execution in real time.

As a sys admin, I've used mytop and mtop, on almost each Linux server with MySQL server installed.
Both tools has helped me many times in debugging oddities with sql servers. Therefore my personal view is mytop and mtop should be along with the Linux sysadmin most useful command tools outfit, still I'm sure many administrators still haven't heard about this nice goodies.

1. Installing mytop on Debian, Ubuntu and other deb based GNU / Linux-es

mytop is available for easy install on Debian and across all debian / ubuntu and deb derivative distributions via apt.

Here is info obtained with apt-cache show

debian:~# apt-cache show mytop|grep -i description -A 3
Description: top like query monitor for MySQL
Mytop is a console-based tool for monitoring queries and the performance
of MySQL. It supports version 3.22.x, 3.23.x, 4.x and 5.x servers.
It's written in Perl and support connections using TCP/IP and UNIX sockets.

Installing the tool is done with the trivial:

debian:~# apt-get --yes install mytop
....

mtop used to be available for apt-get-ting in Debian Lenny and prior Debian releases but in Squeeze onwards, only mytop is included (probably due to some licensing incompitabilities with mtop??).

For those curious on how mtop / mytop works – both are perl scripts written to periodically connects to the SQL server and run commands similar to SHOW FULL PROCESSLIST;. Then, the output is parsed and displayed to the user.

Here how mytop running, looks like:

MyTOP showing queries running on Ubuntu 8.04 Linux - Debugging interactively top like MySQL

2. Installing mytop on RHEL and CentOS

By default in RHEL and CentOS and probably other RedHat based Linux-es, there is neither mtop nor mytop available in package repositories. Hence installing the tools on those is only available from 3rd parties. As of time of writting an rpm builds for RHEL and CentOS, as well as (universal rpm distros) src.rpm package is available on http://pkgs.repoforge.org/mytop/. For the sake of preservation – if in future those RPMs disappear, I made a mirror of mytop rpm's here

Mytop rpm builds depend on a package perl(Term::ReadKey), my attempt to install it on CentOS 5.6, returned following err:

[root@cenots ~]# rpm -ivh mytop-1.4-2.el5.rf.noarch.rpm
warning: mytop-1.4-2.el5.rf.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
error: Failed dependencies:
perl(Term::ReadKey) is needed by mytop-1.4-2.el5.rf.noarch

The perl(Term::ReadKey package is not available in CentOS 5.6 and (probably other centos releases default repositories so I had to google perl(Term::ReadKey) I found it on http://rpm.pbone.net/ package repository, the exact url to the rpm dependency as of time of writting this post is:

ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpm

Quickest, way to install it is:

[root@centos ~]# rpm -ivh ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpmRetrieving ftp://ftp.pbone.net/mirror/yum.trixbox.org/centos/5/old/perl-Term-ReadKey-2.30-2.rf.i386.rpmPreparing... ########################################### [100%]
1:perl-Term-ReadKey ########################################### [100%]

This time mytop, install went fine:

[root@centos ~]# rpm -ivh mytop-1.4-2.el5.rf.noarch.rpm
warning: mytop-1.4-2.el5.rf.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6
Preparing... ########################################### [100%]
1:mytop ########################################### [100%]

To use it further, it is the usual syntax:

mytop -u username -p 'secret_password' -d database

CentOS Linux MyTOP MySQL query benchmark screenshot - vpopmail query

3. Installing mytop and mtop on FreeBSD and other BSDs

To debug the running SQL queries in a MySQL server running on FreeBSD, one could use both mytop and mtop – both are installable via ports:

a) To install mtop exec:

freebsd# cd /usr/ports/sysutils/mtop
freebsd# make install clean
....

b) To install mytop exec:

freebsd# cd /usr/ports/databases/mytop
freebsd# make install clean
....

I personally prefer to use mtop on FreeBSD, because once run it runs prompts the user to interactively type in the user/pass

freebsd# mtop

Then mtop prompts the user with "interactive" dialog screen to type in user and pass:

Mtop interactive type in username and password screenshot on FreeBSD 7.2

It is pretty annoying, same mtop like syntax don't show user/pass prompt:

freebsd# mytop
Cannot connect to MySQL server. Please check the:

* database you specified "test" (default is "test")
* username you specified "root" (default is "root")
* password you specified "" (default is "")
* hostname you specified "localhost" (default is "localhost")
* port you specified "3306" (default is 3306)
* socket you specified "" (default is "")
The options my be specified on the command-line or in a ~/.mytop
config file. See the manual (perldoc mytop) for details.
Here's the exact error from DBI. It might help you debug:
Unknown database 'test'

The correct syntax to run mytop instead is:

freebsd# mytop -u root -p 'secret_password' -d 'blog'

Or the longer more descriptive:

freebsd# mytop --user root --pass 'secret_password' --database 'blog'

By the way if you take a look at mytop's manual you will notice a tiny error in documentation, where the three options –user, –pass and –database are wrongly said to be used as -user, -pass, -database:

freebsd# mytop -user root -pass 'secret_password' -database 'blog'
Cannot connect to MySQL server. Please check the:

* database you specified "atabase" (default is "test")
* username you specified "ser" (default is "root")
* password you specified "ass" (default is "")
* hostname you specified "localhost" (default is "localhost")
* port you specified "3306" (default is 3306)
* socket you specified "" (default is "")a
...
Access denied for user 'ser'@'localhost' (using password: YES)

Actually it is interesting mytop, precededed historically mtop.
mtop was later written (probably based on mytop), to run on FreeBSD OS by a famous MySQL (IT) spec — Jeremy Zawodny .
Anyone who has to do frequent MySQL administration tasks, should already heard Zawodny's name.
For those who haven't, Jeremy used to be a head database administrators and developer in Yahoo! Inc. some few years ago.
His website contains plenty of interesting thoughts and writtings on MySQL server and database management
 

Fix Apache [error] [client xx.xxx.xxx.xx] PHP Warning: Unknown: Input variables exceeded 1000. To increase the limit change max_input_vars in php.ini. in Unknown on line 0

Wednesday, August 14th, 2013

I have a busy Linux server with 24 cores each running on ..4 Ghz. Server is configured to server  Apache and MySQL queries and a dozen of high traffic websites are hosted on it. Until just recently server worked fine and since about few days I started getting SMS notifications that server is inaccessible few times a day. To check what's wrong I checked in /var/log/apache2/error.log  and there I found  following error:

[error] [client 95.107.233.2] PHP Warning:  Unknown: Input variables exceeded 1000.

To increase the limit change max_input_vars in php.ini. in Unknown on line 0 referer: http://www.site-domain-name.com/predict/2013-08-10
 

Before I check Apache error.log, I had a guess that ServerLimit of 256 (spawned servers max) is reached so solution would be raise of ServerLimit to more than MaxClients setting defined in /etc/apache2/apache2.conf. After checking /var/log/apache2/error.log I've realized problem is because the many websites hosted on server exceed maximum defined variables possible to assign by libphp in php.ini. maximum possible defined variables before PHP stops servering is set through max_input_vars variable

As I'm running a Debian Squeeze Linux server, here is what is set as default for max_input_vars in /etc/php5/apache2/php.ini:

; How many GET/POST/COOKIE input variables may be accepted
; max_input_vars = 1000

So to fix it in php.ini just raised a bit setting to 1500, i.e.:

max_input_vars = 1500

Though I hit the error on Debian I assume same error occurs on Redhat RPM based (Fedora, CentOS, RHEL Linux) servers.
Hence I assume

max_input_vars = 1500

or higher should fix on those servers too. Looking forward to hear if same error is hit on RedHats.

Enjoy 🙂
 

Client Denied By Server Configuration Linux Apache error solution

Thursday, August 1st, 2013

Client denied by server configuration fix solution Apache feather logo

If you run Apache server on Debian Linux / Ubuntu / CentOS whatever Linux OS and you try to install a new PHP application under lets say /var/www/ getting an error in Apache error.log like:
 

[Wed Jul 31 03:36:21 2013] [error] [client 192.168.10.2] client denied by server configuration: /var/www/vea/index.php, referer: http://192.168.10.9/vea/


This is due to misconfigured AllowOverrides in some of your main configuration files.

So what is causing the error?

Reason is by default in most current Linux distributions Apache is configured to have restrictive policy following the good security practice (Restrictive by default).
Apache is configured by default to not accept AllowOverrides – i.e. AllowOverride None for DocumentRoot /, because there are plenty of administrators who run Apache without having profound understanding leaving it to interpret by default mod_rewrite rules from .htaccess files etc.

To fix this issue, hence you have to add extra configuration for AllowOverride directive for directory giving the err. In this case /vea:

<Directory /var/www/vea/> 
Options -Indexes FollowSymLinks
AllowOverride AuthConfig
FileInfo Order allow,deny
Allow from all
</Directory> 

Above rules are a bit restrictive and will allow only to have .htaccess with only for protecting directory with htaccess passsword for exmpl. – (AuthUserFile, AuthGroupFile, AuthName, AuthType) .htaccess.
-Indexes – instructs /var/www/vea directory listing to be disabled, below two lines:

Order allow, deny
Allow from all

Makes the directory Allowed to be visible by all, however note that it is possible in some of other Apache configuration files to have other rules configured for /vea documentroot /var/www/ which are preventive (Default Deny) – if this is the case just walk through all Apache configs and change where necessary to Allow from all.

In some cases it is possible Web application or Website placed requires AllowOverride All directive instead. If above <Directory>

does not help and you continue to get:

[Wed Jul 31 03:36:21 2013] [error] [client xxx.xxx.xx.x] client denied by server configuration: /var/www/php-application/index.php, referer: http://xxx.xxx.xx.xx/php-application/  

Try setting Directory rules with AllowOverride All ;

<Directory /var/www/php-application/> 
Options -Indexes FollowSymLinks
AllowOverride All
FileInfo Order allow,deny
Allow from all
</Directory> 

Debian / Ubuntu server admins should check closely for AllowOverride rules in files /etc/apache2/conf.d/*

as well as in /etc/apache2/mods-available/*:

Usually there are AllowOverride rules set from files:

/etc/apache2/conf.d/apache2-doc
/etc/apache2/conf.d/localized-error-pages
/etc/apache2/conf.d/security

and also in

/etc/apache2/mods-available/alias.conf
/etc/apache2/mods-available/userdir.conf

On Debian GNU / Linux, very common reason for getting client denied by server configuration is AllowOverride definitions in /etc/apache2/conf.d/security, default AllowOverride there are set to None, i.e.:

<Directory />
AllowOverride None
Order Deny,Allow
Deny from all
</Directory>

If that's the case with you make sure you config rules to become:

# <Directory />
# AllowOverride None
# Order Deny,Allow
# Deny from all
# </Directory>

A very useful command to find where there is occurance of AllowOverride in Apache many configs is:

root@linux:~# cd /etc/apache2
root@linux:/etc/apache2# grep -rli AllowOverride *
apache2.conf
conf.d/localized-error-pages
conf.d/apache2-doc
conf.d/security
mods-available/userdir.conf
mods-available/alias.conf
sites-available/www.website1.com
sites-available/www.website2.com
...


 

Once you did all necessary config Restart Apache:

root@linux:~# /etc/init.d/apache2 restart
....