Posts Tagged ‘kernel’

Linux Bond network interfaces to merge multiple interfaces ISPs traffic – Combine many interfaces NIC into one on Debian / Ubuntu / CentOS / Fedora / RHEL Linux

Tuesday, December 16th, 2014

Bonding Network Traffic
 (link aggregation) or NIC teaming is used to increase connection thoroughput and as a way to provide redundancy for a services / applications in case of some of the network connection (eth interfaces) fail. Networking Bonding is mostly used in large computer network providers (ISPs), infrastructures, university labs or big  computer network accessible infrastructures or even by enthusiatst to run home-server assuring its >= ~99% connectivity to the internet by bonding few Internet Providers links into single Bonded Network interface. One of most common use of Link Aggreegation nowadays is of course in Cloud environments.  

 Boding Network Traffic is a must know and (daily use) skill for the sys-admin of both Small Company Office network environment up to the large Professional Distributed Computing networks, as novice GNU /  Linux sys-admins would probably have never heard it and sooner or later they will have to, I've created this article as a quick and dirty guide on configuring Linux bonding across most common used Linux distributions.

It is assumed that the server where you need network boding to be configured has at least 2 or more PCI Gigabyte NICs with hardware driver for Linux supporting Jumbo Frames and some relatively fresh up2date Debian Linux >=6.0.*, Ubuntu 10+ distro, CentOS 6.4, RHEL 5.1, SuSE etc.

1. Bond Network ethernet interfaces on Debian / Ubutnu and Deb based distributions

To make network bonding possible on Debian and derivatives you need to install support for it through ifenslave package (command).

apt-cache show ifenslave-2.6|grep -i descript -A 8
Description: Attach and detach slave interfaces to a bonding device
 This is a tool to attach and detach slave network interfaces to a bonding
 device. A bonding device will act like a normal Ethernet network device to
 the kernel, but will send out the packets via the slave devices using a simple
 round-robin scheduler. This allows for simple load-balancing, identical to
 "channel bonding" or "trunking" techniques used in switches.
 The kernel must have support for bonding devices for ifenslave to be useful.
 This package supports 2.6.x kernels and the most recent 2.4.x kernels.


apt-get –yes install ifenslave-2.6


Bonding interface works by creating a "Virtual" network interface on a Linux kernel level, it sends and receives packages via special
slave devices using simple round-robin scheduler. This makes possible a very simple network load balancing also known as "channel bonding" and "trunking"
supported by all Intelligent network switches

Below is a text diagram showing tiny Linux office network router configured to bond ISPs interfaces for increased thoroughput:


 |         (eth0)
ISP Router/Firewall (eth1)
                              | -----+------ Server 1 (Debian FTP file server w/ eth0 & eth1)
      +------------------+ --- |
      | Gigabit Ethernet       |------+------ Server 2 (MySQL)
      | with Jumbo Frame       |
      +------------------+     |------+------ Server 3 (Apache Webserver)
                               |------+-----  Server 4 (Squid Proxy / Qmail SMTP / DHCP)
                               |------+-----  Server 5 (Nginx CDN static content Webserver)
                               |------+-----  WINDOWS Desktop PCs / Printers & Scanners, Other network devices 


Next to configure just installed ifenslave Bonding  

vim /etc/modprobe.d/bonding.conf

alias bond0 bonding
  options bonding mode=0 arp_interval=100 arp_ip_target=,,,,


  1. mode=0 : Set the bonding policies to balance-rr (round robin). This is default mode, provides load balancing and fault tolerance.
  2. arp_interval=100 : Set the ARP link monitoring frequency to 100 milliseconds. Without option you will get various warning when start bond0 via /etc/network/interfaces
  3. arp_ip_target=,, … : Use the (router ip) and IP addresses to use as ARP monitoring peers when arp_interval is > 0. This is used determine the health of the link to the targets. Multiple IP addresses must be separated by a comma. At least one IP address must be given (usually I set it to router IP) for ARP monitoring to function. The maximum number of targets that can be specified is 16.

Next to make bonding work its necessery to load the bonding kernel module:

modprobe -v bonding mode=0 arp_interval=100 arp_ip_target=,,,,


Loading the bonding module should spit some good output in /var/log/messages (check it out with tail -f /var/log/messages)

Now to make bonding active it is necessery to reload networking (this is extremely risky if you don't have some way of Console Web Java / VPN Access such as IPKVM / ILO / IDRAC), so reloading the network be absolutely sure to either do it through a cronjob which will automatically do the network restart with new settings and revert back to old configuration whether network is inaccessible or assure physical access to the server console if the server is at your disposal.

Whatever the case make sure you backup:

 cp /etc/network/interfaces /etc/network/interfaces.bak

vim /etc/network/interfaces

############ WARNING ####################
# You do not need an "iface eth0" nor an "iface eth1" stanza.
# Setup IP address / netmask / gateway as per your requirements.
auto lo
iface lo inet loopback
# The primary network interface
auto bond0
iface bond0 inet static
    slaves eth0 eth1
    # jumbo frame support
    mtu 9000
    # Load balancing and fault tolerance
    bond-mode balance-rr
    bond-miimon 100
    bond-downdelay 200
    bond-updelay 200


As you can see from config there are some bond specific configuration variables that can be tuned, they can have positive / negative impact in some cases on network thoroughput. As you can see bonding interfaces has slaves (this are all other ethXX) interfaces. Bonded traffic will be available via one single interface, such configuration is great for webhosting providers with multiple hosted sites as usually hosting thousand websites on the same server or one single big news site requires a lot of bandwidth and of course requires a redundancy of data (guarantee it is up if possible 7/24h.

Here is what of configs stand for

  • mtu 9000 : Set MTU size to 9000. This is related to Jumbo Frames.
  • bond-mode balance-rr : Set bounding mode profiles to "Load balancing and fault tolerance". See below for more information.
  • bond-miimon 100 : Set the MII link monitoring frequency to 100 milliseconds. This determines how often the link state of each slave is inspected for link failures.
  • bond-downdelay 200 : Set the time, t0 200 milliseconds, to wait before disabling a slave after a link failure has been detected. This option is only valid for the bond-miimon.
  • bond-updelay 200 : Set the time, to 200 milliseconds, to wait before enabling a slave after a link recovery has been detected. This option is only valid for the bond-miimon.
  • dns-nameservers : Use as dns server.
  • dns-search : Use as default host-name lookup (optional).

To get the best network thorougput you might want to play with different bounding policies. To learn more and get the list of all bounding policies check out Linux ethernet Bounding driver howto

To make the new bounding active restart network:

/etc/init.d/networking stop
sleep 5;
/etc/init.d/networking start

2. Fedora / CentOS RHEL Linux network Bond 

Configuring eth0, eth1, eth2 into single bond0 NIC network virtual device is with few easy steps:

a) Create following bond0 configuration file:

vim /etc/sysconfig/network-scripts/ifcfg-bond0



b) Modify ifcfg-eth0 and ifcfg-eth0 files /etc/sysconfig/network-scripts/

– Edit ifcfg-eth0

vim /etc/sysconfig/network-scripts/ifcfg-eth0


– Edit ifcfg-eth1

vim /etc/sysconfig/network-scripts/ifcfg-eth1


c) Load bond driver through modprobe.conf

vim /etc/modprobe.conf

alias bond0 bonding
options bond0 mode=balance-alb miimon=100

Manually load the bonding kernel driver to make it affective without server reboot:

modprobe bonding

d) Restart networking to load just configured bonding 

service network restart

3. Testing Bond Success / Fail status

Periodically if you have to administrate a bonded interface Linux server it is useful to check Bonds Link Status:

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:0b:d6:6c:8f

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:1e:0b:d6:6c:8c

To check out which interfaces are bonded you can either use (on older Linux kernels)

/sbin/ifconfig -a

If ifconfig is not returning IP addresses / interfaces of teamed up eths, to check NICs / IPs:

/bin/ip a show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet brd scope host lo
    inet brd scope host secondary lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether 00:1e:0b:d6:6c:8c brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP qlen 1000
    link/ether 00:1e:0b:d6:6c:8c brd ff:ff:ff:ff:ff:ff
7: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:1e:0b:d6:6c:8c brd ff:ff:ff:ff:ff:ff
    inet brd scope global bond0
    inet brd scope global secondary bond0:7156web
    inet6 fe80::21e:bff:fed6:6c8c/64 scope link
       valid_lft forever preferred_lft forever

In case of Bonding interface failure you will get output like:

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:xx:yy:zz:tt:31
Slave Interface: eth1
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:xx:yy:zz:tt:30

Failure to start / stop bonding is also logged in /var/log/messages so its a good idea to check there too once launched:

tail -f /var/log/messages
Dec  15 07:18:15 nas01 kernel: [ 6271.468218] e1000e: eth1 NIC Link is Down
Dec 15 07:18:15 nas01 kernel: [ 6271.548027] bonding: bond0: link status down for interface eth1, disabling it in 200 ms.
Dec  15 07:18:15 nas01 kernel: [ 6271.748018] bonding: bond0: link status definitely down for interface eth1, disabling it

On bond failure you will get smthing like:

Dec  15 04:19:15 micah01 kernel: [ 6271.468218] e1000e: eth1 NIC Link is Down
Dec  15 04:19:15 micah01 kernel: [ 6271.548027] bonding: bond0: link status down for interface eth1, disabling it in 200 ms.
Dec  15 04:19:15 micah01 kernel: [ 6271.748018] bonding: bond0: link status definitely down for interface eth1, disabling it

4. Adding removing interfaces to the bond interactively

You can set the mode through sysfs virtual filesystem with:

echo active-backup > /sys/class/net/bond0/bonding/mode

If you want to try adding an ethernet interface to the bond, type:

echo +ethN > /sys/class/net/bond0/bonding/slaves

To remove an interface type:

echo -ethN > /sys/class/net/bond0/bonding/slaves

In case if you're wondering how many bonding devices you can have, well the "sky is the limit" you can have, it is only limited by the number of NIC cards Linux kernel / distro support and ofcourse how many physical NIC slots are on your server.

To monitor (in real time) adding  / removal of new ifaces to the bond use:

watch -n 1 ‘cat /proc/net.bonding/bond0′


How to determine which processes make most writes on the hard drive in GNU / Linux using kernel variable

Thursday, November 13th, 2014

In Linux there are plenty of tools to measure input / ouput – read / write server bottlenecks. Just to mention a few such are, the native part of all Linux distributions IOSTAT – which is a great tool to measure hard disk bottlenecks. However as iostat requires certain sysadmin skills for novice sys-admins, there is also ofcourse more interactive tools such as DSTAT or even better GLANCE which monitors not only disk writes but memory use, CPU load and Network use.

This tools can help you measure which processes are writting most (a lot) to hard disk drive but there is another quick and efficient way to track disk i/o by directly using the Linux kernel this is done via kernel parameter :


To enable block_dump kernel logging:

echo 1 > /proc/sys/vm/block_dump

To later track in real time output from kernel interactively on which running process calling the kernel is writing to server hard drive

tail -f /var/log/syslog

The output  looks like so:

Nov 13 12:25:51 pcfreak kernel: [1075037.701056] kjournald(297): WRITE block 482293496 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701059] kjournald(297): WRITE block 482293504 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701062] kjournald(297): WRITE block 482293512 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701066] kjournald(297): WRITE block 482293520 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701069] kjournald(297): WRITE block 482293528 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701072] kjournald(297): WRITE block 482293536 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.702824] kjournald(297): WRITE block 482293544 on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.219288] apache2(3377): dirtied inode 3571740 (_index.html.old) on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.436133] mysqld(22945): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.436826] mysqld(22945): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.662832] mysqld(22945): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.663297] mysqld(22945): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.817120] apache2(3377): dirtied inode 3571754 (_index.html) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.819968] apache2(3377): dirtied inode 3571740 (_index.html_gzip) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.820016] apache2(3377): dirtied inode 3571730 (?) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075040.491378] mysqld(22931): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075040.492309] mysqld(22931): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:54 pcfreak kernel: [1075041.551513] apache2(3377): dirtied inode 1474706 (_index.html_gzip.old) on sda1
Nov 13 12:25:54 pcfreak kernel: [1075041.551566] apache2(3377): dirtied inode 1474712 (_index.html.old) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.769036] mysqld(22941): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.769804] mysqld(22941): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.985857] apache2(3282): dirtied inode 4063282 (data_9d97a7f62d54bc5fd791fba3245ba591-SMF-modSettings.php) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.987460] apache2(3282): dirtied inode 29010186 (data_9d97a7f62d54bc5fd791fba3245ba591-SMF-permissions–1.php) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.988357] flush-8:0(289): WRITE block 51350632 on sda1

Using the kernel method to see which processes are stoning your server is great way especially for servers without connectivity to the Internet where you have no possibility to install sysstat package (contaning iostat),  dstat or glance.
Thanks to Marto's blog for  this nice hack.

Install VMWare tools on Debian and Ubuntu Linux – Enable VMWare Fullscreen and copy paste between OS host and Virtual machine

Wednesday, May 28th, 2014


If you need to use Virtual Machine to run some testing on heterogenous Operating Systems and you have chosen VMWare as a Virtual Machine. You will soon notice some of Virtual Machines functionality like copy between host operating system and Virtual Machine, true fullscreen mode and most importantly Copy paste between your host operating system and VMWare is not working. I'm not too much into Virtualization these days so for me it was truely shocking that a proprietary software like VMWare, claimed to be the best and most efficient Virtual Machine nowadays is not supporting copy / paste, fullscreen and copy between host and guest OS.  For those arguing why I'm using VMWare at all as it is proprietary and there is already free software Virtual Machines like QEMU and Oracle's VirtualBox its simply because now I have the chance to install and use VMWare 9 Enterprise on my work place at HP with a free Corporate license – in other words I'm using VMWare just for the sake of educating myself and would always recommend VirtualBox for those looking for good substitute free alternative to VMWare.

Before trying out VMWare, I tried Virtualbox to emulate Linux on my HP work PC running Windows with VirtualBox I was having issues with keyboard not working (because of lack of support of USB, no full screen support and lack of copy / paste between OS-es), I've just recently understood this is not because Virtualbox is bad Virtualization solution but because I forgot to install VirtualBox Oracle VM VirtualBox Extension Pack which allows support for USB, enables copy paste and full screen support. The equivalent to Virtualbox Oracle VM VirtualBox in VMWare world is called VMWare-Tools and once the guest operating system is installed inside VMWare VM, its necessary to install vmware-tools to enable better screen resolution and copy paste.

In Windows Virtual Machine installation of vmware-tools is pretty straight forward you go through VMWare's menus


VM -> Install Vmware-tools


follow the instructions and you're done, however as always installing VMWare-tools on Linux is little bit more complicated you need to run few commands from Linux installed inside the Virtual Machine to install vmware-tools. Here is how vmware-tools is installed on Debian / Ubuntu / Linux Mint and rest of Debian based operating systems:

  1. Install Build essentials and gcc You need to have this installed some developer tools as well as GCC compiler in order for the vmware-tools to compile a special Linux kernel module which enables extra support (integration) between the VMWare VM and the installed inside VM Linux distro

apt-get install --yes build-essential gcc

2. Install appropriate Linux headers corresponding to current Linux OS installed kernel

apt-get install --yes linux-headers-$(uname -r)

3. Mount CD (Virtual) Content to obtain the vmware-tools version for your Linux

Be sure to have first checked from VMWare menus on menus VM -> Intall Vmware Tools
This step is a little bit strange but just do it without too much questioning …

mount /dev/cdrom /mnt/
umount /media/cdrom0/
mount /media/cdrom
mount /dev/sr0 /mnt/cdrom/
mount /dev/sr0 /mnt/


Note that /dev/sr0, might already be mounted and sometimes it might be necessary to unmount it first (don't remember exactly if I unmounted it or not)

4. Copy and Untar VMwareTools-9.2.0-799703.tar.gz

cp -rpf /media/cdrom/VMwareTools-9.2.0-799703.tar.gz /tmp/
cd /tmp/
tar -zxvvf VMwareTools-9.2.0-799703.tar.gz

5. Run vmware-tools installer

cd vmware-tools-distrib/

You will be asked multiple questions you can safely press enter to answer with default settings to all settings, hopefully if all runs okay this will make VMWare Tools installed

Creating a new VMware Tools installer database using the tar4 format.
Installing VMware Tools.
In which directory do you want to install the binary files?
What is the directory that contains the init directories (rc0.d/ to rc6.d/)?
What is the directory that contains the init scripts?
In which directory do you want to install the daemon files?
In which directory do you want to install the library files?
The path "/usr/lib/vmware-tools" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
In which directory do you want to install the documentation files?
The path "/usr/share/doc/vmware-tools" does not exist currently. This program
is going to create it, including needed parent directories. Is this what you
want? [yes]
The installation of VMware Tools 9.2.0 build-799703 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command: "/usr/bin/".
Before running VMware Tools for the first time, you need to configure it by
invoking the following command: "/usr/bin/". Do you want
this program to invoke the command for you now? [yes]
Making sure services for VMware Tools are stopped.
Stopping VMware Tools services in the virtual machine:
Guest operating system daemon: done
Unmounting HGFS shares: done
Guest filesystem driver: done
[EXPERIMENTAL] The VMware FileSystem Sync Driver (vmsync) is a new feature that creates backups of virtual machines. Please refer to the VMware Knowledge Base for more details on this capability. Do you wish to enable this feature?
Before you can compile modules, you need to have the following installed…
kernel headers of the running kernel
Searching for GCC…
Detected GCC binary at "/usr/bin/gcc-4.6".
The path "/usr/bin/gcc-4.6" appears to be a valid path to the gcc binary.
Would you like to change it? [no]
Searching for a valid kernel header path…
Detected the kernel headers at "/lib/modules/3.2.0-4-amd64/build/include".
The path "/lib/modules/3.2.0-4-amd64/build/include" appears to be a valid path
to the 3.2.0-4-amd64 kernel headers.
Would you like to change it? [no]
The vmblock enables dragging or copying files between host and guest in a
Fusion or Workstation virtual environment. Do you wish to enable this feature?
[no] yes
make: Leaving directory `/tmp/vmware-root/modules/vmblock-only'

No X install found.
Creating a new initrd boot image for the kernel.
update-initramfs: Generating /boot/initrd.img-3.2.0-4-amd64
Checking acpi hot plug done
Starting VMware Tools services in the virtual machine:
Switching to guest configuration: done
VM communication interface: done
VM communication interface socket family: done
File system sync driver: done
Guest operating system daemon: done
The configuration of VMware Tools 8.6.10 build-913593 for Linux for this
running kernel completed successfully.
You must restart your X session before any mouse or graphics changes take
You can now run VMware Tools by invoking "/usr/bin/vmware-toolbox-cmd" from the
command line or by invoking "/usr/bin/vmware-toolbox" from the command line
during an X server session.
To enable advanced X features (e.g., guest resolution fit, drag and drop, and
file and text copy/paste), you will need to do one (or more) of the following:
1. Manually start /usr/bin/vmware-user
2. Log out and log back into your desktop session; and,
3. Restart your X session.
–the VMware team
Found VMware Tools CDROM mounted at /mnt. Ejecting device /dev/sr0 …

.To make sure vmware-tools compiled modules are loaded into Linux kernel inside VM, restart the Virtual Machine. Once Linux boots again and you login to gnome-terminal to check what is vmware-tools status (e.g. if properly loaded) run:

service vmware-tools status
vmtoolsd is running


This method of installing works on Debian 7 (Wheezy) but same steps should work on any Ubuntu and rest of Debian derivatives. For Redhat (RPM) based Linux distributions to install vmware-tools after mounting cdrom drive following above instructions you will have an rpm package instead of .tar.gz archive so all you have to do is install the rpm, e.g. launch smth. like:

rpm -Uhv /mnt/cdrom/VMwareTools-9.2.0-799703.i386.rpm
Cheers 😉

How to detect failing storage LUN on Linux – multipath

Wednesday, April 16th, 2014


If you login to server and after running dmesg – show kernel log command you get tons of messages like:

# dmesg


end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0x00010000
end_request: I/O error, dev sda, sector 0
sd 0:0:0:2: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0x00010000
end_request: I/O error, dev sda, sector 0
sd 0:0:0:2: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdc, sector 0
sd 0:0:0:1: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdb, sector 0
sd 0:0:0:0: SCSI error: return code = 0x00010000
end_request: I/O error, dev sda, sector 0


In /var/log/messages there are also log messages present like:

# cat /var/log/messages


Apr 13 09:45:49 sl02785 kernel: end_request: I/O error, dev sda, sector 0
Apr 13 09:45:49 sl02785 kernel: klogd 1.4.1, ———- state change ———-
Apr 13 09:45:50 sl02785 kernel: sd 0:0:0:1: SCSI error: return code = 0x00010000
Apr 13 09:45:50 sl02785 kernel: end_request: I/O error, dev sdb, sector 0
Apr 13 09:45:55 sl02785 kernel: sd 0:0:0:2: SCSI error: return code = 0x00010000
Apr 13 09:45:55 sl02785 kernel: end_request: I/O error, dev sdc, sector 0



This is a sure sign something is wrong with SCSI hard drives or SCSI controller, to further debug the situation you can use the multipath command, i.e.:

multipath -ll | grep -i failed
_ 0:0:0:2 sdc 8:32 [failed][faulty]
_ 0:0:0:1 sdb 8:16 [failed][faulty]
_ 0:0:0:0 sda 8:0 [failed][faulty]


As you can see all 3 drives merged (sdc, sdb and sda) to show up on 1 physical drive via the remote network connectedLUN to the server is showing as faulty. This is a clear sign something is either wrong with 3 hard drive members of LUN – (Logical Unit Number) (which is less probable)  or most likely there is problem with the LUN network  SCSI controller. It is common error that LUN SCSI controller optics cable gets dirty and needs a physical clean up to solve it.

In case you don't know what is storage LUN? – In computer storage, a logical unit number, or LUN, is a number used to identify a logical unit, which is a device addressed by the SCSI protocol or protocols which encapsulate SCSI, such as Fibre Channel or iSCSI. A LUN may be used with any device which supports read/write operations, such as a tape drive, but is most often used to refer to a logical disk as created on a SAN. Though not technically correct, the term "LUN" is often also used to refer to the logical disk itself.

What LUN's does is very similar to Linux's Software LVM (Logical Volume Manager).

Linux: Add routing from different class network A (192.168.1.x) to network B (192.168.10.x) with ip route command

Friday, July 12th, 2013

adding routing from one network to other linux with ip route

I had a Linux router which does NAT for a local network located behind a CISCO router receiving internet via its WAN interface routing traffic  to Linux with IP The Linux router has few network interfaces and routes traffic for networks; and Another Linux with IP had to talk to (because it was necessary to be able access  ISCO's router web interface accessible via a local network interface with IP ( Access to wasn't possible from because routing on NAT-ting Linux ( to network was missing. To make Linux communicate with,  had to add following routing rules with ip command on both the Linux with IP and Linux host behind NAT (

1. On Server ( run in root shell and add to /etc/rc.local

# /sbin/ip r add via
And then copy paste same line before exit 0 in /etc/rc.local

Its good idea always to check routing, after adding anything new, here is mine:

# ip r show dev eth0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth1  proto kernel  scope link  src
default via dev eth1 

2. And also on Second Linux host ( 

# /sbin/ip r add via
To make routing permanent again paste in /etc/rc.local before exit 0

After above rules, I can normally ping and access hosts on class C network  from

Fix FTP client error 425 Unable to build data connection: No route to host (Cause and Solution)

Monday, July 8th, 2013

ftp fix client 425 unable to-build data connection no route to host Linux
I just configured new dedicated server. One of requirement was dedicated server to support connections via FTP transfer protocol and few users to have access via it.
I added users with required permissions to directory structure and went on to test it with Linux ftp command, i.e.:


hipo@pcfreak:~$ ftp remote-host-name

Connected to
220 ProFTPD 1.3.3a Server (Matusala) []
Name (Matusala:hipo): testing-user
331 Password required for testing-user
230 User testing-user logged in

Remote system type is UNIX.
Using binary mode to transfer files.


ftp> ls


200 PORT command successful
425 Unable to build data connection: No route to host

ftp> exit


221 Goodbye.


As you see from above FTP paste even simple commands like "ls" failed to work with error:

425 Unable to build data connection: No route to host

I thought somehow I have configured on server some paranoid firewall, so tested same connection with iptables rules flushed, e.g.: matusala:~# iptables -F
matusala:~# iptables -t nat -F

Even after flushing firewall message persisted, so I googled around to see the what causes the error. The same error was hit by many users, and as I read, understand what causes is FTP server host is located behind some DMZ or firewall (as it was not my firewall I suppose, the dedicated provider has some firewall rules which are blocking FTP traffic on standard FTP TCP / UDP ports, port 20 and 21 ).
Fix is to enable in Linux kernel ip_nat_ftp or in newer Linux kernels module nf_nat_ftp:

matusala:~# modprobe ip_nat_ftp
matusala:~# modprobe nf_nat_ftp

To make ip_nat_ftp load permanently on Debian and Ubuntu servers:

matusala:~# echo 'ip_nat_ftp' >> /etc/modules
matusala:~# echo 'nf_nat_ftp' >> /etc/modules

One important note to make here is if you're testing the connection between two Linux servers it is possible the server from which you're running the client to have missing nf_nat_ftp loaded, so if error persist and you're testing remote FTP server from local ftp client on linux load same modules on localhost and you're done 🙂

Linux webcam take pictures from tty console or terminal / How to make pictures of yourself using plain console and web-camera

Saturday, December 15th, 2012

how to take webCAM console / terminal pictures on gnu linux (Debian, Ubuntu, Fedora)

I'm a great command line enthusiast, I share the believe of many other command line geeks thinking keyboard is the quickest way to access a computer. Historically keys were first and mouse second and I think there is definitely a good reason for that. Thus today I was curious if it is possible to take pictures from my external web-camera on my Debian GNU / Linux? I did a quick research and this little article springed out as result.

The answer is YES! It is possible and besides that there are many ways to take a webcamera picture using the console; Lastly it is very easy to achieve even for novice Linux buddies 😉 My little research on the topic show me there are 4 straightforward ways one can use to use his extended or embedded WebCam to take pictuers – using (vlc, mplayer, camshot, fswebcam and ffmpeg).

1. Taking a webcam picture using vlc

Invoke vlc with following arguments:

# vlc -I dummy v4l2:///dev/video0 --video-filter scene --no-audio --scene-path /home/hipo --scene-prefix webcam-taken-picture-prefix --scene-format png vlc://quit --run-time=1

I've prepared a little wrapper script, for the sake of simplifying the long and hard to remember vlc options. Below is the script;

# This little script will take picture whilst in gnome-terminal / mlterm or any console tty
# As program uses vlc you need to have vlc properly configured and installed
# as well as the webcam video be properly working (detected by Linux kernel)
# licensed under GPLv2 script modified by hip0 14.12.2012
# Path where to store taken snapshots
# Device locatation of webcam many webcams have default device in /dev/video0
# Stored grabbed picture filename prefix
# gets the current date and adds to set filename prefix
date_cur=$(date +%k_%d_%m_%Y|sed -e 's/^ *//');
vlc -I dummy v4l2://$WEBCAM_DEV --video-filter scene --no-audio --scene-path $STORE_PATH --scene-prefix $FILE_NAME_PREF.$date_cur --scene-format png vlc://quit --run-time=1
echo "WebCam picture taken and stored in $STORE_PATH/$FILE_NAME_PREF.$date_cur*.png";
echo '';
echo "To view picture in Gnome with Eye of GNOME type: eog $STORE_PATH/$FILE_NAME_PREF.$date_cur*.png";

You can also download copy of the script here.

As you see the script uses, vlc's (dummy interface), and –video-filter-scene option to make the snapshot. The script can be stored in let's say /usr/local/bin/ and aliased through ~/.bashrc with some short alias, i.e.:

alias console-picture='/usr/local/bin/'

Then at any time, when you run console-picture you will have a short way to make pictures of your room, your friends or whatever needed. One good application of script is whether you're in coffee with friends and you want to take a snapshot of them without them realizing (assuming, the webcam is embedded) 🙂

Another great application is whether you want to take a snapshot of the WebCam, from another shell script or little application using dialog ncurses interfaces etc.

Even just for the sake of fun it is so nice to take a picture from webcam, whether in plain tty console bash shell 🙂

One small note to make here is should be run as non-root user (for security reasons vlc developers made smartly VLC this way), running it as root drops an err:

>VLC is not supposed to be run as root. Sorry.
If you need to use real-time priorities and/or privileged TCP ports
you can use /usr/bin/vlc-wrapper (make sure it is Set-UID root and
cannot be run by non-trusted users first).

By default, vlc resolution used is the automatically set to the maximum supported to the camera, with mine this is 640×480 SRGB
The quality of pictures taken is a bit low but my camera is a cheap one and even with some GUI program snapshot taking programs like GNOME's cheese, taken pictures are with low quality (though I think the brightnes of the ones taken with vlc is a bit poorer than the ones done with cheese).

Happily it is possible to correct picture brightness and lightning with v42l-ctl (v42l-ctl is not installed by default and on Debian you will have to install deb pack v4l-utils), e.g.:

apt-get install --yes v4l-utils

Further, check out the possible options available with:

v4l2-ctl -L - (list all possible options)

and to set a concrete option do:

v4l2-ctl -c <options>=<value> (set an option)

I don't have a BSD (FreeBSD, NetBSD, OpenBSD) at hand, but with a working supported Webcam, correct location to the webcam /dev/  and installed VLC from ports :

vlc -I dummy v4l2:///dev/video0--video-filter scene --no-audio --scene-path $STORE_PATH --scene-prefix $FILE_NAME_PREF.$date_cur --scene-format png vlc://quit --run-time=1

should be working fine as well.
If someone has access to a BSD with a working installed webcam, please test it and drop a comment to confirm if working …

2. Creating pictures from WebCamera using mplayer cmd

Theoretically mplayer, should be able to take snapshots from the Cam with:

mplayer -vo png -frames 1 tv://

There is possibility to pass output webcam picture (resolution) dimensions too:

mplayer tv:// -tv driver=v4l:device=/dev/video0:width=320:height=240:outfmt=rgb24 -frames 1 -vo jpeg


With my "NoName" (Eltron Technology) webcam the produced images were filled up with solid green color  (maybe due to bug of my webcam used driver). Normally it should be working; I've seen many posts around claiming using both of above cmd lines to produce pictures normally, but not for me.

3. Making pictures with WebCamera (camshot) console tool

I've seen around also another tiny tool (camshot) especially written to take pictures from webcam. The tool is available only to be compiled from source (whether source is fetched through Subversion repo (SVN)). I took a minute to test it as well, e.g.:


hipo@noah:~Desktop$ svn checkout camshot-read-only

hipo@noah:~/Desktop/camshot-read-only$ make
hipo@noah:~/Desktop/camshot-read-only$ ls
arguments.c  arguments.o  camera.h  camshot  image.h  main.c  Makefile  shmem.h  shmem_test.c
arguments.h  camera.c     camera.o  image.c  image.o  main.o  shmem.c   shmem.o

hipo@noah:~Desktop/camshot-read-only$ ./camshot
Letting the camera automaticaly adjust the picture:..........Done.
Command (h for help): h

    x    Capture a picture from camera.
    h    Prints this help.
    q    Quits the program.

Command (h for help): x
Command (h for help): q


Don't know why, but for me camshot did not produce, any output picture from webcam. Maybe my Webcam which is a cheap (all OS) compatible one is not detected fine by the tool? As you see from above help output there are not many options so it is definitely something with webcam detection or just it needs some kind of little "hack" in the source to make it working, I was lazy to further investigate so I leave it.

4. Making pictures from terminal using fswebcam

fswebcam is not so popular as vlc and mplayer, but is existent from default repostiries on both Debian and Ubuntu Linuces. Here is it how it is described when pkg info requested with apt-cache:

apt-cache show fswebcam | grep -i descrip -A 8
Description: Tiny and flexible webcam program
 Fswebcam is a tiny and flexible webcam command-line program for capturing
 images from a V4L1/V4L2 device. It accepts a number of formats, can skip
 the first (possibly bad) frames before performing the actual capture, and
 can perform simple manipulation on the captured image, such as resizing,
 averaging multiple frames or overlaying a caption or an image.
Tag: implemented-in::c, role::program

To use it first install it with apt-get or yum (yes it is available also for RedHat based Linux distros via yum).
Depending whether on Debian or Fedora etc. do:

apt-get install --yes fswebcam


yum -y install fswebcam

fswebcam's syntax is much easier than all of rest cmd tools available around; to create picture from webcam;

# fswebcam -r 640x480 --jpeg 85 -D 1 web-cam-shot.jpg
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Delaying 1 seconds.
--- Capturing frame...
Captured frame in 0.00 seconds.
--- Processing captured image...
Setting output format to JPEG, quality 85
Unable to load font 'luxisr': Could not find/open font
Disabling the the banner.
Writing JPEG image to 'web-cam-shot.jpg'.

I liked supports, saving in multiple formats, can set resolution and is probably the number 1 choice for anyone looking for high level of customization of cam taken picture.. Saying this I think fswebcam is definitely the tool of choice as it is written with the one and only aim to take webcam pictures from console.

5. Capturing picture from Webcam using ffmpeg

With ffmpeg, there are plenty of things possible;
Just to mention few interesting ones, I've written about earlier, ffmeg is capable of;

convert .OGG vorbis to MP3 
convert .FLV to .AVI and .AVI to .FLV
convert .AVI .MP4 and .FLV to OGG Vorbis (Free Format)
convert .OGG video to .FLV Video
extract sounds / music from .FLV to a MP3 / MP4
Add .SRT and .SUB files subtitles to Flash Videos

Along with all this, interestingly ffmpeg can get content using command line from WebCamera (nomatter if it's a VIDEO stream or just a Picture snapshot).

The syntax to take a picture with it is:

ffmpeg -f video4linux2 -i /dev/v4l/by-id/usb-Etron_Technology__Inc._USB2.0_Camera-video-index0 -vframes 1 output-picture.jpeg

The precise /dev/(v4l – video 4 linux) assigned to different cameras will differ so in order to find what kind of /dev, to use ls it:

# ls -al /dev/v4l/by-id/*

lrwxrwxrwx 1 root root 12 Dec 14 22:40 /dev/v4l/by-id/usb-Etron_Technology__Inc._USB2.0_Camera-video-index0 -> ../../video0

The picture resolution taken on my Eltron Technology Webcam is same like with vlc – the cam optimum 640×480, the quality and brightness gamma is also identical to pics taken using VLC.
Therefore if you're wondering if one tool, might make a better pictures from command line than the other the answer, according to my tests is they produce identical quality and all can be customized easily for different set of resolution. It is possible thought, this is not so with other Web Cam models, if you happen to read this post and take the time to try taking pictures with 5 methods and some of the 5 progs is making superior pictures, please drop a comment with the tool you used and the WebCam exact version as detected in dmesg or lsusb

As a sort of Outtro, from purely functional / usability point of view I think fswebcam is probably be the tool of choice for mostly all as it is most simple, easily customizable and especially crafted for creating webcam console shots. That's all Enjoy, taking pics from GUI terminal or console 🙂 

P.S. – I know there are plenty of people who have written on the topic, so this article is nothing new under the sun, but as I couldn't find a post synthesizing in one all of the 5 methods I've come up with this little article. Feedback is mostly welcome
Happy picturing 😉

Disabling sound kernel modules on Debian and Ubuntu GNU / Linux servers

Friday, October 19th, 2012

First step is to list modules related to sound (snd):

root@pcfreak:/var/www# lsmod|grep -i snd
snd_hda_codec_realtek 235234 1
snd_hda_intel 20035 0
snd_hda_codec 53940 2 snd_hda_codec_realtek,snd_hda_intel
snd_hwdep 5220 1 snd_hda_codec
snd_pcm_oss 32415 0
snd_mixer_oss 12478 1 snd_pcm_oss
snd_pcm 60151 3 snd_hda_intel,snd_hda_codec,snd_pcm_oss
snd_seq_midi 4256 0
snd_rawmidi 15323 1 snd_seq_midi
snd_seq_midi_event 4628 1 snd_seq_midi
snd_seq 41281 2 snd_seq_midi,snd_seq_midi_event
snd_timer 15502 2 snd_pcm,snd_seq
snd_seq_device 4493 3 snd_seq_midi,snd_rawmidi,snd_seq
snd 45998 11
soundcore 4566 1 snd
snd_page_alloc 6217 2 snd_hda_intel,snd_pcm

Then snd modules could be removed from current live system, for my Intel RealTek SoundBlaster, I had to remove following modules in (remove) order as follows:

root@pcfreak:/var/www# rmmod soundcore
ERROR: Module soundcore is in use by snd
root@pcfreak:/var/www# rmmod snd_hda_codec_realtec
ERROR: Module snd_hda_codec_realtec does not exist in /proc/modules
root@pcfreak:/var/www# rmmod snd_hda_intel
root@pcfreak:/var/www# rmmod snd_hda_codec
ERROR: Module snd_hda_codec is in use by snd_hda_codec_realtek
root@pcfreak:/var/www# rmmod snd_hda_codec_realtek
root@pcfreak:/var/www# rmmod snd_hda_codec
root@pcfreak:/var/www# rmmod snd_pcm_oss
root@pcfreak:/var/www# rmmod snd_seq_midi
root@pcfreak:/var/www# rmmod snd_rawmidi
root@pcfreak:/var/www# rmmod snd_seq_midi_event
root@pcfreak:/var/www# rmmod snd_seq
root@pcfreak:/var/www# rmmod snd_timer
ERROR: Module snd_timer is in use by snd_pcm
root@pcfreak:/var/www# rmmod snd_seq_device
root@pcfreak:/var/www# rmmod snd_pcm
root@pcfreak:/var/www# rmmod snd_seq_device
ERROR: Module snd_seq_device does not exist in /proc/modules
root@pcfreak:/var/www# rmmod snd_hda_intel
ERROR: Module snd_hda_intel does not exist in /proc/modules
rmmod snd_hwdep
root@pcfreak:/var/www# rmmod snd_mixer_oss
root@pcfreak:/var/www# rmmod snd_timer
root@pcfreak:/var/www# rmmod snd
root@pcfreak:/var/www# rmmod soundcore

Next step is to permanently disable all (blacklist) on system boot time loaded kernel modules, to do so in file /etc/modprobe.d/snd-bkaclist.conf, put:

blacklist soundcore
blacklist snd
blacklist snd_pcm
blacklist snd_pcsp
blacklist pcspkr

You can do it from shell ‘echo’-ing into file, like so:

# touch /etc/modprobe.d/snd-blacklist.conf
# cd /etc/modprobe.d/
# echo ‘blacklist soundcore’ >> snd-blacklist.conf
# echo ‘blacklist snd’ >> snd-blacklist.conf
# echo ‘blacklist snd_pcm’ >> snd-blacklist.conf
# echo ‘blacklist snd_pcsp’ >> snd-blacklist.conf
# echo ‘blacklist pcspkr’ >> snd-blacklist.conf

Another way is to use a tiny shellscript containing all previously shown rmmod commands and set the script to be executed via /etc/rc.local by adding the rmmod modules script, before exit 0 rc.local line.

Shell script removing my Realtek ICH sound driversis here
If you like to use it download it in /usr/local/bin or somewhere and invoke it via rc.local.
Removing sound blaster kernel support does not impact the overall machine performance, but in terms of security. Having sound driver modules constantly loaded in memory is a point which a possible attacker can use to root the host, so in my view always sound driver support should be removed.
Well that’s it hope this post helps someone 🙂

Decrease Debian Linux boot time through decreasing GRUB kernel boot timeout

Friday, October 5th, 2012

Default timeout set for booting GRUB in Debian Linux is 5 seconds. This is really long for (mobile) Desktop Linux users like me who frequently start and stop computer while moving from location to location. Thus I found it useful to decrease the boot timeout set in GRUB in order to decrease my overall boot waiting time.

hipo@noah:~/Desktop$ grep -i timeout /boot/grub/grub.cfg
set timeout=5

To decrease GRUB boot timeout I had to change timeout=5 to timeout=1.
That’s all enjoy 🙂

How to do world and kernel binary upgrade of FreeBSD 7.2 RELEASE to FreeBSD 9.0

Tuesday, October 2nd, 2012

FreeBSD 7.2 to FreeBSD 9.0 kernel world and kernel binary upgrade fbsd logo

1. Use FreeBSD upgrade with 9.0-RELEASE as an argument

freebsd# freebsd-update upgrade -r 9.0-RELEASE

Note that you will get on screen an error message like so:

The update metadata is correctly signed, but failed an integrity check.
Cowardly refusing to proceed any further.

You can safely ignore the message but anyways it is good to do following substitution in freebsd-update sript:

2. Patch freebsd-update script with sed

freebsd# sed -i '' -e 's/=_/=%@_/' /usr/sbin/freebsd-update

After that, run freebsd-update, one more time:

freebsd# freebsd-update upgrade -r 9.0-RELEASE

3. Confirm merges and changes from 7.2 to 9.0

Here during the upgrade you will be prompted for about 20-30 questions, which in most cases will be okay if you answer with yes. Most questions, are just notifications that in FreeBSD 9.0, some extra users were added some version release numbers are changed and the old files need to get merged with the new ones. It is good idea to read all the prompts and be sure to know what you’re doing before answering. Anyhow in my case I was quite in hurry so I simply did yes and q (to quit each of the diff notification screens pop-ping up after each yes answer.

4. Install all the freebsd-update upgrade fetched binaries

Once this process is completed you can safely install FBSD 9.0 kernel and world binaries:

freebsd# freebsd-update install

5. Restart the system to test if FreeBSD 9.0 RELEASE just installed new kernel boots

To test if all is okay with the update restart system:

# shutdown -r now

After system boots, do uname and uptime to assure system boots with 9.0 kernel:

freebsd# uptime
7:37PM up 2 days, 22:37, 1 user, load averages: 1.05, 1.02, 1.00
freebsd# uname -a;
FreeBSD freebsd 9.0-RELEASE-p3 FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 01:47:53 UTC 2012 i386

5. Finalize binary and world upgrade

Then to finalize the binary BSD world and kernel update once again issue:

freebsd# freebsd-update install
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run
"/usr/sbin/freebsd-update install" again to
finish installing updates.

freebsd# freebsd-update install

That’s all FreeBSD 7.2 -> 9.0 (kernel and world) upgrade is succesful. After the upgrade some packages might not be working and you will have to check and manually update ports which are not working (due to linked to old FreeBSD 7.2 world libraries and so on). It is also good to upgrade all installed ports. I will explain how I did ports upgrade a separate article. Cheers 😉