Posts Tagged ‘echo 1’

How to extend LVM full partition to bigger size on Linux Virtual machine Guest running in VMware vSphere

Tuesday, September 20th, 2022

lvm-filesystem-extend-on-linux-virtual-machine-vmware-physical-group-volume-group-logical-volume-partitions-picture

Lets say you have to resize a partition that is wrongly made by some kind of automation like ansible or puppet,
because the Linux RHEL family OS template was prepared with a /home (or other partition with some very small size)  on VMware Vsphere Hypervisor hosting the Guest linux VM and the partition got quickly out of space.

To resolve the following question comes for the sysadmin

I. How to extend the LVM parititon that run out of space (without rebooting the VM Guest Linux Host)

II. how to add new disk partition space to the vSphere hypervisor OS. 

In below article i'll shortly describe that trivials steps to take to achieve that. Article won't show anything new original but I wrote it,
because I want it to have it logged for myself in case I need to LVM extend the space of my own Virtual machines and 
cause hopefully that might be of help to someone else from the Linux community that has to complete the same task.
 

I . Extending a LVM parititon that run out of space on a Linux Guest VM
 
1. Check the current parititon size that you want to extend

[root@linux-hostname home]# df -h /home/
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/vg00-home
                       4.7G  4.5G     0 100% /home

2. Check the Virtualization platform

[root@vm-hostname ~]# lshw |head -3
linux-hostname
    description: Computer
    product: VMware Virtual Platform

3. Check the Operating System Linux OS type and version 

In this specific case this is a bit old Redhat -like CentOS 6.9 Linux
 

[root@vmware-host ~]# cat /etc/*release*
CentOS release 6.9 (Final)
CentOS release 6.9 (Final)
CentOS release 6.9 (Final)
cpe:/o:centos:linux:6:GA

4. Find out the type of target filesystem is EXT3, EXT4 or XFS etc.?

[root@vm-hostname ~]# grep home /proc/mounts
/dev/mapper/vg00-home /home ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0


Filesystem is handled by LVM thus

5. Check the size of the LVM partition we want to exchange

[root@vm-hostname ~]# lvs |grep home
home vg00 -wi-ao—- 5.00g

6. Check whether free space is available space in the volume group ?

[root@vm-hostname ~]# vgdisplay vg00
  — Volume group —
  VG Name               vg00
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  15
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                10
  Open LV               10
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               128.99 GiB
  PE Size               4.00 MiB
  Total PE              33022
  Alloc PE / Size       30976 / 121.00 GiB
  Free  PE / Size       2046 / 7.99 GiB
  VG UUID               1F89PB-nIP2-7Hgu-zEVR-5H0R-7GdB-Lfj7t4


Extend VMWare space configured for additional hard disk on Hypervisor (if necessery)

In order for to extend the LVM of course you need to have a pre-existing additional hard-drive on VM (sdb,sdc etc. attached )

– If you need to extend on Vmware Vsphere Hypervisor:
Extend additional harddrive by entering the new size and Validate.

If you have previously extended the size of the Virtual Disk from VMWare to make the Linux guest vm find out about the change
you have to rerun rescan for the respective device that was grown on the HV.

7. Rescan on Linux VM host for changes in disk size from Hypervisor

Rescan disk for new size :

[root@vm-hostname ~]# echo 1> /sys/block/sdX/device/rescan

(where sdX is the extended additional harddrive)

8. Resize LVM physical volume

[root@vm-hostname ~]# pvresize /dev/sdX

9. Enlarge Logical Volume size 

[root@vm-hostname ~]# lvextend -L+5G /dev/mapper/vg00-home
     Extending logical volume LogVol00 to 10.77 GiB
     Logical volume LogVol00 successfully resized

10. Enlarge LVM hosted filesystem size

Filesystem is ext3 or ext4 :

[root@vm-hostname ~]# resize2fs /dev/mapper/vg00-home

– If the filesystem is not ext3 / ext4 but XFS you have to use xfs_growfs to let the FS know about the change.

Filesystem is XFS :
 

[root@vm-hostname ~]# xfs_growfs /dev/mapper/vg00-home

11. Check the additional filespace is already active on the Linux Guest VM

[root@vm-hostname ~]# df -h /home/
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/vg_cloud-LogVol00
                        10G  4.2G  4.9G  48% /home


12. Verify  the extension of filesystem completed without errors


Check of system log:

[root@vm-hostname ~]# grep -i error /var/log/messages
…

Check if filesystem is writable.

[root@vm-hostname ~]# touch /home/test

[root@vm-hostname ~]# ls -al /home/test
-rw-r—– 1 root root 0 Sep 20 13:39 /home/test
[root@vm-hostname ~]# rm -f /home/test


II.  How to add additional sdb drive to a Linux host from vSPhere HV lets say (sdb)


1.  On VSphere GUI  interface

-> Select New hard drive and click Add

Enter the desired size for the new disk then unpack the disk parameters to choose Thin provision. Validate and Apply the recommendations.

basic-lvm-create-volume_group-diagram-on-linux-explained

2. On Linux system VM guest host to detect the new added sdb available space

Discover new disk :

[root@vm-hostname ~]# echo "- – -"> /sys/class/scsi_host/host2/scan && echo "- – -"> /sys/class/scsi_host/host1/scan && echo "- – -"> /sys/class/scsi_host/host0/scan

See  if discovered disk is found in /var/log/messages :

[…]
Nov 8 17:33:26 bict4004s kernel: scsi 2:0:2:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2
Nov 8 17:33:26 bict4004s kernel: scsi target2:0:2: Beginning Domain Validation
Nov 8 17:33:26 bict4004s kernel: scsi target2:0:2: Domain Validation skipping write tests
Nov 8 17:33:26 bict4004s kernel: scsi target2:0:2: Ending Domain Validation
Nov 8 17:33:26 bict4004s kernel: scsi target2:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
Nov 8 17:33:26 bict4004s kernel: sd 2:0:2:0: Attached scsi generic sg3 type 0
Nov 8 17:33:26 bict4004s kernel: sd 2:0:2:0: [sdb] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
Nov 8 17:33:26 bict4004s kernel: sd 2:0:2:0: [sdb] Write Protect is off
Nov 8 17:33:26 bict4004s kernel: sd 2:0:2:0: [sdb] Cache data unavailable
Nov 8 17:33:26 bict4004s kernel: sd 2:0:2:0: [sdb] Assuming drive cache: write through
Nov 8 17:33:26 bict4004s kernel: sd 2:0:2:0: [sdb] Attached SCSI disk
[…]

3. Create new LVM Physical Volume

[root@vm-hostname ~]# pvcreate /dev/sdb

4. Enlarge LVM Volume Group to the max available size of /dev/sdb

[root@vm-hostname ~]# vgextend vg00 /dev/sdb

Enlarge LVM Logical Volume

[root@vm-hostname ~]# lvextend -L+10G /dev/mapper/vg00-home

5. Enlarge filesystem to max size of just created LVM

If Filesystem is ext3 or ext4 :

[root@vm-hostname ~]# resize2fs /dev/mapper/vg00-home


Again if we work with XFS additionally do:

[root@vm-hostname ~]# xfs_growfs /dev/mapper/vg00-home

6. Checking filesystem extension completed correct

 [root@vm-hostname ~]# df -h /home
…


7. Check filesystem is writtable and no errors produced in logs

Check of system log:

[root@vm-hostname ~]# grep -i error /var/log/messages
…


Check if filesystem is writable.

[root@vm-hostname ~]# touch /home/test
…

Tags: , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | 1 Comment »

Fix FTP active connection issues “Cannot create a data connection: No route to host” on ProFTPD Linux dedicated server

Tuesday, October 1st, 2019

proftpd-linux-logo

Earlier I've blogged about an encounter problem that prevented Active mode FTP connections on CentOS
As I'm working for a client building a brand new dedicated server purchased from Contabo Dedi Host provider on a freshly installed Debian 10 GNU / Linux, I've had to configure a new FTP server, since some time I prefer to use Proftpd instead of VSFTPD because in my opinion it is more lightweight and hence better choice for a small UNIX server setups. During this once again I've encounted the same ACTIVE FTP not working from FTP server to FTP client host machine. But before shortly explaining, the fix I find worthy to explain briefly what is ACTIVE / PASSIVE FTP connection.

 

1. What is ACTIVE / PASSIVE FTP connection?
 

Whether in active mode, the client specifies which client-side port the data channel has been opened and the server starts the connection. Or in other words the default FTP client communication for historical reasons is in ACTIVE MODE. E.g.
Client once connected to Server tells the server to open extra port or ports locally via which the overall FTP data transfer will be occuring. In the early days of networking when FTP protocol was developed security was not of such a big concern and usually Networks did not have firewalls at all and the FTP DATA transfer host machine was running just a single FTP-server and nothing more in this, early days when FTP was not even used over the Internet and FTP DATA transfers happened on local networks, this was not a problem at all.

In passive mode, the server decides which server-side port the client should connect to. Then the client starts the connection to the specified port.

But with the ever increasing complexity of Internet / Networks and the ever tightening firewalls due to viruses and worms that are trying to own and exploit networks creating unnecessery bulk loads this has changed …

active-passive-ftp-explained-diagram
 

2. Installing and configure ProFTPD server Public ServerName

I've installed the server with the common cmd:

 

apt –yes install proftpd

 

And the only configuration changed in default configuration file /etc/proftpd/proftpd.conf  was
ServerName          "Debian"

I do this in new FTP setups for the logical reason to prevent the multiple FTP Vulnerability Scan script kiddie Crawlers to know the exact OS version of the server, so this was changed to:

 

ServerName "MyServerHostname"

 

Though this is the bad security through obscurity practice doing so is a good practice.
 

3. Create iptable firewall rules to allow ACTIVE FTP mode


But anyways, next step was to configure the firewall to be allowed to communicate on TCP PORT 21 and 20 to incoming source ports range 1024:65535 (to enable ACTIVE FTP) on firewal level with iptables on INPUT and OUTPUT chain rules, like this:

 

iptables -A INPUT -p tcp –sport 1024:65535 -d 0/0 –dport 21 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 0/0 –sport 1024:65535 -d 0/0 –dport 20 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 0/0 –sport 21 -d 0/0 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 0/0 –sport 20 -d 0/0 –dport 1024:65535 -m state –state ESTABLISHED,RELATED -j ACCEPT


Talking about Active and Passive FTP connections perhaps for novice Linux users it might be worthy to say few words on Active and Passive FTP connections

Once firewall has enabled FTP Active / Passive connections is on and FTP server is listening, to test all is properly configured check iptable rules and FTP listener:
 

/sbin/iptables -L INPUT |grep ftp
ACCEPT     tcp  —  anywhere             anywhere             tcp spts:1024:65535 dpt:ftp state NEW,ESTABLISHED
ACCEPT     tcp  —  anywhere             anywhere             tcp spts:1024:65535 dpt:ftp-data state NEW,ESTABLISHED
ACCEPT     tcp  —  anywhere             anywhere             tcp dpt:ftp
ACCEPT     tcp  —  anywhere             anywhere             tcp dpt:ftp-data

netstat -l | grep "ftp"
tcp6       0      0 [::]:ftp                [::]:*                  LISTEN    

 

4. Loading nf_nat_ftp module and net.netfilter.nf_conntrack_helper (for backward compitability)


Next step of course was to add the necessery modules nf_nat_ftp nf_conntrack_sane that makes FTP to properly forward ports with respective Firewall states on any of above source ports which are usually allowed by firewalls, note that the range of ports given 1024:65535 might be too much liberal for paranoid sysadmins and in many cases if ports are not filtered, if you are a security freak you can use some smaller range such as 60000-65535.

 

Here is time to say for sysadmins who haven't recently had a task to configure a new (unecrypted) File Transfer Server as today Secure FTP is almost alltime used for file transfers for the sake of security might be puzzled to find out the old Linux kernel ip_conntrack_ftp which was the standard module used to make FTP Active connections work is substituted nowadays with  nf_nat_ftp and nf_conntrack_sane.

To make the 2 modules permanently loaded on next boot on Debian Linux they have to be added to /etc/modules

Here is how sample /etc/modules that loads the modules on next system boot looks like

cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
softdog
nf_nat_ftp
nf_conntrack_sane


Next to say is that in newer Linux kernels 3.x / 4.x / 5.x the nf_nat_ftp and nf_conntrack-sane behaviour changed so  simply loading the modules would not work and if you do the stupidity to test it with some FTP client (I used gFTP / ncftp from my Linux desktop ) you are about to get FTP No route to host errors like:

 

Cannot create a data connection: No route to host

 

cannot-create-a-data-connection-no-route-to-host-linux-error-howto-fix


Sometimes, instead of No route to host error the error FTP client might return is:

 

227 entering passive mode FTP connect connection timed out error


To make the nf_nat_ftp module on newer Linux kernels hence you have to enable backwards compatibility Kernel variable

 

 

/proc/sys/net/netfilter/nf_conntrack_helper

 

echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper

 

To make it permanent if you have enabled /etc/rc.local legacy one single file boot place as I do on servers – for how to enable rc.local on newer Linuxes check here

or alternatively add it to load via sysctl

sysctl -w net.netfilter.nf_conntrack_helper=1

And to make change permanent (e.g. be loaded on next boot)

echo 'net.netfilter.nf_conntrack_helper=1' >> /etc/sysctl.conf

 

5. Enable PassivePorts in ProFTPD or PassivePortRange in PureFTPD


Last but not least open /etc/proftpd/proftpd.conf find PassivePorts config value (commented by default) and besides it add the following line:

 

PassivePorts 60000 65534

 

Just for information if instead of ProFTPd you experience the error on PureFTPD the configuration value to set in /etc/pure-ftpd.conf is:
 

PassivePortRange 30000 35000


That's all folks, give the ncftp / lftp / filezilla or whatever FTP client you prefer and test it the FTP client should be able to talk as expected to remote server in ACTIVE FTP mode (and the auto passive mode) will be not triggered anymore, nor you will get a strange errors and failure to connect in FTP clients as gftp.

Cheers 🙂

Tags: , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | No Comments »

How to determine which processes make most writes on the hard drive in GNU / Linux using kernel variable

Thursday, November 13th, 2014

how-to-determine-which-processes-writes-most-to-hard-drive-Linux-Kernel
In Linux there are plenty of tools to measure input / ouput – read / write server bottlenecks. Just to mention a few such are, the native part of all Linux distributions IOSTAT – which is a great tool to measure hard disk bottlenecks. However as iostat requires certain sysadmin skills for novice sys-admins, there is also ofcourse more interactive tools such as DSTAT or even better GLANCE which monitors not only disk writes but memory use, CPU load and Network use.

This tools can help you measure which processes are writting most (a lot) to hard disk drive but there is another quick and efficient way to track disk i/o by directly using the Linux kernel this is done via kernel parameter :

/proc/sys/vm/block_dump

To enable block_dump kernel logging:

echo 1 > /proc/sys/vm/block_dump

To later track in real time output from kernel interactively on which running process calling the kernel is writing to server hard drive
 

tail -f /var/log/syslog

The output  looks like so:
 

Nov 13 12:25:51 pcfreak kernel: [1075037.701056] kjournald(297): WRITE block 482293496 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701059] kjournald(297): WRITE block 482293504 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701062] kjournald(297): WRITE block 482293512 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701066] kjournald(297): WRITE block 482293520 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701069] kjournald(297): WRITE block 482293528 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.701072] kjournald(297): WRITE block 482293536 on sda1
Nov 13 12:25:51 pcfreak kernel: [1075037.702824] kjournald(297): WRITE block 482293544 on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.219288] apache2(3377): dirtied inode 3571740 (_index.html.old) on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.436133] mysqld(22945): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:52 pcfreak kernel: [1075039.436826] mysqld(22945): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.662832] mysqld(22945): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.663297] mysqld(22945): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.817120] apache2(3377): dirtied inode 3571754 (_index.html) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.819968] apache2(3377): dirtied inode 3571740 (_index.html_gzip) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075039.820016] apache2(3377): dirtied inode 3571730 (?) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075040.491378] mysqld(22931): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:53 pcfreak kernel: [1075040.492309] mysqld(22931): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:54 pcfreak kernel: [1075041.551513] apache2(3377): dirtied inode 1474706 (_index.html_gzip.old) on sda1
Nov 13 12:25:54 pcfreak kernel: [1075041.551566] apache2(3377): dirtied inode 1474712 (_index.html.old) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.769036] mysqld(22941): dirtied inode 21546676 (#sql_c0a_0.MYI) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.769804] mysqld(22941): dirtied inode 21546677 (#sql_c0a_0.MYD) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.985857] apache2(3282): dirtied inode 4063282 (data_9d97a7f62d54bc5fd791fba3245ba591-SMF-modSettings.php) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.987460] apache2(3282): dirtied inode 29010186 (data_9d97a7f62d54bc5fd791fba3245ba591-SMF-permissions–1.php) on sda1
Nov 13 12:25:55 pcfreak kernel: [1075041.988357] flush-8:0(289): WRITE block 51350632 on sda1

Using the kernel method to see which processes are stoning your server is great way especially for servers without connectivity to the Internet where you have no possibility to install sysstat package (contaning iostat),  dstat or glance.
Thanks to Marto's blog for  this nice hack.

Tags: , , , , , , , , , , ,
Posted in System Administration, Various, Web and CMS | No Comments »

Disable bluetooth on Linux IBM / Lenovo Thinkpad laptops

Thursday, February 14th, 2013

bluetooth gnu linux disable bluetooth linux how to tux logo bluetooth thinkpad

I have a Debian GNU / Linux squeeze with bluetooth and bluetooth is started automatically on system boot. This is pretty annoying, cause I use bluetooth quite rarely.
 disable / enable bluetooth via terminal is controlled via Linux sysfs virtual filesystem. The command to disable bluetooth one time is:

debian:~# echo 0 > /sys/devices/platform/thinkpad_acpi/bluetooth_enable

It is efficient in terms of energy saving especially if you use often your notebook on battery to turn off bluetooth permanently and only enable it when needed with:

debian:~# echo 1 > /sys/devices/platform/thinkpad_acpi/bluetooth_enable

To permanently disable bluetooth on Linux boot use:

# service bluetooth stop

In /etc/rc.local before exit 0 line place:

echo 0 > /sys/devices/platform/thinkpad_acpi/bluetooth_enable

An alternative method to permanently disable bluetooth (on other non-Thinkpad – any brand laptops) is via rfkill (bluetooth device control interface), on Ubuntu rfkill is installed by default but Debian users has to explicitly install it via apt:

debian:~# apt-get install –yes rfkill

Once rfkill is installed on host put a line before exit 0 in /etc/local:

rfkill block bluetooth
 

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in Bluetooth, Linux and FreeBSD Desktop, Various | No Comments »

How to set a crontab to execute commands on a seconds time interval on GNU / Linux and FreeBSD

Sunday, October 30th, 2011

crontab-execute-cron-jobs-every-second-on-linux-cron-logo
Have you ever been in need to execute some commands scheduled via a crontab, every let’s say 5 seconds?, naturally this is not possible with crontab, however adding a small shell script to loop and execute a command or commands every 5 seconds and setting it up to execute once in a minute through crontab makes this possible.
Here is an example shell script that does execute commands every 5 seconds:

#!/bin/bash
command1_to_exec='/bin/ls';
command2_to_exec='/bin/pwd';
for i in $(echo 1 2 3 4 5 6 7 8 9 10 11); do
sleep 5;
$command1_to_exec; $command2_to_exec;
done

This script will issue a sleep every 5 seconds and execute the two commands defined as $command1_to_exec and $command2_to_exec

Copy paste the script to a file or fetch exec_every_5_secs_cmds.sh from here

The script can easily be modified to execute on any seconds interval delay, the record to put on cron to use with this script should look something like:

# echo '* * * * * /path/to/exec_every_5_secs_cmds.sh' | crontab -

Where of course /path/to/exec_every_5_secs_cmds.sh needs to be modified to a proper script name and path location.

Another way to do the on a number of seconds program / command schedule without using cron at all is setting up an endless loop to run/refresh via /etc/inittab with a number of predefined commands inside. An example endless loop script to run via inittab would look something like:

while [ 1 ]; do
/bin/ls
sleep 5;
done

To run the above sample never ending script using inittab, one needs to add to the end of inittab, some line like:

mine:234:respawn:/path/to/script_name.sh

A quick way to add the line from consone would be with echo:

echo 'mine:234:respawn:/path/to/script' >> /etc/inittab

Of course the proper paths, should be put in:

Then to load up the newly added inittab line, inittab needs to be reloaded with cmd:

# init q

I've also red, some other methods suggested to run programs on a periodic seconds basis using just cron, what I found in stackoverflow.com's  as a thread proposed as a solution is:

* * * * * /foo/bar/your_script
* * * * * sleep 15; /foo/bar/your_script
* * * * * sleep 30; /foo/bar/your_script
* * * * * sleep 45; /foo/bar/your_script

One guy, even suggested a shorted way with cron:

0/15 * * * * * /path/to/my/script

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, System Administration, Various | 1 Comment »

How to configure and enable Xen Linux dedicated server’s Virtual machines Internet to work / Enable multipe real IPs and one MAC only in (SolusVM) through NAT routed and iptables

Saturday, June 4th, 2011

Xen Linux Virtual Machine Logo

I’ve been hired as a consultant recently to solve a small task on a newly bought Xen based dedicated server.
The server had installed on itself SolusVM

The server was a good hard-iron machine running with CentOS Linux with enabled Xen virtualization support.
The Data Center (DC) has provided the client with 4 IP public addresses, whether the machine was assigned to possess only one MAC address!

The original idea was the dedicated server is supposed to use 4 of the IP addresses assigned by the DC whether only one of the IPs has an external internet connected ethernet interface with assigned MAC address.

In that case using Xen’s bridging capabilities was pretty much impossible and therefore Xen’s routing mode has to be used, plus an Iptables Network Address Translation or an IP MASQUERADE .

In overall the server would have contained 3 virtual machines inside the Xen installed with 3 copies of:

  • Microsoft Windows 2008

The scenario I had to deal with is pretty much explained in Xen’s Networking wiki Two Way Routed Network

In this article I will describe as thoroughfully as I can how I configured the server to be able to use the 3 qemu virtual machines (running inside the Xen) with their respective real interner visible public IP addresses.

1. Enable Proxyarp for the eth0 interface

To enable proxyarp for eth0 on boot time and in real time on the server issue the commands:

[root@centos ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp[root@centos ~]# echo 'net.ipv4.conf.all.proxy_arp = 1' >> /etc/sysctl.conf

2. Enable IP packet forwarding for eth interfaces

This is important pre-requirement in order to make the iptables NAT to work.

[root@centos ~]# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
[root@centos ~]# echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf

If you get errors during execution of /etc/init.d/xendomains , like for example:

[root@centos ~]# /etc/init.d/xendomains restart
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv4/conf/eth0/proxy_arp: No such file or directory
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv6/conf/eth0/proxy_arp: No such file or directory

in order to get rid of the message you will have to edit /etc/xen/scripts/network-route and comment out the lines:

echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp
e.g.
#echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
#echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp

3. Edit /etc/xen/xend-config.sxp, disable ethernet bridging and enable eth0 routing (route mode) and NAT for Xen’s routed mode

Make absolutely sure that in /etc/xen/xend-config.sxp the lines related to bridging are commented.
The lines you need to comment out are:

(network-script network-bridge)
(vif-script vif-bridge)

make them look like:

#(network-script network-bridge)
#(vif-script vif-bridge)br />

Now as bridging is disabled let’s enable Xen routed network traffic as an bridged networking alternative.

Find the commented (network-script network-route) and (vif-script vif-route) lines and uncomment them:

#(network-script network-route)
#(vif-script vif-route)

The above commented lines should become:

(network-script network-route)
(vif-script vif-route)

Next step is to enable NAT for routed traffic in Xen (necessery to make routed mode work).
Below commented two lines in /etc/xen/xend-config.sxp, should be uncommented e.g.:

#(network-script network-nat)
#(vif-script vif-nat)

Should become:

(network-script network-nat)
(vif-script vif-nat)

4. Restart Xen control daemon and reload installed Xen’s Virtual Machines installed domains

To do so invoke the commands:

[root@centos ~]# /etc/init.d/xend
[root@centos ~]# /etc/init.d/xendomains restart

This two commands will probably take about 7 to 10 minutes (at least they took this serious amount of time in my case).
If you think this time is too much to speed-up the procedure of restarting Xen and qemu attached virtual machines, restart the whole Linux server, e.g.:

[root@centos ~]# restart

5. Configure iptables NAT rules on the CentOS host

After the server boots up, you will have to initiate the following ifconfig & iptables rules in order to make the Iptables NAT to work out:

echo > > /proc/sys/net/ipv4/conf/tap1.0/proxy_arp
/sbin/ifconfig eth0:1 11.22.33.44 netmask 255.255.252.0
/sbin/ifconfig eth0:2 22.33.44.55 netmask 255.255.252.0
/sbin/ifconfig eth0:3 33.44.55.66 netmask 255.255.252.0

/sbin/iptables -t nat -A PREROUTING -d 11.22.33.44 -i eth0 -j DNAT --to-destination 192.168.1.2
/sbin/iptables -t nat -A PREROUTING -d 22.33.44.55 -i eth0 -j DNAT --to-destination 192.168.1.3
/sbin/iptables -t nat -A PREROUTING -d 33.44.55.66 -i eth0 -j DNAT --to-destination 192.168.1.4
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.2 -o eth0 -j SNAT --to-source 11.22.33.44
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.3 -o eth0 -j SNAT --to-source 22.33.44.55
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.4 -o eth0 -j SNAT --to-source 33.44.55.66

In the above ifconfig and iptables rules the IP addresses:

11.22.33.44, 22.33.44.55, 33.44.55.66 are real IP addresses visible from the Internet.
In the above rules eth0:1, eth0:2 and eth0:3 are virtual ips assigned to the main eth0 interface.

This ifconfig and iptables setup assumes that the 3 Windows virtual machines running inside the Xen dedicated server will be configured to use (local) private network IP addresses:

192.168.1.2, 192.168.1.3 and 192.168.1.4

You will have also to substitute the 11.22.33.44, 22.33.44.55 and 33.44.55.66 with your real IP addreses.

To store the iptables rules permanently on the fedora you can use the iptables-save command:

[root@centos ~]# /sbin/iptables-save

However I personally did not use this approach to save my inserserted iptable rules for later boots but I use my small script set_ips.sh to add virtual interfaces and iptables rules via the /etc/rc.local invokation:

If you like the way I have integrated my virtual eths initiation and iptables kernel firewall inclusion, download my script and set it to run in /etc/rc.local, like so:

[root@centos ~]# cd /usr/sbin
[root@centos sbin]# wget https://www.pc-freak.net/bshscr/set_ips.sh
...
[root@centos ~]# chmod +x /usr/sbin/set_ips.sh
[root@centos ~]# mv set_ips.sh /usr/sbin
[root@centos ~]# echo '/usr/sbin/set_ips.sh' >> /etc/rc.local

Note that you will have to modify my set_ips.sh script to substitute the 11.22.33.44, 22.33.44.55 and 33.44.55.66 with your real IP address.

So far so good, one might think that all this should be enough for the Virtual Machines Windows hosts to be able to connect to the Internet and Internet requests to the virtual machines to arrive, but no it’s not!!

6. Debugging Limited Connectivity Windows LAN troubles on the Xen dedicated server

Even though the iptables rules were correct and the vif route and vif nat was enabled inside the Xen node, as well as everything was correctly configured in the Windows 2008 host Virtual machines, the virtual machines’s LAN cards were not able to connect properly to connect to the internet and the Windows LAN interface kept constantly showing Limited Connectivity! , neither a ping was available to the gateway configured for the Windows VM host (which in my case was: 192.168.1.1).

You see the error with Limited connectivity inside the Windows on below’s screenshot:

Limited Connectivty Windows error Lan Interface, status screenshot

Here is also a screenshot of my VNC connection to the Virtual machine with the correct IP settings – (TCP/IPv4) Properties Window:

Windows Xen Network Connections Windows VNC TCP/IPv4 Properties Window

This kind of Limited Connectivity VM Windows error was really strange and hard to diagnose, thus I started investigating what is wrong with this whole situation and why is not able the Virtualized Windows to connect properly to the Internet, through the Iptables NAT inbound and outbound traffic redirection.

To diagnose the problem, I started up with listing the exact network interfaces showing to be on the Xen Dedicated server:


[root@centos ~]# /sbin/ifconfig |grep -i 'Link encap' -A 1
eth0 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:111.22.33.55 Bcast:111.22.33.255
Mask:255.255.252.0
--
eth0:1 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:11.22.33.44 Bcast:11.22.33.255
Mask:255.255.252.0
--
eth0:2 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:22.33.44.55 Bcast:22.33.44.255
Mask:255.255.252.0
--
eth0:3 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:33.44.55.66 Bcast:33.44.55.255
Mask:255.255.252.0
--
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
--
tap1.0 Link encap:Ethernet HWaddr FA:07:EF:CA:13:31
--
vifvm101.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr:111.22.33.55 Bcast:111.22.33.55
Mask:255.255.255.255

I started debugging the issue, using the expelling logic.
In the output concerning my interfaces via ifconfig on eth0, I have my primary server IP address 111.22.33.55 , this one is working for sure as I was currently connected to the server through it.

The other virtual IP addresses assigned on the virtual network interfaces eth0:1, eth0:2 and eth0:3 were also assigned correctly as I was able to ping this ips from my Desktop machine from the Internet.

The lo , interface was also properly configured as I could ping without a problem the loopback ip – 127.0.0.1

The rest of the interfaces displayed by my ifconfig output were: tap1.0, vifvm101.0

After a bit of ressearch, I’ve figured out that they’re virtual interfaces and they belong to the Xen domains which are running qemu virtual machines with the Windows host.

I used tcpdump to debug what kind of traffic does flow through the tap1.0 and vifvm101.0 interfaces, like so

[root@centos ~]# tcpdump -i vifvm101.0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vifvm101.0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@centos ~]# tcpdump -i tap1.0
cpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1.0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
08:55:52.490249 IP 229.197.34.95.customer.cdi.no.15685 > 192.168.1.2.12857: UDP, length 42

I’ve figured out as it’s also observable in above’s two tcpdump commands output, that nothing flows through the vifvm101.0 interface, and that there was some traffic passing by tap1.0 interface.

7. Solving the Limited Connectivy Windows Internet network connection problems

As below’s ifconfig output reveals, there is no IP address assigned to tap1.0 interface, using some guidelines and suggestions from guys in irc.freenode.net’s #netfilter irc channel, I’ve decided to give a go to set up an IP address of 192.168.1.1 to tap1.0 .

I choose for a reason as this IP address is configured to be my Gateway’s IP Address inside the Emulated Windows 2008 hosts

To assign the 192.168.1.1 to tap1.0, I issued:

[root@centos ~]# /sbin/ifconfig tap1.0 192.168.1.1 netmask 255.255.255.0
 To test if there is difference I logged in to the Virtual Machine host with gtkvncviewer (which by the way is a very nice VNC client for Gnome) and noticed there was an established connection to the internet inside the Virtual Machine 😉

I issued a ping to google which was also returned and opened a browser to really test if everything is fine with the Internet.
Thanks God! I could browse and everything was fine 😉

8. Making tap1.0 192.168.1.1 (VM hosts gateway to be set automatically, each time server reboots)

After rebooting the server the tap1.0 assignmend of 192.168.1.1 disappeared thus I had to make the 192.168.1.1, be assigned automatically each time the CentoS server boots.

To give it a try, I decided to place /sbin/ifconfig tap1.0 192.168.1.1 netmask 255.255.255.0 into /etc/rc.local, but this worked not as the tap1.0 interface got initialized a while after all the xendomains gets initialized.

I tried few times to set some kind of sleep time interval with the sleep , right before the /sbin/ifconfig tap1.0 … ip initialization but this did not worked out, so I finally completely abandoned this methodology and make the tap1.0 get initialized with an IP through a cron daemon.
For that purpose I’ve created a script to be invoked, every two minutes via cron which checked if the tap1.0 interface is up and if not issues the ifconfig command to initialize the interface and assign the 192.168.1.1 IP to it.

Here is my set_tap_1_iface.sh shell script

To set it up on your host in /usr/sbin issue:

[root@centos ~]# cd /usr/sbin/
[root@centos sbin]# wget https://www.pc-freak.net/bshscr/set_tap_1_iface.sh
...
 In order to set it on cron to make the tap1.0 initialization automatically every two minutes use the cmd:

[root@centos ~]# crontab -u root -e

After the cronedit opens up, place the set_tap_1_iface.sh cron invokation rules:

*/2 * * * * /usr/sbin/set_tap_1_iface.sh >/dev/null 2>&1

and save.

That’s all now your Xen dedicated and the installed virtual machines with their public internet IPs will work 😉
If this article helped you to configure your NAT routing in Xen drop me a thanks message, buy me a beer or hire me! Cheers 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | 2 Comments »