Posts Tagged ‘Netmask’

FreeBSD Jumbo Frames network configuration short how to

Wednesday, March 14th, 2012

FreeBSD Jumbo Frames Howto configure FreeBSD

Recently I wrote a post on how to enable Jumbo Frames on GNU / Linux , therefore I thought it will be useful to write how Jumbo Frames network boost can be achieved on FreeBSD too.

I will skip the details of what is Jumbo Frames, as in the previous article I have thoroughfully explained. Just in short to remind you what is Jumbo Frames and why you might need it? – it is a way to increase network MTU transfer frames from the MTU 1500 to MTU of 9000 bytes

It is interesting to mention that according to specifications, the maximum Jumbo Frames MTU possible for assignment are of MTU=16128
Just like on Linux to be able to take advantage of the bigger Jumbo Frames increase in network thoroughput, you need to have a gigabyt NIC card/s on the router / server.

1. Increasing MTU to 9000 to enable Jumbo Frames "manually"

Just like on Linux, the network tool to use is ifconfig. For those who don't know ifconfig on Linux is part of the net-tools package and rewritten from scratch especially for GNU / Linux OS, whether BSD's ifconfig is based on source code taken from 4.2BSD UNIX

As you know, network interface naming on FreeBSD is different, as there is no strict naming like on Linux (eth0, eth1, eth2), rather the interfaces are named after the name of the NIC card vendor for instance (Intel(R) PRO/1000 NIC is em0), RealTek is rl0 etc.

To set Jumbro Frames Maximum Transmission Units of 9000 on FreeBSD host with a Realtek and Intel gigabyt ethernet cards use:freebsd# /sbin/ifconfig em0 192.168.1.2 mtu 9000
freebsd# /sbin/ifconfig rl0 192.168.2.2 mtu 9000

!! Be very cautious here, as if you're connected to the system remotely over ssh you might loose connection to it because of broken routing.

To prevent routing loss problems, if you're executing the above two commands remotely, you better run them in GNU screen session:

freebsd# screen
freebsd# /sbin/ifconfig em0 192.168.1.2 mtu 9000; /sbin/ifconfig rl0 192.168.1.2 mtu 9000; \
/etc/rc.d/netif restart; /etc/rc.d/routed restart

2. Check MTU settings are set to 9000

If everything is fine the commands will return empty output, to check further the MTU is properly set to 9000 issue:

freebsd# /sbin/ifconfig -a|grep -i em0em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000freebsd# /sbin/ifconfig -a|grep -i rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000

3. Reset routing for default gateway

If you have some kind of routing assigned for em0 and rl0, network interfaces they will be affected by the MTU change and the routing will be gone. To reset the routing to the previously properly assigned routing, you have to restart the BSD init script taking care for assigning routing on system boot time:

freebsd# /etc/rc.d/routing restart
default 192.168.1.1 done
add net default: gateway 192.168.1.1
Additional routing options: IP gateway=YES.

4. Change MTU settings for NIC card with route command

There is also a way to assign higher MTU without "breaking" the working routing, e.g. avoiding network downtime with bsd route command:

freebsd# grep -i defaultrouter /etc/rc.conf
defaultrouter="192.168.1.1"
freebsd# /sbin/route change 192.168.1.1 -mtu 9000
change host 192.168.1.1

5. Finding the new MTU NIC settings on the FreeBSD host

freebsd# /sbin/route -n get 192.168.1.1
route to: 192.168.1.1
destination: 192.168.1.1
interface: em0
flags: <UP,HOST,DONE,LLINFO,WASCLONED>
recvpipe sendpipe ssthresh rtt,msec rttvar hopcount mtu expire
0 0 0 0 0 0 9000 1009

6. Set Jumbo Frames to load automatically on system load

To make the increased MTU to 9000 for Jumbo Frames support permanent on a FreeBSD system the /etc/rc.conf file is used:

The variable for em0 and rl0 NICs are ifconfig_em0 and ifconfig_rl0.
The lines to place in /etc/rc.conf should be similar to:

ifconfig_em0="inet 192.168.1.1 netmask 255.255.255.0 media 1000baseTX mediaopt half-duplex mtu 9000"
ifconfig_em0="inet 192.168.1.1 netmask 255.255.255.0 media 1000baseTX mediaopt half-duplex mtu 9000"

Change in the above lines the gateway address 192.168.1.1 and the netmask 255.255.255.0 to yours corresponding gw and netmask.
Also in the above example you see the half-duplex ifconfig option is set insetad of full-duplex in order to prevent some duplex mismatches. A full-duplex could be used instead, if you're completely sure on the other side of the host is configured to support full-duplex connections. Otherwise if you try to set full-duplex with other side set to half-duplex or auto-duplex a duplex mismatch will occur. If this happens insetad of taking the advantage of the Increase Jumbo Frames MTU the network connection could become slower than originally with standard ethernet MTU of 1500. One other bad side if you end up with duplex-mismatch could be a high number of loss packets and degraded thoroughout …

7. Setting Jumbo Frames for interfaces assigning dynamic IP via DHCP

If you need to assign an MTU of 9000 for a gigabyt network interfaces, which are receiving its TCP/IP network configuration over DHCP server.
First, tell em0 and rl0 network interfaces to dynamically assign IP addresses via DHCP proto by adding in /etc/rc.conf:

ifconfig_em0="DHCP"
ifconfig_rl0="DHCP"

Secondly make two files /etc/start_if.em0 and /etc/start_if.rl0 and include in each file:

ifconfig em0 media 1000baseTX mediaopt full-duplex mtu 9000
ifconfig rl0 media 1000baseTX mediaopt full-duplex mtu 9000

Copy / paste in root console:

echo 'ifconfig em0 media 1000baseTX mediaopt full-duplex mtu 9000' >> /etc/start_if.em0
echo 'ifconfig rl0 media 1000baseTX mediaopt full-duplex mtu 9000' >> /etc/start_if.rl0

Finally, to load the new MTU for both interfaces, reload the IPs with the increased MTUs:

freebsd# /etc/rc.d/routing restart
default 192.168.1.1 done
add net default: gateway 192.168.1.1

8. Testing if Jumbo Frames is working correctly

To test if an MTU packs are transferred correctly through the network you can use ping or tcpdumpa.) Testing Jumbo Frames enabled packet transfers with tcpdump

freebsd# tcpdump -vvn | grep -i 'length 9000'

You should get output like:

16:40:07.432370 IP (tos 0x0, ttl 50, id 63903, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 85825:87285(1460) ack 668 win 14343
16:40:07.432588 IP (tos 0x0, ttl 50, id 63904, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 87285:88745(1460) ack 668 win 14343
16:40:07.433091 IP (tos 0x0, ttl 50, id 63905, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 23153:24613(1460) ack 668 win 14343
16:40:07.568388 IP (tos 0x0, ttl 50, id 63907, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 88745:90205(1460) ack 668 win 14343
16:40:07.568636 IP (tos 0x0, ttl 50, id 63908, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 90205:91665(1460) ack 668 win 14343
16:40:07.569012 IP (tos 0x0, ttl 50, id 63909, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 91665:93125(1460) ack 668 win 14343
16:40:07.569888 IP (tos 0x0, ttl 50, id 63910, offset 0, flags [DF], proto TCP (6), length 9000) 192.168.1.2.80 > 192.168.1.1.60213: . 93125:94585(1460) ack 668 win 14343

b.) Testing if Jumbo Frames are enabled with ping

Testing Jumbo Frames with ping command on Linux

linux:~# ping 192.168.1.1 -M do -s 8972
PING 192.168.1.1 (192.168.1.1) 8972(9000) bytes of data.
9000 bytes from 192.168.1.1: icmp_req=1 ttl=52 time=43.7 ms
9000 bytes from 192.168.1.1: icmp_req=2 ttl=52 time=43.3 ms
9000 bytes from 192.168.1.1: icmp_req=3 ttl=52 time=43.5 ms
9000 bytes from 192.168.1.1: icmp_req=4 ttl=52 time=44.6 ms
--- 192.168.0.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 2.397/2.841/4.066/0.708 ms

If you get insetad an an output like:

From 192.168.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 192.168.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 192.168.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 192.168.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)

--- 192.168.1.1 ping statistics ---
0 packets transmitted, 0 received, +4 errors

This means a packets with maximum MTU of 1500 could be transmitted and hence something is not okay with the Jumbo Frames config.
Another helpful command in debugging MTU and showing which host in a hop queue support jumbo frames is Linux's traceroute

To debug a path between host and target, you can use:

linux:~# traceroute --mtu www.google.com
...

If you want to test the Jumbo Frames configuration from a Windows host use ms-windows ping command like so:

C:\>ping 192.168.1.2 -f -l 8972
Pinging 192.168.1.2 with 8972 bytes of data:
Reply from 192.168.1.2: bytes=8972 time=2ms TTL=255
Reply from 192.168.1.2: bytes=8972 time=2ms TTL=255
Reply from 192.168.1.2: bytes=8972 time=2ms TTL=255
Reply from 192.168.1.2: bytes=8972 time=2ms TTL=255
Ping statistics for 192.168.1.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 2ms, Average = 2ms

Here -l 8972 value is actually equal to 9000. 8972 = 9000 – 20 (20 byte IP header) – 8 (ICMP header)

Boost local network performance (Increase network thoroughput) by enabling Jumbo Frames on GNU / Linux

Saturday, March 10th, 2012

Jumbo Frames boost local network performance in GNU / Linux

So what is Jumbo Frames? and why, when and how it can increase the network thoroughput on Linux?

Jumbo Frames are Ethernet frames with more than 1500 bytes of payload. They can carry up to 9000 bytes of payload. Many Gigabit switches and network cards supports them.
Jumbo frames is a networking standard for many educational networks like AARNET. Unfortunately most commercial ISPs doesn't support them and therefore enabling Jumbo frames will rarely increase bandwidth thoroughput for information transfers over the internet.
Hopefully in the years to come with the constant increase of bandwidths and betterment of connectivity, jumbo frames package transfers will be supported by most ISPs as well.
Jumbo frames network support is just great for is small local – home networks and company / corporation office intranets.

Thus enabling Jumbo Frame is absolutely essential for "local" ethernet networks, where large file transfers occur frequently. Such networks are networks where, there is often a Video or Audio streaming with high quality like HD quality on servers running File Sharing services like Samba, local FTP sites,Webservers etc.

One other advantage of enabling jumbo frames is reduce of general server overhead and decrease in CPU load / (CPU usage), when transferring large or enormous sized files.Therefore having jumbo frames enabled on office network routers with GNU / Linux or any other *nix OS is vital.

Jumbo Frames traffic is supported in GNU / Linux kernel since version 2.6.17+ in earlier 2.4.x it was possible through external third party kernel patches.

1. Manually increase MTU to 9000 with ifconfig to enable Jumbo frames

debian:~# /sbin/ifconfig eth0 mtu 9000

The default MTU on most GNU / Linux (if not all) is 1500, to check the default set MTU with ifconfig:

linux:~# /sbin/ifconfig eth0|grep -i mtu
UP BROADCAST MULTICAST MTU:1500 Metric:1

To take advantage of Jumbo Frames, all that has to be done is increase the default Maximum Transmission Unit from 1500 to 9000

For those who don't know MTU is the largest physical packet size that can be transferred over the network. MTU is measured by default in bytes. If a information has to be transferred over the network which exceeds the lets say 1500 MTU (bytes), it will be chopped and transferred in few packs each of 1500 size.

MTUs differ on different netework topologies. Just for info here are the few main MTUs for main network types existing today:
 

  • 16 MBit/Sec Token Ring – default MTU (17914)
  • 4 Mbits/Sec Token Ring – default MTU (4464)
  • FDDI – default MTU (4352)
  • Ethernet – def MTU (1500)
  • IEEE 802.3/802.2 standard – def MTU (1492)
  • X.25 (dial up etc.) – def MTU (576)
  • Jumbo Frames – def max MTU (9000)

Setting the MTU packet frames to 9000 to enable Jumbo Frames is done with:

linux:~# /sbin/ifconfig eth0 mtu 9000

If the command returns nothing, this most likely means now the server can communicate on eth0 with MTUs of each 9000 and therefore the network thoroughput will be better. In other case, if the network card driver or card is not a gigabit one the cmd will return error:

SIOCSIFMTU: Invalid argument

2. Enabling Jumbo Frames on Debian / Ubuntu etc. "the Debian way"

a.) Jumbo Frames on ethernet interfaces with static IP address assigned Edit /etc/network/interfaces and you should have for each of the interfaces you would like to set the Jumbo Frames, records similar to:

Raising the MTU to 9000 if for one time can be done again manually with ifconfig

debian:~# /sbin/ifconfig eth0 mtu 9000

iface eth0 inet static
address 192.168.0.5
network 192.168.0.0
gateway 192.168.0.254
netmask 255.255.255.0
mtu 9000

For each of the interfaces (eth1, eth2 etc.), add a chunk similar to one above changing the changing the IPs, Gateway and Netmask.

If the server is with two gigabit cards (eth0, eth1) supporting Jumbo frames add to /etc/network/interfaces :

iface eth0 inet static
address 192.168.0.5
network 192.168.0.0
gateway 192.168.0.254
netmask 255.255.255.0
mtu 9000

iface eth1 inet static
address 192.168.0.6
network 192.168.0.0
gateway 192.168.0.254
netmask 255.255.255.0
mtu 9000

b.) Jumbo Frames on ethernet interfaces with dynamic IP obtained via DHCP

Again in /etc/network/interfaces put:

auto eth0
iface eth0 inet dhcp
post-up /sbin/ifconfig eth0 mtu 9000

3. Setting Jumbo Frames on Fedora / CentOS / RHEL "the Redhat way"

Enabling jumbo frames on all Gigabit lan interfaces (eth0, eth1, eth2 …) in Fedora / CentOS / RHEL is done through files:
 

  • /etc/sysconfig/network-script/ifcfg-eth0
  • /etc/sysconfig/network-script/ifcfg-eth1

etc. …
append in each one at the end of the respective config:

MTU=9000

[root@fedora ~]# echo 'MTU=9000' >> /etc/sysconfig/network-scripts/ifcfg-eth


a quick way to set Maximum Transmission Unit to 9000 for all network interfaces on on Redhat based distros is by executing the following loop:

[root@centos ~]# for i in $(echo /etc/sysconfig/network-scripts/ifcfg-eth*); do \echo 'MTU=9000' >> $i
done

P.S.: Be sure that all your interfaces are supporting MTU=9000, otherwise increase while the MTU setting is set will return SIOCSIFMTU: Invalid argument err.
The above loop is to be used only, in case you have a group of identical machines with Lan Cards supporting Gigabit networks and loaded kernel drivers supporting MTU up to 9000.

Some Intel and Realtek Gigabit cards supports only a maximum MTU of 7000, 7500 etc., so if you own a card like this check what is the max MTU the card supports and set it in the lan device configuration.
If increasing the MTU is done on remote server through SSH connection, be extremely cautious as restarting the network might leave your server inaccessible.

To check if each of the server interfaces are "Gigabit ready":

[root@centos ~]# /sbin/ethtool eth0|grep -i 1000BaseT
1000baseT/Half 1000baseT/Full
1000baseT/Half 1000baseT/Full

If you're 100% sure there will be no troubles with enabling MTU > 1500, initiate a network reload:

[root@centos ~]# /etc/init.d/network restart
...

4. Enable Jumbo Frames on Slackware Linux

To list the ethernet devices and check they are Gigabit ones issue:

bash-4.1# lspci | grep [Ee]ther
0c:00.0 Ethernet controller: D-Link System Inc Gigabit Ethernet Adapter (rev 11)
0c:01.0 Ethernet controller: D-Link System Inc Gigabit Ethernet Adapter (rev 11)

Setting up jumbo frames on Slackware Linux has two ways; the slackware way and the "universal" Linux way:

a.) the Slackware way

On Slackware Linux, all kind of network configurations are done in /etc/rc.d/rc.inet1.conf

Usual config for eth0 and eth1 interfaces looks like so:

# Config information for eth0:
IPADDR[0]="10.10.0.1"
NETMASK[0]="255.255.255.0"
USE_DHCP[0]=""
DHCP_HOSTNAME[1]=""
# Config information for eth1:
IPADDR[1]="10.1.1.1"
NETMASK[1]="255.255.255.0"
USE_DHCP[1]=""
DHCP_HOSTNAME[1]=""

To raise the MTU to 9000, the variables MTU[0]="9000" and MTU[1]="9000" has to be included after each interface config block, e.g.:

# Config information for eth0:
IPADDR[0]="172.16.1.1"
NETMASK[0]="255.255.255.0"
USE_DHCP[0]=""
DHCP_HOSTNAME[1]=""
MTU[0]="9000"
# Config information for eth1:
IPADDR[1]="10.1.1.1"
NETMASK[1]="255.255.255.0"
USE_DHCP[1]=""
DHCP_HOSTNAME[1]=""
MTU[1]="9000"

bash-4.1# /etc/rc.d/rc.inet1 restart
...

b.) The "Universal" Linux way

This way is working on most if not all Linux distributions.
Insert in /etc/rc.local:

/sbin/ifconfig eth0 mtu 9000 up
/sbin/ifconfig eth1 mtu 9000 up

5. Check if Jumbo Frames are properly enabled

There are at least two ways to display the MTU settings for eths.

a.) Using grepping the MTU from ifconfig

linux:~# /sbin/ifconfig eth0|grep -i mtu
UP BROADCAST MULTICAST MTU:9000 Metric:1
linux:~# /sbin/ifconfig eth1|grep -i mtu
UP BROADCAST MULTICAST MTU:9000 Metric:1

b.) Using ip command from iproute2 package to get MTU

linux:~# ip route get 192.168.2.134
local 192.168.2.134 dev lo src 192.168.2.134
cache mtu 9000 advmss 1460 hoplimit 64

linux:~# ip route show dev wlan0
192.168.2.0/24 proto kernel scope link src 192.168.2.134
default via 192.168.2.1

You see MTU is now set to 9000, so the two server lans, are now able to communicate with increased network thoroughput.
Enjoy the accelerated network transfers 😉

 

How to configure manually static IP address on Debian GNU/Linux / How to fix eth0 interface not brought up with error (networking restart is deprecated)

Friday, July 29th, 2011

I’ve recently had to manually assign a static IP address on one of the servers I manage, here is how I did it:             

debian:~# vim /etc/network/interfaces

Inside the file I placed:

# The primary network interface
allow-hotplug eth0
auto eth0
iface eth0 inet static address 192.168.0.2 netmask 255.255.255.0 broadcast 192.168.0.0 gateway 192.168.0.1 dns-nameservers 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220

The broadcast and gateway configuration lines are not obligitory.
dns-nameservers would re-create /etc/resolv.conf file with the nameserver values specified which in these case are Google Public DNS servers and OpenDNS servers.

Very important variable is allow-hotplug eth0
If these variable with eth0 lan interface is omitted or missing (due to some some weird reason), the result would be the output you see from the command below:

debian:~# /etc/init.d/networking restart
Running /etc/init.d/networking restart is deprecated because it may not enable again some interfaces ... (warning).
Reconfiguring network interfaces...

Besides the /etc/init.d/networking restart is deprecated because it may not enable again some interfaces … (warning). , if the allow-hotplug eth0 variable is omitted the eth0 interface would not be brough up on next server boot or via the networking start/stop/restart init script.

My first reaction when I saw the message was that probably I’ll have to use invoke-rc.d, e.g.:
debian:~# invoke-rc.d networking restart
Running invoke-rc.d networking restart is deprecated because it may not enable again some interfaces ... (warning).

However as you see from above’s command output, running invoke-rc.d helped neither.

I was quite surprised with the inability to bring my network up for a while with the networking init script.
Interestingly using the command:

debian:~# ifup eth0

was able to succesfully bring up the network interface, whether still invoke-rc.d networking start failed.

After some wondering I finally figured out that the eth0 was not brought up by networking init script, because auto eth0 or allow-hotplug eth0 (which by the way are completely interchangable variables) were missing.

I added allow-hotplug eth0 and afterwards the networking script worked like a charm 😉

How to add a range of virtual IPs to a CentOS and Fedora Linux server

Monday, July 18th, 2011

Recently I had the task to add a range of few IP addresses to as a virtual interface IPs.

The normal way to do that is of course using the all well known ifconfig eth0:0, ifconfig eth0:1 or using a tiny shell script which does it and set it up to run through /etc/rc.local .

However the Redhat guys could omit all this mambo jambo and do it The Redhat way TM 😉 by using a standard method documented in CentOS and RHEL documentation.
Here is how:

# go to network-script directory[root@centos ~]# cd /etc/sysconfig/network-scripts
# create ifcfg-eth0-range (if virtual ips are to be assigned on eth0 lan interface[root@centos network-scripts]# touch ifcfg-eth0-range

Now inside ifcfg-eth0-range, open up with a text editor or use the echo command to put inside:

IPADDR_START=192.168.1.120
IPADDR_END=192.168.1.250
NETMASK=255.255.255.25
CLONENUM_START=0

Now save the /etc/sysconfig/network-scripts/ifcfg-eth0-range file and finally restart centos networking via the network script:

[root@centos network-scripts]# service network restart

That’s all now after the network gets reinitialized all the IPs starting with 192.168.1.120 and ending in 192.168.1.250< will get assigned as virtual IPs for eth0 interface
Cheers 😉

How to delete entries from routing table on Linux

Thursday, July 7th, 2011

Every now and then I had to tamper routing tables in Linux and every time I had to do it I forgot how I did it last time so finally I decided to put it on my blog and find how I can delete from Linux routing table easier

Deleting a record from a wrong routing table on Linux is a piece of cake basicly, here is an example:

linux:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.1 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0

Let’s say this is the routing table and it’s incorrect as the routing through the node 169.254.0.0 should not be there.
Here is how the routing through 169.254.0.0 can be deleted:

linux:~# /sbin/route del -net 169.254.0.0 netmask 255.255.0.0

Now here is the difference after deletion:

linux:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.1 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0

If you want to delete the default gateway just use 0.0.0.0, e.g. -net 0.0.0.0 .. in above cmd example.
This would delete default gateway record from routing table which by the way in routing tables is marked with the UG flag.
Cheers 😉

How to configure and enable Xen Linux dedicated server’s Virtual machines Internet to work / Enable multipe real IPs and one MAC only in (SolusVM) through NAT routed and iptables

Saturday, June 4th, 2011

Xen Linux Virtual Machine Logo

I’ve been hired as a consultant recently to solve a small task on a newly bought Xen based dedicated server.
The server had installed on itself SolusVM

The server was a good hard-iron machine running with CentOS Linux with enabled Xen virtualization support.
The Data Center (DC) has provided the client with 4 IP public addresses, whether the machine was assigned to possess only one MAC address!

The original idea was the dedicated server is supposed to use 4 of the IP addresses assigned by the DC whether only one of the IPs has an external internet connected ethernet interface with assigned MAC address.

In that case using Xen’s bridging capabilities was pretty much impossible and therefore Xen’s routing mode has to be used, plus an Iptables Network Address Translation or an IP MASQUERADE .

In overall the server would have contained 3 virtual machines inside the Xen installed with 3 copies of:

  • Microsoft Windows 2008

The scenario I had to deal with is pretty much explained in Xen’s Networking wiki Two Way Routed Network

In this article I will describe as thoroughfully as I can how I configured the server to be able to use the 3 qemu virtual machines (running inside the Xen) with their respective real interner visible public IP addresses.

1. Enable Proxyarp for the eth0 interface

To enable proxyarp for eth0 on boot time and in real time on the server issue the commands:

[root@centos ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp[root@centos ~]# echo 'net.ipv4.conf.all.proxy_arp = 1' >> /etc/sysctl.conf

2. Enable IP packet forwarding for eth interfaces

This is important pre-requirement in order to make the iptables NAT to work.

[root@centos ~]# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
[root@centos ~]# echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf

If you get errors during execution of /etc/init.d/xendomains , like for example:

[root@centos ~]# /etc/init.d/xendomains restart
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv4/conf/eth0/proxy_arp: No such file or directory
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv6/conf/eth0/proxy_arp: No such file or directory

in order to get rid of the message you will have to edit /etc/xen/scripts/network-route and comment out the lines:

echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp
e.g.
#echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
#echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp

3. Edit /etc/xen/xend-config.sxp, disable ethernet bridging and enable eth0 routing (route mode) and NAT for Xen’s routed mode

Make absolutely sure that in /etc/xen/xend-config.sxp the lines related to bridging are commented.
The lines you need to comment out are:

(network-script network-bridge)
(vif-script vif-bridge)

make them look like:

#(network-script network-bridge)
#(vif-script vif-bridge)br />

Now as bridging is disabled let’s enable Xen routed network traffic as an bridged networking alternative.

Find the commented (network-script network-route) and (vif-script vif-route) lines and uncomment them:

#(network-script network-route)
#(vif-script vif-route)

The above commented lines should become:

(network-script network-route)
(vif-script vif-route)

Next step is to enable NAT for routed traffic in Xen (necessery to make routed mode work).
Below commented two lines in /etc/xen/xend-config.sxp, should be uncommented e.g.:

#(network-script network-nat)
#(vif-script vif-nat)

Should become:

(network-script network-nat)
(vif-script vif-nat)

4. Restart Xen control daemon and reload installed Xen’s Virtual Machines installed domains

To do so invoke the commands:

[root@centos ~]# /etc/init.d/xend
[root@centos ~]# /etc/init.d/xendomains restart

This two commands will probably take about 7 to 10 minutes (at least they took this serious amount of time in my case).
If you think this time is too much to speed-up the procedure of restarting Xen and qemu attached virtual machines, restart the whole Linux server, e.g.:

[root@centos ~]# restart

5. Configure iptables NAT rules on the CentOS host

After the server boots up, you will have to initiate the following ifconfig & iptables rules in order to make the Iptables NAT to work out:

echo > > /proc/sys/net/ipv4/conf/tap1.0/proxy_arp
/sbin/ifconfig eth0:1 11.22.33.44 netmask 255.255.252.0
/sbin/ifconfig eth0:2 22.33.44.55 netmask 255.255.252.0
/sbin/ifconfig eth0:3 33.44.55.66 netmask 255.255.252.0

/sbin/iptables -t nat -A PREROUTING -d 11.22.33.44 -i eth0 -j DNAT --to-destination 192.168.1.2
/sbin/iptables -t nat -A PREROUTING -d 22.33.44.55 -i eth0 -j DNAT --to-destination 192.168.1.3
/sbin/iptables -t nat -A PREROUTING -d 33.44.55.66 -i eth0 -j DNAT --to-destination 192.168.1.4
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.2 -o eth0 -j SNAT --to-source 11.22.33.44
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.3 -o eth0 -j SNAT --to-source 22.33.44.55
/sbin/iptables -t nat -A POSTROUTING -s 192.168.1.4 -o eth0 -j SNAT --to-source 33.44.55.66

In the above ifconfig and iptables rules the IP addresses:

11.22.33.44, 22.33.44.55, 33.44.55.66 are real IP addresses visible from the Internet.
In the above rules eth0:1, eth0:2 and eth0:3 are virtual ips assigned to the main eth0 interface.

This ifconfig and iptables setup assumes that the 3 Windows virtual machines running inside the Xen dedicated server will be configured to use (local) private network IP addresses:

192.168.1.2, 192.168.1.3 and 192.168.1.4

You will have also to substitute the 11.22.33.44, 22.33.44.55 and 33.44.55.66 with your real IP addreses.

To store the iptables rules permanently on the fedora you can use the iptables-save command:

[root@centos ~]# /sbin/iptables-save

However I personally did not use this approach to save my inserserted iptable rules for later boots but I use my small script set_ips.sh to add virtual interfaces and iptables rules via the /etc/rc.local invokation:

If you like the way I have integrated my virtual eths initiation and iptables kernel firewall inclusion, download my script and set it to run in /etc/rc.local, like so:

[root@centos ~]# cd /usr/sbin
[root@centos sbin]# wget http://www.pc-freak.net/bshscr/set_ips.sh
...
[root@centos ~]# chmod +x /usr/sbin/set_ips.sh
[root@centos ~]# mv set_ips.sh /usr/sbin
[root@centos ~]# echo '/usr/sbin/set_ips.sh' >> /etc/rc.local

Note that you will have to modify my set_ips.sh script to substitute the 11.22.33.44, 22.33.44.55 and 33.44.55.66 with your real IP address.

So far so good, one might think that all this should be enough for the Virtual Machines Windows hosts to be able to connect to the Internet and Internet requests to the virtual machines to arrive, but no it’s not!!

6. Debugging Limited Connectivity Windows LAN troubles on the Xen dedicated server

Even though the iptables rules were correct and the vif route and vif nat was enabled inside the Xen node, as well as everything was correctly configured in the Windows 2008 host Virtual machines, the virtual machines’s LAN cards were not able to connect properly to connect to the internet and the Windows LAN interface kept constantly showing Limited Connectivity! , neither a ping was available to the gateway configured for the Windows VM host (which in my case was: 192.168.1.1).

You see the error with Limited connectivity inside the Windows on below’s screenshot:

Limited Connectivty Windows error Lan Interface, status screenshot

Here is also a screenshot of my VNC connection to the Virtual machine with the correct IP settings – (TCP/IPv4) Properties Window:

Windows Xen Network Connections Windows VNC TCP/IPv4 Properties Window

This kind of Limited Connectivity VM Windows error was really strange and hard to diagnose, thus I started investigating what is wrong with this whole situation and why is not able the Virtualized Windows to connect properly to the Internet, through the Iptables NAT inbound and outbound traffic redirection.

To diagnose the problem, I started up with listing the exact network interfaces showing to be on the Xen Dedicated server:


[root@centos ~]# /sbin/ifconfig |grep -i 'Link encap' -A 1
eth0 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:111.22.33.55 Bcast:111.22.33.255
Mask:255.255.252.0
--
eth0:1 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:11.22.33.44 Bcast:11.22.33.255
Mask:255.255.252.0
--
eth0:2 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:22.33.44.55 Bcast:22.33.44.255
Mask:255.255.252.0
--
eth0:3 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr:33.44.55.66 Bcast:33.44.55.255
Mask:255.255.252.0
--
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
--
tap1.0 Link encap:Ethernet HWaddr FA:07:EF:CA:13:31
--
vifvm101.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr:111.22.33.55 Bcast:111.22.33.55
Mask:255.255.255.255

I started debugging the issue, using the expelling logic.
In the output concerning my interfaces via ifconfig on eth0, I have my primary server IP address 111.22.33.55 , this one is working for sure as I was currently connected to the server through it.

The other virtual IP addresses assigned on the virtual network interfaces eth0:1, eth0:2 and eth0:3 were also assigned correctly as I was able to ping this ips from my Desktop machine from the Internet.

The lo , interface was also properly configured as I could ping without a problem the loopback ip – 127.0.0.1

The rest of the interfaces displayed by my ifconfig output were: tap1.0, vifvm101.0

After a bit of ressearch, I’ve figured out that they’re virtual interfaces and they belong to the Xen domains which are running qemu virtual machines with the Windows host.

I used tcpdump to debug what kind of traffic does flow through the tap1.0 and vifvm101.0 interfaces, like so

[root@centos ~]# tcpdump -i vifvm101.0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vifvm101.0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@centos ~]# tcpdump -i tap1.0
cpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1.0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
08:55:52.490249 IP 229.197.34.95.customer.cdi.no.15685 > 192.168.1.2.12857: UDP, length 42

I’ve figured out as it’s also observable in above’s two tcpdump commands output, that nothing flows through the vifvm101.0 interface, and that there was some traffic passing by tap1.0 interface.

7. Solving the Limited Connectivy Windows Internet network connection problems

As below’s ifconfig output reveals, there is no IP address assigned to tap1.0 interface, using some guidelines and suggestions from guys in irc.freenode.net’s #netfilter irc channel, I’ve decided to give a go to set up an IP address of 192.168.1.1 to tap1.0 .

I choose for a reason as this IP address is configured to be my Gateway’s IP Address inside the Emulated Windows 2008 hosts

To assign the 192.168.1.1 to tap1.0, I issued:

[root@centos ~]# /sbin/ifconfig tap1.0 192.168.1.1 netmask 255.255.255.0
To test if there is difference I logged in to the Virtual Machine host with gtkvncviewer (which by the way is a very nice VNC client for Gnome) and noticed there was an established connection to the internet inside the Virtual Machine 😉

I issued a ping to google which was also returned and opened a browser to really test if everything is fine with the Internet.
Thanks God! I could browse and everything was fine 😉

8. Making tap1.0 192.168.1.1 (VM hosts gateway to be set automatically, each time server reboots)

After rebooting the server the tap1.0 assignmend of 192.168.1.1 disappeared thus I had to make the 192.168.1.1, be assigned automatically each time the CentoS server boots.

To give it a try, I decided to place /sbin/ifconfig tap1.0 192.168.1.1 netmask 255.255.255.0 into /etc/rc.local, but this worked not as the tap1.0 interface got initialized a while after all the xendomains gets initialized.

I tried few times to set some kind of sleep time interval with the sleep , right before the /sbin/ifconfig tap1.0 … ip initialization but this did not worked out, so I finally completely abandoned this methodology and make the tap1.0 get initialized with an IP through a cron daemon.
For that purpose I’ve created a script to be invoked, every two minutes via cron which checked if the tap1.0 interface is up and if not issues the ifconfig command to initialize the interface and assign the 192.168.1.1 IP to it.

Here is my set_tap_1_iface.sh shell script

To set it up on your host in /usr/sbin issue:

[root@centos ~]# cd /usr/sbin/
[root@centos sbin]# wget http://www.pc-freak.net/bshscr/set_tap_1_iface.sh
...
In order to set it on cron to make the tap1.0 initialization automatically every two minutes use the cmd:

[root@centos ~]# crontab -u root -e

After the cronedit opens up, place the set_tap_1_iface.sh cron invokation rules:

*/2 * * * * /usr/sbin/set_tap_1_iface.sh >/dev/null 2>&1

and save.

That’s all now your Xen dedicated and the installed virtual machines with their public internet IPs will work 😉
If this article helped you to configure your NAT routing in Xen drop me a thanks message, buy me a beer or hire me! Cheers 😉

Universal way to configure a static IP address on ethernet lan (eth0) interface in Linux

Friday, April 29th, 2011

One of the most precious commands I ever learned to use in Linux is ifconfig and route .

They have saved my life in configuring the static IP based internet of numerous Desktop Linux computers & notebooks.

Though the usage is very much known by most of the people who are into Linux, I believe it’s likely that the newer people who entered the world of Linux or some Unix system administrators are still lacking the knowledge on how to manually configure their eth0 lan card, thus I thought it might be handy for someone to share it, I know that for most unix users & admins especially the advanced ones this post might be funny, so if you’re an advanced administrator just skip the post and don’t laught at it 😉

Now the universal commands (works on each and every Linux host) to configure manually static IP internet connection on Linux are:

linux:~# /sbin/ifconfig eth0 192.168.0.3 netmask 255.255.255.0
linux:~# /sbin/route add default gw 192.168.0.1
linux:~# echo 'nameserver 192.168.0.1' >> /etc/resolv.conf

I’ve used this simple commands on thousands ot Linux hosts and it’s still handy 🙂

In above example 192.168.0.3 is the static IP address provided by the ISP, netmask is the netmask and the second /sbin/route add default gw would set the default gateway to the example ip 192.168.0.1

The third final line would add up a resolver nameserver the Linux host would use.

Cheers 😉

How to configure static IP address on Lan card eth0 on Ubuntu and Debian Linux

Wednesday, April 27th, 2011

Does your provider provides you with a connection to the internet via a static IP address? Are you an Ubuntu or Debian user like me? Are you looking for a way to configure your eth0 Linux network card with the static ISP provided IP address? That was the scenario with me and in this article I will explain, how you can configure your Home internet access with your Ubuntu/Debian based Linux.

Both Ubuntu and Debian does have a graphic tools, which also can be used to set a static IP address to your network interface, however I find it easier to do it straight from the command line.

To configure your internet static IP via a command line, what you will need to modify is the file:

/etc/network/interfaces

In order to configure a static IP address, your provider should have equipped you with few IP addresses like let’s say the example values below:

Host IP Address: 192.168.0.5
Netmask Address: 255.255.255.0
Gateway: 192.168.0.1
Primary DNS Server: 192.168.0.1
Secondary DNS Server: 192.168.0.2

Now edit with vim, nano or mcedit /etc/network/interfaces e.g.:

root@ubuntu:~# mcedit /etc/network/interfaces

A plain /etc/network/interfaces file should contain something similar to:

auto lo
iface lo inet loopback

In order to be able to set your static IP address, Netmask, Gateway and DNS servers you will have to append in the interfaces file, the settings:

iface eth0 inet static
address 192.168.0.1
netmask 255.255.255.0
network 192.168.0.0
gateway 192.168.0.1

The eth0 sets the lan card on which the values will be assigned, address variable is the IP address assigned by your ISP, netmask is logically the netmask, network should always be configured same as the value set for address but the last ip block should be always .0 , gateway as you already know is the gateway (the ISP router).

One more thing you need to do is to configure your DNS servers by including the DNS ip addresses to /etc/resolv.conf , just issue something like:

root@ubuntu:~# echo 'nameserver 192.168.0.1' >> /etc/resolv.conf
root@ubuntu:~# echo 'nameserver 192.168.0.2' >> /etc/resolv.conf

To test that your new Linux static ip configuration is correct exec:

root@ubuntu:~# /etc/init.d/networking restart

Next use ping or (if ping is disabled by ISP), use matt’s traceroute (mtr) or a browser to test if the Linux is connected to the net.

ubuntu:~# ping google.com
...
ubuntu:~# mtr google.com

If none of the two are not able to show either ping requests flowing around, or routes to google, then something is either wrong with your internet configuration or you forgot to pay your internet bill 😉