Posts Tagged ‘ipv’

Resolving “nf_conntrack: table full, dropping packet.” flood message in dmesg Linux kernel log

Wednesday, March 28th, 2012

On many busy servers, you might encounter in /var/log/syslog or dmesg kernel log messages like

nf_conntrack: table full, dropping packet

to appear repeatingly:

[1737157.057528] nf_conntrack: table full, dropping packet.
[1737157.160357] nf_conntrack: table full, dropping packet.
[1737157.260534] nf_conntrack: table full, dropping packet.
[1737157.361837] nf_conntrack: table full, dropping packet.
[1737157.462305] nf_conntrack: table full, dropping packet.
[1737157.564270] nf_conntrack: table full, dropping packet.
[1737157.666836] nf_conntrack: table full, dropping packet.
[1737157.767348] nf_conntrack: table full, dropping packet.
[1737157.868338] nf_conntrack: table full, dropping packet.
[1737157.969828] nf_conntrack: table full, dropping packet.
[1737157.969928] nf_conntrack: table full, dropping packet
[1737157.989828] nf_conntrack: table full, dropping packet
[1737162.214084] __ratelimit: 83 callbacks suppressed

There are two type of servers, I've encountered this message on:

1. Xen OpenVZ / VPS (Virtual Private Servers)
2. ISPs – Internet Providers with heavy traffic NAT network routers

I. What is the meaning of nf_conntrack: table full dropping packet error message

In short, this message is received because the nf_conntrack kernel maximum number assigned value gets reached.
The common reason for that is a heavy traffic passing by the server or very often a DoS or DDoS (Distributed Denial of Service) attack. Sometimes encountering the err is a result of a bad server planning (incorrect data about expected traffic load by a company/companeis) or simply a sys admin error…

– Checking the current maximum nf_conntrack value assigned on host:

linux:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max

– Alternative way to check the current kernel values for nf_conntrack is through:

linux:~# /sbin/sysctl -a|grep -i nf_conntrack_max
error: permission denied on key 'net.ipv4.route.flush'
net.netfilter.nf_conntrack_max = 65536
error: permission denied on key 'net.ipv6.route.flush'
net.nf_conntrack_max = 65536

– Check the current sysctl nf_conntrack active connections

To check present connection tracking opened on a system:


linux:~# /sbin/sysctl net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 12742

The shown connections are assigned dynamicly on each new succesful TCP / IP NAT-ted connection. Btw, on a systems that work normally without the dmesg log being flooded with the message, the output of lsmod is:

linux:~# /sbin/lsmod | egrep 'ip_tables|conntrack'
ip_tables 9899 1 iptable_filter
x_tables 14175 1 ip_tables

On servers which are encountering nf_conntrack: table full, dropping packet error, you can see, when issuing lsmod, extra modules related to nf_conntrack are shown as loaded:

linux:~# /sbin/lsmod | egrep 'ip_tables|conntrack'
nf_conntrack_ipv4 10346 3 iptable_nat,nf_nat
nf_conntrack 60975 4 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1073 1 nf_conntrack_ipv4
ip_tables 9899 2 iptable_nat,iptable_filter
x_tables 14175 3 ipt_MASQUERADE,iptable_nat,ip_tables


II. Remove completely nf_conntrack support if it is not really necessery

It is a good practice to limit or try to omit completely use of any iptables NAT rules to prevent yourself from ending with flooding your kernel log with the messages and respectively stop your system from dropping connections.

Another option is to completely remove any modules related to nf_conntrack, iptables_nat and nf_nat.
To remove nf_conntrack support from the Linux kernel, if for instance the system is not used for Network Address Translation use:

/sbin/rmmod iptable_nat
/sbin/rmmod ipt_MASQUERADE
/sbin/rmmod rmmod nf_nat
/sbin/rmmod rmmod nf_conntrack_ipv4
/sbin/rmmod nf_conntrack
/sbin/rmmod nf_defrag_ipv4

Once the modules are removed, be sure to not use iptables -t nat .. rules. Even attempt to list, if there are any NAT related rules with iptables -t nat -L -n will force the kernel to load the nf_conntrack modules again.

Btw nf_conntrack: table full, dropping packet. message is observable across all GNU / Linux distributions, so this is not some kind of local distribution bug or Linux kernel (distro) customization.

III. Fixing the nf_conntrack … dropping packets error

– One temporary, fix if you need to keep your iptables NAT rules is:

linux:~# sysctl -w net.netfilter.nf_conntrack_max=131072

I say temporary, because raising the nf_conntrack_max doesn't guarantee, things will get smoothly from now on.
However on many not so heavily traffic loaded servers just raising the net.netfilter.nf_conntrack_max=131072 to a high enough value will be enough to resolve the hassle.

– Increasing the size of nf_conntrack hash-table

The Hash table hashsize value, which stores lists of conntrack-entries should be increased propertionally, whenever net.netfilter.nf_conntrack_max is raised.

linux:~# echo 32768 > /sys/module/nf_conntrack/parameters/hashsize
The rule to calculate the right value to set is:
hashsize = nf_conntrack_max / 4

– To permanently store the made changes ;a) put into /etc/sysctl.conf:

linux:~# echo 'net.netfilter.nf_conntrack_count = 131072' >> /etc/sysctl.conf
linux:~# /sbin/sysct -p

b) put in /etc/rc.local (before the exit 0 line):

echo 32768 > /sys/module/nf_conntrack/parameters/hashsize

Note: Be careful with this variable, according to my experience raising it to too high value (especially on XEN patched kernels) could freeze the system.
Also raising the value to a too high number can freeze a regular Linux server running on old hardware.

– For the diagnosis of nf_conntrack stuff there is ;

/proc/sys/net/netfilter kernel memory stored directory. There you can find some values dynamically stored which gives info concerning nf_conntrack operations in "real time":

linux:~# cd /proc/sys/net/netfilter
linux:/proc/sys/net/netfilter# ls -al nf_log/

total 0
dr-xr-xr-x 0 root root 0 Mar 23 23:02 ./
dr-xr-xr-x 0 root root 0 Mar 23 23:02 ../
-rw-r--r-- 1 root root 0 Mar 23 23:02 0
-rw-r--r-- 1 root root 0 Mar 23 23:02 1
-rw-r--r-- 1 root root 0 Mar 23 23:02 10
-rw-r--r-- 1 root root 0 Mar 23 23:02 11
-rw-r--r-- 1 root root 0 Mar 23 23:02 12
-rw-r--r-- 1 root root 0 Mar 23 23:02 2
-rw-r--r-- 1 root root 0 Mar 23 23:02 3
-rw-r--r-- 1 root root 0 Mar 23 23:02 4
-rw-r--r-- 1 root root 0 Mar 23 23:02 5
-rw-r--r-- 1 root root 0 Mar 23 23:02 6
-rw-r--r-- 1 root root 0 Mar 23 23:02 7
-rw-r--r-- 1 root root 0 Mar 23 23:02 8
-rw-r--r-- 1 root root 0 Mar 23 23:02 9


IV. Decreasing other nf_conntrack NAT time-out values to prevent server against DoS attacks

Generally, the default value for nf_conntrack_* time-outs are (unnecessery) large.
Therefore, for large flows of traffic even if you increase nf_conntrack_max, still shorty you can get a nf_conntrack overflow table resulting in dropping server connections. To make this not happen, check and decrease the other nf_conntrack timeout connection tracking values:

linux:~# sysctl -a | grep conntrack | grep timeout
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_events_retry_timeout = 15
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30

All the timeouts are in seconds. net.netfilter.nf_conntrack_generic_timeout as you see is quite high – 600 secs = (10 minutes).
This kind of value means any NAT-ted connection not responding can stay hanging for 10 minutes!

The value net.netfilter.nf_conntrack_tcp_timeout_established = 432000 is quite high too (5 days!)
If this values, are not lowered the server will be an easy target for anyone who would like to flood it with excessive connections, once this happens the server will quick reach even the raised up value for net.nf_conntrack_max and the initial connection dropping will re-occur again …

With all said, to prevent the server from malicious users, situated behind the NAT plaguing you with Denial of Service attacks:

Lower net.ipv4.netfilter.ip_conntrack_generic_timeout to 60 – 120 seconds and net.ipv4.netfilter.ip_conntrack_tcp_timeout_established to stmh. like 54000

linux:~# sysctl -w net.ipv4.netfilter.ip_conntrack_generic_timeout = 120
linux:~# sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 54000

This timeout should work fine on the router without creating interruptions for regular NAT users. After changing the values and monitoring for at least few days make the changes permanent by adding them to /etc/sysctl.conf

linux:~# echo 'net.ipv4.netfilter.ip_conntrack_generic_timeout = 120' >> /etc/sysctl.conf
linux:~# echo 'net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 54000' >> /etc/sysctl.conf

Share this on

How to disable IPv6 on Debian / Ubuntu / CentOS and RHEL Linux

Friday, December 9th, 2011

I have few servers, which have automatically enabled IPv6 protocols (IPv6 gets automatically enabled on Debian), as well as on most latest Linux distribituions nowdays.

Disabling IPv6 network protocol on Linux if not used has 2 reasons:

1. Security (It’s well known security practice to disable anything not used on a server)
Besides that IPv6 has been known for few criticil security vulnerabilities, which has historically affected the Linux kernel.
2. Performance (Sometimes disabling IPv6 could have positive impact on IPv4 especially on heavy traffic network servers).
I’ve red people claiming disabling IPv6 improves the DNS performance, however since this is not rumors and did not check it personally I cannot positively confirm this.

Disabling IPv6 on all GNU / Linuces can be achieved by changing the kernel sysctl settings net.ipv6.conf.all.disable_ipv6 by default net.ipv6.conf.all.disable_ipv6 equals 1 which means IPv6 is enabled, hence to disable IPv6 I issued:

server:~# sysctl net.ipv6.conf.all.disable_ipv6=0

To set it permanently on system boot I put the setting also in /etc/sysctl.conf :

server:~# echo 'net.ipv6.conf.all.disable = 1 >> /etc/sysctl.conf

The aforedescribed methods should be working on most Linux kernels version > 2.6.27 in that number it should work 100% on recent versions of Fedora, CentOS, Debian and Ubuntu.

To disable IPv6 protocol on Debian Lenny its necessery to blackist the ipv6 module in /etc/modprobe.d/blacklist by issuing:

echo 'blacklist ipv6' >> /etc/modprobe.d/blacklist

On Fedora / CentOS there is a another universal “Redhat” way disable IPv6.

On them disabling IPv6 is done by editting /etc/sysconfig/network and adding:


I would be happy to hear how people achieved disabling the IPv6, since on earlier and (various by distro) Linuxes the way to disable the IPv6 is probably different.

Alto to stop Iptables IPV6 on CentOS / Fedora and RHEL issue:

# service ip6tables stop

# service ip6tables off

Share this on

How to install Microsoft Windows XP SP3 on Debian GNU / Linux Squeeze

Thursday, December 1st, 2011

Windows XP Service Pack 3 QEMU Debian Squeeze

I have never did a proper install of Windows XP on Debian before hand. Even though I experimented once long time ago. I had zero success with installing Windows XP Service Pack 2 . The only Windows I can make correctly working before hand on these early days on my Debian powered notebook with qemu virtual machine emulator was Windows 2000 .

I decided to give it another go today as I hoped the qemu has advanced and I’ve seen many reports online of people who were able to correctly make Windows XP SP2 work out.

As I’ve seen many blog posts online of people who succesfully run with qemu Windows XP SP2, in order to escape from repeating the other guys experience and conduct a fresh experiment, I decided to give qemu a try with Microsoft.Windows.XP.Professional.SP3.Integrated.June.2011.Corporate

Before I proceed with using latest qemu I,

1. Installed qemu using the usual:

debian:~# apt-get install --yes qemu qemu-keymaps qemu-system qemu-user qemu-utils uml-utilities


2. Created a new directory where the qemu Windows image will be stored:

debian:~# su hipo
hipo@noah:~$ mkdir windows
hipo@noah:~$ cd windows

As a following step I loaded the tun kernel module which is necessery for Qemu to properly handle the Windows LAN networking.

3. Load and set proper permissions for tun kernel module

In case if /dev/net is not existing first step is to create the proper device, however in most cases /dev/net should be there:

debian:~# mkdir -p /dev/net
debian:~# mknod /dev/net/tun c 10 200

As a next step its necessery to load tun kernel module and set the proper permissions:

debian:~# modprobe tun
debian:~# echo 'tun' >> /etc/modules
debian:~# chgrp users /dev/net/tun
debian:~# chmod g+w /dev/net/tun

Next step is to create an image file with dd or with qemu-img which will be holding the Virtual Machine Windows installation.

4. Create image file for Windows using dd

I decided to create a the image file to be with a size of 5 Gigabytes, this is of course custom so other people might prefer having it less or more the absolute minimum for a proper Windows XP SP3 install is 2000 Megabytes.

debian:~# su hipo -; cd windows;
debian:/home/hipo/windows$ dd of=hd.img bs=1024 seek=5000000 count=0
0+0 records in
0+0 records out
0 bytes (0 B) copied, 1.5505e-05 s, 0.0 kB/s

Notice here that the file dd will create will appear like 0 kb file until the Windows install from a BootCD is run with qemu.

5. Download an image of Microsoft.Windows.XP.Professional.SP3.Integrated.June.2011.Corporate from

Microsoft.Windows.XP.Professional.SP3.Integrated.June.2011.Corporate is currently available for download from the if in the times to come it is not available it will most likely be available from, etc. so I’ll skip more explanations with this step and let you use your favourite torrent program of choice to download the MS Windows iso. Just to make a note here I used transmission as this is my favourite torrent client. After downloading the iso I used K3B to burn the Image file as Bootable ISO. I’m naturally a GNOME user so to burn it as Image I just open it with K3B by using the GNOME menu and selecting Open with K3B

Next I instructed qemu to boot from the just burnt CD.

6. Boot windows Installation with Qemu from the Boot CD

debian:/home/hipo/windows$ qemu -boot d -cdrom /dev/cdrom -hda hd.img

Notice here that I’m running the qemu virtual machine emulator with a non-privileged reasons. This is important as qemu might have holes in the emulation of Windows Networking stack which if executed as root superuser. Can allow some malicious attacker to remotely compromise your GNU / Linux PC …

Qemu window will pop-up where one installs the Windows as it will install it using an ordinary PC. To switch qemu to fullscreen mode to have the complete feeling like installing Windows on an non-emulated PC ctrl + alt + f can be pressed.

The Windows installation took like 1 hour 20 minutes on my dual core 1.8 Ghz notebook with 2 GB of RAM. But I should say while installing I had multiple applications running; xmms, transmission, epiphany, icedove, evince etc. probably if I just run the Virtual Machine with no other applications to extra load my PC, probably the Windows install would have been done in max 50 minutes time.

After the installation is complete. To

7. Further run the installed Windowsdebian:/home/hipo/windows$ qemu -hda hd.img -boot c

As a next step its necessery to;

8. Bring up the tap0 interface and configure it for the user

I’m running my qemu emulator with my user hipo , so I run cmds:

debian:/home/hipo/windows$ su - root
debian:~# tunctl -u hipo
Set 'tap0' persistent and owned by uid 1000

9. Enable ip_forwarding and arp proxy and for wlan0 and tap0

debian:~# echo 1 > /proc/sys/net/ipv4/conf/wlan0/proxy_arp
debian:~# echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp
debian:~# echo 1 > /proc/sys/net/ipv4/conf/tap0/proxy_arp

10. Install the proper Network Drivers inside Windows

That’s just in case, if they’re not supported by the Windows default existing drivers.
To do so, I downloaded my LAN drivers from the Vendor and put it on USB and sticked the USB drive to my laptop. In order to make the Kingston USB drive I used to transfer my LAN and Video drivers. I had to restart qemu with the parameter -usb -usbdevice host:0951:1625 , where I used lsusb to check and get the correct USB ID 0951:1625, like shown in the command below:

debian:~# lsusb |grep -i kingston
Bus 001 Device 006: ID 0951:1625 Kingston Technology DataTraveler 101 II

After on I booted again the Windows XP with the following command line in order to make qemu detect the USB Drive:

debian:/home/hipo/windows# sudo qemu -boot c -hda hd.img -usb -usbdevice host:0951:1625

One oddity here is that in order for qemu to detect the USB stick, I had to run it via sudo with super user privileges.Don’t ask me why this is the only way it worked …
Next on used the Windows device manager from Control Panel -> System -> Device Manager to point my undetected hardware to the correct Win drivers.

For the GUI preferring user qemu has a nice GNOME GUI interface called qemu-launcher, if you like to use qemu via it instead of scripting the qemu launcher commands, you can install and use via:

debian:~# apt-get install --yes qemu-launcher qemuctl
debian:/home/hipo/windows$ qemu-launcher

QEMU-Launcher Debian Squeeze Screenshot

Another GUI alternative to qemu-launcher, which easifys the work with qemu is qemulator; here is a screenshot:

Qemulator 0.5 - Qemu GUI Screenshot Debian Squeeze

Share this on

Possible way to increase Linux TCP/IP port thoroughput via sysctl kernel variable

Thursday, August 25th, 2011

Sysctl is a great way to optimize Linux. sysctl has a dozens of values which could drastically improve server networking and overall performance.

One of the many heplful variables to optimize the way the Linuz kernel works on busy servers is net.ipv4.ip_local_port_range .

The default sysctl setting for net.ipv4.ip_local_port_range on Debian, Ubuntu Fedora, RHEL, CentOS is:

net.ipv4.ip_local_port_range = 32768 65536

This means that the kernel and the corresponding server running services instructing the Linuz kernel open new port sockets can only open local ports in the range of 32768 – 65536 .
On a regular Desktop GNU/Linux machine or a not high iron server this settins is perfectly fine, however on a high scale servers the local port range in the interval of 32768-65536 might be insufficient at times, especially if there are programs which require binding of many local ports.

Therefore on a high load servers, generally it’s a good to raise the port range to be assigned by kernel to 8912 – 65536 , to do so the setting has to be changed like shown below:

linux:~# sysctl -w net.ipv4.ip_local_port_range = 8192 65536

If changing this setting on the server doesn’t show any negative impact on performance in few hours time or a day or even better decreases the server average load, it’s a good idea that it be added to sysctl.conf to load up the setting on next kernel boot.

linux:~# echo 'net.ipv4.ip_local_port_range' >> /etc/sysctl.conf

Enjoy 😉

Share this on

How to make NAT enable hosts in a local network to access the internet, create port forwarding to local IPs behind the router using iptables

Tuesday, August 23rd, 2011

I’m bulding new iptables firewall on one Linux server. The Debian GNU/Linux is required to act as firewall do Network Adress Translation for a small network of office PCs as well as forward some of the inbound ports to hosts from the local network located behind the router.

The local network besides the router had an IP addressing in the class C network e.g. (

First I procceded and enabled the Network Address Translation via the Linux kernel variable:

linux:~# sysctl -w net.ipv4.ip_forward=1
linux:~# echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf

Initially I even forgot to switch on the net.ipv4.ip_forward to 1 (by default this value is set to 0) – GNU/Linux’s default network behaviour is not predetermined to act as network router.
However, since I haven’t configured Network Address Translation for quite some time it completely slipped my mind!

Anyways next the actual iptables rule which makes NAT possible I used is:

linux:~# /sbin/iptables -t nat -A POSTROUTING -s ! -d -j SNAT --to-source

Whether is the External IP address assigned to the router on eth0

With this very simple rules now Network the local network is capable of accessing the Internet withotu problem.

It’s a good time to say that still many system administrators, still erroneously use MASQUERADE rules instead of SNAT .
IP MASQUERADING is an ancestry from ipchains and these days should be completely abandonded, especially where no often change of primary IP address to access the internet is made.
For dial-ups or other kind of networking, where the IP addresses are often changed still IP MASQUERADING might be a good idea though.

My next goal was to make the Linux router to do port forwarding of Traffic which arrives on port 80 to a IIS server assigned with a local IP address of
I did the webserver (port 80), port forwarding from IP to with the iptables rule:

linux:~# /sbin/iptables -t nat -A PREROUTING -d -p tcp -m tcp --dport 80 -j DNAT --to-destination

There was a requirement to do port forwarding for a Windows remote Desktop running on standard port 3389 from the router to the internal Windows IP address running the IIS webserver, however the company required me to only allow access to the rdesktop 3389 port to certain real IP addresses.
Initially I thought about using the above PREROUTING rule which makes the port redirection to the IIS server and only change port 80 to port 3389 , and then use filter table INPUT chain rules like:

/sbin/iptables -A INPUT -s xx1.xx2.xx3.xx4,1xx,2xx,3xx,4xx, -p tcp -m tcp --dport 3389 -j ACCEPT/sbin/iptables -A INPUT -p tcp -m tcp --dport 3389 -j REJECT --reject-with icmp-port-unreachable

However this did not work out, so I decided to give a try to do the same within the filter table using the FORWARD chain, like so:

FORWARD/sbin/iptables -A FORWARD -p tcp -m tcp -s xx1.xx2.xx3.xx4,1xx,2xx,3xx,4xx, -p tcp -m tcp --dport 3389 -j ACCEPT
/sbin/iptables -A FORWARD -p tcp -m tcp --dport 3389 -j REJECT --reject-with icmp-port-unreachable

Adding this rules did not added any filtering to the forwarded remote desktop port. I suspected that somehow probably my above PREROUTING nat rules are read before any other rules and therefore automatically allows any IP address to port fortward traffic.
I’ve checked the iptables documentation and it seems my guess was partially right.

When some kind of network traffic enters the iptables firewall it first goes through the PREROUTING channel and then the traffic flows in a certain order.
iptables packet flow diagram

The iptables network packets flow is clearly seen in above’s diagram a thorough looks gives a very good idea on how packet is being processed by iptables

Finally as I couldn’t think about a good solution on how to only filter the port redirected traffic, which always firstly entered in the POSTROUTING chain, I’ve consulted with the guys in in #Netfilter.

I’m quite thanksful as a guy nicknamed Olipro has given me a pretty good picture on the port forwarding POSTROUTING problem and has provided me with a very logical easy and great fix.
He suggested that I only do port forwarding for certain IP addresses instead of allowing all IP addresses and then lookup for a way to allow only some of them and filter the rest.

The iptables rule to restrict the incoming traffic to the remote desktop forwarded port 3389 to few only allowed IP addresses looks like so:

linux:~# /sbin/iptables -t nat -A PREROUTING -d -s xx1.xx2.xx3.xx4,1xx,2xx,3xx,4xx, -p tcp -m tcp –dport 3389 -j DNAT –to-destination

Now the three sample IPs passed xx1.xx2.xx3.xx4,1xx,2xx,3xx,4xx, has added to port forward traffic on 3389 to

By the way I did not know that newer versions of iptables support passing by multiple IP addresses to the –source or –destination IP. This is really great feature I’ve learned from the good guys from #Netfilter. However one should be careful when using the multiple IPs with -s or -d, it’s really important that the passed consequent IPs has no space between the , delimiter.

Now that’s all my task is completed. All computerse inside the Network on the Linux router freely can access the Internet, all IPs are also capable to access the IIS server located behind the NAT as well as only certain IPs are capable of accessing to the IIS remote desktop.
Hope the article helps somebody 😉

Share this on

How to harden Linux Security and imprpove network efficiency on Kernel sysctl Level to Stop SYN flood

Friday, July 8th, 2011

Power up Linux and protect against DDoS with sysctl var optimization

Some long time ago I’ve written an article Optimizing Linux tcp/ip networking

In the article I’ve examined a number of Linux kernel sysctl variables, which significantly improve the way TCP/IP networking is handled by a non router Linux based servers.

As the time progresses I’ve been continuing to read materials on blogs and internet sites on various tips and anti Denial of Service rules which one could apply on newly installed hosting (Apache/MySql/Qmail/Proxy) server to improve webserver responce times and tighten the overall security level.

In my quest for sysctl 😉 I found a few more handy sysctl variables apart from the old ones I incorporate on every Linux server I adminstrate.
The sysctl variables improves the overall network handling efficiency and protects about common SYN/ACK Denial of service attacks.

Here are the extra sysctl variables I started incorporating just recently:

############ IPv4 Sysctl Settings ################
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routers
net.ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignments
net.ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interface
net.ipv6.conf.default.max_addresses = 1


To use this settings paste the above sysctl variables in /etc/sysctl.conf and ask sysctl command to read and apply the newly added conf settings:

server:~# sysctl -p

Hopefully you should not get errors while applying the sysctl settings, if you get some errors, it’s possible some of the variable is differently named (depending on the Linux kernel version) or the Linux distribution on which sysctl’s are implemented.

For some convenience I’ve created unified sysctl variables /etc/sysct.conf containing the newly variables I started implementing to servers with the ones I already exlpained in my previous post Optimizing Linux TCP/IP Networking

Here is the optimized / hardened sysctl.conf file for download

I use this exact sysctl.conf these days on both Linux hosting / VPS / Mail servers etc. as well as on my personal notebook 😉

Here is also the the complete content of above’s sysctl.conf file, just in case if somebody wants to directly copy/paste it in his /etc/sysctl.conf

# Sysctl kernel variables to improve network performance and protect against common Denial of Service attacks
# It's possible that not all of the variables are working on all Linux distributions, test to make sure
# Some of the variables might need a slight modification to match server hardware, however in most cases it should be fine
# variables list compiled by hip0
#### date 08.07.2011
############ IPv4 Sysctl Kernel Settings ################
net.ipv4.ip_forward = 0
# ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0
# ( same as above )
net.ipv4.conf.default.accept_source_route = 0
# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
# ( - || - )net.ipv4.tcp_fin_timeout = 40
# ( Decrease FIN timeout ) - Useful on busy/high load server
net.ipv4.tcp_keepalive_time = 4000
# ( keepalive tcp timeout )
net.core.rmem_default = 786426
# Receive memory stack size ( a good idea to increase it if your server receives big files )
##net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608
#( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
# ( maximum amount of option memory buffers )
# tcp reordering, increase max buckets, increase the amount of backlost
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
##net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routersnet.
ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignmentsnet.
ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interfacenet.
ipv6.conf.default.max_addresses = 1
# Reboot if kernel panic
kernel.panic = 20

These sysctl settings will tweaken the Linux kernel default network settings performance and you will notice the improvements in website responsiveness immediately in some cases implementing this kernel level goodies will make the server perform better and the system load might decrease even 😉

This optimizations on a kernel level are not only handy for servers, their implementation on Linux Desktop should also have a positive influence on the way the network behaves and could improve significantly the responce times of opening pages in Firefox/Opera/Epiphany Torrent downloads etc.

Hope this kernel tweakenings are helpful to someone.
Cheers 😉

Share this on

How to configure and enable Xen Linux dedicated server’s Virtual machines Internet to work / Enable multipe real IPs and one MAC only in (SolusVM) through NAT routed and iptables

Saturday, June 4th, 2011

Xen Linux Virtual Machine Logo

I’ve been hired as a consultant recently to solve a small task on a newly bought Xen based dedicated server.
The server had installed on itself SolusVM

The server was a good hard-iron machine running with CentOS Linux with enabled Xen virtualization support.
The Data Center (DC) has provided the client with 4 IP public addresses, whether the machine was assigned to possess only one MAC address!

The original idea was the dedicated server is supposed to use 4 of the IP addresses assigned by the DC whether only one of the IPs has an external internet connected ethernet interface with assigned MAC address.

In that case using Xen’s bridging capabilities was pretty much impossible and therefore Xen’s routing mode has to be used, plus an Iptables Network Address Translation or an IP MASQUERADE .

In overall the server would have contained 3 virtual machines inside the Xen installed with 3 copies of:

  • Microsoft Windows 2008

The scenario I had to deal with is pretty much explained in Xen’s Networking wiki Two Way Routed Network

In this article I will describe as thoroughfully as I can how I configured the server to be able to use the 3 qemu virtual machines (running inside the Xen) with their respective real interner visible public IP addresses.

1. Enable Proxyarp for the eth0 interface

To enable proxyarp for eth0 on boot time and in real time on the server issue the commands:

[root@centos ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp[root@centos ~]# echo 'net.ipv4.conf.all.proxy_arp = 1' >> /etc/sysctl.conf

2. Enable IP packet forwarding for eth interfaces

This is important pre-requirement in order to make the iptables NAT to work.

[root@centos ~]# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
[root@centos ~]# echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf

If you get errors during execution of /etc/init.d/xendomains , like for example:

[root@centos ~]# /etc/init.d/xendomains restart
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv4/conf/eth0/proxy_arp: No such file or directory
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv6/conf/eth0/proxy_arp: No such file or directory

in order to get rid of the message you will have to edit /etc/xen/scripts/network-route and comment out the lines:

echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp
#echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
#echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp

3. Edit /etc/xen/xend-config.sxp, disable ethernet bridging and enable eth0 routing (route mode) and NAT for Xen’s routed mode

Make absolutely sure that in /etc/xen/xend-config.sxp the lines related to bridging are commented.
The lines you need to comment out are:

(network-script network-bridge)
(vif-script vif-bridge)

make them look like:

#(network-script network-bridge)
#(vif-script vif-bridge)br />

Now as bridging is disabled let’s enable Xen routed network traffic as an bridged networking alternative.

Find the commented (network-script network-route) and (vif-script vif-route) lines and uncomment them:

#(network-script network-route)
#(vif-script vif-route)

The above commented lines should become:

(network-script network-route)
(vif-script vif-route)

Next step is to enable NAT for routed traffic in Xen (necessery to make routed mode work).
Below commented two lines in /etc/xen/xend-config.sxp, should be uncommented e.g.:

#(network-script network-nat)
#(vif-script vif-nat)

Should become:

(network-script network-nat)
(vif-script vif-nat)

4. Restart Xen control daemon and reload installed Xen’s Virtual Machines installed domains

To do so invoke the commands:

[root@centos ~]# /etc/init.d/xend
[root@centos ~]# /etc/init.d/xendomains restart

This two commands will probably take about 7 to 10 minutes (at least they took this serious amount of time in my case).
If you think this time is too much to speed-up the procedure of restarting Xen and qemu attached virtual machines, restart the whole Linux server, e.g.:

[root@centos ~]# restart

5. Configure iptables NAT rules on the CentOS host

After the server boots up, you will have to initiate the following ifconfig & iptables rules in order to make the Iptables NAT to work out:

echo > > /proc/sys/net/ipv4/conf/tap1.0/proxy_arp
/sbin/ifconfig eth0:1 netmask
/sbin/ifconfig eth0:2 netmask
/sbin/ifconfig eth0:3 netmask

/sbin/iptables -t nat -A PREROUTING -d -i eth0 -j DNAT --to-destination
/sbin/iptables -t nat -A PREROUTING -d -i eth0 -j DNAT --to-destination
/sbin/iptables -t nat -A PREROUTING -d -i eth0 -j DNAT --to-destination
/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source
/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source
/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source

In the above ifconfig and iptables rules the IP addresses:,, are real IP addresses visible from the Internet.
In the above rules eth0:1, eth0:2 and eth0:3 are virtual ips assigned to the main eth0 interface.

This ifconfig and iptables setup assumes that the 3 Windows virtual machines running inside the Xen dedicated server will be configured to use (local) private network IP addresses:, and

You will have also to substitute the, and with your real IP addreses.

To store the iptables rules permanently on the fedora you can use the iptables-save command:

[root@centos ~]# /sbin/iptables-save

However I personally did not use this approach to save my inserserted iptable rules for later boots but I use my small script to add virtual interfaces and iptables rules via the /etc/rc.local invokation:

If you like the way I have integrated my virtual eths initiation and iptables kernel firewall inclusion, download my script and set it to run in /etc/rc.local, like so:

[root@centos ~]# cd /usr/sbin
[root@centos sbin]# wget
[root@centos ~]# chmod +x /usr/sbin/
[root@centos ~]# mv /usr/sbin
[root@centos ~]# echo '/usr/sbin/' >> /etc/rc.local

Note that you will have to modify my script to substitute the, and with your real IP address.

So far so good, one might think that all this should be enough for the Virtual Machines Windows hosts to be able to connect to the Internet and Internet requests to the virtual machines to arrive, but no it’s not!!

6. Debugging Limited Connectivity Windows LAN troubles on the Xen dedicated server

Even though the iptables rules were correct and the vif route and vif nat was enabled inside the Xen node, as well as everything was correctly configured in the Windows 2008 host Virtual machines, the virtual machines’s LAN cards were not able to connect properly to connect to the internet and the Windows LAN interface kept constantly showing Limited Connectivity! , neither a ping was available to the gateway configured for the Windows VM host (which in my case was:

You see the error with Limited connectivity inside the Windows on below’s screenshot:

Limited Connectivty Windows error Lan Interface, status screenshot

Here is also a screenshot of my VNC connection to the Virtual machine with the correct IP settings – (TCP/IPv4) Properties Window:

Windows Xen Network Connections Windows VNC TCP/IPv4 Properties Window

This kind of Limited Connectivity VM Windows error was really strange and hard to diagnose, thus I started investigating what is wrong with this whole situation and why is not able the Virtualized Windows to connect properly to the Internet, through the Iptables NAT inbound and outbound traffic redirection.

To diagnose the problem, I started up with listing the exact network interfaces showing to be on the Xen Dedicated server:

[root@centos ~]# /sbin/ifconfig |grep -i 'Link encap' -A 1
eth0 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
eth0:1 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
eth0:2 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
eth0:3 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
lo Link encap:Local Loopback
inet addr: Mask:
tap1.0 Link encap:Ethernet HWaddr FA:07:EF:CA:13:31
vifvm101.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr: Bcast:

I started debugging the issue, using the expelling logic.
In the output concerning my interfaces via ifconfig on eth0, I have my primary server IP address , this one is working for sure as I was currently connected to the server through it.

The other virtual IP addresses assigned on the virtual network interfaces eth0:1, eth0:2 and eth0:3 were also assigned correctly as I was able to ping this ips from my Desktop machine from the Internet.

The lo , interface was also properly configured as I could ping without a problem the loopback ip –

The rest of the interfaces displayed by my ifconfig output were: tap1.0, vifvm101.0

After a bit of ressearch, I’ve figured out that they’re virtual interfaces and they belong to the Xen domains which are running qemu virtual machines with the Windows host.

I used tcpdump to debug what kind of traffic does flow through the tap1.0 and vifvm101.0 interfaces, like so

[root@centos ~]# tcpdump -i vifvm101.0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vifvm101.0, link-type EN10MB (Ethernet), capture size 96 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@centos ~]# tcpdump -i tap1.0
cpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1.0, link-type EN10MB (Ethernet), capture size 96 bytes
08:55:52.490249 IP > UDP, length 42

I’ve figured out as it’s also observable in above’s two tcpdump commands output, that nothing flows through the vifvm101.0 interface, and that there was some traffic passing by tap1.0 interface.

7. Solving the Limited Connectivy Windows Internet network connection problems

As below’s ifconfig output reveals, there is no IP address assigned to tap1.0 interface, using some guidelines and suggestions from guys in’s #netfilter irc channel, I’ve decided to give a go to set up an IP address of to tap1.0 .

I choose for a reason as this IP address is configured to be my Gateway’s IP Address inside the Emulated Windows 2008 hosts

To assign the to tap1.0, I issued:

[root@centos ~]# /sbin/ifconfig tap1.0 netmask
To test if there is difference I logged in to the Virtual Machine host with gtkvncviewer (which by the way is a very nice VNC client for Gnome) and noticed there was an established connection to the internet inside the Virtual Machine 😉

I issued a ping to google which was also returned and opened a browser to really test if everything is fine with the Internet.
Thanks God! I could browse and everything was fine 😉

8. Making tap1.0 (VM hosts gateway to be set automatically, each time server reboots)

After rebooting the server the tap1.0 assignmend of disappeared thus I had to make the, be assigned automatically each time the CentoS server boots.

To give it a try, I decided to place /sbin/ifconfig tap1.0 netmask into /etc/rc.local, but this worked not as the tap1.0 interface got initialized a while after all the xendomains gets initialized.

I tried few times to set some kind of sleep time interval with the sleep , right before the /sbin/ifconfig tap1.0 … ip initialization but this did not worked out, so I finally completely abandoned this methodology and make the tap1.0 get initialized with an IP through a cron daemon.
For that purpose I’ve created a script to be invoked, every two minutes via cron which checked if the tap1.0 interface is up and if not issues the ifconfig command to initialize the interface and assign the IP to it.

Here is my shell script

To set it up on your host in /usr/sbin issue:

[root@centos ~]# cd /usr/sbin/
[root@centos sbin]# wget
In order to set it on cron to make the tap1.0 initialization automatically every two minutes use the cmd:

[root@centos ~]# crontab -u root -e

After the cronedit opens up, place the cron invokation rules:

*/2 * * * * /usr/sbin/ >/dev/null 2>&1

and save.

That’s all now your Xen dedicated and the installed virtual machines with their public internet IPs will work 😉
If this article helped you to configure your NAT routing in Xen drop me a thanks message, buy me a beer or hire me! Cheers 😉

Share this on

Optimizing Linux TCP/IP Networking to increase Linux Servers Performance

Tuesday, April 8th, 2008


Some time ago I thought of ways to optimize my Linux Servers network performance.

Even though there are plenty of nice articles on the topic on how to better optimize Linux server performance by tunning up the kernel sysctl (variables).

Many of the articles I found was not structed in enough understandable way so I decided togoogle around and  found few interesting websites which gives a good overview on how one can speed up a bit and decrease overall server loads by simply tuning few basic kernel sysctl variables.

Below article is a product of my research on the topic on how to increase my GNU / Linux servers performance which are mostly running LAMP (Linux / Apache / MySQL / PHP) together with Qmail mail servers.

The article is focusing on Networking as networking is usual bottleneck for performance.
Below are the variables I found useful for optimizing the Linux kernel Network stack.

Implementing the variables might reduce your server load or if not decrease server load times and CPU utilization, they would at lease increase thoroughput so more users will be able to access your servers with (hopefully) less interruptions.
That of course would save you some Hardware costs and raise up your Servers efficiency.

Here are the variables themselves and some good example:

# = 0 ( Turn off IP Forwarding )

net.ipv4.conf.default.rp_filter = 1

# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0

# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0 ( same as above )
net.ipv4.conf.default.accept_source_route = 0

# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
( - || - )net.ipv4.tcp_fin_timeout = 40

# ( Decrease FIN timeout ) - Useful on busy/high load
serversnet.ipv4.tcp_keepalive_time = 4000 ( keepalive tcp timeout )
net.core.rmem_default = 786426 - Receive memory stack size ( a good idea to increase it if your server receives big files )
net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608 ( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
( maximum amount of option memory buffers )

# like a homework investigate by yourself what the variables below stand for :)
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024


# Below are newly added experimental
#net.core.rmem_max = 16777216
#net.core.wmem_max = 16777216
##kernel.msgmni = 1024
##kernel.sem = 250 256000 32 1024


Also a good sysctl.conf file which one might want to substitite or use as a skele for some productive server is ready for download here

Even if you can't reap out great CPU reduction benefits from integrating above values or similar ones, your overall LAMP performance to end customers should increase – at some occasions dramatically, at others little bit but still noticable.

If you're unsure on exact kernel variable values to use check yourself what should be the best values that fits you according to your server Hardware – usually this is done by experimenting and reading the kernel documentation as provided for each one of uplisted variables.

Above sysctl.conf is natively created to run on Debian and on other distributions like CentOS, Fedora Slackware some values might either require slight modifications.

Hope this helps and gives you some idea of how network optimization in Linux is usually done. Happy (hacking) tweakening !

Share this on