Posts Tagged ‘ips’

How to install and configure AIDE ( Advanced Intrusion Detection Environment ) on Debian GNU / Linux 11 to monitor files for changes

Thursday, March 9th, 2023


How to install and configure AIDE ( Advanced Intrusion Detection Environment ) on Debian GNU / Linux 11 to monitor files for changes

Having a intrusion detection system is essential to keeping a server security to good level and being compliant with PCI (Payment Card Industry) DSS Standards. It is a great thing for the sake to protect oneself from hackers assaults. 

There is plenty of Intrusion Detection systems available all around since many years, in the past one of main ones for Linux as older system administrators should remember was Tripwire – integrity tool for monitoring and alerting on specific file change(s) on a range of systems

Tripwire is still used today but many today prefer to use AIDE that is a free software replacement for Tripwire under GPL (General Public License), that is starting to become like a "standard"  for many Unix-like systems as an inexpensive baseline control and rootkit detection system.

In this article I'll explain shortly how to Install / Configure and Use AIDE to monitor, changes with files on the system.

But before proceeding it is worthy to mention on some of the alternatives companies and businesses choose to as an IDS (Intrusion Detection Systems), that is useful to give a brief idea of the sysadmins that has to deal with Security, on what is some of the main Intrusion Detection Systems adopted on UNIX OSes today:

  • Samhain

    An integrity checker and host intrusion detection system that can be used on single hosts as well as large, UNIX-based networks. It supports central monitoring as well as powerful (and new) stealth features to run undetected in memory, using steganography. Samhain is an open-source multiplatform application for POSIX systems (Unix, Linux, Cygwin/Windows).

  • OSSEC 
    OSSEC uses a centralized, cross-platform architecture allowing multiple systems to be monitored and managed.
  • Snort
    IDS which has the capabilities to prevent attacks. By taking a particular action based on traffic patterns, it can become an intrusion prevention system (IPS). – written in Pure C.
  • Zeek (Bro)
    Zeek helps to perform security monitoring by looking into the network's activity. It can find suspicious data streams. Based on the data, it alert, react, and integrate with other tools – written in C++.
  • Maltrail (Maltrail monitors for traffic on the network that might indicate system compromise or other bad behavior. It is great for intrusion detection and monitoring. – written in Python).

1. Install aide deb package

# apt -y install aide

root@haproxy2:~# aide -v
Aide 0.17.3

Compiled with the following options:


Default config values:
config file: <none>
database_in: <none>
database_out: <none>

Available hashsum groups:
md5: yes
sha1: yes
sha256: yes
sha512: yes
rmd160: yes
tiger: yes
crc32: yes
crc32b: yes
haval: yes
whirlpool: yes
gost: yes
stribog256: no
stribog512: no

Default compound groups:
R: l+p+u+g+s+c+m+i+n+md5+acl+selinux+xattrs+ftype+e2fsattrs+caps
L: l+p+u+g+i+n+acl+selinux+xattrs+ftype+e2fsattrs+caps
>: l+p+u+g+i+n+acl+S+selinux+xattrs+ftype+e2fsattrs+caps
H: md5+sha1+rmd160+tiger+crc32+haval+gost+crc32b+sha256+sha512+whirlpool
X: acl+selinux+xattrs+e2fsattrs+caps

2. Prepare AIDE configuration and geenrate (initialize) database

Either you can use the default AIDE configuration which already has a preset rules for various files and directories to be monitored,
or you might add up additional ones.

  • For details on configuration of aide.conf accepted options "man aide.conf"

The rules and other configurations resides lays under  /etc/aide/ directory

The AIDE database is located under /var/lib/aide

root@server:~# ls -al /var/lib/aide/
общо 33008
drwxr-xr-x  2 root root     4096  9 мар 12:38 ./
drwxr-xr-x 27 root root     4096  9 мар 12:01 ../
-rw——-  1 root root 16895467  9 мар 16:03 aide.db
-rw——-  1 root root 16895467  9 мар 18:49

Also, details about major setting rules config regarding how AIDE will run via cronjob as with most debian services are into /etc/default/aide

Default aide.conf config is in /etc/aide/aide.conf if you need custom stuff to do with it simply edit it.

Here is an Example:
Lets say you want to omit some directory to not be monitored by aide, which would otherwise do, i.e.
omit /var/log/* from monitoring

# At the end of file /etc/aide/aide.conf



  • Initialize the aide database first time

Run aideinit command, aideinit will create a new baseline database –  /var/lib/aide/ (a baseline)
Note that, /var/lib/aide/aide.db is the old database that aide uses to check against for any changes of files / directories on the configured monitored filesystem objects.

root@server:~# aideinit
Running aide –init…

debug1: client_input_channel_req: channel 0 rtype reply 1
debug1: client_input_channel_req: channel 0 rtype reply 1
debug1: client_input_channel_req: channel 0 rtype reply 1
debug1: client_input_channel_req: channel 0 rtype reply 1
debug1: client_input_channel_req: channel 0 rtype reply 1
Start timestamp: 2023-03-09 12:06:16 +0200 (AIDE 0.17.3)
AIDE initialized database at /var/lib/aide/

Number of entries:      66971

The attributes of the (uncompressed) database(s):

 SHA256    : nVrYljiBFM/KaKCTjbaJtR2w6N8vc8qN
 SHA512    : S1ZNB0DCqb4UTmuqaalTgiQ3UAltTOzO
 RMD160    : xaUnfW1+/DJV/6FEm/nn1k1UKOU=
 TIGER     : nGYEbX281tsQ6T21VPx1Hr/FwBdwF4cK
 CRC32     : fzf7cg==
 HAVAL     : yYQw/87KUmRiRLSu5JcEIvBUVfsW/G9H
 WHIRLPOOL : 6b5y42axPjpUxWFipUs1PtbgP2q0KJWK
 GOST      : sHAzx7hkr5H3q8TCSGCKjndEiZgcvCEL

End timestamp: 2023-03-09 12:38:30 +0200 (run time: 32m 14s)

Be patient now, go grab a coffee / tea or snack as the command might take up to few minutes for the aide to walk through the whole monitored filesystems and built its database.

root@server:~# echo cp /var/lib/aide/aide.db{.new,}
cp /var/lib/aide/ /var/lib/aide/aide.db


root@server:~# cp /var/lib/aide/aide.db{.new,}

root@server:~# aide –check –config /etc/aide/aide.conf

Start timestamp: 2023-03-09 13:01:32 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!

  Total number of entries:      66972
  Added entries:                1
  Removed entries:              0
  Changed entries:              7

Added entries:

f+++++++++++++++++: /var/lib/aide/aide.db

Changed entries:

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /root
f <…. mci.H.. . : /root/.viminfo
f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db
f >b… mc..H.. . : /var/log/sysstat/sa09

Detailed information about changes:

Directory: /etc/aide
 Mtime     : 2023-03-09 12:04:03 +0200        | 2023-03-09 12:51:11 +0200
 Ctime     : 2023-03-09 12:04:03 +0200        | 2023-03-09 12:51:11 +0200

Directory: /root
 Mtime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 12:51:11 +0200
 Ctime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 12:51:11 +0200

File: /root/.viminfo
 Size      : 18688                            | 17764
 Mtime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 12:51:11 +0200
 Ctime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 12:51:11 +0200
 Inode     : 133828                           | 133827
 SHA256    : aV54gi33aA/z/FuBj2ZioU2cTa9H16TT | dnFdLVQ/kx3UlTah09IgEMrJ/aYgczHe
             TzkLSxBDSB4=                     | DdxDAmPOSAM=

3. Test aide detects file changes

Create a new file and append some text and rerun the aide check


root@server:~# touch /root/test.txt
root@server:~# echo aaa > /root/test.txt
root@server:~# aide –check –config /etc/aide/aide.conf


Start timestamp: 2023-03-09 13:07:21 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!

  Total number of entries:      66973
  Added entries:                2
  Removed entries:              0
  Changed entries:              7

Added entries:

f+++++++++++++++++: /root/test.txt
f+++++++++++++++++: /var/lib/aide/aide.db

Changed entries:

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /root
f <…. mci.H.. . : /root/.viminfo
f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db
f >b… mc..H.. . : /var/log/sysstat/sa09


The same command can be shortened for the lazy typist:

root@server:~# aide -c /etc/aide/aide.conf -C

The command will basically try to check the deviation between the AIDE database and the filesystem.

4. Limiting AIDES Integrity Checks to Specific Files / Directories

In order to limit the integrity checks to a specific entries for example /etc, pass the –limit REGEX option to AIDE check command where REGEX is the entry to check.

For example, check and update the database entries matching /etc, you would run aide command as shown below;

root@server:~# aide -c /etc/aide/aide.conf –limit /etc –check


AIDE found differences between database and filesystem!!
Limit: /etc

  Total number of entries:      66791
  Added entries:                0
  Removed entries:              0
  Changed entries:              2

Changed entries:

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /etc/default

Detailed information about changes:

Directory: /etc/aide
 Mtime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 16:43:03 +0200
 Ctime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 16:43:03 +0200

Directory: /etc/default
 Mtime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 18:42:12 +0200
 Ctime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 18:42:12 +0200

The attributes of the (uncompressed) database(s):

 SHA256    : sjCxyIkr0nC/gTkNmn7DNqAQWttreDF6
 SHA512    : vNMpb54qxrbOk6S1Z+m9r0UwGvRarkWY
 RMD160    : anhm5E6UlKmPYYJ4WYnWXk/LT3A=
 TIGER     : 5e1wycoF35/ABrRf7FNypZ45169VTuV4
 CRC32     : EAJlFg==
 HAVAL     : R5imONWRYgNGEfhBTc096K+ABnMFkMmh
 GOST      : F5zO2Ovtvf+f7Lw0Ef++ign1znZAQMHM

End timestamp: 2023-03-09 20:02:18 +0200 (run time: 1m 32s)

5. Add the modified /root/test.txt to AIDE list of known modified files database

root@server:~# aide –update –config /etc/aide/aide.
  ERROR: cannot open config file '/etc/aide/aide.': No such file or directory


root@server:~# ​ aide –update –config /etc/aide/aide.conf

Start timestamp: 2023-03-09 18:45:17 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!
New AIDE database written to /var/lib/aide/

  Total number of entries:      66791
  Added entries:                0
  Removed entries:              0
  Changed entries:              8

Changed entries:

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /etc/default
d =…. mc.. .. . : /root
f >…. mci.H.. . : /root/.viminfo
f >…. mci.H.. . : /root/test.txt
f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db

Detailed information about changes:

Directory: /etc/aide
 Mtime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 16:43:03 +0200
 Ctime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 16:43:03 +0200

Directory: /etc/default
 Mtime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 18:42:12 +0200
 Ctime     : 2023-03-09 12:06:13 +0200        | 2023-03-09 18:42:12 +0200

Directory: /root
 Mtime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 18:44:34 +0200
 Ctime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 18:44:34 +0200

File: /root/.viminfo
 Size      : 16706                            | 16933
 Mtime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 18:44:34 +0200
 Ctime     : 2023-03-09 15:59:53 +0200        | 2023-03-09 18:44:34 +0200
 Inode     : 136749                           | 133828
 SHA256    : KMHGoMVJo10BtafVrWIOLt3Ht9gK8bc+ | rrp8S3VftzZzvjBP1JC+PBpODv9wPKGw
             9uHh/z7iJWA=                     | TA+hyhTiY+U=
 SHA512    : ieDHy7ObSTfYm5d8DtYcHKxHya13CS65 | PDAJjyZ39uU3kKFo2lHBduTqxMDq4i01
             ObMYIRAre6IgvLslEs0ZodQFyrczMyRt | 1Kvm/h6xzFhHtFgjidtcemG8wDcjtfNF
             +d6SrW0gn3skKn2B7G09eQ==         | Z7LO230fgGeO7UepqtxZjQ==
 RMD160    : nUgg/G4zsVGKzVmmrqltuYUDvtg=     | jj61KAFONK92mj+u66RDJmxFhmI=
 TIGER     : 3vPSOrla5k+k2br1E2ES4eNiSZ2novFX | mn4kNCzd8SQr2ID2VSe4f4l0ta7pO/xo
 CRC32     : NDnMgw==                         | AyzVUQ==
 HAVAL     : Q9/KozxRiPbLEkaIfnBUZdEWftaF52Mw | 6jADKV6jg7ZVr/A/oMhR4NXc8TO1AOGW
             7tiR7DXhl0o=                     | NrYe+j6UcO0=
 WHIRLPOOL : vB/ZMCul4hN0aYd39gBu+HmZT/peRUI8 | mg6c1lYYVNZcy4mVzGojwraim8e3X2/R
             KDkaslNb8+YleoFWx0mbhAbkGurc0+jh | urVvEmbsgTuUCJOuf9+OrEACiF0fbe/x
             YPBviZIKcxUbTc2nGthTWw==         | t+BXnSQWk08OL9EI6gMGqA==
 GOST      : owVGTgU9BH3b0If569wQygw3FAbZIZde | ffx29GV2jaCB7XzuNjdiRzziIiZYnbi3
             eAfQfzlRPGY=                     | Ar7jyNMUutk=

File: /root/test.txt
 Size      : 4                                | 8
 Mtime     : 2023-03-09 13:07:12 +0200        | 2023-03-09 18:44:34 +0200
 Ctime     : 2023-03-09 13:07:12 +0200        | 2023-03-09 18:44:34 +0200
 Inode     : 133828                           | 136751
 SHA256    : F+aC8GC1+OR+oExcSFWQiwpa1hICImD+ | jUIZMGfiMdAlWFHu8mmmlml4qAGNQNL5
             UOEeywzAq3Y=                     | 6NhzJ1sYFZE=
 SHA512    : d+UmFKFBzvGadt5hk+nIRbjP//7PSXNl | ixn20lcEMDEtsJo3hO90Ea/wHWLCHcrz
             Pl16XRIUUPq2FCiQ4PeUcVciukJX7ijL | seBWunbBysY0z3BWcfgnN2vH05WfRfvA
             D045ZvGOEcnmL6a6vwp0jw==         | QiNtQS1tStuEdB3Voq54zQ==
 RMD160    : I6waxKN3rMx4WTz4VCUQXoNoxUg=     | urTh1j1t3UHchnJGnBG4lUZnjI4=
 TIGER     : cwUYgfKHcJnWXcA0pr/OKuxuoxh+b9lA | prstKqCfMXL39aVGFPA0kX4Q9x7a+hUn
 CRC32     : UD78Dw==                         | zoYiEA==
 HAVAL     : bdbKR9LvPgsYClViKiHx48fFixfIL/jA | ZdpdeMhw4MvKBgWsM4EeyUgerO86Rt82
             F3tjdc2Gm8Y=                     | W94fJFRWbrM=
 WHIRLPOOL : OLP0Y4oKcqW2yEvme8z419N1KE4TB9GJ | Xk8Ujo3IU2SzSqbJFegq7p1ockmrnxJF
             biHn/9XgrBz4fQiDJ8eHpx+0exA9hXmY | R3Rfstd1jWSwLFNTEwfbRRw+TARtRK50
             EbbakMJJdzLt1ipKWiV9gg==         | iWJeHLsD5dZ+CzV0tf4sUg==
 GOST      : ystISzoeH/ZznYrrXmxe4rwmybWMpGuE | GhMWNxg7Is0svJ+5LP+DVWbgt+CDQO+3
             0PzRnVEqnR8=                     | 08dwBuVAwB8=

File: /var/lib/fail2ban/fail2ban.sqlite3
 Mtime     : 2023-03-09 15:55:01 +0200        | 2023-03-09 18:45:01 +0200
 Ctime     : 2023-03-09 15:55:01 +0200        | 2023-03-09 18:45:01 +0200
 SHA256    : lLilXNleqSgHIP1y4o7c+oG5XyUPGzgi | NCJJ2H6xgCw/NYys1LMA7hOWwoOoxI8Y
             RHYH+zvlAL4=                     | 4SJygfqEioE=
 SHA512    : iQj2pNT4NES4fBcujzdlEEGZhDnkhKgc | ClQZ5HMOSayUNb//++eZc813fiMJcXnj
             QDlGFSAn6vi+RXesFCjCABT7/00eEm5/ | vTGs/2tANojoe6cqpsT/LaJ3QZXpmrfh
             ILcaqlQtBSLJgHjMQehzdg==         | syVak1I4n9yg8cDKEkZUvw==
 RMD160    : Xg4YU8YI935L+DLvkRsDanS4DGo=     | SYrQ27n+/1fvIZ7v+Sar/wQHulI=
 TIGER     : 2WhhPq9kuyeNJkOicDTDeOeJB8HR8zZe | o1LDZtRclri2KfZBe5J3D4YhM05UaP4E
 CRC32     : NQmi4A==                         | tzIsqg==
 HAVAL     : t1ET+84+8WgfwqlLy4R1Qk9qGZQRUbJI | MwVnjtM3dad/RuN2BfgsySX2DpfYq4qi
             z2J0ROGduXc=                     | H1pq6RYsA6o=
 WHIRLPOOL : xKSn71gFIVhk5rWJIBaYQASl0V+pGn+3 | m5LEXfhBbhWFg/d8CFJhklOurmRSkDSG
             N85R0tiCKsTZ2+LRkxDrzcVQdss2k8+z | LC/vICnbEWzLwrCuMwBi1/e5wDNIY8gK
             oqExhoXtPsMaREjpCugd3Q==         | mvGn40x+G4cCYNZ6lGT9Zg==
 GOST      : WptpUlfooIlUjzDHU8XGuOU2waRud5SR | i6K4COXU0nyZ1mL3ZBuGUPz/ZXTj8KKQ
             E/tnoBqk+q0=                     | L6VNyS8/X2Y=

Directory: /var/lib/vnstat
 Mtime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 18:45:01 +0200
 Ctime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 18:45:01 +0200

File: /var/lib/vnstat/vnstat.db
 Mtime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 18:45:00 +0200
 Ctime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 18:45:00 +0200
 SHA256    : X/lnJuuSo4jX4HRzxMBodnKHAjQFvugi | oqtY3HTNds/qDNFCRAEsfN5SuO0U5LRg
             2sh2c0u69x8=                     | otc5z1y+eGY=
 SHA512    : U/g8O6G8cuhsqCUCbrElxgiy+naJKPkI | y+sw4LX8mlDWkRJMX38TsYSo1DQzxPOS
             hG7vdH9rBINjakL87UWajT0s6WSy0pvt | 068otnzw2FSSlM5X5j5EtyJiY6Hd5P+A
             ALaTcDFKHBAmmFrl8df2nQ==         | jFiWStMbx+dQidXYZ4XFAw==
 RMD160    : F6YEjIIQu2J3ru7IaTvSemA9e34=     | bmVSaRKN2qU7qpEWkzfXFoH4ZK4=
 TIGER     : UEwLoeR6Qlf2oOI58pUCEDaWk0pHDkcY | 0Qb4nUqe3cKh/g5CQUnOXGfjZwJHjeWa
 CRC32     : Bv3/6A==                         | jvW6mg==
 HAVAL     : VD7tjHb8o8KTUo5xUH7eJEmTWgB9zjft | rumfiWJvy/sTK/09uj7XlmV3f7vj6KBM
             kOkzKxFWqqU=                     | qeOuKvu0Zjc=
 WHIRLPOOL : wR0qt8u4N8aQn8VQ+bmfrxB7CyCWVwHi | FVWDRE3uY6qHxLlJQLU9i9QggLW+neMj
             ADHpMTUxBEKOpOBlHTWXIk13qYZiD+o/ | Wt+Dj9Rz92BG9EomgLUgUkxfiVFO8cMq
             XtzTB4rMbxS4Z5PAdC/07A==         | WaR/KKq3Z7R8f/50tc9GMQ==
 GOST      : l3ibqMkHMSPpQ+9ok51/xBthET9+JQMd | qn0GyyCg67KRGP13At52tnviZfZDgyAm
             OZtiFGYXmgU=                     | c82NXSzeyV0=

The attributes of the (uncompressed) database(s):

 SHA256    : sjCxyIkr0nC/gTkNmn7DNqAQWttreDF6
 SHA512    : vNMpb54qxrbOk6S1Z+m9r0UwGvRarkWY
 RMD160    : anhm5E6UlKmPYYJ4WYnWXk/LT3A=
 TIGER     : 5e1wycoF35/ABrRf7FNypZ45169VTuV4
 CRC32     : EAJlFg==
 HAVAL     : R5imONWRYgNGEfhBTc096K+ABnMFkMmh
 GOST      : F5zO2Ovtvf+f7Lw0Ef++ign1znZAQMHM

 SHA256    : QRwubXnz8md/08n28Ek6DOsSQKGkLvuc
 SHA512    : 238RmI1PHhd9pXhzcHqM4+VjNzR0es+3
 RMD160    : GJWuX/nIPY05gz62YXxk4tWiH5I=
 TIGER     : l0aOjXlM4/HjyN9bhgBOvvCYeqoQyjpw
 CRC32     : KFz6GA==
 HAVAL     : a//4jwVxF22URf2BRNA612WOOvOrScy7
 WHIRLPOOL : MBf+NeXElUvscJ2khIuAp+NDu1dm4h1f
 GOST      : EQnPh6jQLVUqaAK9B4/U4V89tanTI55N

End timestamp: 2023-03-09 18:49:51 +0200 (run time: 4m 34s)

6. Substitute old aide database with the new that includes the modified files

As you see AIDE detected the changes in /root/test.txt

To apply the changes be known by AIDE for next time (e.g. this file was authorized and supposed to be written there) simply move the new generated database
to current aide database.

# copy generated DB to master DB
root@dlp:~# cp -p /var/lib/aide/ /var/lib/aide/aide.db

7. Check once again to make sure recently modified files are no longer seen as changed by AIDE

Recheck again the database to make sure the files you wanted to omit are no longer mentioned as changed

root@server:~# aide –check –config /etc/aide/aide.conf
Start timestamp: 2023-03-09 16:23:05 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!

  Total number of entries:      66791
  Added entries:                0
  Removed entries:              0
  Changed entries:              3

Changed entries:

f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db

Detailed information about changes:

File: /var/lib/fail2ban/fail2ban.sqlite3
 Mtime     : 2023-03-09 15:55:01 +0200        | 2023-03-09 16:25:02 +0200
 Ctime     : 2023-03-09 15:55:01 +0200        | 2023-03-09 16:25:02 +0200
 SHA256    : lLilXNleqSgHIP1y4o7c+oG5XyUPGzgi | MnWXC2rBMf7DNJ91kXtHXpM2c2xxF60X
             RHYH+zvlAL4=                     | DfLUQLHiSiY=
 SHA512    : iQj2pNT4NES4fBcujzdlEEGZhDnkhKgc | gxHVBxhGTKi0TjRE8/sn6/gtWsRw7Mfy
             QDlGFSAn6vi+RXesFCjCABT7/00eEm5/ | /wCfPlDK0dkRZEbr8IE2BNUhBgwwocCq
             ILcaqlQtBSLJgHjMQehzdg==         | zuazTy4N4x6X8bwOzRmY0w==
 RMD160    : Xg4YU8YI935L+DLvkRsDanS4DGo=     | +ksl9kjDoSU9aL4tR7FFFOK3mqw=
 TIGER     : 2WhhPq9kuyeNJkOicDTDeOeJB8HR8zZe | 9cvXZNbU+cp5dA5PLiX6sGncXd1Ff5QO
 CRC32     : NQmi4A==                         | y6Oixg==
 HAVAL     : t1ET+84+8WgfwqlLy4R1Qk9qGZQRUbJI | aPnCrHfmZAUm7QjROGEl6rd3776wO+Ep
             z2J0ROGduXc=                     | s/TQn7tH1tY=
 WHIRLPOOL : xKSn71gFIVhk5rWJIBaYQASl0V+pGn+3 | 9Hu6NBhz+puja7uandb21Nt6cEW6zEpm
             N85R0tiCKsTZ2+LRkxDrzcVQdss2k8+z | bTsq4xYA09ekhDHMQJHj2WpKpzZbA+t0
             oqExhoXtPsMaREjpCugd3Q==         | cttMDX8J8M/UadqfL8KZkQ==
 GOST      : WptpUlfooIlUjzDHU8XGuOU2waRud5SR | WUQfAMtye4wADUepBvblvgO+vBodS0Ej
             E/tnoBqk+q0=                     | cIbXy4vpPYc=

Directory: /var/lib/vnstat
 Mtime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 16:25:01 +0200
 Ctime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 16:25:01 +0200

File: /var/lib/vnstat/vnstat.db
 Mtime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 16:25:01 +0200
 Ctime     : 2023-03-09 16:00:00 +0200        | 2023-03-09 16:25:01 +0200
 SHA256    : X/lnJuuSo4jX4HRzxMBodnKHAjQFvugi | N1lzhV3+tkDBud3AVlmIpDkU1c3Rqhnt
             2sh2c0u69x8=                     | YqE8naDicoM=
 SHA512    : U/g8O6G8cuhsqCUCbrElxgiy+naJKPkI | +8B9HvHhOp1C/XdlOORjyd3J2RtTbRBF
             hG7vdH9rBINjakL87UWajT0s6WSy0pvt | b0Moo2Gj+cIxaMCu5wOkgreMp6FloqJR
             ALaTcDFKHBAmmFrl8df2nQ==         | UH4cNES/bAWtonmbj4W7Vw==
 RMD160    : F6YEjIIQu2J3ru7IaTvSemA9e34=     | 8M6TIOHt0NWgR5Mo47DxU28cp+4=
 TIGER     : UEwLoeR6Qlf2oOI58pUCEDaWk0pHDkcY | Du9Ue0JA2URO2tiij31B/+663OaWKefR
 CRC32     : Bv3/6A==                         | v0Ai4w==
 HAVAL     : VD7tjHb8o8KTUo5xUH7eJEmTWgB9zjft | XA+vRnMNdVGFrO+IZtEA0icunWqBGaCf
             kOkzKxFWqqU=                     | leR27LN4ejc=
 WHIRLPOOL : wR0qt8u4N8aQn8VQ+bmfrxB7CyCWVwHi | HG31dNEEcak2zZGR24W7FDJx8mh24MaJ
             ADHpMTUxBEKOpOBlHTWXIk13qYZiD+o/ | BQNhqkuS6R/bmlhx+P+eQ/JimwPAPOaM
             XtzTB4rMbxS4Z5PAdC/07A==         | xWG7cMETIXdT9sUOUal8Sw==
 GOST      : l3ibqMkHMSPpQ+9ok51/xBthET9+JQMd | y6Ek/TyAMGV5egkfCu92Y4qqk1Xge8c0
             OZtiFGYXmgU=                     | 3ONXRveOlr0=

The attributes of the (uncompressed) database(s):

 SHA256    : sjCxyIkr0nC/gTkNmn7DNqAQWttreDF6
 SHA512    : vNMpb54qxrbOk6S1Z+m9r0UwGvRarkWY
 RMD160    : anhm5E6UlKmPYYJ4WYnWXk/LT3A=
 TIGER     : 5e1wycoF35/ABrRf7FNypZ45169VTuV4
 CRC32     : EAJlFg==
 HAVAL     : R5imONWRYgNGEfhBTc096K+ABnMFkMmh
 GOST      : F5zO2Ovtvf+f7Lw0Ef++ign1znZAQMHM

End timestamp: 2023-03-09 16:27:33 +0200 (run time: 4m 28s)

As you can see there are no new added entries for /root/test.txt and some other changed records for vnstat service as well as fail2ban ones, so the Intrusion detection system works just as we expected it.

8. Configure Email AIDE changed files alerting Email recipient address

From here on aide package has set its own cron job which is automatically doing the check operation every day and any new file modifications will be captured and alerts sent to local root@localhost mailbox account, so you can check it out later with mail command.

If you want to sent the Email alert for any files modifications occured to another email, assuming that you have a locally running SMTP server with a mail relay to send to external mails, you can do it via /etc/default/aide via:


For example change it to a FQDN email address

9.Force AIDE to run AIDE at specitic more frequent time intervals

You can as well install a cron job to execute AIDE at specific time intervals, as of your choice

Lets say you want to run a custom prepared set of files to monitor in /etc/aide/aide_custom_config.conf configure a new cronjob like below:

root@server:~# crontab -u root -e
*/5 * * * * aide -c /etc/aide/aide_custom_config.conf -u && cp /var/lib/custom-aide/aide.db{.new,}

This will execute AIDE system check every 5 minutse and email the report to ealier configured email via /etc/default/aide

10. Check the output of AIDE for changes – useful for getting a files changes from aide from scripts

Check the command exit status.

root@server:~# echo $?

According to AIDE man pages, the AIDE’s exit status is normally 0 if no errors occurred. Except when the –check, –compare or –update command was requested, in which case the exit status is defined as:

   1 * (new files detected?)     +

   2 * (removed files detected?) +

   4 * (changed files detected?)

   Since  those three cases can occur together, the respective error codes are added. For example, if there are new files and removed files detected, the exit status will be 1 + 2 = 3.

   Additionally, the following exit codes are defined for generic error conditions in aide help manual:

   14 Error writing error

   15 Invalid argument error

   16 Unimplemented function error

   17 Invalid configureline error

   18 IO error

   19 Version mismatch error


  • That AIDE checks might be resource intensive
    and could cause a peak in CPU use and have a negative effect on lets very loaded application server machines,
    thus causing a performance issuea during integrity checks !
  • If you are scanning file system wide and you do it frequent, be sure to provide “enough” resources or schedule the scan at a times that the Linux host will be less used !
  • Whenever you made any AIDE configuration changes, remember to initialize the database to create a baseline !

How to check if your Linux WebServer is under a DoS attack

Friday, July 22nd, 2011

There are few commands I usually use to track if my server is possibly under a Denial of Service attack or under Distributed Denial of Service

Sys Admins who still have not experienced the terrible times of being under a DoS attack are happy people for sure …

1. How to Detect a TCP/IP Denial of Service Attack This are the commands I use to find out if a loaded Linux server is under a heavy DoS attack, one of the most essential one is of course netstat.
To check if a server is under a DoS attack with netstat, it’s common to use:

linux:~# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n|wc -l

If the output of below command returns a result like 2000 or 3000 connections!, then obviously it’s very likely the server is under a DoS attack.

To check all the IPS currently connected to the Apache Webserver and get a very brief statistics on the number of times each of the IPs connected to my server, I use the cmd:

linux:~# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
221 233 540

As you could see from the above command output the IP is either connected 221 times to the server or is in state of connecting or disconnecting to the node.

Another possible way to check, if a Linux or BSD server is under a Distributed DoS is with the list open files command lsof
Here is how lsof can be used to list the approximate number of ESTABLISHED connections to port 80.

linux:~# lsof -i TCP:80
litespeed 241931 nobody 17u IPv4 18372655 TCP (LISTEN)
litespeed 241931 nobody 25u IPv4 18372659 TCP (LISTEN)
litespeed 241931 nobody 30u IPv4 29149647 TCP> (ESTABLISHED)
litespeed 241931 nobody 33u IPv4 18372647 TCP (LISTEN)
litespeed 241931 nobody 34u IPv4 29137514 TCP> (ESTABLISHED)
litespeed 241931 nobody 35u IPv4 29137831 TCP> (ESTABLISHED)
litespeed 241931 nobody 37w IPv4 29132085 TCP> (ESTABLISHED)

Another way to get an approximate number of established connections to let’s say Apache or LiteSpeed webserver with lsof can be achieved like so:

linux:~# lsof -i TCP:80 |wc -l

I find it handy to keep track of above lsof command output every few secs with gnu watch , like so:

linux:~# watch "lsof -i TCP:80"

2. How to Detect if a Linux server is under an ICMP SMURF attack

ICMP attack is still heavily used, even though it’s already old fashioned and there are plenty of other Denial of Service attack types, one of the quickest way to find out if a server is under an ICMP attack is through the command:

server:~# while :; do netstat -s| grep -i icmp | egrep 'received|sent' ; sleep 1; done
120026 ICMP messages received
1769507 ICMP messages sent
120026 ICMP messages received
1769507 ICMP messages sent

As you can see the above one liner in a loop would check for sent and recieved ICMP packets every few seconds, if there are big difference between in the output returned every few secs by above command, then obviously the server is under an ICMP attack and needs to hardened.

3. How to detect a SYN flood with netstat

linux:~# netstat -nap | grep SYN | wc -l

1032 SYNs per second is quite a high number and except if the server is not serving let’s say 5000 user requests per second, therefore as the above output reveals it’s very likely the server is under attack, if however I get results like 100/200 SYNs, then obviously there is no SYN flood targetting the machine 😉

Another two netstat command application, which helps determining if a server is under a Denial of Service attacks are:

server:~# netstat -tuna |wc -l


server:~# netstat -tun |wc -l

Of course there also some other ways to check the count the IPs who sent SYN to the webserver, for example:

server:~# netstat -n | grep :80 | grep SYN |wc -l

In many cases of course the top or htop can be useful to find, if many processes of a certain type are hanging around.

4. Checking if UDP Denial of Service is targetting the server

server:~# netstat -nap | grep 'udp' | awk '{print $5}' | cut -d: -f1 | sort |uniq -c |sort -n

The above command will list information concerning possible UDP DoS.

The command can easily be accustomed also to check for both possible TCP and UDP denial of service, like so:

server:~# netstat -nap | grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort |uniq -c |sort -n

If after getting an IP that has too many connections to the server and is almost certainly a DoS host you would like to filter this IP.

You can use the /sbin/route command to filter it out, using route will probably be a better choice instead of iptables, as iptables would load up the CPU more than simply cutting the route to the server.

Here is how I remove hosts to not be able to route packets to my server:

route add reject

The above command would null route the access of IP to my server.

Later on to look up for a null routed IP to my host, I use:

route -n |grep -i

Well hopefully this should be enough to give a brief overview on how, one can dig in his server and find if he is under a Distributed Denial of Service, hope it’s helpful to somebody out there.
Cheers 😉

How to check the IP address of Skype (user / Contacts) on GNU / Linux with netstat and whois

Thursday, May 3rd, 2012

netstat check skype contact IP info with netstat Linux xterm Debian Linux

Before I explain how netstat and whois commands can be used to check information about a remote skype user – e.g. (skype msg is send or receved) in Skype. I will say in a a few words ( abstract level ), how skype P2P protocol is designed.
Many hard core hackers, certainly know how skype operates, so if this is the case just skip the boring few lines of explanation on how skype proto works.

In short skype transfers its message data as most people know in Peer-to-Peer "mode" (P2P)  – p2p is unique with this that it doesn't require a a server to transfer data from one peer to another. Most classical use of p2p networks in the free software realm are the bittorrents.

Skype way of connecting to peer client to other peer client is done via a so called "transport points". To make a P-to-P connection skype wents through a number of middle point destinations. This transport points (peers) are actually other users logged in Skype and the data between point A and point B is transferred via this other logged users in encrypted form. If a skype messages has to be transferred  from Peer A (point A) to Peer B (Point B) or (the other way around), the data flows in a way similar to:

 A -> D -> F -> B


B -> F -> D -> A

(where D and F are simply other people running skype on their PCs).
The communication from a person A to person B chat in Skype hence, always passes by at least few other IP addresses which are owned by some skype users who happen to be located in the middle geographically between the real geographic location of A (the skype peer sender) and B (The skype peer receiver)..

The exact way skypes communicate is way more complex, this basics however should be enough to grasp the basic skype proto concept for most ppl …

In order to find the IP address to a certain skype contact – one needs to check all ESTABLISHED connections of type skype protocol with netsat within the kernel network stack (connection) queue.

netstat displays few IPs, when skype proto established connections are grepped:

noah:~# netstat -tupan|grep -i skype | grep -i established| grep -v ''
tcp 0 0 ESTABLISHED 3606/skype
tcp 0 0 ESTABLISHED 3606/skype
tcp 0 0 ESTABLISHED 3606/skype

Now, as few IPs are displayed, one needs to find out which exactly from the list of the ESTABLISHED IPs is the the Skype Contact from whom are received or to whom are sent the messages in question.

The blue colored IP address:port is the local IP address of my host running the Skype client. The red one is the IP address of the remote skype host (Skype Name) to which messages are transferred (in the the exact time the netstat command was ran.

The easiest way to find exactly which, from all the listed IP is the IP address of the remote person is to send multiple messages in a low time interval (let's say 10 secs / 10 messages to the remote Skype contact).

It is a hard task to write 10 msgs for 10 seconds and run 10 times a netstat in separate terminal (simultaneously). Therefore it is a good practice instead of trying your reflex, to run a tiny loop to delay 1 sec its execution and run the prior netstat cmd.

To do so open a new terminal window and type:

noah:~# for i in $(seq 1 10); do \
sleep 1; echo '-------'; \
netstat -tupan|grep -i skype | grep -i established| grep -v ''; \

tcp 0 0 ESTABLISHED 3606/skype
tcp 0 0 ESTABLISHED 3606/skype
tcp 0 0 ESTABLISHED 3606/skype
tcp 0 0 ESTABLISHED 3606/skype
tcp 0 0 ESTABLISHED 3606/skype

You see on the first netstat (sequence) exec, there is only 1 IP address to which a skype connection is established, once I sent some new messages to my remote skype friend, another IP immediatelly appeared. This other IP is actually the IP of the person to whom, I'm sending the "probe" skype messages.
Hence, its most likely the skype chat at hand is with a person who has an IP address of the newly appeared

Later to get exact information on who owns and administrative contact info as well as address of the ISP or person owning the IP, do a RIPE  whois

noah:~# whois
% This is the RIPE Database query service.
% The objects are in RPSL format.
% The RIPE Database is subject to Terms and Conditions.
% See

% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to ' -'
inetnum: -
descr: BTC Broadband Service
country: BG
admin-c: LG700-RIPE
tech-c: LG700-RIPE
tech-c: SS4127-RIPE
mnt-by: BT95-ADM
mnt-domains: BT95-ADM
mnt-lower: BT95-ADM
source: RIPE # Filteredperson: Lyubomir Georgiev

Note that this method of finding out the remote Skype Name IP to whom a skype chat is running is not always precise.

If for instance you tend to chat to many people simultaneously in skype, finding the exact IPs of each of the multiple Skype contacts will be a very hard not to say impossible task.
Often also by using netstat to capture a Skype Name you're in chat with, there might be plenty of "false positive" IPs..
For instance, Skype might show a remote Skype contact IP correct but still this might not be the IP from which the remote skype user is chatting, as the remote skype side might not have a unique assigned internet IP address but might use his NET connection over a NAT or DMZ.

The remote skype user might be hard or impossible to track also if skype client is run over skype tor proxy for the sake of anonymity
Though it can't be taken as granted that the IP address obtained would be 100% correct with the netstat + whois method, in most cases it is enough to give (at least approximate) info on a Country and City origin of the person you're skyping with.

The end of the work week :)

Friday, February 1st, 2008

One more week passed without serious server problems. Yesterday after upgrade to debian 4.0rc2 with

apt-get dist-upgrade and reboot the pc-freak box became unbootable.

I wasn’t able to fix it until today because the machine’s box seemed not to read cds well.The problem was consisted of this that after the boot process of the linux kernel has started the machine the boot up was interrupted with a message saying
/sbin/init is missing

and I was dropped to a busybox without being able to read nothing from my filesystem.Thankfully nomen came to Dobrich for the weekend and today he bring me his cdrom-drive I booted with the debian.

Using Debian’s linux rescue I mounted the partition to check what’s wrong. I suspected something is terribly wrong with the lilo’s conf.

Looking closely to it I saw it’s the lilo conf file it was setupped to load a initrd for the older kernel. changing the line to thenew initrd in /etc/lilo.conf and rereading the lilo; /sbin/lilo -C; /sbin/lilo;

fixed the mess and pc-freak booted succesfully! 🙂

Yesterday I had to do something kinky. It was requested from a client to have access to a mysql service of one of the company servers,the problem was that the client didn’t have static IP so I didn’t have a good way to put into the current firewall.

Everytime the adsl they use got restarted a new absolutely random IP from all the BTC IP ranges was assigned.

The solution was to make a port redirect to a non-standard mysql port (XXXXX) which pointed to the standard 3306 service. I had to tell the firewall not to check the coming IPs on the non-standard port (XXXXX) against the 3306 service fwall rules.

Thanks to the help of a guy #iptables jengelh I figured out the solution.

To complete the requested task it was needed to mark all packagescoming into port (XXXXX) using the iptables mangle option and to add a rule to ACCEPT all marked packages.

The rules looked like this

/sbin/iptables -t mangle -A PREROUTING -p tcp –dport XXXXX -j MARK –set-mark 123456/sbin/iptables -t nat -A PREROUTING -d EXTERNAL_IP -i eth0 -p tcp –dport XXXXX -j DNAT –to-destination EXTERNAL_IP:3306

/sbin/iptables -t filter -A INPUT -p tcp –dport 3306 -m mark –mark 123456 -j ACCEPT .

Something I wondered a bit was should /proc/sys/net/ipv4/ip_forward in order for the above redirect to be working, in case you’re wondering too well it doesn’t 🙂 The working week was a sort of quiteful no serious problems with servers and work no serious problems at school (although I see me and my collegues become more and more unserious) at studying. My grand parentsdecided to make me a gift and give me money to buy a laptop and I’m pretty happy for this 🙂 All that is left is to choose a good machine with hardware supported both by FreeBSD and Linux.


How to prevent SSH and FTP bruteforce attacks with iptables on Linux

Friday, December 30th, 2011

Earlier I've blogged about how to prevent brute force attacks with fail2ban, denohosts and blockhosts , however there is easier way to secure against basic brute force attacks by not installing or configuring any external programs.
The way I'm talking about uses simple iptables rules to filter out brute force attacks.

Here is a small script to stop ssh and FTP invaders which try to initiate more than 3 consequential connections in 5 minutes time to port 22 or port 23:

/sbin/iptables -A INPUT -p tcp --dport 22 --syn -m recent --name sshbr --set
/sbin/iptables -A INPUT -p tcp --dport 22 --syn -j SSH_WHITELIST
/sbin/iptables -A INPUT -p tcp --dport 22 --syn -m recent --name sshbr \
--update --rttl --hitcount 3 --seconds 300 -j REJECT --reject-with tcp-reset
/sbin/iptables -A SSH_WHITELIST -s $SERVER_MAIN_IP -p tcp --dport 22 --syn -m recent --rttl --remove

The only thinIf the rules are matched iptables filter rules will be added to the iptables CHAIN SSH_WHITELIST
In case if you want to add some more truested IPs add some more iptables rules, like:

/sbin/iptables -A SSH_WHITELIST -s $ALLOW_IP -p tcp --dport 22 --syn -m recent --rttl --remove

Each filtered IP that matches the rules will be filtered for 5 minutes, if 5 minutes is enough, the 300 value has to be increased.

How to block IP address with pf on FreeBSD, NetBSD and OpenBSD

Wednesday, July 27th, 2011

Pf Firewall BSD logo

I’ve noticed some IPs which had a kind of too agressive behaviour towards my Apache webserver and thus decided to filter them out with the Firewall.
As the server is running FreeBSD and my firewall choise is bsd’s pf I added the following lines to my /etc/pf.conf to filter up the abiser IP:

table persist file "/etc/pf.blocked.ip.conf"
EXT_NIC="ml0" # interface connected to internet
block drop in log (all) quick on $EXT_NIC from to any
echo '' >> /etc/pf.blocked.ip.conf

As you see I’m adding the malicious IP to /etc/pf.blocked.ip.conf, if I later decide to filter some other IPs I can add them up there and they will be loaded and filtered by pf on next pf restart.

Next I restarted my pf firewall definitions to make the newly added rules in pf.conf to load up.

freebsd# pfctl -d
freebsd# pfctl -e -f /etc/pf.conf

To show all IPs which will be inside the blockips filtering tables, later on I used:

pfctl -t blockips -T show

I can also later use pf to add later on new IPs to be blocked without bothering to restart the firewall with cmd:

freebsd# pfctl -t blockedips -T add 111.222.333.444

Deleting an IP is analogous and can be achieved with:

freebsd# pfctl -t blockedips -T delete 111.222.333.444

There are also logs stored about pf IP blocking as well as the other configured firewall rules in /var/log/pflog file.
Hope this is helpful to somebody.

How to add a range of virtual IPs to a CentOS and Fedora Linux server

Monday, July 18th, 2011

Recently I had the task to add a range of few IP addresses to as a virtual interface IPs.

The normal way to do that is of course using the all well known ifconfig eth0:0, ifconfig eth0:1 or using a tiny shell script which does it and set it up to run through /etc/rc.local .

However the Redhat guys could omit all this mambo jambo and do it The Redhat way TM 😉 by using a standard method documented in CentOS and RHEL documentation.
Here is how:

# go to network-script directory[root@centos ~]# cd /etc/sysconfig/network-scripts
# create ifcfg-eth0-range (if virtual ips are to be assigned on eth0 lan interface[root@centos network-scripts]# touch ifcfg-eth0-range

Now inside ifcfg-eth0-range, open up with a text editor or use the echo command to put inside:


Now save the /etc/sysconfig/network-scripts/ifcfg-eth0-range file and finally restart centos networking via the network script:

[root@centos network-scripts]# service network restart

That’s all now after the network gets reinitialized all the IPs starting with and ending in< will get assigned as virtual IPs for eth0 interface
Cheers 😉

How to configure and enable Xen Linux dedicated server’s Virtual machines Internet to work / Enable multipe real IPs and one MAC only in (SolusVM) through NAT routed and iptables

Saturday, June 4th, 2011

Xen Linux Virtual Machine Logo

I’ve been hired as a consultant recently to solve a small task on a newly bought Xen based dedicated server.
The server had installed on itself SolusVM

The server was a good hard-iron machine running with CentOS Linux with enabled Xen virtualization support.
The Data Center (DC) has provided the client with 4 IP public addresses, whether the machine was assigned to possess only one MAC address!

The original idea was the dedicated server is supposed to use 4 of the IP addresses assigned by the DC whether only one of the IPs has an external internet connected ethernet interface with assigned MAC address.

In that case using Xen’s bridging capabilities was pretty much impossible and therefore Xen’s routing mode has to be used, plus an Iptables Network Address Translation or an IP MASQUERADE .

In overall the server would have contained 3 virtual machines inside the Xen installed with 3 copies of:

  • Microsoft Windows 2008

The scenario I had to deal with is pretty much explained in Xen’s Networking wiki Two Way Routed Network

In this article I will describe as thoroughfully as I can how I configured the server to be able to use the 3 qemu virtual machines (running inside the Xen) with their respective real interner visible public IP addresses.

1. Enable Proxyarp for the eth0 interface

To enable proxyarp for eth0 on boot time and in real time on the server issue the commands:

[root@centos ~]# echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp[root@centos ~]# echo 'net.ipv4.conf.all.proxy_arp = 1' >> /etc/sysctl.conf

2. Enable IP packet forwarding for eth interfaces

This is important pre-requirement in order to make the iptables NAT to work.

[root@centos ~]# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
[root@centos ~]# echo 'net.ipv6.conf.all.forwarding=1' >> /etc/sysctl.conf

If you get errors during execution of /etc/init.d/xendomains , like for example:

[root@centos ~]# /etc/init.d/xendomains restart
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv4/conf/eth0/proxy_arp: No such file or directory
/etc/xen/scripts/network-route: line 29: /proc/sys/net/ipv6/conf/eth0/proxy_arp: No such file or directory

in order to get rid of the message you will have to edit /etc/xen/scripts/network-route and comment out the lines:

echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp
#echo 1 >/proc/sys/net/ipv4/conf/${netdev}/proxy_arp
#echo 1 > /proc/sys/net/ipv6/conf/eth0/proxy_arp

3. Edit /etc/xen/xend-config.sxp, disable ethernet bridging and enable eth0 routing (route mode) and NAT for Xen’s routed mode

Make absolutely sure that in /etc/xen/xend-config.sxp the lines related to bridging are commented.
The lines you need to comment out are:

(network-script network-bridge)
(vif-script vif-bridge)

make them look like:

#(network-script network-bridge)
#(vif-script vif-bridge)br />

Now as bridging is disabled let’s enable Xen routed network traffic as an bridged networking alternative.

Find the commented (network-script network-route) and (vif-script vif-route) lines and uncomment them:

#(network-script network-route)
#(vif-script vif-route)

The above commented lines should become:

(network-script network-route)
(vif-script vif-route)

Next step is to enable NAT for routed traffic in Xen (necessery to make routed mode work).
Below commented two lines in /etc/xen/xend-config.sxp, should be uncommented e.g.:

#(network-script network-nat)
#(vif-script vif-nat)

Should become:

(network-script network-nat)
(vif-script vif-nat)

4. Restart Xen control daemon and reload installed Xen’s Virtual Machines installed domains

To do so invoke the commands:

[root@centos ~]# /etc/init.d/xend
[root@centos ~]# /etc/init.d/xendomains restart

This two commands will probably take about 7 to 10 minutes (at least they took this serious amount of time in my case).
If you think this time is too much to speed-up the procedure of restarting Xen and qemu attached virtual machines, restart the whole Linux server, e.g.:

[root@centos ~]# restart

5. Configure iptables NAT rules on the CentOS host

After the server boots up, you will have to initiate the following ifconfig & iptables rules in order to make the Iptables NAT to work out:

echo > > /proc/sys/net/ipv4/conf/tap1.0/proxy_arp
/sbin/ifconfig eth0:1 netmask
/sbin/ifconfig eth0:2 netmask
/sbin/ifconfig eth0:3 netmask

/sbin/iptables -t nat -A PREROUTING -d -i eth0 -j DNAT --to-destination
/sbin/iptables -t nat -A PREROUTING -d -i eth0 -j DNAT --to-destination
/sbin/iptables -t nat -A PREROUTING -d -i eth0 -j DNAT --to-destination
/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source
/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source
/sbin/iptables -t nat -A POSTROUTING -s -o eth0 -j SNAT --to-source

In the above ifconfig and iptables rules the IP addresses:,, are real IP addresses visible from the Internet.
In the above rules eth0:1, eth0:2 and eth0:3 are virtual ips assigned to the main eth0 interface.

This ifconfig and iptables setup assumes that the 3 Windows virtual machines running inside the Xen dedicated server will be configured to use (local) private network IP addresses:, and

You will have also to substitute the, and with your real IP addreses.

To store the iptables rules permanently on the fedora you can use the iptables-save command:

[root@centos ~]# /sbin/iptables-save

However I personally did not use this approach to save my inserserted iptable rules for later boots but I use my small script to add virtual interfaces and iptables rules via the /etc/rc.local invokation:

If you like the way I have integrated my virtual eths initiation and iptables kernel firewall inclusion, download my script and set it to run in /etc/rc.local, like so:

[root@centos ~]# cd /usr/sbin
[root@centos sbin]# wget
[root@centos ~]# chmod +x /usr/sbin/
[root@centos ~]# mv /usr/sbin
[root@centos ~]# echo '/usr/sbin/' >> /etc/rc.local

Note that you will have to modify my script to substitute the, and with your real IP address.

So far so good, one might think that all this should be enough for the Virtual Machines Windows hosts to be able to connect to the Internet and Internet requests to the virtual machines to arrive, but no it’s not!!

6. Debugging Limited Connectivity Windows LAN troubles on the Xen dedicated server

Even though the iptables rules were correct and the vif route and vif nat was enabled inside the Xen node, as well as everything was correctly configured in the Windows 2008 host Virtual machines, the virtual machines’s LAN cards were not able to connect properly to connect to the internet and the Windows LAN interface kept constantly showing Limited Connectivity! , neither a ping was available to the gateway configured for the Windows VM host (which in my case was:

You see the error with Limited connectivity inside the Windows on below’s screenshot:

Limited Connectivty Windows error Lan Interface, status screenshot

Here is also a screenshot of my VNC connection to the Virtual machine with the correct IP settings – (TCP/IPv4) Properties Window:

Windows Xen Network Connections Windows VNC TCP/IPv4 Properties Window

This kind of Limited Connectivity VM Windows error was really strange and hard to diagnose, thus I started investigating what is wrong with this whole situation and why is not able the Virtualized Windows to connect properly to the Internet, through the Iptables NAT inbound and outbound traffic redirection.

To diagnose the problem, I started up with listing the exact network interfaces showing to be on the Xen Dedicated server:

[root@centos ~]# /sbin/ifconfig |grep -i 'Link encap' -A 1
eth0 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
eth0:1 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
eth0:2 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
eth0:3 Link encap:Ethernet HWaddr 00:19:99:9C:08:3A
inet addr: Bcast:
lo Link encap:Local Loopback
inet addr: Mask:
tap1.0 Link encap:Ethernet HWaddr FA:07:EF:CA:13:31
vifvm101.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr: Bcast:

I started debugging the issue, using the expelling logic.
In the output concerning my interfaces via ifconfig on eth0, I have my primary server IP address , this one is working for sure as I was currently connected to the server through it.

The other virtual IP addresses assigned on the virtual network interfaces eth0:1, eth0:2 and eth0:3 were also assigned correctly as I was able to ping this ips from my Desktop machine from the Internet.

The lo , interface was also properly configured as I could ping without a problem the loopback ip –

The rest of the interfaces displayed by my ifconfig output were: tap1.0, vifvm101.0

After a bit of ressearch, I’ve figured out that they’re virtual interfaces and they belong to the Xen domains which are running qemu virtual machines with the Windows host.

I used tcpdump to debug what kind of traffic does flow through the tap1.0 and vifvm101.0 interfaces, like so

[root@centos ~]# tcpdump -i vifvm101.0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vifvm101.0, link-type EN10MB (Ethernet), capture size 96 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel
[root@centos ~]# tcpdump -i tap1.0
cpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1.0, link-type EN10MB (Ethernet), capture size 96 bytes
08:55:52.490249 IP > UDP, length 42

I’ve figured out as it’s also observable in above’s two tcpdump commands output, that nothing flows through the vifvm101.0 interface, and that there was some traffic passing by tap1.0 interface.

7. Solving the Limited Connectivy Windows Internet network connection problems

As below’s ifconfig output reveals, there is no IP address assigned to tap1.0 interface, using some guidelines and suggestions from guys in’s #netfilter irc channel, I’ve decided to give a go to set up an IP address of to tap1.0 .

I choose for a reason as this IP address is configured to be my Gateway’s IP Address inside the Emulated Windows 2008 hosts

To assign the to tap1.0, I issued:

[root@centos ~]# /sbin/ifconfig tap1.0 netmask
To test if there is difference I logged in to the Virtual Machine host with gtkvncviewer (which by the way is a very nice VNC client for Gnome) and noticed there was an established connection to the internet inside the Virtual Machine 😉

I issued a ping to google which was also returned and opened a browser to really test if everything is fine with the Internet.
Thanks God! I could browse and everything was fine 😉

8. Making tap1.0 (VM hosts gateway to be set automatically, each time server reboots)

After rebooting the server the tap1.0 assignmend of disappeared thus I had to make the, be assigned automatically each time the CentoS server boots.

To give it a try, I decided to place /sbin/ifconfig tap1.0 netmask into /etc/rc.local, but this worked not as the tap1.0 interface got initialized a while after all the xendomains gets initialized.

I tried few times to set some kind of sleep time interval with the sleep , right before the /sbin/ifconfig tap1.0 … ip initialization but this did not worked out, so I finally completely abandoned this methodology and make the tap1.0 get initialized with an IP through a cron daemon.
For that purpose I’ve created a script to be invoked, every two minutes via cron which checked if the tap1.0 interface is up and if not issues the ifconfig command to initialize the interface and assign the IP to it.

Here is my shell script

To set it up on your host in /usr/sbin issue:

[root@centos ~]# cd /usr/sbin/
[root@centos sbin]# wget
In order to set it on cron to make the tap1.0 initialization automatically every two minutes use the cmd:

[root@centos ~]# crontab -u root -e

After the cronedit opens up, place the cron invokation rules:

*/2 * * * * /usr/sbin/ >/dev/null 2>&1

and save.

That’s all now your Xen dedicated and the installed virtual machines with their public internet IPs will work 😉
If this article helped you to configure your NAT routing in Xen drop me a thanks message, buy me a beer or hire me! Cheers 😉