Archive for July, 2011
Thursday, July 14th, 2011 
Just recently it was necessery to load up a tun kernel module on few CentOS Linux servers.
I’m using Debian on daily basis, and everybody that had even little of experience with Debian should already be aware about the existence of the handy:
/etc/modules file.
On Debian to enable a certain kernel module to load up on Linux boot, all necessery is to just place the kernel module name in /etc/modules.
For example loading the tun tunneling kernel module I issue the command:
debian:~# echo tun >> /etc/modules
I wondered if CentOS, also supports /etc/modules as it was necessery now to add this tun module to load up on CentOS’s boot.
After a bit of research I’ve figured out CentOS does not have support for adding modules names in /etc/modules , anyhow after consulting CentOS documentation on http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-kernel-modules-persistant.html , I found CentOS and RHEL use /etc/rc.modules instead of Debian’s /etc/modules to load up any custom kernel modules not loaded by default during system boot.
Therefore instructing the RHEL Linux to load up my desired tun module in kernel on next boot was as easy as executing:
[root@centos ~]# echo 'modprobe tun' >> /etc/rc.modules
[root@centos ~]# chmod +x /etc/rc.modules
Now on next boot CentOS will load up the tun module in kernel. Achiving the same module load up is also possible through /etc/rc.local , but it’s not recommended way as /etc/rc.local would load up the kernel module after all of the rest init boot scripts complete and therefore will load up the module slightly later, at the final boot stage.
Tags: basis, boot, boot scripts, boot stage, CentOS, command, custom, custom kernel, daily basis, Debian, deployment guide, everybody, existence, experience, final boot, kernel, kernel module, kernel modules, Linux, linux servers, modprobe, Module, modulesNow, name, necessery, rhel, root, stage, support, system boot, use, way
Posted in Linux, System Administration | No Comments »
Thursday, July 14th, 2011 
My WordPress blog had an odd problem, all on the blog, worked just fine except the Widgets add new widget drag and drop functionality.
As there was new version of wordpress available I tried updating but this fixed the missing drag and drop widget functionality neither, so finally after a bit of googling and experimentation to follow suggested fixes like using the enable Accessiblity Mode button on top Widgets.Thus the common suggested fix that worked for most of the people who stumbled upon this annoying wordpress bug was to simply nagivate to:
Appearance -> Widgets -> Screen Options -> Enable Accessibility Mode
As for me pressing over the Screen Options was impossible (nothing happened when I pressed over it), finally I’ve found a hack 😉
The hack is quite simple, all I had to do is just pass a variable ?widgets-access=on to the widget.php php script
Therefore the fix to the problem was as follows http://blog.mysite.com/wp-admin/widgets.php?widgets-access=on
And done, FIXED! 😉
Tags: Appearance, blog, Button, common, drag, drop functionality, fine, fix, FIXED, functionality, googling, hack, Mode, mode button, ModeAs, mysite com, nagivate, odd problem, php, problem, screen, screen options, script, scriptTherefore, version, widget, Widgets, Wordpress, wordpress blog
Posted in Web and CMS, Wordpress | 3 Comments »
Wednesday, July 13th, 2011 One server with a broken Raid array was having troubles with it’s software raid.
I tried to scan the raid array via a rescue cd like so:
server:~# mdadm --assemble --scan /dev/md1
just to be suprised by the message:
mdadm: /dev/md1 assembled from 2 drives – not enough to start the array.
In /proc/mdstat respectively the raid was showing inactive, e.g.:
server:~# cat /proc/mdstat
Personalities : [raid10] [raid1]
md1 : inactive sda2[0] sdc2[2] sdb2[1]
12024384 blocks
Respectively trying to activate the software Linux raid array with:
server:~# mdadm -A -s
Couldn’t be completed because of the same annoying error:
/dev/md1 assembled from 2 drives – not enough to start the array.
Thanksfully finally thanks to some Russian, who posted having same issues reported to be able to active his software RAID with mdadm’s –force option.
Thus enabling the problematic RAID 5 array was possible with:
server:~# mdadm -A -s --force
This solution of course is temporary and will have to further check what’s wrong with the array, however at least now I can chroot to the server’s / directory. 😉
Tags: annoying error, cat, check, course, dev, drive, ERROR, force option, Linux, md1, mdstat, option, Personalities, proc, Raid, raid 5 array, raid array, rescue, rescue cd, sCould, sda, sdb, sdc, software, software linux, software raid, solution, Thanksfully, way
Posted in Linux, System Administration | 1 Comment »
Tuesday, July 12th, 2011 
Are you an administrator of servers and it happens a server is DOWN.
You request the Data Center to reboot, however suddenly the server fails to boot properly and you have to request for IPKVM or some web java interface to directly access the server physical terminal …
This is a very normal admin scenario and many people who have worked in the field of remote system administrators (like me), should have experienced that bad times multiple times.
Sadly enough only a insignifant number of administrators try to do their best to reduce this down times to resolve client stuff downtime but prefer spending time playing the ztype! game or watching some porn website 😉
Anyways there are plenty of things like Server Auto Reboot on Crash with software Watchdog etc., that we as sysadmins can do to reduce server downtimes and most of the manual human interactions on server boot time.
In that manner of thougts a very common thing when setting up a new Linux server that many server admins forget or don’t know is to enable all the server partition filesystems to be auto fscked during server boot time.
By not enabling the auto filesystem check options in Linux the server filesystems did not automatically scan and fix hard drive partitions for fs innode inconsistencies.
Even though the filesystems are tuned to automatically get checked on every 38 system reboots, still if some kind of filesystem errors are found that require a manual confirmation the boot process is interrupted and the admin ends up with a server which is not reachable remotely via ssh !
For the remote system administrator, this times are a terrible times of waitings, prayers and hopes that the server hardware is fine 😉 as well as being on hold to get a KVM to get into the server manually and enter the necessery input to fsck prompt.
Many of this bad times can be completely avoided with a very simple fix through /etc/fstab by enabling all server partitions containing any filesystem to be automatically checked and fixed in case if inconsistencies or errors are found by fsck.ext3, fsck.ext4, fsck.reiserfs etc. commands.
A very typical default /etc/fstab file you will find on many servers should look something like:
/dev/sda8 / ext3 errors=remount-ro 0 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda1 /home ext3 defaults 0 0
Notice the line:
/dev/sda1 /home ext3 defaults 0 0
The first column in the example contains the device name, the second one its mount point, third its filesystem type, fourth the mount options, fifth (a number) dump options, and sixth (another number) filesystem check options. Let’s take a closer look at this stuff.
The ones which are interesting to enable auto fsck checking and error resolving is provided usually by the last sixth variable (filesystem check option) which in the above example equals 0 .
When the filesystem check option equals 0 this means the auto fsck and repair for the respective filesystem is disabled.
Some time in the past the dump backup option (5th option in the example) was also used but as far as I can understand today it’s not that important in modern GNU/Linux distributions.
Now having the above sample crontab in order to enable the fsck file checking on Linux boot for /dev/sda1 , we will need to modify the above line’s filesystem check option be 2, e.g. the line would afterwards look like:
/dev/sda1 /home ext3 defaults 0 2
Setting the 2 as an option for filesystem check is necessery for every filesystem which is not mounted as a root filesystem /
In above example /etc/fstab you already see that auto filesystem fsck is enabled for root partition:
/dev/sda8 / ext3 errors=remount-ro 0 1
(notice the 1 in the end of the line)
Finally a modified version of the default sample /etc/fstab which will check the extra /dev/sda1 /home partition would look like so:
/dev/sda8 / ext3 errors=remount-ro 0 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda1 /home ext3 defaults 0 2
Making sure all Linux server partitions has the auto filesystem check option enabled is something absoultely necessery!
Enabling the auto fsck on servers always makes me sleep calmer 😉
Hope it helps your too. 🙂
Tags: auto reboot, boot process, boot time, center, client, crash, data, ext, file, filesystem errors, hard drive partitions, human interactions, inconsistencies, java interface, linux server, multiple times, necessery, number, option, partition, physical terminal, porn website, reiserfs, root, sda, server boot, server downtimes, server hardware, shm, software, something, spending, spending time, system administrators, terminal, terrible times, time, Watchdog, web java, ztype
Posted in Linux, System Administration | 2 Comments »
Monday, July 11th, 2011 
If you are stuffed with errors in /var/log/mysqld.log similar to:
110711 11:00:48 [ERROR] /usr/libexec/mysqld: Incorrect information in file: './anyboots_moncler_spaccio/zen_seo_cache.frm'
110711 11:00:48 [ERROR] /usr/libexec/mysqld: Incorrect information in file: './anyboots_moncler_spaccio/zen_sessions.frm'
This is a sure sign something terrible has happened with your mysql database tables that lead to corruption.
Having corrupt table in mysql installation can severely lead to data loss as well as significantly reduce the speed and performance of a MySQL server in this awful times mysqlcheck is the best friend of the administrator, here is how you can check and repair broken tables in MySQL server:
mysql-server:~# mysqlcheck --all-databases -u root -p
chillor_hjbgl.vn_users OK
chillor_lul.mybb_adminlog OK
chillor_lul.mybb_adminoptions OK
chillor_lul.mybb_adminsessions OK
chillor_lul.mybb_adminviews OK
chillor_lul.mybb_announcements OK
...
You will notice the corrupt sql tables will be reported as corrupt by the tool and mysqlcheck will try it’s best to recover the corrupt tables.
In most cases this should be enough to recover corrupt tables.
Tags: adminlog, best friend, corruption, database tables, databases, ERROR, file, hjbgl, installation, libexec, loss, lul, moncler, mybb, Mysql, mysql database, mysql server, mysqld, OKchillor, pchillor, performance, repair, root, SEO, server mysql, sessions, something, spaccio, speed and performance, sql tables, tool, usr, var, zen
Posted in Linux, MySQL, System Administration | No Comments »
Saturday, July 9th, 2011 
These days, I’m managing many, many servers. The servers are ordered in few groups. Each of the servers in the server groups contains identical hardware, identical Linux distribution as well as identical configuration.
Since managing multiple servers normally, takes a lot of time, applying changes to every single host loosing time in looking for the password is not a a good idea.
Thus I was forced to start managing the servers in a cluster like fashion, by executing commands on a server group using a simple for bash loop etc.
To be able to use this mass execution of course I needed away either to pass on the server group password just once and issue a command on the whole server group or use a passwordless authentication ssh key pair.
Before I switched to using SSH keys to authenticate passwordless, I first tried to use a bit of tools which were claimed to be helpful in the task of executing the same commands to a group of servers. I have tested the tools pssh, sudossh and dsh but none of them was capable of logging in and executing a bunch of commands to the group of remote servers.
I gave my best to make pssh work on Debian and CentOS distributions, but even though all my experiemnts and efforts to make the so much talked about pssh to work were bad!
I’ve seen also another tool called Cluster SSH which was said to be capable of issuing commands simultaneously on a number of hosts.
Cluster SSH looked promising, however the only problem was it’s supposed to run under xterm or some kind of X graphics based terminal and therefore it did not matched my desired.
Finally I got pissed of trying these mostly useless mass command linux server administration tools and decided to come COME BACK TO THE PRIMITIVE 😉 and use the good all known, well established method of Passwordless SSH server login with ssh public/private DSA key auth.
Therefore here the problem come to this to generate one single DSA ssh authenticatoin key and replicate/copy it to the whole group of 50 servers.
These task initially seemed quite complex, but with the help of a one liner bash shell script, it seemed to be a piece of cake 😉
To achieve this task, all I had to do is:
a. Generate an SSH key with ssh-keygen command
and
b. Use a one liner shell script to copy the generated id_rsa.pub file, to each server.
and
c. Create a file containig all server IP addresses to pass to the shell script.
Here are the two lines of code you will have to use to achieve these tasks:
1. Generate a DSA ssh key
linux:~# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/hipo/.ssh/id_dsa): y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in y.
Your public key has been saved in y.pub.
The key fingerprint is:
b0:28:48:a2:60:65:5a:ed:1b:9d:6c:ff:5f:37:03:e3 hipo@www.pc-freak.net
Here press enter few times and be sure not to fill in any passphrase when asked ’bout it.
2. Create a file containing all server IPs
Just create a file let’s say server-list.txt and make sure you include all the server IPs, one per line.
3. Use the following little script to upload the newly generated id_dsa.pub to the server list
linux:~# while read line; do ssh-copy-id -i ~/.ssh/id_dsa.pub root@"$line"; done < server-list.txt
Now you will have to paste the server password for about 50 times (if you have a file with 50 servers), however the good news is it will be just a wait and paste 50 times, if the servers are all configured to have the same root administrator pass (which was the case with me).
So until you do the boring pasting part, you can start up a nice music and enjoy 😉
Cheers 😉
Tags: administration tools, authentication, BACK, CentOS, course, distributions, DSA, dsh, fashion, file, hipo, host, hosts, Linux, linux distribution, mass, mass execution, none, passphrase, password, PRIMITIVE, pssh, root, server administration, server group, server groups, server login, servers, Shell, ssh server, terminal, time, tool, work, xterm
Posted in Linux, Linux and FreeBSD Desktop, Programming, System Administration | 2 Comments »
Saturday, July 9th, 2011 I’m currently writting a script which is supposed to be adding new crontab jobs and do a bunch of other mambo jambo.
By so far I’ve been aware of only one way to add a cronjob non-interactively like so:
linux:~# echo '*/5 * * * * /root/myscript.sh' | crontab -
Though using the | crontab – would work it has one major pitfall, I did completely forgot | crontab – OVERWRITES CURRENT CRONTAB! with the crontab passed by with the echo command.
One must be extremely careful if he decides to use the above example as you might loose your crontab definitions permanently!
Thanksfully it seems there is another way to add crontabs non interactively via a script, as I couldn’t find any good blog which explained something different from the classical example with pipe to crontab –, I dropped by in the good old irc.freenode.net to consult the bash gurus there 😉
So I entered irc and asked the question how can I add a crontab via bash shell script without overwritting my old existing crontab definitions less than a minute later one guy with a nickname geirha was kind enough to explain me how to get around the annoying overwridding.
The solution to the ovewrite was expected, first you use crontab to dump current crontab lines to a file and then you append the new cron job as a new record in the file and finally you ask the crontab program to read and insert the crontab definitions from the newly created files.
So here is the exact code one could run inside a script to include new crontab jobs, next to the already present ones:
linux:~# crontab -l > file; echo '*/5 * * * * /root/myscript.sh >/dev/null 2>&1' >> file; crontab file
The above definition as you could read would make the new record of */5 * * * * /root/myscript.sh >/dev/null be added next to the existing crontab scheduled jobs.
Now I’ll continue with my scripting, in the mean time I hope this will be of use to someone out there 😉
Tags: bash scripts, bash shell script, blog, classical example, cron, cron job, cron jobs, crontab, definitions, dev, echo, echo 5, echo command, exact code, file, fileThe, gurus, irc, jambo, kind, line, Linux, mambo jambo, mean time, minute, nbsp nbsp nbsp nbsp nbsp, nickname, ovewrite, pipe, pitfall, root, scripting, Shell, solution, someone, Thanksfully, time, use, way
Posted in Linux, System Administration | 9 Comments »
Friday, July 8th, 2011 
Some long time ago I’ve written an article Optimizing Linux tcp/ip networking
In the article I’ve examined a number of Linux kernel sysctl variables, which significantly improve the way TCP/IP networking is handled by a non router Linux based servers.
As the time progresses I’ve been continuing to read materials on blogs and internet sites on various tips and anti Denial of Service rules which one could apply on newly installed hosting (Apache/MySql/Qmail/Proxy) server to improve webserver responce times and tighten the overall security level.
In my quest for sysctl 😉 I found a few more handy sysctl variables apart from the old ones I incorporate on every Linux server I adminstrate.
The sysctl variables improves the overall network handling efficiency and protects about common SYN/ACK Denial of service attacks.
Here are the extra sysctl variables I started incorporating just recently:
############ IPv4 Sysctl Settings ################
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
##################################################
#
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routers
net.ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignments
net.ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interface
net.ipv6.conf.default.max_addresses = 1
#####################################################
To use this settings paste the above sysctl variables in /etc/sysctl.conf and ask sysctl command to read and apply the newly added conf settings:
server:~# sysctl -p
...
Hopefully you should not get errors while applying the sysctl settings, if you get some errors, it’s possible some of the variable is differently named (depending on the Linux kernel version) or the Linux distribution on which sysctl’s are implemented.
For some convenience I’ve created unified sysctl variables /etc/sysct.conf containing the newly variables I started implementing to servers with the ones I already exlpained in my previous post Optimizing Linux TCP/IP Networking
Here is the optimized / hardened sysctl.conf file for download
I use this exact sysctl.conf these days on both Linux hosting / VPS / Mail servers etc. as well as on my personal notebook 😉
Here is also the the complete content of above’s sysctl.conf file, just in case if somebody wants to directly copy/paste it in his /etc/sysctl.conf
# Sysctl kernel variables to improve network performance and protect against common Denial of Service attacks
# It's possible that not all of the variables are working on all Linux distributions, test to make sure
# Some of the variables might need a slight modification to match server hardware, however in most cases it should be fine
# variables list compiled by hip0
### https://www.pc-freak.net
#### date 08.07.2011
############ IPv4 Sysctl Kernel Settings ################
net.ipv4.ip_forward = 0
# ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0
# ( same as above )
net.ipv4.conf.default.accept_source_route = 0
# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
# ( - || - )net.ipv4.tcp_fin_timeout = 40
# ( Decrease FIN timeout ) - Useful on busy/high load server
net.ipv4.tcp_keepalive_time = 4000
# ( keepalive tcp timeout )
net.core.rmem_default = 786426
# Receive memory stack size ( a good idea to increase it if your server receives big files )
##net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608
#( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
# ( maximum amount of option memory buffers )
# tcp reordering, increase max buckets, increase the amount of backlost
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
##net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
###################################################
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routersnet.
ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignmentsnet.
ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interfacenet.
ipv6.conf.default.max_addresses = 1
#####################################################
# Reboot if kernel panic
kernel.panic = 20
These sysctl settings will tweaken the Linux kernel default network settings performance and you will notice the improvements in website responsiveness immediately in some cases implementing this kernel level goodies will make the server perform better and the system load might decrease even 😉
This optimizations on a kernel level are not only handy for servers, their implementation on Linux Desktop should also have a positive influence on the way the network behaves and could improve significantly the responce times of opening pages in Firefox/Opera/Epiphany Torrent downloads etc.
Hope this kernel tweakenings are helpful to someone.
Cheers 😉
Tags: adminstrate, amount, anti, apache mysql, autoconf, default, default network, default router, Denial, denial of service, denial of service attacks, exploits, file, harden, How to, imprpove, internet sites, ip networking, ipv, ipv4, ipv6, kernel, kernel level, level, Linux, Linux Security, linux server, memory, network efficiency, non, number, protection, proxy server, Qmail, quot, ra, randomize, reordering, responce, responce times, router linux, routers, rtr, security, security level, serverskernel, Service, solicitations, Source, SYN, syn flood attack, sysctl, time, variables, wmem
Posted in Computer Security, Linux, Linux and FreeBSD Desktop, System Administration | 4 Comments »
Thursday, July 7th, 2011 Every now and then I had to tamper routing tables in Linux and every time I had to do it I forgot how I did it last time so finally I decided to put it on my blog and find how I can delete from Linux routing table easier
Deleting a record from a wrong routing table on Linux is a piece of cake basicly, here is an example:
linux:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.1 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
Let’s say this is the routing table and it’s incorrect as the routing through the node 169.254.0.0 should not be there.
Here is how the routing through 169.254.0.0 can be deleted:
linux:~# /sbin/route del -net 169.254.0.0 netmask 255.255.0.0
Now here is the difference after deletion:
linux:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.1 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
If you want to delete the default gateway just use 0.0.0.0, e.g. -net 0.0.0.0 .. in above cmd example.
This would delete default gateway record from routing table which by the way in routing tables is marked with the UG flag.
Cheers 😉
Tags: basicly, blog, cake, Cheers, default gateway, deletion, difference, eth, flags, Genmask, Iface, last time, Linux, Metric, Netmask, nKernel, piece, piece of cake, Ref, routing, routing table, sbin, TABLE, tableDestination, time, way
Posted in Linux, System Administration | No Comments »
Thursday, July 7th, 2011 
If you're an administrator of a shared hosting server running mail server on localhost, you've definitely come across to issues with your mail server ip entering into public blacklists like spamhaus's CBL,XBL, PBL etc.
The usual procedure after one's ip gets listed in blacklists is to delist it manually following spamhaus or any other blacklist website's web interface, however often even after delisting yourself from blacklists you get back into them in a couple of hours, since your mail server continues to send a mass amounts of spam.
To track issues like as a system administrator I always use the good old network swiss army of knife tool tcpdump
tcpdump is really precious in tracking all kind of traffic oddities or mail server traffic.
To check if there are oddities with traffic flowing from a mail server on localhost after I login to a mail server with issues I use tcpdump command with following options:
tcpdump -nNxXi eth0 -s 0 proto TCP and port 25
The usual output of it should look something like:
root@hosting:/home/hipo/public_html:# tcpdump -nNxXi eth0 -s 0 proto TCP and port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:37:51.692685 IP xxx.xxx.xxx.xxx.smtp > 212.235.67.205.53745: P 2645817175:2645817203(28) ack 31168958 win 7632
0x0000: 4500 0044 92b4 4000 4006 9ae8 5511 9f4d E..D..@.@...U..M
0x0010: d4eb 43cd 0019 d1f1 9db3 f757 01db 99be ..C........W....
0x0020: 5018 1dd0 0d4e 0000 3235 3020 4f4b 2069 P....N..250.OK.i
0x0030: 643d 3151 656c 3150 2d30 3033 7666 412d d=1Qel1P-003vfA-
0x0040: 4730 0d0a G0..
11:37:52.175038 IP 212.235.67.205.53745 > xxx.xxx.xxx.xxx.smtp: . ack 28 win 65064
0x0000: 4500 0028 1bb4 4000 7706 db04 d4eb 43cd E..(..@.w.....C.
0x0010: 5511 9f4d d1f1 0019 01db 99be 9db3 f773 U..M...........s
0x0020: 5010 fe28 a1c8 0000 0000 0000 0000 P..(..........
In this example the xxx.xxx.xxx.xxx is the IP address of the hosting server (my mail server) and the other IP is the interaction of my mail server's smtp port 25 with tther machine 212.235.67.205.
If after issuing this command there are tons of repeating address IPs the mail server interacts with this is possible sign of spammers who sent traffic via the mail server.
Of course this is not always the case as sometimes, some clients use to send large newsletters or just some planned advertisements, however in most cases as I said it's a spammer.
To futher get the abuser I check Apache logs and the mail server logs. Also in many cases a spammer can be catched via observing the mail server logs (/var/log/maillog, /var/log/qmail/current or wherever the mail server logs it's interactions).
In the above tcpdump output you can even read some of the information flowing in between mail servers in a very raw form for example in above tcpdump output notice the 250.OK . This is obviously an interaction between the two mail servers where the server running on my hosting server with ip (xxx.xxx.xxx.xxx) sends to the remote mail server the command 250 OK
Hope this article is helpful to somebody 😉
Tags: ack, blacklists, cbl, command, course, delisting, eth, hipo, host, hosting server, How to, interaction, Knife, knife tool, localhost, log, login, logs, look, machine, mail server, mail servers, mass, mass amounts, n 250, oddities, PBL, proto, Qel, root, server ip, server traffic, size, smtp, something, spamhaus, spammer, swiss army, system administrator, tcpdump, tool, tther, verbose, vv, web interface, xxx
Posted in Linux, Monitoring, Networking, Qmail, System Administration | 3 Comments »
