Walking in Light With Christ - hip0's personal blog on System Administration, Free Software, Computer Security and Religion

rkhunter, chkrootkit and unhide three Linux handy tools to find out if a Linux server is compromised

I've been hired, just recently by a company to make them a quick security audit analysis as they complain of having severe Denial of Service problems against an online e-store.

As a mean of securing the local network of the company, the old Linux server had to be replaced with a freshly installed one, which I had to configure to run smooth firewall and create DMZ zone.
The company System Administrator has installed Debian Linux and brought up sshd to allow me to futher take the lead and secure the server
After few hours, I've noticed unusual activities. For example processes like:

./a 207
as well as processes like ./ssh 212

debian:~# ps axu|grep -i ./ssh|wc -l
216


On the server and after checking in /etc/shadow I found that users like admin, test had a password string assigned , interestingly even system users like postgresql and man was enabled and ready for login and had a password hash assigned in /etc/shadow

I've seen the ./ssh shit before and I know pretty well this is a famous romanian script kiddies tools which circulates in the wild for at least 5 years already.

A quick look up in the usual places, where script kiddies store their files, like: /var/tmp, /tmp, /proc, /sys and /var/run has helped me find the ssh brute forcer the kiddie has issues on the machine in my case it was located in /var/run/2012 inside I found of course the usual sk bad stuff:

root@debian:/var/run/2012# ls 1 3 5 a data.conf find internet.txt pass_file screen ssh stricat.txt vuln.txt
2 4 64.37.find.22 atack derbedeu.txt full mfu.txt ronet13.txt ss start tefac.txt x


As well as an archive in /var/run/2012.tgz which was obviously quickly extracted and the brute force ssh tool was started to lookup for some more targets.

As I like keeping script kiddie stuff to expose and share with people, to help them get a better understanding what they can expect on their servers I made a copy of the 2012.tgz brute forcer tool, download the romanian ssh brute forcer shit is here if somebody wants to take a look, though I suggest to be very careful with it as some files might contain rootkits and other unwanted tool.
If someone wants to give it a try be sure not to launch it as root.

Anyways, after finding the abuser I quickly removed all the active users, which were not supposed to be existent on the newly installed system from /etc/shadow changed the server root password, give a reinstall the openssh-server and openssh-client (as I was not if they had not been substituted with some rootkited version), e.g.:

debian:~# apt-get install --reinstall openssh-client openssh-server


Consequentially I killed all the active ./ssh and ./a processes to assure the script kiddie ssh brute forcer is no longer running on the server as well killed all connections to the ssh server manually with kill cmd to the unknown IPs.

Finally to make sure, the server is not rootkited or not backdoored I run some tests with chkrootkit, rkhunter and unhide :

On Debian the 3 tools are available as packages, so this saved me time and I proceeded installing like so:

debian:~# apt-get install rkhunter unhide chkrootkit
...


In few seconds the tools were installed so I used them to check if the server is irreversably damaged or root kitted by the kiddie.

1. unhide

First I used unhide to make sure there are no hidden backdoor processes listening on the server:

for i in $(echo proc sys brute); do unhide $i; done
Unhide 20100201
http://www.security-projects.com/?Unhide
[*]Searching for Hidden processes through /proc scanning
Unhide 20100201
http://www.security-projects.com/?Unhide
[*]Searching for Hidden processes through kill(..,0) scanning
[*]Searching for Hidden processes through comparison of results of system calls
[*]Searching for Hidden processes through getpriority() scanning
[*]Searching for Hidden processes through getpgid() scanning
[*]Searching for Hidden processes through getsid() scanning
[*]Searching for Hidden processes through sched_getaffinity() scanning
[*]Searching for Hidden processes through sched_getparam() scanning
[*]Searching for Hidden processes through sched_getscheduler() scanning
[*]Searching for Hidden processes through sched_rr_get_interval() scanning
[*]Searching for Hidden processes through sysinfo() scanning
HIDDEN Processes Found: 1
Unhide 20100201
http://www.security-projects.com/?Unhide
[*]Starting scanning using brute force against PIDS with fork()
[*]Starting scanning using brute force against PIDS with Threads


The above command checks with unhide in /proc and /sys for hidden processes as well as next uses brute option to try to brute force all PIDs on the server attempting to locate listening backdoors processes.

There are 2 more unhide binaries that can be used to check for hidden backdoors, unhide-posix and unhide-tcp

For some reason unhide-posix sys detected the /usr/sbin/rsyslogd -c4 process as some kind of hidden process, which is most probably a false positive, though I can't be one hundred sure until I try to scan completely the server remotely after mounting the filesystem via ssh and scan it with clamav chkrootkit and rkhunter and even maybe with drweb .

I assume the HIDDEN Processes Found: 1 bells the alarm, however even though I did profound look up on the server with lsof, netstat and fuser cmds I cannot find nothing suspicious any more.

2. chkrootkit scan

Next I used chkrootkit to check if some common rootkit is not installed on the server:

debian:~# chkrootkit
ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `crontab'... not infected
Checking `date'... not infected
...


The chkrootkit output displayed thanksfully failed to detect that the system is rootkitted.

3. rkhunter

Finally, tested with rkhunter to assure chkrootkit to expand the range of rootkits the system is scanned for:

debian:~# rkhunter --check
[ Rootkit Hunter version 1.3.6 ]

Checking system commands...

Performing 'strings' command checks
Checking 'strings' command [ OK ]

Performing 'shared libraries' checks
Checking for preloading variables [ None found ]
Checking for preloaded libraries [ None found ]
Checking LD_LIBRARY_PATH variable [ Not found ]
...


After a long screen of consequential rootkit tests, each of which requires the user to press enter rkhunter was unable to detect any rootkits installed on host as well.

Finally to make sure for one more time there are no some backdoors left or some unwanted users still connected to the server I used netstat:

debian:~# netstat -tanp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 7269/sshd
tcp 0 0 192.168.0.1:22 83.228.93.76:54913 ESTABLISHED 24545/1
tcp 0 0 192.168.0.1:22 83.228.93.76:50240 ESTABLISHED 7059/0
tcp6 0 0 :::22 :::* LISTEN 7269/sshd


As the output shows, I coulnd't find anything suspiciou, still I can't be 100% percent sure the system is clean and there is no something left from the cracker, probably the Debian server will be re-installed as it doesn't matter since it's newly installed Debian machine.

What is amazing is that the server was compromised immediately after it was installed. This means that either the very easy default root password the admin who installed the server was cracked, or his whole network has been compromised by the script kiddie.
Tomorrow will have to investigate further to assure the security breach is closed hopefully once and for all.