Posts Tagged ‘running processes’

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness

Wednesday, July 24th, 2019

linux-why-du-and-df-shows-different-result-inconsincy-explained-filesystem-full-oddity

If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reports its file use different when checked with
df
cmd than when checked with du command, like for example:
 

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total

 

1. Simple debug on what might be the root cause for df / du inconsistency reporting

 

Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:

 

# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do

 

 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less

 

I've also investigated on the complete /var directory contents sorted by size with:

 

 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache


Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:
 

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql

 

So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).
 

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:
 

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www


FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:
 

# netstat -tupanl


Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files

 

# lsof -nP | grep '(deleted)'

COMMAND   PID   USER   FD   TYPE DEVICE    SIZE NLINK  NODE NAME
mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)

 

Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:

 

# lsof -nP | grep '(deleted)' |wc -l
438


As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):

 

# lsof +L1|less


The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.
 

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:

 

– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/libnss_files-2.28.so

 

# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/libm-2.28.so

Other command that helped to track the discrepancy between df and du different file usage on FS is:
 

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'
 

 

3. Fixing large files kept in memory filesystem problem


What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:

 

 

# systemctl restart mysqld.service


or on older init.d system V ones:

# /etc/init.d/service restart


For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l script.sh
7977 script.sh


# kill -HUP PID

 

Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.

 

# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total

 

What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?
 

Configure Linux users to see only their own user processes with Hidepid – Stop users to see what others are doing

Tuesday, December 23rd, 2014

configure-Linux-users-to-see-only-ther-own-processes-with-hidepid-ps-aux-stop-system-users-to-see-what-others-are-doing
If you administer a university shared free shell Linux server, have a small community of *NIX users offering free accounts for them, or responsible for Linux software company with development servers, where programmers login and use daily to program software / websites its necessery to have tightened security rules with a major goal to keep the different user accounts processes separate one from other (hide all system and user processes from single logged in user).

Preventing users to see other users processes is essential for Linux servers which are at high risk to be hacked. At earlier times to achieve hiding all processes besides own ones from a logged in user was possible by using A kernel security module Grsecurity.
In latest currenlt Linux kernel version 3.2+ (on both Debian (unstable) / Ubuntu 14.04 / RHEL/CentOS v6.5+ above) you can hide process from other user so only root (useruser) can see all running process with (ps auxwwf) with a native kernel option hidepid. 

Configuring Hidepid

To enable hidepid option you have to remount the /proc filesystem with the Linux kernel hardening hidepid option, to make it one time setting on already running server issue:
 

 mount -o remount,rw,hidepid=2 /proc


To make the hidepid setting permanently active its necessery to modify /proc filesystem settings in /etc/fstab


 

vim /etc/fstab

proc    /proc    proc    defaults,hidepid=2     0     0
 

  • hidepid=0 – Anybody may read all world-readable /proc/PID/* files (default).
  • hidepid=1 – Means users may not access any /proc/ / directories, but only ones owned by them.Important  files like cmdline, sched*, status are now protected to read from other other users.
  • hidepid=2 – Means hidepid=1 plus all /proc/PID/ will be invisible to other users besides logged in. Using this options stops Cracker's from gathering info about running processes, indication of daemon (services) which runs with elevated privileges, other user running processes (some might contain password) passed as argument or some sensitive data. Revealing such data is frequently used to get versions of local / remote running services that can be exploited.
     

Below is output of htop of a logged in user on hidepid activated server:

:htop_screenshot_on_hideid_showing-only-own-user-credentials-gnu-linux-debian

How to check what process is listening on network port with: lsof & fuser commands in Linux / BSD

Saturday, March 16th, 2013

It is a common thing for me as a Linux and FreeBSD sysadmin to know what process assignes to which port number? I'm sure many novice system administrators will end up sooner or later with same question. Knowing what kind of processes has listening to TCP and UDP protocol / ports is a must to have a proper configured Linux / BSD system. In Linux there are two commands ( lsof and fuser) with which you can get various extra PID information on running processes (i.e. get information which cannot otherwise be obtained via the usual ps and netstat  To use them on most Linux distributions, you will have to have them installed.

1. Install fuser / lsof on Linux / BSD

a) On RPM based Linux distros – Fedora, CentOS, RHEL, SuSE /sbin/fuser is usually part of base install psmisc rpm package, however /usr/sbin/lsof is not among standard installed rpms, so you have to manually install via yum:

[root@centos ~]# yum install -y lsof
….

b) On Deb based Linuxes (Debian, Ubuntu, Mint, ArchLinux etc.). both lsof and fuser has to be installed via a separate packages non-part of Debian base install packs.

server:~# apt-get --yes install lsof fuser
....

On Debian full path location of both is in /bin/fuser and /usr/bin/lsof.

Two tools are precious swiss army knife outfit to use, whether you doubt someone cracked into a server or in doubt you might have "hidden" cracker processes on server.

c) Install fuser on Free/Net/Open/ BSD

bsd# cd /usr/ports/sysutils/fuser
bsd# make install clean
....
bsd# cd /usr/ports/sysutils/lsof
bsd# make install clean
....

2. Using fuser and lsof to look up process PIDs assigned to port numbers

lsof name is derived from List Open Files, as we know in UNIX, GNU / Linux,  *BSD everything on the system is a file, thus to get all kind of information concerning all active files (i.e. ports, PIDs, procotols and process names;

server:~# lsof +M -i4 | less

COMMAND     PID     USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
portmap    1317   daemon    4u  IPv4     3930      0t0  UDP *:sunrpc[portmapper]
portmap    1317   daemon    5u  IPv4     3939      0t0  TCP *:sunrpc[portmapper] (LISTEN)
rpc.statd  1329    statd    4u  IPv4     3974      0t0  UDP *:657
rpc.statd  1329    statd    6u  IPv4     3983      0t0  UDP *:28530[status]
rpc.statd  1329    statd    7u  IPv4     3986      0t0  TCP *:58515[status] (LISTEN)
apache2    1625 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    1625 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
sshd       1918     root    3u  IPv4     4945      0t0  TCP *:ssh (LISTEN)
miniserv.  2155     root    5u  IPv4     5299      0t0  TCP *:20000 (LISTEN)
miniserv.  2155     root    6u  IPv4     5300      0t0  UDP *:20000
miniserv.  2161     root    6u  IPv4     5367      0t0  TCP *:webmin (LISTEN)
miniserv.  2161     root    7u  IPv4     5368      0t0  UDP *:10000
ntpd       2172      ntp   16u  IPv4     5395      0t0  UDP *:ntp
ntpd       2172      ntp   18u  IPv4     5402      0t0  UDP localhost:ntp
ntpd       2172      ntp   19u  IPv4     5403      0t0  UDP iqtest.soccerfame.com:ntp
ntpd       2172      ntp   20u  IPv4    16028      0t0  UDP secure.broomlake.com:ntp
apache2    4505 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4505 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4539 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4539 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4780 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4780 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4900 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4900 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4907 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4907 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    4915 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    4915 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5067 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5067 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5133 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5133 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5134 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5134 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5148 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5148 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5152 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5152 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5259 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5259 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5265 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5265 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5266 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5266 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5346 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5346 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5356 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5356 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5467 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5467 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5523 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5523 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5568 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5568 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5715 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5715 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5716 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5716 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5758 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5758 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    5789 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    5789 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2    6106 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2    6106 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   16608 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   16608 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   16904 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   16904 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   17124 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   17124 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   17280 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   17280 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   20855 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   20855 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   20920 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   20920 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   21023 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   21023 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   22182 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   22182 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23307 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23307 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23366 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23366 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23408 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23408 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23419 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23419 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23428 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23428 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23452 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23452 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23561 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23561 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23579 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23579 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   23851 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   23851 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   24103 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   24103 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   24659 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   24659 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
sshd      25073     root    3u  IPv4 29855891      0t0  TCP iqtest.soccerfame.com:ssh->pc-freak.net:50176 (ESTABLISHED)
sshd      25084     hipo    3u  IPv4 29855891      0t0  TCP iqtest.soccerfame.com:ssh->pc-freak.net:50176 (ESTABLISHED)
apache2   25089 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   25089 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   26737 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   26737 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   27243 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   27243 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   27282 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   27282 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   27633 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   27633 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   28205 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   28205 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29244 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29244 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29372 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29372 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29411 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29411 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29462 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29462 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   29548 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   29548 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   30161 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   30161 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   31876 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   31876 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   31958 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   31958 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32052 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32052 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32061 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32061 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32143 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32143 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32149 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32149 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32440 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32440 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32635 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32635 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   32790 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   32790 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40211 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40211 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40309 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40309 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40432 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40432 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   40476 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   40476 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46319     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46319     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46438 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46438 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46439 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46439 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46440 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46440 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46441 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46441 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46442 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46442 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46443 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46443 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46509     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46509     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46510     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46510     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   46515     root    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   46515     root    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   51287 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   51287 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   51485 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   51485 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   51804 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   51804 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
named     54418     bind   20u  IPv4 31298857      0t0  TCP localhost:domain (LISTEN)
named     54418     bind   21u  IPv4 31298859      0t0  TCP iqtest.soccerfame.com:domain (LISTEN)
named     54418     bind   22u  IPv4 31298861      0t0  TCP secure.broomlake.com:domain (LISTEN)
named     54418     bind   23u  IPv4 31298865      0t0  TCP localhost:953 (LISTEN)
named     54418     bind  512u  IPv4 31298856      0t0  UDP localhost:domain
named     54418     bind  513u  IPv4 31298858      0t0  UDP iqtest.soccerfame.com:domain
named     54418     bind  514u  IPv4 31298860      0t0  UDP secure.broomlake.com:domain
named     54418     bind  515u  IPv4 31298864      0t0  UDP *:domain
proftpd   62010  proftpd    1u  IPv4 31306260      0t0  TCP *:ftp (LISTEN)
mysqld    62420    mysql   11u  IPv4 31306903      0t0  TCP *:mysql (LISTEN)
apache2   62582 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   62582 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   62845 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   62845 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)
apache2   64748 www-data    3u  IPv4     5456      0t0  TCP *:www (LISTEN)
apache2   64748 www-data    4u  IPv4     5458      0t0  TCP *:https (LISTEN)

Above lsof command lists all active listening processes port number on UDP and TCP/IP 4 proto with the assigned process PID number (in second column). This is very useful if you find out to have listening service on port number and you cannot figure out what process name exactly is listening.

A classic example, where this is very helpful is if you have a listening process on SMTP port 25 and you cannot identify what kind of mail server is taking up the port? This has happened me many times on Debian Linux based hosts, which by default had priorly installed sendmail and I later removed sendmail to install Postfix or Exim SMTP.
To find out what is assigning port 25, you had to grep the protocol name from all binded host processes, like so:

 

server:~# lsof +M -i4 | grep -i smtp

exim4     17550     root    3u  IPv4 31577966      0t0  TCP localhost:smtp (LISTEN)

Whether you want to get information on Process ID, binding other random port lets say port 10000, following same logic you can grep it:

server:~# lsof +M -i4 |grep -i 10000
miniserv.  2161     root    7u  IPv4     5368      0t0  UDP *:10000

To get rid of a process for which you're unsure what kind of (/etc/init.d/service-name) init script is starting it, you can then use kill cmd to stop it;

server:~# kill -9 2161

Second approach to find out what kind of process is listening on a random port or socket, lets say port num 58515 is by using fuser.

 

 

server:~# netstat -ltn4
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:58515           0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN    
tcp        0      0 77.92.85.71:53          0.0.0.0:*               LISTEN    
tcp        0      0 109.123.106.44:53       0.0.0.0:*               LISTEN    
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN    
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN    
tcp        0      0 0.0.0.0:20000           0.0.0.0:*               LISTEN    

Below netstat cmmand, lists all active listening processes on respective IP address and port for protocol TCPIP v.4.

Hence on to get more information on what process is listening on port 58515?

server:~#  fuser -v 58515/tcp

                     USER        PID ACCESS COMMAND
58515/tcp:           statd      1329 F…. rpc.statd

Once you know what is listening, whether you want to kill it this is also possible directly through fuser;

 

server:~# fuser -vk 58515/tcp

As a close-up, I will say fuser and lsof are two must have software tools on any productive server. It is among the critical applications admin has to install during initial server set-up. fuser and lsof helps me often in my sysadmin work, it was more than once I've used them to identify script-kiddies exploit scanners running as a standard process names, as well secretly listening on weird port number  cracker back-doors.

Hopefully this little article, helps someone learn something new. Plenty is written and will be written and if one takes the time to research he can learn much, much more. I'm sure my tiny article is nothing new under the sun for old-school admins, I still hope it will be of use to novice. I'm looking forward to hear if I'm missing some neat use or some interesting case, when lsof or fuser "saved your ass" 🙂
 

How to find out all programs bandwidth use with (nethogs) top like utility on Linux

Friday, September 30th, 2011

Just run across across a super nice top like, program for system administrators, its called nethogs and is definitely entering my “l337” admin outfit next to tools like iftop, nettop, ettercap, darkstat htop, iotop etc.

nethogs is ultra easy to use, to get immediately in console statistics about running processes UPLOAD and DOWNLOAD bandwidth consumption just run it:

linux:~# nethogs

Nethogs screenshot on Linux Server with Nginx
Nethogs running on Debian GNU/Linux serving static web content with Nginx

If you need to check what program is using what amount of network bandwidth, you will definitely love this tool. Having information of bandwidth consumption is also viewable partially with iftop, however iftop is unable to track the bandwidth consumption to each process using the network thus it seems nethogs is unique at what it does.

Nethogs supports IPv4 and IPv6 as well as supports network traffic over ppp. The tool is available via package repositories for Debian GNU/Lenny 5 and Debian Squeeze 6.

To install Nethogs on CentOS and Fedora distributions, you will have to install it from source. On CentOS 5.7, latest nethogs which as of time of writting this article is 0.8.0 compiles and installs fine with make && make install commands.

In the manner of thoughts of network bandwidth monitoring, another very handy tool to add extra understanding on what kind of traffic is crossing over a Linux server is jnettop
jnettop shows which hosts/ports is taking up the most network traffic.
It is available for install via apt in Debian 5/6).

Here is a screenshot on jnettop in action:

Jnettop check network traffic in console

To install jnettop on latest Fedoras / CentOS / Slackware Linux it has to be download and compiled from source via jnettop’s official wiki page
I’ve tested jnettop install from source on CentOS release 5.7 and it seems to compile just fine using the usual compile commands:

[root@prizebg jnettop-0.13.0]# ./configure
...
[root@prizebg jnettop-0.13.0]# make
...
[root@prizebg jnettop-0.13.0]# make install

If you need to have an idea on the network traffic passing by your Linux server distringuished by tcp/udp/icmp network protocols and services like ssh / ftp / apache, then you will definitely want to take a look at nettop (if of course not familiar with it yet).
Nettop is not provided as a deb package in Debian and Ubuntu, where it is included as rpm for CentOS and presumably Fedora?
Here is a screenshot on nettop network utility in action:

Nettop server traffic division by protocol screenshot
FreeBSD users should be happy to find out that jnettop and nettop are part of the ports tree and the two can be installed straight, however nethogs would not work on FreeBSD, I searched for a utility capable of what Nethogs can, but couldn’t find such.
It seems the only way on FreeBSD to track bandwidth back and from originating process is using a combination of iftop and sockstat utilities. Probably there are other tools which people use to track network traffic to the processes running on a hos and do general network monitoringt, if anyone knows some good tools, please share with me.