Posts Tagged ‘total’

How much memory users uses in GNU / Linux and FreeBSD – Commands and Scripts to find user memory usage on Linux

Tuesday, February 17th, 2015

 

how-much-memory-users-use-in-gnu-linux-freebsd-command-to-find-and-show-ascending-descending-usage-of-system-memory-tux-memory-logo

 


If you have to administrate a heterogenous network with Linux and FreeBSD or other UNIX like OSes you should sooner or later need for scripting purposes to have a way to list how much memory separate users take up on your system. Listing memory usage per user is very helpful for admins who manager free-shells or for companies where you have developers, developing software directly on the server via ssh. Being able to check which process eats up most memory is essential for every UNIX / Linux sysadmin, because often we as admins setup (daemons) on servers and we forgot about their existence, just to remember they exist 2 years later and see the server is crashing because of memory exhaustion. Tracking server bottlenecks where RAM memory and Swapping is the bottleneck is among the main swiss amry knives of admins. Checking which user occupies all server memory is among the routine tasks we're forced to do as admins, but because nowdays servers have a lot of memory and we put on servers often much more memory than ever will be used many admins forget to routinely track users / daemons memory consumption or even many probably doesn't know how.  Probably all are aware of the easiest wy to get list of all users memory in console non interactively with free command, e.g.:
 

free -m
             total       used       free     shared    buffers     cached
Mem:         32236      26226       6010          0        983       8430
-/+ buffers/cache:      16812      15424
Swap:        62959        234      62725

 

but unfortunately free command only shows overall situation with memory and doesn't divide memory usage by user

Thus probably to track memory users the only known way for most pepole is to (interactively) use good old top command or if you like modern (colorful) visualization with htop:

debian:~# top

 

linux-check_memory_usage_by_logged-in-user-with-top-process-command-gnu-linux-freebsd-screenshot

Once top runs interactive press 'm' to get ordered list of processes which occupy most system memory on Linux server.Top process use status statistics will refresh by default every '3.0' seconds to change that behavior to '1' second press  s and type '1.0'. To get Sort by Memory Use in htop also press 'm'
 

[root@mail-server ~]# htop


htop_show_users_memory_usage_order_ascending-gnu-linux-screenshot

 

However if you need to be involved in scripting and setting as a cron job tasks to be performed in case if high memroy consumption by a service you will need to use few lines of code. Below are few examples on how Linux user memory usage can be shown with ps cmd.

Probably the most universal way to see memory usage by users on Debian / Ubuntu / CentOS / RHEL and BSDs (FreeBSD / NetBSD) is with below one liner:

 

server:~# ps hax -o rss,user | awk '{a[$2]+=$1;}END{for(i in a)print i” “int(a[i]/1024+0.5);}' | sort -rnk2
daemon 0
debian-tor 63
dnscache 1
dnslog 0
hipo 21
messagebus 1
mysql 268
ntp 2
privoxy 1
proftpd 1
qmaill 0
qmailq 0
qmailr 0
qmails 0
qscand 291
root 94
shellinabox 1
snmp 1
statd 1
vpopmail 80
www-data 6765

 

Output is in MBs

Below is output from machine where this blog is running, the system runs ( Apache + PHP + MySQL Webserver + Qmail Mail server and Tor) on Debian GNU / Linux.

 To get more human readable (but obscure to type – useful for scripting) output list of which user takes how much memory use on deb / rpm etc. based Linux :

 

server:~# echo "USER                 RSS      PROCS" ; echo "——————– ——– —–" ; \
ps hax -o rss,user | awk '{rss[$2]+=$1;procs[$2]+=1;}END{for(user in rss) printf “%-20s %8.0f %5.0f\n”, user, rss[user]/1024, procs[user];}' | sort -rnk2

 

USER                 RSS      PROCS
——————– ——– —–
www-data                 6918   100
qscand                    291     2
mysql                     273     1
root                       95   120
vpopmail                   81     4
debian-tor                 63     1
hipo                       21    15
ntp                         2     1
statd                       1     1
snmp                        1     1
shellinabox                 1     2
proftpd                     1     1
privoxy                     1     1
messagebus                  1     1
dnscache                    1     1
qmails                      0     2
qmailr                      0     1
qmailq                      0     2
qmaill                      0     4
dnslog                      0     1
daemon                      0     2

 

It is possible to get the list of memory usage listed in percentage proportion, with a tiny for bash loop and some awk + process list command
 

TOTAL=$(free | awk '/Mem:/ { print $2 }')
for USER in $(ps haux | awk '{print $1}' | sort -u)
do
    ps hux -U $USER | awk -v user=$USER -v total=$TOTAL '{ sum += $6 } END { printf "%s %.2f\n", user, sum / total * 100; }'
done

107 1.34
115 2.10
119 1.34
daemon 1.32
dnscache 1.34
dnslog 1.32
hipo 1.59
mysql 4.79
ntp 1.34
privoxy 1.33
proftpd 1.32
qmaill 1.33
qmailq 1.33
qmailr 1.32
qmails 1.33
qscand 4.98
root 1.33
snmp 1.33
statd 1.33
vpopmail 2.35
www-data 86.48

Also a raw script which can be easily extended to give you some custom information on memory use by user list_memory_use_by_user.sh is here.
You can also want to debug further how much memory a certain users (lets say user mysql and my username hipo) is allocating, this can easily be achieved ps like so:
 

root@pcfreak:~# ps -o size,pid,user,command -u mysql –sort -size
 SIZE   PID USER     COMMAND
796924 14857 mysql   /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plugin-dir=/usr/lib/mysql/plugin –user=mysql –pid-file=/var/run/mysqld/mysqld.pid –socket=/var/run/mysqld/mysqld.sock –port=3306

 

root@pcfreak~# ps -o size,pid,user,command -u hipo –sort -size|less
 SIZE   PID USER     COMMAND
13408 19063 hipo     irssi
 3168 19020 hipo     SCREEN
 2940  2490 hipo     -bash
 1844 19021 hipo     /bin/bash
 1844 19028 hipo     /bin/bash
 1844 19035 hipo     /bin/bash
 1844 19042 hipo     /bin/bash
 1844 19491 hipo     /bin/bash
 1844 22952 hipo     /bin/bash
  744  2487 hipo     sshd: hipo@pts/0
  744  2516 hipo     sshd: hipo@notty
  524  2519 hipo     screen -r
  412  2518 hipo     /usr/lib/openssh/sftp-server

You see from below output user running with www-data (this is Apache Webserver user in Debian) is eating 86.48% of overall system memory and MySQL server user is using only 4.79% of available memory

Output is shown in Megabytes per username memory usage, and user memory usage is ordered (stepping-down / descentive) from top to bottom

Getting more thoroughful and easier to read reporting without beeing a 31337 bash coder you can install and use on Linux smem – memory reporting tool .

SMEM can provide you with following memory info:

  • system overview listing
  • listings by process, mapping, user
  • filtering by process, mapping, or user
  • configurable columns from multiple data sources
  • configurable output units and percentages
  • configurable headers and totals
  • reading live data from /proc
  • reading data snapshots from directory mirrors or compressed tarballs
  • lightweight capture tool for embedded systems
  • built-in chart generation


Installing smem on Debian 6 / 7 / Ubuntu 14.04 / Turnkey Linux etc. servers is done with standard:

 

debian:~# apt-get install –yes smem
….

 

 

To install smem on CentOS 6 / 7:

 

[root@centos ~ ]# yum -y install smem
….


On Slackware and other Linux-es where smem is not available as a package you can install it easily from binary archive with:

 

cd /tmp/
wget http://www.selenic.com/smem/download/smem-1.3.tar.gz
tar xvf smem-1.3.tar.gz
sudo cp /tmp/smem-1.3/smem /usr/local/bin/
sudo chmod +x /usr/local/bin/smem

 


Two most common smem uses are:

 

root@mail:~# smem -u
User     Count     Swap      USS      PSS      RSS
dnslog       1       44       48       54      148
qmaill       4      232      124      145      464
hipo        11    13552     8596     9171    13160
qscand       2     4500   295336   295602   297508
root       188   217312  4521080  4568699  7712776

 

Below command shows (-u – Report memory usage by user, -t – show totals, -k – show unix suffixes)

root@mail:~# smem -u -t -k
User     Count     Swap      USS      PSS      RSS
dnslog       1    44.0K    48.0K    54.0K   148.0K
qmaill       4   232.0K   124.0K   145.0K   464.0K
hipo        11    13.2M     8.4M     9.0M    12.9M
qscand       2     4.4M   288.4M   288.7M   290.5M
root       188   212.2M     4.3G     4.4G     7.4G
—————————————————
           206   230.1M     4.6G     4.6G     7.7G


To get users memory use by percentage with smem:
 

root@mail:~# smem -u -p
User     Count     Swap      USS      PSS      RSS
dnslog       1    0.00%    0.00%    0.00%    0.00%
qmaill       4    0.00%    0.00%    0.00%    0.01%
hipo        11    0.17%    0.11%    0.11%    0.16%
qscand       2    0.05%    3.63%    3.63%    3.66%
root       194    2.64%   56.18%   56.77%   95.56%

It is also useful sometimes when you want to debug system overloads caused by external hardware drivers loaded into kernel causing issues to get list of system wide memory use sorted by user

 

 root@mail:~# smem -w -p
Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        38.30%     36.01%      2.28%
userspace memory             60.50%      0.98%     59.53%
free memory                   1.20%      1.20%      0.00%


smem is very nice as if you're running it on a Desktop Linux system with Xserver installed you can see also graphical output of memory use by application:
 

root@desktop-pc:~# smem –bar pid -c "pss uss"


smem_graphical_representation-of-which-user-application-is-consuming-most-memory-gnu-linux-kde-screenshot-smem-command-line-tool

smem can even generate graphical pie charts to visualize better memory use
 

root@desktop-pc:~# smem -P '^k' –pie=name

 

generate-graphical-staticstics-linux-memory-use-by-pie-chart

If there is a high percentage shown in firmware/hardware this means some buggy module is loaded in kernel eating up memory, to fix it debug further and remove the problematic module.
userspace memory actually shows the percantage of memory out of all server available RAM that is being consumed by applications (non kernel and other system processes which make the system move). You see in above example the kernel itself is consuming about 40% of system overall available memory. 

We all know the SWAP field stands for hard disk drive used as a memory when system is out, but there are 3 fields which smem will report which will be probably unclear for most here is also explanation on what USS / PSS / RSS means?

RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. It does include memory from shared libraries as long as the pages from those libraries are actually in memory. It does include all stack and heap memory too.

There is also PSS (proportional set size). This is a newer measure which tracks the shared memory as a proportion used by the current process. So if there were two processes using the same shared library from before.

USS stands for Unique set size, USS is just the unshared page count, i.e. memory returned when process is killed 

PSS = Proportional set size, (PSS),  is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system.  
Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. The parameter PSS instead measures each application’s “fair share” of each shared area to give a realistic measure. For most admins checking out the output from RSS (output) should be enough, it will indicate which user and therefore which daemon is eating up all your memory and will help you to catch problematic services which are cause your server to run out of RAM and start swapping to disk.

Linux: how to show all users crontab – List all cronjobs

Thursday, May 22nd, 2014

linux-unix-list-all-crontab-users-and-scripts
I'm doing another server services decomissioning and part of decomissioning plan is: Removing application and all related scripts from related machines (FTP, RSYNC, …). In project documentation I found a list with Cron enabled shell scripts:

#Cron tab excerpt:
1,11,21,31,41,51 * * * */webservices/tools/scripts/rsync_portal_sync.sh

that has to be deleted, however there was nowhere mentioned under what kind of credentials (with what kind of user) are the cron scripts running? Hence I had to look up all users that has cronjobs and find inside each user's cronjobs whether respective script is set to run. Herein I will explain shortly how I did that.

Cronjobs by default has few locations from where cronjobs are setupped depending on their run time schedule. First place I checked for the scripts is

/etc/crontabs # cat /etc/crontabs SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1
59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly
14 4 * * * root rm -f /var/spool/cron/lastrun/cron.daily
29 4 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly
44 4 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly

I was not really user via what user is shell script run, therefore I looked first if someone doesn't set the script to run via crontab's standard locations for Daily, Hourly,Weekly and Monthly cronjobs:
 

a) Daily set cron jobs are in:

/etc/cron.daily/

b) Hourly set cron jobs:

/etc/cron.hourly

c) Weekly cron jobs are in:

/etc/cron.weekly/

d) Monthly cron jobs:

/etc/cron.monthly

There is also a location read by crontab for all Software (package distribution) specific cronjobs – all run under root user privileges.:

e) Software specific script cron jobs are in:

/etc/cron.d/  
As the system has about 327 users in /etc/passwd, checking each user's cronjob manually with:

# crontab -u UserName -l

was too much time consuming thus it is a good practice to list

/var/spool/cron/*

directory and to see which users has cron jobs defined

 

# ls -al /var/spool/cron/*
-rw——- 1 root root 11 2007-07-09 17:08 /var/spool/cron/deny

/var/spool/cron/lastrun:
total 0
drwxr-xr-x 2 root root 80 2014-05-22 11:15 .
drwx—— 4 root root 120 2008-02-25 15:45 ..
-rw-r–r– 1 root root 0 2014-05-22 04:15 cron.daily

/var/spool/cron/tabs:
total 8
drwx—— 2 root root 72 2014-04-03 03:43 .
drwx—— 4 root root 120 2008-02-25 15:45 ..
-rw——- 1 root root 4901 2014-04-03 03:43 root
 


/var/spool/cron – is crond (/usr/bin/cron/)'s spool directory.

# ls -al /var/spool/cron/tabs/ total 8
drwx------ 2 root root 72 2014-04-03 03:43 .
drwx------ 4 root root 120 2008-02-25 15:45 ..
-rw------- 1 root root 4901 2014-04-03 03:43 root

Above output shows only root superuser has defined crons.

Alternative way to check all user crontabs is via quick Linux one liner shell script show all user cron jobs

for i in $(cat /etc/passwd | sed -e "s#:# #g" | awk '{ print $1 }'); do
echo "user $i --- crontab ---";
crontab -u $i -l 2>&1 >/dev/null;
echo '----------';
done|less

Note that above short script has to run with root user. Enjoy 🙂

What is the real development costs of Debian GNU / Linux – How much costs the development of a Free Software projects

Friday, February 17th, 2012

Free Software (FS) is free as in freedom as well as free as in price. Free and Open Source Software (FOSS) is developed by geek hobbyist which voluntarily put their time and effort in writting, testing and sharing with anyone for free million of lines of programming code. This doesn't mean however the price of free software costs is 0 (zero). Though the "end product" –  Free Software developed is FREE, "real" software costs as with any other product costs huge money.

I've recently read on Jeb's blog an estimation on how much is the cost of one of the major Free Software project efforts – Debian GNU / Linux
According to James E. Brombergerthe whole Debian project was estimated to be at the shocking price of $19 billion – $19 000, 000, 000 !!!

Here is how JEB got the $19 billions, a quote taken from his blog:

"By using David A Wheeler’s sloccount tool and average wage of a developer of US$72,533 (using median estimates from Salary.com and PayScale.com for 2011) I summed the individual results to find a total of 419,776,604 source lines of code for the ‘pristine’ upstream sources, in 31 programming languages — including 429 lines of Cobol and 1933 lines of Modula3!

In my analysis the projected cost of producing Debian Wheezy in February 2012 is US$19,070,177,727 (AU$17.7B, EUR€14.4B, GBP£12.11B), making each package’s upstream source code worth an average of US$1,112,547.56 (AU$837K) to produce. Impressively, this is all free (of cost).

James has done incredible job with this great research and he deserves applause.
However I believe the numbers proposed by his research are slightly different if we speak about realistic cost of Debian GNU / Linux.
The real costs of the working software ready to install on a user PC are way higher, as according to Jeb's research only the software cost based on code line count is considered.

Hence James software estimation calculates only the programming costs and miss many, many factors that constitute the software end cost.
Some of the many, many REAL COST / expenses for developing a huge Free Software project like Debian GNU / Linux to be considered are:
 

a) bandwidth costs for hosting free software (on the server side)b) bandwidth cost for developers or FS users downloading the software

a) Time spend to spread the word of the great added value of Debian and bundled software (Mouth by Mouth Marketing)

b) Time spend to advertise Debian and its free software components on blogs, social networks (identi.ca, facebook, twitter) etc.(Voluntary online Marketing, SEO etc.)

c) Time spend on generating ideas on future program versions and reporting them to Debian FS community

d) Time on evaluation and feedback on software

e) Time spend on managing free software repository (download) servers voluntarily (by system administrators)

f) Time spend by users on Bug Tracking & Bug Reporting

g) Time spend on research and self-actualization by software developer)

h) Time spend on software Quality Assurance

This are most of the multiple factors which should probably influence the cost of any non-free (proprietary software) project. No matter this costs apply for non-free software, it perfectly applies for free software as well.With all said if if we assume the non-programming costs are equal to the programming costs of $ 19 000 000 000 (suggested by Jeb). This means the real cost of Debian will presumably be at least $32 000 000 000. Putting $ 19 billion for all this long list of "additional" costs (besides pure source) factors is probably still very under-scored number.
 

  • the developers use of their own computers (hardware depreciation)
  • electricity bill of the volunteer (developer) working on the program or project
  • electricity bills for servers where free software is stored and available for download
  • volunteer developer IT skills and tech knowledge (KNOW HOW)
  • Internet, network, dial up bandwidth cost
  • personal time put in FS development (programming, design, creativity etc.)! here the sub costs are long:
  • Costs for Project Management Leaders / Project Coordination
  • The complexity of each of the projects constituting Debian

Very interesting figure from Jeb's research is the Programming Languages break down by source code figure.
Jamesresearch reveals on the 4 major programming languages used in the 17000+ software projects (part of Debian GNU / Linux):

 

  • ANSI C with 168,536,758 – (40% of all projects source code)
  • C++ at 83,187,329 – (20% of all projects source)
  • Java 34,698,990 – (lines of code 8% of sources)
  • Lisp – (7% of all projects source code)

  His research also provides a general idea on how much the source code of some of the major FOSS projects costs. Here is a copy of his figures
 

Individual Projects

Other highlights by project included:

ProjectVersionThousands
of SLOC
Projected cost
at US$72,533/developer/year
Samba3.6.12,000US$101 (AU$93M)
Apache2.2.9693US$33.5M (AU$31M)
MySQL5.5.171,200US$64.2M (AU$59.7M)
Perl5.14.2669US$32.3M (AU$30M)
PHP5.3.9693US$33.5M (AU$31.1M)
Bind9.7.3319US$14.8M (AU$13.8M)
Moodle1.9.9396US$18.6M (AU$17.3M)
Dasher4.11109US$4.8M (AU$4.4M)
DVSwitch0.8.3.66US$250K (AU$232K)

 

As you can imagine all the source evaluation results, are highly biased and are open for discussion, since evaluating a free software project/s is a hard not to say impossible task. The "open" model of development makes a project very hard to track, open source model implies too many unexpected variables missing from the equation for clear calculation on costs. What is sure however if turned in money it is very expensive to produce.  At present moment Debian Project is sponsored only through donations. The usual yearly budget 5 years ago for Debian  was only $80 000 dollars a year!! You can check Debian Project annual reports throughout the years here , for year 2012 Debian Project budget is as low as $ 222, 677 (US Dollars)! The output price of the software the project provides is enormous high if compared to the low project expenses!

For us the free software users, price is not a concern, Debian is absolutely free both  as in freedom and free as in beer 😉