Posts Tagged ‘bash script’

Show directory structure bash script on Linux howto – See hierarchical directory tree structure one liner shell script

Friday, February 24th, 2017

show-directory-structure-see-hierarchical-directory-tree-structure-on-linux-with-tree-command-and-with-bash-shell-scripts

If you have Sys Adminned Linux or *Nix OS like, whether for some shell scripting purpose or just for sake of keeping a backup you should have definitely come
into some need to list a tree of a directories content in a hierarchical order.

The most obvious way to do that on Linux is by simply using:

1.  "tree" command (not installed by default on most Linux distributions so in order to have it on Deb / Debian based Linux do:
 

# apt-get install –yes tree


On Fedora / CentOS Redhat Linux (RHEL) etc. install with:

# yum –yes install tree

 

By the way for those that needs tree on FreeBSD / BSD UNIX, tree is also available on that platform you can install it with:
 

pkg_add -vr tree


Then simply check man tree to get idea on how to use it, the easiest way to use the command tree once package is installed is to run tree inside directory of choice, i.e.

 

$ cd /somedir
$ tree -a

.
├── acpi
│   ├── events
│   │   └── powerbtn-acpi-support
│   └── powerbtn-acpi-support.sh
├── adduser.conf
├── adjtime
├── aliases
├── alternatives
│   ├── ABORT.7.gz -> /usr/share/postgresql/9.5/man/man7/ABORT.7.gz
│   ├── aclocal -> /usr/bin/aclocal-1.11
│   ├── aclocal.1.gz -> /usr/share/man/man1/aclocal-1.11.1.gz
│   ├── ALTER_AGGREGATE.7.gz -> /usr/share/postgresql/9.5/man/man7/ALTER_AGGREGATE.7.gz
│   ├── ALTER_COLLATION.7.gz -> /usr/share/postgresql/9.5/man/man7/ALTER_COLLATION.7.gz
│   ├── ALTER_CONVERSION.7.gz -> /usr/share/postgresql/9.5/man/man7/ALTER_CONVERSION.7.gz
 

 

To get a list of only directories with tree use:
 

$ tree -d /

 

  │   ├── bin
│   │   ├── boot
│   │   │   └── grub
│   │   │       └── locale
│   │   ├── disk
│   │   │   ├── Books
│   │   │   │   ├── 200 E-BOOKS
│   │   │   │   │   ├── McGraw-Hill – Windows Server 2003
│   │   │   │   │   ├── Oreilly.Access.Cookbook.2nd.Edition-LiB
│   │   │   │   │   ├── Oreilly.ActionScript.Cookbook.eBook-LiB
│   │   │   │   │   ├── OReilly.ActionScript.The.Definative.Guide.WinAll.Retail-EAT
│   │   │   │   │   ├── Oreilly.Active.Directory.2nd.Edition.eBook-LiB
│   │   │   │   │   ├── Oreilly.Active.Directory.Cookbook.eBook-LiB
│   │   │   │   │   ├── Oreilly.ADO.Dot.NET.Cookbook.eBook-LiB
│   │   │   │   │   ├── Oreilly.Amazon.Hacks.eBook-LiB
│   │   │   │   │   ├── OREILLY.ANT.THE.DEFINITIVE.GUIDE-JGT
│   │   │   │   │   ├── Oreilly.Apache.Cookbook.eBook-LiB
│   │   │   │   │   ├── Oreilly.AppleScript.The.Definitive.Guide.eBook-LiB
│   │   │   │   │   ├── Oreilly.ASP.Dot.NET.In.A.Nutshell.2nd.Edition.eBook-LiB
│   │   │   │   │   ├── OReilly.Better.Faster.Lighter.Java.Jun.2004.eBook-DDU
│   │   │   │   │   ├── Oreilly.BLAST.eBook-LiB
│   │   │   │   │   ├── OReilly.BSD.Hacks.May.2004.eBook-DDU
│   │   │   │   │   ├── Oreilly.Building.Embedded.Linux.Systems.eBook-LiB

If you have a colorful terminal and you like colors for readability the -C option is quite handy

 

$ tree -C /

 

tree-command-linux-hierarchical-structure-directory-tree
 

To list the directory tree with permissions included use tree cmd like so:

 

$ tree -L 2 -p /usr

/usr/
├── [drwxr-xr-x]  bin
│   ├── [-rwxr-xr-x]  [
│   ├── [lrwxrwxrwx]  2to3 -> 2to3-2.6
│   ├── [-rwxr-xr-x]  2to3-2.6
│   ├── [-rwxr-xr-x]  411toppm
│   ├── [-rwxr-xr-x]  7z
│   ├── [-rwxr-xr-x]  7za
│   ├── [-rwxr-xr-x]  a2p
│   ├── [-rwxr-xr-x]  ab
│   ├── [-rwxr-xr-x]  ac
│   ├── [lrwxrwxrwx]  aclocal -> /etc/alternatives/aclocal
│   ├── [-rwxr-xr-x]  aclocal-1.11
│   ├── [-rwxr-xr-x]  acpi


Another truly handy option of tree is to list the directory structure index with included file sizes information

 

$ tree -L 2 -sh /bin

/bin
├── [903K]  bash
├── [147K]  bsd-csh
├── [ 30K]  bunzip2
├── [681K]  busybox
├── [ 30K]  bzcat
├── [   6]  bzcmp -> bzdiff
├── [2.1K]  bzdiff
├── [   6]  bzegrep -> bzgrep
├── [4.8K]  bzexe
├── [   6]  bzfgrep -> bzgrep
├── [3.6K]  bzgrep
├── [ 30K]  bzip2
├── [ 14K]  bzip2recover
├── [   6]  bzless -> bzmore
├── [1.3K]  bzmore
├── [ 51K]  cat
├── [ 59K]  chgrp
├── [ 55K]  chmod
├── [ 63K]  chown
├── [ 10K]  chvt
├── [127K]  cp
├── [134K]  cpio
├── [  21]  csh -> /etc/alternatives/csh
├── [104K]  dash


To list a directory tree of a search pattern, lets say all files with .conf extensions use:
 

$ tree -P *.conf

/etc/ca-certificates.conf [error opening dir]
/etc/dante.conf [error opening dir]
/etc/debconf.conf [error opening dir]
/etc/deluser.conf [error opening dir]
/etc/discover-modprobe.conf [error opening dir]
/etc/fuse.conf [error opening dir]
/etc/gai.conf [error opening dir]
/etc/gpm.conf [error opening dir]
/etc/gssapi_mech.conf [error opening dir]
/etc/hdparm.conf [error opening dir]
/etc/host.conf [error opening dir]
/etc/idmapd.conf [error opening dir]
/etc/inetd.conf [error opening dir]
/etc/insserv.conf [error opening dir]
/etc/irssi.conf [error opening dir]
/etc/kernel-img.conf [error opening dir]
/etc/ld.so.conf [error opening dir]
/etc/libao.conf [error opening dir]
/etc/libaudit.conf [error opening dir]
/etc/logrotate.conf [error opening dir]
/etc/memcached.conf [error opening dir]
/etc/mke2fs.conf [error opening dir]
/etc/mongodb.conf [error opening dir]
/etc/mtools.conf [error opening dir]
/etc/multitail.conf [error opening dir]
/etc/nsswitch.conf [error opening dir]
/etc/ntp.conf [error opening dir]
/etc/ocamlfind.conf [error opening dir]

 

 

tree -I option does exclude all petterns you don't want tree to list

Here are few other tree useful options:

  • tree -u /path/to/file – displays the users owning the files
  • tree -g /path/to/file – display the groups owning the files
  • tree -a /path/to/file – display the hidden files/folders
  • tree -d /path/to/file – display only the directories in the hierarchy


However there might be some cases where you have to support a Linux server or you just have to write a script for a non-root user and you might not have the permissins to install the tree command to make your life confortable. If that's the case then you can still use a couple of command line tools and tricks (assuming you have permissions) to list a log a directory / files and subdirectories tree structure in a hierarchical tree like command order

2. Print a list of all sub-directories and files within a directory tree

To print all directories within any path of choise on a server use
 

$ find /path/ -type d -print

 

To print all files within a root filesystem hierarchically with find command

Another way to do it in a more beautiful output is by using find in conjunction with awk
 

$ find . -type d -print 2>/dev/null|awk '!/\.$/ {for (i=1;i<NF;i++){d=length($i);if ( d < 5  && i != 1 )d=5;printf("%"d"s","|")}print "—"$NF}'  FS='/'

 

|—bashscripts
|          |—not-mine
|          |—various
|          |—output
|          |—examples
|          |—fun
|          |—educational
|          |—backdoor-cgi
|          |—fork-bombs
|          |—tmp
|          |    |—old
|          |—bullshits
|—packages
|       |—ucspi-ssl-0.70.2
|       |               |—package
|       |               |—compile
|       |               |      |—rts-tmp
|       |               |—command
|       |               |—src
|—bin
|—package
|      |—host
|      |    |—superscript.com
|      |    |              |—command
|—mnt
|    |—tmpfs
|    |—disk
|    |—flash_drive
|    |—ramfs
|    |—cdrom

3. Get a list of the directories on filesystem structure with one-liner ls + sed script
 

$ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//–/g' -e 's/^/ /' -e 's/-/|/'

 

 |-bin
 |-boot
 |-dev
 |—net
 |—pts
 |-downloads
 |—autorespond-2.0.5
 |—–debian
 |—deb-packages
 |—–IP-Country-2.28
 |——-bin
 |——-blib
 |———arch
 |———–auto

….
 

4. Print all files within root filesystem and issue any command on each of the files
 

ls -R1 / |    while read l; do case $l in *:) d=${l%:};; "") d=;; *) echo "$d/$l";; esac; done


Above command just prints all the found files with full-path if you want to check the file size or file type you just check echo command with any command you need to execute on each of the listed file

5. Get a list hierarchical directory Linux tree with bash shell scripts: Assuming that the server where you need to have a list of the directory filesystem structure in a tree fashion has bash you could use this little script called tree.sh to do the job or for a full filesystem hierarchical tree like directory structure use fulltree.sh

Remove \r (Carriage Return) from string with standard bash shell / sed / tr / vim or awk – Replace \r hidden messy characters from files

Tuesday, February 10th, 2015

remove_r_carriage_return_from_string_with-standard-bash_shell_sed_tr_or_awk_replace_annoying_hidden_messy_characters_from_files

I've been recently writting this Apache webserver / Tomcat / JBoss / Java decomissioning bash script. Part of the script includes extraction from httpd.conf of DocumentRoot variable configured for Apache host.
I was using following one liner to grep and store DocumentRoot set directory into new variable:

documentroot=$(grep -i documentroot /usr/local/apache/conf/httpd.conf | awk '{ print $2 }' |sed -e 's#"##g');

Above line greps for documentroot prints 2nd column of the matchi (which is the Apache server set docroot and then removes any " chars).

However I faced the issue that parsed string contained in $documentroot variable there was mysteriously containing r – return carriage – this is usually Carriage Return (CR) sent by Mac OS and Apple computers. For those who don't know the End of Line of files in UNIX / Linux OS-es is LF – often abreviated as n – often translated as return new line), while Windows PCs use for EOF CR + LF – known as the infamous  rn. I was running the script from the server which is running SuSE SLES 11 Linux, meaning the CR + LF end of file is standardly used, however it seem someone has editted the httpd.conf earlier with a text editor from Mac OS X (Terminal). Thus I needed a way to remove the r from CR character out of the variable, because otherwise I couldn't use it to properly exec tar to archive the documentroot set directory, cause the documentroot directory was showing unexistent.

Opening the httpd.conf in standard editor didn't show the r at the end of
"directory", e.g. I could see in the file when opened with vim

DocumentRoot "/usr/local/apache/htdocs/site/www"

However obviously the r character was there to visualize it I had to use cat command -v option (–show-nonprinting):

cat -v /usr/local/apache/conf/httpd.conf

DocumentRoot "/usr/local/apache/htdocs/site/wwwr"


1. Remove the r CR with bash

To solve that with bash, I had to use another quick bash parsing that scans through $directory and removes r, here is how:

documentroot=${documentroot%$'r'}

It is also possible to use same example to remove "broken" Windows rn Carriage Returns after file is migrated from Windows to Liunx /  FreeBSD host:

documentroot=${documentroot%$'rn'}

 

2. Remove r Carriage Return character with sed

Other way to do remove (del) Windows / Mac OS Carriage Returns in case if Migrating to UNIX is with sed (stream editor).

sed -i s/r// filename >> filename_out.txt


3. Remove r CR character with tr

There is a third way also to do it with (tr) – translate or delete characters old shool *nix command:

tr -d 'r' < file_with_carriagereturns > file_without_carriage_returns

 

4. Remove r CRs with awk (pattern scanning and processing language)

 awk 'sub("$", "r")' inputf_with_crs.txt > outputf_without_crs.txt


5. Delete r CR with VIM editor

:%s/r//g


6. Converting  file DOS / UNIX OSes with dos2unix and unix2dos command line tools

For sysadmins who don't want to bother with writting code to convert CR when moving files between Windows and UNIX hosts there are dos2unix and unix2dos installable commands.

All done Cheers ! 🙂

How to determine WordPress blogs with most spam on multiple blog hosting server

Thursday, November 27th, 2014

determine_find_blogs_with_most_spam-on-multiple-wordpress-blogs-hosting-server-stop-and-clea-large-amounts-ofrcomment-spam
If you're a hosting company that is hosts Joomla / WordPress / ModX websites (each) on separate servers and thus you end up with servers hosting multiple WordPress customer Blogs only, lets say (100+ WP blogs per host) soon your MySQL blogs databases will be full (overfilled) with spam comments. Blogs with multitude of spam comments reduces the WordPress site attractiveness, takes useless disk space, makes wp databases hard to backup and slowing drastically the SQL server.

As our duty as system administrators is to keep the servers optimized (improve performance) and prevent spam-bots to hammer your Linux servers, its is always a good idea to keep an eye on which hosted blogs attract more spammers and cause server overheads and bad hardware optimization.

WordPress blogs keeps logged comments under database_name.wp_comments  (table) thus the quickest way to find out blogs with largest comments tables is to use Linux's find command and print out only comments tables larger than set size.

Here is how:

find /var/lib/mysql/ -type f -size +1024k -name "*_comments.MYD" -exec ls -lh {} ; | awk '{ print $9 ": " $5 }'


/var/lib/mysql/funny-blog/wp_comments.MYD: 15,7M
/var/lib/mysql/wordblogger/wp_comments.MYD: 5,3M
/var/lib/mysql/loveblog/wp_comments.MYD: 50,5M

A comments database of 1MB means about at least 500+ comments, thus the blog loveblog's wp_comments.MYD = 50,5 Mbs contains probably about 10000! comments and should be definitely checked in a browser, if its overfilled with spam because of bad anti-spam policy or missing currently best wordpress spam catcher plugin Akismet. In cases of lack of client to protect his spam you can write quickly a script to auto mail him and ask him kindly to check / fix his blog spam.
In some cases it is useful to write a few liners bash script to automatically disable users with extraordinary blog spam comments databases (quickest way to do it is to move users blog data under quarantine directory and adding a Blog Suspended static html webpage with text like "Please contact support for more info".

1024k find arguments is 1MB, on a big hosted blogs this might be low and you might want to use (100Mb) = 102400kbytes.
You should note that *_comments.MYD in above find cmd is because though standardly wordpress sets wp_ as a prefix to its created skele table structures it is not always the case. 

In above command example find looks for spam comments in /var/lib/mysql (because this is a Debian Linux server), however on other MySQL custom installs, it might be in another dir i.e. /usr/local/mysql/data etc.

It is useful to set the wp_comments statistics output to execute at least once a day as a cronjob:

crontab -u root -e 00 24 * * * /usr/sbin/check_spammed_blogs.sh

vim /usr/check_soammed_blogs.sh

Set a script like:

#!/bin/sh
find /var/lib/mysql/ -type f -size +1024k -name "*_comments.MYD" -exec ls -lh {} ; | awk '{ print $9 ": " $5 }' | tee -a /var/log/blogs_with_most_spam_comments.log

Though above commands is to run on GNU / Linux, for Windows servers based hosting you can  install GNUWin tools and adapt above cmd using windows standard commands or PowerShell to do the same.
Finally you can might want to use some other SQL script to clear blogs with enormously large tables from spam or clear all unapproved spam comments

Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

Saturday, March 24th, 2012

auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

server~:# uptime
13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
load average: 0.09, 0.08, 0.08

Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

#!/bin/sh
# script to check server for extremely high load and restart Apache if the condition is matched
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
# define max load avarage when script is triggered
max_load='25'
# log file
high_load_log='/var/log/apache_high_load_restart.log';
# location of inidex.php to overwrite with temporary message
index_php_loc='/home/site/www/index.php';
# location to Apache init script
apache_init='/etc/init.d/apache2';
#
site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
if [ $check -gt "$max_load" ]; then>
#25 is load average on 5 minutes
cp -rpf $index_php_loc $index_php_loc.bak_ap
echo "$site_maintenance_msg" > $index_php_loc
sleep 15;
if [ $check -gt "$max_load" ]; then
$apache_init stop
sleep 5;
$apache_init restart
echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
cp -rpf $index_php_loc.bak_ap $index_php_loc
fi
fi

The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

# restart Apache if load is higher than 25
*/5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?