Posts Tagged ‘bash shell’

Command to get CPU server load in % percentage using bash and /proc/stat on Linux

Wednesday, March 11th, 2015

Command-to-get-CPU-server-load-in-percentage-using-bash-shell-script-and-linux-proc-stat

Getting load avarage is easy with uptime command, however since nowadays Linux servers are running on multiple CPU machines and Dual cores, returned load avarage shows only information concerning a single processor. Of course seeing overall CPU server load is possible with TOP / TLoad command  / HTOP and a bunch of other monitoring commands, but how you can get a CPU percentage server load using just  /proc/stat and bash scripting? Here is hwo:
 

:;sleep=1;CPU=(`cat /proc/stat | head -n 1`);PREV_TOTAL=0;for VALUE in "${CPU[@]}”; do let “PREV_TOTAL=$PREV_TOTAL+$VALUE”;done;PREV_IDLE=${CPU[4]};sleep $sleep; CPU=(`cat /proc/stat | head -n 1`);unset CPU[0];IDLE=${CPU[4]};TOTAL=0; for VALUE in “${CPU[@]}"; do let "TOTAL=$TOTAL+$VALUE"; done;echo $(echo "scale=2; ((($sleep*1000)*(($TOTAL-$PREV_TOTAL)-($IDLE-$PREV_IDLE))/($TOTAL-$PREV_TOTAL))/10)" | bc -l );

52.45

As you can see command output shows CPU is loaded on 52.45%, so this server will soon have to be replaced with better hardware, because it gets CPU loaded over 50%

It is useful to use above bash shell command one liner together with little for loop to refresh output every few seconds and see how the CPU is loaded in percentage over time.

 

for i in $(seq 0 10); do :;sleep=1;CPU=(`cat /proc/stat | head -n 1`);PREV_TOTAL=0;for VALUE in "${CPU[@]}”; do let “PREV_TOTAL=$PREV_TOTAL+$VALUE”;done;PREV_IDLE=${CPU[4]};sleep $sleep; CPU=(`cat /proc/stat | head -n 1`);unset CPU[0];IDLE=${CPU[4]};TOTAL=0; for VALUE in “${CPU[@]}"; do let "TOTAL=$TOTAL+$VALUE"; done;echo $(echo "scale=2; ((($sleep*1000)*(($TOTAL-$PREV_TOTAL)-($IDLE-$PREV_IDLE))/($TOTAL-$PREV_TOTAL))/10)" | bc -l ); done

47.50

13.86
27.36
82.67
77.18

To monitor "forever" output from all server processor overall load use:
 

while [ 1 ]; do :;sleep=1;CPU=(`cat /proc/stat | head -n 1`);PREV_TOTAL=0;for VALUE in “${CPU[@]}”; do let “PREV_TOTAL=$PREV_TOTAL+$VALUE”;done;PREV_IDLE=${CPU[4]};sleep $sleep; CPU=(`cat /proc/stat | head -n 1`);unset CPU[0];IDLE=${CPU[4]};TOTAL=0; for VALUE in “${CPU[@]}"; do let "TOTAL=$TOTAL+$VALUE"; done;echo $(echo "scale=2; ((($sleep*1000)*(($TOTAL-$PREV_TOTAL)-($IDLE-$PREV_IDLE))/($TOTAL-$PREV_TOTAL))/10)" | bc -l ); done

 

 

Remove \r (Carriage Return) from string with standard bash shell / sed / tr / vim or awk – Replace \r hidden messy characters from files

Tuesday, February 10th, 2015

remove_r_carriage_return_from_string_with-standard-bash_shell_sed_tr_or_awk_replace_annoying_hidden_messy_characters_from_files

I've been recently writting this Apache webserver / Tomcat / JBoss / Java decomissioning bash script. Part of the script includes extraction from httpd.conf of DocumentRoot variable configured for Apache host.
I was using following one liner to grep and store DocumentRoot set directory into new variable:

documentroot=$(grep -i documentroot /usr/local/apache/conf/httpd.conf | awk '{ print $2 }' |sed -e 's#"##g');

Above line greps for documentroot prints 2nd column of the matchi (which is the Apache server set docroot and then removes any " chars).

However I faced the issue that parsed string contained in $documentroot variable there was mysteriously containing r – return carriage – this is usually Carriage Return (CR) sent by Mac OS and Apple computers. For those who don't know the End of Line of files in UNIX / Linux OS-es is LF – often abreviated as n – often translated as return new line), while Windows PCs use for EOF CR + LF – known as the infamous  rn. I was running the script from the server which is running SuSE SLES 11 Linux, meaning the CR + LF end of file is standardly used, however it seem someone has editted the httpd.conf earlier with a text editor from Mac OS X (Terminal). Thus I needed a way to remove the r from CR character out of the variable, because otherwise I couldn't use it to properly exec tar to archive the documentroot set directory, cause the documentroot directory was showing unexistent.

Opening the httpd.conf in standard editor didn't show the r at the end of
"directory", e.g. I could see in the file when opened with vim

DocumentRoot "/usr/local/apache/htdocs/site/www"

However obviously the r character was there to visualize it I had to use cat command -v option (–show-nonprinting):

cat -v /usr/local/apache/conf/httpd.conf

DocumentRoot "/usr/local/apache/htdocs/site/wwwr"


1. Remove the r CR with bash

To solve that with bash, I had to use another quick bash parsing that scans through $directory and removes r, here is how:

documentroot=${documentroot%$'r'}

It is also possible to use same example to remove "broken" Windows rn Carriage Returns after file is migrated from Windows to Liunx /  FreeBSD host:

documentroot=${documentroot%$'rn'}

 

2. Remove r Carriage Return character with sed

Other way to do remove (del) Windows / Mac OS Carriage Returns in case if Migrating to UNIX is with sed (stream editor).

sed -i s/r// filename >> filename_out.txt


3. Remove r CR character with tr

There is a third way also to do it with (tr) – translate or delete characters old shool *nix command:

tr -d 'r' < file_with_carriagereturns > file_without_carriage_returns

 

4. Remove r CRs with awk (pattern scanning and processing language)

 awk 'sub("$", "r")' inputf_with_crs.txt > outputf_without_crs.txt


5. Delete r CR with VIM editor

:%s/r//g


6. Converting  file DOS / UNIX OSes with dos2unix and unix2dos command line tools

For sysadmins who don't want to bother with writting code to convert CR when moving files between Windows and UNIX hosts there are dos2unix and unix2dos installable commands.

All done Cheers ! 🙂

Fixing Shellshock new critical remote bash shell exploitable vulnerability on Debian / Ubuntu / CentOS / RHEL / Fedora / OpenSuSE and Slackware

Friday, October 10th, 2014

Bash-ShellShock-remote-exploitable-Vulnerability-affecting-Linux-Mac-OSX-and-BSD-fixing-shellshock-bash-vulnerability-debian-redhat-fedora-centos-ubuntu-slackware-and-opensuse
If you still haven’t heard about the ShellShock Bash (Bourne Again) shell remote exploit vulnerability and you admin some Linux server, you will definitely have to read seriously about it. ShellShock Bash Vulnerabily has become public on Sept 24 and is described in details here.

The vulnerability allows remote malicious attacker to execute arbitrary code under certain conditions, by passing strings of code following environment variable assignments. Affected are most of bash versions starting with bash 1.14 to bash 4.3.
Even if you have patched there are some reports, there are other bash shell flaws in the way bash handles shell variables, so probably in the coming month there will be even more patches to follow.

Affected bash flaw OS-es are Linux, Mac OS and BSDs;

• Some DHCP clients

• OpenSSL servers that use ForceCommand capability in (Webserver config)

• Apache Webservers that use CGi Scripts through mod_cgi and mod_cgid as well as cgis written in bash or launching bash subshells

• Network exposed services that use bash somehow

Even though there is patch there are futher reports claiming patch ineffective from both Google developers and RedHat devs, they say there are other flaws in how batch handles variables which lead to same remote code execution.

There are a couple of online testing tools already to test whether your website or certain script from a website is vulnerable to bash remote code executions, one of the few online remote bash vulnerability scanner is here and here. Also a good usable resource to test whether your webserver is vulnerable to ShellShock remote attack is found on ShellShocker.Net.

As there are plenty of non-standard custom written scripts probably online and there is not too much publicity about the problem and most admins are lazy the vulnerability will stay unpatched for a really long time and we’re about to see more and more exploit tools circulating in the script kiddies irc botnets.

Fixing bash Shellcode remote vulnerability on Debian 5.0 Lenny.

Follow the article suggesting how to fix the remote exploitable bash following few steps on older unsupported Debian 4.0 / 3.0 (Potato) etc. – here.

Fixing the bash shellcode vulnerability on Debian 6.0 Squeeze. For those who never heard since April 2014, there is a A Debian LTS (Long Term Support) repository. To fix in Debian 6.0 use the LTS package repository, like described in following article.

If you have issues patching your Debian Wheezy 6.0 Linux bash, it might be because you already have a newer installed version of bash and apt-get is refusing to overwrite it with an older version which is provided by Debian LTS repos. The quickest and surest way to fix it is to do literally the following:


vim /etc/apt/sources.list

Paste inside to use the following LTS repositories:

deb http://http.debian.net/debian/ squeeze main contrib non-free
deb-src http://http.debian.net/debian/ squeeze main contrib non-free
deb http://security.debian.org/ squeeze/updates main contrib non-free
deb-src http://security.debian.org/ squeeze/updates main contrib non-free
deb http://http.debian.net/debian squeeze-lts main contrib non-free
deb-src http://http.debian.net/debian squeeze-lts main contrib non-free

Further on to check the available installable deb package versions with apt-get, issue:



apt-cache showpkg bash
...
...
Provides:
4.1-3+deb6u2 -
4.1-3 -
Reverse Provides:

As you see there are two installable versions of bash one from default Debian 6.0 repos 4.1-3 and the second one 4.1-3+deb6u2, another way to check the possible alternative installable versions when more than one version of a package is available is with:



apt-cache policy bash
...
*** 4.1-3+deb6u2 0
500 http://http.debian.net/debian/ squeeze-lts/main amd64 Packages
100 /var/lib/dpkg/status
4.1-3 0
500 http://http.debian.net/debian/ squeeze/main amd64 Packages

Then to install the LTS bash version on Debian 6.0 run:



apt-get install bash=4.1-3+deb6u2

Patching Ubuntu Linux supported version against shellcode bash vulnerability:
A security notice addressing Bash vulnerability in Ubuntus is in Ubuntu Security Notice (USN) here
USNs are a way Ubuntu discloses packages affected by a security issues, thus Ubuntu users should try to keep frequently an eye on Ubuntu Security Notices

apt-get update
apt-get install bash

Patching Bash Shellcode vulnerability on EOL (End of Life) versions of Ubuntu:

mkdir -p /usr/local/src/dist && cd /usr/local/src/dist
wget http://ftpmirror.gnu.org/bash/bash-4.3.tar.gz.sig
wget http://ftpmirror.gnu.org/bash/bash-4.3.tar.gz
wget http://tiswww.case.edu/php/chet/gpgkey.asc
gpg --import gpgkey.asc
gpg --verify bash-4.3.tar.gz.sig
cd ..
tar xzvf dist/bash-4.3.tar.gz
cd bash-4.3
mkdir patches && cd patches
wget -r --no-parent --accept "bash43-*" -nH -nd
ftp.heanet.ie/mirrors/gnu/bash/bash-4.3-patches/ # Use a local mirror
echo *sig | xargs -n 1 gpg --verify --quiet # see note 2

cd ..
echo patches/bash43-0?? | xargs -n 1 patch -p0 -i # see note 3 below

./configure --prefix=/usr --bindir=/bin
--docdir=/usr/share/doc/bash-4.3
--without-bash-malloc
--with-installed-readline

make
make test && make install

To solve bash vuln in recent Slackware Linux:

slackpkg update
slackpkg upgrade bash

For old Slacks, either download a patched version of bash or download the source for current installed package and apply the respective patch for the shellcode vulnerability.
There is also a GitHub project “ShellShock” Proof of Concept code demonstrating – https://github.com/mubix/shellshocker-pocs
There are also non-confirmed speculations for bash vulnerability bug to impact also:

Speculations:(Non-confirmed possibly vulnerable common server services):

• XMPP(ejabberd)

• Mailman

• MySQL

• NFS

• Bind9

• Procmail

• Exim

• Juniper Google Search

• Cisco Gear

• CUPS

• Postfix

• Qmail

Fixing ShellShock bash vulnerability on supported versions of CentOS, Redhat, Fedora

In supported versions of CentOS where EOL has not reached:

yum –y install bash

In Redhat, Fedoras recent releases to patch:

yum update bash

To upgrade the bash vulnerability in OpenSUSE:

zipper patch –cve=CVE-2014-7187

Shellcode is worser vulnerability than recent SSL severe vulnerability Hearbleed. According to Redhat and other sources this new bash vulnerability is already actively exploited in the wild and probably even worms are crawling the net stealing passwords, data and building IRC botnets for remote control and UDP flooding.

Windows Explorer (Open directory in command prompt preserving dir PATH) – Add Dos Prompt Here feature via tiny registry tweak

Friday, January 10th, 2014

Windows explorer dos prompt here open directory in windows command line

If you have to use Windows on system administration level, you had to use command prompt daily, thus its useful to be able to be able to open Command Line starting from desired directory with no need to copy directory Path by hand and CD to it manually.

By default Command Prompt, cmd.exe always opens itself setting a path to user home directory, reading what is defined by win system variable %USERPROFILE% or %HOMEPATH% – MS Windows equivalent of UNIX's $HOME shell variable.

To add open in  DOS Prompt Here Command Prompt option to Windows Explorer menus its necessary to apply few rules to Windows registry DB
Use above Download link and launch it and from there on clicking with right Mouse button to any directory will enable you to open directory in CMD.EXE.

Here is content of little registry tweak adding the new menu Dos Prompt Here button
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\DosHere]
@="Dos &Prompt Here"

[HKEY_CLASSES_ROOT\Directory\shell\DosHere\Command]
@="cmd.exe /k cd %1"

[HKEY_CLASSES_ROOT\Drive\shell\DosHere]
@="Dos &Prompt Here"

[HKEY_CLASSES_ROOT\Drive\shell\DosHere\Command]
@="cmd.exe /k cd %1"

Windows explorer open program files or any specific directory in windows command line

windows opening directory in command line program files screenshot win 7

This little registry code is originally for Windows 2000, anyways it is compatible with all NT technology based Windowses, Add DOS Prompt Here tweak works fine on Windows XP, Windows 7 and Windows 8 (Home, Pro and Business editions).

By Mentioning $HOME its interesting to say Windows equivalent of Linux's as it might be useful to know:

linux:~# echo $HOME
/home/hipo

is

C:\\> echo %USERPROFILE%
C:\\Users\\georgi

To list all Windows Command Prompt environment variable equivalent to Linux's bash shell env / setenv is SET command), here is example output from my Winblows;

C:\Users\georgi> SET
ALLUSERSPROFILE=C:\\ProgramData
APPDATA=C:\\Users\georgi\AppData\Roaming
CLASSPATH=.;C:\\Program Files (x86)\\Java\\jre6\\lib\\ext\\QTJava.zip
CommonProgramFiles=C:\\Program Files\Common Files
CommonProgramFiles(x86)=C:\\Program Files (x86)\\Common Files
CommonProgramW6432=C:\\Program Files\\Common Files
COMPUTERNAME=GEORGI
ComSpec=C:\\Windows\\system32\\cmd.exe
DEFLOGDIR=C:\\ProgramData\\McAfee\\DesktopProtection
FP_NO_HOST_CHECK=NO
HOMEDRIVE=C:
HOMEPATH=\\Users\\georgi
LOCALAPPDATA=C:\\Users\\georgi\\AppData\\Local
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=C:\\Program Files\\RA2HP\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\S
Wbem;C:\\Windows\\System32\\WindowsPowerShell\v1.0\;C:\\Program Files\\WIDCOM
oth Software\\;C:\\Program Files\\WIDCOMM\\Bluetooth Software\\syswow64;C:\\Pr
les (x86)\\Hewlett-Packard\\HP ProtectTools Security Manager\\Bin\\;C:\\Progr
\\ActivIdentity\\ActivClient\\;C:\\Program Files (x86)\\ActivIdentity\\ActivCl
\\Program Files (x86)\\QuickTime\\QTSystem\\
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=3a09
ProgramData=C:\\ProgramData
ProgramFiles=C:\\Program Files
ProgramFiles(x86)=C:\\Program Files (x86)
ProgramW6432=C:\\Program Files
PROMPT=$P$G
PSModulePath=C:\\Windows\\system32\\WindowsPowerShell\\v1.0\Modules\\
PTSMInstallPath_X86=C:\\Program Files (x86)\\Hewlett-Packard\\HP ProtectToo
ity Manager\\
PUBLIC=C:\\Users\\Public
QTJAVA=C:\\Program Files (x86)\\Java\\jre6\\lib\\ext\\QTJava.zip
SESSIONNAME=Console
SystemDrive=C:
SystemRoot=C:\\Windows
TEMP=C:\\Users\\georgi\\AppData\\Local\\Temp
TMP=C:\\Users\\georgi\\AppData\\Local\\Temp
USERDNSDOMAIN=EMEA.CPQCORP.NET
USERDOMAIN=EMEA
USERNAME=georgi
USERPROFILE=C:\\Users\\georgi
VSEDEFLOGDIR=C:\\ProgramData\\McAfee\\DesktopProtection
windir=C:\\Windows
windows_tracing_flags=3
windows_tracing_logfile=C:\\BVTBin\\Tests\\installpackage\\csilogfile.log

Hope this little trick hopes someone out there.
I will be glad to hear of other cool useful windows registry tweaks?

 

Linux bash: Add line numbering to text file with NL – number lines command

Friday, March 22nd, 2013

Lets say you have a plainjt text TXT file with sentences and you would like to put quickly line numbering in a Linux bash shell. First thing that might pop up to your mind is to use a loop cat text file and print it line by one together with output of incremental values counted from 0 to the number of lines the file contains. This would work but is useless work. As coreutils package already contains a tool called nl ( /usr/bin/nl ). Here is paste from header of nl manual page;

 

NAME
       nl – number lines of files

SYNOPSIS
       nl [OPTION]… [FILE]…
 

Thus to put line numbering to lets say this fortune's bible quotes plain text file;

$ nl bible > bible_quotes_numbered.txt

Here is a small chunk of bible_quotes_numbered.txt;

 

$ cat bible_quotes_numbered.txt

 1  In the beginning God created the heaven and the earth.
     2                  — Genesis 1:1
     3  %
     4  And the earth was without form, and void; and darkness was upon
     5  the face of the deep. And the Spirit of God moved upon the face of
     6  the waters.
     7                  — Genesis 1:2
     8  %
     9  And God said, Let there be light: and there was light.
    10                  — Genesis 1:3
    11  %
    12  And God saw the light, that it was good: and God divided the light
    13  from the darkness.
    14                  — Genesis 1:4
    15  %
    16  And God called the light Day, and the darkness he called Night.
    17  And the evening and the morning were the first day.
    18                  — Genesis 1:5
    19  %
    20  And God said, Let there be a firmament in the midst of the waters,
    21  and let it divide the waters from the waters.
    22                  — Genesis 1:6
    23  %
    24  And God made the firmament, and divided the waters which were under
    25  the firmament from the waters which were above the firmament: and
    26  it was so.
    27                  — Genesis 1:7
    28  %
    29  And God called the firmament Heaven. And the evening and the morning
    30  were the second day.
    31                  — Genesis 1:8
    32  %
    33  And God said, Let the waters under the heaven be gathered together
    34  unto one place, and let the dry land appear: and it was so.
    35                  — Genesis 1:9
    36  %
    37  And God called the dry land Earth; and the gathering together of
    38  the waters called he Seas: and God saw that it was good.
    39                  — Genesis 1:10
    40  %
    41  And God said, Let the earth bring forth grass, the herb yielding
    42  seed, and the fruit tree yielding fruit after his kind, whose seed
    43  is in itself, upon the earth: and it was so.
    44                  — Genesis 1:11
 

 

It is worthy mentioning the same opeartion can be achieved with good old /bin/cat with cat -n, e.g.;

$ cat -n bible >bible_quotes_numbered.txt

 

On *BSDs nl command is not available, thus cat -n is very good to know of since it works for numbering ASCII plain text files on Free/Net/OpenBSDs

How to set repository to install binary packages on amd64 FreeBSD 9.1

Friday, January 11th, 2013

Though, it is always good idea to build from source for better performance of Apache + MySQL + PHP, its not worthy the time on installing minor things like; trafshow, tcpdump or deco (MC – midnight commander like native freebsd BSD program).

If you're on a 64 bit version of FreeBSD ( amd64) 9.1 and you try to install a binary package with;

freebsd# pkg_add -vr vim

Ending up with an error;

Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9.1-release/Latest/vim.tbz: File unavailable (e.g., file not found, no access)
pkg_add: unable to fetch 'ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9.1-release/Latest/vim.tbz' by URL
pkg_add: 1 package addition(s) failed

The error is caused by lack of special packages-9.1-release directory existing on FreeBSD.org servers. I've realized this after doing a quick manual check opening ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64. The existing URL containing working fbsd 9.1 binaries is:

ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/
h

You will have to set a repository for FreeBSD 9.1 amd64 packages manually with cmd:
freebsd# echo $SHELL
/bin/csh
freebsd# setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/

If you're on bash shell use export instead:

freebsd# export PACKAGESITE="ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"

To make ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/ as a permanent binary repository:

echo 'setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/' >> /root/.cshrc

or

echo 'export PACKAGESITE="ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9-current/Latest/"' >> /root/.bashrc

Now, pkg_add as much as you like 😉

How to search text strings only in hidden files dot (.) files within a directory on Linux and FreeBSD

Saturday, April 28th, 2012

how-to-search-hidden-files-linux-freebsd-logo_grep
If there is necessity to look for a string in all hidden files with all sub-level subdirectories (be aware this will be time consuming and CPU stressing) use:
 

hipo@noah:~$ grep -rli 'PATH' .*

./.gftp/gftprc
./.gftp/cache/cache.OOqZVP
….

Sometimes its necessery to only grep for variables within the first-level directories (lets say you would like to grep a 'PATH' variable set, string within the $HOME directory, the command is:

hipo@noah:~$ grep PATH .[!.]*

.profile:PATH=/bin:/usr/bin/:${PATH}
.profile:export PATH
.profile:# set PATH so it includes user's private bin if it exists
.profile: PATH="$HOME/bin:$PATH"
.profile.language-env-bak:# set PATH so it includes user's private bin if it exists
.profile.language-env-bak: PATH="$HOME/bin:$PATH"
.viminfo:?/PATH.xcyrillic: XNLSPATH=/usr/X11R6/lib/X11/nls
.xcyrillic: export XNLSPATH

The regular expression .[!.]*, means exclude any file or directory name starting with '..', e.g. match only .* files

Note that to use the grep PATH .[!.]* on FreeBSD you will have to use this regular expression in bash shell, the default BSD csh or tsch shells will not recognize the regular expression, e.g.:

grep PATH '.[!.]*'
grep: .[!.]*: No such file or directory

Hence on BSD, if you need to look up for a string within the home directory, hidden files: .profile .bashrc .bash_profile .cshrc run it under bash shell:

freebsd# /usr/local/bin/bash
[root@freebsd:/home/hipo]# grep PATH .[!.]*

.bash_profile:# set PATH so it includes user's private bin if it exists
.bash_profile:# PATH=~/bin:"${PATH}"
.bash_profile:# do the same with …

Another easier to remember, alternative grep cmd is:

hipo@noah:~$ grep PATH .*
.profile:PATH=/bin:/usr/bin/:${PATH}
.profile:export PATH
.profile:# set PATH so it includes user's private bin if it exists
.profile: PATH="$HOME/bin:$PATH"
….

Note that grep 'string' .* is a bit different in meaning, as it will not prevent grep to match filenames with names ..filename1, ..filename2 etc.
Though grep 'string' .* will work note that it will sometimes output some unwanted matches if filenames with double dot in the beginning of file name are there …
That's all folks 🙂

Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

Saturday, March 24th, 2012

auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

server~:# uptime
13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
load average: 0.09, 0.08, 0.08

Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

#!/bin/sh
# script to check server for extremely high load and restart Apache if the condition is matched
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
# define max load avarage when script is triggered
max_load='25'
# log file
high_load_log='/var/log/apache_high_load_restart.log';
# location of inidex.php to overwrite with temporary message
index_php_loc='/home/site/www/index.php';
# location to Apache init script
apache_init='/etc/init.d/apache2';
#
site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
if [ $check -gt "$max_load" ]; then>
#25 is load average on 5 minutes
cp -rpf $index_php_loc $index_php_loc.bak_ap
echo "$site_maintenance_msg" > $index_php_loc
sleep 15;
if [ $check -gt "$max_load" ]; then
$apache_init stop
sleep 5;
$apache_init restart
echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
cp -rpf $index_php_loc.bak_ap $index_php_loc
fi
fi

The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

# restart Apache if load is higher than 25
*/5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?

Convert single PDF pages to multiple SVG files on Debian Linux with pdf2svg

Sunday, February 26th, 2012

In my last article, I've explained How to create PNG, JPG, GIF pictures from one single PDF document
Convertion of PDF to images is useful, however as PNG and JPEG graphic formats are raster graphics the image quality gets crappy if the picture is zoomed to lets say 300%.
This means convertion to PNG / GIF etc. is not a good practice especially if image quality is targetted.

I myself am not a quality freak but it was interesting to find out if it is possible to convert the PDF pages to SVG (Scalable Vector Graphics) graphics format.

Converting PDF to SVG is very easy as for GNU / Linux there is a command line tool called pdf2svg
pdf2svg's official page is here

The traditional source way compile and install is described on the homepage. For Debian users pdf2svg has already existing a deb package.

To install pdf2svg on Debian use:

debian:~# apt-get install --yes pdf2svg
...

Once installed usage of pdf2svg to convert PDF to multiple SVG files is analogous to imagemagick's convert .
To convert the 44 pages Projects.pdf to multiple SVG pages – (each PDF page to a separate SVG file) issue:

debian:~/project-pdf-to-images$ for i in $(seq 1 44); do \
pdf2svg Projects.pdf Projects-$i.SVG $i; \
done

This little loop tells each page number from the 44 PDF document to be stored in separate SVG vector graphics file:

debian:~/project-pdf-to-images$ ls -1 *.svg|wc -l
44

For BSD users and in particular FreeBSD ones png2svg has a bsd port in:

/usr/ports/graphics/pdf2svg

Installing on BSD is possible directly via the port and convertion of PDF to SVG on FreeBSD, should be working in the same manner. The only requirement is that bash shell is used for the above little bash loop, as by default FreeBSD runs the csh. 
On FreeBSD launch /usr/local/bin/bash, before following the Linux instructions if you're not already in bash.

Now the output SVG files are perfect for editting with Inkscape or Scribus and the picture quality is way superior to old rasterized (JPEG, PNG) images