Posts Tagged ‘instance’

Shutdown tomcat server node in case of memory depletion – Avoiding Tomcat Out of memory

Friday, June 6th, 2014

fix-avoid-tomcat-out-of-memory-logo

Out Of Memory Errors, or OOMEs, are one of the most common problems faced by Apache Tomcat users. Tomcat cluster behind Apache unreachable (causing customer downtimes). OOME errors occur on production servers that are experiencing an unusually high spike of traffic.

Out of memory errors are usually a problem of application and not of Tomcat server. OMEs have become such a persistent topic of discussion in the Apache Tomcat community cause its so difficult to trace to their root cause. Usually 'incorrect' web app code causing Tomcat to run out of memory is usually technically correct.

Most common reasons for Out of Memory errors in application code are:
 

  •     the heap size being too small
  •     running out of file descriptors
  •     more open threads than the host OS allows
  •     code with high amounts of recursion
  •     code that loads a very large file into memory
  •     code that retaining references to objects or classloaders
  •     a large number of web apps and a small PermGen


The following java option -XX:OnOutOfMemoryError= could be added to any of tomcat java application servers in setenv.sh in  JAVA_OPTS= variable in case of regular Out of Memory errors occur making an application unstable.

-XX:OnOutOfMemoryError=<path_to_tomcat_shutdown_script.sh>

Where < path_to tomcat_shutdown_script.sh > is shutdown script(which performs kill <tomcat_pid> if normal shutdown fails) for the tomcat instance.

With this setup if any tomcat instance run out of memory it will be shutdown (shutdown script invoked) – as result the Apache proxy infront of Tomcats should not pass any further requests to this instance and application will visualize / work properly for end customers.

Usually a tomcat_shutdown_script.sh to invoke in case of OOM would initiate a Tomcat server restart something like:

for i in `ps -ef |grep tomcat |grep /my_path_to_my_instance | awk '{print $2}'`
do
kill -9 "$i"
#path and script to start tomcat
done

To prevent blank pages returned to customer because of shutdown_script.sh starting stopping Tomcat you can set in Reverse Apache Proxy something like:
 

<Proxy balancer://mycluster>
   BalancerMember ajp://10.16.166.48:11010/ route=delivery1 timeout=30 retry=1
   BalancerMember ajp://10.16.166.70:11010/ route=delivery2 timeout=30 retry=1
</Proxy>

Where in above example I assume, there are only two tomcat nodes, for more just add respective ones.

Note that if the deployed application along all servers is having some code making it crash all tomcat nodes can get shutdown all time and you can get in a client havoc 🙂

Trip to Romania ( TTR ) – Few of my impressions from Ro-Mania :)

Thursday, June 21st, 2012

I just came home from a short 4 days trip to Romania. I'm truly impressed by the beauty, the people goodness and their piety.

Over the last few years, I've visited Romania numerous times, most of the times I just crossed by car while traveling to Holland. My impressions the previous years were quite good already but I didn't have the opportunity until now to see how the country looks "from within" – I mean how the life flows there etc.

I've heard quite a lot of BAD thinks about corruption (from european parliament critics in new) and gipsy like people and plenty of bad stuff over the years.
This is all lies, the country people are not corrupt, they're just smart and find plenty of ways to earn money. For instance while I was traveling there were Romanians people selling raspberries brought by the nearby woods, the raspberry selling is not entering the treasury of the country and this is considered by the European Union corruption and a crime 🙂
Well talking about police, they're a bit corrupt and stopping you for no reason sometimes to ask if you can give them some coffee or cigarettes but i wouldn't put this police act as corruption it is rather an act the patrol police do in order to find some extra cash near the low sallaries received for serving in police 🙂

Well anyways my trip to Romania here are few of my travel notes. The whole travel Route was Bulgaria – Romania, the aim of my travel can be classified as business trip, since I was travelling with a relative's husband who was their to transfer some physical goods (a kind of logistics). The wohle trip started from my hometown Dobrich; we went through Silistra and Ruse and then the so famous Danube bridge. The roads in Bulgaria are not perfect and some regions were full of road holes, but this is normal since we have a harsh winter here and now a road recovery works are in progress on the roads. Crossing the Danube bridge, there is a custom police and they charge something like 23 BGN (Bulgarian levs) for crossing it. Then there is a border police check which as most people who travelled by bus or car over borders (out of Schengen) knows pretty well. You have to hand in your ID and it is checked by a policeman and if you're not in the list of WANTED CRIMINALS 🙂 they let you go. Some 500 meters or a 1 kilometer after that is the Romanian border police which (since not part of Schengen yet checks your traveller ID again).

In Romania we travelled through Gurgevo which is one of the closest cities to Danube river bridgeroad and one of nearest romanian cities to Bulgaria.
Our first destination was a place in Romania called Targo Mures, to go their our route passed through the side road of Bucuresht – Bururesti as Domneles (misters) calls it.

As I'm trying to live an Orthodox Christian life the most striking think from the moment we entered Romania to the one we went off it was the number of Churches, Crucifix and religious references one can see by just crossing the country roads.
Many villages in Romania had 2 or 3 Churches a small chapels a Cross on the road sidewalks etc. Even many Romanian houses almost everywhere had a Christian Cross on top of the house or on the house door. Putting a Christian cross on top of your home is something I myself has thought of and wondered why Christians did not put here in Bulgaria and other Christian countries seeing this being a reality in RO-MANIA 🙂 was unexpected.

Other from my impression is that Romanian people work hard and a lot of them live in the village, owning a small land to grow vegetables and animals (sheeps and cows) mostly. While driving we saw on many places shepherds with large herds of sheeps, people mowing and generally many people who seemed to be working hard to grow their own food. This really made sense since here in Bulgaria grievously animals are rarely grown by country people and most people are living in cities (our rural population is very small).
Growing myself vegetables and having sheeps and living a village life is one of my dreams so seeing so many people growing big portion (if not all of their food on their own) especially in this crazy super-market times is really something that gives me hope, we're not completely doomed yet.

Biggest part of Romania is Orthodox Christian, while the deep central part and the western part is Roman Catholic. Nomatter the differences between the two the people in both parts seemed to live in piece. The western part was visibly more developed than the eastern.
In my observations the western part of Romania is highly influenced by Hungarian culture and architecture, while a lot of Hungarians live their today. We went through Sigishuara which seemed to be mostly Roman Catholic though I saw some Orthodox Church too on the city center. Sigishuara is amazingly beautiful. The people we met all throughout the trip was extremely friendly and always wanted to help and threated us like true Christians, something one can rarely see happening nowdays …

Most of Romania we travelled is mountain covered and the air was extremely clean, so when you're even breathing you feel very different and alive if compared to whether in my hometown.
Brasov is one of the big cities in Romania but its construction is not too much concentrated, the city is quite scattered consisting of neap houses up to few stages per hight. Generally in Romania I haven't seen too high buildings almost anywhere, in Bucurest there are some but still they're not so tall as the blocks in Bulgaria or some other ex-communist countries.

Something interesting about Brasov is that near it is Transilvania and a castle in a small village (or town) called Bran, the castle is dating back from the 12 century and is told to be the original castle where Count Dracula lived. Perhaps the region is filled with Vampire stories but unfortunately my knowledge in Romanesco was so poor so I couldn't get into contact to locals to ask for that (neither I had the time to).

Bran Count Dracula Castle

Bran castle was restored a couple of times and has been a host for the royal family of Romania (many Romanian kings, princesses) through the middle ages in Communistic time it was abandoned and just recently it was turned to a cultural museum (probably part of UNESCO).

The castle is located on a high rock and near it is a beatiful garden and an old times mention. In the castle there are plenty of objects left and traditional princess dresses, pictures, a cinema room, an ancient torture room and plenty of other medievel furniture.
The price was normal 25 LEU (the romanian currency), this equals to something like 6,5 EURO.

On the road, while travelling in Romania on many places you see the signs reading <b> DRUM BUN</b>: which in English means <b>"Have you a good trip"</b>.
During our way back from Bran we travelled through another mountain region – Sinaia.
The prices of food in the super-markets in Romania are quite almost same like in Bulgaria and if compared to western europe many products can be considered even cheap. The quality of food I found to be quite satisfying. A lot of people in Romania are still selling home made yellow cheese and natural grown products in villages as well as I've seen plenty of this street old fashioned tradesman which I like so much on the streets. The country roads especially the central ones and highways were quite good too. Well there is a lot to be said but unfortunately I don't have the time to prolonge this post anymore. In the mountain areas there were some sinaguture for monasteries according to what I've heard from some priests here in Bulgaria in Romania currently they have 7000 Orthodox Christian MONKS! ! ! The number is amazing high just for a comparison in Bulgaria as of time of writting we have not more than 300 monks.

Our trip thanksfully was safe and every romanian we met or ask anything was more than welcome to us and tried his best to help us. Finally it was time and we hit the road back to Bulgaria through Danube riverbridge – the bridge road is a bit better than before some holes are filled in but still there are plenty of holes.
We had to pass through the Romanian Customs and pay a fee for passing by and later were checked by Bulgarian border police – thanksfully with no problems. We had to deliver some cargo to Karnobat in Bulgaria so we passed by there and then through Burgas headed back through Stara Planina (Old Mountain) which is amazingly beautiful mountain and is a must visit place for any keen on mountain tourism.

Thanks God I came back home alive and well and here I am writting this post. To sum it up if I have to grade America and being asked if it is worthy as a tourist destination I would say not only worthy but it is a real pearl you must see!
 

How to disable PC Spaker on Debian and Ubuntu Linux

Sunday, May 13th, 2012

 

How to disable pc-speaker on Linux / PC-Speaker Old Desktop Computer picture

A PC Speaker is helpful as it could be used as a tool for diagnosing system hardware failures (different systems produce different beep sequences depending on the machine BIOS type).
Using the instructions for the respective BIOS vendor and version one could determine the type of problem experienced by a machine based on the sequence and frequency of sounds produced by the SPEAKER.
Lets say a hardware component on a server is down with no need for a monitor or screen to be attached you can say precisely if it is the hard drive, memory or fan malfunctioning…

Generally speaking historically embedded PC Speaker was inseparatable part of the Personal Computers, preceding the soundblasters, now this is changing but for compitability sake many comp equipment vendors still produce machines with pc-speaker in.
Some newer machines (mostly laptops) are factory produced with no PC-SPEAKER component anymore.
For those who don't know what is PC SPEAKER, it is a hardware device capable of emitting very simple short beep sounds at certain system occasions.

Talking about PC-Speaker, it reminds me of the old computer days, where we used pc-speakers to play music in DOS quite frequently.
It was wide practice across my friends and myself to use the pc-speaker to play Axel Folly and other mod files because we couldn't afford to pay 150$ for a sound cards. Playing a song over pc-speaker is quite a nice thing and it will be a nice thing if someone writes a program to be able to play songs on Linux via the pc-speaker for the sake of experiment.

As of time of writting, I don't know of any application capable of playing music files via the pc-speaker if one knows of something like this please, drop me a comment..

As long as it is used for hardware failure diagnosis the speaker is useful, however there are too many occasions where its just creating useless annoying sounds.
For instance whether one uses a GUI terminal or console typing commands and hits multiple times backspace to delete a mistyped command. The result is just irritating beeps, which could be quite disturbing for other people in the room (for example if you use Linux as Desktop in heterogeneous OS office).
When this "unplanned" glitchering beeps are experienced 100+ times a day you really want to break the computer, as well as your collegues are starting to get mad (if not using their headphones) 🙂

Hence you need sometimes to turn off the pc-speaker to save some nerves.

Here is how this is done on major Linux distros.

On Debian and most other distros, the PC SPEAKER is controlled by a kernel module, so to disable communication with the speaker you have to remove the kernel module.

On Debian and Fedora disabling pcspeaker is done with:

# modprobe -r pcspkr

Then to permanently disable load of the pcspkr module on system boot:

debian:~# echo 'blacklist pcspkr' >> /etc/modprobe.d/blacklist.conf

On Ubuntu to disable load on boot /etc/modprobe.d/blacklist, file should be used:

ubuntu:~# echo 'blacklist pcspkr' >> /etc/modprobe.d/blacklist
Well that's all folks …

How to convert FLV to AVI and AVI to FLV Videos on Linux and BSD with avidemux and ffmpeg – Simple video editting with LiVES

Tuesday, May 8th, 2012

I'm starting to learn some video editing, as I need it sometimes for building client websites.
As a Linux user I needed to have some kind of software for amateur video editing.
For Microsoft Windows OS, there are tons of video editor programs both free and proprietary (paid).
Windows users can for instance use the free software program VirtualDub (licensed under GPL license) to easily cut movie scenes from a video.

Unfortunately VirtualDub didn't have a Linux or BSD version so in my case I had to look for another soft.

VirtualDub running on Microsoft Windows XP Screenshot (Biomassa)

I consulted a friend of mine who recommended a video editor program called LiVES.

If you haven't done any video editing previously on Linux (like my case was), you will certainly be happy to try LiVES

Debian GNU / Linux LiVES video editor logo bootscreen shot

LiVES can extract only sound from videos, cut selected parts (frames) from videos and do plenty of other nice stuff. It is just great piece of software for anyone, who needs to do simply (newbie) video editting.

With LiVES even an amateur video editor like me could, immediately learn how to chop a movie scenes

Screenshot opened video for editting with LiVES Linux movie editor Debian Squeeze Linux shot

To master the basics and edit one video in FLV format it took me about 1 hour of time, as in the beginning it was confusing to get confortable with the program scenes selector.

One downside of LiVES it failure to open a FLV file I wanted to edit.
In order to be able to edit the flv movie hence I first had to convert the FLV to AVI or MPEG, as this two (video multimedia formats) are supported by LiVES video editor.

After completing my video scenes chopping to the AVI file I had to convert back to FLV.

In order to complete the convertion between FLV to AVI format on my Debian Linux, I used a program called avidemux

Avidemux has a nice GUI interface and also like Lives has support for video editting, though I have never succesfully done any video edits with it.

Avidemux IMHO is user (completely intuitive). To convert the FLV to AVI, all I had to do was simply open the file FLV file, press (CTRL+S) select my FLV video file format and select the output file extension format to be AVI.

Further on, used LiVES to cut my desired parts from my video of choice. Once the cuts were complete I saved the new cutted version of video to AVI.
Then I needed the video again in FLV to upload it in Joomla, so used ffmpegcommand line tool to do the AVI to FLV file converstion, like so:

hipo@noah:~$ /usr/bin/ffmpeg -i my_media_file.avi my_video_file.flv

Hope this article helps someone aiming to do basic video editting on Linux with LiVES and just like needed FLV to AVI and AVI to FLV convertions.

How to make screenshot in /dev/tty console on GNU / Linux – Taking picture JPEG / PNG snapshot of text console in systems without graphical environment

Monday, April 30th, 2012

I'm used to making picture screenshots in GNOME desktop environment. As I've said in my prior posts, I'm starting to return to my old habits of using console ttys for regular daily jobs in order to increase my work efficiency. In that manner of thoughts sometimes I need to take a screenshot of what I'm seeing in my physical (TTY consoles) to be able to later reuse this. I did some experimenting and this is how this article got born.

In this post, I will shortly explain how a picture of a command running in console or terminal in GNU / Linux can be made

Before proceeding to the core of the article, I will say few words on ttys as I believe they might be helpful someone.
The abbreviation of tty comes after TeleTYpewritter phrase and is dating back somewhere near the 1960s. The TTY was invented to help people with impaired eyesight or hearing to use a telephone like typing interface.

In Unix / Linux / BSD ttys are the physical consoles, where one logs in (typing in his user/password). There are physical ttys and virtual vtys in today *nixes. Today ttys, are used everywhere in a modern Unixes or Unix like operating system with or without graphical environments.
Various Linux distributions have different number of physical consoles (TTYs) (terminals connected to standard output) and this depends mostly on the distro major contributors, developers or surrounding OS community philosophy.
Most modern Linux distributions have at least 5 to 7 physical ttys. Some Linux distributions like Debian for instance as of time of writting this, had 7 active by default physical consoles.
Adding 3 more ttys in Debian / Ubuntu Linux is done by adding the following lines in /etc/inittab:
 

7:23:respawn:/sbin/getty 38400 tty7
8:23:respawn:/sbin/getty 38400 tty8
9:23:respawn:/sbin/getty 38400 tty9

On some Linux distributions like Fedora version 9 and newer ones, new ttys can no longer be added via /etc/inittab,as the RedHat guys changed it for some weird reason, but I guess this is too broad issue to discuss ….

In graphical environments ttys are called methaphorically "virtual". For instance in gnome-terminal or while connecting to a remote SSH server, a common tty naming would be /dev/pts/8 etc.

tty command in Linux and BSDs can be used to learn which tty, one is operating in.

Here is output from my tty command, issued on 3rd TTY (ALT+F3) on my notebook:
 

noah:~# tty
/dev/tty3

A tty cmd output from mlterm GUI terminal is like so:
 

hipo@noah:~$ tty/dev/pts/9

Now as mentioned few basic things on ttys I will proceed further to explain how I managed to:

a) Take screenshot of a plain text tty screen into .txt file format
b) take a (picture) JPG / PNG screenshot of my Linux TTY consoles content

1. Take screenshot of plain text tty screen into a plain (ASCII) .txt file:

To take a screenshot of tty1, tty2 and tty3 text consoles in a txt plain text format, cat + a standard UNIX redirect is all necessery:
 

noah:~# cat /dev/vcs1 > /home/hipo/tty1_text_screenshot.txt
noah:~# cat /dev/vcs2 > /home/hipo/tty2_text_screenshot.txt
noah:~# cat /dev/vcs3 > /home/hipo/tty3_text_screenshot.txt

This will dump the text content of the console into the respective files, if however you try to dump an ncurses library like text interactive interfaces you will end up with a bunch of unreadable mess.
In order to read the produced text 'shots' onwards less command can be used …
 

noah:~# less /home/hipo/tty1_text_screenshot.txt
noah:~# less /home/hipo/tty2_text_screenshot.txt
noah:~# less /home/hipo/tty3_text_screenshot.txt

2. Take picture JPG / PNG snapshot of Linux TTY console content

To take a screenshot of my notebook tty consoles I had to first install a "third party program" snapscreenshot . There is no deb / rpm package available as of time of writting this post for the 4 major desktop linux distributions Ubuntu, Debian, Fedora and Slackware.
Hence to install snapscreenshot,I had to manually download the latest program tar ball source and compile e.g.:
 

noah:~# cd /usr/local/src
noah:/usr/local/src# wget -q http://bisqwit.iki.fi/src/arch/snapscreenshot-1.0.14.3.tar.bz2
noah:/usr/local/src# tar -jxvvvf snapscreenshot-1.0.14.3.tar.bz2

noah:/usr/local/src# cd snapscreenshot-1.0.14.3
noah:/usr/local/src/snapscreenshot-1.0.14# ./configure && make && make install
Configuring…
Fine. Done. make.
make: Nothing to be done for `all'.
if [ ! "/usr/local/bin" = "" ]; then mkdir –parents /usr/local/bin 2>/dev/null; mkdir /usr/local/bin 2>/dev/null; \
for s in snapscreenshot ""; do if [ ! "$s" = "" ]; then \
install -c -s -o bin -g bin -m 755 "$s" /usr/local/bin/"$s";fi;\
done; \
fi; \
if [ ! "/usr/local/man" = "" ]; then mkdir –parents /usr/local/man 2>/dev/null; mkdir /usr/local/man 2>/dev/null; \
for s in snapscreenshot.1 ""; do if [ ! "$s" = "" ]; then \
install -m 644 "$s" /usr/local/man/man"`echo "$s"|sed 's/.*\.//'`"/"$s";fi;\
done; \
fi

By default snapscreenshot command is made to take screenshot in a tga image format, this format is readable by most picture viewing programs available today, however it is not too common and not so standartized for the web as the JPEG and PNG.
Therefore to make the text console tty snapshot taken in PNG or JPEG one needs to use ImageMagick's convert tool. The convert example is also shown in snapscreenshot manual page Example section.

To take a .png image format screenshot of lets say Midnight Commander interactive console file manager running in console tty1, I used the command:
 

noah:/home/hipo# snapscreenshot -c1 -x1 > ~/console-screenshot.tga && convert ~/console-screenshot.tga console-screenshot.png

Linux text console tty mc screenshot with snapscreenshot terminal / console snapshotting program

Note that you need to have read/write permissions to the /dev/vcs* otherwise the snapscreenshot will be unable to read the tty and produce an error:
 

hipo@noah:~/Desktop$ snapscreenshot -c2 -x1 > snap.tga && convert snap.tga snap.pngGeometry will be: 1x2Reading font…/dev/console: Permission denied

To take simultaneous picture screenshot of everything contained in all text consoles, ranging from tty1 to tty5, issue:
 

noah:/home/hipo# snapscreenshot -c5 -x1 > ~/console-screenshot.tga && convert ~/console-screenshot.tga console-screenshot.png

Here is a resized 480×320 pixels version of the original screenshot the command produces:

All text Consoles tty1 to tty5 merged screenshot png image with snapscreenshot taken on Debian GNU / Linux

Storing a picture shot of the text (console) screen in JPEG (JPG) format is done analogously just the convert command output extension has to be changed to jpeg i.e.:
 

noah:/home/hipo# snapscreenshot -c5 -x1 > ~/console-screenshot.tga && convert ~/console-screenshot.tga console-screenshot.jpeg

I've also written a tiny wrapper shell script, to facilitate myself picture picture taking as I didn't like to type each time I want to take a screenshot of a tty the above long line.

Here is the wrapper script I wrote:
 

#!/bin/sh
### Config
# .tga produced file name
output_f_name='console-screenshot.tga';
# gets current date
cur_date=$(date +%d_%m_%Y|sed -e 's/^ *//');
# png output f name
png_f_name="console-screenshot-$cur_date.png";
### END Config
snapscreenshot -c$arg1 -x1 > $output_f_name && convert $output_f_name $png_f_name;
echo "Output png screenshot from tty1 console produced in";
echo "$PWD/$png_f_name";
/bin/rm -f $output_f_name;

You can also download my console-screenshot.sh snapscreenshot wrapper script here

The script is quite simplistic to use, it takes just one argument which is the number of the tty you would like to screenshot.
To use my script download it in /usr/local/bin and set it executable flag:
 

noah:~# cd /usr/local/bin
noah:/usr/local/bin# wget -q https://www.pc-freak.net/~bshscr/console-screenshot.sh
noah:/usr/local/bin# chmod +x console-screenshot.sh

Onwards to use the script to snapshot console terminal (tty1) type:
 

noan:~# console-screenshot.sh

I've made also mirror of latest version of snapscreenshot-1.0.14.3.tar.bz2 here just in case this nice little program disappears from the net in future times.

 

How to copy / clone installed packages from one Debian server to another

Friday, April 13th, 2012

1. Dump all installed server packages from Debian Linux server1

First it is necessery to dump a list of all installed packages on the server from which the intalled deb packages 'selection' will be replicated.

debian-server1:~# dpkg --get-selections \* > packages.txt

The format of the produced packages.txt file will have only two columns, in column1 there will be the package (name) installed and in column 2, the status of the package e.g.: install or deinstall

Note that you can only use the –get-selections as root superuser, trying to run it with non-privileged user I got:

hipo@server1:~$ dpkg --set-selections > packages.txt
dpkg: operation requires read/write access to dpkg status area

2. Copy packages.txt file containing the installed deb packages from server1 to server2

There is many way to copy the packages.txt package description file, one can use ftp, sftp, scp, rsync … lftp or even copy it via wget if placed in some Apache directory on server1.

A quick and convenient way to copy the file from Debian server1 to server2 is with scp as it can also be used easily for an automated script to do the packages.txt file copying (if for instance you have to implement package cloning on multiple Debian Linux servers).

root@debian-server1:~# scp ./packages.txt hipo@server-hostname2:~/packages.txt
The authenticity of host '83.170.97.153 (83.170.97.153)' can't be established. RSA key fingerprint is 38:da:2a:79:ad:38:5b:64:9e:8b:b4:81:09:cd:94:d4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '83.170.97.153' (RSA) to the list of known hosts. hipo@83.170.97.153's password:
packages.txt

As this is the first time I make connection to server2 from server1, I'm prompted to accept the host RSA unique fingerprint.

3. Install the copied selection from server1 on server2 with apt-get or dselect

debian-server2:/home/hipo# apt-get update
...
debian-server2:/home/hipo# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
debian-server2:/home/hipo# dpkg --set-selections < packages.txt
debian-server2:/home/hipo# apt-get -u dselect-upgrade --yes

The first apt-get update command assures the server will have the latest version of the packages currently installed, this will save you from running an outdated versions of the installed packages on debian-server2

Bear in mind that using apt-get sometimes, might create dependency issues. This is depending on the exact package names, being replicated in between the servers

Therefore it is better to use another approach with bash for loop to "replicate" installed packages between two servers, like so:

debian-server2:/home/hipo# for i in $(cat packages.txt |awk '{ print $1 }'); do aptitude install $i; done

If you want to automate the questioning about aptitude operations pass on the -y

debian-server2:/home/hipo# for i in $(cat packages.txt |awk '{ print $1 }'); do aptitude -y install $i; done

Be cautious if the -y is passed as sometimes some packages might be removed from the server to resolve dependency issues, if you need this packages you will have to again install them manually.

4. Mirroring package selection from server1 to server2 using one liner

A quick one liner, that does replicate a set of preselected packages from server1 to server2 is also possible with either a combination of apt, ssh, awk and dpkg or with ssh + dpkg + dselect :

a) One-liner code with apt-get unifying the installed packages between 2 or more servers

debian-server2:~# apt-get --yes install `ssh root@debian-server1 "dpkg -l | grep -E ^ii" | awk '{print $2}'`
...

If it is necessery to install on more than just debian-server2, copy paste the above code to all servers you want to have identical installed packages as with debian-server1 or use a shor for loop to run the commands for each and every host of multiple servers group.

In some cases it might be better to use dselect instead as in some situations using apt-get might not correctly solve the package dependencies, if encountering problems with dependencies better run:

debian-server2:/home/hipo# ssh root@debian-server1 'dpkg --get-selections' | dpkg --set-selections && dselect install

As you can see using this second dselect installed "package" mirroring is also way easier to read and understand than the prior "cryptic" method with apt-get, hence I personally think using dselect method is a better.

Well that's basically it. If you need to synchronize also configurations, either an rsync/scp shell script, should be used with all defined server1 config files or in case if a cloning of packages between identical server machines is necessery dd or some other tool like Norton Ghost could be used.
Hope this helps, someone.

Resolving “nf_conntrack: table full, dropping packet.” flood message in dmesg Linux kernel log

Wednesday, March 28th, 2012

nf_conntrack_table_full_dropping_packet
On many busy servers, you might encounter in /var/log/syslog or dmesg kernel log messages like

nf_conntrack: table full, dropping packet

to appear repeatingly:

[1737157.057528] nf_conntrack: table full, dropping packet.
[1737157.160357] nf_conntrack: table full, dropping packet.
[1737157.260534] nf_conntrack: table full, dropping packet.
[1737157.361837] nf_conntrack: table full, dropping packet.
[1737157.462305] nf_conntrack: table full, dropping packet.
[1737157.564270] nf_conntrack: table full, dropping packet.
[1737157.666836] nf_conntrack: table full, dropping packet.
[1737157.767348] nf_conntrack: table full, dropping packet.
[1737157.868338] nf_conntrack: table full, dropping packet.
[1737157.969828] nf_conntrack: table full, dropping packet.
[1737157.969928] nf_conntrack: table full, dropping packet
[1737157.989828] nf_conntrack: table full, dropping packet
[1737162.214084] __ratelimit: 83 callbacks suppressed

There are two type of servers, I've encountered this message on:

1. Xen OpenVZ / VPS (Virtual Private Servers)
2. ISPs – Internet Providers with heavy traffic NAT network routers
 

I. What is the meaning of nf_conntrack: table full dropping packet error message

In short, this message is received because the nf_conntrack kernel maximum number assigned value gets reached.
The common reason for that is a heavy traffic passing by the server or very often a DoS or DDoS (Distributed Denial of Service) attack. Sometimes encountering the err is a result of a bad server planning (incorrect data about expected traffic load by a company/companeis) or simply a sys admin error…

– Checking the current maximum nf_conntrack value assigned on host:

linux:~# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
65536

– Alternative way to check the current kernel values for nf_conntrack is through:

linux:~# /sbin/sysctl -a|grep -i nf_conntrack_max
error: permission denied on key 'net.ipv4.route.flush'
net.netfilter.nf_conntrack_max = 65536
error: permission denied on key 'net.ipv6.route.flush'
net.nf_conntrack_max = 65536

– Check the current sysctl nf_conntrack active connections

To check present connection tracking opened on a system:

:

linux:~# /sbin/sysctl net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 12742

The shown connections are assigned dynamicly on each new succesful TCP / IP NAT-ted connection. Btw, on a systems that work normally without the dmesg log being flooded with the message, the output of lsmod is:

linux:~# /sbin/lsmod | egrep 'ip_tables|conntrack'
ip_tables 9899 1 iptable_filter
x_tables 14175 1 ip_tables

On servers which are encountering nf_conntrack: table full, dropping packet error, you can see, when issuing lsmod, extra modules related to nf_conntrack are shown as loaded:

linux:~# /sbin/lsmod | egrep 'ip_tables|conntrack'
nf_conntrack_ipv4 10346 3 iptable_nat,nf_nat
nf_conntrack 60975 4 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4 1073 1 nf_conntrack_ipv4
ip_tables 9899 2 iptable_nat,iptable_filter
x_tables 14175 3 ipt_MASQUERADE,iptable_nat,ip_tables

 

II. Remove completely nf_conntrack support if it is not really necessery

It is a good practice to limit or try to omit completely use of any iptables NAT rules to prevent yourself from ending with flooding your kernel log with the messages and respectively stop your system from dropping connections.

Another option is to completely remove any modules related to nf_conntrack, iptables_nat and nf_nat.
To remove nf_conntrack support from the Linux kernel, if for instance the system is not used for Network Address Translation use:

/sbin/rmmod iptable_nat
/sbin/rmmod ipt_MASQUERADE
/sbin/rmmod rmmod nf_nat
/sbin/rmmod rmmod nf_conntrack_ipv4
/sbin/rmmod nf_conntrack
/sbin/rmmod nf_defrag_ipv4

Once the modules are removed, be sure to not use iptables -t nat .. rules. Even attempt to list, if there are any NAT related rules with iptables -t nat -L -n will force the kernel to load the nf_conntrack modules again.

Btw nf_conntrack: table full, dropping packet. message is observable across all GNU / Linux distributions, so this is not some kind of local distribution bug or Linux kernel (distro) customization.
 

III. Fixing the nf_conntrack … dropping packets error

– One temporary, fix if you need to keep your iptables NAT rules is:

linux:~# sysctl -w net.netfilter.nf_conntrack_max=131072

I say temporary, because raising the nf_conntrack_max doesn't guarantee, things will get smoothly from now on.
However on many not so heavily traffic loaded servers just raising the net.netfilter.nf_conntrack_max=131072 to a high enough value will be enough to resolve the hassle.

– Increasing the size of nf_conntrack hash-table

The Hash table hashsize value, which stores lists of conntrack-entries should be increased propertionally, whenever net.netfilter.nf_conntrack_max is raised.

linux:~# echo 32768 > /sys/module/nf_conntrack/parameters/hashsize
The rule to calculate the right value to set is:
hashsize = nf_conntrack_max / 4

– To permanently store the made changes ;a) put into /etc/sysctl.conf:

linux:~# echo 'net.netfilter.nf_conntrack_count = 131072' >> /etc/sysctl.conf
linux:~# /sbin/sysct -p

b) put in /etc/rc.local (before the exit 0 line):

echo 32768 > /sys/module/nf_conntrack/parameters/hashsize

Note: Be careful with this variable, according to my experience raising it to too high value (especially on XEN patched kernels) could freeze the system.
Also raising the value to a too high number can freeze a regular Linux server running on old hardware.

– For the diagnosis of nf_conntrack stuff there is ;

/proc/sys/net/netfilter kernel memory stored directory. There you can find some values dynamically stored which gives info concerning nf_conntrack operations in "real time":

linux:~# cd /proc/sys/net/netfilter
linux:/proc/sys/net/netfilter# ls -al nf_log/

total 0
dr-xr-xr-x 0 root root 0 Mar 23 23:02 ./
dr-xr-xr-x 0 root root 0 Mar 23 23:02 ../
-rw-r--r-- 1 root root 0 Mar 23 23:02 0
-rw-r--r-- 1 root root 0 Mar 23 23:02 1
-rw-r--r-- 1 root root 0 Mar 23 23:02 10
-rw-r--r-- 1 root root 0 Mar 23 23:02 11
-rw-r--r-- 1 root root 0 Mar 23 23:02 12
-rw-r--r-- 1 root root 0 Mar 23 23:02 2
-rw-r--r-- 1 root root 0 Mar 23 23:02 3
-rw-r--r-- 1 root root 0 Mar 23 23:02 4
-rw-r--r-- 1 root root 0 Mar 23 23:02 5
-rw-r--r-- 1 root root 0 Mar 23 23:02 6
-rw-r--r-- 1 root root 0 Mar 23 23:02 7
-rw-r--r-- 1 root root 0 Mar 23 23:02 8
-rw-r--r-- 1 root root 0 Mar 23 23:02 9

 

IV. Decreasing other nf_conntrack NAT time-out values to prevent server against DoS attacks

Generally, the default value for nf_conntrack_* time-outs are (unnecessery) large.
Therefore, for large flows of traffic even if you increase nf_conntrack_max, still shorty you can get a nf_conntrack overflow table resulting in dropping server connections. To make this not happen, check and decrease the other nf_conntrack timeout connection tracking values:

linux:~# sysctl -a | grep conntrack | grep timeout
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_events_retry_timeout = 15
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent2 = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30

All the timeouts are in seconds. net.netfilter.nf_conntrack_generic_timeout as you see is quite high – 600 secs = (10 minutes).
This kind of value means any NAT-ted connection not responding can stay hanging for 10 minutes!

The value net.netfilter.nf_conntrack_tcp_timeout_established = 432000 is quite high too (5 days!)
If this values, are not lowered the server will be an easy target for anyone who would like to flood it with excessive connections, once this happens the server will quick reach even the raised up value for net.nf_conntrack_max and the initial connection dropping will re-occur again …

With all said, to prevent the server from malicious users, situated behind the NAT plaguing you with Denial of Service attacks:

Lower net.ipv4.netfilter.ip_conntrack_generic_timeout to 60 – 120 seconds and net.ipv4.netfilter.ip_conntrack_tcp_timeout_established to stmh. like 54000

linux:~# sysctl -w net.ipv4.netfilter.ip_conntrack_generic_timeout = 120
linux:~# sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 54000

This timeout should work fine on the router without creating interruptions for regular NAT users. After changing the values and monitoring for at least few days make the changes permanent by adding them to /etc/sysctl.conf

linux:~# echo 'net.ipv4.netfilter.ip_conntrack_generic_timeout = 120' >> /etc/sysctl.conf
linux:~# echo 'net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 54000' >> /etc/sysctl.conf

Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

Saturday, March 24th, 2012

auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

server~:# uptime
13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
load average: 0.09, 0.08, 0.08

Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

#!/bin/sh
# script to check server for extremely high load and restart Apache if the condition is matched
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
# define max load avarage when script is triggered
max_load='25'
# log file
high_load_log='/var/log/apache_high_load_restart.log';
# location of inidex.php to overwrite with temporary message
index_php_loc='/home/site/www/index.php';
# location to Apache init script
apache_init='/etc/init.d/apache2';
#
site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
if [ $check -gt "$max_load" ]; then>
#25 is load average on 5 minutes
cp -rpf $index_php_loc $index_php_loc.bak_ap
echo "$site_maintenance_msg" > $index_php_loc
sleep 15;
if [ $check -gt "$max_load" ]; then
$apache_init stop
sleep 5;
$apache_init restart
echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
cp -rpf $index_php_loc.bak_ap $index_php_loc
fi
fi

The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

# restart Apache if load is higher than 25
*/5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?

What is the facebook recommended profile picture (logo) size for better SEO Marketing / Few basic Facebook marketing tips

Wednesday, February 29th, 2012

Facebook marketing Likes good recommended logo sizes, Facebook profile logo

I hate facebook, from the deepness of my guts! However, unfortunately in one of the companies I'm employed, occasionally I have to use it as an advertisement media to improve the Search Engine visibility for their websites (gather them some more likes). As a not big lover of facebook, I'm also not a facebook-pro, anyways with this circumstances, I'm starting to learn basic tips on, making a facebook page / profile more user friendly and hence more attracting to facebook users.

From what, i"ve read there are plenty of factors that can affect on how attractive a facebook page / company profile , anyways some of the factors are more important to tamper than others. Such two factors of top importance are:
 

Let me evaluate a bit on each of the three aforementioned fb marketing weight factors.

1. Using Page likes for popularization.

– Lets say that you would like to make marketing to a company websites which is involved in Financial or Investment market.
Enter with the company created page or profile and facebook and think for and search for as many keywords related to the company business as possible.
Check each of the results if the Page / Group is liked by hundred thousands of other people, then just give it a like too.
By doing so for a couple of seconds all this (hundred thousands or millions) of people who has also liked it will see for a second (or less), what hangs on your Facebook profile 😉
The chance someone gets interested into what is your profile saying here is high especially if this pops up to a some 100 000+ of logged people who previously liked a fb page 🙂

2. Joining groups as a way to drive more visitors to a facebook profile

Joining big groups consisting ot thousands or millions's logic is the same like with the page likes. The only difference is some groups are kept private the so called (Closed Group).
Finding a number of similar (Open Groups) to your company activities and joining them will possibly display your company facebook profile to as many profiles as the group has.
Most of the groups are not too active therefore joing groups as a way to drive attention of facebook users to a desired profile is not so efficient as with Page Likes Adjusting a profile picture logo dimensions to a wide skyscraper.

It might seem strange but actually the size of profile picture set to pages or profiles in facebook matters, different profiles dimensions could have impact on Page Likes 🙂
The reason for the fb picture profile having influence on the marketing is very simple. If you have a wider picture you can graphically include more data visible for the user that is always stuck to the page and hence seen by the user.

I've realized this after, I've red few articles online on the same topic along with that I've noticed many big brand pages on Facebook like for instance BMW (Cars) with over 8 000 000 of likes, Adidas and Nike – with over 6 000 000 likes, have set up this wide skyscraper dimensions logos.

Here is a crop taken displaying the wide skyscraper profile picture set by BMW cars.The profile picture used by BWM (cars) has an image dimensions of: 180×500 heigh x widthAdidas's profile logo has a size of 180×489.

Nike uses a bit of unstandard 390×720 (but while uploaded facebook website automatically crops the image to a size which is fittable for a wide skyscraper) with dimensions like (200×400)Another franchising company with big likes base, I've seen is McDonald's with 19 900 000+ likes! again with a banner logo consisting of wide scraper with picture dimenions (180×439 – height / width )There are some comparatively big user base facebook pages, like Ubuntu's non-profit organization with 555 000+ thousands of likes, and their logo has dimensions of 180×181 pixels.

Not all of the highly liked (visited) facebook pages however use a wide skyscraper as a profile logo and still has enormous number of likes.
One contra-example showing that there is no clear relation between facebook profile picture dimensions and page popularity (number of likes) is Metallica (Musician/Band) fan page – (http://www.facebook.com/Metallica), which as of the time of writting this article has the UNIQUE HIGH NUMBER of 23,725,897 LIKES! 🙂

As far as, I've further red on many blogs, there are two general facebook standard banner profile picture recommendations:

 

1. One is facebook square like profile picture

– (this is the classics), if you just place a picture that is not too wide automatically facebook upload scripts will tailer the picture to have dimensions like: 180×180 pixels

The other recommended facebook pictures size is the wide wide-scraper and it should possess an image dimensions size similar to:
width x height -180×500 pixels

This is pretty much the few basics I  so far, know of can help to easily gather a custom fb company Page Likes.
As facebook is really HUGE nowdays  fb marketing has advanced tremendously.

I'll be glad to hear some cool tips & tricks you know? which can help a facebook page / profile popularity rise up.
Looking forward to hear for your good or bad facebook experiences 🙂

  • Likes of pages, which are liked by thousands of hundreds or millions of people
  • Belongings to groups / Membering to big groups (consisting of thousands)
    which hold similar interest or business activity
  • Profile picture logo dimensions