Posts Tagged ‘condition’

Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

Saturday, March 24th, 2012

auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

server~:# uptime
13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
load average: 0.09, 0.08, 0.08

Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

#!/bin/sh
# script to check server for extremely high load and restart Apache if the condition is matched
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
# define max load avarage when script is triggered
max_load='25'
# log file
high_load_log='/var/log/apache_high_load_restart.log';
# location of inidex.php to overwrite with temporary message
index_php_loc='/home/site/www/index.php';
# location to Apache init script
apache_init='/etc/init.d/apache2';
#
site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
if [ $check -gt "$max_load" ]; then>
#25 is load average on 5 minutes
cp -rpf $index_php_loc $index_php_loc.bak_ap
echo "$site_maintenance_msg" > $index_php_loc
sleep 15;
if [ $check -gt "$max_load" ]; then
$apache_init stop
sleep 5;
$apache_init restart
echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
cp -rpf $index_php_loc.bak_ap $index_php_loc
fi
fi

The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

# restart Apache if load is higher than 25
*/5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?

How to check MASTER / SLAVE MySQL nodes status – Check MySQL Replication Status

Thursday, April 19th, 2012

I'm doing replication for one server. Its not the first time I do configure replication between two MySQL database nodes, however since I haven't done it for a few years, my "know how" has mostly vanished so I had some troubles in setting it up. Once I followed some steps to configure replication I had to check if the two MASTER / Slave MySQL db nodes communicate properly. Hence I decided to drop a short post on that just in case if someone has to do the same or if I myself forget how I did it so I can check later on:

1. Check if MASTER MySQL server node is configured properly

The standard way to check a MySQL master node status info is with:
 

mysql> show master status;
+——————+———-+———————————————————+——————+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+——————+———-+———————————————————+——————+
| mysql-bin.000007 | 106 | database1,database2,database3 | |
+——————+———-+———————————————————+——————+
1 row in set (0.00 sec)

By putting \G some extra status info is provided:
 

mysql> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.000007
Position: 106
Binlog_Do_DB: database1,database2,database3
Binlog_Ignore_DB:
1 row in set (0.00 sec)

ERROR:
No query specified

2. Check if Slave MySQL node is configured properly

To check status of the slave the cmd is:
 

mysql> show slave status;

The command returns an output like:
 

mysql> show slave status;+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+| Slave_IO_State | Master_Host | Master_User | Master_Port | Connect_Retry | Master_Log_File | Read_Master_Log_Pos | Relay_Log_File | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error |+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+| Waiting for master to send event | HOST_NAME.COM | slave_user | 3306 | 10 | mysql-bin.000007 | 106 | mysqld-relay-bin.000002 | 251 | mysql-bin.000007 | Yes | Yes | database1,database2,database3 | | | | | | 0 | | 0 | 106 | 407 | None | | 0 | No | | | | | | 0 | No | 0 | | 0 | |+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+

As you can see the output is not too readable, as there are too many columns and data to be displayed and this doesn't fit neither a text console nor a graphical terminal emulator.

To get more readable (more verbose) status for the SQL SLAVE, its better to use command:
 

mysql> show slave status\G;

Here is a sample returned output:
 

mysql> show slave status\G;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: HOST_NAME.COM Master_User: slave_user Master_Port: 3306 Connect_Retry: 10 Master_Log_File: mysql-bin.000007 Read_Master_Log_Pos: 106 Relay_Log_File: mysqld-relay-bin.000002 Relay_Log_Pos: 251 Relay_Master_Log_File: mysql-bin.000007 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: database1,database2,database3 Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 106 Relay_Log_Space: 407 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)ERROR: No query specified

If show master status or shwo slave status commands didn't reveal replication issue, one needs to stare at the mysql log for more info.

Tricks of the supermarket trade or how supermarket lie to you to buy more than you need (My few memories on government food stores in communism times)

Friday, June 22nd, 2012

I've personally always hated super markets from the very early days I entered one. Here in Bulgaria we've been somehow partly lucky not to have supermarket due to the different philosophy of the communistic regime in which we lived until 1991. The philosophy of then governing communist back then was to distribute food in local middle size or small stores owned by the government. The food has to be approved by governmental body and heavily checked if it matches the governmental set standards. The variety of food we were offered in the stores was very little. Most of the small stores (which were owned by the government) only contained basic products like;;;

bread, youghurt, milk, meat, butter, lemonade, beer and few more,

This was done probably intentionally and was a result probably of our long years Orthodox Christian faith which (has saved us and preserved over the centuries). Orthodox Christianity has always teached for simplicity. Even though the communist party rejected the faith and even did severe persecution against the Bulgarian Orthodox Church. The people on the top elite who were planning the communism had been a descent of an orthodox christian raised up living and therefore the philosophy of simplicity was inherited (even genetically) so the way the COM party behaved based on its leaders and the governing decisions about the nations belonging to communism (Russia, (Yugoslavia) – Serbia Koso , Bulgaria, Greece, Egypt etc.) were in a material expression in conjunction with the Christian tradition of simplicity (i say material because Orthodox Christian Church even visiting back then in the countries was usually prohibited and the Com. party did their best to ruin up the believe in Christ in people).

In communism the local suburb grocery stores, were the only "points" of distribution of food. Since buying and owning a car took at least few years of waiting and big money for it. Not more than 10 to 20% of families owned car and owning a car was considered to be granted mostly to people belonging to the then governing communistic elite or the Communist Party.

The public transportation was encouraged because according to communism so called "geometry" which was a main drive for how the systems in communistic countries should work it was estimated public transportation is cheaper and more environmental friendly than if a car is given to anyone, also public transportation (trains, bus-es) etc. did a good thing in the aim of the party to exersice higher control and surveilance over the population.

Because of that existence of big super-market was scarce and the culture to go and buy from a shop with your own family car was not existing.

The equivalent of nowdays mall shops (big retail stores) was usually located on the city center and the size of this shop was much smaller and the variety of products one can find there was only few. Hence the customer didn't have so much options to choose between, neither had to spend too much of time on choosing, the com party wanted the people to work more and buy less, so spending time in stores was not encouraged.

With the entrance of democracy and the moving up of large super-market food and good retails shops like Metro was among the first ones which entered the "freed" market. It was really a bizarre experience for us the ex U.S.S.R belonging people to enter such a large sized stores and to face such a big variety of choice.

Before that in communism we never saw such stores even on the TV, since the TV was governed by communistic party and only information which was useful for the establishment and protection of the regime was projected on the only available National TV channel.

Sorry to distract a bit from the major topic of this post, but I thought this is necessery to explain before continuing because it is important to understand that unintentinonally the communist gave us an advantage to have a different view on things than Western free world, a view which is in many things contrary and more correct (in terms of hard logic) than to Western Europe.

My first encounter with a super-market was only about 8 or 10 years ago and I believe many people who lived in the province of Bulgaria and other ex-communist countries did for a first time experience super-markets approximate 10y. ago.
Though all looked so nice inside the shop and the shop emploees were so nice I always felt something very cold dead inside this big stores. Ptonsnlu my intuition (spirit) if you want has always pointed me out that there is something very wrong with this super markets.

With the increase of products on the market for us people who lived up to our 8 to 12 years (young age) in communism it was a great shock, since we were raised in a society based on some communist false ideology and suddenly the markets were opened and the products variety to choose between become extremely high.
All previously said to be good and true in communism time was said to be anti-freedom and many people did their best to destroy as much as possible mostly everything somehow connected with previous communistic time.
My homeland Bulgaria as a result of this become a very hard to live place and we were forced to adapt in fast paces and learn things on how we have to live like democrats for a very little short of time.
One severe "damage" most of the growing people in Bulgaria in communism had was the hardness and inability to make choices.
For many people between the age of 25 to 35 it is very hard to make choices even for small things.
This was a direct consequence of a long years (45 years) of communist propaganda, and few generations who lived in a non-freedom respecting regime which conditioned and prepared a specific plan and place for every individual living in the country.

Seeing communism as the ultimate evil and manipulation to make us dependent we didn't know that in the free democratic world the manipulation and conditioning in society was heavy too.

"The Programming" of western society through institutions, stores and order in society was a fact and in a sense the only difference between the Democratic world and communistic world was one of the worlds seemed to have a larger frame in which the individual can work and live, whether the communistic regime seem to be more restrictive.
One main thing used to condition people choices and buying behavior and make them over-consume to make the democratic economy constantly growth was big super-markets. Though they were privately owned and not like in communist country, the only drive behind each shop was PROFIT, PROFIT and agian PROFIT.

Higher profits would mean higher consumptions, higher consumptions means higher production and more work places opened as well as more expenses for working individuals would mean bigger necessity to work more and earn more, because though the sallaries incomes were high in the free world. The individual necessities there were even increasing too.

Since the about 1950s 1960s the super-markets started entering the "free" democrat countries world. As a result the small shops which were prior a main food and beverage distribution source started closing, because they couldn't be a concurrence to the big "brothers" super-markets which were able to buy higher quantities of goods in lesser price and therefore sell many of the goods in much lower prices than the little grocery stores.

The result with the years was a huge shift in people buying behavior from local groce and open markets to buy everything from the big super-markets. The big store chains had to be in a fight for customers all the time, so businesses involved started customer wars and tried their best to "steal" customers from each other. In this financial wars a main thing that most super-market chains and malls did was constant optimization of efficiency of both buying and reselling to the end customer. This is a 3 step business process actually.

1;;; They buy from the commodity producer
2;;; They prepare the food for the store (packaging, cleaning up whatever)
3;;; They put it in a certain way in the store and organize the store in a way to always sell more and more

As a result a huge number of tricks were employed by each and every super-market aiming to deceive the customer to ever buy more
The deception of the client in store is very tricky and smart organized so it cannot really be called a lie but its rather a tricky smartness.

Since the super-markets entered in my country too and the tiny grocery stores are mostly closing unable to compete with the large super-market 'money making machines'. Even I am sometimes forced to buy stuff from super markets. Since here in Bulgaria the culture of purchasing from super-markets is not so strong yet. In order to be able to consume the little grocery stores. Many large super-markets started investing in making the little groceries their retail shops with their brand or (sometimes a new make up brand).
By doing so the little grocery stores loose their authoritarity and independance and become dependant on the big super-market on the type of products they will sell, this obviously means this strategy of the big super-market in the long run will lead the little grocery stores we still have to ruin up. This is obvious because if the small grocery stores start selling the exact same products (brands) of the big super-markets, there will be nothing left to make the little grocery unique. Neither the variety of products choice inside or pricing will be any different from super-market. This will mean the tiny grocer will be no different from the big super-markets in terms of product and prices and therefore. The clients which now are loyal to small grocery stores like me will then have not such big motivation to buy from small super-markets, since everything inside is somehow similar (not to say) the same as with big supermarket the only difference would be the lesser variety which is obviously a good reason why the customer would definitely prefer to buy from the large supermarket …

On the videos you're about to see below, they explain some of the probably thousands of tricks nowdays existing the super-market chains employs to trick us to exponentiolly consume their goods.

Old Tricks of the SuperMarket Trade

As you can see the video is presenting a reality of the tricks which was used in the 1960 and now technology and knowledge in the field has largely increased and since they there are surely many new trick 'developments' which are working for the masses of customers daily.
There is even a whole science centered on buying behaviour already existent called buyology!

Here is a short introduction to buyology to give you an idea what its purpose is:

Short introduction video on buyology – The Science of buying part 1 of 2

Short introduction video on buyology – The Science of buying part 2 of 2


Seeing all this is very precious information, since being informed one can escape the bad "tricks and traps" pawn in the markets.

All the information about the communistic regime and the 'shopping system' there was mentioned priorly because I wanted to explain a bit of the difference of then and now to point you to my opinion that the Western buying model and expectation to sell more based on the tricks are probably not working very well here in the ex-communistic countries, especially with the older generations and the one like me which somehow grow a small part of their youth in communism.

The reason is we have a different 'social programming' than westerners. Also the different spirituality (The orthodox Christian faith) plays another role. Also there is a difference in the type of buying behaviour. In Bulgaria there is no culture to buy once or twice per week. People have culture to buy daily, though this is starting to change slowly these days.

For a bit more of videos and a bit of my thougths on my research on Super Market Psychology check my previous post containing educational videos on how the supermarkets cheat you to buy more

The topic is actually very huge and I'm sure what I'm saying is just the 'tip of the iceberg'. As I'm a firm opponent of big industries and large business I'm firmly against the buying from big stores any goods. The reason is that simply by doing so one makes the already RICH people even more RICHER. By increasing the richness of a certain small group of people daily, we as society are un-consciously somehow involuntarily letting them increase the financial control that is already there in some degree to the society.
As democracy's main drive is money this means that by helping the large business-es marge and become even BIGGER, we're doing something against ourselves and our interests as society.
I don't think that any ordinary citizen in the free world want to live in a fully controlled one World State (one world country), similar to the ex-communism I lived through, so I think people concerned about our freedoms should oppose the big businesses according to the society level they're in. Even if we're not in position to change things with money, we as society are in a position to change ourselves and our understanding of world, our desires and our behaviour. If we change our value system in a way, where money are not a top priority and the highest value the outcome will be positive in both spiritual terms and overall world state.

The modern day however wants us to falsely believe that we're divided, helpless and money dependent. This is a big delusion which we're constanty repeated. Just like with the super-market this big super-market company owners were able to convince us through time that super-market is better than the grocery stores, because of the less money spending advantage. Even if in the past there were possibility to spend less via supermarket nowdays spending less by using the supermarket instead of the small groce-store is a BIG LIE. Even if one spends less in super-market on certain products, the amount of related products he is cheated* to buy in store highly exceed the simply money one would spend on even higher priced products in the small groce.
The outcome results from the super-market is also a big waste, as I lived in the west I've seen most of the households are buying more than they need, spending more than they need, cooking more food and more frequently than necessery and throughing large portions of food in the garbage (e.g. a huge amount of food waster).

If it was possible that people were aware they are buying all this non-sense food because of deception, they would have bought less, the super-markets would bought less and distribute less and the food (waster) remaining could be distributed to poor-er countries to help the starving kids and suffering people in Africa.
Helping the suffering and starving, we would have helped each other as it is a nowdays well known that even on molecular level the whole world is connected, therefore helping our poorer brothers and sisters is actually helping ourselves ***