Posts Tagged ‘scale’

Command to get CPU server load in % percentage using bash and /proc/stat on Linux

Wednesday, March 11th, 2015

Command-to-get-CPU-server-load-in-percentage-using-bash-shell-script-and-linux-proc-stat

Getting load avarage is easy with uptime command, however since nowadays Linux servers are running on multiple CPU machines and Dual cores, returned load avarage shows only information concerning a single processor. Of course seeing overall CPU server load is possible with TOP / TLoad command  / HTOP and a bunch of other monitoring commands, but how you can get a CPU percentage server load using just  /proc/stat and bash scripting? Here is hwo:
 

:;sleep=1;CPU=(`cat /proc/stat | head -n 1`);PREV_TOTAL=0;for VALUE in "${CPU[@]}”; do let “PREV_TOTAL=$PREV_TOTAL+$VALUE”;done;PREV_IDLE=${CPU[4]};sleep $sleep; CPU=(`cat /proc/stat | head -n 1`);unset CPU[0];IDLE=${CPU[4]};TOTAL=0; for VALUE in “${CPU[@]}"; do let "TOTAL=$TOTAL+$VALUE"; done;echo $(echo "scale=2; ((($sleep*1000)*(($TOTAL-$PREV_TOTAL)-($IDLE-$PREV_IDLE))/($TOTAL-$PREV_TOTAL))/10)" | bc -l );

52.45

As you can see command output shows CPU is loaded on 52.45%, so this server will soon have to be replaced with better hardware, because it gets CPU loaded over 50%

It is useful to use above bash shell command one liner together with little for loop to refresh output every few seconds and see how the CPU is loaded in percentage over time.

 

for i in $(seq 0 10); do :;sleep=1;CPU=(`cat /proc/stat | head -n 1`);PREV_TOTAL=0;for VALUE in "${CPU[@]}”; do let “PREV_TOTAL=$PREV_TOTAL+$VALUE”;done;PREV_IDLE=${CPU[4]};sleep $sleep; CPU=(`cat /proc/stat | head -n 1`);unset CPU[0];IDLE=${CPU[4]};TOTAL=0; for VALUE in “${CPU[@]}"; do let "TOTAL=$TOTAL+$VALUE"; done;echo $(echo "scale=2; ((($sleep*1000)*(($TOTAL-$PREV_TOTAL)-($IDLE-$PREV_IDLE))/($TOTAL-$PREV_TOTAL))/10)" | bc -l ); done

47.50

13.86
27.36
82.67
77.18

To monitor "forever" output from all server processor overall load use:
 

while [ 1 ]; do :;sleep=1;CPU=(`cat /proc/stat | head -n 1`);PREV_TOTAL=0;for VALUE in “${CPU[@]}”; do let “PREV_TOTAL=$PREV_TOTAL+$VALUE”;done;PREV_IDLE=${CPU[4]};sleep $sleep; CPU=(`cat /proc/stat | head -n 1`);unset CPU[0];IDLE=${CPU[4]};TOTAL=0; for VALUE in “${CPU[@]}"; do let "TOTAL=$TOTAL+$VALUE"; done;echo $(echo "scale=2; ((($sleep*1000)*(($TOTAL-$PREV_TOTAL)-($IDLE-$PREV_IDLE))/($TOTAL-$PREV_TOTAL))/10)" | bc -l ); done

 

 

What is Vertical scaling and Horizontal scaling – Vertical and Horizontal hardware / services scaling

Friday, June 13th, 2014

horizontal-vs-vertical-scaling-vertical-and-horizontal-scaling-explained-diagram

If you're coming from a small or middle-sized company to a corporations like HP or IBM probably you will not a clear defined idea on the 2 types (2 dimensions) of system Scaling (Horizontal and Vertical scaling). I know from my pesronal experience that in small companies – all needed is to guarantee a model for as less probels as possible without too much of defining things and with much less planning. Other thing is being a sysadmin in middle-sized companies, often doesn't give you opportunity to discuss issues to solve with other admins but you have to deal as "one man (machine) for all" and thus often to solve office server and services tasks you do some custom solution.
hence for novice system administrators probably it will be probably unclear what is the difference between Horizontal and Vertical Scaling?

horizontal-vertical-scaling-scale-up-and-scale-out-server-infrastructure-diagram

 

Vertical Scaling (scale vertically or scale up) :- adding more resources(CPU/RAM/DISK) to your server (database or application server is still remains one).
Vertical Scaling is much more used in small and middle-sized companies and in applications and products of middle-range. Very common example for Virtual Scaling nowdays is to buy an expensive hardware and use it as a Virtual Machine hypervisor (VMWare ESX). Where a database is involved using Vertical Scaling without use of multiple virtual machines might be not the best solution, as even though hardware might suffice (creation of database locks might impose problems). Reasons to scale vertically include increasing IOPS (Input / Ouput Operations), increasing CPU/RAM capacity, and increasing disk capacity.
Because Vertical Scaling usually means upgrade of server hardware – whenever an improved performance is targeted, even though if Virtualization is used, the risk for downtimes with it is much higher than whenever horizontal scaling.

Horizontal Scaling (scale horizontally or scale out):- adding more processing units (phyiscal machine) to your server (infrastructure be it application web/server or database).
Horizontal scaling, means increasing  the number of nodes in the cluster, reduces the responsibilities of each member node by spreading the keyspace wider and providing additional end-points for client connections. The capacity of each individual node does not change, but its load is decreased (because load is distributed between separate server nodes). Reasons to scale horizontally include increasing I/O concurrency, reducing the load on existing nodes, and increasing disk capacity.
Horizontal Scaling has been historically much more used for high level of computing and for application and services. The Internet and particular web services gave a boom of Horizontal Scaling use, most companies nowadays that provide well known web services like Google (Gmail, Youtube), Yahoo, Facebook, Ebay, Amazon etc. are using heavily horizontal scaling. Horizontal Scaling is a must use technology – whenever a high availability of (server) services are required.

Web and Middleware JBoss Training at Hewlett Packard – Intro to JBoss JAVA application server

Thursday, November 21st, 2013

jboss application server logo- serve java servlet pages on Linux and Windows

I and my TEAM Web and Middleware Implementation Team @Hewlett Packard are assigned an online training to follow on topic of JBoss Application server.It is my first online training of this kind where a number of people are streamed a video from a trainer who explains in real time concepts of JBossA Community Drive open source middleware (Application Server), since some time JBoss is known under a new name (WildFly).

Wildfly new name of jboss application java servlet server

In short what is JBoss? – It is an application server similar to Apache Tomcat  -an open source software implementation of the Java Servlet and JavaServer Pages technologies.

Apache Tomcat java servlet application server logo

In case you wonder about what is Middleware it is a buzzword well established in Corporate world referring to all kind of servers in the middle between Servers on pure OS and hardware Level and end client. Middleware includes all kind of Web and Application servers like Apache, JBoss, Tomcat, Adobe's WebLogic Webserver, IBM WebSphere application server etc..

What this means is JBOSS is very similar to Tomcat but it is designed to run interpret through (Java Virtual Machine), higher scale of Java Applications and then return content to a a web browser. In other words if you need to have a Webserver with support for Java VM. JBoss is one of the open source technologies available which can be a substitute for Tomcat. In Fact Jboss itself started as a fork of Tomcat and n owadays, Jboss has an implementation of Tomcat embedded into itself. Jboss is mainly developed and supported by Redhat. It has 3 major releases used in IT Companies. Jboss 5, JBoss 6 and JBoss 7. In most production server systems running some kind of Java servlets currently still Jboss ver. 5 and Jboss v. 6 is used. Just like Tomcat, the server is messy in its structure. But if we have to compare Tomcat with Jboss then JBoss is at least 100 times more messy and hard to configure tune than Tomcat. Actually after getting to know JBoss 6 I would not advice anyone to use this Application server. Its too complex and all configuration and performance tuning is done through hundred of XML so it is like a hell for the usual System Administrator who likes clearness and simplicity. JBoss has a Web configuration interface which in version 7 is a bit advanced and easier to configure and get to know the server compared to previous versions. But same web interface for older releases is lousy and not nice. Just like Tomcat, JBoss supports clustering, here is full list of all features it supports:

  • Full clustering support for both traditional J2EE applications and EJB 3.0 POJO applications
  • Automatic discovery. Nodes in cluster find each other with no additional configuration.
  • Cluster-wide replicated JNDI context
  • Failover and load-balancing for JNDI, RMI and all EJB types
  • Stateful Session Bean state replication
  • HTTP Session replication
  • High Availability JMS
  • Farming. Distributed deployment of JBoss components. Deploying on one node deploys on all nodes.
     

Looks like JBoss is among the few Application Servers supporting deployment of Java JSP, WAR Archive files, SAR Archives, JMS (Java Message Service), JNDI (Java Naming and Directory Interface). Jboss supports load balancing between clustered nodes, supports SOAP, Java servlet faces and Java MQ (Messaging Queue). JBoss can be installed on GNU / Linux, FreeBSD and Windows. So far from what I've learned for JBOSS I prefer not to use it and don't recommend this Application server to anyone. Its too complex and doesn't worth the effort to learn. Proprietary products like WebLogic and Webspehere are in light years better.

Video Review of Top Cities, Places and Things – must see in the Netherlands

Thursday, January 17th, 2013

South of Haarlem, makes the ultimate outing for ogling Holland's famed flower. The bulb fields in and around Keukenhof burst into colour from the end of March to mid May.
 

HOLLAND: Keukenhof – world's largest flower garden [HD]

 


 

hoe leuk is nederland , delft holland (How nice is Holland – Delft (one of the ancient cities in Holland)


 

Utrecht, The Netherlands

Views of the city of Utrecht and its tourist attractions like the Oudegracht canal, the Domkerk, Domtoren, Pieterkerk, Muziek Centrum. Check out the old tower museum and try to spot the Gothic churches spread around the city.


 

Utrecht Netherlands, City, Life and University


 

HOLLAND in MINIATURE – MADURODAM – The Hague, Netherlands

Madurodam is a park and tourist attraction in Scheveningen, The Hague, Netherlands, home to a range of perfect 1:25 scale model replicas of famous Dutch castles, public buildings, and large industrial projects as found at various locations in the country. The park was opened in 1952 and has been visited by tens of millions of visitors since that date. Madurodam was named after George Maduro, a Jewish law student from Curaçao who fought the Nazi occupation forces as a member of the Dutch resistance and died at Dachau concentration camp in 1945. In 1946 Maduro was posthumously awarded the medal of Knight 4th-class of the Military Order of William, the highest and oldest military decoration in the Kingdom of the Netherlands, for the valor he had demonstrated in the Battle of the Netherlands against German troops. His parents donated the funds necessary for the Madurodam project.


 

Scheveningen Beach (The Hague) Netherlands

Very surprisingly in Holland there are beaches too 🙂 It is a bit funny in such a Northern Country, there could be beach but there is. As you can see the sun lack takes off from enjoying the beach bathing. If you never have tried a Northern Beach destination and you happen to drop off Netherlands in the Summer give it a try 🙂


 

Haarlem The Netherlands – An Old Dutch city who most likely give birth to famous American suburb


 

Holland Gouda, The World famous Cheese Market

Nationaal park De Hoge Veluwe

near Arnhem, among whose attractions is an underground museum dedicated to subterranean life, and the Kröller-Müller Museum (www.kmm.nl) which contains 280 Van Gogh paintings as well as numerous other works. Bicycles are available free of charge to visitors.


 

Maastricht, Netherlands – Ancient Dutch city, place where European Union creation agreement was signed

Holland's oldest fortified city – Full of Medieval interesting architecture


 

NEMO Museum Amsterdam

 


 

Rotterdam Holland Tourist Attractions


 

Amsterdam Holland

Holland's emblematic capital (www.iamsterdam.com) brims with romance, culture and kicks. Take a candlelit cruise along the canals, lined with narrow-fronted buildings dating from the city's heyday as a trading centre, browse the paintings of the Dutch masters at the Rijksmuseum or join the punters prowling the alleyways of the Red Light District.


 

Hermitage Amsterdam Tour – Sibling of the Original Hermitage in St. Peterburg


 

Open air museum Arnhem in the Netherlands

Life was easier when Blackberry and Apple were just fruits

Saturday, June 30th, 2012

Life was much easier when blackberry and Apple were just fruits

Sad but true fact Life was much easier when BlackBerry and Apple were just fruits!.

I don’t know about others but I have difficulties to adjust my body to the fast pace life has taken. Life would was easy and more fun without all this technology. Now even going to a pub you can’t escape the “doom” of technology. If the tech hype continues the way it does or something drastic doesn’t happen I guess all people on earth will get completely crazy …

I thougth about how this techno-mambo jumbo can be stopped on a wide scale and the only idea I have is a weapon that blocks all tech equipment world-wide. I hope someone will develop it soon and press the destory button SOON ! 🙂

How road signs evolved / short history of roadsigns

Friday, June 29th, 2012

how the road signs evolved brief history of road signs Ancient Roman Road of Tall Aqibrin

As a person interested in history and antrophology. Just recently on my last trip to Romania as I travelled a very interesting question poped up in my mind – How it happened that RoadSigns we use on every street highway and practically everywhere on the road came to be. Interestingly now with the standartization of road signs often the most popular road signs are used as a basis for development on other popular prohibit or allowance signs, we read on airports public institutions, pubs and mostly everywhere.

So in short I did a short research on Road Sign History, just to find out once again that the ancients, were wiser than we think. The first road signs probably came to existence with the existence of humanity, however officially, there was no standartization of using signs to point on road locations travellers before it was introudced in the Roman Empire. In Rome a pillars on the roads were placed to point to major road arteries leading to Rome and various important empire city centers.

During the middle ages, milestones pillars were no longer used, but for practical reasons wooden markers placed across european cities instructed tradesman and travellers to major city important centers and were used to show a general road direction leading to nearby city.
The wooden signs practice had been in use until the first modern roadsigns erected  on a wide scale designed for riders of 'high' and ordinary bicycles in the late 1870 and 1880s. The modern road signs as we know it today however emerged as a result of the  first International Road Congress meeting that occured in Rome in 1908.
On the meeting a four standard pictures were selected to note the basic for road signs further development. The need for the meeting was the large increase of roads across european artery cities. The road signs developed on the meeting were bump, curve, intersection and railroad crossings. The invention and adoption of cars and the boom of the car producing industry quickened the need for international road sign standard. The intensive work on international road signs that took place between 1926 and 1949 eventually led to the development of the European road sign system as we know it. The signs were quicky spread to America and in 1960, the road signs become universal in America and almost everywhere all around the developed and developing world.

As of today 2012 it can be said road signs exist all around the civilized world.Though most of road signs are identical across all countries around the world today still some road symbols varies from country to country. I remember seeing some very unique road signs during my travelling through Serbia, 2 years ago.
 

Possible way to increase Linux TCP/IP port thoroughput via sysctl kernel variable

Thursday, August 25th, 2011

Sysctl is a great way to optimize Linux. sysctl has a dozens of values which could drastically improve server networking and overall performance.

One of the many heplful variables to optimize the way the Linuz kernel works on busy servers is net.ipv4.ip_local_port_range .

The default sysctl setting for net.ipv4.ip_local_port_range on Debian, Ubuntu Fedora, RHEL, CentOS is:

net.ipv4.ip_local_port_range = 32768 65536

This means that the kernel and the corresponding server running services instructing the Linuz kernel open new port sockets can only open local ports in the range of 32768 – 65536 .
On a regular Desktop GNU/Linux machine or a not high iron server this settins is perfectly fine, however on a high scale servers the local port range in the interval of 32768-65536 might be insufficient at times, especially if there are programs which require binding of many local ports.

Therefore on a high load servers, generally it’s a good to raise the port range to be assigned by kernel to 8912 – 65536 , to do so the setting has to be changed like shown below:

linux:~# sysctl -w net.ipv4.ip_local_port_range = 8192 65536
...

If changing this setting on the server doesn’t show any negative impact on performance in few hours time or a day or even better decreases the server average load, it’s a good idea that it be added to sysctl.conf to load up the setting on next kernel boot.

linux:~# echo 'net.ipv4.ip_local_port_range' >> /etc/sysctl.conf

Enjoy 😉