The Athonite Monks in Holy Mount Athos serves an All Night Vigil prayer to God to stop Coronavirus COVID-19 epidemic

March 28th, 2020


In the Eastern Orthodox World to which my homeland Bulgaria belongs is deeply saddened and worried about the all time worsening situation with the New Worldwide pandemics which caused suffering and death to already thousands of people. We the Orthodox Christians have the rule that if you can't manage a problem you have to pray God to fix the problem and repent and try to fix what is at your power. This is exactly what pushed the Monks from Biggest on Earth Monk Republic situated on an Island near Greece the Holy Mount Athos to create a All Monasteries Night Vigil for concillation of the The God the Holy Trinity – The Father, The Son and The Holy Spirit with the mankind who has done a terrible deeds over the last 30 years against Gods well known law. 

All kind of well known and prohibited sins about which the Holy Church, The Saint Writtings, Church Tradition and The Holy Bible warned us would cause a mass suffering, deaths and a havoc, that their unfollowing will bring. As the warnings and signs given by God to mankind in the form of various Cataclysms especially visible in Italy over the last few years and in virtually all countries has been ignored and the attention of the people was falsely led to material stuff and for the never ending consumer needs by the spiritually highly blind mankind, well it is not strange the payback for the Godlessness of mankind is at hand with the quickly spreading new infectious disease codenamed Coronavirus COVID-19.

Of course atheists reject that and they are giving the result of the Coronavirus to Biological Warfare / animal origin or simply a human hygiene breach, but nomatter that for anyone that has the desire to see it, it is clear that what is happening is one of the many God's mercies to mankind. 

As over the centuries many times we were send the Lord, other misfortunes wars and epidemics as an attempt to save us and bring up our eyes out of the material things which are doomed to be gone one day to the Heavenly and All time lasting spiritual things.
Since the boom of the COVID-19 pandemics, Monks from Holy Mount Athos has made a Call to all Orthodox Christians world wide from All the Holy Eastern Orthodox country 14 Churches and every Orthodox Christian on Earth to read every day at 21:00 o'clock psalms and a special prayers to The well Known helper in Epidemics and Deadly diaseses Saint Haralambos (Св. Харалампий) / Saint Charalampos of Magnesia on the Maeander (Greek city in Ionia so called Asia Minor).


and the Mother of God (Theotokos) called Acathist to Panagia ( Всецарица ) Pantanassa – a miraculous icon of the Virgin Mary from the holy and great Monastery of Vatopedi, Mount Athos, Greece depicting the Theotokos enthroned and holding her son Lord Jesus Christ.


As the result of this Pandemics would be a prison like state for humanity economics and a decay of civillization as we know it and the clear realization of the Monks on Holy Athos whose spiritual seeing has been widely open to see the things happening on earth such as they're and the overall late temptations within the Church and the actions of some of the Hierarchs that are trying to destroy the unity and love we had for the last 2020 years …

The athonites today decided to pray all night long continuously as they do on a Big Christian feasts such as Resurrection, Nativity and Dormition etc. (which by the way for the Athos island are relatively common but all monasteries all Night vigil together is unique and only happened in most harsh times in human history, where full spiritual backup was necessery).

In each and every monastery just like in every Holy Liturgy served we pray for the inceasingly praying for the sick with Coronavirus, the lonely, the suffering the hungry, the saddened, people in pay for our parents, grandma, grandpas, sons daughters all Nations the God-fearing governmental powers and us the childs of the Holy Eastern Orthodox Church.

This prayer is to be streamed straight from the Monasteries is to be streamed online for everyone who can to be unite to become part of the prayer by listening it and secretly repeating deep in his heart the prayer that is said to be giving the highest goods on earth, the so called Jesus Prayer.

This prayer is very simple and the long version os if consists of the words "Lord Jesus Christ, Son of the Blessed God, have mercy of me the sinner!".
The shorter one is simply "Lord Jesus Christ Have mercy on me", and the shortest one is "Lord Have Mercy".
Monks on Athos depending on their nationality repeat "Lord Have Mercy, Lord Have Mercy, Lord Jesus Christ have mercy on me …" in Greek Bulgarian, Romanian, Serbian, Greek, Georgian, Russian etc. praying for themselves and benefitting from the spiritual well of Eternity spreading this Grace through themselves for the world.


Lets not forget that the Holy Bible and The experience from the Church history and tradition, tells us about multiple epidemics such as Black Death who were destroyed by many miracles for the humility and prayers of the saintly man and the merfiness of Christ and his Holy Mother The Holy Theotoks Virgin Mary and numerous Saints. 

It is in the Church tradition in this heavy times of pestilence to make Lithia (Holy Cross Procession, Procession with Miracle making icons, Holy relics from saints) has been used to put an end to the diseases and the God's grace which has been attarcted for the faith of People and their timely repentance stopped the diseases, this can be checked in multiple ancient manuscripts and history books, written by various historions.

Also the story of the paralytic, the blindly born, the lepper, the deaf and innumerous other sick people is found in the Bible and we read that they were easily healed by the savior, so we need to come back to our faith, which we have changed with the Friday lustful enjoyment meet in Discos, Caffeterias, Bars Cinema, Theather and kind of spiritually unhealthy and most of time unusuable activities from the perspective of the Eternal Soul

Perhaps it is not a coincidence that this difficult for every living soul situation happened exactly during the time of the Great Lent in which the whole Church is praying to God for forgiveness of the multitude of sins done since the last year Lent time …

Let us pray that this terrible disease who is physically separating us, do on the contrary unite us and bring us spiritual-growth, Knowledge of God, turning out from the sinful habits, thoughts and deeds to do what is pleasant good and wonderful as the Bible says and which has been the ultimate reason why we Human were created in the Beginning.

Lord Jesus Christ son of the Blessed God have mercy on us the Sinners !!!! Lord Jesus Save and Heal the Sick, protect the Healthy, give us repentance, give us your love and strenght, forgive us our terrible sins, be with us always until the End of Ages and in all Eternity !



Find when cron.daily cron.weekly and cron.monthly run on Redhat / CentOS / Debian Linux and systemd-timers

March 25th, 2020



The problem – Apache restart at random times

I've noticed today something that is occuring for quite some time but was out of my scope for quite long as I'm not directly involved in our Alert monitoring at my daily job as sys admin. Interestingly an Apache HTTPD webserver is triggering alarm twice a day for a short downtime that lasts for 9 seconds.

I've decided to investigate what is triggering WebServer restart in such random time and investigated on the system for any background running scripts as well as reviewed the system logs. As I couldn't find nothing there the only logical place to check was cron jobs.
The usual

crontab -u root -l

Had no configured cron jobbed scripts so I digged further to check whether there isn't cron jobs records for a script that is triggering the reload of Apache in /etc/crontab /var/spool/cron/root and /var/spool/cron/httpd.
Nothing was found there and hence as there was no anacron service running but /usr/sbin/crond the other expected place to look up for a trigger even was /etc/cron*


1. Configured default cron execution times, every day, every hour every month


# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 feb 27 10:54 /etc/cron.d/
drwxr-xr-x 2 root root 4096 dec 27 10:55 /etc/cron.daily/
drwxr-xr-x 2 root root 4096 dec  7 23:04 /etc/cron.hourly/
drwxr-xr-x 2 root root 4096 dec  7 23:04 /etc/cron.monthly/
drwxr-xr-x 2 root root 4096 dec  7 23:04 /etc/cron.weekly/


After a look up to each of above directories, finally I found the very expected logrorate shell script set to execute from /etc/cron.daily/logrotate and inside it I've found after the log files were set to be gzipped and moved to execute WebServer restart with:

systemctl reload httpd 


My first reaction was to ponder seriously why the script is invoking systemctl reload httpd instead of the good oldschool

apachectl -k graceful


But it seems on Redhat and CentOS since RHEL / CentOS version 6.X onwards systemctl reload httpd is supposed to be identical and a substitute for apachectl -k graceful.
Okay the craziness of innovation continued as obviously the reload was causing a Downtime to be visible in the Zabbix HTTPD port Monitoring graph …
Now as the problem was identified the other logical question poped up how to find out what is the exact timing scheduled to run the script in that unusual random times each time ??

2. Find out cron scripts timing Redhat / CentOS / Fedora / SLES


/etc/cron.{daily,monthly,weekly} placed scripts's execution method has changed over the years, causing a chaos just like many Linux standard things we know due to the inclusion of systemd and some other additional weird OS design changes. The result is the result explained above scripts are running at a strange unexpeted times … one thing that was intruduced was anacron – which is also executing commands periodically with a different preset frequency. However it is considered more thrustworhty by crond daemon, because anacron does not assume the machine is continuosly running and if the machine is down due to a shutdown or a failure (if it is a Virtual Machine) or simply a crond dies out, some cronjob necessery for overall set environment or application might not run, what anacron guarantees is even though that and even if crond is in unworking defunct state, the preset scheduled scripts will still be served.
anacron's default file location is in /etc/anacrontab.

A standard /etc/anacrontab looks like so:

[root@centos ~]:# cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
# the maximal random delay added to the base delay of the jobs
# the jobs will be started during the following hours only
#period in days   delay in minutes   job-identifier   command
1    5    cron.daily        nice run-parts /etc/cron.daily
7    25    cron.weekly        nice run-parts /etc/cron.weekly
@monthly 45    cron.monthly        nice run-parts /etc/cron.monthly


START_HOURS_RANGE : The START_HOURS_RANGE variable sets the time frame, when the job could started. 
The jobs will start during the 3-22 (3AM-10PM) hours only.

  • cron.daily will run at 3:05 (After Midnight) A.M. i.e. run once a day at 3:05AM.
  • cron.weekly will run at 3:25 AM i.e. run once a week at 3:25AM.
  • cron.monthly will run at 3:45 AM i.e. run once a month at 3:45AM.

If the RANDOM_DELAY env var. is set, a random value between 0 and RANDOM_DELAY minutes will be added to the start up delay of anacron served jobs. 
For instance RANDOM_DELAY equels 45 would therefore add, randomly, between 0 and 45 minutes to the user defined delay. 

Delay will be 5 minutes + RANDOM_DELAY for cron.daily for above cron.daily, cron.weekly, cron.monthly config records, i.e. 05:01 + 0-45 minutes

A full detailed explanation on automating system tasks on Redhat Enterprise Linux is worthy reading here.

!!! Note !!! that listed jobs will be running in queue. After one finish, then next will start.

3. SuSE Enterprise Linux cron jobs not running at desired times why?

in SuSE it is much more complicated to have a right timing for standard default cron jobs that comes preinstalled with a service 

In older SLES release /etc/crontab looked like so:



# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

As time of writting article it looks like:


# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
-*/15 * * * *   root  test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1



This runs any scripts placed in /etc/cron.{hourly, daily, weekly, monthly} but it may not run them when you expect them to run. 
/usr/lib/cron/run-crons compares the current time to the /var/spool/cron/lastrun/cron.{time} file to determine if those jobs need to be run.

For hourly, it checks if the current time is greater than (or exactly) 60 minutes past the timestamp of the /var/spool/cron/lastrun/cron.hourly file.

For weekly, it checks if the current time is greater than (or exactly) 10080 minutes past the timestamp of the /var/spool/cron/lastrun/cron.weekly file.

Monthly uses a caclucation to check the time difference, but is the same type of check to see if it has been one month after the last run.

Daily has a couple variations available – By default it checks if it is more than or exactly 1440 minutes since lastrun.
If DAILY_TIME is set in the /etc/sysconfig/cron file (again a suse specific innovation), then that is the time (within 15minutes) when daily will run.

For systems that are powered off at DAILY_TIME, daily tasks will run at the DAILY_TIME, unless it has been more than x days, if it is, they run at the next running of run-crons. (default 7days, can set shorter time in /etc/sysconfig/cron.)
Because of these changes, the first time you place a job in one of the /etc/cron.{time} directories, it will run the next time run-crons runs, which is at every 15mins (xx:00, xx:15, xx:30, xx:45) and that time will be the lastrun, and become the normal schedule for future runs. Note that there is the potential that your schedules will begin drift by 15minute increments.

As you see this is very complicated stuff and since God is in the simplicity it is much better to just not use /etc/cron.* for whatever scripts and manually schedule each of the system cron jobs and custom scripts with cron at specific times.

4. Debian Linux time start schedule for cron.daily / cron.monthly / cron.weekly timing

As the last many years many of the servers I've managed were running Debian GNU / Linux, my first place to check was /etc/crontab which is the standard cronjobs file that is setting the { daily , monthly , weekly crons } 


 debian:~# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 фев 27 10:54 /etc/cron.d/
drwxr-xr-x 2 root root 4096 фев 27 10:55 /etc/cron.daily/
drwxr-xr-x 2 root root 4096 дек  7 23:04 /etc/cron.hourly/
drwxr-xr-x 2 root root 4096 дек  7 23:04 /etc/cron.monthly/
drwxr-xr-x 2 root root 4096 дек  7 23:04 /etc/cron.weekly/


debian:~# cat /etc/crontab 
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin# Example of job definition:
# .—————- minute (0 – 59)
# |  .————- hour (0 – 23)
# |  |  .———- day of month (1 – 31)
# |  |  |  .——- month (1 – 12) OR jan,feb,mar,apr …
# |  |  |  |  .—- day of week (0 – 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed
17 *    * * *    root    cd / && run-parts –report /etc/cron.hourly
25 6    * * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.daily )
47 6    * * 7    root    test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.weekly )
52 6    1 * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.monthly )

What above does is:

– Run cron.hourly once at every hour at 1:17 am
– Run cron.daily once at every day at 6:25 am.
– Run cron.weekly once at every day at 6:47 am.
– Run cron.monthly once at every day at 6:42 am.

As you can see if anacron is present on the system it is run via it otherwise it is run via run-parts binary command which is reading and executing one by one all scripts insude /etc/cron.hourly, /etc/cron.weekly , /etc/cron.mothly

anacron – few more words

Anacron is the canonical way to run at least the jobs from /etc/cron.{daily,weekly,monthly) after startup, even when their execution was missed because the system was not running at the given time. Anacron does not handle any cron jobs from /etc/cron.d, so any package that wants its /etc/cron.d cronjob being executed by anacron needs to take special measures.

If anacron is installed, regular processing of the /etc/cron.d{daily,weekly,monthly} is omitted by code in /etc/crontab but handled by anacron via /etc/anacrontab. Anacron's execution of these job lists has changed multiple times in the past:

debian:~# cat /etc/anacrontab 
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.


# These replace cron's entries
1    5    cron.daily    run-parts –report /etc/cron.daily
7    10    cron.weekly    run-parts –report /etc/cron.weekly
@monthly    15    cron.monthly    run-parts –report /etc/cron.monthly

In wheezy and earlier, anacron is executed via init script on startup and via /etc/cron.d at 07:30. This causes the jobs to be run in order, if scheduled, beginning at 07:35. If the system is rebooted between midnight and 07:35, the jobs run after five minutes of uptime.
In stretch, anacron is executed via a systemd timer every hour, including the night hours. This causes the jobs to be run in order, if scheduled, beween midnight and 01:00, which is a significant change to the previous behavior.
In buster, anacron is executed via a systemd timer every hour with the exception of midnight to 07:00 where anacron is not invoked. This brings back a bit of the old timing, with the jobs to be run in order, if scheduled, beween 07:00 and 08:00. Since anacron is also invoked once at system startup, a reboot between midnight and 08:00 also causes the jobs to be scheduled after five minutes of uptime.
anacron also didn't have an upstream release in nearly two decades and is also currently orphaned in Debian.

As of 2019-07 (right after buster's release) it is planned to have cron and anacron replaced by cronie.

cronie – Cronie was forked by Red Hat from ISC Cron 4.1 in 2007, is the default cron implementation in Fedora and Red Hat Enterprise Linux at least since Version 6. cronie seems to have an acive upstream, but is currently missing some of the things that Debian has added to vixie cron over the years. With the finishing of cron's conversion to quilt (3.0), effort can begin to add the Debian extensions to Vixie cron to cronie.

Because cronie doesn't have all the Debian extensions yet, it is not yet suitable as a cron replacement, so it is not in Debian.

5. systemd-timers – The new crazy systemd stuff for script system job scheduling

Timers are systemd unit files with a suffix of .timer. systemd-timers was introduced with systemd so older Linux OS-es does not have it.
 Timers are like other unit configuration files and are loaded from the same paths but include a [Timer] section which defines when and how the timer activates. Timers are defined as one of two types:


  • Realtime timers (a.k.a. wallclock timers) activate on a calendar event, the same way that cronjobs do. The option OnCalendar= is used to define them.
  • Monotonic timers activate after a time span relative to a varying starting point. They stop if the computer is temporarily suspended or shut down. There are number of different monotonic timers but all have the form: OnTypeSec=. Common monotonic timers include OnBootSec and OnActiveSec.



    For each .timer file, a matching .service file exists (e.g. foo.timer and foo.service). The .timer file activates and controls the .service file. The .service does not require an [Install] section as it is the timer units that are enabled. If necessary, it is possible to control a differently-named unit using the Unit= option in the timer’s [Timer] section.

    systemd-timers is a complex stuff and I'll not get into much details but the idea was to give awareness of its existence for more info check its manual man systemd.timer

Its most basic use is to list all configured systemd.timers, below is from my home Debian laptop

debian:~# systemctl list-timers –all
NEXT                         LEFT         LAST                         PASSED       UNIT                         ACTIVATES
Tue 2020-03-24 23:33:58 EET  18s left     Tue 2020-03-24 23:31:28 EET  2min 11s ago laptop-mode.timer            lmt-poll.service
Tue 2020-03-24 23:39:00 EET  5min left    Tue 2020-03-24 23:09:01 EET  24min ago    phpsessionclean.timer        phpsessionclean.service
Wed 2020-03-25 00:00:00 EET  26min left   Tue 2020-03-24 00:00:01 EET  23h ago      logrotate.timer              logrotate.service
Wed 2020-03-25 00:00:00 EET  26min left   Tue 2020-03-24 00:00:01 EET  23h ago      man-db.timer                 man-db.service
Wed 2020-03-25 02:38:42 EET  3h 5min left Tue 2020-03-24 13:02:01 EET  10h ago      apt-daily.timer              apt-daily.service
Wed 2020-03-25 06:13:02 EET  6h left      Tue 2020-03-24 08:48:20 EET  14h ago      apt-daily-upgrade.timer      apt-daily-upgrade.service
Wed 2020-03-25 07:31:57 EET  7h left      Tue 2020-03-24 23:30:28 EET  3min 11s ago anacron.timer                anacron.service
Wed 2020-03-25 17:56:01 EET  18h left     Tue 2020-03-24 17:56:01 EET  5h 37min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service


8 timers listed.

N ! B! If a timer gets out of sync, it may help to delete its stamp-* file in /var/lib/systemd/timers (or ~/.local/share/systemd/ in case of user timers). These are zero length files which mark the last time each timer was run. If deleted, they will be reconstructed on the next start of their timer.


In this article, I've shortly explain logic behind debugging weird restart events etc. of Linux configured services such as Apache due to configured scripts set to run with a predefined scheduled job timing. I shortly explained on how to figure out why the preset default install configured cron jobs such as logrorate – the service that is doing system logs archiving and nulling run at a certain time. I shortly explained the mechanism behind cron.{daily, monthy, weekly} and its execution via anacron – runner program similar to crond that never misses to run a scheduled job even if a system downtime occurs due to a crashed Docker container etc. run-parts command's use was shortly explained. A short look at systemd.timers was made which is now essential part of almost every new Linux release and often used by system scripts for scheduling time based maintainance tasks.

Great Lent third Sunday of the Veneration of the Holy Christ Cross

March 23rd, 2020
Today is the third Sunday of the Holy Christ Cross Holy Feast (Кръстовден / The day of the Cross).
Each of the Great Lent time period has its own Sunday dedicated to very important events or saints.
This one has been set over the centuries by the Holy Eastern Orthodox Church to accent the Great unbeatable power of the Holy Cross. Which is the greatest power through which each of us Christians is victorious over sin, flesh, passions, temptations and everything visible and invisible.
The cross provides every person and us as socieity with 'The power of God and Wisdowm of God' to those who venerate it for those being saved as pointed in Holy Bible in scriptures (1 Corintians 1:24).
This Sunday was set to help people recharge their spiritual powers in feat of Lentin hardship weakness and temptations. It is to remind the faithful Christians of the redemption received through Christ's sufferings for our sins. But besides the joy of the Cross it is to remind us another important fact – "He who does not take up his cross and follow me is not worthy of me" (Mt 10:38).

The hymnology (the Church songs) accent over the Holy Liturgy morning service is the victory and joy of the cross, not the suffering. The Church fathers equate the life-giving cross with the tree of life and plant it in the middle of the Lenten pilgrimage. It was the tree that was planted in Paradise; it is to remind the faithful of both Adam's bliss and how he was deprived from it.

The Church teaches that it is Christ's cross that saves. One cannot take up his own cross and follow Christ unless one has Christ's cross which he took up to save mankind. Partaking of this tree, one will no longer die, but will be kept alive.


History of the Veneration of Holy Cross

The commemoration and ceremonies of the Third Sunday of Lent are closely parallel to the feasts of the Veneration of the Cross (September 14) and the Procession of the Cross (August 1). Not only does the Sunday of the Holy Cross prepare us for commemoration of the Crucifixion, but it also reminds us that the whole of Lent is a period when we are crucified with Christ.

As we have “crucified the flesh with its passions and desires” (Galatians 5:24), and will have mortified ourselves during these forty days of the Fast, the precious and life-giving Cross is now placed before us to refresh our souls and encourage us who may be filled with a sense of bitterness, resentment, and depression. The Cross reminds us of the Passion of our Lord, and by presenting to us His example, it encourages us to follow Him in struggle and sacrifice, being refreshed, assured, and comforted. In other words, we must experience what the Lord experienced during His Passion – being humiliated in a shameful manner. The Cross teaches us that through pain and suffering we shall see the fulfillment of our hopes: the heavenly inheritance and eternal glory.

As they who walk on a long and hard way and are bowed down by fatigue find great relief and strengthening under the cool shade of a leafy tree, so do we find comfort, refreshment, and rejuvenation under the Life-giving Cross, which our Fathers “planted” on this Sunday. Thus, we are fortified and enabled to continue our Lenten journey with a light step, rested and encouraged.



As at the feast of the Elevation of the Holy Cross on September 14, the Trisagion is replaced by the hymn:

Before Thy Cross we bow down in worship, O Master, and Thy holy Resurrection we glorify. (Thrice)
Glory to the Father, and to the Son, and to the Holy Spirit, both now and ever, and unto the ages of ages. Amen.
And Thy holy Resurrection we glorify.
Before Thy Cross we bow down in worship, O Master, and Thy holy Resurrection we glorify.

Тропар Църковнославянски (Българска Православна Църква) / Troparion Old Bulgarian (Church Slavonic) Bulgarian Orthodox Church

Спаси, Господи, люди Твоя и благослови достояние Твое, победи православному болгарскому народу на сопротивния даруя, и Твое сохраняя Крестом Твоим жительство.

Troparion (Tone 4) [1]

O Lord, save Your people,
And bless Your inheritance.
Grant victories to the Orthodox Christians,
over their adversaries.
And by virtue of Your Cross
Preserve Your habitation!
Kontakion (Tone 7) [2]

Now the flaming sword no longer guards the gates of Eden;
It has mysteriously been quenched by the wood of the Cross!
The sting of death and the victory of hell have been vanquished;
For You, O my Savior, have come and cried to those in hell:
"Enter again into paradise."


It is worthy to remind my dear readers for the Cross, because nowadays everyone is scared from the Coronavirus COVID-19 mysterious virus who is plaguing the death and who makes people who have no faith to tremble besides a simple flu, but do not tremble in front of the Cross of Christ and the near coming Great Judgement day in which each of us will have to give account for his good and bad deeds …
Also through the centuries the Cross has been a reason for many miracles. In this scareful days where constitutions of many countries has been breached and a military regimes are pushed into our countres.

Lets pray and bow the holy cross of Christ to protect our lands, Countries families and loved grand parents, mothers, fathers, children, friends and every living creature from the evil that is lurking the earth!


Check if server is Physical Bare Metal or a Virtual Machine and its type

March 17th, 2020


In modern times the IT employee system administrator / system engineer / security engineer or a developer who has to develop and test code remotely on UNIX hosts, we have to login to multiple of different servers located in separate data centers around the world situated in Hybrid Operating system environments running multitude of different Linux OSes. Often especially for us sysadmins it is important to know whether the remote machine we have SSHed to is physical server (Bare Metal) or a virtual machines running on top of different kind of Hypervisor node OpenXen / Virtualbox / Virtuosso  / VMWare etc.

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?

1. Using the dmesg kernel log utility

The good old dmesg that is used to examine and control the kernel ring buffer detects plenty of useful information which gives you the info whether a server is Virtual or Bare Metal. It is present and accessible on every Linux server out there, thus using it is the best and simplest way to determine the OS system node type.

To grep whether a machine is Virtual and the Hypervisor type use:


nginx:~# dmesg | grep "Hypervisor detected"
[0.000000] Hypervisor detected: KVM

As you see above OS installed is using the KVM Virtualization technology.

An empty output of this command means the Remote OS is installed on a physical computer.


2. Detecting the OS platform the systemd way

Systemd along with the multiple over-complication of things that nearly all sysadmins (including me hate) so much introduced something useful in the fact of hostnamectl command
that could give you the info about the OS chassis platform.


root@pcfreak:~# hostnamectl status
 Static hostname: pcfreak
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 02425d67037b8e67cd98bd2800002671
           Boot ID: 34a83b9a79c346168082f7605c2f557c
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-amd64
      Architecture: x86-64



Below is output of a VM running on a Oracle Virtualbox HV.


linux:~# hostnamectl status
Static hostname: ubuntuserver
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 2befe86cf8887ca098f509e457554beb
 Boot ID: 8021c02d65dc46b1885afb25fddcf18c
 Virtualization: oracle
 Operating System: Ubuntu 16.04.1 LTS
 Kernel: Linux 4.4.0-78-generic
 Architecture: x86-64


3. Detect concrete container virtualization with systemd-detect-virt 

Another Bare Metal or VM identify tool that was introducted some time ago by freedesktop project is systemd-detect-virt (usually command is part of systemd package).
It is useful to detect the exact virtualization on a systemd running OS systemd-detect-virt is capable to detect many type of Virtualization type that are rare like: IBM zvm S390 Z/VM, bochs, bhyve (a FreeBSD hypervisor), Mac OS's parallels, lxc (linux containers), docker containers, podman etc.

The output from the command is either none (if no virtualization is present or the VM Hypervisor Host type):


server:~# systemd-detect-virt


quake:~# systemd-detect-virt


4. Install and use facter to report per node facts


debian:~# apt-cache show facter|grep -i desc -A2
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and
 reports per-node facts, which are collected by the Puppet agent and are made

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Tag: devel::lang:ruby, devel::library, implemented-in::ruby,
root@jeremiah:/home/hipo# apt-cache show facter|grep -i desc -A1
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and

Description-md5: 88cdf9a1db3df211de4539a0570abd0a


– Install facter on Debian / Ubuntu / deb based Linux


# apt install facter –yes

– Install facter on RedHat / CentOS RPM based distros

# yum install epel-release


# yum install facter

– Install facter on OpenSuSE / SLES

# zypper install facter

Once installed on the system to find out whether the remote Operating System is Virtual:

# facter 2> /dev/null | grep virtual
is_virtual => false
virtual => physical

If the machine is a virtual machine you will get some different output like:

# facter 2> /dev/null | grep virtual
is_virtual => true
virtual => kvm

If you're lazy to grep you can use it with argument.

# facter virtual


6. Use lshw and dmidecode (list hardware configuration tool)

If you don't have the permissions to install facter on the system and you can see whether lshw (list hardware command) is not already present on remote host.

# lshw -class system  
    description: Computer
    width: 64 bits
    capabilities: smbios-2.7 vsyscall32

If the system is virtual you'll get an output similar to:

# lshw -class system  
 description: Computer
 product: VirtualBox
 vendor: innotek GmbH
 version: 1.2
 serial: 0
 width: 64 bits
 capabilities: smbios-2.5 dmi-2.5 vsyscall32
 configuration: family=Virtual Machine uuid=78B58916-4074-42E2-860F-7CAF39F5E6F5

Of course as it provides a verbosity of info on Memory / CPU type / Caches / Cores / Motherboard etc. virtualization used or not can be determined also with dmidecode / hwinfo and other tools that detect the system hardware this is described thoroughfully in my  previous article Get hardware system info on Linux.

7. Detect virtualziation using virt-what or imvirt scripts

imvirt is a little script to determine several virtualization it is pretty similar to virt-what the RedHat own script for platform identification. Even though virt-what is developed for RHEL it is available on other distros, Fedoda, Debian, Ubuntu, Arch Linux (AUR) just like is imvirt.

installing both of them is with the usual apt-get / yum or on Arch Linux with yay package manager (yay -S virt-what) …

Once run the output it produces for physical Dell / HPE / Fujitsu-Siemens Bare Metal servers would be just empty string.

# virt-what

Or if the system is Virtual Machine, you'll get the type, for example KVM (Kernel-based Virtual Machine) / virtualbox / qemu etc.




It was explained how to do a simple check whether the server works on a physical hardware or on a virtual Host hypervisor. The most basic and classic way is with dmesg. If no access to dmesg is due to restrictions you can try the other methods for systemd enabled OSes with hostnamectl / systemd-detect-virt. Other means if the tools are installed or you have the permissions to install them is with facter / lshw or with virt-what / imvirt scripts.
There definitely perhaps much more other useful tools to grasp hardware and virtualization information but this basics could be useful enough for shell scripting purposes.
If you know other tools, please share.

Linux: Compress website images for better responsiveness with Trimage Graphical tool

March 10th, 2020


If you run a Website or a Blog with images sooner or later you will end up with in looking for better ways to optimize the SEO of the website. I had a small discussion today with a friend of mine Mitko Ivanov who is working as SEO consultant expert,  we had a small discussion on the good practice of optimizing website pictures to reduce the website opening time. Ingeral part of Website responsiveness is the time the Browser needs to fetch all the page Images. Thus if your site is with multiple images, like this blog here, picture comperssion is definitely something that could make miracles in how website visualize for end user and increase rank in Search Engines. The easiest way to compress images of an amateur website of course is to use external picture compression service such as, this requires no knowledge at any computer technology and you can do it easy, but the problem is it shares your image to the remote website used for conversion and I personally think this is not the best idea.
For WordPress website owners of course there is plenty of plugins such as eWWW Image Optimizer that does realtime reduce of size of picture by chunking out the unnecessery bits.
Alternative to especially for people who have a little bit of technical knowledge is is to use some command line tool as optipng together with some kind of shell for loopfor details see my previous article Optimize PNG images by compressing on GNU / Linux, FreeBSD server to Improve Website overall Performance.
But for Many of Webmaster site owners this solution takes too much time as well many people just don't have even basic command line knowledge / are kinda of scared from the console but need to do image compression in a simple GUI way for those the good news are there is  Graphical cross-platform tool for losslessly optimizing PNG and JPG files for web. Trimage.
To use it it even unexperienced non enthusiast could simply roll out a new Virtual Machine on top of some VM Host machine such as Virtual Box and roll out some kind of Linux distribution via a graphical installer which is mega easy well guided and takes 15-20 minutes time.

Once machine is set-up either the Graphical Distribution tool for page management or via apt you can fetch Trimage. It is now existing in most Linux distributions so, to install it on any deb based distribution Debian / Mint / Ubuntu etc. do the usual:

# apt-get install –yes trimage


Once you have it, just move the pictures you want to compress for losslessly optimizing from your website to your Computer with Linux. Trimage GUI on the background will run commands optipng, pngcrush, advpng or jpegoptim, imageoptim and depending on the filetype remove the unnecessery file data that are appended by the program with which image was produced Gimp / Photoshop / Camera software etc. All image files are losslessy compressed on the highest available compression levels, and EXIF and other metadata is removed so you just have to recopy ( upload ) the optimized images back to the website.


That's all folks Enjoy ! 🙂


IBM TSM dsmc console client use for listing configured backups, checking set scheduled backups and backup and restore operations howto

March 6th, 2020


Creating a simple home based backup solution with some shell scripting and rsync is a common use. However as a sysadmin in a middle sized or large corporations most companies use some professional backup service such as IBM Tivoli Storage Manager TSM – recently IBM changed the name of the product to IBM Spectrum.

IBM TSM  is a data protection platform that gives enterprises a single point of control and administration for backup and recovery that is used for Privare Clouds backup and other high end solutions where data criticality is top.
Usually in large companies TSM backup handling is managed by a separate team or teams as managing a large TSM infrastructure is quite a complex task, however my experience as a sysadmin show me that even if you don't have too much of indepth into tsm it is very useful to know how to manage at least basic Incremental backup operations such as view what is set to be backupped, set-up a new directory structure for backup, check the backup schedule configured, check what files are included and which excluded from the backup store etc. 

TSM has multi OS support ans you can use it on most streamline Operating systems Windows / Mac OS X and Linux in this specific article I'll be talking concretely about backing up data with tsm on Linux, tivoli can be theoretically brought up even on FreeBSD machines via the Linuxemu BSD module and the 64-Bit Tivoli Storage Manager RPMs.
Therefore in this small article I'll try to give few useful operations for the novice admin that stumbles on tsm backupped server that needs some small maintenance.

1. Starting up the dsmc command line client


Nomatter the operating system on which you run it to run the client run:

# dsmc



Note that usually dsmc should run as superuser so if you try to run it via a normal non-root user you will get an error message like:


[ user@linux ~]$ dsmc
ANS1398E Initialization functions cannot open one of the Tivoli Storage Manager logs or a related file: /var/tsm/dsmerror.log. errno = 13, Permission denied


Tivoli SM has an extensive help so to get the use basics, type help

tsm> help
1.0 New for IBM Tivoli Storage Manager Version 6.4
2.0 Using commands
  2.1 Start and end a client command session
    2.1.1 Process commands in batch mode
    2.1.2 Process commands in interactive mode
  2.2 Enter client command names, options, and parameters
    2.2.1 Command name
    2.2.2 Options
    2.2.3 Parameters
    2.2.4 File specification syntax
  2.3 Wildcard characters
  2.4 Client commands reference
  2.5 Archive
  2.6 Archive FastBack

Enter 'q' to exit help, 't' to display the table of contents,
press enter or 'd' to scroll down, 'u' to scroll up or
enter a help topic section number, message number, option name,
command name, or command and subcommand:    


2. Listing files listed for backups


A note to make here is as in most corporate products tsm supports command aliases so any command supported described in the help like query, could be
abbreviated with its first letters only, e.g. query filespace tsm cmd can be abbreviated as

tsm> q fi

Commands can be run non-interactive mode also so if you want the output of q fi you can straight use:

tsm> dsmc q fi



This shows the directories and files that are set for backup creation with Tivoli.


3. Getting included and excluded backup set files


It is useful to know what are the exact excluded files from tsm set backup this is done with query inclexcl



4. Querying for backup schedule time

Tivoli as every other backup solution is creating its set to backup files in a certain time slot periods. 
To find out what is the time slot for backup creation use;

tsm> q sched
Schedule Name: WEEKLY_ITSERV
      Description: ITSERV weekly incremental backup
   Schedule Style: Classic
           Action: Incremental
         Priority: 5
   Next Execution: 180 Hours and 35 Minutes
         Duration: 15 Minutes
           Period: 1 Week  
      Day of Week: Wednesday
     Day of Month:
    Week of Month:
           Expire: Never  




5. Check which files have been backed up

If you want to make sure backups are really created it is a good to check, which files from the selected backup files have already
a working backup copy.

This is done with query backup like so:

tsm> q ba /home/*



If you want to query all the current files and directories backed up under a directory and all its subdirectories you need to add the -subdir=yes option as below:


tsm> q ba /home/hipo/projects/* -subdir=yes
Size      Backup Date        Mgmt Class A/I File
   —-      ———–        ———- — —-
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106
  1,024  08-12-2011 02:46:53    STANDARD    A  /home/hipo/projects/hsm41perf
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hsm41test
    512  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test/test2
 12,048  04-12-2011 02:01:29    STANDARD    A  /home/hipo/projects/hsm41perf/tables
 50,326  30-04-2012 01:35:26    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70023
 50,326  27-04-2012 00:28:15    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70099
 11,013  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg/md5check  


  • To make tsm, backup some directories on Linux / AIX other unices:


tsm> incr /  /usr  /usr/local  /home /lib


  • For tsm to backup some standard netware drives, use:


tsm> incr NDS:  USR:  SYS:  APPS:  


  • To backup C:\ D:\ E:\ F:\ if TSM is running on Windows


tsm> incr C:  D:  E: F:  -incrbydate 


  • To back up entire disk volumes irrespective of whether files have changed since the last backup, use the selective command with a wildcard and -subdir=yes as below:


tsm> sel /*  /usr/*   /home/*  -su=yes   ** Unix/Linux


7. Backup selected files from a backup location


It is intuitive to think you can just add some wildcard characters to select what you want
to backup from a selected location but this is not so, if you try something like below
you will get an err.


tsm> incr /home/hipo/projects/*/* -su=yes      
ANS1071E Invalid domain name entered: '/home/hipo/projects/*/*'

The proper way to select a certain folder / file for backup is with:


tsm> sel /home/hipo/projects/*/* -su=yes


8. Restoring tsm data from backup


To restore the config httpd.conf to custom directory use:


tsm> rest /etc/httpd/conf/httpd.conf  /home/hipo/restore/


N!B! that in order for above to work you need to have the '/' trailing slash at the end.

If you want to restore a file under a different name:


tsm> rest /etc/ntpd.conf  /home/hipo/restore/


9. Restoring a whole backupped partition


tsm> rest /home/*  /tmp/restore/ -su=yes


This is using the Tivoli 'Restoring multiple files and directories', and the files to restore '*'
are kept till the one that was recovered (saying this in case if you accidently cancel the restore)


10. Restoring files with back date 


By default the restore function will restore the latest available backupped file, if you need
to recover a specific file, you need the '-inactive' '-pick' options.
The 'pick' interface is interactive so once listed you can select the exact file from the date
you want to restore.

General restore command syntax is:

tsm> restore [source-file] [destination-file]


tsm> rest /home/hipo/projects/*  /tmp/restore/ -su=yes  -inactive -pick

TSM Scrollable PICK Window – Restore

     #    Backup Date/Time        File Size A/I  File
   170. | 12-09-2011 19:57:09        650  B  A   /home/hipo/projects/hsm41test/inclexcl.test
   171. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.ORIG
   172. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.TEST
   173. | 12-09-2011 19:57:09       1.13 KB  A   /home/hipo/projects/hsm41test/md5.out
   174. | 30-04-2012 01:35:26        512  B  A   /home/hipo/projects/hsm42125upg/PMR70023
   175. | 26-04-2012 01:02:08        512  B  I   /home/hipo/projects/hsm42125upg/PMR70023
   176. | 27-04-2012 00:28:15        512  B  A   /home/hipo/projects/hsm42125upg/PMR70099
   177. | 24-04-2012 19:17:34        512  B  I   /home/hipo/projects/hsm42125upg/PMR70099
   178. | 24-04-2012 00:22:56       1.35 KB  A   /home/hipo/projects/hsm42125upg/dsm.opt
   179. | 24-04-2012 00:22:56       4.17 KB  A   /home/hipo/projects/hsm42125upg/dsm.sys
   180. | 24-04-2012 00:22:56       1.13 KB  A   /home/hipo/projects/hsm42125upg/dsmmigfstab
   181. | 24-04-2012 00:22:56       7.30 KB  A   /home/hipo/projects/hsm42125upg/filesystems
   182. | 24-04-2012 00:22:56       1.25 KB  A   /home/hipo/projects/hsm42125upg/inclexcl
   183. | 24-04-2012 00:22:56        198  B  A   /home/hipo/projects/hsm42125upg/inclexcl.dce
   184. | 24-04-2012 00:22:56        291  B  A   /home/hipo/projects/hsm42125upg/inclexcl.ox_sys
   185. | 24-04-2012 00:22:56        650  B  A   /home/hipo/projects/hsm42125upg/inclexcl.test
   186. | 24-04-2012 00:22:56        670  B  A   /home/hipo/projects/hsm42125upg/inetd.conf
   187. | 24-04-2012 00:22:56       2.71 KB  A   /home/hipo/projects/hsm42125upg/inittab
   188. | 24-04-2012 00:22:56       1.00 KB  A   /home/hipo/projects/hsm42125upg/md5check
   189. | 24-04-2012 00:22:56      79.23 KB  A   /home/hipo/projects/hsm42125upg/mkreport.020423.out
   190. | 24-04-2012 00:22:56       4.27 KB  A   /home/hipo/projects/hsm42125upg/ssamap.020423.out
   191. | 26-04-2012 01:02:08      12.78 MB  A   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
   192. | 25-04-2012 16:33:36      12.78 MB  I   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
<U>=Up  <D>=Down  <T>=Top  <B>=Bottom  <R#>=Right  <L#>=Left
<G#>=Goto Line #  <#>=Toggle Entry  <+>=Select All  <->=Deselect All
<#:#+>=Select A Range <#:#->=Deselect A Range  <O>=Ok  <C>=Cancel

To navigate in pick interface you can select individual files to restore via the number seen leftside.
To scroll up / down use 'U' and 'D' as described in the legenda.


11. Restoring your data to another machine


In certain circumstances, it may be necessary to restore some, or all, of your data onto a machine other than the original from which it was backed up.

In ideal case the machine platform should be identical to that of the original machine. Where this is not possible or practical please note that restores are only possible for partition types that the operating system supports. Thus a restore of an NTFS partition to a Windows 9x machine with just FAT support may succeed but the file permissions will be lost.
TSM does not work fine with cross-platform backup / restore, so better do not try cross-platform restores.
 Trying to restore files onto a Windows machine that have previously been backed up with a non-Windows one. TSM created backups on Windows sent by other OS platforms can cause  backups to become inaccessible from the host system.

To restore your data to another machine you will need the TSM software installed on the target machine. Entries in Tivoli configuration files dsm.sys and/or dsm.opt need to be edited if the node that you are restoring from does not reside on the same server. Please see our help page section on TSM configuration files for their locations for your operating system. 

To access files from another machine you should then start the TSM client as below:


# dsmc -virtualnodename=RESTORE.MACHINE      

You will then be prompted for the TSM password for this machine.


You will probably want to restore to a different destination to the original files to prevent overwriting files on the local machine, as below:


  • Restore of D:\ Drive to D:\Restore ** Windows 


tsm> rest D:\*   D:\RESTORE\    -su=yes 


  • Restore user /home/* to /scratch on ** Mac, Unix/Linux


tsm> rest /home/* /scratch/     -su=yes  


  • Restoring Tivoli data on old netware


tsm> rest SOURCE-SERVER\USR:*  USR:restore/   -su=yes  ** Netware


12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

The easiest way is to check the produced dschmed.log if everything is okay there should be records in the log that Tivoli backup was scheduled in a some hours time
A normally produced backup scheduled in log should look something like:


14-03-2020 23:03:04 — SCHEDULEREC STATUS BEGIN
14-03-2020 23:03:04 Total number of objects inspected:   91,497
14-03-2020 23:03:04 Total number of objects backed up:      113
14-03-2020 23:03:04 Total number of objects updated:          0
14-03-2020 23:03:04 Total number of objects rebound:          0
14-03-2020 23:03:04 Total number of objects deleted:          0
14-03-2020 23:03:04 Total number of objects expired:         53
14-03-2020 23:03:04 Total number of objects failed:           6
14-03-2020 23:03:04 Total number of bytes transferred:    19.38 MB
14-03-2020 23:03:04 Data transfer time:                    1.54 sec
14-03-2020 23:03:04 Network data transfer rate:        12,821.52 KB/sec
14-03-2020 23:03:04 Aggregate data transfer rate:        114.39 KB/sec
14-03-2020 23:03:04 Objects compressed by:                    0%
14-03-2020 23:03:04 Elapsed processing time:           00:02:53
14-03-2020 23:03:04 — SCHEDULEREC STATUS END
14-03-2020 23:03:04 — SCHEDULEREC OBJECT END WEEKLY_23_00 14-12-2010 23:00:00
14-03-2020 23:03:04 Scheduled event 'WEEKLY_23_00' completed successfully.
14-03-2020 23:03:04 Sending results for scheduled event 'WEEKLY_23_00'.
14-03-2020 23:03:04 Results sent to server for scheduled event 'WEEKLY_23_00'.


in case of errors you should check dsmerror.log


In this article I've briefly evaluated some basics of IBM Commercial Tivoli Storage Manager (TSM) to be able to  list backups, check backup schedules and how to the files set to be
excluded from a backup location and most importantly how to check that data backed up data is in a good shape and accessible.
It was explained how backups can be restored on a local and remote machine as well as how to  append new files to be set for backup on next incremental scheduled backup.
It was shown how the pick interactive cli interface could be used to restore files at a certain data back in time as well as how full partitions can be restored and how some
certain file could be retrieved from the TSM data copy.

Improve DNS lookup domain resolve speed on Linux / UNIX servers through /etc/resolv.conf timeout, attempts, rorate options

February 27th, 2020

If you're an performance optimization freak and you want to optimize your Linux servers to perform better in terms of DNS resolve slowness because of failing DNS resolve queries due to Domain Name Server request overload or due to Denial of Service attack towards it. It might be interesting to mention about some little known functionalities of /etc/resolv.conf described in the manual page.

The defined nameservers under /etc/resolv.conf are queried one by one waiting for responce of the sent DNS resolve request if it is not replied from the first one for some time, the 2nd one is queried until a responce is received by any of the defined nameserver IPs

A default /etc/resolv.conf on a new Linux server install looks something like this:


However one thing is that defined if NS1 dies out due to anything, it takes timeout time until the second or 3rd working one takes over to resolve the query.
This is controlled by the timeout value.

Below is description from man page

sets the amount of time the resolver will wait for a
response from a remote name server before retrying the
query via a different name server.  Measured in
seconds, the default is RES_TIMEOUT (currently 5, see
<resolv.h>).  The value for this option is silently
capped to 30.


  • In other words Timeout value is time to resolving IP address from hostname through DNS server,timeout option is to reduce hostname lookup time

As you see from manual default is 5 seconds which is quite high, thus reducing the value to 3 secs or even 1 seconds is a good sysadmin practice IMHO.

Another value that could be tuned in /etc/resolv.conf is attempts value below is what the manual says about it: 

                     Sets the number of times the resolver will send a query to its name servers before giving up and returning an error to the calling application.  The default is RES_DFLRETRY (cur‐
                     rently 2, see <resolv.h>).  The value for this option is silently capped to 5.



  • This means default behaviour on a failing DNS query resolve is to try to resend the DNS resolve request to the failing nameserver 5 more times, that is quite high thus it is a good practice from my experience to reduce it to something as 2 or 1

Another very useful resolv.conf value is rotate
The default behavior of how DNS outgoing Domain requests are handled is to use only the primary defined DNS, instead if you need to do a load balancing in a round-robin manner add to conf rotate option.

The final /etc/resolv.conf optimized would look like so:


linux# cat /etc/resolv.conf

options ndots:1
options timeout:1
options attempts:1
options rotate

The search opt. placement is also important to be placed in the right location in the file. The correct placement is after the nameservers defined, I have to say in older Linux distributions the correct placement of search option was to be on top of resolv.conf.

Note that this configuration is good and fits not only Linux but also is a good DNS lookup optimization speed on other UNIX derivatives such as FreeBSD / NetBSD as well as other Proprietary OS UNIX machines running IBM AIX etc.

On Linux it is also possible to place the options given in one single line like so, below is the config I have on my running Lenovo server:


options timeout:2 attempts:1 rotate


When is /etc/hosts record venerated and when is /etc/resolv.conf DNS defined queried for a defined DNS host?


One important thing to know when dealing with /etc/resolv.conf  is what happens if a Name domain is defined in both /etc/hosts and /etc/resolv.conf.
For example you have a domain record in /etc/hosts to a certain domain
but the DNS nameserver in Google has a record to an IP that is the real IP


# dig @

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> @
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54656
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 512
;                  IN      A

;; ANSWER SECTION:           3599    IN      A

;; Query time: 40 msec
;; WHEN: чт фев 27 18:04:23 EET 2020
;; MSG SIZE  rcvd: 57


  • Which of the 2 different IPs will the applications installed on the server such as Apache / Squid / MySQL / tinyproxy for their DNS resolve operations?


Now it is time to say few words about /etc/nsswitch.conf (The Nameserver switching configuration file). This file defines the DNS resolve file used order in which the Operationg System does IP to domain translation and backwards.

# grep -i hosts: /etc/nsswitch.conf

hosts:          files dns myhostname

As you can see first the local defined in files like /etc/hosts record is venerated when resolving, then it is the externally configured DNS resolver IPs from /etc/resolv.conf.

nsswitch.conf  is used also for defining where the OS will look up for user / passwd (e.g. login credentials) on login, on systems which are having an LDAP authentication via the sssd (system security services daemon) via definitions like:


passwd:     files sss
shadow:     files sss
group:      files sss

E.g. the user login will be first try to read from local /etc/passwd , /etc/shadow , /etc/groups and if no matched record is found then the LDAP service the sssd is queried.

Fix Blocked Unresponsive keyboard keys on Windows 7 / 10

February 25th, 2020



The Problem

If you're still using Windows 7 Operating system in your company due to some weird security concern policies and your company gets custom end user updates from Microsoft due to a special EULA agreement and you are new to using Windows E.g. have for many users used for your daily work Linux and Mac OS you might hit a strange issue wtih many of the keyboard keys strangely being locked with some of the keys such as Num Lock, Escape tab working where the Alphabet keys then don't panic. This is not a Windows bug its a feature (as usual) 🙂

Reason behind Blocked Unresponsive Keyboard


First logical thought I had is maybe my Logitech K120 Membrane 17 EURO cheap keyboard externally attached keyboard broke up thus I've tried to connect another LOGIC keyboard I had at hand just to assure myself the problem with partial keys on kbd reacting was present with the other Working keyboard as well.
This was an indicator that either the custom installed Windows by the company Helpdesk Office with the preassumed common additional features for corporations such as Keyloggers on this Laptop has messed up somohow the Windows service that is managing the keyboard or some kind of mechanical error or electronic circuit on the laptop embedded keyboard has occured or the KBD DLL Loaded driver damaged
I have to say here a colleague of mine was having a weird keyboard problems back in the day when I was still working in Project Services as a Web and Middleware in Hewlett Packard, where misteriously a character was added to his typed content just like a key on his keyboard has stuck and he experienced this issue for quite some time, he opened the keyboard to physically check whether all is okay and even checked the keyboard electricity whether levels on each of the keys and he couldn't find anything, and after running Malware Bytes anti-malware and a couple of other anti-malware programs which found his computer was infected with Malware and issue resolved.


Hard Fix – Reboot ..


As as a solution most times so far I've restarted the Windows which was reloading the Windows kernel / DLL libraries etc.

However just hit to this Windows accessibility feature once again today and since this is not the first time I end up with unworking keyboard (perhaps due to my often) furious fast typing – where I press sometimes multiple keys in parallel as a typing error then you trigger the Windows Disabled accessibiltiy Windows Feature, which as I thought makes the PC only usable for Mouse but unusuable for providing any meaningful keyboard input.

This problem I've faced already multiple times and usually the work around was the good known Windows User recipee phrase "Restart and It will get fixed", this time I was pissed off and didn't wanted to loose another 5 minutes in Restarting Reconnecting to the Company's Cisco Secure VPN reopening all my used files Notepad++ / Outlook / Browsers etc plus I was already part of online Lync (Skype) Meeting in which Colleague was Sharing his remote Desktop checking some important stuff about Zabbix Monitored AIX machine, hence didn't wanted to restart but still wanted to use my computer and type some stuff to send Email and do a simple googling.

Temporary work around to complete work with Virtual Keyboard

Hence as a temporary work around, I've used the Windows Virtual Keyboard, I've mentioned it in the earlier  blog post – How to run Virtual Keyboard in Windows XP / Vista article
To do so I'verun by typing osk command in cmd.exe command Prompt:

Either Search for osk.exe from Start menu


or run via command line via

Windows Button (on the Keyboard) + R and run

cmd.exe -> osk




Solution Without PC Restart

After a bit of thought and Googling I've found the fix  here

From the Start -> Control Panel from here I had to go to Accessibility Options.
Select Ease of Access Center.


Select the keyboard settings and
Ensure the following options are unchecked: Turn on Sticky Keys, Turn on Toggle Keys and Turn on Filter Keys.


I've found in the Turn on Toggle Keys tick present (e.g. service was enabled) – hence  after unticking it and 

Press Apply and OK, keyboard restored its usual functions.
Now all left was to  Enjoy as your keyboard was back usable and I could conitnue my Citrix sessions and SSH console Superputty terminals  and complete my started to write E-mail
without loosing time meanlessly for reboot.


Windows 10 Disable the Filter Keys option


This feature makes your keyboard ignore brief or repeated keystrokes, which might have led to your WinKey issue in Windows 10. To disable filter keys, use the instructions below:

1. Right-click on your Start menu icon.
2. Select Settings from the menu.
3. Navigate to Ease of Access and click on it.
4. Go to the left pane and click Keyboard.
5. Locate the Filter Keys feature.
6. Toggle it off.
7. Check if this manoeuvre has resolved your issue.

Closing Notes 

Of course this might be not always the fix, as sometimes it could be that the Winblows just blows your keyboard buffer due to some buggy application or a bug, but in most of the times that should solve it 🙂
If it didn't go through and debug all the other possible reasons, check whether you have a faulty keyboard cable (if you're still on a non-bluetooth Wired Keyboard), unplug and plug the keyboard again,
scan the computer for spyware and malware, rethink what really happened or what have you done until the problem occured and whether blocked keyboard is triggered by your user action or was triggered
by some third party software anti-virus stuff that did it as an attempt to prevent keylog sniffer / Virus or other weird stuff.

How to check if shared library is loaded in AIX OS – Fix missing

February 20th, 2020


I've had to find out whether an externally Linux library is installed  on AIX system and whether something is not using it.
The returned errors was like so:


# gpg –export -a

Could not load program gpg:
Dependent module /opt/custom/lib/libreadline.a( could not be loaded.
Member is not found in archive

After a bit of investigation, I found that gpg was failing cause it linked to older version of, the workaround was to just substitute the newer version of over the original installed one.

Thus I had a plan to first find out whether this libreadline.a is loaded and recognized by AIX UNIX first and second find out whether some of the running processes is not using that library.
I've come across this interesting IBM official documenation that describes pretty good insights on how to determine whether a shared library  is currently loaded on the system. which mentions the genkld command that is doing
exactly what I needed.

In short:
genkld – creates a list that is printed to the console that shows all loaded shared libraries


Next I used lsof (list open files) command to check whether there is in real time opened libraries by any of the running programs on the system.

After not finding anything and was sure the library is neither loaded as a system library in AIX nor it is used by any of the currently running AIX processes, I was sure I could proceed to safely overwrite libreadline.a ( with libreadline.a with (

The result of that is again a normally running gpg as ldd command shows the binary is again normally linked to its dependend system libraries.

aix# ldd /usr/bin/gpg
/usr/bin/gpg needs:



# gpg –version
gpg (GnuPG) 1.4.22
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


Home: ~/.gnupg
Supported algorithms:
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2



How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages

February 19th, 2020

haproxy  logging can be managed in different form the most straight forward way is to directly use /dev/log either you can configure it to use some log management service as syslog or rsyslogd for that.

If you don't use rsyslog yet to install it: 

# apt install -y rsyslog

Then to activate logging via rsyslogd we can should add either to /etc/rsyslogd.conf or create a separte file and include it via /etc/rsyslogd.conf with following content:

Enable haproxy logging from rsyslogd

Log haproxy messages to separate log file you can use some of the usual syslog local0 to local7 locally used descriptors inside the conf (be aware that if you try to use some wrong value like local8, local9 as a logging facility you will get with empty haproxy.log, even though the permissions of /var/log/haproxy.log are readable and owned by haproxy user.

When logging to a local Syslog service, writing to a UNIX socket can be faster than targeting the TCP loopback address. Generally, on Linux systems, a UNIX socket listening for Syslog messages is available at /dev/log because this is where the syslog() function of the GNU C library is sending messages by default. To address UNIX socket in haproxy.cfg use:

log /dev/log local2 

If you want to log into separate log each of multiple running haproxy instances with different haproxy*.cfg add to /etc/rsyslog.conf lines like:

local2.* -/var/log/haproxylog2.log
local3.* -/var/log/haproxylog3.log

One important note to make here is since rsyslogd is used for haproxy logging you need to have enabled in rsyslogd imudp and have a UDP port listener on the machine.

E.g. somewhere in rsyslog.conf or via rsyslog include file from /etc/rsyslog.d/*.conf needs to have defined following lines:

$ModLoad imudp
$UDPServerRun 514

I prefer to use external /etc/rsyslog.d/20-haproxy.conf include file that is loaded and enabled rsyslogd via /etc/rsyslog.conf:

# vim /etc/rsyslog.d/20-haproxy.conf

$ModLoad imudp
$UDPServerRun 514​
local2.* -/var/log/haproxy2.log

It is also possible to produce different haproxy log output based on the severiy to differentiate between important and less important messages, to do so you'll need to rsyslog.conf something like:

# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log


Prevent Haproxy duplicate messages to appear in /var/log/messages

If you use local2 and some default rsyslog configuration then you will end up with the messages coming from haproxy towards local2 facility producing doubled simultaneous records to both your pre-defined /var/log/haproxy.log and /var/log/messages on Proxy servers that receive few thousands of simultanous connections per second.
This is a problem since doubling the log will produce too much data and on systems with smaller /var/ partition you will quickly run out of space + this haproxy requests logging to /var/log/messages makes the file quite unreadable for normal system events which are so important to track clearly what is happening on the server daily.

To prevent the haproxy duplicate messages you need to define somewhere in rsyslogd usually /etc/rsyslog.conf local2.none near line of facilities configured to log to file:

*.info;mail.none;authpriv.none;cron.none;local2.none     /var/log/messages

This configuration should work but is more rarely used as most people prefer to have haproxy log being written not directly to /dev/log which is used by other services such as syslogd / rsyslogd.

To use /dev/log to output logs from haproxy configuration in global section use config like:

        log /dev/log local2 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy

The log global directive basically says, use the log line that was set in the global section for whole config till end of file. Putting a log global directive into the defaults section is equivalent to putting it into all of the subsequent proxy sections.

Using global logging rules is the most common HAProxy setup, but you can put them directly into a frontend section instead. It can be useful to have a different logging configuration as a one-off. For example, you might want to point to a different target Syslog server, use a different logging facility, or capture different severity levels depending on the use case of the backend application. 

Insetad of using /dev/log interface that is on many distributions heavily used by systemd to store / manage and distribute logs,  many haproxy server sysadmins nowdays prefer to use rsyslogd as a default logging facility that will manage haproxy logs.
Admins prefer to use some kind of mediator service to manage log writting such as rsyslogd or syslog, the reason behind might vary but perhaps most important reason is  by using rsyslogd it is possible to write logs simultaneously locally on disk and also forward logs  to a remote Logging server  running rsyslogd service.

Logging is defined in /etc/haproxy/haproxy.cfg or the respective configuration through global section but could be also configured to do a separate logging based on each of the defined Frontend Backends or default section. 
A sample exceprt from this section looks something like:

# Global settings
    log local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/
    maxconn     4000
    user        haproxy
    group       haproxy

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    mode                    tcp
    log                     global
    option                  tcplog
    #option                  dontlognull
    #option http-server-close
    #option forwardfor       except
    option                  redispatch
    retries                 7
    #timeout http-request    10s
    timeout queue           10m
    timeout connect         30s
    timeout client          20m
    timeout server          10m
    #timeout http-keep-alive 10s
    timeout check           30s
    maxconn                 3000



# HAProxy Monitoring Config
listen stats                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard


# frontend which proxys to the backends
frontend ft_DKV_PROD_WLPFO
    mode tcp
    option tcplog
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
    default_backend Default_Bakend_Name

# round robin balancing between the various backends
backend bk_DKV_PROD_WLPFO
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
    stick on src
    # (5) Server List
    # (5.1) Backend
    server Backend_Server1 check port 18088
    server Backend_Server2 check port 18088 backup

The log directive in above config instructs HAProxy to send logs to the Syslog server listening at Messages are sent with facility local2, which is one of the standard, user-defined Syslog facilities. It’s also the facility that our rsyslog configuration is expecting. You can add more than one log statement to send output to multiple Syslog servers.

Once rsyslog and haproxy logging is configured as a minumum you need to restart rsyslog (assuming that haproxy config is already properly loaded):

# systemctl restart rsyslogd.service

To make sure rsyslog reloaded successfully:

systemctl status rsyslogd.service

Restarting HAproxy

If the rsyslogd logging to port 514 was recently added a HAProxy restart should also be run, you can do it with:

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/ -sf $(cat /var/run/

Or to restart use systemctl script (if haproxy is not used in a cluster with corosync / heartbeat).

# systemctl restart haproxy.service

You can control how much information is logged by adding a Syslog level by

    log local2 info

The accepted values are the standard syslog security level severity:

Value Severity Keyword Deprecated keywords Description Condition
0 Emergency emerg panic System is unusable A panic condition.
1 Alert alert   Action must be taken immediately A condition that should be corrected immediately, such as a corrupted system database.
2 Critical crit   Critical conditions Hard device errors.
3 Error err error Error conditions  
4 Warning warning warn Warning conditions  
5 Notice notice   Normal but significant conditions Conditions that are not error conditions, but that may require special handling.
6 Informational info   Informational messages  
7 Debug debug   Debug-level messages Messages that contain information normally of use only when debugging a program.


Logging only errors / timeouts / retries and errors is done with option:

Note that if the rsyslog is configured to listen on different port for some weird reason you should not forget to set the proper listen port, e.g.:

  log local2 info

option dontlog-normal

in defaults or frontend section.

You most likely want to enable this only during certain times, such as when performing benchmarking tests.

(or log-format-sd for structured-data syslog) directive in your defaults or frontend

Haproxy Logging shortly explained

The type of logging you’ll see is determined by the proxy mode that you set within HAProxy. HAProxy can operate either as a Layer 4 (TCP) proxy or as Layer 7 (HTTP) proxy. TCP mode is the default. In this mode, a full-duplex connection is established between clients and servers, and no layer 7 examination will be performed. When in TCP mode, which is set by adding mode tcp, you should also add option tcplog. With this option, the log format defaults to a structure that provides useful information like Layer 4 connection details, timers, byte count and so on.

Below is example of configured logging with some explanations:

Log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

Example of Log-Format configuration as shown above outputted of haproxy config:

Log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"


To understand meaning of this abbreviations you'll have to closely read  haproxy-log-format.txt. More in depth info is to be found in HTTP Log format documentation


Logging HTTP request headers

HTTP request header can be logged via:

 http-request capture

frontend website
    bind :80
    http-request capture req.hdr(Host) len 10
    http-request capture req.hdr(User-Agent) len 100
    default_backend webservers

The log will show headers between curly braces and separated by pipe symbols. Here you can see the Host and User-Agent headers for a request: [20/Dec/2018:22:20:00.899] website~ webservers/server1 0/0/1/0/1 200 462 – – —- 1/1/0/0/0 0/0 {|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.80 } "GET / HTTP/1.1"


Haproxy Stats Monitoring Web interface

Haproxy is having a simplistic stats interface which if enabled produces some general useful information like in above screenshot, through which
you can get a very basic in browser statistics and track potential issues with the proxied traffic for all configured backends / frontends incoming outgoing
network packets configured nodes
 experienced downtimes etc.


The basic configuration to make the stats interface accessible would be like pointed in above config for example to enable network listener on address

with hproxyuser / password config would be:

# HAProxy Monitoring Config
listen stats                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard



Sessions states and disconnect errors on new application setup

Both TCP and HTTP logs include a termination state code that tells you the way in which the TCP or HTTP session ended. It’s a two-character code. The first character reports the first event that caused the session to terminate, while the second reports the TCP or HTTP session state when it was closed.

Here are some essential termination codes to track in for in the log:

Here are some termination code examples most commonly to see on TCP connection establishment errors:

Two-character code    Meaning
—    Normal termination on both sides.
cD    The client did not send nor acknowledge any data and eventually timeout client expired.
SC    The server explicitly refused the TCP connection.
PC    The proxy refused to establish a connection to the server because the process’ socket limit was reached while attempting to connect.

To get all non-properly exited codes the easiest way is to just grep for anything that is different from a termination code –, like that:

tail -f /var/log/haproxy.log | grep -v ' — '

This should output in real time every TCP connection that is exiting improperly.

There’s a wide variety of reasons a connection may have been closed. Detailed information about all possible termination codes can be found in the HAProxy documentation.
To get better understanding a very useful reading to haproxy Debug errors with  is in haproxy-logging.txt in that small file are collected all the cryptic error messages codes you might find in your logs when you're first time configuring the Haproxy frontend / backend and the backend application behind.

Another useful analyze tool which can be used to analyze Layer 7 HTTP traffic is halog for more on it just google around.