Great Lent third Sunday of the Veneration of the Holy Christ Cross


March 23rd, 2020

http://bg-patriarshia.bg/img/ftp/Krustopoklonna.
Today is the third Sunday of the Holy Christ Cross Holy Feast (Кръстовден / The day of the Cross).
Each of the Great Lent time period has its own Sunday dedicated to very important events or saints.
This one has been set over the centuries by the Holy Eastern Orthodox Church to accent the Great unbeatable power of the Holy Cross. Which is the greatest power through which each of us Christians is victorious over sin, flesh, passions, temptations and everything visible and invisible.
The cross provides every person and us as socieity with 'The power of God and Wisdowm of God' to those who venerate it for those being saved as pointed in Holy Bible in scriptures (1 Corintians 1:24).
This Sunday was set to help people recharge their spiritual powers in feat of Lentin hardship weakness and temptations. It is to remind the faithful Christians of the redemption received through Christ's sufferings for our sins. But besides the joy of the Cross it is to remind us another important fact – "He who does not take up his cross and follow me is not worthy of me" (Mt 10:38).

The hymnology (the Church songs) accent over the Holy Liturgy morning service is the victory and joy of the cross, not the suffering. The Church fathers equate the life-giving cross with the tree of life and plant it in the middle of the Lenten pilgrimage. It was the tree that was planted in Paradise; it is to remind the faithful of both Adam's bliss and how he was deprived from it.

The Church teaches that it is Christ's cross that saves. One cannot take up his own cross and follow Christ unless one has Christ's cross which he took up to save mankind. Partaking of this tree, one will no longer die, but will be kept alive.

 

History of the Veneration of Holy Cross
 

The commemoration and ceremonies of the Third Sunday of Lent are closely parallel to the feasts of the Veneration of the Cross (September 14) and the Procession of the Cross (August 1). Not only does the Sunday of the Holy Cross prepare us for commemoration of the Crucifixion, but it also reminds us that the whole of Lent is a period when we are crucified with Christ.

As we have “crucified the flesh with its passions and desires” (Galatians 5:24), and will have mortified ourselves during these forty days of the Fast, the precious and life-giving Cross is now placed before us to refresh our souls and encourage us who may be filled with a sense of bitterness, resentment, and depression. The Cross reminds us of the Passion of our Lord, and by presenting to us His example, it encourages us to follow Him in struggle and sacrifice, being refreshed, assured, and comforted. In other words, we must experience what the Lord experienced during His Passion – being humiliated in a shameful manner. The Cross teaches us that through pain and suffering we shall see the fulfillment of our hopes: the heavenly inheritance and eternal glory.

As they who walk on a long and hard way and are bowed down by fatigue find great relief and strengthening under the cool shade of a leafy tree, so do we find comfort, refreshment, and rejuvenation under the Life-giving Cross, which our Fathers “planted” on this Sunday. Thus, we are fortified and enabled to continue our Lenten journey with a light step, rested and encouraged.

Elevation-of-the-Holy-Cross-feast

Hymns

As at the feast of the Elevation of the Holy Cross on September 14, the Trisagion is replaced by the hymn:

Before Thy Cross we bow down in worship, O Master, and Thy holy Resurrection we glorify. (Thrice)
Glory to the Father, and to the Son, and to the Holy Spirit, both now and ever, and unto the ages of ages. Amen.
And Thy holy Resurrection we glorify.
Before Thy Cross we bow down in worship, O Master, and Thy holy Resurrection we glorify.

Тропар Църковнославянски (Българска Православна Църква) / Troparion Old Bulgarian (Church Slavonic) Bulgarian Orthodox Church

Спаси, Господи, люди Твоя и благослови достояние Твое, победи православному болгарскому народу на сопротивния даруя, и Твое сохраняя Крестом Твоим жительство.

Troparion (Tone 4) [1]

O Lord, save Your people,
And bless Your inheritance.
Grant victories to the Orthodox Christians,
over their adversaries.
And by virtue of Your Cross
Preserve Your habitation!
Kontakion (Tone 7) [2]

Now the flaming sword no longer guards the gates of Eden;
It has mysteriously been quenched by the wood of the Cross!
The sting of death and the victory of hell have been vanquished;
For You, O my Savior, have come and cried to those in hell:
"Enter again into paradise."

Elevation-of-the-Holy-Cross-of-Christ

It is worthy to remind my dear readers for the Cross, because nowadays everyone is scared from the Coronavirus COVID-19 mysterious virus who is plaguing the death and who makes people who have no faith to tremble besides a simple flu, but do not tremble in front of the Cross of Christ and the near coming Great Judgement day in which each of us will have to give account for his good and bad deeds …
Also through the centuries the Cross has been a reason for many miracles. In this scareful days where constitutions of many countries has been breached and a military regimes are pushed into our countres.

Lets pray and bow the holy cross of Christ to protect our lands, Countries families and loved grand parents, mothers, fathers, children, friends and every living creature from the evil that is lurking the earth!

 

Check if server is Physical Bare Metal or a Virtual Machine and its type


March 17th, 2020

check-if-linux-operating-system-is-running-on-physical-bare-metal-or-virtual-machine

In modern times the IT employee system administrator / system engineer / security engineer or a developer who has to develop and test code remotely on UNIX hosts, we have to login to multiple of different servers located in separate data centers around the world situated in Hybrid Operating system environments running multitude of different Linux OSes. Often especially for us sysadmins it is important to know whether the remote machine we have SSHed to is physical server (Bare Metal) or a virtual machines running on top of different kind of Hypervisor node OpenXen / Virtualbox / Virtuosso  / VMWare etc.
 

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?
 

1. Using the dmesg kernel log utility


The good old dmesg that is used to examine and control the kernel ring buffer detects plenty of useful information which gives you the info whether a server is Virtual or Bare Metal. It is present and accessible on every Linux server out there, thus using it is the best and simplest way to determine the OS system node type.

To grep whether a machine is Virtual and the Hypervisor type use:

 

nginx:~# dmesg | grep "Hypervisor detected"
[0.000000] Hypervisor detected: KVM


As you see above OS installed is using the KVM Virtualization technology.

An empty output of this command means the Remote OS is installed on a physical computer.

 

2. Detecting the OS platform the systemd way


Systemd along with the multiple over-complication of things that nearly all sysadmins (including me hate) so much introduced something useful in the fact of hostnamectl command
that could give you the info about the OS chassis platform.

 

root@pcfreak:~# hostnamectl status
 
 Static hostname: pcfreak
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 02425d67037b8e67cd98bd2800002671
           Boot ID: 34a83b9a79c346168082f7605c2f557c
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-amd64
      Architecture: x86-64

 

 

Below is output of a VM running on a Oracle Virtualbox HV.

 

linux:~# hostnamectl status
Static hostname: ubuntuserver
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 2befe86cf8887ca098f509e457554beb
 Boot ID: 8021c02d65dc46b1885afb25fddcf18c
 Virtualization: oracle
 Operating System: Ubuntu 16.04.1 LTS
 Kernel: Linux 4.4.0-78-generic
 Architecture: x86-64

 

3. Detect concrete container virtualization with systemd-detect-virt 


Another Bare Metal or VM identify tool that was introducted some time ago by freedesktop project is systemd-detect-virt (usually command is part of systemd package).
It is useful to detect the exact virtualization on a systemd running OS systemd-detect-virt is capable to detect many type of Virtualization type that are rare like: IBM zvm S390 Z/VM, bochs, bhyve (a FreeBSD hypervisor), Mac OS's parallels, lxc (linux containers), docker containers, podman etc.

The output from the command is either none (if no virtualization is present or the VM Hypervisor Host type):

 

server:~# systemd-detect-virt
none

 

quake:~# systemd-detect-virt
oracle

 

4. Install and use facter to report per node facts

 

debian:~# apt-cache show facter|grep -i desc -A2
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and
 reports per-node facts, which are collected by the Puppet agent and are made

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter
Tag: devel::lang:ruby, devel::library, implemented-in::ruby,
root@jeremiah:/home/hipo# apt-cache show facter|grep -i desc -A1
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter

 


– Install facter on Debian / Ubuntu / deb based Linux

 

# apt install facter –yes


– Install facter on RedHat / CentOS RPM based distros

# yum install epel-release

 

# yum install facter


– Install facter on OpenSuSE / SLES

# zypper install facter


Once installed on the system to find out whether the remote Operating System is Virtual:

# facter 2> /dev/null | grep virtual
is_virtual => false
virtual => physical


If the machine is a virtual machine you will get some different output like:

# facter 2> /dev/null | grep virtual
is_virtual => true
virtual => kvm


If you're lazy to grep you can use it with argument.

# facter virtual
physical

 

6. Use lshw and dmidecode (list hardware configuration tool)


If you don't have the permissions to install facter on the system and you can see whether lshw (list hardware command) is not already present on remote host.

# lshw -class system  
storage-host                  
    description: Computer
    width: 64 bits
    capabilities: smbios-2.7 vsyscall32

If the system is virtual you'll get an output similar to:

# lshw -class system  
debianserver 
 description: Computer
 product: VirtualBox
 vendor: innotek GmbH
 version: 1.2
 serial: 0
 width: 64 bits
 capabilities: smbios-2.5 dmi-2.5 vsyscall32
 configuration: family=Virtual Machine uuid=78B58916-4074-42E2-860F-7CAF39F5E6F5


Of course as it provides a verbosity of info on Memory / CPU type / Caches / Cores / Motherboard etc. virtualization used or not can be determined also with dmidecode / hwinfo and other tools that detect the system hardware this is described thoroughfully in my  previous article Get hardware system info on Linux.


7. Detect virtualziation using virt-what or imvirt scripts


imvirt is a little script to determine several virtualization it is pretty similar to virt-what the RedHat own script for platform identification. Even though virt-what is developed for RHEL it is available on other distros, Fedoda, Debian, Ubuntu, Arch Linux (AUR) just like is imvirt.

installing both of them is with the usual apt-get / yum or on Arch Linux with yay package manager (yay -S virt-what) …

Once run the output it produces for physical Dell / HPE / Fujitsu-Siemens Bare Metal servers would be just empty string.

# virt-what
#

Or if the system is Virtual Machine, you'll get the type, for example KVM (Kernel-based Virtual Machine) / virtualbox / qemu etc.

#imvirt
Physical

 

Conclusion


It was explained how to do a simple check whether the server works on a physical hardware or on a virtual Host hypervisor. The most basic and classic way is with dmesg. If no access to dmesg is due to restrictions you can try the other methods for systemd enabled OSes with hostnamectl / systemd-detect-virt. Other means if the tools are installed or you have the permissions to install them is with facter / lshw or with virt-what / imvirt scripts.
There definitely perhaps much more other useful tools to grasp hardware and virtualization information but this basics could be useful enough for shell scripting purposes.
If you know other tools, please share.
 

Linux: Compress website images for better responsiveness with Trimage Graphical tool


March 10th, 2020

trimage-compress-reduce-lossless-encoding-of-pictures-for-seo-linux-screenshot0

If you run a Website or a Blog with images sooner or later you will end up with in looking for better ways to optimize the SEO of the website. I had a small discussion today with a friend of mine Mitko Ivanov who is working as SEO consultant expert,  we had a small discussion on the good practice of optimizing website pictures to reduce the website opening time. Ingeral part of Website responsiveness is the time the Browser needs to fetch all the page Images. Thus if your site is with multiple images, like this blog here, picture comperssion is definitely something that could make miracles in how website visualize for end user and increase rank in Search Engines. The easiest way to compress images of an amateur website of course is to use external picture compression service such as tinypng.com, this requires no knowledge at any computer technology and you can do it easy, but the problem is it shares your image to the remote website used for conversion and I personally think this is not the best idea.
For WordPress website owners of course there is plenty of plugins such as eWWW Image Optimizer that does realtime reduce of size of picture by chunking out the unnecessery bits.
Alternative to especially for people who have a little bit of technical knowledge is is to use some command line tool as optipng together with some kind of shell for loopfor details see my previous article Optimize PNG images by compressing on GNU / Linux, FreeBSD server to Improve Website overall Performance.
But for Many of Webmaster site owners this solution takes too much time as well many people just don't have even basic command line knowledge / are kinda of scared from the console but need to do image compression in a simple GUI way for those the good news are there is  Graphical cross-platform tool for losslessly optimizing PNG and JPG files for web. Trimage.
To use it it even unexperienced non enthusiast could simply roll out a new Virtual Machine on top of some VM Host machine such as Virtual Box and roll out some kind of Linux distribution via a graphical installer which is mega easy well guided and takes 15-20 minutes time.

Once machine is set-up either the Graphical Distribution tool for page management or via apt you can fetch Trimage. It is now existing in most Linux distributions so, to install it on any deb based distribution Debian / Mint / Ubuntu etc. do the usual:

# apt-get install –yes trimage


trimage-compress-reduce-lossless-encoding-of-pictures-for-seo-linux-screenshot1

Once you have it, just move the pictures you want to compress for losslessly optimizing from your website to your Computer with Linux. Trimage GUI on the background will run commands optipng, pngcrush, advpng or jpegoptim, imageoptim and depending on the filetype remove the unnecessery file data that are appended by the program with which image was produced Gimp / Photoshop / Camera software etc. All image files are losslessy compressed on the highest available compression levels, and EXIF and other metadata is removed so you just have to recopy ( upload ) the optimized images back to the website.

trimage-compress-reduce-lossless-encoding-of-pictures-for-seo-linux-screenshot2

That's all folks Enjoy ! 🙂

 

IBM TSM dsmc console client use for listing configured backups, checking set scheduled backups and backup and restore operations howto


March 6th, 2020

tsm-ibm-logo_tivoli-dsmc-console-client-listing-backups-create-backups-and-restore-on-linux-unix-windows

Creating a simple home based backup solution with some shell scripting and rsync is a common use. However as a sysadmin in a middle sized or large corporations most companies use some professional backup service such as IBM Tivoli Storage Manager TSM – recently IBM changed the name of the product to IBM Spectrum.

IBM TSM  is a data protection platform that gives enterprises a single point of control and administration for backup and recovery that is used for Privare Clouds backup and other high end solutions where data criticality is top.
Usually in large companies TSM backup handling is managed by a separate team or teams as managing a large TSM infrastructure is quite a complex task, however my experience as a sysadmin show me that even if you don't have too much of indepth into tsm it is very useful to know how to manage at least basic Incremental backup operations such as view what is set to be backupped, set-up a new directory structure for backup, check the backup schedule configured, check what files are included and which excluded from the backup store etc. 

TSM has multi OS support ans you can use it on most streamline Operating systems Windows / Mac OS X and Linux in this specific article I'll be talking concretely about backing up data with tsm on Linux, tivoli can be theoretically brought up even on FreeBSD machines via the Linuxemu BSD module and the 64-Bit Tivoli Storage Manager RPMs.
Therefore in this small article I'll try to give few useful operations for the novice admin that stumbles on tsm backupped server that needs some small maintenance.
 

1. Starting up the dsmc command line client

 

Nomatter the operating system on which you run it to run the client run:

# dsmc

 

tsm-check-backup-schedule-set-time

Note that usually dsmc should run as superuser so if you try to run it via a normal non-root user you will get an error message like:

 

[ user@linux ~]$ dsmc
ANS1398E Initialization functions cannot open one of the Tivoli Storage Manager logs or a related file: /var/tsm/dsmerror.log. errno = 13, Permission denied

 

Tivoli SM has an extensive help so to get the use basics, type help
 

tsm> help
1.0 New for IBM Tivoli Storage Manager Version 6.4
2.0 Using commands
  2.1 Start and end a client command session
    2.1.1 Process commands in batch mode
    2.1.2 Process commands in interactive mode
  2.2 Enter client command names, options, and parameters
    2.2.1 Command name
    2.2.2 Options
    2.2.3 Parameters
    2.2.4 File specification syntax
  2.3 Wildcard characters
  2.4 Client commands reference
  2.5 Archive
  2.6 Archive FastBack

Enter 'q' to exit help, 't' to display the table of contents,
press enter or 'd' to scroll down, 'u' to scroll up or
enter a help topic section number, message number, option name,
command name, or command and subcommand:    

 

2. Listing files listed for backups

 

A note to make here is as in most corporate products tsm supports command aliases so any command supported described in the help like query, could be
abbreviated with its first letters only, e.g. query filespace tsm cmd can be abbreviated as

tsm> q fi

Commands can be run non-interactive mode also so if you want the output of q fi you can straight use:

tsm> dsmc q fi

 

tsm-check-included-excluded-files-q-file-if-backupped-list-backup-set-directories

This shows the directories and files that are set for backup creation with Tivoli.

 

3. Getting included and excluded backup set files

 

It is useful to know what are the exact excluded files from tsm set backup this is done with query inclexcl

tsm-check-excluded-included-files

 

4. Querying for backup schedule time

Tivoli as every other backup solution is creating its set to backup files in a certain time slot periods. 
To find out what is the time slot for backup creation use;

tsm> q sched
Schedule Name: WEEKLY_ITSERV
      Description: ITSERV weekly incremental backup
   Schedule Style: Classic
           Action: Incremental
          Options: 
          Objects: 
         Priority: 5
   Next Execution: 180 Hours and 35 Minutes
         Duration: 15 Minutes
           Period: 1 Week  
      Day of Week: Wednesday
            Month:
     Day of Month:
    Week of Month:
           Expire: Never  

 

tsm-query-partitions-backupeed-or-not

 

5. Check which files have been backed up

If you want to make sure backups are really created it is a good to check, which files from the selected backup files have already
a working backup copy.

This is done with query backup like so:

tsm> q ba /home/*

 

tsm-dsmc-query-user-home-for-backups

If you want to query all the current files and directories backed up under a directory and all its subdirectories you need to add the -subdir=yes option as below:

 

tsm> q ba /home/hipo/projects/* -subdir=yes
   
Size      Backup Date        Mgmt Class A/I File
   —-      ———–        ———- — —-
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106
  1,024  08-12-2011 02:46:53    STANDARD    A  /home/hipo/projects/hsm41perf
    512  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hsm41test
    512  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test
  1,024  12-09-2011 19:57:09    STANDARD    A  /home/hipo/projects/hfs0106/test/test2
 12,048  04-12-2011 02:01:29    STANDARD    A  /home/hipo/projects/hsm41perf/tables
 50,326  30-04-2012 01:35:26    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70023
 50,326  27-04-2012 00:28:15    STANDARD    A  /home/hipo/projects/hsm42upg/PMR70099
 11,013  24-04-2012 00:22:56    STANDARD    A  /home/hipo/projects/hsm42upg/md5check  

 

  • To make tsm, backup some directories on Linux / AIX other unices:

 

tsm> incr /  /usr  /usr/local  /home /lib

 

  • For tsm to backup some standard netware drives, use:

 

tsm> incr NDS:  USR:  SYS:  APPS:  

 

  • To backup C:\ D:\ E:\ F:\ if TSM is running on Windows

 

tsm> incr C:  D:  E: F:  -incrbydate 

 

  • To back up entire disk volumes irrespective of whether files have changed since the last backup, use the selective command with a wildcard and -subdir=yes as below:

 

tsm> sel /*  /usr/*   /home/*  -su=yes   ** Unix/Linux

 

7. Backup selected files from a backup location

 

It is intuitive to think you can just add some wildcard characters to select what you want
to backup from a selected location but this is not so, if you try something like below
you will get an err.

 

tsm> incr /home/hipo/projects/*/* -su=yes      
ANS1071E Invalid domain name entered: '/home/hipo/projects/*/*'


The proper way to select a certain folder / file for backup is with:

 

tsm> sel /home/hipo/projects/*/* -su=yes

 

8. Restoring tsm data from backup

 

To restore the config httpd.conf to custom directory use:

 

tsm> rest /etc/httpd/conf/httpd.conf  /home/hipo/restore/

 

N!B! that in order for above to work you need to have the '/' trailing slash at the end.

If you want to restore a file under a different name:

 

tsm> rest /etc/ntpd.conf  /home/hipo/restore/

 

9. Restoring a whole backupped partition

 

tsm> rest /home/*  /tmp/restore/ -su=yes

 

This is using the Tivoli 'Restoring multiple files and directories', and the files to restore '*'
are kept till the one that was recovered (saying this in case if you accidently cancel the restore)

 

10. Restoring files with back date 

 

By default the restore function will restore the latest available backupped file, if you need
to recover a specific file, you need the '-inactive' '-pick' options.
The 'pick' interface is interactive so once listed you can select the exact file from the date
you want to restore.

General restore command syntax is:
 

tsm> restore [source-file] [destination-file]

 


tsm> rest /home/hipo/projects/*  /tmp/restore/ -su=yes  -inactive -pick

TSM Scrollable PICK Window – Restore

     #    Backup Date/Time        File Size A/I  File
   ————————————————————————————————–
   170. | 12-09-2011 19:57:09        650  B  A   /home/hipo/projects/hsm41test/inclexcl.test
   171. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.ORIG
   172. | 12-09-2011 19:57:09       2.74 KB  A   /home/hipo/projects/hsm41test/inittab.TEST
   173. | 12-09-2011 19:57:09       1.13 KB  A   /home/hipo/projects/hsm41test/md5.out
   174. | 30-04-2012 01:35:26        512  B  A   /home/hipo/projects/hsm42125upg/PMR70023
   175. | 26-04-2012 01:02:08        512  B  I   /home/hipo/projects/hsm42125upg/PMR70023
   176. | 27-04-2012 00:28:15        512  B  A   /home/hipo/projects/hsm42125upg/PMR70099
   177. | 24-04-2012 19:17:34        512  B  I   /home/hipo/projects/hsm42125upg/PMR70099
   178. | 24-04-2012 00:22:56       1.35 KB  A   /home/hipo/projects/hsm42125upg/dsm.opt
   179. | 24-04-2012 00:22:56       4.17 KB  A   /home/hipo/projects/hsm42125upg/dsm.sys
   180. | 24-04-2012 00:22:56       1.13 KB  A   /home/hipo/projects/hsm42125upg/dsmmigfstab
   181. | 24-04-2012 00:22:56       7.30 KB  A   /home/hipo/projects/hsm42125upg/filesystems
   182. | 24-04-2012 00:22:56       1.25 KB  A   /home/hipo/projects/hsm42125upg/inclexcl
   183. | 24-04-2012 00:22:56        198  B  A   /home/hipo/projects/hsm42125upg/inclexcl.dce
   184. | 24-04-2012 00:22:56        291  B  A   /home/hipo/projects/hsm42125upg/inclexcl.ox_sys
   185. | 24-04-2012 00:22:56        650  B  A   /home/hipo/projects/hsm42125upg/inclexcl.test
   186. | 24-04-2012 00:22:56        670  B  A   /home/hipo/projects/hsm42125upg/inetd.conf
   187. | 24-04-2012 00:22:56       2.71 KB  A   /home/hipo/projects/hsm42125upg/inittab
   188. | 24-04-2012 00:22:56       1.00 KB  A   /home/hipo/projects/hsm42125upg/md5check
   189. | 24-04-2012 00:22:56      79.23 KB  A   /home/hipo/projects/hsm42125upg/mkreport.020423.out
   190. | 24-04-2012 00:22:56       4.27 KB  A   /home/hipo/projects/hsm42125upg/ssamap.020423.out
   191. | 26-04-2012 01:02:08      12.78 MB  A   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
   192. | 25-04-2012 16:33:36      12.78 MB  I   /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
        0———10——–20——–30——–40——–50——–60——–70——–80——–90–
<U>=Up  <D>=Down  <T>=Top  <B>=Bottom  <R#>=Right  <L#>=Left
<G#>=Goto Line #  <#>=Toggle Entry  <+>=Select All  <->=Deselect All
<#:#+>=Select A Range <#:#->=Deselect A Range  <O>=Ok  <C>=Cancel
pick> 


To navigate in pick interface you can select individual files to restore via the number seen leftside.
To scroll up / down use 'U' and 'D' as described in the legenda.

 

11. Restoring your data to another machine

 

In certain circumstances, it may be necessary to restore some, or all, of your data onto a machine other than the original from which it was backed up.

In ideal case the machine platform should be identical to that of the original machine. Where this is not possible or practical please note that restores are only possible for partition types that the operating system supports. Thus a restore of an NTFS partition to a Windows 9x machine with just FAT support may succeed but the file permissions will be lost.
TSM does not work fine with cross-platform backup / restore, so better do not try cross-platform restores.
 Trying to restore files onto a Windows machine that have previously been backed up with a non-Windows one. TSM created backups on Windows sent by other OS platforms can cause  backups to become inaccessible from the host system.

To restore your data to another machine you will need the TSM software installed on the target machine. Entries in Tivoli configuration files dsm.sys and/or dsm.opt need to be edited if the node that you are restoring from does not reside on the same server. Please see our help page section on TSM configuration files for their locations for your operating system. 

To access files from another machine you should then start the TSM client as below:

 

# dsmc -virtualnodename=RESTORE.MACHINE      


You will then be prompted for the TSM password for this machine.

 

You will probably want to restore to a different destination to the original files to prevent overwriting files on the local machine, as below:

 

  • Restore of D:\ Drive to D:\Restore ** Windows 

 

tsm> rest D:\*   D:\RESTORE\    -su=yes 
 

 

  • Restore user /home/* to /scratch on ** Mac, Unix/Linux

 

tsm> rest /home/* /scratch/     -su=yes  
 

 

  • Restoring Tivoli data on old netware

 

tsm> rest SOURCE-SERVER\USR:*  USR:restore/   -su=yes  ** Netware

 

12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

The easiest way is to check the produced dschmed.log if everything is okay there should be records in the log that Tivoli backup was scheduled in a some hours time
succesfully.
A normally produced backup scheduled in log should look something like:

 

14-03-2020 23:03:04 — SCHEDULEREC STATUS BEGIN
14-03-2020 23:03:04 Total number of objects inspected:   91,497
14-03-2020 23:03:04 Total number of objects backed up:      113
14-03-2020 23:03:04 Total number of objects updated:          0
14-03-2020 23:03:04 Total number of objects rebound:          0
14-03-2020 23:03:04 Total number of objects deleted:          0
14-03-2020 23:03:04 Total number of objects expired:         53
14-03-2020 23:03:04 Total number of objects failed:           6
14-03-2020 23:03:04 Total number of bytes transferred:    19.38 MB
14-03-2020 23:03:04 Data transfer time:                    1.54 sec
14-03-2020 23:03:04 Network data transfer rate:        12,821.52 KB/sec
14-03-2020 23:03:04 Aggregate data transfer rate:        114.39 KB/sec
14-03-2020 23:03:04 Objects compressed by:                    0%
14-03-2020 23:03:04 Elapsed processing time:           00:02:53
14-03-2020 23:03:04 — SCHEDULEREC STATUS END
14-03-2020 23:03:04 — SCHEDULEREC OBJECT END WEEKLY_23_00 14-12-2010 23:00:00
14-03-2020 23:03:04 Scheduled event 'WEEKLY_23_00' completed successfully.
14-03-2020 23:03:04 Sending results for scheduled event 'WEEKLY_23_00'.
14-03-2020 23:03:04 Results sent to server for scheduled event 'WEEKLY_23_00'.

 

in case of errors you should check dsmerror.log
 

Conclusion


In this article I've briefly evaluated some basics of IBM Commercial Tivoli Storage Manager (TSM) to be able to  list backups, check backup schedules and how to the files set to be
excluded from a backup location and most importantly how to check that data backed up data is in a good shape and accessible.
It was explained how backups can be restored on a local and remote machine as well as how to  append new files to be set for backup on next incremental scheduled backup.
It was shown how the pick interactive cli interface could be used to restore files at a certain data back in time as well as how full partitions can be restored and how some
certain file could be retrieved from the TSM data copy.

Improve DNS lookup domain resolve speed on Linux / UNIX servers through /etc/resolv.conf timeout, attempts, rorate options


February 27th, 2020

improve-dns-lookup-speed-on-Linux-UNIX-servers-resolv.conf-change-dns-settings-linux
If you're an performance optimization freak and you want to optimize your Linux servers to perform better in terms of DNS resolve slowness because of failing DNS resolve queries due to Domain Name Server request overload or due to Denial of Service attack towards it. It might be interesting to mention about some little known functionalities of /etc/resolv.conf described in the manual page.

The defined nameservers under /etc/resolv.conf are queried one by one waiting for responce of the sent DNS resolve request if it is not replied from the first one for some time, the 2nd one is queried until a responce is received by any of the defined nameserver IPs

A default /etc/resolv.conf on a new Linux server install looks something like this:
 

nameserver      10.10.8.1
nameserver      10.10.8.2
nameserver      10.10.8.3
search          sub.subdomain.com subdom.dom.domain.com


However one thing is that defined if NS1 dies out due to anything, it takes timeout time until the second or 3rd working one takes over to resolve the query.
This is controlled by the timeout value.

Below is description from man page

timeout:n
sets the amount of time the resolver will wait for a
response from a remote name server before retrying the
query via a different name server.  Measured in
seconds, the default is RES_TIMEOUT (currently 5, see
<resolv.h>).  The value for this option is silently
capped to 30.

 

  • In other words Timeout value is time to resolving IP address from hostname through DNS server,timeout option is to reduce hostname lookup time

As you see from manual default is 5 seconds which is quite high, thus reducing the value to 3 secs or even 1 seconds is a good sysadmin practice IMHO.

Another value that could be tuned in /etc/resolv.conf is attempts value below is what the manual says about it: 
 

attempts:n
                     Sets the number of times the resolver will send a query to its name servers before giving up and returning an error to the calling application.  The default is RES_DFLRETRY (cur‐
                     rently 2, see <resolv.h>).  The value for this option is silently capped to 5.

 

 

  • This means default behaviour on a failing DNS query resolve is to try to resend the DNS resolve request to the failing nameserver 5 more times, that is quite high thus it is a good practice from my experience to reduce it to something as 2 or 1


Another very useful resolv.conf value is rotate
The default behavior of how DNS outgoing Domain requests are handled is to use only the primary defined DNS, instead if you need to do a load balancing in a round-robin manner add to conf rotate option.

The final /etc/resolv.conf optimized would look like so:

 

linux# cat /etc/resolv.conf

nameserver      10.10.8.1
nameserver      10.10.8.2
nameserver      10.10.8.3
search          sub.subdomain.com subdom.dom.domain.com
options ndots:1
options timeout:1
options attempts:1
options rotate


The search opt. placement is also important to be placed in the right location in the file. The correct placement is after the nameservers defined, I have to say in older Linux distributions the correct placement of search option was to be on top of resolv.conf.

Note that this configuration is good and fits not only Linux but also is a good DNS lookup optimization speed on other UNIX derivatives such as FreeBSD / NetBSD as well as other Proprietary OS UNIX machines running IBM AIX etc.

On Linux it is also possible to place the options given in one single line like so, below is the config I have on my pc-freak.net running Lenovo server:

 

domain pc-freak.net
search pc-freak.net
#nameserver 192.168.0.1
nameserver 127.0.0.1
nameserver 83.228.92.2
nameserver 8.8.8.8
nameserver 83.228.92.1
nameserver 208.67.222.222
nameserver 208.67.220.220
options timeout:2 attempts:1 rotate

 

When is /etc/hosts record venerated and when is /etc/resolv.conf DNS defined queried for a defined DNS host?

 

One important thing to know when dealing with /etc/resolv.conf  is what happens if a Name domain is defined in both /etc/hosts and /etc/resolv.conf.
For example you have a pc-freak.net domain record in /etc/hosts to a certain domain
but the DNS nameserver 8.8.8.8 in Google has a record to an IP that is the real IP 83.228.93.76

 

83.228.93.75 irc.pc-freak.net pc-freak.net pcfreak.biz www.pc-freak.net pcfreak.us services.pc-freak.net jabber.pc-freak.net

 

# dig @8.8.8.8 pc-freak.net

; <<>> DiG 9.11.5-P4-5.1-Debian <<>> @8.8.8.8 pc-freak.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54656
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;pc-freak.net.                  IN      A

;; ANSWER SECTION:
pc-freak.net.           3599    IN      A       83.228.93.76

;; Query time: 40 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: чт фев 27 18:04:23 EET 2020
;; MSG SIZE  rcvd: 57
 

 

  • Which of the 2 different IPs will the applications installed on the server such as Apache / Squid / MySQL / tinyproxy for their DNS resolve operations?

 


Now it is time to say few words about /etc/nsswitch.conf (The Nameserver switching configuration file). This file defines the DNS resolve file used order in which the Operationg System does IP to domain translation and backwards.
 

# grep -i hosts: /etc/nsswitch.conf

hosts:          files dns myhostname

As you can see first the local defined in files like /etc/hosts record is venerated when resolving, then it is the externally configured DNS resolver IPs from /etc/resolv.conf.

nsswitch.conf  is used also for defining where the OS will look up for user / passwd (e.g. login credentials) on login, on systems which are having an LDAP authentication via the sssd (system security services daemon) via definitions like:

 

passwd:     files sss
shadow:     files sss
group:      files sss


E.g. the user login will be first try to read from local /etc/passwd , /etc/shadow , /etc/groups and if no matched record is found then the LDAP service the sssd is queried.

Fix Blocked Unresponsive keyboard keys on Windows 7 / 10


February 25th, 2020

Scroll-lock-keboard-work-around-windows-issues

 

The Problem


If you're still using Windows 7 Operating system in your company due to some weird security concern policies and your company gets custom end user updates from Microsoft due to a special EULA agreement and you are new to using Windows E.g. have for many users used for your daily work Linux and Mac OS you might hit a strange issue wtih many of the keyboard keys strangely being locked with some of the keys such as Num Lock, Escape tab working where the Alphabet keys then don't panic. This is not a Windows bug its a feature (as usual) 🙂
 

Reason behind Blocked Unresponsive Keyboard

 

First logical thought I had is maybe my Logitech K120 Membrane 17 EURO cheap keyboard externally attached keyboard broke up thus I've tried to connect another LOGIC keyboard I had at hand just to assure myself the problem with partial keys on kbd reacting was present with the other Working keyboard as well.
This was an indicator that either the custom installed Windows by the company Helpdesk Office with the preassumed common additional features for corporations such as Keyloggers on this Laptop has messed up somohow the Windows service that is managing the keyboard or some kind of mechanical error or electronic circuit on the laptop embedded keyboard has occured or the KBD DLL Loaded driver damaged
I have to say here a colleague of mine was having a weird keyboard problems back in the day when I was still working in Project Services as a Web and Middleware in Hewlett Packard, where misteriously a character was added to his typed content just like a key on his keyboard has stuck and he experienced this issue for quite some time, he opened the keyboard to physically check whether all is okay and even checked the keyboard electricity whether levels on each of the keys and he couldn't find anything, and after running Malware Bytes anti-malware and a couple of other anti-malware programs which found his computer was infected with Malware and issue resolved.

 

Hard Fix – Reboot ..

 

As as a solution most times so far I've restarted the Windows which was reloading the Windows kernel / DLL libraries etc.

However just hit to this Windows accessibility feature once again today and since this is not the first time I end up with unworking keyboard (perhaps due to my often) furious fast typing – where I press sometimes multiple keys in parallel as a typing error then you trigger the Windows Disabled accessibiltiy Windows Feature, which as I thought makes the PC only usable for Mouse but unusuable for providing any meaningful keyboard input.

This problem I've faced already multiple times and usually the work around was the good known Windows User recipee phrase "Restart and It will get fixed", this time I was pissed off and didn't wanted to loose another 5 minutes in Restarting Reconnecting to the Company's Cisco Secure VPN reopening all my used files Notepad++ / Outlook / Browsers etc plus I was already part of online Lync (Skype) Meeting in which Colleague was Sharing his remote Desktop checking some important stuff about Zabbix Monitored AIX machine, hence didn't wanted to restart but still wanted to use my computer and type some stuff to send Email and do a simple googling.

Temporary work around to complete work with Virtual Keyboard

Hence as a temporary work around, I've used the Windows Virtual Keyboard, I've mentioned it in the earlier  blog post – How to run Virtual Keyboard in Windows XP / Vista article
To do so I'verun by typing osk command in cmd.exe command Prompt:

Either Search for osk.exe from Start menu

windows-7-osk-virtual-keyboard-screenshot-from-start-menu

or run via command line via

Windows Button (on the Keyboard) + R and run
 

cmd.exe -> osk

 


windows-7-osk-virtual-keyboard-screenshot2

 

Solution Without PC Restart


After a bit of thought and Googling I've found the fix  here

From the Start -> Control Panel from here I had to go to Accessibility Options.
Select Ease of Access Center.

Keyboard_Locked-Windows_7_Accessibility_Options-HP-Customer-Support

Select the keyboard settings and
Ensure the following options are unchecked: Turn on Sticky Keys, Turn on Toggle Keys and Turn on Filter Keys.

Keyboard-Locked_Find-Out-How-to-Unlock-make-keyboard-easier-to-use

I've found in the Turn on Toggle Keys tick present (e.g. service was enabled) – hence  after unticking it and 

Press Apply and OK, keyboard restored its usual functions.
Now all left was to  Enjoy as your keyboard was back usable and I could conitnue my Citrix sessions and SSH console Superputty terminals  and complete my started to write E-mail
without loosing time meanlessly for reboot.


N.B. !!!! A VERY IMPORTANT NOTE TO MAKE IS IF NOTHING ELSE HELPS PLEASE TRY TO RUN OSK ViRTUAL KEYBOARD
UNDER SOME OCCASIONS THE VIRTUAL KEYBOARD FORCES THE WINDOWS KEYBOARD DRIVER TO RELOAD AND THAT WILL FIX THE KEYBOARD !!!

Windows 10 Disable the Filter Keys option

 

This feature makes your keyboard ignore brief or repeated keystrokes, which might have led to your WinKey issue in Windows 10. To disable filter keys, use the instructions below:

1. Right-click on your Start menu icon.
2. Select Settings from the menu.
3. Navigate to Ease of Access and click on it.
4. Go to the left pane and click Keyboard.
5. Locate the Filter Keys feature.
6. Toggle it off.
7. Check if this manoeuvre has resolved your issue.
 

Closing Notes 


Of course this might be not always the fix, as sometimes it could be that the Winblows just blows your keyboard buffer due to some buggy application or a bug, but in most of the times that should solve it 🙂
If it didn't go through and debug all the other possible reasons, check whether you have a faulty keyboard cable (if you're still on a non-bluetooth Wired Keyboard), unplug and plug the keyboard again,
scan the computer for spyware and malware, rethink what really happened or what have you done until the problem occured and whether blocked keyboard is triggered by your user action or was triggered
by some third party software anti-virus stuff that did it as an attempt to prevent keylog sniffer / Virus or other weird stuff.

How to check if shared library is loaded in AIX OS – Fix missing libreadline.so.7


February 20th, 2020

ibm-aix-logo1

I've had to find out whether an externally Linux library is installed  on AIX system and whether something is not using it.
The returned errors was like so:

 

# gpg –export -a

Could not load program gpg:
Dependent module /opt/custom/lib/libreadline.a(libreadline.so.7) could not be loaded.
Member libreadline.so.7 is not found in archive


After a bit of investigation, I found that gpg was failing cause it linked to older version of libreadline.so.6, the workaround was to just substitute the newer version of libreadline.so.7 over the original installed one.

Thus I had a plan to first find out whether this libreadline.a is loaded and recognized by AIX UNIX first and second find out whether some of the running processes is not using that library.
I've come across this interesting IBM official documenation that describes pretty good insights on how to determine whether a shared library  is currently loaded on the system. which mentions the genkld command that is doing
exactly what I needed.

In short:
genkld – creates a list that is printed to the console that shows all loaded shared libraries

genkld-screenshot-aix-unix

Next I used lsof (list open files) command to check whether there is in real time opened libraries by any of the running programs on the system.

After not finding anything and was sure the library is neither loaded as a system library in AIX nor it is used by any of the currently running AIX processes, I was sure I could proceed to safely overwrite libreadline.a (libreadline.so.6) with libreadline.a with (libreadline.so.7).

The result of that is again a normally running gpg as ldd command shows the binary is again normally linked to its dependend system libraries.
 

aix# ldd /usr/bin/gpg
/usr/bin/gpg needs:
         /usr/lib/threads/libc.a(shr.o)
         /usr/lib/libpthreads.a(shr_comm.o)
         /usr/lib/libpthreads.a(shr_xpg5.o)
         /opt/freeware/lib/libintl.a(libintl.so.1)
         /opt/freeware/lib/libreadline.a(libreadline.so.7)
         /opt/freeware/lib/libiconv.a(libiconv.so.2)
         /opt/freeware/lib/libz.a(libz.so.1)
         /opt/freeware/lib/libbz2.a(libbz2.so.1)
         /unix
         /usr/lib/libcrypt.a(shr.o)
         /opt/freeware/lib/libiconv.a(shr4.o)
         /usr/lib/libcurses.a(shr42.o)

 

 

# gpg –version
gpg (GnuPG) 1.4.22
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

 

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

 

 

How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages


February 19th, 2020

haproxy-logging-basics-how-to-log-to-separate-file-prevent-duplicate-messages-haproxy-haproxy-weblogo-squares
haproxy  logging can be managed in different form the most straight forward way is to directly use /dev/log either you can configure it to use some log management service as syslog or rsyslogd for that.

If you don't use rsyslog yet to install it: 

# apt install -y rsyslog

Then to activate logging via rsyslogd we can should add either to /etc/rsyslogd.conf or create a separte file and include it via /etc/rsyslogd.conf with following content:
 

Enable haproxy logging from rsyslogd


Log haproxy messages to separate log file you can use some of the usual syslog local0 to local7 locally used descriptors inside the conf (be aware that if you try to use some wrong value like local8, local9 as a logging facility you will get with empty haproxy.log, even though the permissions of /var/log/haproxy.log are readable and owned by haproxy user.

When logging to a local Syslog service, writing to a UNIX socket can be faster than targeting the TCP loopback address. Generally, on Linux systems, a UNIX socket listening for Syslog messages is available at /dev/log because this is where the syslog() function of the GNU C library is sending messages by default. To address UNIX socket in haproxy.cfg use:

log /dev/log local2 


If you want to log into separate log each of multiple running haproxy instances with different haproxy*.cfg add to /etc/rsyslog.conf lines like:

local2.* -/var/log/haproxylog2.log
local3.* -/var/log/haproxylog3.log


One important note to make here is since rsyslogd is used for haproxy logging you need to have enabled in rsyslogd imudp and have a UDP port listener on the machine.

E.g. somewhere in rsyslog.conf or via rsyslog include file from /etc/rsyslog.d/*.conf needs to have defined following lines:

$ModLoad imudp
$UDPServerRun 514


I prefer to use external /etc/rsyslog.d/20-haproxy.conf include file that is loaded and enabled rsyslogd via /etc/rsyslog.conf:

# vim /etc/rsyslog.d/20-haproxy.conf

$ModLoad imudp
$UDPServerRun 514​
local2.* -/var/log/haproxy2.log


It is also possible to produce different haproxy log output based on the severiy to differentiate between important and less important messages, to do so you'll need to rsyslog.conf something like:
 

# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log

 

Prevent Haproxy duplicate messages to appear in /var/log/messages

If you use local2 and some default rsyslog configuration then you will end up with the messages coming from haproxy towards local2 facility producing doubled simultaneous records to both your pre-defined /var/log/haproxy.log and /var/log/messages on Proxy servers that receive few thousands of simultanous connections per second.
This is a problem since doubling the log will produce too much data and on systems with smaller /var/ partition you will quickly run out of space + this haproxy requests logging to /var/log/messages makes the file quite unreadable for normal system events which are so important to track clearly what is happening on the server daily.

To prevent the haproxy duplicate messages you need to define somewhere in rsyslogd usually /etc/rsyslog.conf local2.none near line of facilities configured to log to file:

*.info;mail.none;authpriv.none;cron.none;local2.none     /var/log/messages

This configuration should work but is more rarely used as most people prefer to have haproxy log being written not directly to /dev/log which is used by other services such as syslogd / rsyslogd.

To use /dev/log to output logs from haproxy configuration in global section use config like:
 

global
        log /dev/log local2 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

The log global directive basically says, use the log line that was set in the global section for whole config till end of file. Putting a log global directive into the defaults section is equivalent to putting it into all of the subsequent proxy sections.

Using global logging rules is the most common HAProxy setup, but you can put them directly into a frontend section instead. It can be useful to have a different logging configuration as a one-off. For example, you might want to point to a different target Syslog server, use a different logging facility, or capture different severity levels depending on the use case of the backend application. 

Insetad of using /dev/log interface that is on many distributions heavily used by systemd to store / manage and distribute logs,  many haproxy server sysadmins nowdays prefer to use rsyslogd as a default logging facility that will manage haproxy logs.
Admins prefer to use some kind of mediator service to manage log writting such as rsyslogd or syslog, the reason behind might vary but perhaps most important reason is  by using rsyslogd it is possible to write logs simultaneously locally on disk and also forward logs  to a remote Logging server  running rsyslogd service.

Logging is defined in /etc/haproxy/haproxy.cfg or the respective configuration through global section but could be also configured to do a separate logging based on each of the defined Frontend Backends or default section. 
A sample exceprt from this section looks something like:

#———————————————————————
# Global settings
#———————————————————————
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#———————————————————————
defaults
    mode                    tcp
    log                     global
    option                  tcplog
    #option                  dontlognull
    #option http-server-close
    #option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 7
    #timeout http-request    10s
    timeout queue           10m
    timeout connect         30s
    timeout client          20m
    timeout server          10m
    #timeout http-keep-alive 10s
    timeout check           30s
    maxconn                 3000

 

 

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

 

#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_DKV_PROD_WLPFO
    mode tcp
    bind 192.168.233.5:30000-31050
    option tcplog
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
    default_backend Default_Bakend_Name


#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_DKV_PROD_WLPFO
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
    stick on src
    # (5) Server List
    # (5.1) Backend
    server Backend_Server1 10.10.10.1 check port 18088
    server Backend_Server2 10.10.10.2 check port 18088 backup


The log directive in above config instructs HAProxy to send logs to the Syslog server listening at 127.0.0.1:514. Messages are sent with facility local2, which is one of the standard, user-defined Syslog facilities. It’s also the facility that our rsyslog configuration is expecting. You can add more than one log statement to send output to multiple Syslog servers.

Once rsyslog and haproxy logging is configured as a minumum you need to restart rsyslog (assuming that haproxy config is already properly loaded):

# systemctl restart rsyslogd.service

To make sure rsyslog reloaded successfully:

systemctl status rsyslogd.service


Restarting HAproxy

If the rsyslogd logging to 127.0.0.1 port 514 was recently added a HAProxy restart should also be run, you can do it with:
 

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)


Or to restart use systemctl script (if haproxy is not used in a cluster with corosync / heartbeat).

# systemctl restart haproxy.service

You can control how much information is logged by adding a Syslog level by

    log         127.0.0.1 local2 info


The accepted values are the standard syslog security level severity:

Value Severity Keyword Deprecated keywords Description Condition
0 Emergency emerg panic System is unusable A panic condition.
1 Alert alert   Action must be taken immediately A condition that should be corrected immediately, such as a corrupted system database.
2 Critical crit   Critical conditions Hard device errors.
3 Error err error Error conditions  
4 Warning warning warn Warning conditions  
5 Notice notice   Normal but significant conditions Conditions that are not error conditions, but that may require special handling.
6 Informational info   Informational messages  
7 Debug debug   Debug-level messages Messages that contain information normally of use only when debugging a program.

 

Logging only errors / timeouts / retries and errors is done with option:

Note that if the rsyslog is configured to listen on different port for some weird reason you should not forget to set the proper listen port, e.g.:
 

  log         127.0.0.1:514 local2 info

option dontlog-normal

in defaults or frontend section.

You most likely want to enable this only during certain times, such as when performing benchmarking tests.

(or log-format-sd for structured-data syslog) directive in your defaults or frontend
 

Haproxy Logging shortly explained


The type of logging you’ll see is determined by the proxy mode that you set within HAProxy. HAProxy can operate either as a Layer 4 (TCP) proxy or as Layer 7 (HTTP) proxy. TCP mode is the default. In this mode, a full-duplex connection is established between clients and servers, and no layer 7 examination will be performed. When in TCP mode, which is set by adding mode tcp, you should also add option tcplog. With this option, the log format defaults to a structure that provides useful information like Layer 4 connection details, timers, byte count and so on.

Below is example of configured logging with some explanations:

Log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

haproxy-logged-fields-explained
Example of Log-Format configuration as shown above outputted of haproxy config:

Log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

haproxy_http_log_format-explained1

To understand meaning of this abbreviations you'll have to closely read  haproxy-log-format.txt. More in depth info is to be found in HTTP Log format documentation


haproxy_logging-explained

Logging HTTP request headers

HTTP request header can be logged via:
 

 http-request capture

frontend website
    bind :80
    http-request capture req.hdr(Host) len 10
    http-request capture req.hdr(User-Agent) len 100
    default_backend webservers


The log will show headers between curly braces and separated by pipe symbols. Here you can see the Host and User-Agent headers for a request:

192.168.150.1:57190 [20/Dec/2018:22:20:00.899] website~ webservers/server1 0/0/1/0/1 200 462 – – —- 1/1/0/0/0 0/0 {mywebsite.com|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.80 } "GET / HTTP/1.1"

 

Haproxy Stats Monitoring Web interface


Haproxy is having a simplistic stats interface which if enabled produces some general useful information like in above screenshot, through which
you can get a very basic in browser statistics and track potential issues with the proxied traffic for all configured backends / frontends incoming outgoing
network packets configured nodes
 experienced downtimes etc.

haproxy-statistics-report-picture

The basic configuration to make the stats interface accessible would be like pointed in above config for example to enable network listener on address
 

https://192.168.0.5:8080/stats


with hproxyuser / password config would be:

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

 

 

Sessions states and disconnect errors on new application setup

Both TCP and HTTP logs include a termination state code that tells you the way in which the TCP or HTTP session ended. It’s a two-character code. The first character reports the first event that caused the session to terminate, while the second reports the TCP or HTTP session state when it was closed.

Here are some essential termination codes to track in for in the log:
 

Here are some termination code examples most commonly to see on TCP connection establishment errors:

Two-character code    Meaning
—    Normal termination on both sides.
cD    The client did not send nor acknowledge any data and eventually timeout client expired.
SC    The server explicitly refused the TCP connection.
PC    The proxy refused to establish a connection to the server because the process’ socket limit was reached while attempting to connect.


To get all non-properly exited codes the easiest way is to just grep for anything that is different from a termination code –, like that:

tail -f /var/log/haproxy.log | grep -v ' — '


This should output in real time every TCP connection that is exiting improperly.

There’s a wide variety of reasons a connection may have been closed. Detailed information about all possible termination codes can be found in the HAProxy documentation.
To get better understanding a very useful reading to haproxy Debug errors with  is in haproxy-logging.txt in that small file are collected all the cryptic error messages codes you might find in your logs when you're first time configuring the Haproxy frontend / backend and the backend application behind.

Another useful analyze tool which can be used to analyze Layer 7 HTTP traffic is halog for more on it just google around.

Procedure Instructions to safe upgrade CentOS / RHEL Linux 7 Core to latest release


February 13th, 2020

safe-upgrade-CentOS-and_Redhat_Enterprise_Linux_RHEL-7-to-latest-stable-release

Generally upgrading both RHEL and CentOS can be done straight with yum tool just we're pretty aware and mostly anyone could do the update, but it is good idea to do some
steps in advance to make backup of any old basic files that might help us to debug what is wrong in case if the Operating System fails to boot after the routine Machine OS restart
after the upgrade that is usually a good idea to make sure that machine is still bootable after the upgrade.

This procedure can be shortened or maybe extended depending on the needs of the custom case but the general framework should be useful anyways to someone that's why
I decided to post this.

Before you go lets prepare a small status script which we'll use to report status of  sysctl installed and enabled services as well as the netstat connections state and
configured IP addresses and routing on the system.

The script show_running_services_netstat_ips_route.sh to be used during our different upgrade stages:
 

# script status ###
echo "STARTED: $(date '+%Y-%m-%d_%H-%M-%S'):" | tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
systemctl list-unit-files –type=service | grep enabled
systemctl | grep ".service" | grep "running"
netstat -tulpn
netstat -r
ip a s
/sbin/route -n
echo "ENDED $(date '+%Y-%m-%d_%H-%M-%S'):" | tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
####

 

– Save the script in any file like /root/status.sh

– Make the /root/logs directoriy.
 

[root@redhat: ~ ]# mkdir /root/logs
[root@redhat: ~ ]# vim /root/status.sh
[root@redhat: ~ ]# chmod +x /root/status.sh

 

1. Get a dump of CentOS installed version release and grub-mkconfig generated os_probe

 

[root@redhat: ~ ]# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
[root@redhat: ~ ]# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

2. Clear old versionlock marked RPM packages (if there are such)

 

On servers maintained by multitude of system administrators just like the case is inside a Global Corporations and generally in the corporate world , where people do access the systems via LDAP and more than a single person
has superuser privileges. It is a good prevention measure to use yum package management  functionality to RPM based Linux distributions called  versionlock.
versionlock for those who hear it for a first time is locking the versions of the installed RPM packages so if someone by mistake or on purpose decides to do something like :

[root@redhat: ~ ]# yum install packageversion

Having the versionlock set will prevent the updated package to be installed with a different branch package version.

Also it will prevent a playful unknowing person who just wants to upgrade the system without any deep knowledge to be able to
run

[root@redhat: ~ ]# yum upgrade

update and leave the system in unbootable state, that will be only revealed during the next system reboot.

If you haven't used versionlock before and you want to use it you can do it with:

[root@redhat: ~ ]# yum install yum-plugin-versionlock

To add all the packages for compiling C code and all the interdependend packages, you can do something like:

 

[root@redhat: ~ ]# yum versionlock gcc-*

If you want to clear up the versionlock, once it is in use run:

[root@redhat: ~ ]#  yum versionlock clear
[root@redhat: ~ ]#  yum versionlock list

 

3.  Check RPC enabled / disabled

 

This step is not necessery but it is a good idea to check whether it running on the system, because sometimes after upgrade rpcbind gets automatically started after package upgrade and reboot. 
If we find it running we'll need to stop and mask the service.

 

# check if rpc enabled
[root@redhat: ~ ]# systemctl list-unit-files|grep -i rpc
var-lib-nfs-rpc_pipefs.mount                                      static
auth-rpcgss-module.service                                        static
rpc-gssd.service                                                  static
rpc-rquotad.service                                               disabled
rpc-statd-notify.service                                          static
rpc-statd.service                                                 static
rpcbind.service                                                   disabled
rpcgssd.service                                                   static
rpcidmapd.service                                                 static
rpcbind.socket                                                    disabled
rpc_pipefs.target                                                 static
rpcbind.target                                                    static

[root@redhat: ~ ]# systemctl status rpcbind.service
● rpcbind.service – RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

 

[root@redhat: ~ ]# systemctl status rpcbind.socket
● rpcbind.socket – RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; disabled; vendor preset: enabled)
   Active: inactive (dead)
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)
           0.0.0.0:111 (Datagram)
           [::]:111 (Stream)
           [::]:111 (Datagram)

 

4. Check any previously existing downloaded / installed RPMs (check yum cache)

 

yum install package-name / yum upgrade keeps downloaded packages via its operations inside its cache directory structures in /var/cache/yum/*.
Hence it is good idea to check what were the previously installed packages and their count.

 

[root@redhat: ~ ]# cd /var/cache/yum/x86_64/;
[root@redhat: ~ ]# find . -iname '*.rpm'|wc -l

 

5. List RPM repositories set on the server

 

 [root@redhat: ~ ]# yum repolist
Loaded plugins: fastestmirror, versionlock
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Determining fastest mirrors
repo id                                                                                 repo name                                                                                                            status
!atos-ac/7/x86_64                                                                       Atos Repository                                                                                                       3,128
!base/7/x86_64                                                                          CentOS-7 – Base                                                                                                      10,019
!cr/7/x86_64                                                                            CentOS-7 – CR                                                                                                         2,686
!epel/x86_64                                                                            Extra Packages for Enterprise Linux 7 – x86_64                                                                          165
!extras/7/x86_64                                                                        CentOS-7 – Extras                                                                                                       435
!updates/7/x86_64                                                                       CentOS-7 – Updates                                                                                                    2,500

 

This step is mandatory to make sure you're upgrading to latest packages from the right repositories for more concretics check what is inside in confs /etc/yum.repos.d/ ,  /etc/yum.conf 
 

6. Clean up any old rpm yum cache packages

 

This step is again mandatory but a good to follow just to have some more clearness on what packages is our upgrade downloading (not to mix up the old upgrades / installs with our newest one).
For documentation purposes all deleted packages list if such is to be kept under /root/logs/yumclean-install*.out file

[root@redhat: ~ ]# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

7. List the upgradeable packages's latest repository provided versions

 

[root@redhat: ~ ]# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

Then to be aware how many packages we'll be updating:

 

[root@redhat: ~ ]#  yum check-update | wc -l

 

8. Apply the actual uplisted RPM packages to be upgraded

 

[root@redhat: ~ ]# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

Again output is logged to /root/logs/yumcheckupate-*.out 

 

9. Monitor downloaded packages count real time

 

To make sure yum upgrade is not in some hanging state and just get some general idea in which state of the upgrade is it e.g. Download / Pre-Update / Install  / Upgrade/ Post-Update etc.
in mean time when yum upgrade is running to monitor,  how many packages has the yum upgrade downloaded from remote RPM set repositories:

 

[root@redhat: ~ ]#  watch "ls -al /var/cache/yum/x86_64/7Server/…OS-repository…/packages/|wc -l"

 

10. Run status script to get the status again

 

[root@redhat: ~ ]# sh /root/status.sh |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

11. Add back versionlock for all RPM packs

 

Set all RPM packages installed on the RHEL / CentOS versionlock for all packages.

 

#==if needed
# yum versionlock \*

 

 

12. Get whether old software configuration is not messed up during the Package upgrade (Lookup the logs for .rpmsave and .rpmnew)

 

During the upgrade old RPM configuration is probably changed and yum did automatically save .rpmsave / .rpmnew saves of it thus it is a good idea to grep the prepared logs for any matches of this 2 strings :
 

[root@redhat: ~ ]#   grep -i ".rpm" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out
[root@redhat: ~ ]#  grep -i ".rpmsave" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out
[root@redhat: ~ ]#  grep -i ".rpmnew" /root/logs/yumupdate-server-host-2020-01-20_14-30-41.out


If above commands returns output usually it is fine if there is is .rpmnew output but, if you get grep output of .rpmsave it is a good idea to review the files compare with the original files that were .rpmsaved with the 
substituted config file and atune the differences with the changes manually made for some program functionality.

What are the .rpmsave / .rpmnew files ?
This files are coded files that got triggered by the RPM install / upgrade due to prewritten procedures on time of RPM build.

 

If a file was installed as part of a rpm, it is a config file (i.e. marked with the %config tag), you've edited the file afterwards and you now update the rpm then the new config file (from the newer rpm) will replace your old config file (i.e. become the active file).
The latter will be renamed with the .rpmsave suffix.

If a file was installed as part of a rpm, it is a noreplace-config file (i.e. marked with the %config(noreplace) tag), you've edited the file afterwards and you now update the rpm then your old config file will stay in place (i.e. stay active) and the new config file (from the newer rpm) will be copied to disk with the .rpmnew suffix.
See e.g. this table for all the details. 

In both cases you or some program has edited the config file(s) and that's why you see the .rpmsave / .rpmnew files after the upgrade because rpm will upgrade config files silently and without backup files if the local file is untouched.

After a system upgrade it is a good idea to scan your filesystem for these files and make sure that correct config files are active and maybe merge the new contents from the .rpmnew files into the production files. You can remove the .rpmsave and .rpmnew files when you're done.


If you need to get a list of all .rpmnew .rpmsave files on the server do:

[root@redhat: ~ ]#  find / -print | egrep "rpmnew$|rpmsave$

 

13. Reboot the system 

To check whether on next hang up or power outage the system will boot normally after the upgrade, reboot to test it.

 

you can :

 

[root@redhat: ~ ]#  reboot

 

either

[root@redhat: ~ ]#  shutdown -r now


or if on newer Linux with systemd in ues below systemctl reboot.target.

[root@redhat: ~ ]#  systemctl start reboot.target

 

14. Get again the system status with our status script after reboot

[root@redhat: ~ ]#  sh /root/status.sh |tee /root/logs/status-after-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

15. Clean up any versionlocks if earlier set

 

[root@redhat: ~ ]# yum versionlock clear
[root@redhat: ~ ]# yum versionlock list

 

16. Check services and logs for problems

 

After the reboot Check closely all running services on system make sure every process / listening ports and services on the system are running fine, just like before the upgrade.
If the sytem had firewall,  check whether firewall rules are not broken, e.g. some NAT is not missing or anything earlier configured to automatically start via /etc/rc.local or some other
custom scripts were run and have done what was expected. 
Go through all the logs in /var/log that are most essential /var/log/boot.log , /var/log/messages … yum.log etc. that could reveal any issues after the boot. In case if running some application server or mail server check /var/log/mail.log or whenever it is configured to log.
If the system runs apache closely check the logs /var/log/httpd/error.log or php_errors.log for any strange errors that occured due to some issues caused by the newer installed packages.
Usually most of the cases all this should be flawless but a multiple check over your work is a stake for good results.