System Administration Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Archive for the ‘System Administration’ Category

Monitoring Linux hardware Hard Drives / Temperature and Disk with lm_sensors / smartd / hddtemp and Zabbix Userparameter lm_sensors report script

Thursday, April 30th, 2020

monitoring-linux-hardware-with-software-temperature-disk-cpu-health-zabbix-userparameter-script

I'm part of a SysAdmin Team that is partially doing some minor Zabbix imrovements on a custom corporate installed Zabbix in an ongoing project to substitute the previous HP OpenView monitoring for a bunch of Legacy Linux hosts.
As one of the necessery checks to have is regarding system Hardware, the task was to invent some simplistic way to monitor hardware with the Zabbix Monitoring tool. Monitoring Bare Metal servers hardware of HP / Dell / Fujituse etc. servers in Linux usually is done with a third party software provided by the Hardware vendor. But as this requires an additional services to run and sometimes is not desired. It was interesting to find out some alternative Linux native ways to do the System hardware monitoring.
Monitoring statistics from the system hardware components can be obtained directly from the server components with ipmi / ipmitool (for more info on it check my previous article Reset and Manage intelligent Platform Management remote board article).
With ipmi hardware health info could be received straight from the ILO / IDRAC / HPMI of the server. However as often the Admin-Lan of the server is in a seperate DMZ secured network and available via only a certain set of routed IPs, ipmitool can't be used.

So what are the other options to use to implement Linux Server Hardware Monitoring?

The tools to use are perhaps many but I know of two which gives you most of the information you ever need to have a prelimitary hardware damage warning system before the crash, these are:

1. smartmontools (smartd)

Smartd is part of smartmontools package which contains two utility programs (smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology system (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks.

Disk monitoring is handled by a special service the package provides called smartd that does query the Hard Drives periodically aiming to find a warning signs of hardware failures.
The downside of smartd use is that it implies a little bit of extra load on Hard Drive read / writes and if misconfigured could reduce the the Hard disk life time.

linux:~# /usr/sbin/smartctl -a /dev/sdb2
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-5-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: KINGSTON SA400S37240G
Serial Number: 50026B768340AA31
LU WWN Device Id: 5 0026b7 68340aa31
Firmware Version: S1Z40102
User Capacity: 240,057,409,536 bytes [240 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Thu Apr 30 14:05:01 2020 EEST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 10) minutes.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0032 100 100 000 Old_age Always – 100
9 Power_On_Hours 0x0032 100 100 000 Old_age Always – 2820
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always – 21
148 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 0
149 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 0
167 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 0
168 Unknown_Attribute 0x0012 100 100 000 Old_age Always – 0
169 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 0
170 Unknown_Attribute 0x0000 100 100 010 Old_age Offline – 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always – 0
173 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 0
181 Program_Fail_Cnt_Total 0x0032 100 100 000 Old_age Always – 0
182 Erase_Fail_Count_Total 0x0000 100 100 000 Old_age Offline – 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always – 0
192 Power-Off_Retract_Count 0x0012 100 100 000 Old_age Always – 16
194 Temperature_Celsius 0x0022 034 052 000 Old_age Always – 34 (Min/Max 19/52)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always – 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always – 0
218 Unknown_Attribute 0x0032 100 100 000 Old_age Always – 0
231 Temperature_Celsius 0x0000 097 097 000 Old_age Offline – 97
233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always – 2104
241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always – 1857
242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always – 1141
244 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 32
245 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 107
246 Unknown_Attribute 0x0000 100 100 000 Old_age Offline – 15940
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
Selective Self-tests/Logging not supported

2. hddtemp

Usually if smartd is used it is useful to also use hddtemp which relies on smartd data.
The hddtemp program monitors and reports the temperature of PATA, SATA
or SCSI hard drives by reading Self-Monitoring Analysis and Reporting
Technology (S.M.A.R.T.) information on drives that support this feature.

linux:~# /usr/sbin/hddtemp /dev/sda1
/dev/sda1: Hitachi HDS721050CLA360: 31°C
linux:~# /usr/sbin/hddtemp /dev/sdc6
/dev/sdc6: KINGSTON SV300S37A120G: 25°C
linux:~# /usr/sbin/hddtemp /dev/sdb2
/dev/sdb2: KINGSTON SA400S37240G: 34°C
linux:~# /usr/sbin/hddtemp /dev/sdd1
/dev/sdd1: WD Elements 10B8: S.M.A.R.T. not available

3. lm-sensors / i2c-tools

Lm-sensors is a hardware health monitoring package for Linux. It allows you
to access information from temperature, voltage, and fan speed sensors.
i2c-tools was historically bundled in the same package as lm_sensors but has been seperated cause not all hardware monitoring chips are I²C devices, and not all I²C devices are hardware monitoring chips.

The most basic use of lm-sensors is with the sensors command

linux:~# sensors
i350bb-pci-0600
Adapter: PCI adapter
loc1: +55.0 C (high = +120.0 C, crit = +110.0 C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0: +28.0 C (high = +78.0 C, crit = +88.0 C)
Core 0: +26.0 C (high = +78.0 C, crit = +88.0 C)
Core 1: +28.0 C (high = +78.0 C, crit = +88.0 C)
Core 2: +28.0 C (high = +78.0 C, crit = +88.0 C)
Core 3: +28.0 C (high = +78.0 C, crit = +88.0 C)

On CentOS Linux useful tool is also lm_sensors-sensord.x86_64 – A Daemon that periodically logs sensor readings to syslog or a round-robin database, and warns of sensor alarms.

In Debian Linux there is also the psensors-server (an HTTP server providing JSON Web service which can be used by GTK+ Application to remotely monitor sensors) useful for developers
psesors-server

If you have a Xserver installed on the Server accessed with Xclient or via VNC though quite rare,
You can use xsensors or Psensor – a GTK+ (Widget Toolkit for creating Graphical User Interface) application software.

With this 3 tools it is pretty easy to script one liners and use the Zabbix UserParameters functionality to send hardware report data to a Company's Zabbix Sserver, though Zabbix has already some templates to do so in my case, I couldn't import this templates cause I don't have Zabbix Super-Admin credentials, thus to work around that a sample work around is use script to monitor for higher and critical considered temperature.
Here is a tiny sample script I came up in 1 min time it can be used to used as 1 liner UserParameter and built upon something more complex.

SENSORS_HIGH=`sensors | awk '{ print $6 }'| grep '^+' | uniq`;
SENSORS_CRIT=`sensors | awk '{ print $9 }'| grep '^+' | uniq`; ;SENSORS_STAT=`sensors|grep -E 'Core\s' | awk '{ print $1" "$2" "$3 }' | grep "$SENSORS_HIGH|$SENSORS_CRIT"`;
if [ ! -z $SENSORS_STAT ]; then
echo 'Temperature HIGH';
else
echo 'Sensors OK';
fi
Of course there is much more sophisticated stuff to use for monitoring out there

Below script can be easily adapted and use on other Monitoring Platforms such as Nagios / Munin / Cacti / Icinga and there are plenty of paid solutions, but for anyone that wants to develop something from scratch just like me I hope this
article will be a good short introduction.
If you know some other Linux hardware monitoring tools, please share.

Tags: Adapter, around, Auto Offline Data Collection Disabled, awk, CPU, data, developers, Disk, Extended, firmware version, hard drives, hardware, hardware health, health, information, ISA, linux?, Monitoring, Monitoring Linux, nagios, package, pci, script, sensors, Short, software, system hardware, temperature, zabbix, Zabbix Userparameter
Posted in Linux, Monitoring, System Administration | No Comments »

Linux Local User Accounts Password Security policies Hardening – Set Password expiry, password quality, limit repatead access attempts, add directionary check, increase logged history command size

Friday, April 10th, 2020

For everyone that has to to manage Linux servers which are holding thousand of users for example UNIX Jump hosts (hop stations) or Free Shells, VPS shared developer hosts etc. setting a good password policies is a basis for proprietary confidential data protection .
Password expiry in jump host machines that are using LDAP authentication to grant access to users, 2 factor authentication or passwordless authentication policies is not a scope of this article, as they are handled by the LDAP authentication service sssd daemon on the login Unix host.
However there are many machines nowadays holding shared local users and setting a good password expiry policy on them is an essential to guarantee less probability of some external unathenticated intruder to be able to break in with some old user account of an employee that is no longer working for the company. Even for home brew Linux machines setting a password policy is a must if you don't want to end up some day with a strange logins from Asia unknown IPs and at a worst case a rooted machine …

1. Change user password / passphrases at least one every 90 days

Perhaps the best practice of local password security, practiced in most IT companies like IBM, Hewlett Packard, SAP, Amazon is to have a password policy expiry of 3 months this is done to fulfill the recommended PCI security standards recomendations.
A more paranoid ones perhaps prefer even shorter password expiry spans like 1 month but I think this is a bit too much and 3 months of local existing user password expiry is in the middle of too permissive and too restrictive pwd approach.
It is a very good security practice to set a password complexity and length
Control over password expiry on Linux is handled from /etc/login.defs, default settings for password policy is to never expire, below is a configuration from my Debian Linux.

PASS_MAX_DAYS 99999
PASS_MIN_DAYS 0
PASS_WARN_AGE 7

Here is /etc/login.defs that is a good security practice to implement on RHEL or CentOS

PASS_MAX_DAYS 90
PASS_MIN_DAYS 0
PASS_MIN_LEN 10
PASS_WARN_AGE 14

One note to make here is PASS_MAX_DAYS is not understood on .deb based Linuxes but will be understand by the OS only on RPM distros RHEL / Fedora etc.

Config options meaning is:

PASS_MIN_LEN – Minimum password length
PASS_MIN_DAYS – The minimum number of days allowed between password changes. Any password changes attempted sooner than this will be rejected. If not specified, -1 will be assumed (which disables the
restriction).
PASS_WARN_AGE – The number of days warning given before a password expires. A zero means warning is given only upon the day of expiration, a negative value means no warning is given. If not specified, no
warning will be provided.

Getting the user set user expiry date is done with the chage cmd

root@linux:~# chage -l hipo
Last password change : окт 17, 2017
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
root@linux:~#

root@linux:/home/test# chage -l test
Last password change : окт 07, 2017
Password expires : яну 05, 2018
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 90
Number of days of warning before password expires : 7
root@linux#

To set a certain special user, to lets say non-privileged user used by sysadmin to never expire:

root@linux:~# chage -M 99999 username

2. Increase requirements for password quality

This is done through the libpwquality system library that is present on virtually any modern fresh new GNU / Linux distribution install.
The paswrod requirements to authenticate access to system is configured via /etc/security/pwqualtity.conf or for more verbose user specific pwd authentication requirements with rules to be set in /etc/security/pwquality.conf.d/*.conf.
Many things could be configured via this simple name = value config just to mention few

minlen – minimum lenght of changing passowrd
minclass – the minimum amount of characters required for new pass (digit, uppercase, lowercase, others).
maxrepeat – maximum number of same repeatable consecutive characters in new set pass.
dcredit – require at least one digit in the new pass

A good set of configuration options one could benit from is as follows:

minlen = 10
dcredit = -1
lcredit = -1
ucredit = -1
ocredit = -1
minclass = 2
maxrepeat = 4
minclass = 4
maxclassrepeat = 0

For full supported capabitilies of libpwquality check out man pwquality.conf

3. Limit repeated access attempts by locking out the user ID if more than 6 unsuccesful login

In Red Hat Enterprise Linux 7 and CentOS the pam_faillock PAM module allows system administrators to lock out user accounts after a specified number of failed attempts. Limiting user login attempts serves mainly as a security measure that aims to prevent possible brute force attacks targeted to obtain a user's account password.

With the pam_faillock module, failed login attempts are stored in a separate file for each user in the /var/run/faillock directory.
Locking account if a number of unsuccessful login attempts are made is a very good security practice as this will block long lasting attempts to brute force local user credentials.

To do so add the following line immediately before the pam_unix.so statement in the AUTH section of /etc/pam.d/system-auth and /etc/pam.d/password-auth:

auth required pam_faillock.so preauth silent deny=6 unlock_time=1800 fail_interval=900

Now try to ssh to the machine. After the number of 6 failed attemp threshold is reached, user account will be locked.
In /var/log/secure the default location where RHEL / CentOS / Fedora and other RPM based distros store their login success and failure messages you should get a message like:

Aug 15 10:40:43 linux sshd[29038]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.43.135.1 user=hipo
Aug 15 10:40:43 linux sshd[29038]: pam_faillock(sshd:auth): Consecutive login failures for user deepak account temporarily locked

4. Set blockout user lockout duration to 30 minutes

Once a user is blocked after a number of failure attempts, it is normal to block that user to some reasonable duration like 30 minutes in most cases, however some paranoid administrators
prefer to block the user forever until the user contacts and requests the administrator to unlock him his "by mistake" blocked user by the login policies.

To automatically set a clerance of blocked user, add the following line immediately after the pam_unix.so statement in the AUTH section of /etc/pam.d/system-auth and /etc/pam.d/password-auth:

auth [default=die] pam_faillock.so authfail deny=6 unlock_time=1800 fail_interval=900

And add the following line immediately before the pam_unix.so statement in the ACCOUNT section of /etc/pam.d/system-auth and /etc/pam.d/password-auth:
account required pam_faillock.so

5. Check and reset fail lock number attempts from user

To check failure counts from user

root@linux:~# pam_tally –user hipo
Login Failures Latest failure From
hipo 3 08/28/19 22:00:50 10.55.135.3

To reset the faillog counter

root@linux:~# pam_tally2 –user hipo –reset

To check now the login failure counter

root@linux:~# pam_tally2 –user hipo
User hipo (1000) has 0

P.S. You can use also faillock instead of pam_tally

6. Check user new set passwords against dictionary with pam_pwquality.so

pam_pwquality – is the PAM module to perform password quality checking

To have it installed and possible to use on a Linux system you will need to have some kind of variation of the pam_pwquality package installed, for example on Debian Linux this is provided by libpam-pwquality package.
Default Debian installs does not have it installed so to have it do the usual:

debian:~# apt-get install –yes libpam-pwquality

Most RPM based distributions RHEL / Fedora etc. comes preinstalled with it, so it can be straight used.

To get an idea about what kind of checks it does on new password set, here is an extract from man pam_pwquality

      The action of this module is to prompt the user for a password and check its strength against a system dictionary and a set of rules for identifying poor choices.
       The first action is to prompt for a single password, check its strength and then, if it is considered strong, prompt for the password a second time (to verify that it was typed correctly on the
       first occasion). All being well, the password is passed on to subsequent modules to be installed as the new authentication token.
       The checks for strength are:
       Palindrome
           Is the new password a palindrome?
       Case Change Only
           Is the new password the the old one with only a change of case?
       Similar
           Is the new password too much like the old one? This is primarily controlled by one argument, difok which is a number of character changes (inserts, removals, or replacements) between the old
           and new password that are enough to accept the new password.
       Simple
           Is the new password too small? This is controlled by 6 arguments minlen, maxclassrepeat, dcredit, ucredit, lcredit, and ocredit. See the section on the arguments for the details of how these
           work and there defaults.
       Rotated
           Is the new password a rotated version of the old password?
       Same consecutive characters
           Optional check for same consecutive characters.
       Too long monotonic character sequence
           Optional check for too long monotonic character sequence.
       Contains user name
           Check whether the password contains the user's name in some form.
       Dictionary check
           The Cracklib routine is called to check if the password is part of a dictionary.
       These checks are configurable either by use of the module arguments or by modifying the /etc/security/pwquality.conf configuration file. The module arguments override the settings in the
       configuration file.

The benefit you will get once it is installed is the new Improved password enforcement policies that every sysadmin or security expert needs, below is output of libpam pwquality security strict policies in action.

[root@host ~]# passwd build
Changing password for user build.
New password: @teztpassy1
BAD PASSWORD: The password contains less than 1 uppercase letters
New password: @teztPassy
BAD PASSWORD: The password contains less than 1 digits
New password: teztPassy1
BAD PASSWORD: The password contains less than 1 non-alphanumeric characters
passwd: Have exhausted maximum number of retries for service

Once a strict password is given that is matching the enforced password security policy password will have to be repeated 3 times and not 2 times as the usual passwd user command does.

[root@host ~]# passwd build
Changing password for user build.
New password: @teztPassy1
Retype new password: @teztPassy1
passwd: all authentication tokens updated successfully.

7. Set User Session Timeout in 15 minutes

Another good security practice is to set the SSH logged in users active sessions to disconnect after lets say 20 minutes (1200 seconds).
This is done by using the TMOUT embedded variable in bash shell.
To give it a try how it works on any of your active bash session export TMOUT variable to 300 secs.

export TMOUT=300

Keep the shell opened for 5 minutes without any activity and you'll get automatically disconnected.
TMOUT variable can be set as a default for all users through placing it in /etc/bashrc or /etc/profile.
or only to certain users by appending it to selected users ~/.bashrc or ~/.bash_profile.
However as some of the most advanced users could notice its existence by issuing env command and checking the current environment variables
and decide to get rid of it with unset TMOUT it is good idea to make the variable readonly to do so append in /etc/bashrc

TMOUT=900
readonly TMOUT
export TMOUT

Now the risk that logged in user forgot his console active on a public place and someone got access to his account through that is minimized, however
for some users that might be quite annoying as any forgot console will get dropped off after short time and that could be quite irritating ..

8. Increase size of logged User commands history HISTFILE and HISTFILESIZE variables

There are 2 embedded bash variables that controls the size of logged commands after user logs in to their bash shell this is HISTFILE and HISTFILESIZE
The difference between HISTSIZE and HISTFILESIZE is that HISTSIZE limits the number of commands shown by the command history while HISTFILESIZE limits the number of commands which can be saved in $HISTFILE.

When one exits the bash, if there are more than $HISTSIZE number of commands which have been executed in the single bash session, the contents of $HISTFILE will be replaced by the $HISTSIZE number of commands. If there are less than or equal to $HISTSIZE number of commands in the bash session, these commands will be appended to $HISTFILE as long as $HISTFILESIZE permits.

The number of commands to be remembered in the history can be specified by the environment variable HISTSIZE.
Below is example of some values that will log up to 20000 of commands run by the user, this is much desired as the default logged in commands is only 1000 and if you tend to run many commands daily and this history size is exceeded in a couple of days.

export HISTSIZE=2000
export HISTFILESIZE=20000

To make this permanent for all server existing users add the records to /etc/profile or /etc/bashrc to make custom HISTORYFILESIZE add it to $HOME/.bashrc or ~/.bash_profile.

To apply (test) new exported changes either relogin to the shell or run:

linux~:# source /etc/profile

You can run shopt command to force to append the history commands to $HISTFILE even though there are more than $HISTSIZE number of commands which have been executed in the bash session:

# shopt -s histappend

9. Display TIMESTAMP in command history using HISTTIMEFORMAT

The normal way history command does show the history of previously user run commands is in line ordered form. This is not very useful for auditing purpose.
If for example you need to track a certain user activity as you cannot exactly see when the history logged command was issued.

Here is example what I mean:

linux~:# history|head -n 5
7 tail -f /var/log/apache2/error.log|grep -i seg
8 php -v
9 /usr/bin/php5.6 -v
10 w
11 tail -f /var/log/apache2/php_error.log

To set a basic date-month-year time timestamp formatting you can append

linux:~# export HISTTIMEFORMAT=’%F %T ‘

in /etc/profile /etc/bash.bashrc to apply it for all users or
for local users to ~/.bashrc ~/.bash_profile

The resulted formatting will be:

linux:~# history| tail -n 5
31 2020-05-07 10:06:34 sudo su – root
32 2020-05-07 10:06:34 w
33 2020-05-07 10:06:34 sudo su – root
34 2020-05-07 10:06:36 history
35 2020-05-07 10:06:45 history| tail -n 5

To show printed the exact day of week and written with letters you instead of numeric:

linux:~# export HISTTIMEFORMAT="%a %h %d – %r "

history log output results then will be as so:

linux:~# history |tail -n 2
101 Monday May 07 – 07:39:53 PM top
102 Monday May 22 – 07:39:54 PM history
That's all folks Enjoy !

Tags: attempts, authentication, change user, checking, configuration file, configuration options, good security, How to, installed, Login Failures Latest, logins, maximum number, middle, New, passwordless, root linux, Security Hardening, security practice, sshd, system administrators, uid, user accounts, username
Posted in Computer Security, Linux, Remote System Administration, System Administration | No Comments »

How to Import Remove List archive signing keys on CentOS / RHEL / Fedora RPM based Linux distributions

Wednesday, April 8th, 2020

how-to-import-remove-list-archiving-signing-keys-on-CentOS-RHEL-Fedora-rpm-based-Linux-distros-package
If you plan to build and distribute own RPMs securely, it is strongly recommended that all custom RPMs are signed using GNU Privacy Guard (GPG). Generating GPG keys and building GPG-signed packages matching it.
Hence, If you have to deal with some of the RPM based package management Linux distribution like CentOS / RHEL / Fedora etc. you will sooner or later end up in a situation where some of the archive signing keys for a package provided by some of the repositories is missing or it is not matching the keys provided for the RPM repo.

As a result you will be unable to install some package like lets say zabbix-sender or you won't be able to update a certain package to the latest available version, because the Archive Signing key is not found.
The usual naming for a RPM file with a GPG key in is YOUR-RPM-GPG-KEY.

A typical PGP Public key file content looks something like this:

—–BEGIN PGP PUBLIC KEY BLOCK—–
Version: GnuPG v1.0.0 (GNU/Linux)
Comment: For info see http://www.gnupg.org
mQGiBDfqVEqRBADBKr3Bl6PO8BQ0H8sJoD6p9U7Yyl7pjtZqioviPwXP+DCWd4u8
HQzcxAZ57m8ssA1LK1Fx93coJhDzM130+p5BG9mYSPShLabR3N1KXdXAYYcowTOM
GxdwYRGr1Spw8QydLhjVfU1VSl4xt6bupPbFJbyjkg5Z3P7BlUOUJmrx3wCgobNV
EDGaWYJcch5z5B1of/41G8kEAKii6q7Gu/vhXXnLS6m15oNnPVybyngiw/23dKjS
ti/PYrrL2J11P2ed0x7zm8v3gLrY0cue1iSba+8glY+p31ZPOr5ogaJw7ZARgoS8
BwjyRymXQp+8Dete0TELKOL2/itDOPGHW07SsVWOR6cmX4VlRRcWB5KejaNvdrE5
4XFtOd04NMgWI63uqZc4zkRa+kwEZtmbz3tHSdWCCE+Y7YVP6IUf/w6YPQFQriWY
FiA6fD10eB+BlIUqIw80EqjsBKmCwvKkn4jg8kibUgj4/TzQSx77uYokw1EqQ2wk
OZoaEtcubsNMquuLCMWijYhGBBgRAgAGBQI36lRyAAoJECGRgM3bQqYOhyYAnj7h
VDY/FJAGqmtZpwVp9IlitW5tAJ4xQApr/jNFZCTksnI+4O1765F7tA==
=3AHZ
—–END PGP PUBLIC KEY BLOCK—–

The usual naming for a RPM file with a GPG key in is YOUR-RPM-GPG-KEY

1. List RPM gpg keys installed on system

To list all the installed RPM gpg keys on the system do:

rpm -q gpg-pubkey

To get a list of the number of installed keys with verbose info with key description::

rpm -qa gpg-pubkey –qf "%{version}-%{release} %{summary}\n"|wc -l

rpm -qa gpg-pubkey –qf "%{version}-%{release} %{summary}\n"
fdb19c98-56fd6333 gpg(Fedora 25 Primary (25) <fedora-25-primary@fedoraproject.org>)
7fac5991-4615767f gpg(Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>)
64dab85d-57d33e22 gpg(Fedora 26 Primary (26) <fedora-26-primary@fedoraproject.org>)
fa7a179a-562bcd6e gpg(RPM Fusion nonfree repository for Fedora (25) <rpmfusion-buildsys@lists.rpmfusion.org>)
6806a9cb-562bce39 gpg(RPM Fusion free repository for Fedora (25) <rpmfusion-buildsys@lists.rpmfusion.org>)
d38b4796-570c8cd3 gpg(Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>)

[root@host ~:]# rpm -q gpg-pubkey –qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-f4a80eb5-53a7ff4b gpg(CentOS-7 Key (CentOS 7 Official Signing Key) )
gpg-pubkey-b6792c39-53c4fbdd gpg(CentOS-7 Debug (CentOS-7 Debuginfo RPMS) )
gpg-pubkey-8fae34bd-538f1e51 gpg(CentOS-7 Testing (CentOS 7 Testing content) )

To list all OS installed gpg keys do:

[user@host ~:]$ $ rpm -qa | grep -i gpg
gpg-pubkey-db42a60e-37ea5438

2. Import RPM-GPG-KEY

A new key be it official archive keys issued from Fedora or a custom own build RPM package can be imported Redhat Package Manager like so:

[root@host ~:]# rpm –import RPM-GPG-KEY

It is possible to also import multiple GPG signature keys, for example on CentOS the usual path containg keys is /etc/pki/rpm-gpg/ to import all of the contained files there:

[root@host ~:]# rpm –import /etc/pki/rpm-gpg/*

3. Check package with imported gpg arch key

Once the RPM-GPG-KEY is imported you can compare whether a RPM package matches with the key signature.

[root@host ~:]# rpm –checksig package-1.3-3.src.rpm

[root@host ~:]# rpm –checksig xtoolwait-1.3-3.src.rpm
package-1.3-3.src.rpm: (sha1) dsa sha1 md5 gpg OK

4. Remove RPM installed arch key

If you have installed some gpg arch. key by mistake and you need to remove it:

[root@host ~:]#rpm -e gpg-pubkey-b6792c39-53c4fbdd

To make sure it is remove do a Listing once again signing archive keys, it should not show anymore:

[root@host ~:]# rpm -q gpg-pubkey –qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-f4a80eb5-53a7ff4b gpg(CentOS-7 Key (CentOS 7 Official Signing Key) )
gpg-pubkey-8fae34bd-538f1e51 gpg(CentOS-7 Testing (CentOS 7 Testing content) )

Tags: archive signing key, centos linux, fedora, GPG, remove gpg rpm key, rhel, rpm import
Posted in Linux, SuSE Linux, System Administration | No Comments »

Linux: How to sync user files with rsync on every root user logout – useful for Clusters

Monday, April 6th, 2020

rsync-how-to-transfer-locally-remotely-and-synchronize

For those who run Redhat Linux Enterprise servers / Suse Enterprise Linux or any other Linux distribution with some kind of Application clustering such as Pacemaker ( PCM ) / Corosync and OpenSSH
which are supposed to be always identical in terms of configurations files in /etc/ configurations of haproxy /etc/hosts* below are few steps to follow to always keep files from server1 to server2 the same.

1. Create rsyncuser on 1st machine and second machine

– Create user with useradd on node1 and node2 and generate RSA SSH key pair with ssh-keygen command.

– Copy id_rsa.pub from local server1 host to remote host server2

– Test and confirm remote passwordless authentication is working

– To make rsyncuser be able to copy files with root privileges

echo ‘rsyncuser ALL=NOPASSWD:/usr/bin/rsync’ >> /etc/sudoers

For indepth descriptions of above steps please check my previous artice
Rsync copy files with root privileges between servers with root superuser account disabled.

2. CREATE sync-commands file desired to be executed on logout from the system of the user into
some file like /root/sync_check.sh

root@host # vim /root/sync_check.sh
#!/bin/sh
# sync logout script
echo "###################checking sync Line1 and Line2 now ################"
rsync –dry-run –delete -avze 'ssh -i /home/rsyncuser/.ssh/id_rsa' '–rsync-path=sudo rsync' /etc/hosts.deny rsyncuser@server2:/etc/hosts.deny
rsync –dry-run –delete -avze 'ssh -i /home/rsyncuser/.ssh/id_rsa' '–rsync-path=sudo rsync' /etc/haproxy/certs/*.* rsyncuser@server2:/etc/haproxy/certs/
rsync –dry-run –delete -avze 'ssh -i /home/rsyncuser/.ssh/id_rsa' '–rsync-path=sudo rsync' /etc/haproxy/*.cfg rsyncuser@server2:/etc/haproxy/
echo "################### done sync_check.sh ################"

3. Edit root lougout-script /root/.bash_logout

root@host # vi /root/.bash_logout
== Append the following lines:
# ~ / .bash_logout
echo "################### run root bash_logout ################"
/root/sync_check.sh
echo "################### root logged out ################"
== Save and close the file.

– To make the rsync to sync some specific files such as user's personal home directory etc. and hence you need it to be user wide you can add the file to be executed for all users via
/etc/bashrc or /etc/profile

What you will have as a result is an an override copy of /etc/hosts.deny /etc/haproxy/certs* /etc/haproxy/*.cfg on each and every root logout.
The meaning of all this is to make sure this 3 files always stay identical, to prevent someone to mess up some configuration by mistake and on next haproxy standby / reload to not load with unworking config due to the user stupidity.

That's all folks. Test it and be happy 🙂

Tags: Account, ALL, and, ANY, application, are, authentication, bashrc, Below, between, bin, certs, cfg, check, checking, copy files, logout, rsync, script, synchronize user data on logout
Posted in Everyday Life, Linux, System Administration | No Comments »

Find when cron.daily cron.weekly and cron.monthly run on Redhat / CentOS / Debian Linux and systemd-timers

Wednesday, March 25th, 2020

The problem – Apache restart at random times

I've noticed today something that is occuring for quite some time but was out of my scope for quite long as I'm not directly involved in our Alert monitoring at my daily job as sys admin. Interestingly an Apache HTTPD webserver is triggering alarm twice a day for a short downtime that lasts for 9 seconds.

I've decided to investigate what is triggering WebServer restart in such random time and investigated on the system for any background running scripts as well as reviewed the system logs. As I couldn't find nothing there the only logical place to check was cron jobs.
The usual

crontab -u root -l

Had no configured cron jobbed scripts so I digged further to check whether there isn't cron jobs records for a script that is triggering the reload of Apache in /etc/crontab /var/spool/cron/root and /var/spool/cron/httpd.
Nothing was found there and hence as there was no anacron service running but /usr/sbin/crond the other expected place to look up for a trigger even was /etc/cron*

1. Configured default cron execution times, every day, every hour every month

# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 feb 27 10:54 /etc/cron.d/
drwxr-xr-x 2 root root 4096 dec 27 10:55 /etc/cron.daily/
drwxr-xr-x 2 root root 4096 dec 7 23:04 /etc/cron.hourly/
drwxr-xr-x 2 root root 4096 dec 7 23:04 /etc/cron.monthly/
drwxr-xr-x 2 root root 4096 dec 7 23:04 /etc/cron.weekly/

After a look up to each of above directories, finally I found the very expected logrorate shell script set to execute from /etc/cron.daily/logrotate and inside it I've found after the log files were set to be gzipped and moved to execute WebServer restart with:

systemctl reload httpd

My first reaction was to ponder seriously why the script is invoking systemctl reload httpd instead of the good oldschool

apachectl -k graceful

But it seems on Redhat and CentOS since RHEL / CentOS version 6.X onwards systemctl reload httpd is supposed to be identical and a substitute for apachectl -k graceful.
Okay the craziness of innovation continued as obviously the reload was causing a Downtime to be visible in the Zabbix HTTPD port Monitoring graph …
Now as the problem was identified the other logical question poped up how to find out what is the exact timing scheduled to run the script in that unusual random times each time ??

2. Find out cron scripts timing Redhat / CentOS / Fedora / SLES

/etc/cron.{daily,monthly,weekly} placed scripts's execution method has changed over the years, causing a chaos just like many Linux standard things we know due to the inclusion of systemd and some other additional weird OS design changes. The result is the result explained above scripts are running at a strange unexpeted times … one thing that was intruduced was anacron – which is also executing commands periodically with a different preset frequency. However it is considered more thrustworhty by crond daemon, because anacron does not assume the machine is continuosly running and if the machine is down due to a shutdown or a failure (if it is a Virtual Machine) or simply a crond dies out, some cronjob necessery for overall set environment or application might not run, what anacron guarantees is even though that and even if crond is in unworking defunct state, the preset scheduled scripts will still be served.
anacron's default file location is in /etc/anacrontab.

A standard /etc/anacrontab looks like so:

[root@centos ~]:# cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command
1   5   cron.daily       nice run-parts /etc/cron.daily
7   25   cron.weekly       nice run-parts /etc/cron.weekly
@monthly 45   cron.monthly       nice run-parts /etc/cron.monthly

START_HOURS_RANGE : The START_HOURS_RANGE variable sets the time frame, when the job could started.
The jobs will start during the 3-22 (3AM-10PM) hours only.

cron.daily will run at 3:05 (After Midnight) A.M. i.e. run once a day at 3:05AM.
cron.weekly will run at 3:25 AM i.e. run once a week at 3:25AM.
cron.monthly will run at 3:45 AM i.e. run once a month at 3:45AM.

If the RANDOM_DELAY env var. is set, a random value between 0 and RANDOM_DELAY minutes will be added to the start up delay of anacron served jobs.
For instance RANDOM_DELAY equels 45 would therefore add, randomly, between 0 and 45 minutes to the user defined delay.

Delay will be 5 minutes + RANDOM_DELAY for cron.daily for above cron.daily, cron.weekly, cron.monthly config records, i.e. 05:01 + 0-45 minutes

A full detailed explanation on automating system tasks on Redhat Enterprise Linux is worthy reading here.

!!! Note !!! that listed jobs will be running in queue. After one finish, then next will start.

3. SuSE Enterprise Linux cron jobs not running at desired times why?

in SuSE it is much more complicated to have a right timing for standard default cron jobs that comes preinstalled with a service

In older SLES release /etc/crontab looked like so:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/
# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

As time of writting article it looks like:

SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root
#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1

This runs any scripts placed in /etc/cron.{hourly, daily, weekly, monthly} but it may not run them when you expect them to run.
/usr/lib/cron/run-crons compares the current time to the /var/spool/cron/lastrun/cron.{time} file to determine if those jobs need to be run.

For hourly, it checks if the current time is greater than (or exactly) 60 minutes past the timestamp of the /var/spool/cron/lastrun/cron.hourly file.

For weekly, it checks if the current time is greater than (or exactly) 10080 minutes past the timestamp of the /var/spool/cron/lastrun/cron.weekly file.

Monthly uses a caclucation to check the time difference, but is the same type of check to see if it has been one month after the last run.

Daily has a couple variations available – By default it checks if it is more than or exactly 1440 minutes since lastrun.
If DAILY_TIME is set in the /etc/sysconfig/cron file (again a suse specific innovation), then that is the time (within 15minutes) when daily will run.

For systems that are powered off at DAILY_TIME, daily tasks will run at the DAILY_TIME, unless it has been more than x days, if it is, they run at the next running of run-crons. (default 7days, can set shorter time in /etc/sysconfig/cron.)
Because of these changes, the first time you place a job in one of the /etc/cron.{time} directories, it will run the next time run-crons runs, which is at every 15mins (xx:00, xx:15, xx:30, xx:45) and that time will be the lastrun, and become the normal schedule for future runs. Note that there is the potential that your schedules will begin drift by 15minute increments.

As you see this is very complicated stuff and since God is in the simplicity it is much better to just not use /etc/cron.* for whatever scripts and manually schedule each of the system cron jobs and custom scripts with cron at specific times.

4. Debian Linux time start schedule for cron.daily / cron.monthly / cron.weekly timing

As the last many years many of the servers I've managed were running Debian GNU / Linux, my first place to check was /etc/crontab which is the standard cronjobs file that is setting the { daily , monthly , weekly crons }

debian:~# ls -ld /etc/cron.*
drwxr-xr-x 2 root root 4096 фев 27 10:54 /etc/cron.d/
drwxr-xr-x 2 root root 4096 фев 27 10:55 /etc/cron.daily/
drwxr-xr-x 2 root root 4096 дек 7 23:04 /etc/cron.hourly/
drwxr-xr-x 2 root root 4096 дек 7 23:04 /etc/cron.monthly/
drwxr-xr-x 2 root root 4096 дек 7 23:04 /etc/cron.weekly/

debian:~# cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin# Example of job definition:
# .—————- minute (0 – 59)
# | .————- hour (0 – 23)
# | | .———- day of month (1 – 31)
# | | | .——- month (1 – 12) OR jan,feb,mar,apr …
# | | | | .—- day of week (0 – 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 *   * * *   root cd / && run-parts –report /etc/cron.hourly
25 6   * * *   root   test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.daily )
47 6   * * 7   root   test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.weekly )
52 6   1 * *   root   test -x /usr/sbin/anacron || ( cd / && run-parts –report /etc/cron.monthly )

What above does is:

– Run cron.hourly once at every hour at 1:17 am
– Run cron.daily once at every day at 6:25 am.
– Run cron.weekly once at every day at 6:47 am.
– Run cron.monthly once at every day at 6:42 am.

As you can see if anacron is present on the system it is run via it otherwise it is run via run-parts binary command which is reading and executing one by one all scripts insude /etc/cron.hourly, /etc/cron.weekly , /etc/cron.mothly .

anacron – few more words

Anacron is the canonical way to run at least the jobs from /etc/cron.{daily,weekly,monthly) after startup, even when their execution was missed because the system was not running at the given time. Anacron does not handle any cron jobs from /etc/cron.d, so any package that wants its /etc/cron.d cronjob being executed by anacron needs to take special measures.

If anacron is installed, regular processing of the /etc/cron.d{daily,weekly,monthly} is omitted by code in /etc/crontab but handled by anacron via /etc/anacrontab. Anacron's execution of these job lists has changed multiple times in the past:

debian:~# cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
HOME=/root
LOGNAME=root
# These replace cron's entries
1   5   cron.daily   run-parts –report /etc/cron.daily
7   10   cron.weekly   run-parts –report /etc/cron.weekly
@monthly   15   cron.monthly   run-parts –report /etc/cron.monthly

In wheezy and earlier, anacron is executed via init script on startup and via /etc/cron.d at 07:30. This causes the jobs to be run in order, if scheduled, beginning at 07:35. If the system is rebooted between midnight and 07:35, the jobs run after five minutes of uptime.
In stretch, anacron is executed via a systemd timer every hour, including the night hours. This causes the jobs to be run in order, if scheduled, beween midnight and 01:00, which is a significant change to the previous behavior.
In buster, anacron is executed via a systemd timer every hour with the exception of midnight to 07:00 where anacron is not invoked. This brings back a bit of the old timing, with the jobs to be run in order, if scheduled, beween 07:00 and 08:00. Since anacron is also invoked once at system startup, a reboot between midnight and 08:00 also causes the jobs to be scheduled after five minutes of uptime.
anacron also didn't have an upstream release in nearly two decades and is also currently orphaned in Debian.

As of 2019-07 (right after buster's release) it is planned to have cron and anacron replaced by cronie.

cronie – Cronie was forked by Red Hat from ISC Cron 4.1 in 2007, is the default cron implementation in Fedora and Red Hat Enterprise Linux at least since Version 6. cronie seems to have an acive upstream, but is currently missing some of the things that Debian has added to vixie cron over the years. With the finishing of cron's conversion to quilt (3.0), effort can begin to add the Debian extensions to Vixie cron to cronie.

Because cronie doesn't have all the Debian extensions yet, it is not yet suitable as a cron replacement, so it is not in Debian.

5. systemd-timers – The new crazy systemd stuff for script system job scheduling

Timers are systemd unit files with a suffix of .timer. systemd-timers was introduced with systemd so older Linux OS-es does not have it.
Timers are like other unit configuration files and are loaded from the same paths but include a [Timer] section which defines when and how the timer activates. Timers are defined as one of two types:

Realtime timers (a.k.a. wallclock timers) activate on a calendar event, the same way that cronjobs do. The option OnCalendar= is used to define them.
Monotonic timers activate after a time span relative to a varying starting point. They stop if the computer is temporarily suspended or shut down. There are number of different monotonic timers but all have the form: OnTypeSec=. Common monotonic timers include OnBootSec and OnActiveSec.

For each .timer file, a matching .service file exists (e.g. foo.timer and foo.service). The .timer file activates and controls the .service file. The .service does not require an [Install] section as it is the timer units that are enabled. If necessary, it is possible to control a differently-named unit using the Unit= option in the timer’s [Timer] section.
systemd-timers is a complex stuff and I'll not get into much details but the idea was to give awareness of its existence for more info check its manual man systemd.timer

Its most basic use is to list all configured systemd.timers, below is from my home Debian laptop

debian:~# systemctl list-timers –all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2020-03-24 23:33:58 EET 18s left Tue 2020-03-24 23:31:28 EET 2min 11s ago laptop-mode.timer lmt-poll.service
Tue 2020-03-24 23:39:00 EET 5min left Tue 2020-03-24 23:09:01 EET 24min ago phpsessionclean.timer phpsessionclean.service
Wed 2020-03-25 00:00:00 EET 26min left Tue 2020-03-24 00:00:01 EET 23h ago logrotate.timer logrotate.service
Wed 2020-03-25 00:00:00 EET 26min left Tue 2020-03-24 00:00:01 EET 23h ago man-db.timer man-db.service
Wed 2020-03-25 02:38:42 EET 3h 5min left Tue 2020-03-24 13:02:01 EET 10h ago apt-daily.timer apt-daily.service
Wed 2020-03-25 06:13:02 EET 6h left Tue 2020-03-24 08:48:20 EET 14h ago apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2020-03-25 07:31:57 EET 7h left Tue 2020-03-24 23:30:28 EET 3min 11s ago anacron.timer anacron.service
Wed 2020-03-25 17:56:01 EET 18h left Tue 2020-03-24 17:56:01 EET 5h 37min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service

8 timers listed.

N ! B! If a timer gets out of sync, it may help to delete its stamp-* file in /var/lib/systemd/timers (or ~/.local/share/systemd/ in case of user timers). These are zero length files which mark the last time each timer was run. If deleted, they will be reconstructed on the next start of their timer.

Summary

In this article, I've shortly explain logic behind debugging weird restart events etc. of Linux configured services such as Apache due to configured scripts set to run with a predefined scheduled job timing. I shortly explained on how to figure out why the preset default install configured cron jobs such as logrorate – the service that is doing system logs archiving and nulling run at a certain time. I shortly explained the mechanism behind cron.{daily, monthy, weekly} and its execution via anacron – runner program similar to crond that never misses to run a scheduled job even if a system downtime occurs due to a crashed Docker container etc. run-parts command's use was shortly explained. A short look at systemd.timers was made which is now essential part of almost every new Linux release and often used by system scripts for scheduling time based maintainance tasks.

Tags: checks, crontab, daily, EET, execution, form, last time, list, multiple times, Nothing, root root, running, sbin, scripts, shutdown, system startup, timer, today
Posted in Bash Scripting, Curious Facts, Linux, System Administration, Various | No Comments »

Check if server is Physical Bare Metal or a Virtual Machine and its type

Tuesday, March 17th, 2020

check-if-linux-operating-system-is-running-on-physical-bare-metal-or-virtual-machine

In modern times the IT employee system administrator / system engineer / security engineer or a developer who has to develop and test code remotely on UNIX hosts, we have to login to multiple of different servers located in separate data centers around the world situated in Hybrid Operating system environments running multitude of different Linux OSes. Often especially for us sysadmins it is important to know whether the remote machine we have SSHed to is physical server (Bare Metal) or a virtual machines running on top of different kind of Hypervisor node OpenXen / Virtualbox / Virtuosso / VMWare etc.

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?

1. Using the dmesg kernel log utility

The good old dmesg that is used to examine and control the kernel ring buffer detects plenty of useful information which gives you the info whether a server is Virtual or Bare Metal. It is present and accessible on every Linux server out there, thus using it is the best and simplest way to determine the OS system node type.

To grep whether a machine is Virtual and the Hypervisor type use:

nginx:~# dmesg | grep "Hypervisor detected"
[0.000000] Hypervisor detected: KVM

As you see above OS installed is using the KVM Virtualization technology.

An empty output of this command means the Remote OS is installed on a physical computer.

2. Detecting the OS platform the systemd way

Systemd along with the multiple over-complication of things that nearly all sysadmins (including me hate) so much introduced something useful in the fact of hostnamectl command
that could give you the info about the OS chassis platform.

root@pcfreak:~# hostnamectl status
Static hostname: pcfreak
Icon name: computer-desktop
Chassis: desktop
Machine ID: 02425d67037b8e67cd98bd2800002671
Boot ID: 34a83b9a79c346168082f7605c2f557c
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-5-amd64
Architecture: x86-64

Below is output of a VM running on a Oracle Virtualbox HV.

linux:~# hostnamectl status
Static hostname: ubuntuserver
Icon name: computer-vm
Chassis: vm
Machine ID: 2befe86cf8887ca098f509e457554beb
Boot ID: 8021c02d65dc46b1885afb25fddcf18c
Virtualization: oracle
Operating System: Ubuntu 16.04.1 LTS
Kernel: Linux 4.4.0-78-generic
Architecture: x86-64

3. Detect concrete container virtualization with systemd-detect-virt

Another Bare Metal or VM identify tool that was introducted some time ago by freedesktop project is systemd-detect-virt (usually command is part of systemd package).
It is useful to detect the exact virtualization on a systemd running OS systemd-detect-virt is capable to detect many type of Virtualization type that are rare like: IBM zvm S390 Z/VM, bochs, bhyve (a FreeBSD hypervisor), Mac OS's parallels, lxc (linux containers), docker containers, podman etc.

The output from the command is either none (if no virtualization is present or the VM Hypervisor Host type):

server:~# systemd-detect-virt
none

quake:~# systemd-detect-virt
oracle

4. Install and use facter to report per node facts

debian:~# apt-cache show facter|grep -i desc -A2
Description-en: collect and display facts about the system
Facter is Puppet’s cross-platform system profiling library. It discovers and
reports per-node facts, which are collected by the Puppet agent and are made
—
Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter
Tag: devel::lang:ruby, devel::library, implemented-in::ruby,
root@jeremiah:/home/hipo# apt-cache show facter|grep -i desc -A1
Description-en: collect and display facts about the system
Facter is Puppet’s cross-platform system profiling library. It discovers and
—
Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Homepage: https://github.com/puppetlabs/facter

– Install facter on Debian / Ubuntu / deb based Linux

# apt install facter –yes

– Install facter on RedHat / CentOS RPM based distros

# yum install epel-release
…

# yum install facter
…

– Install facter on OpenSuSE / SLES

# zypper install facter
…

Once installed on the system to find out whether the remote Operating System is Virtual:

# facter 2> /dev/null | grep virtual
is_virtual => false
virtual => physical

If the machine is a virtual machine you will get some different output like:

# facter 2> /dev/null | grep virtual
is_virtual => true
virtual => kvm

If you're lazy to grep you can use it with argument.

# facter virtual
physical

6. Use lshw and dmidecode (list hardware configuration tool)

If you don't have the permissions to install facter on the system and you can see whether lshw (list hardware command) is not already present on remote host.

# lshw -class system
storage-host
description: Computer
width: 64 bits
capabilities: smbios-2.7 vsyscall32
If the system is virtual you'll get an output similar to:
# lshw -class system
debianserver
description: Computer
product: VirtualBox
vendor: innotek GmbH
version: 1.2
serial: 0
width: 64 bits
capabilities: smbios-2.5 dmi-2.5 vsyscall32
configuration: family=Virtual Machine uuid=78B58916-4074-42E2-860F-7CAF39F5E6F5

Of course as it provides a verbosity of info on Memory / CPU type / Caches / Cores / Motherboard etc. virtualization used or not can be determined also with dmidecode / hwinfo and other tools that detect the system hardware this is described thoroughfully in my previous article Get hardware system info on Linux.

7. Detect virtualziation using virt-what or imvirt scripts

imvirt is a little script to determine several virtualization it is pretty similar to virt-what the RedHat own script for platform identification. Even though virt-what is developed for RHEL it is available on other distros, Fedoda, Debian, Ubuntu, Arch Linux (AUR) just like is imvirt.

installing both of them is with the usual apt-get / yum or on Arch Linux with yay package manager (yay -S virt-what) …

Once run the output it produces for physical Dell / HPE / Fujitsu-Siemens Bare Metal servers would be just empty string.

# virt-what
#

Or if the system is Virtual Machine, you'll get the type, for example KVM (Kernel-based Virtual Machine) / virtualbox / qemu etc.

#imvirt
Physical

Conclusion

It was explained how to do a simple check whether the server works on a physical hardware or on a virtual Host hypervisor. The most basic and classic way is with dmesg. If no access to dmesg is due to restrictions you can try the other methods for systemd enabled OSes with hostnamectl / systemd-detect-virt. Other means if the tools are installed or you have the permissions to install them is with facter / lshw or with virt-what / imvirt scripts.
There definitely perhaps much more other useful tools to grasp hardware and virtualization information but this basics could be useful enough for shell scripting purposes.
If you know other tools, please share.

Tags: Below, installed, kvm, Linux Check, oracle, servers, system administrator, system hardware, utility, virtual machines, virtualization
Posted in Linux, System Administration, Various | No Comments »

IBM TSM dsmc console client use for listing configured backups, checking set scheduled backups and backup and restore operations howto

Friday, March 6th, 2020

tsm-ibm-logo_tivoli-dsmc-console-client-listing-backups-create-backups-and-restore-on-linux-unix-windows

Creating a simple home based backup solution with some shell scripting and rsync is a common use. However as a sysadmin in a middle sized or large corporations most companies use some professional backup service such as IBM Tivoli Storage Manager TSM – recently IBM changed the name of the product to IBM Spectrum.

IBM TSM is a data protection platform that gives enterprises a single point of control and administration for backup and recovery that is used for Privare Clouds backup and other high end solutions where data criticality is top.
Usually in large companies TSM backup handling is managed by a separate team or teams as managing a large TSM infrastructure is quite a complex task, however my experience as a sysadmin show me that even if you don't have too much of indepth into tsm it is very useful to know how to manage at least basic Incremental backup operations such as view what is set to be backupped, set-up a new directory structure for backup, check the backup schedule configured, check what files are included and which excluded from the backup store etc.

TSM has multi OS support ans you can use it on most streamline Operating systems Windows / Mac OS X and Linux in this specific article I'll be talking concretely about backing up data with tsm on Linux, tivoli can be theoretically brought up even on FreeBSD machines via the Linuxemu BSD module and the 64-Bit Tivoli Storage Manager RPMs.
Therefore in this small article I'll try to give few useful operations for the novice admin that stumbles on tsm backupped server that needs some small maintenance.

1. Starting up the dsmc command line client

Nomatter the operating system on which you run it to run the client run:

# dsmc

tsm-check-backup-schedule-set-time

Note that usually dsmc should run as superuser so if you try to run it via a normal non-root user you will get an error message like:

[ user@linux ~]$ dsmc
ANS1398E Initialization functions cannot open one of the Tivoli Storage Manager logs or a related file: /var/tsm/dsmerror.log. errno = 13, Permission denied

Tivoli SM has an extensive help so to get the use basics, type help

tsm> help
1.0 New for IBM Tivoli Storage Manager Version 6.4
2.0 Using commands
2.1 Start and end a client command session
2.1.1 Process commands in batch mode
2.1.2 Process commands in interactive mode
2.2 Enter client command names, options, and parameters
2.2.1 Command name
2.2.2 Options
2.2.3 Parameters
2.2.4 File specification syntax
2.3 Wildcard characters
2.4 Client commands reference
2.5 Archive
2.6 Archive FastBack

Enter 'q' to exit help, 't' to display the table of contents,
press enter or 'd' to scroll down, 'u' to scroll up or
enter a help topic section number, message number, option name,
command name, or command and subcommand:

2. Listing files listed for backups

A note to make here is as in most corporate products tsm supports command aliases so any command supported described in the help like query, could be
abbreviated with its first letters only, e.g. query filespace tsm cmd can be abbreviated as

tsm> q fi

Commands can be run non-interactive mode also so if you want the output of q fi you can straight use:

tsm> dsmc q fi

tsm-check-included-excluded-files-q-file-if-backupped-list-backup-set-directories

This shows the directories and files that are set for backup creation with Tivoli.

3. Getting included and excluded backup set files

It is useful to know what are the exact excluded files from tsm set backup this is done with query inclexcl

tsm-check-excluded-included-files

4. Querying for backup schedule time

Tivoli as every other backup solution is creating its set to backup files in a certain time slot periods.
To find out what is the time slot for backup creation use;

tsm> q sched
Schedule Name: WEEKLY_ITSERV
Description: ITSERV weekly incremental backup
Schedule Style: Classic
Action: Incremental
Options:
Objects:
Priority: 5
Next Execution: 180 Hours and 35 Minutes
Duration: 15 Minutes
Period: 1 Week
Day of Week: Wednesday
Month:
Day of Month:
Week of Month:
Expire: Never

tsm-query-partitions-backupeed-or-not

5. Check which files have been backed up

If you want to make sure backups are really created it is a good to check, which files from the selected backup files have already
a working backup copy.

This is done with query backup like so:

tsm> q ba /home/*

tsm-dsmc-query-user-home-for-backups

If you want to query all the current files and directories backed up under a directory and all its subdirectories you need to add the -subdir=yes option as below:

tsm> q ba /home/hipo/projects/* -subdir=yes
Size Backup Date Mgmt Class A/I File
—- ———– ———- — —-
512 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hfs0106
1,024 08-12-2011 02:46:53 STANDARD A /home/hipo/projects/hsm41perf
512 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hsm41test
512 24-04-2012 00:22:56 STANDARD A /home/hipo/projects/hsm42upg
1,024 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hfs0106/test
1,024 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hfs0106/test/test2
12,048 04-12-2011 02:01:29 STANDARD A /home/hipo/projects/hsm41perf/tables
50,326 30-04-2012 01:35:26 STANDARD A /home/hipo/projects/hsm42upg/PMR70023
50,326 27-04-2012 00:28:15 STANDARD A /home/hipo/projects/hsm42upg/PMR70099
11,013 24-04-2012 00:22:56 STANDARD A /home/hipo/projects/hsm42upg/md5check

To make tsm, backup some directories on Linux / AIX other unices:

tsm> incr / /usr /usr/local /home /lib

For tsm to backup some standard netware drives, use:

tsm> incr NDS: USR: SYS: APPS:

To backup C:\ D:\ E:\ F:\ if TSM is running on Windows

tsm> incr C: D: E: F: -incrbydate

To back up entire disk volumes irrespective of whether files have changed since the last backup, use the selective command with a wildcard and -subdir=yes as below:

tsm> sel /* /usr/* /home/* -su=yes ** Unix/Linux

7. Backup selected files from a backup location

It is intuitive to think you can just add some wildcard characters to select what you want
to backup from a selected location but this is not so, if you try something like below
you will get an err.

tsm> incr /home/hipo/projects/*/* -su=yes
ANS1071E Invalid domain name entered: '/home/hipo/projects/*/*'

The proper way to select a certain folder / file for backup is with:

tsm> sel /home/hipo/projects/*/* -su=yes

8. Restoring tsm data from backup

To restore the config httpd.conf to custom directory use:

tsm> rest /etc/httpd/conf/httpd.conf /home/hipo/restore/

N!B! that in order for above to work you need to have the '/' trailing slash at the end.

If you want to restore a file under a different name:

tsm> rest /etc/ntpd.conf /home/hipo/restore/

9. Restoring a whole backupped partition

tsm> rest /home/* /tmp/restore/ -su=yes

This is using the Tivoli 'Restoring multiple files and directories', and the files to restore '*'
are kept till the one that was recovered (saying this in case if you accidently cancel the restore)

10. Restoring files with back date

By default the restore function will restore the latest available backupped file, if you need
to recover a specific file, you need the '-inactive' '-pick' options.
The 'pick' interface is interactive so once listed you can select the exact file from the date
you want to restore.

General restore command syntax is:

tsm> restore [source-file] [destination-file]

tsm> rest /home/hipo/projects/* /tmp/restore/ -su=yes -inactive -pick
TSM Scrollable PICK Window – Restore
# Backup Date/Time File Size A/I File
————————————————————————————————–
170. | 12-09-2011 19:57:09 650 B A /home/hipo/projects/hsm41test/inclexcl.test
171. | 12-09-2011 19:57:09 2.74 KB A /home/hipo/projects/hsm41test/inittab.ORIG
172. | 12-09-2011 19:57:09 2.74 KB A /home/hipo/projects/hsm41test/inittab.TEST
173. | 12-09-2011 19:57:09 1.13 KB A /home/hipo/projects/hsm41test/md5.out
174. | 30-04-2012 01:35:26 512 B A /home/hipo/projects/hsm42125upg/PMR70023
175. | 26-04-2012 01:02:08 512 B I /home/hipo/projects/hsm42125upg/PMR70023
176. | 27-04-2012 00:28:15 512 B A /home/hipo/projects/hsm42125upg/PMR70099
177. | 24-04-2012 19:17:34 512 B I /home/hipo/projects/hsm42125upg/PMR70099
178. | 24-04-2012 00:22:56 1.35 KB A /home/hipo/projects/hsm42125upg/dsm.opt
179. | 24-04-2012 00:22:56 4.17 KB A /home/hipo/projects/hsm42125upg/dsm.sys
180. | 24-04-2012 00:22:56 1.13 KB A /home/hipo/projects/hsm42125upg/dsmmigfstab
181. | 24-04-2012 00:22:56 7.30 KB A /home/hipo/projects/hsm42125upg/filesystems
182. | 24-04-2012 00:22:56 1.25 KB A /home/hipo/projects/hsm42125upg/inclexcl
183. | 24-04-2012 00:22:56 198 B A /home/hipo/projects/hsm42125upg/inclexcl.dce
184. | 24-04-2012 00:22:56 291 B A /home/hipo/projects/hsm42125upg/inclexcl.ox_sys
185. | 24-04-2012 00:22:56 650 B A /home/hipo/projects/hsm42125upg/inclexcl.test
186. | 24-04-2012 00:22:56 670 B A /home/hipo/projects/hsm42125upg/inetd.conf
187. | 24-04-2012 00:22:56 2.71 KB A /home/hipo/projects/hsm42125upg/inittab
188. | 24-04-2012 00:22:56 1.00 KB A /home/hipo/projects/hsm42125upg/md5check
189. | 24-04-2012 00:22:56 79.23 KB A /home/hipo/projects/hsm42125upg/mkreport.020423.out
190. | 24-04-2012 00:22:56 4.27 KB A /home/hipo/projects/hsm42125upg/ssamap.020423.out
191. | 26-04-2012 01:02:08 12.78 MB A /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
192. | 25-04-2012 16:33:36 12.78 MB I /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
0———10——–20——–30——–40——–50——–60——–70——–80——–90–
<U>=Up <D>=Down <T>=Top <B>=Bottom <R#>=Right <L#>=Left
<G#>=Goto Line # <#>=Toggle Entry <+>=Select All <->=Deselect All
<#:#+>=Select A Range <#:#->=Deselect A Range <O>=Ok <C>=Cancel
pick>

To navigate in pick interface you can select individual files to restore via the number seen leftside.
To scroll up / down use 'U' and 'D' as described in the legenda.

11. Restoring your data to another machine

In certain circumstances, it may be necessary to restore some, or all, of your data onto a machine other than the original from which it was backed up.

In ideal case the machine platform should be identical to that of the original machine. Where this is not possible or practical please note that restores are only possible for partition types that the operating system supports. Thus a restore of an NTFS partition to a Windows 9x machine with just FAT support may succeed but the file permissions will be lost.
TSM does not work fine with cross-platform backup / restore, so better do not try cross-platform restores.
Trying to restore files onto a Windows machine that have previously been backed up with a non-Windows one. TSM created backups on Windows sent by other OS platforms can cause backups to become inaccessible from the host system.

To restore your data to another machine you will need the TSM software installed on the target machine. Entries in Tivoli configuration files dsm.sys and/or dsm.opt need to be edited if the node that you are restoring from does not reside on the same server. Please see our help page section on TSM configuration files for their locations for your operating system.

To access files from another machine you should then start the TSM client as below:

# dsmc -virtualnodename=RESTORE.MACHINE

You will then be prompted for the TSM password for this machine.

You will probably want to restore to a different destination to the original files to prevent overwriting files on the local machine, as below:

Restore of D:\ Drive to D:\Restore ** Windows

tsm> rest D:\* D:\RESTORE\ -su=yes

Restore user /home/* to /scratch on ** Mac, Unix/Linux

tsm> rest /home/* /scratch/ -su=yes

Restoring Tivoli data on old netware

tsm> rest SOURCE-SERVER\USR:* USR:restore/ -su=yes ** Netware

12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

The easiest way is to check the produced dschmed.log if everything is okay there should be records in the log that Tivoli backup was scheduled in a some hours time
succesfully.
A normally produced backup scheduled in log should look something like:

14-03-2020 23:03:04 — SCHEDULEREC STATUS BEGIN
14-03-2020 23:03:04 Total number of objects inspected: 91,497
14-03-2020 23:03:04 Total number of objects backed up: 113
14-03-2020 23:03:04 Total number of objects updated: 0
14-03-2020 23:03:04 Total number of objects rebound: 0
14-03-2020 23:03:04 Total number of objects deleted: 0
14-03-2020 23:03:04 Total number of objects expired: 53
14-03-2020 23:03:04 Total number of objects failed: 6
14-03-2020 23:03:04 Total number of bytes transferred: 19.38 MB
14-03-2020 23:03:04 Data transfer time: 1.54 sec
14-03-2020 23:03:04 Network data transfer rate: 12,821.52 KB/sec
14-03-2020 23:03:04 Aggregate data transfer rate: 114.39 KB/sec
14-03-2020 23:03:04 Objects compressed by: 0%
14-03-2020 23:03:04 Elapsed processing time: 00:02:53
14-03-2020 23:03:04 — SCHEDULEREC STATUS END
14-03-2020 23:03:04 — SCHEDULEREC OBJECT END WEEKLY_23_00 14-12-2010 23:00:00
14-03-2020 23:03:04 Scheduled event 'WEEKLY_23_00' completed successfully.
14-03-2020 23:03:04 Sending results for scheduled event 'WEEKLY_23_00'.
14-03-2020 23:03:04 Results sent to server for scheduled event 'WEEKLY_23_00'.

in case of errors you should check dsmerror.log

Conclusion

In this article I've briefly evaluated some basics of IBM Commercial Tivoli Storage Manager (TSM) to be able to list backups, check backup schedules and how to the files set to be
excluded from a backup location and most importantly how to check that data backed up data is in a good shape and accessible.
It was explained how backups can be restored on a local and remote machine as well as how to append new files to be set for backup on next incremental scheduled backup.
It was shown how the pick interactive cli interface could be used to restore files at a certain data back in time as well as how full partitions can be restored and how some
certain file could be retrieved from the TSM data copy.

Tags: cause, checking, Commands, console, custom, file permissions, General, How to, ITSERV, Listing, machine, NDS, operating system, order, Schedule Name, subdirectories, tar, target, test, transfer, Wednesday, work
Posted in Linux, Linux Backup tools, System Administration, Various | 2 Comments »

Improve DNS lookup domain resolve speed on Linux / UNIX servers through /etc/resolv.conf timeout, attempts, rorate options

Thursday, February 27th, 2020

improve-dns-lookup-speed-on-Linux-UNIX-servers-resolv.conf-change-dns-settings-linux
If you're an performance optimization freak and you want to optimize your Linux servers to perform better in terms of DNS resolve slowness because of failing DNS resolve queries due to Domain Name Server request overload or due to Denial of Service attack towards it. It might be interesting to mention about some little known functionalities of /etc/resolv.conf described in the manual page.

The defined nameservers under /etc/resolv.conf are queried one by one waiting for responce of the sent DNS resolve request if it is not replied from the first one for some time, the 2nd one is queried until a responce is received by any of the defined nameserver IPs

A default /etc/resolv.conf on a new Linux server install looks something like this:

nameserver 10.10.8.1
nameserver 10.10.8.2
nameserver 10.10.8.3
search sub.subdomain.com subdom.dom.domain.com

However one thing is that defined if NS1 dies out due to anything, it takes timeout time until the second or 3rd working one takes over to resolve the query.
This is controlled by the timeout value.

Below is description from man page

timeout:n
sets the amount of time the resolver will wait for a
response from a remote name server before retrying the
query via a different name server. Measured in
seconds, the default is RES_TIMEOUT (currently 5, see
<resolv.h>). The value for this option is silently
capped to 30.

In other words Timeout value is time to resolving IP address from hostname through DNS server,timeout option is to reduce hostname lookup time

As you see from manual default is 5 seconds which is quite high, thus reducing the value to 3 secs or even 1 seconds is a good sysadmin practice IMHO.

Another value that could be tuned in /etc/resolv.conf is attempts value below is what the manual says about it:

attempts:n
Sets the number of times the resolver will send a query to its name servers before giving up and returning an error to the calling application. The default is RES_DFLRETRY (cur‐
rently 2, see <resolv.h>). The value for this option is silently capped to 5.

This means default behaviour on a failing DNS query resolve is to try to resend the DNS resolve request to the failing nameserver 5 more times, that is quite high thus it is a good practice from my experience to reduce it to something as 2 or 1

Another very useful resolv.conf value is rotate
The default behavior of how DNS outgoing Domain requests are handled is to use only the primary defined DNS, instead if you need to do a load balancing in a round-robin manner add to conf rotate option.

The final /etc/resolv.conf optimized would look like so:

linux# cat /etc/resolv.conf
nameserver 10.10.8.1
nameserver 10.10.8.2
nameserver 10.10.8.3
search sub.subdomain.com subdom.dom.domain.com
options ndots:1
options timeout:1
options attempts:1
options rotate

The search opt. placement is also important to be placed in the right location in the file. The correct placement is after the nameservers defined, I have to say in older Linux distributions the correct placement of search option was to be on top of resolv.conf.

Note that this configuration is good and fits not only Linux but also is a good DNS lookup optimization speed on other UNIX derivatives such as FreeBSD / NetBSD as well as other Proprietary OS UNIX machines running IBM AIX etc.

On Linux it is also possible to place the options given in one single line like so, below is the config I have on my pc-freak.net running Lenovo server:

domain pc-freak.net
search pc-freak.net
#nameserver 192.168.0.1
nameserver 127.0.0.1
nameserver 83.228.92.2
nameserver 8.8.8.8
nameserver 83.228.92.1
nameserver 208.67.222.222
nameserver 208.67.220.220
options timeout:2 attempts:1 rotate

When is /etc/hosts record venerated and when is /etc/resolv.conf DNS defined queried for a defined DNS host?

One important thing to know when dealing with /etc/resolv.conf is what happens if a Name domain is defined in both /etc/hosts and /etc/resolv.conf.
For example you have a pc-freak.net domain record in /etc/hosts to a certain domain but the DNS nameserver 8.8.8.8 in Google has a record to an IP that is the real IP 83.228.93.76

83.228.93.75 irc.pc-freak.net pc-freak.net pcfreak.biz www.pc-freak.net pcfreak.us services.pc-freak.net jabber.pc-freak.net

# dig @8.8.8.8 pc-freak.net
; <<>> DiG 9.11.5-P4-5.1-Debian <<>> @8.8.8.8 pc-freak.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54656
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;pc-freak.net. IN A
;; ANSWER SECTION:
pc-freak.net. 3599 IN A 83.228.93.76
;; Query time: 40 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: чт фев 27 18:04:23 EET 2020
;; MSG SIZE rcvd: 57

Which of the 2 different IPs will the applications installed on the server such as Apache / Squid / MySQL / tinyproxy for their DNS resolve operations?

Now it is time to say few words about /etc/nsswitch.conf (The Nameserver switching configuration file). This file defines the DNS resolve file used order in which the Operationg System does IP to domain translation and backwards.

# grep -i hosts: /etc/nsswitch.conf

hosts: files dns myhostname

As you can see first the local defined in files like /etc/hosts record is venerated when resolving, then it is the externally configured DNS resolver IPs from /etc/resolv.conf.

nsswitch.conf is used also for defining where the OS will look up for user / passwd (e.g. login credentials) on login, on systems which are having an LDAP authentication via the sssd (system security services daemon) via definitions like:

passwd: files sss
shadow: files sss
group: files sss

E.g. the user login will be first try to read from local /etc/passwd , /etc/shadow , /etc/groups and if no matched record is found then the LDAP service the sssd is queried.

Tags: DNS, improve aix dns resolve time, improve bsd ns query time, improve resolve times, nsswitch.conf, optimize dns resolve time linux, resolv.conf
Posted in Everyday Life, FreeBSD, Linux, System Administration | No Comments »

How to check if shared library is loaded in AIX OS – Fix missing libreadline.so.7

Thursday, February 20th, 2020

ibm-aix-logo1

I've had to find out whether an externally Linux library is installed on AIX system and whether something is not using it.
The returned errors was like so:

# gpg –export -a
Could not load program gpg:
Dependent module /opt/custom/lib/libreadline.a(libreadline.so.7) could not be loaded.
Member libreadline.so.7 is not found in archive

After a bit of investigation, I found that gpg was failing cause it linked to older version of libreadline.so.6, the workaround was to just substitute the newer version of libreadline.so.7 over the original installed one.

Thus I had a plan to first find out whether this libreadline.a is loaded and recognized by AIX UNIX first and second find out whether some of the running processes is not using that library.
I've come across this interesting IBM official documenation that describes pretty good insights on how to determine whether a shared library is currently loaded on the system. which mentions the genkld command that is doing
exactly what I needed.

In short:
genkld – creates a list that is printed to the console that shows all loaded shared libraries

genkld-screenshot-aix-unix

Next I used lsof (list open files) command to check whether there is in real time opened libraries by any of the running programs on the system.

After not finding anything and was sure the library is neither loaded as a system library in AIX nor it is used by any of the currently running AIX processes, I was sure I could proceed to safely overwrite libreadline.a (libreadline.so.6) with libreadline.a with (libreadline.so.7).

The result of that is again a normally running gpg as ldd command shows the binary is again normally linked to its dependend system libraries.

aix# ldd /usr/bin/gpg
/usr/bin/gpg needs:
/usr/lib/threads/libc.a(shr.o)
/usr/lib/libpthreads.a(shr_comm.o)
/usr/lib/libpthreads.a(shr_xpg5.o)
/opt/freeware/lib/libintl.a(libintl.so.1)
/opt/freeware/lib/libreadline.a(libreadline.so.7)
/opt/freeware/lib/libiconv.a(libiconv.so.2)
/opt/freeware/lib/libz.a(libz.so.1)
/opt/freeware/lib/libbz2.a(libbz2.so.1)
/unix
/usr/lib/libcrypt.a(shr.o)
/opt/freeware/lib/libiconv.a(shr4.o)
/usr/lib/libcurses.a(shr42.o)

# gpg –version
gpg (GnuPG) 1.4.22
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Tags: AES, aes256, AIX, check, Cipher, command, find, first, found, freeware, How to, idea, installed, libraries, RSA, running processes, substitute, system, using, version
Posted in AIX, Everyday Life, System Administration | No Comments »

How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages

Wednesday, February 19th, 2020

haproxy-logging-basics-how-to-log-to-separate-file-prevent-duplicate-messages-haproxy-haproxy-weblogo-squares
haproxy logging can be managed in different form the most straight forward way is to directly use /dev/log either you can configure it to use some log management service as syslog or rsyslogd for that.

If you don't use rsyslog yet to install it:

# apt install -y rsyslog

Then to activate logging via rsyslogd we can should add either to /etc/rsyslogd.conf or create a separte file and include it via /etc/rsyslogd.conf with following content:

Enable haproxy logging from rsyslogd

Log haproxy messages to separate log file you can use some of the usual syslog local0 to local7 locally used descriptors inside the conf (be aware that if you try to use some wrong value like local8, local9 as a logging facility you will get with empty haproxy.log, even though the permissions of /var/log/haproxy.log are readable and owned by haproxy user.

When logging to a local Syslog service, writing to a UNIX socket can be faster than targeting the TCP loopback address. Generally, on Linux systems, a UNIX socket listening for Syslog messages is available at /dev/log because this is where the syslog() function of the GNU C library is sending messages by default. To address UNIX socket in haproxy.cfg use:

log /dev/log local2

If you want to log into separate log each of multiple running haproxy instances with different haproxy*.cfg add to /etc/rsyslog.conf lines like:

local2.* -/var/log/haproxylog2.log
local3.* -/var/log/haproxylog3.log

One important note to make here is since rsyslogd is used for haproxy logging you need to have enabled in rsyslogd imudp and have a UDP port listener on the machine.

E.g. somewhere in rsyslog.conf or via rsyslog include file from /etc/rsyslog.d/*.conf needs to have defined following lines:

$ModLoad imudp
$UDPServerRun 514

I prefer to use external /etc/rsyslog.d/20-haproxy.conf include file that is loaded and enabled rsyslogd via /etc/rsyslog.conf:

# vim /etc/rsyslog.d/20-haproxy.conf
$ModLoad imudp
$UDPServerRun 514
local2.* -/var/log/haproxy2.log

It is also possible to produce different haproxy log output based on the severiy to differentiate between important and less important messages, to do so you'll need to rsyslog.conf something like:

# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log

Prevent Haproxy duplicate messages to appear in /var/log/messages

If you use local2 and some default rsyslog configuration then you will end up with the messages coming from haproxy towards local2 facility producing doubled simultaneous records to both your pre-defined /var/log/haproxy.log and /var/log/messages on Proxy servers that receive few thousands of simultanous connections per second.
This is a problem since doubling the log will produce too much data and on systems with smaller /var/ partition you will quickly run out of space + this haproxy requests logging to /var/log/messages makes the file quite unreadable for normal system events which are so important to track clearly what is happening on the server daily.

To prevent the haproxy duplicate messages you need to define somewhere in rsyslogd usually /etc/rsyslog.conf local2.none near line of facilities configured to log to file:

*.info;mail.none;authpriv.none;cron.none;local2.none /var/log/messages

This configuration should work but is more rarely used as most people prefer to have haproxy log being written not directly to /dev/log which is used by other services such as syslogd / rsyslogd.

To use /dev/log to output logs from haproxy configuration in global section use config like:

global
log /dev/log local2 debug
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

The log global directive basically says, use the log line that was set in the global section for whole config till end of file. Putting a log global directive into the defaults section is equivalent to putting it into all of the subsequent proxy sections.

Using global logging rules is the most common HAProxy setup, but you can put them directly into a frontend section instead. It can be useful to have a different logging configuration as a one-off. For example, you might want to point to a different target Syslog server, use a different logging facility, or capture different severity levels depending on the use case of the backend application.

Insetad of using /dev/log interface that is on many distributions heavily used by systemd to store / manage and distribute logs, many haproxy server sysadmins nowdays prefer to use rsyslogd as a default logging facility that will manage haproxy logs.
Admins prefer to use some kind of mediator service to manage log writting such as rsyslogd or syslog, the reason behind might vary but perhaps most important reason is by using rsyslogd it is possible to write logs simultaneously locally on disk and also forward logs to a remote Logging server running rsyslogd service.

Logging is defined in /etc/haproxy/haproxy.cfg or the respective configuration through global section but could be also configured to do a separate logging based on each of the defined Frontend Backends or default section.
A sample exceprt from this section looks something like:

#———————————————————————
# Global settings
#———————————————————————
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#———————————————————————
defaults
mode tcp
log global
option tcplog
#option dontlognull
#option http-server-close
#option forwardfor except 127.0.0.0/8
option redispatch
retries 7
#timeout http-request 10s
timeout queue 10m
timeout connect 30s
timeout client 20m
timeout server 10m
#timeout http-keep-alive 10s
timeout check 30s
maxconn 3000

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080 #Haproxy Monitoring run on port 8080
mode http
option httplog
option http-server-close
stats enable
stats show-legends
stats refresh 5s
stats uri /stats #URL for HAProxy monitoring
stats realm Haproxy\ Statistics
stats auth hproxyauser:Password___ #User and Password for login to the monitoring dashboard

#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_DKV_PROD_WLPFO
mode tcp
bind 192.168.233.5:30000-31050
option tcplog
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
default_backend Default_Bakend_Name

#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_DKV_PROD_WLPFO
mode tcp
# (0) Load Balancing Method.
balance source
# (4) Peer Sync: a sticky session is a session maintained by persistence
stick-table type ip size 1m peers hapeers expire 60m
stick on src
# (5) Server List
# (5.1) Backend
server Backend_Server1 10.10.10.1 check port 18088
server Backend_Server2 10.10.10.2 check port 18088 backup

The log directive in above config instructs HAProxy to send logs to the Syslog server listening at 127.0.0.1:514. Messages are sent with facility local2, which is one of the standard, user-defined Syslog facilities. It’s also the facility that our rsyslog configuration is expecting. You can add more than one log statement to send output to multiple Syslog servers.

Once rsyslog and haproxy logging is configured as a minumum you need to restart rsyslog (assuming that haproxy config is already properly loaded):

# systemctl restart rsyslogd.service

To make sure rsyslog reloaded successfully:

systemctl status rsyslogd.service

Restarting HAproxy

If the rsyslogd logging to 127.0.0.1 port 514 was recently added a HAProxy restart should also be run, you can do it with:

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

Or to restart use systemctl script (if haproxy is not used in a cluster with corosync / heartbeat).

# systemctl restart haproxy.service

You can control how much information is logged by adding a Syslog level by

log 127.0.0.1 local2 info

The accepted values are the standard syslog security level severity:

Value Severity Keyword Deprecated keywords Description Condition
0 Emergency emerg panic System is unusable A panic condition.
1 Alert alert Action must be taken immediately A condition that should be corrected immediately, such as a corrupted system database.
2 Critical crit Critical conditions Hard device errors.
3 Error err error Error conditions
4 Warning warning warn Warning conditions
5 Notice notice Normal but significant conditions Conditions that are not error conditions, but that may require special handling.
6 Informational info Informational messages
7 Debug debug Debug-level messages Messages that contain information normally of use only when debugging a program.

Value	Severity	Keyword	Deprecated keywords	Description	Condition
0	Emergency	`emerg`	`panic`	System is unusable	A panic condition.
1	Alert	`alert`		Action must be taken immediately	A condition that should be corrected immediately, such as a corrupted system database.
2	Critical	`crit`		Critical conditions	Hard device errors.
3	Error	`err`	`error`	Error conditions
4	Warning	`warning`	`warn`	Warning conditions
5	Notice	`notice`		Normal but significant conditions	Conditions that are not error conditions, but that may require special handling.
6	Informational	`info`		Informational messages
7	Debug	`debug`		Debug-level messages	Messages that contain information normally of use only when debugging a program.

Logging only errors / timeouts / retries and errors is done with option:

Note that if the rsyslog is configured to listen on different port for some weird reason you should not forget to set the proper listen port, e.g.:

log 127.0.0.1:514 local2 info

option dontlog-normal

in defaults or frontend section.

You most likely want to enable this only during certain times, such as when performing benchmarking tests.

(or log-format-sd for structured-data syslog) directive in your defaults or frontend

Haproxy Logging shortly explained

The type of logging you’ll see is determined by the proxy mode that you set within HAProxy. HAProxy can operate either as a Layer 4 (TCP) proxy or as Layer 7 (HTTP) proxy. TCP mode is the default. In this mode, a full-duplex connection is established between clients and servers, and no layer 7 examination will be performed. When in TCP mode, which is set by adding mode tcp, you should also add option tcplog. With this option, the log format defaults to a structure that provides useful information like Layer 4 connection details, timers, byte count and so on.

Below is example of configured logging with some explanations:

Log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

Example of Log-Format configuration as shown above outputted of haproxy config:

Log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

haproxy_http_log_format-explained1

To understand meaning of this abbreviations you'll have to closely read haproxy-log-format.txt. More in depth info is to be found in HTTP Log format documentation

haproxy_logging-explained

Logging HTTP request headers

HTTP request header can be logged via:

http-request capture

frontend website
bind :80
http-request capture req.hdr(Host) len 10
http-request capture req.hdr(User-Agent) len 100
default_backend webservers

The log will show headers between curly braces and separated by pipe symbols. Here you can see the Host and User-Agent headers for a request:

192.168.150.1:57190 [20/Dec/2018:22:20:00.899] website~ webservers/server1 0/0/1/0/1 200 462 – – —- 1/1/0/0/0 0/0 {mywebsite.com|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.80 } "GET / HTTP/1.1"

Haproxy Stats Monitoring Web interface

Haproxy is having a simplistic stats interface which if enabled produces some general useful information like in above screenshot, through which
you can get a very basic in browser statistics and track potential issues with the proxied traffic for all configured backends / frontends incoming outgoing
network packets configured nodes experienced downtimes etc.

haproxy-statistics-report-picture

The basic configuration to make the stats interface accessible would be like pointed in above config for example to enable network listener on address

https://192.168.0.5:8080/stats

with hproxyuser / password config would be:

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080 #Haproxy Monitoring run on port 8080
mode http
option httplog
option http-server-close
stats enable
stats show-legends
stats refresh 5s
stats uri /stats #URL for HAProxy monitoring
stats realm Haproxy\ Statistics
stats auth hproxyauser:Password___ #User and Password for login to the monitoring dashboard

Sessions states and disconnect errors on new application setup

Both TCP and HTTP logs include a termination state code that tells you the way in which the TCP or HTTP session ended. It’s a two-character code. The first character reports the first event that caused the session to terminate, while the second reports the TCP or HTTP session state when it was closed.

Here are some essential termination codes to track in for in the log:

Here are some termination code examples most commonly to see on TCP connection establishment errors:

Two-character code   Meaning
—   Normal termination on both sides.
cD   The client did not send nor acknowledge any data and eventually timeout client expired.
SC   The server explicitly refused the TCP connection.
PC   The proxy refused to establish a connection to the server because the process’ socket limit was reached while attempting to connect.

To get all non-properly exited codes the easiest way is to just grep for anything that is different from a termination code –, like that:

tail -f /var/log/haproxy.log | grep -v ' — '

This should output in real time every TCP connection that is exiting improperly.

There’s a wide variety of reasons a connection may have been closed. Detailed information about all possible termination codes can be found in the HAProxy documentation.
To get better understanding a very useful reading to haproxy Debug errors with is in haproxy-logging.txt in that small file are collected all the cryptic error messages codes you might find in your logs when you're first time configuring the Haproxy frontend / backend and the backend application behind.

Another useful analyze tool which can be used to analyze Layer 7 HTTP traffic is halog for more on it just google around.

Tags: about, action, Activate, adding, admin, application, based, configured, enable, errors, facility, How to, log, log messages, logging, option, separate, setup, something, var
Posted in Haproxy, System Administration, Various, Web and CMS | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Archive for the ‘System Administration’ Category

How to check if shared library is loaded in AIX OS – Fix missing libreadline.so.7

How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages

Enable haproxy logging from rsyslogd

Prevent Haproxy duplicate messages to appear in /var/log/messages

Restarting HAproxy

Haproxy Logging shortly explained

Logging HTTP request headers

Haproxy Stats Monitoring Web interface

Sessions states and disconnect errors on new application setup

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites

Archive for the ‘System Administration’ Category

1. smartmontools (smartd)

2. hddtemp

3. lm-sensors / i2c-tools

1. Change user password / passphrases at least one every 90 days

2. Increase requirements for password quality

3. Limit repeated access attempts by locking out the user ID if more than 6 unsuccesful login

4. Set blockout user lockout duration to 30 minutes

5. Check and reset fail lock number attempts from user

6. Check user new set passwords against dictionary with pam_pwquality.so

7. Set User Session Timeout in 15 minutes

8. Increase size of logged User commands history HISTFILE and HISTFILESIZE variables

9. Display TIMESTAMP in command history using HISTTIMEFORMAT

To show printed the exact day of week and written with letters you instead of numeric:

1. List RPM gpg keys installed on system

2. Import RPM-GPG-KEY

3. Check package with imported gpg arch key

4. Remove RPM installed arch key

1. Create rsyncuser on 1st machine and second machine

2. CREATE sync-commands file desired to be executed on logout from the system of the user into some file like /root/sync_check.sh

3. Edit root lougout-script /root/.bash_logout

1. Configured default cron execution times, every day, every hour every month

2. Find out cron scripts timing Redhat / CentOS / Fedora / SLES

3. SuSE Enterprise Linux cron jobs not running at desired times why?

4. Debian Linux time start schedule for cron.daily / cron.monthly / cron.weekly timing

5. systemd-timers – The new crazy systemd stuff for script system job scheduling

Summary

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?

1. Using the dmesg kernel log utility

2. Detecting the OS platform the systemd way

3. Detect concrete container virtualization with systemd-detect-virt

4. Install and use facter to report per node facts

6. Use lshw and dmidecode (list hardware configuration tool)

7. Detect virtualziation using virt-what or imvirt scripts

Conclusion

1. Starting up the dsmc command line client

2. Listing files listed for backups

3. Getting included and excluded backup set files

4. Querying for backup schedule time

5. Check which files have been backed up

7. Backup selected files from a backup location

8. Restoring tsm data from backup

9. Restoring a whole backupped partition

10. Restoring files with back date

11. Restoring your data to another machine

12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

Conclusion

When is /etc/hosts record venerated and when is /etc/resolv.conf DNS defined queried for a defined DNS host?

Enable haproxy logging from rsyslogd

Prevent Haproxy duplicate messages to appear in /var/log/messages

Restarting HAproxy

Haproxy Logging shortly explained

Logging HTTP request headers

Haproxy Stats Monitoring Web interface

Sessions states and disconnect errors on new application setup

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Tags

Top Post Views

blogtopsites

2. CREATE sync-commands file desired to be executed on logout from the system of the user into
some file like /root/sync_check.sh