Posts Tagged ‘file’

How to monitor Haproxy Application server backends with Zabbix userparameter autodiscovery scripts

Friday, May 13th, 2022


Haproxy is doing quite a good job in High Availability tasks where traffic towards multiple backend servers has to be redirected based on the available one to sent data from the proxy to. 

Lets say haproxy is configured to proxy traffic for App backend machine1 and App backend machine2.

Usually in companies people configure a monitoring like with Icinga or Zabbix / Grafana to keep track on the Application server is always up and running. Sometimes however due to network problems (like burned Network Switch / router or firewall misconfiguration) or even an IP duplicate it might happen that Application server seems to be reporting reachable from some monotoring tool on it but unreachable from  Haproxy server -> App backend machine2 but reachable from App backend machine1. And even though haproxy will automatically switch on the traffic from backend machine2 to App machine1. It is a good idea to monitor and be aware that one of the backends is offline from the Haproxy host.
In this article I'll show you how this is possible by using 2 shell scripts and userparameter keys config through the autodiscovery zabbix legacy feature.
Assumably for the setup to work you will need to have as a minimum a Zabbix server installation of version 5.0 or higher.

1. Create the required  and scripts 

You will have to install the two scripts under some location for example we can put it for more clearness under /etc/zabbix/scripts

[root@haproxy-server1 ]# mkdir /etc/zabbix/scripts

[root@haproxy-server1 scripts]# vim 
# Get list of Frontends and Backends from HAPROXY
# Example: ./ [/var/lib/haproxy/stats] FRONTEND|BACKEND|SERVERS
# First argument is optional and should be used to set location of your HAPROXY socket
# Second argument is should be either FRONTEND, BACKEND or SERVERS, will default to FRONTEND if not set
# !! Make sure the user running this script has Read/Write permissions to that socket !!
## haproxy.cfg snippet
#  global
#  stats socket /var/lib/haproxy/stats  mode 666 level admin

[ -n “$1” ] && echo $1 | grep -q ^/ && HAPROXY_SOCK="$(echo $1 | tr -d '\040\011\012\015')"

if [[ “$1” =~ (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?):[0-9]{1,5} ]];


query_stats() {
    if [[ ${QUERYING_METHOD} == “SOCKET” ]]; then
        echo "show stat" | socat ${HAPROXY_SOCK} stdio 2>/dev/null
    elif [[ ${QUERYING_METHOD} == “TCP” ]]; then
        echo "show stat" | nc ${HAPROXY_STATS_IP//:/ } 2>/dev/null

get_stats() {
        echo "$(query_stats)" | grep -v "^#"

[ -n “$2” ] && shift 1
case $1 in
        B*) END="BACKEND" ;;
        F*) END="FRONTEND" ;;
                for backend in $(get_stats | grep BACKEND | cut -d, -f1 | uniq); do
                        for server in $(get_stats | grep "^${backend}," | grep -v BACKEND | grep -v FRONTEND | cut -d, -f2); do
                echo -e '{\n\t"data":[\n’${serverlist#,}’]}'
                exit 0
        *) END="FRONTEND" ;;

for frontend in $(get_stats | grep "$END" | cut -d, -f1 | uniq); do
echo -e '{\n\t"data":[\n’${felist#,}’]}'


[root@haproxy-server1 scripts]# vim 
set -o pipefail

if [[ “$1” = /* ]]
  shift 0
  if [[ “$1” =~ (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?):[0-9]{1,5} ]];
    shift 1


CACHE_INFO_FILEPATH="${CACHE_INFO_FILEPATH:-/var/tmp/haproxy_info.cache}" ## unused
GET_STATS=${GET_STATS:-1} # when you update stats cache outsise of the script
SOCAT_BIN="$(which socat)"
NC_BIN="$(which nc)"
FLOCK_BIN="$(which flock)"
FLOCK_WAIT=15 # maximum number of seconds that "flock" waits for acquiring a lock
CUR_TIMESTAMP="$(date '+%s')"

debug() {
  [ “${DEBUG}” -eq 1 ] && echo "DEBUG: $@" >&2 || true

debug "SOCAT_BIN        => $SOCAT_BIN"
debug "NC_BIN           => $NC_BIN"
debug "FLOCK_BIN        => $FLOCK_BIN"
debug "FLOCK_WAIT       => $FLOCK_WAIT seconds"
debug "pxname   => $pxname"
debug "svname   => $svname"
debug "stat     => $stat"

# check if socat is available in path
if [ “$GET_STATS” -eq 1 ] && [[ $QUERYING_METHOD == “SOCKET” && -z “$SOCAT_BIN” ]] || [[ $QUERYING_METHOD == “TCP” &&  -z “$NC_BIN” ]]
  echo 'ERROR: cannot find socat binary'
  exit 126

# if we are getting stats:
#   check if we can write to stats cache file, if it exists
#     or cache file path, if it does not exist
#   check if HAPROXY socket is writable
# if we are NOT getting stats:
#   check if we can read the stats cache file
if [ “$GET_STATS” -eq 1 ]
  if [ -e “$CACHE_FILEPATH” ] && [ ! -w “$CACHE_FILEPATH” ]
    echo 'ERROR: stats cache file exists, but is not writable'
    exit 126
  elif [ ! -w ${CACHE_FILEPATH%/*} ]
    echo 'ERROR: stats cache file path is not writable'
    exit 126
    echo "ERROR: haproxy socket is not writable"
    exit 126
elif [ ! -r “$CACHE_FILEPATH” ]
  echo 'ERROR: cannot read stats cache file'
  exit 126

# index:name:default

_STAT=$(echo -e "$MAP" | grep :${stat}:)

debug "_STAT    => $_STAT"
debug "_INDEX   => $_INDEX"
debug "_DEFAULT => $_DEFAULT"

# check if requested stat is supported
if [ -z “${_STAT}” ]
  echo "ERROR: $stat is unsupported"
  exit 127

# method to retrieve data from haproxy stats
# usage:
# query_stats "show stat"
query_stats() {
    if [[ ${QUERYING_METHOD} == “SOCKET” ]]; then
        echo $1 | socat ${HAPROXY_SOCKET} stdio 2>/dev/null
    elif [[ ${QUERYING_METHOD} == “TCP” ]]; then
        echo $1 | nc ${HAPROXY_STATS_IP//:/ } 2>/dev/null

# a generic cache management function, that relies on 'flock'
check_cache() {
  local cache_type="${1}"
  local cache_filepath="${2}"
  local cache_expiration="${3}"  
  local cache_filemtime
  cache_filemtime=$(stat -c '%Y' "${cache_filepath}" 2> /dev/null)
  if [ $((cache_filemtime+60*cache_expiration)) -ge ${CUR_TIMESTAMP} ]
    debug "${cache_type} file found, results are at most ${cache_expiration} minutes stale.."
  elif "${FLOCK_BIN}" –exclusive –wait "${FLOCK_WAIT}" 200
    cache_filemtime=$(stat -c '%Y' "${cache_filepath}" 2> /dev/null)
    if [ $((cache_filemtime+60*cache_expiration)) -ge ${CUR_TIMESTAMP} ]
      debug "${cache_type} file found, results have just been updated by another process.."
      debug "no ${cache_type} file found, querying haproxy"
      query_stats "show ${cache_type}" > "${cache_filepath}"
  fi 200> "${cache_filepath}${FLOCK_SUFFIX}"

# generate stats cache file if needed
get_stats() {

# generate info cache file
## unused at the moment
get_info() {

# get requested stat from cache file using INDEX offset defined in MAP
# return default value if stat is ""
get() {
  # $1: pxname/svname
  local _res="$("${FLOCK_BIN}" –shared –wait "${FLOCK_WAIT}" "${CACHE_STATS_FILEPATH}${FLOCK_SUFFIX}" grep $1 "${CACHE_STATS_FILEPATH}")"
  if [ -z “${_res}” ]
    echo "ERROR: bad $pxname/$svname"
    exit 127
  _res="$(echo $_res | cut -d, -f ${_INDEX})"
  if [ -z “${_res}” ] && [[ “${_DEFAULT}” != “@” ]]
    echo "${_DEFAULT}"  
    echo "${_res}"

# not sure why we'd need to split on backslash
# left commented out as an example to override default get() method
# status() {
#   get "^${pxname},${svnamem}," $stat | cut -d\  -f1
# }

# this allows for overriding default method of getting stats
# name a function by stat name for additional processing, custom returns, etc.
if type get_${stat} >/dev/null 2>&1
  debug "found custom query function"
  get_stats && get_${stat}
  debug "using default get() method"
  get_stats && get "^${pxname},${svname}," ${stat}

! NB ! Substitute in the script /var/run/haproxy/haproxy.sock with your haproxy socket location

You can download the here and here

2. Create the userparameter_haproxy_backend.conf

[root@haproxy-server1 zabbix_agentd.d]# cat userparameter_haproxy_backend.conf 
# Discovery Rule

# HAProxy Frontend, Backend and Server Discovery rules
UserParameter=haproxy.list.discovery[*],sudo /etc/zabbix/scripts/ SERVER
UserParameter=haproxy.stats[*],sudo /etc/zabbix/scripts/  $2 $3 $4

# support legacy way

UserParameter=haproxy.stat.downtime[*],sudo /etc/zabbix/scripts/  $2 $3 downtime

UserParameter=haproxy.stat.status[*],sudo /etc/zabbix/scripts/  $2 $3 status

UserParameter=haproxy.stat.last_chk[*],sudo /etc/zabbix/scripts/  $2 $3 last_chk


3. Create new simple template for the Application backend Monitoring and link it to monitored host




Go to Configuration -> Hosts (find the host) and Link the template to it

4. Restart Zabbix-agent, in while check autodiscovery data is in Zabbix Server

[root@haproxy-server1 ]# systemctl restart zabbix-agent

Check in zabbix the userparameter data arrives, it should not be required to add any Items or Triggers as autodiscovery zabbix feature should automatically create in the server what is required for the data regarding backends to be in.

To view data arrives go to Zabbix config menus:

Configuration -> Hosts -> Hosts: (lookup for the haproxy-server1 hostname)


The autodiscovery should have automatically created the following prototypes

Now if you look inside Latest Data for the Host you should find some information like:

HAProxy Backend [backend1] (3 Items)
HAProxy Server [backend-name_APP/server1]: Connection Response
2022-05-13 14:15:04            History
HAProxy Server [backend-name/server2]: Downtime (hh:mm:ss)
2022-05-13 14:13:57    20:30:42        History
HAProxy Server [bk_name-APP/server1]: Status
2022-05-13 14:14:25    Up (1)        Graph
        ccnrlb01    HAProxy Backend [bk_CCNR_QA_ZVT] (3 Items)
HAProxy Server [bk_name-APP3/server1]: Connection Response
2022-05-13 14:15:05            History
HAProxy Server [bk_name-APP3/server1]: Downtime (hh:mm:ss)
2022-05-13 14:14:00    20:55:20        History
HAProxy Server [bk_name-APP3/server2]: Status
2022-05-13 14:15:08    Up (1)

To make alerting in case if a backend is down which usually you would like only left thing is to configure an Action to deliver alerts to some email address.

Configure own Media streaming minidlna Linux server to access data from your Smart TV

Friday, February 18th, 2022


If you happen to buy or already own or just have to install a Smart TV to be connected with a LAN Network to a Linux based custom built NAS (Network Attached Storage) server. You might benefit of the smart TV to Share and Watching the Disk Storage Pictures, Music, Video files from the NAS  to the Smart TV using the Media Server protocol.

You have certainly already faced the Media Server at your life on many locations in stores and Mall Buildings, because virtually any reoccuring advertisements, movies projected on the TVs, Kids entertainment or Floor and Buildings Room location schedules or timeline promition schedules are streamed using the Media Server protocol, for many years now. Thus having a brief idea about Media Server proto existence is foundamental stuff to be aware of for sysadmins and programmers.

Shortly about DLNA UPnP Media Streaming Protocol

Assuming that your Smart TV has been already connected to your Wireless Router 2.4Ghz or 5Ghz Wifi, one would think that the easiest way to share the files with the SmartTV is via something like a simple SAMBA Linux server via smb:// cifs:// protocols or via the good old NFS Server, however most of Samsung Smart TV and many other in year 2022 does not have embedded support for Samba SMB / CIFS Protocol but instead have support for the DLNA (Digital Living Network Alliance) streaming support. DLNA is part of the UPnP (Universal Plug and Play) Protocols, UPnP is also known to those using and familiar with Windows Operating Systems realm simply as UPnP AV Media server or Windows Media server.
Windows Media server for those who never heard it or used it 
 allows you to build a Playlists with Media files Video and Audio data files, that can be then later played remotely via a Local LAN or even long distance over TCP / IP remote side connected Internet network.

1. Set up and Stream data via Media server on  Windows PC / notebook with integrated Windows Media server 

Windows Media server configuration on Windows 7, 10 and 11 is a relatively easy to configure via:

Network and Sharing Center -> Media Streaming Options -> Turn on Media Streaming 

Then you have to define the name of the Media Library, configure whether Media server should show
on the Local Netework
for other conected devices and Allow or Block access from the other network present devices.

 2. Using a more advanced Media Server to get rid about the limitation of DLNA set of supported file codecs.

The Windows default embedded DLNA server is the easiest and fastest one to set up, but it’s not necessarily the best option.
Due to the way DLNA works, you can only stream certain types of media codecs supported by the server. If you have other types of media not defaultly supported and defined by DLNA win server, it just won’t work.

Thus thanksfully it was developed other DLNA servers improve this by offering real-time transcoding.
If you try to play an unsupported file, they’ll transcode it on-the-fly, streaming the video in a supported format to your DLNA device.
Just to name few of the DLNA Media Streaming servers that have supported for larger MPG Video, MP3 / MP4 and other Audio formats encodings,
you can try Plex or the Universal Media Server both of which are free to use under freeware license and have versions for Linux and Mac OS.



3. Setting up a free as in freedom DLNA server MiniDLNA (ReadyMedia) on GNU / Linux

ReadyMedia (formerly known as MiniDLNA) is a simple media server software, with the aim of being fully compliant with DLNA/UPnP-AV clients. It was originally developed by a NETGEAR employee for the ReadyNAS product line.

MiniDNLA daemon serves media files (music, pictures, and video) to clients on a network. Linux Media servers clients you can use to test or scan your network for existent Media servers are multiple perhaps the most famous ones are applications such as totem (for QT users) and Kodi (for KDE).
The devices that can be used with minidlna are devices such as portable media players (iPod), Smartphones, Televisions, Tablets, and gaming systems (such as PS3 and Xbox 360) etc.

ReadyMedia is a simple, lightweight, the downside of it is It does not have a web interface for administration and must be configured by editing a text file. But for a simple Video streaming in most cases does a great job.

3.1 Install the minidlna software package 

Minidlna is available out of the box on most linux distributions (Fedora / CentOS / Debian / Ubuntu etc.) as of year 2022.

  • Install on Debian Linux (Deb based distro)

media-server:~# apt install minidlna –yes

  • Install on Fedora / CentOS (other RPM based distro)

media-server:~# yum install -y minidlna

3.2 Configure minidlna

– /etc/minidlna.conf – main config file
Open with text editor and set user= ,  media_dir= ,  port=, friendly_name= ,  network_interface= variables as minimum.
To be add minidlnad support symlinks to external file locations, set also wide_links=yes

media-server:~# vim /etc/minidlna.conf


# Port number for HTTP traffic (descriptions, SOAP, media transfer).
# This option is mandatory (or it must be specified on the command-line using
# "-p").
# Name that the DLNA server presents to clients.
# Defaults to "hostname: username".
friendly_name=DLNAServer Linux
# set this to yes to allow symlinks that point outside user-defined media_dirs.
# Automatic discovery of new files in the media_dir directory.

Keep in mind that it is supported to provide separete media_dir and provide different USB / External Hard Drive or SD Card sources separated only by content be it Video, Audio or Pictures short named in config as (A,V,P).


You might want to diasble / ineable the inotify depending on your liking, if you don't plan to place new files automated to the NAS and don't care to get indexed and streamed from the Media server you can disable it with inotify=no otherwise keep that on.

– /etc/default/minidlna – additional startup config to set minidlnad (daemon) options such as setup to run with admin superuser root:root 
(usually it is safe to leave it empty and set the user=root, whether needed straight from /etc/minidlna.conf
That's all now go on and launch the minidlna and enable it to automatically boot on Linux boot.

media-server:~# systemctl start minidlna
media-server:~# systemctl enable minidlna
media-server:~# systemctl status minidlna


3.3 Rebuilt minidlna database with data indexed files

If you need to re- generate minidlna's database.
To do so stop the minidlna server with the

media-server:~# systemctop stop minidlna

 command, then issue the following command (both commands should be run as root):

media-server:~# minidlna -R

Since this command might kept in the background and keep the minidlna server running with incorrect flags, after a minute or two kill minidlna process and relaunch the server via sysctl.

media-server:~#  killall -9 minidlna
media-server:~#  systemctl start minidlna


3.4 Permission Issues / Scanning issues

If you plan to place files in /home directory. You better have a seperate partition or folder *outside* your "home" directory devoted to your media. Default user with which minidlna runs is minidlna, this could prevent some files with root or other users being red. So either run minidlna daemon as root or as other user with whom all media files should be accessible.
If service runs as root:root, and still getting some scanning issues, check permissions on your files and remove special characters from file names.

media-server:~# tail -10 /var/log/minidlna/minidlna.log 
[2022/02/17 22:51:36] scanner.c:489: warn: Unsuccessful getting details for /var/www/owncloud/data/Videos/Family-Videos/FILE006.MPG
[2022/02/17 22:52:08] scanner.c:819: warn: Scanning /var/www/owncloud/data finished (10637 files)!
[2022/02/17 22:52:08] playlist.c:135: warn: Parsing playlists…
[2022/02/17 22:52:08] playlist.c:269: warn: Finished parsing playlists.
minidlna.c:1126: warn: Starting MiniDLNA version 1.3.0.
minidlna.c:1186: warn: HTTP listening on port 8200
scanner.c:489: warn: Unsuccessful getting details for /var/www/owncloud/data/admin/files/origin/External SD card/media/Viber Images/IMG-4477de7b1eee273d5e6ae25236c5c223-V.jpg
scanner.c:489: warn: Unsuccessful getting details for /var/www/owncloud/data/Videos/Family-Video/FILE006.MPG
playlist.c:135: warn: Parsing playlists…
playlist.c:269: warn: Finished parsing playlists.


3.5. Fix minidlna Inotify errors

In /etc/sysctl.conf 



in a blank line at end of file and do 

media-server:~# sysctl -p

Debugging minidlna problems, index errors, warnings etc

minidlna does write by default to /var/log/minidlna/minidlna.log inspect the log closely and you should get most of the time what is wrong with it.
Note that some files might not get indexed because minidlna won't support the strange file codecs such as SWF encoding, if you have some important files to stream that are not indexed by minidlna, then install and try one of the more sophisticated free software Media Servers for Linux:


Note that most Linux users from my quick research shows, MediaTomb is the preferred advanced features Open Source Linux Media Server of choice for most of the guys.



4. Test minidlna Linux servers works, getting information of other DLNA Servers on the network

media-server:~# lynx -dump
MiniDLNA status

  Media library

   Audio files 0
   Video files 455
   Image files 10182

  Connected clients

   ID Type                   IP Address    HW Address        Connections
   0  Samsung Series [CDEFJ]  7C:0A:3D:88:A6:FA 0
   1  Generic DLNA 1.5 00:16:4E:1D:48:05 0
   2  Generic DLNA 1.5  00:16:3F:0D:45:05 0
   3  Unknown           FF:FF:FF:FF:FF:FF 0

   -1 connections currently open

Note that there is -1 connections (no active connections) currently to the server. 
The 2 Generic DLNA 1.5 IPs are another DLNA servers provided by a OpenXEN hosted Windows 7 Virtual machines, that are also broadcasting their existence in the network. The Samsung Series [CDEFJ] is the DLNA client on the Samsung TV found, used to detect and stream data from the just configured Linux dlna server.

The DLNA Protocol enabled devices on a network as you can see are quite easy to access, querying localhost on the 8200 server dumps, what minidlna knows, the rest of IPs connecting should not be able to receive this info. But anyways since the minidlna does not have a special layers of security to access it, but the only way to restrict is filtering the 8200 port, it is a very good idea to put a good iptables firewall on the machine to allow only the devices that should have access to the data.

Further more if you happen to need to access the Media files on Linux from GUI you might use some client as upmentioned totem, VLC or if you need something more feature rich Java eezUPnP .


That's all folks !
Enjoy your media on the TV 🙂

Log rsyslog script incoming tagged string message to separate external file to prevent /var/log/message from string flood

Wednesday, December 22nd, 2021


If you're using some external bash script to log messages via rsyslogd to some of the multiple rsyslog understood data tubes (called in rsyslog language facility levels) and you want Rsyslog to move message string to external log file, then you had the same task as me few days ago.

For example you have a bash shell script that is writting a message to rsyslog daemon to some of the predefined facility levels be it:

kern,user,cron, auth etc. or some local

and your logged script data ends under the wrong file location /var/log/messages , /var/log/secure , var/log/cron etc. However  you need to log everything coming from that service to a separate file based on the localX (fac. level) the usual way to do it is via some config like, as you would usually do it with rsyslog variables as:                                            /var/log/custom-log.log

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none        /var/log/messages

Note the local1.none is instructing the rsyslog not to log anything from local1 facility towards /var/log/message. 
But what if this due to some weirdness in configuration of rsyslog on the server or even due to some weird misconfiguration in

/etc/systemd/journald.conf such as:


Due to that config and especially the FowardToSyslog=yes, the messages sent via the logger tool to local1 still end up inside /var/log/messages, not nice huh ..

The result out of that is anything being sent with a predefined TAGGED string via the script which uses the logger command  (if you never use it check man logger) to enter message into rsyslog with cmd like:

# logger -p -t TAG_STRING

# logger -p local2.warn test
# tail -2 /var/log/messages
Dec 22 18:58:23 pcfreak rsyslogd: — MARK —
Dec 22 19:07:12 pcfreak hipo: test

was nevertheless logged to /var/log/message.
Of course /var/log/message becomes so overfilled with "junk" shell script data not related to real basic Operating system adminsitration, so this prevented any critical or important messages that usually should come under /var/log/message / /var/log/syslog to be lost among the big quantities of other tagged tata reaching the log.

After many attempts to resolve the issue by modifying /etc/rsyslog.conf as well as the messed /etc/systemd/journald.conf (which by the way was generated with this strange values with an OS install time automation ansible stuff). It took me a while until I found the solution on how to tell rsyslog to log the tagged message strings into an external separate file. From my 20 minutes of research online I have seen multitudes of people in different Linux OS versions to experience the same or similar issues due to whatever, thus this triggered me to write this small article on the solution to rsyslog.

The solution turned to be pretty easy but requires some further digging into rsyslog, Redhat's basic configuration on rsyslog documentation is a very nice reading for starters, in my case I've used one of the Propery-based compare-operations variable contains used to select my tagged message string.

1. Add msg contains compare-operations to output log file and discard the messages

[root@centos bin]# vi /etc/rsyslog.conf

# config to log everything logged to rsyslog to a separate file
:msg, contains, "tag_string:/"         /var/log/custom-script-log.log
:msg, contains, "tag_string:/"    ~

Substitute quoted tag_string:/ to whatever your tag is and mind that it is better this config is better to be placed somewhere near the beginning of /etc/rsyslog.conf and touch the file /var/log/custom-script-log.log and give it some decent permissions such as 755, i.e.

1.1 Discarding a message

The tilda sign –  

as placed to the end of the msg, contains is the actual one to tell the string to be discarded so it did not end in /var/log/messages.

Alternative rsyslog config to do discard the unwanted message once you have it logged is with the
rawmsg variable, like so:


# config to log everything logged to rsyslog to a separate file
:msg, contains, "tag_string:/"         /var/log/custom-script-log.log
:rawmsg, isequal, "tag_string:/" stop

Other way to stop logging immediately after log is written to custom file across some older versions of rsyslog is via the &stop

:msg, contains, "tag_string:/"         /var/log/custom-script-log.log
& stop

I don't know about other versions but Unfortunately the &stop does not work on RHEL 7.9 with installed rpm package rsyslog-8.24.0-57.el7_9.1.x86_64.

1.2 More with property based filters basic exclusion of string 

Property based filters can do much more, you can for example, do regular expression based matches of strings coming to rsyslog and forward to somewhere.

To select syslog messages which do not contain any mention of the words fatal and error with any or no text between them (for example, fatal lib error), type:

:msg, !regex, "fatal .* error"


2. Create file where tagged data should be logged and set proper permissions

[root@centos bin]# touch /var/log/custom-script-log.log
[root@centos bin]# chmod 755 /var/log/custom-script-log.log

3. Test rsyslogd configuration for errors and reload rsyslog

[root@centos ]# rsyslogd -N1
rsyslogd: version 8.24.0-57.el7_9.1, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

[root@centos ]# systemctl restart rsyslog
[root@centos ]#  systemctl status rsyslog 
● rsyslog.service – System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-12-22 13:40:11 CET; 3h 5min ago
     Docs: man:rsyslogd(8)
 Main PID: 108600 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─108600 /usr/sbin/rsyslogd -n


4. Property-based compare-operations supported by rsyslog table

Compare-operation Description
contains Checks whether the provided string matches any part of the text provided by the property. To perform case-insensitive comparisons, use  contains_i .
isequal Compares the provided string against all of the text provided by the property. These two values must be exactly equal to match.
startswith Checks whether the provided string is found exactly at the beginning of the text provided by the property. To perform case-insensitive comparisons, use  startswith_i .
regex Compares the provided POSIX BRE (Basic Regular Expression) against the text provided by the property.
ereregex Compares the provided POSIX ERE (Extended Regular Expression) regular expression against the text provided by the property.
isempty Checks if the property is empty. The value is discarded. This is especially useful when working with normalized data, where some fields may be populated based on normalization result.


5. Rsyslog understanding Facility levels

Here is a list of facility levels that can be used.

Note: The mapping between Facility Number and Keyword is not uniform over different operating systems and different syslog implementations, so among separate Linuxes there might be diference in the naming and numbering.

Facility Number Keyword Facility Description
0 kern kernel messages
1 user user-level messages
2 mail mail system
3 daemon system daemons
4 auth security/authorization messages
5 syslog messages generated internally by syslogd
6 lpr line printer subsystem
7 news network news subsystem
8 uucp UUCP subsystem
9   clock daemon
10 authpriv security/authorization messages
11 ftp FTP daemon
12 NTP subsystem
13 log audit
14 log alert
15 cron clock daemon
16 local0 local use 0 (local0)
17 local1 local use 1 (local1)
18 local2 local use 2 (local2)
19 local3 local use 3 (local3)
20 local4 local use 4 (local4)
21 local5 local use 5 (local5)
22 local6 local use 6 (local6)
23 local7 local use 7 (local7)

6. rsyslog Severity levels (sublevels) accepted by facility level

As defined in RFC 5424, there are eight severity levels as of year 2021:

Code Severity Keyword Description General Description
0 Emergency emerg (panic) System is unusable. A "panic" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call.
1 Alert alert Action must be taken immediately. Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection.
2 Critical crit Critical conditions. Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection.
3 Error err (error) Error conditions. Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time.
4 Warning warning (warn) Warning conditions. Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full – each item must be resolved within a given time.
5 Notice notice Normal but significant condition. Events that are unusual but not error conditions – might be summarized in an email to developers or admins to spot potential problems – no immediate action required.
6 Informational info Informational messages. Normal operational messages – may be harvested for reporting, measuring throughput, etc. – no action required.
7 Debug debug Debug-level messages. Info useful to developers for debugging the application, not useful during operations.

7. Sample well tuned configuration using severity and facility levels and immark, imuxsock, impstats

Below is sample config using severity and facility levels

# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none        /var/log/messages

Note the local0.none; local1.none tells rsyslog to not log from that facility level to /var/log/messages.

If you need a complete set of rsyslog configuration fine tuned to have a proper logging with increased queues and included configuration for loggint to remote log aggegator service as well as other measures to prevent the system disk from being filled in case if something goes wild with a logging service leading to a repeatedly messages you might always contact me and I can help 🙂
 Other from that sysadmins might benefit from a sample set of configuration prepared with the Automated rsyslog config builder  or use some fine tuned config  for rsyslog-8.24.0-57.el7_9.1.x86_64 on Redhat 7.9 (Maipo)   rsyslog_config_redhat-2021.tar.gz.

To sum it up rsyslog though looks simple and not an important thing to pre

CentOS disable SELinux permanently or one time on grub Linux kernel boot time

Saturday, July 24th, 2021



1. Office 365 cloud connected computer and a VirtualBox hosted machine with SELINUX preventing it to boot

At my job we're in process of migrating my old Lenovo Laptop Thinkpad model L560 Laptop to Dell Latitude 5510 wiith Intel Core i5 vPro CPU and 256 Gb SSD Hard Drive.  The new laptops are generally fiine though they're not even a middle class computers and generally I prefer thinkpads. The sad thing out of this is our employee decided to migrate to Office 365 (again perhaps another stupid managerial decision out of an excel sheet wtih a balance to save some money … 

As you can imagine Office 365 is not really PCI Standards compliant and not secure since our data is stored in Microsoft cloud and theoretically Microsoft has and owns our data or could wipe loose the data if they want to. The other obvious security downside I've noticed with the new "Secure PCI complaint laptop" is the initial PC login screen which by default offers fingerprint authentication or the even worse  and even less secure face recognition, but obviosly everyhing becomes more and more crazy and people become less and less cautious for security if that would save money or centralize the data … In the name of security we completely waste security that is very dubious paradox I don't really understand but anyways, enough rant back to the main topic of this article is how to and I had to disable selinux?

As part of Migration I've used Microsoft OneDrive to copy old files from the Thinkpad to the Latitude (as on the old machine USB's are forbidden and I cannot copy over wiith a siimple USB driive, as well as II have no right to open the laptop and copy data from the Hard driive, and even if we had this right without breaking up some crazy company policy that will not be possible as the hard drive data on old laptop is encrypted, the funny thing is that the new laptop data comes encrypted and there is no something out of the box as BitDefender or McAffee incryption (once again, obviously our data security is a victim of some managarial decisions) …

2. OneDrive copy problems unable to sync some of the copied files to Onedrive

Anyways as the Old Laptop's security is quite paranoid and we're like Fort Nox, only port 80 and port 443 connections to the internet can be initiated to get around this harsh restrictions it was as simple to use a Virtualbox Virtual Machine. So on old laptop I've installed a CentOS 7 image which I used so far and I used one drive to copy my vbox .vdi image on the new laptop work machine.

The first head buml was the .vdi which seems to be prohibited to be copied to OneDrive, so to work around this I had to rename the origianl CentOS7.vdi to CentOS7.vdi-renamed on old laptop and once the data is in one drive copy my Vitualbox VM/ directory from one drive to the Dell Latitude machine and rename the .vdi-named towards .vdi as well as import it from the latest installed VirtualBox on the new machine.

3. Disable SELINUX from initial grub boot

So far so good but as usual happens with miigrations I've struck towards another blocker, the VM image once initiated to boot from Virtualbox badly crashed with some complains that selinux cannot be loaded.
Realizing CentOS 7 has the more or less meaningless Selinux, I've took the opportunity to disable SeLinux.

To do so I've booted the Kernel with Selinux disabled from GRUB2 loader prompt before Kernel and OS Userland boots.



I thought I need to type the information on the source in grub. What I did is very simple, on the Linux GRUB boot screen I've pressed

'e' keyboard letter

that brought the grub boot loader into edit mode.

Then I had to add selinux=0 on the edited selected kernel version, as shown in below screenshot:


Next to boot the Linux VM without Selinux enabled one time,  just had to press together

Ctrl+X then add selinux=0 on the edited selected kernel version, that should be added as shown in the screenshot somewhere after the line of

4. Permanently Disable Selinux on CentOS 7

Once I managed to boot Virtual Machine properly with Oracle Virtualbox, to permanently disabled selinux I had to:


Once booted into CentOS, to check the status of selinux run:


# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31


5. Disable SELinux one time with setenforce command

You can temporarily change the SELinux mode from targeted to permissive with the following command:


# setenforce 0

Next o permanently disable SELinux on your CentOS 7 next time the system boots, Open the /etc/selinux/config file and set the SELINUX mod parameter to disabled.

On CentOS 7 you can  edit the kernel parameters in /etc/default/grub (in the GRUB_CMDLINE_LINUX= key) and set selinux=0 so on next VM / PC boot we boot with a SELINUX disabled for example add   RUB_CMDLINE_LINUX=selinux=0 to the file then you have to regenerate your Grub config like this:

# grub2-mkconfig -o /etc/grub2.cfg
# grub2-mkconfig -o /etc/grub2-efi.cfg

Further on to disable SeLinux on OS level edit /etc/selinux

Default /etc/selinux/config with selinux enabled should look like so:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.

To disable SeLinux modify the file to be something like:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.

6. Check SELINUX status is disabled

# sestatus

SELinux status:                 disabled

So in this article shottly was explained shortly the fake security adopted by using Microsoft Cloud environment Offiice 365, my faced OneDrive copy issues (which prevented even my old laptop Virtual Machine to boot properly and the handy trick to rename the file that is unwilling to get copied from old PC towards m$ OneDrive as well as the grub trick to disable Selinux permanently from grub2.

KVM Virtual Machine RHEL 8.3 Linux install on Redhat 8.3 Linux Hypervisor with custom tailored kickstart.cfg

Friday, January 22nd, 2021


If you don't have tried it yet Redhat and CentOS and other RPM based Linux operationg systems that use anaconda installer is generating a kickstart file after being installed under /root/{anaconda-ks.cfg,initial-setup- ks.cfg,original-ks.cfg} immediately after the OS installation completes. Using this Kickstart file template you can automate installation of Redhat installation with exactly the same configuration as many times as you like by directly loading your /root/original-ks.cfg file in RHEL installer.

Here is the official description of Kickstart files from Redhat:

"The Red Hat Enterprise Linux installation process automatically writes a Kickstart file that contains the settings for the installed system. This file is always saved as /root/anaconda-ks.cfg. You may use this file to repeat the installation with identical settings, or modify copies to specify settings for other systems."

Kickstart files contain answers to all questions normally asked by the text / graphical installation program, such as what time zone you want the system to use, how the drives should be partitioned, or which packages should be installed. Providing a prepared Kickstart file when the installation begins therefore allows you to perform the installation automatically, without need for any intervention from the user. This is especially useful when deploying Redhat based distro (RHEL / CentOS / Fedora …) on a large number of systems at once and in general pretty useful if you're into the field of so called "DevOps" system administration and you need to provision a certain set of OS to a multitude of physical servers or create or recreate easily virtual machines with a certain set of configuration.

1. Create /vmprivate storage directory where Virtual machines will reside

First step on the Hypervisor host which will hold the future created virtual machines is to create location where it will be created:

[root@redhat ~]#  lvcreate –size 140G –name vmprivate vg00
[root@redhat ~]#  mkfs.ext4 -j -b 4096 /dev/mapper/vg00-vmprivate
[root@redhat ~]# mount /dev/mapper/vg00-vmprivate /vmprivate

To view what is the situation with Logical Volumes and  VG group names:

[root@redhat ~]# vgdisplay -v|grep -i vmprivate -A7 -B7
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:0


  — Logical volume —
  LV Path                /dev/vg00/vmprivate
  LV Name                vmprivate
  VG Name                vg00
  LV UUID                VVUgsf-FXq2-TsMJ-QPLw-7lGb-Dq5m-3J9XJJ
  LV Write Access        read/write
  LV Creation host, time, 2021-01-20 17:26:11 +0100
  LV Status              available
  # open                 1
  LV Size                150.00 GiB

Note that you'll need to have the size physically available on a SAS / SSD Hard Drive physically connected to Hypervisor Host.

To make the changes Virtual Machines storage location directory permanently mounted add to /etc/fstab

/dev/mapper/vg00-vmprivate  /vmprivate              ext4    defaults,nodev,nosuid 1 2

[root@redhat ~]# echo '/dev/mapper/vg00-vmprivate  /vmprivate              ext4    defaults,nodev,nosuid 1 2' >> /etc/fstab


2. Second we need to install the following set of RPM packages on the Hypervisor Hardware host

[root@redhat ~]# yum install qemu-kvm qemu-img libvirt virt-install libvirt-client virt-manager libguestfs-tools virt-install virt-top -y

3. Enable libvirtd on the host

[root@redhat ~]#  lsmod | grep -i kvm
[root@redhat ~]#  systemctl enable libvirtd

4. Configure network bridging br0 interface on Hypervisor

In /etc/sysconfig/network-scripts/ifcfg-eth0 you need to include:


Next use nmcli redhat configurator to create the bridge (you can use ip command instead) but since the tool is the redhat way to do it lets do it their way ..

[root@redhat ~]# nmcli connection delete eno3
[root@redhat ~]# nmcli connection add type bridge autoconnect yes con-name br0 ifname br0
[root@redhat ~]# nmcli connection modify br0 ipv4.addresses ipv4.method manual
[root@redhat ~]# nmcli connection modify br0 ipv4.gateway
[root@redhat ~]# nmcli connection modify br0 ipv4.dns
[root@redhat ~]# nmcli connection add type bridge-slave autoconnect yes con-name eno3 ifname eno3 master br0
[root@redhat ~]# nmcli connection up br0

5. Prepare a working kickstart.cfg file for VM

Below is a sample kickstart file I've used to build a working fully functional Virtual Machine with Red Hat Enterprise Linux 8.3 (Ootpa) .

# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda
# Use network installation
#url --url=
##url --url=
# Use text mode install
# System language
#lang en_US.UTF-8
keyboard --vckeymap=us --xlayouts='us'
# Keyboard layouts
##keyboard us
lang en_US.UTF-8
# Root password
rootpw $6$gTiUCif4$YdKxeewgwYCLS4uRc/XOeKSitvDJNHFycxWVHi.RYGkgKctTMCAiY2TErua5Yh7flw2lUijooOClQQhlbstZ81 --iscrypted
# network-stuff
# place ip=your_VM_IP, netmask, gateway, nameserver hostname 
network --bootproto=static --ip= --netmask= --gateway= --nameserver= --device=eth0 --noipv6 --onboot=yes
# if you need just localhost initially configured uncomment and comment above
##network В --device=lo --hostname=localhost.localdomain
# System authorization information
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
# skipx
# Firewall configuration
firewall --disabled
# System timezone
timezone Europe/Berlin
# Clear the Master Boot Record
# Repositories
## Add RPM repositories from KS file if necessery
#repo --name=appstream --baseurl=
#repo --name=baseos --baseurl=
#repo --name=inst.stage2 --baseurl= ff=/dev/vg0/vmprivate
##repo --name=rhsm-baseos В  В --baseurl=
##repo --name=rhsm-appstream --baseurl=
##repo --name=os-baseos В  В  В --baseurl=
##repo --name=os-appstream В  --baseurl=
#repo --name=inst.stage2 --baseurl=
# Disk partitioning information set proper disk sizing
##bootloader --location=mbr --boot-drive=vda
bootloader --append=" crashkernel=auto tsc=reliable divider=10 plymouth.enable=0 console=ttyS0 " --location=mbr --boot-drive=vda
# partition plan
clearpart --all --drives=vda --initlabel
part /boot --size=1024 --fstype=ext4 --asprimary
part swap --size=1024
part pv.01 --size=30000 --grow --ondisk=vda
##part pv.0 --size=80000 --fstype=lvmpv
#part pv.0 --size=61440 --fstype=lvmpv
volgroup s pv.01
logvol / --vgname=s --size=15360 --name=root --fstype=ext4
logvol /var/cache/ --vgname=s --size=5120 --name=cache --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
logvol /var/log --vgname=s --size=7680 --name=log --fstype=ext4 --fsoptions="defaults,nodev,noexec,nosuid"
logvol /tmp --vgname=s --size=5120 --name=tmp --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
logvol /home --vgname=s --size=5120 --name=home --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
logvol /opt --vgname=s --size=2048 --name=opt --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
logvol /var/log/audit --vgname=s --size=3072 --name=audit --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
logvol /var/spool --vgname=s --size=2048 --name=spool --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
logvol /var --vgname=s --size=7680 --name=var --fstype=ext4 --fsoptions="defaults,nodev,nosuid"
# SELinux configuration
selinux --disabled
# Installation logging level
logging --level=debug
# reboot automatically
# Tune Linux vm.dirty_background_bytes (IMAGE-439)
# The following tuning causes dirty data to begin to be background flushed at
# 100 Mbytes, so that it writes earlier and more often to avoid a large build
# up and improving overall throughput.
echo "vm.dirty_background_bytes=100000000" >> /etc/sysctl.conf
# Disable kdump
systemctl disable kdump.service

Important note to make here is the MD5 set root password string in (rootpw) line this string can be generated with openssl or mkpasswd commands :

Method 1: use openssl cmd to generate (md5, sha256, sha512) encrypted pass string

[root@redhat ~]# openssl passwd -6 -salt xyz test

Note: passing -1 will generate an MD5 password, -5 a SHA256 encryption and -6 SHA512 encrypted string (logically recommended for better security)

Method 2: (md5, sha256, sha512)

[root@redhat ~]# mkpasswd –method=SHA-512 –stdin

The option –method accepts md5, sha-256 and sha-512
Theoretically there is also a kickstart file generator web interface on Redhat's site here however I never used it myself but instead use above kickstart.cfg

6. Install the new VM with virt-install cmd

Roll the new preconfigured VM based on above ks template file use some kind of one liner command line  like below:

[root@redhat ~]# virt-install -n RHEL8_3-VirtualMachine –description "CentOS 8.3 Virtual Machine" –os-type=Linux –os-variant=rhel8.3 –ram=8192 –vcpus=8 –location=/vmprivate/rhel-server-8.3-x86_64-dvd.iso –disk path=/vmprivate/RHEL8_3-VirtualMachine.img,bus=virtio,size=70 –graphics none –initrd-inject=/root/kickstart.cfg –extra-args "console=ttyS0 ks=file:/kickstart.cfg"

7. Use a tiny shell script to automate VM creation

For some clarity and better automation in case you plan to repeat VM creation you can prepare a tiny bash shell script:

VM_DESCR='CentOS 8.3 Virtual Machine';
# size is in Gigabytes

virt-install -n "$VMNAME" –description "$VM_DESCR" –os-type=Linux –os-variant=rhel8.3 –ram=8192 –vcpus=8 –location="$ISO_LOCATION" –disk path=$VM_IMG_FILE,bus=virtio,size=$IMG_VM_SIZE –graphics none –initrd-inject=/root/$KS_FILE –extra-args "console=ttyS0 ks=file:/$KS_FILE"

A copy of script can be downloaded here

Wait for the installation to finish it should be visualized and if all installation is smooth you should get a login prompt use the password generated with openssl tool and test to login, then disconnect from the machine by pressing CTRL + ] and try to login via TTY with

[root@redhat ~]# virst list –all
 Id   Name        State
RHEL8_3-VirtualMachine   running

[root@redhat ~]#  virsh console RHEL8_3-VirtualMachine


One last thing I recommend you check the official documentation on Kickstart2 from CentOS official website

In case if you later need to destroy the VM and the respective created Image file you can do it with:

[root@redhat ~]#  virsh destroy RHEL8_3-VirtualMachine
[root@redhat ~]#  virsh undefine RHEL8_3-VirtualMachine

Don't forget to celebreate the success and give this nice article a credit by sharing this nice tutorial with a friend or by placing a link to it from your blog 🙂



Enjoy !

Short SSL generate new and self-signed certificates PEM, view and convert to and from PKCS12 to java key store cookbook commands cheat sheet

Tuesday, January 12th, 2021


Below is a short compilation of common used openssl commands (a kind of cookbook) helpful for sysadmins who has to commonly deal with OpenSSL certificates.

Lets say you have to generate new certificate / key and a PEM files, prepare self-signed certificates, show CSR / PEM or KEY ssl file contents, get information about certificate such as expiry date a type of encryption algorithm or sign certificate with self-signed authority convert PEM to PKCS12, convert from PKCS12 file format to .PEM, convert java X509 to java key store SSL encryptionor convert java key store format certificate to PKCS12, then below will be of use to you.

1. Generate Private RSA Key with 2048 bits

# openssl genrsa -out $ (hostname -f) .key 2048

2. Create CSR file

# openssl req -new -key $ (hostname -f) .key -out $ (hostname -f) .csr

3. Create a Self Certified Certificate:

# openssl x509 -req -days 30 -in $ (hostname -f) .csr -signkey $ (hostname -f) .key -out $ (hostname -f) .crt
Enter password:

# openssl rsa -in key.pem -out newkey.pem

4. Show CSR file content

# openssl req -in newcsr.csr -noout -text

5. Get Certificate version / serial number / signature algorithm / RSA key lenght / modulus / exponent etc.

# openssl x509 -in newcert.pem -noout -text

6. Server certificate as CA self signeded

# openssl ca -in newcert.csr -notext -out newcert.pem

7. Generate a certificate signing request based on an existing certificate

# openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key

8. Convert .pem / .key / .crt file format to pkcs12 format

# openssl pkcs12 -export -in newcert.pem -inkey newkey.key -certfile ca.crt -out newcert.p12

9. Convert pkcs12 pfx to common .pem

# openssl pkcs12 -in mycert.pfx -out mycert.pem

10. The Formats available

# openssl x509 -inform the -in certificate.cer -out certificate.crt

11. Convert a pkcs # 7 certificate into PEM format

# openssl pkcs7 -in cert.p7c -inform DER -outform PEM -out certificate.p7b
# openssl pkcs7 -print_certs -in certificate.p7b -out certificate.pem

12. Convert X509 to java keystore file

# java -cp not-yet-commons-ssl-0.3.11.jar org.apache.commons.ssl.KeyStoreBuilder pass_for_new_keystore key.key certificate.crt

13. Convert java keystore file to pkcs12

# keytool -importkeystore -srckeystore keystore.jks -destkeystore intermediate.p12 -deststoretype PKCS12

Hack: Using ssh / curl or wget to test TCP port connection state to remote SSH, DNS, SMTP, MySQL or any other listening service in PCI environment servers

Wednesday, December 30th, 2020


If you work on PCI high security environment servers in isolated local networks where each package installed on the Linux / Unix system is of importance it is pretty common that some basic stuff are not there in most cases it is considered a security hole to even have a simple telnet installed on the system. I do have experience with such environments myself and thus it is pretty daunting stuff so in best case you can use something like a simple ssh client if you're lucky and the CentOS / Redhat / Suse Linux whatever distro has openssh-client package installed.
If you're lucky to have the ssh onboard you can use telnet in same manner as netcat or the swiss army knife (nmap) network mapper tool to test whether remote service TCP / port is opened or not. As often this is useful, if you don't have access to the CISCO / Juniper or other (networ) / firewall equipment which is setting the boundaries and security port restrictions between networks and servers.

Below is example on how to use ssh client to test port connectivity to lets say the Internet, i.e.  Google / Yahoo search engines.

[root@pciserver: /home ]# ssh -oConnectTimeout=3 -v -p 23
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 23.
debug1: connect to address port 23: Connection timed out
debug1: Connecting to [2a00:1450:4017:80b::200e] port 23.
debug1: connect to address 2a00:1450:4017:80b::200e port 23: Cannot assign requested address
ssh: connect to host port 23: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -oConnectTimeout=3 -v -p 80
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: connect to address port 80: Connection timed out
debug1: Connecting to [2a00:1450:4017:807::200e] port 80.
debug1: connect to address 2a00:1450:4017:807::200e port 80: Cannot assign requested address
ssh: connect to host port 80: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -p 80
ssh_exchange_identification: Connection closed by remote host
root@pcfreak:/var/www/images# ssh -p 80 -v -oConnectTimeout=3
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: connect to address port 80: Connection timed out
debug1: Connecting to [2a00:1450:4017:80b::200e] port 80.
debug1: connect to address 2a00:1450:4017:80b::200e port 80: Cannot assign requested address
ssh: connect to host port 80: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -p 80 -v -oConnectTimeout=5
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: connect to address port 80: Connection timed out
debug1: Connecting to [2a00:1450:4017:80c::200e] port 80.
debug1: connect to address 2a00:1450:4017:80c::200e port 80: Cannot assign requested address
ssh: connect to host port 80: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -p 80 -v
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: ssh_exchange_identification: HTTP/1.0 400 Bad Request


debug1: ssh_exchange_identification: Content-Type: text/html; charset=UTF-8

debug1: ssh_exchange_identification: Referrer-Policy: no-referrer

debug1: ssh_exchange_identification: Content-Length: 1555

debug1: ssh_exchange_identification: Date: Wed, 30 Dec 2020 14:13:25 GMT

debug1: ssh_exchange_identification:

debug1: ssh_exchange_identification: <!DOCTYPE html>

debug1: ssh_exchange_identification: <html lang=en>

debug1: ssh_exchange_identification:   <meta charset=utf-8>

debug1: ssh_exchange_identification:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">

debug1: ssh_exchange_identification:   <title>Error 400 (Bad Request)!!1</title>

debug1: ssh_exchange_identification:   <style>

debug1: ssh_exchange_identification:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(// 10
debug1: ssh_exchange_identification: 0% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.g
debug1: ssh_exchange_identification: no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(// no-repeat 0
debug1: ssh_exchange_identification: % 0%/100% 100%;-moz-border-image:url(// 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//
debug1: ssh_exchange_identification: color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}

debug1: ssh_exchange_identification:   </style>

debug1: ssh_exchange_identification:   <a href=//><span id=logo aria-label=Google></span></a>

debug1: ssh_exchange_identification:   <p><b>400.</b> <ins>That\342\200\231s an error.</ins>

debug1: ssh_exchange_identification:   <p>Your client has issued a malformed or illegal request.  <ins>That\342\200\231s all we know.</ins>

ssh_exchange_identification: Connection closed by remote host


Here is another example on how to test remote host whether a certain service such as DNS (bind) or telnetd is enabled and listening on remote local network  IP with ssh

[root@pciserver: /home ]# ssh -p 53 -v -oConnectTimeout=5
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 53.
debug1: connect to address port 53: Connection timed out
ssh: connect to host port 53: Connection timed out

[root@server: /home ]# ssh -p 23 -v -oConnectTimeout=5
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 23.
debug1: connect to address port 23: Connection timed out
ssh: connect to host port 23: Connection timed out

But what if Linux server you have tow work on is so paranoid that you even the ssh client is absent? Well you can use anything else that is capable of doing a connectivity to remote port such as wget or curl. Some web servers or application servers usually have wget or curl as it is integral part for some local shell scripts doing various operation needed for proper services functioning or simply to test locally a local or remote listener services, if that's the case we can use curl to connect and get output of a remote service simulating a normal telnet connection like this:

host:~# curl -vv 'telnet://remote-server-host5:22'
* About to connect() to remote-server-host5 port 22 (#0)
*   Trying… connected
* Connected to aflpvz625 ( port 22 (#0)

Now lets test whether we can connect remotely to a local net remote IP's Qmail mail server with curls telnet simulation mode:

host:~#  curl -vv 'telnet://'
* Expire in 0 ms for 6 (transfer 0x56066e5ab900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x56066e5ab900)
* Connected to ( port 25 (#0)
220 This is Mail Pc-Freak.NET ESMTP

Fine it works, lets now test whether a remote server who has MySQL listener service on standard MySQL port TCP 3306 is reachable with curl

host:~#  curl -vv 'telnet://'
* Expire in 0 ms for 6 (transfer 0x5601fafae900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x5601fafae900)
* Connected to ( port 3306 (#0)
Warning: Binary output can mess up your terminal. Use "–output -" to tell
Warning: curl to output it to your terminal anyway, or consider "–output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 107)
* Closing connection 0
root@pcfreak:/var/www/images#  curl -vv 'telnet://'
* Expire in 0 ms for 6 (transfer 0x5598ad008900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x5598ad008900)
* Connected to ( port 3306 (#0)
Warning: Binary output can mess up your terminal. Use "–output -" to tell
Warning: curl to output it to your terminal anyway, or consider "–output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 107)
* Closing connection 0

As you can see the remote connection is returning binary data which is unknown to a standard telnet terminal thus to get the output received we need to pass curl suggested arguments.

host:~#  curl -vv 'telnet://' –output –
* Expire in 0 ms for 6 (transfer 0x55b205c02900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x55b205c02900)
* Connected to ( port 3306 (#0)

The curl trick used to troubleshoot remote port to remote host from a Windows OS host which does not have telnet installed by default but have curl instead.

Also When troubleshooting vSphere Replication, it is often necessary to troubleshoot port connectivity as common Windows utilities are not available.
As Curl is available in the VMware vCenter Server Appliance command line interface.

On servers where curl is not there but you have wget is installed you can use it also to test a remote port


# wget -vv -O /dev/null –timeout=5
–2020-12-30 16:54:22–
Resolving (…, 2a00:1450:4017:80b::200e
Connecting to (||:554… failed: Connection timed out.
Connecting to (|2a00:1450:4017:80b::200e|:554… failed: Cannot assign requested address.

–2020-12-30 16:54:28–  (try: 2)
Connecting to (||:554… ^C

As evident from output the port 554 is filtered in google which is pretty normal.

If curl or wget is not there either as a final alternative you can either install some perl, ruby, python or bash script etc. that can opens a remote socket to the remote IP.

How to redirect / forward all postfix emails to one external email address?

Thursday, October 29th, 2020


Lets say you're  a sysadmin doing email migration of a Clustered SMTP and due to that you want to capture for a while all incoming email traffic and redirect it (forward it) towards another single mailbox, where you can review the mail traffic that is flowing for a few hours and analyze it more deeper. This aproach is useful if you have a small or middle sized mail servers and won't be so useful on a mail server that handels few  hundreds of mails hourly. In below article I'll show you how.

How to redirect all postfix mail for a specific domain to single external email address?

There are different ways but if you don't want to just intercept the traffic and a create a copy of email traffic using the always_bcc integrated postfix option (as pointed in my previous article postfix copy every email to a central mailbox).  You can do a copy of email flow via some custom written dispatcher script set to be run by the MTA on each mail arriva, or use maildrop filtering functionality below is very simple example with maildrop in case if you want to filter out and deliver to external email address only email targetted to specific domain.

If you use maildrop as local delivery agent to copy email targetted to specifidc domain to another defined email use rule like:

if ( /^From:.*domain\.com/:h ) {
  cc "!"

To use maildrop to just forward email incoming from a specific sender towards local existing email address on the postfix to an external email address  use something like:

if ( /^From: .**/ )
        dotlock "forward.lock" {
          log "Forward mail"
          to "|/usr/sbin/sendmail"

Then to make the filter active assuming the user has a physical unix mailbox, paste above to local user's  $HOME/.mailfilter.

What to do if your mail delivered via your are sent from a monitoring and alarming scripts that are sending towards many mailboxes that no longer exist after the migration?

To achive capturing all normal attempted to be sent traffic via the mail server, we can forward all served mails towards a single external mail address you can use the nice capability of postfix to understand PCRE perl compatible regular expressions. Regular expressions in postfix of course has its specific I recommend you take a look to the postfix regexp table documentation here, as well as check the Postfix Regex / Tester / Debugger online tool – useful to validate a regexp you want to implement.

How to use postfix regular expression to do a redirect of all sent emails via your postfix mail relayhost towards external mail servers?


In /etc/postfix/ include this line near bottom or as a last line:

virtual_maps = hash:/etc/postfix/virtual, regexp:/etc/postfix/virtual-regexp

One defines the virtual file which can be used to define any of your virtual domains you want to simulate as present on the local postfix, the regexp: does load the file which is read by postfix where you can type the regular expression applied to every incoming email via SMTP port 25 or encrypted MTA ports 385 / 995 etc.

So how to redirect all postfix mail to one external email address for later analysis?

Create file /etc/postfix/virtual-regexp


Next build the mapfile (this will generate /etc/postfix/virtual-regexp.db )

# postmap /etc/postfix/virtual-regexp

This also requires a virtual.db to exist. If it doesn't create an empty file called virtual and run again postmap postfix .db generator

# touch /etc/postfix/virtual && postmap /etc/postfix/virtual

Note in /etc/postfix/virtual you can add your postfix mail domains for which you want the MTA to accept mail as a local mail.

In case you need to view all postfix defined virtual domains configured to accept mail locally on the mail server.

$ postconf -n | grep virtual
virtual_alias_domains =
virtual_alias_maps = hash:/etc/postfix/virtual

The regexp /.+@.+/ applied will start forwarding mails immediately after you reload the MTA with:

# systemctl restart postfix

If you want to exclude target mail domains to not be captured by above regexp, in /etc/postfix/virtual-regexp place:


Time for a test. Send a test email

Next step is to Test it mail forwarding works as expected

# echo -e "Tseting body" | mail -s "testing subject" -r ""