Archive for the ‘System Administration’ Category

Recover lost / forgotten root password for CentOS 7 Linux / Boot CentOS 6 into Single User mode to reset admin pass

Friday, September 27th, 2024

centos-community-enterprise-operating-system-logo.

If you have some old CentOS 7 Virtual machine hanging for a long time and you don't remember the root password or you don't remember where you have stored it, but you have something important as data left over, you might need to recover root password for your CentOS 7 Virtual Machine.

I recently had to resolve that issue and here is the few easy steps to take to recover the lost root password.

Assuming you have tried to boot the VM and the VM boots fine and your few attempts to input manually some default passwords of yours failed, next 

1. Reboot the Virtual Machine to the GRUB boot menu

 

grub.png

The GRUB boot screen should appear and be there for few secs

2. Edit the boot loader kernel options ( add add rd.break enforcing=0 )

 

How to reset root password on CentOS Linux - Clouvider

Press 'e' to Edit the boot loader and modify the boot commands options passed to the linux kernel.

In GRUB edit mode:

add rd.break enforcing=0


to the end of the line starting with linux at the end of passed parameters list as shown in the picture.

When done editing, press Ctrl-x (Control button x key simultaneously) to boot with changed parameters.

ALTERNATIVE WAY TO BOOT THE SYSTEM INTO ROOT WITHOUT PASSWORD PROMPT:

Alternative options to use instead of add rd.break.enforcing=0 are to substitute the rhgb quiet kernel option with init=/bin/bash

Edit CentOS Grub Boot Menu Entries rhgb quiet options shot

Modify kernel parameters pass init=/bin/bash to kernel to boot emergency mode centos linux

 

As you might wonder for the meaning of the passed 2 parameters:

rd.break breaks the boot process at initramfs while
enforcing=0 disables the SELinux (which often enabled by default on CentOS).

Another way is to 

3. Boot in CentOS emergency mode and Reset the root password
 

When done editing, press Ctrl-x to boot with changed parameters.

As you might wonder for the meaning of the passed parameters:

rd.break breaks the boot process at initramfs while
enforcing=0 disables the SELinux (which often enabled by default on CentOS).

Whence system boots up with the modified kernel options cmd, the switch_root prompt will appear.
As the emerency mode boots the filesystem into read-only mode under /sysroot default directory, in order to be able to
modify the MD5 root password stored hash inside RO mounted /sysroot/etc/shadow you need to remount the Filesystme
in read-write mode.

To Remount the read-only file system /sysroot in write mode:

# mount -o remount,rw /sysroot

As the /sysroot is not the root directory to be able to use a standard passwd command you need to make /sysroot
as the default root folder for the booted linux by chrooting into it.
 

  • Generate MD5 password manually (for Hardcore masochistic admins 🙂 )

If you're a hard core linux sysadmin of course, generate your own new md5 password and directly modify /etc/shadow copy pasting the md5 string.

If you want to manually generate the md5 string, you can do it depending on the required encryption algorithm with:

For (md5, sha256, sha512) encrypted pass

# openssl passwd -6 -salt xyz  yourpass

For   (md5, sha256, sha512) encrypted pwd

# mkpasswd –method=SHA-512 –stdin

For (des, md5, sha256, sha512) encrypted pw

# perl -e 'print crypt("YourPasswd", "salt", "sha512"),"\n"'


Once the string is generated;

# vim  /etc/shadow


and exchange the old with new string for MD5

  • Change password with chroot (the easy common way)

remount read write the filesystem in emergency single user mode CentOS LINUX

# chroot /sysroot

That should drop you into another shell bash-4.x

 

Reset root user password in CentOS 7

# passwd
Changing password for user root.
New password:
Retype new password:

We need have to sync the entire filesystem we have to use the sync command, for novice sys admins who never heard about this command, below
short description:

The Linux sync command synchronizes cached data to permanent storage.
This data includes modified superblocks, modified inodes, delayed reads and writes, and others. sync uses several system calls:

sync()
syncfs()
fsync()
fdatasync()


For example, the sync command utilizes the sync() system call to write all buffered modifications to file data and metadata to an underlying storage device.

As a Linux systems administrator or developer, understanding the sync command can be crucial for efficient file synchronization. Additionally, sync can be helpful after crashes or when the file system becomes corrupted.

In this tutorial, we’ll explore the various aspects of the sync command. Also, we’ll see how we can use sync in different scenarios.

# sync

# exec /sbin/init

Try out the root password after booting normally into CentOS and the new set administrator pass should work.


Resetting forgotten (lost) root password on CentOS 6

The process is absolutely the same except on the Step 1 (in the modification of GRUB boot menu by pressing e key), add to

rhgb quiet

at the end one 'S'

This S character means 'boot CentOS into Single user mode'

rhgb quiet S

 

Go to single user mode on CentOS 6 Linux in boot loader S kernel setting

Then, press ENTER key and press b key to boot CentOS 6 into to single user mode.
 

Debug and Fix QMAIL Mail Server qmail-inject: fatal: qq temporary problem (#4.3.0) and ‘reformime[1648048]: segfault at 0 ip 00007fea608bef28 sp 00007fff3c8d4bc0 error 4’ errors after update from Debian 11 to Debian 12

Monday, September 2nd, 2024

finding-qmail-install-problems-common-reasons-for-unworking-qmail-debugging-qmail

For a legacy reasons and lack of time and fact once Qmail is run on a server it works almost forever if you don't do very major upgrades and you still to the same version I have few Qmail SMTP servers that are nowadays are there for historical reasons.

After the last major version upgrade from Debian 11 to Debian 12, I've got the qmail smtpd not completely running fine and I have to follow some of my previous blog notes on how to recover in that situations as well as some common logic to resolve it.

After the upgrade I started getting every few minutes a repeating really annoying error due to reformime crashing in /var/log/messages as well as in qmail logs, the exact error was as so

Sep  1 22:15:34 pcfr_hware_local_ip kernel: [366799.585663] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:17:50 pcfr_hware_local_ip kernel: [366935.524185] reformime[1647438]: segfault at 0 ip 00007f0b9beeff28 sp 00007fff4ffd5850 error 4 in libc.so.6[7f0b9be76000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:17:50 pcfr_hware_local_ip kernel: [366935.524207] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:18:44 pcfr_hware_local_ip kernel: [366989.796532] reformime[1647577]: segfault at 0 ip 00007fe8e14bef28 sp 00007ffc000e9040 error 4 in libc.so.6[7fe8e1445000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:18:44 pcfr_hware_local_ip kernel: [366989.796554] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:20:08 pcfr_hware_local_ip kernel: [367072.889786] reformime[1647888]: segfault at 0 ip 00007efcaa6bef28 sp 00007ffdfe793560 error 4 in libc.so.6[7efcaa645000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:20:08 pcfr_hware_local_ip kernel: [367072.889809] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:21:14 pcfr_hware_local_ip kernel: [367139.010116] reformime[1648048]: segfault at 0 ip 00007fea608bef28 sp 00007fff3c8d4bc0 error 4 in libc.so.6[7fea60845000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:21:14 pcfr_hware_local_ip kernel: [367139.010139] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:22:43 pcfr_hware_local_ip rsyslogd: — MARK —

To debug more concretely what exactly was happening with reformime and why it was crashing with the libc segfault error, I've used the journalctl log with this cmd:

# journalctl -p 3 -xb

сеп 01 22:10:27 pcfrxen qmail-scanner-queue.pl[2170438]: X-Qmail-Scanner-2.10st:[pcfrxen17252178278122170438] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252178278122170438/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:11:11 pcfrxen qmail-scanner-queue.pl[2170631]: X-Qmail-Scanner-2.10st:[pcfrxen17252178718122170631] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252178718122170631/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:15:32 pcfrxen qmail-scanner-queue.pl[2171777]: X-Qmail-Scanner-2.10st:[pcfrxen17252181328122171777] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252181328122171777/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:15:35 pcfrxen qmail-scanner-queue.pl[2171793]: X-Qmail-Scanner-2.10st:[pcfrxen17252181358122171793] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252181358122171793/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:21:21 pcfrxen qmail-scanner-queue.pl[2173427]: X-Qmail-Scanner-2.10st:[pcfrxen17252184788122173427] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252184788122173427/ (Segmentation fau>
                                                            ) – that shouldn't happen!


As you can see this showed that the problem is with reformime's passing on -x argument, and some temporary directory, thus to make sure the crash is not a cause of some mixed permissions, I've had to check the /var/spool/qscan permissions, and clamd permissions and few other permissions of the qmail install, and the wrong permissions (perhaps after the update of clamav after the Debian Linux migration was with /var/lib/clamav which was incorrectly owned by user clamav group clamav instead of the qscand / qscand user group, thus to resolve, I've run:

chown qscand:qscand /var/lib/clamav/ -R


Another thing I've had to correct was the /var/log/qmail permissions which was too permissive (perhaps due to some old install time hurry up stupidity done), so to correct, them:

# chmod 750 /var/log/qmail/


First thing i tried to resolve is of course to reinstall maildrop debian package that provides /usr/bin/reformime binary. 

root@pcfreak:/usr/local/bin# dpkg -l |grep -i maildrop
rc  courier-maildrop                      0.68.2-1                                                                   amd64        Courier mail server – mail delivery agent
ii  maildrop                              2.9.3-2.1                                                                  amd64        mail delivery agent with filtering abilities (set-GID=mail)

In an old post of mine on a similar error Fixing Qmail 451 qq temporary problem (#4.3.0) / @4000000050587780174c60dc status: qmail-todo stop processing asap / status: exiting, part of the solution was to reinstall maildrop, so tried this one:

root@pcfreak:/usr/local/bin# apt install –reinstall maildrop


Of course to try it out restarted qmail with the usual 

# qmailctl restart

Sadly enough this doesn't solve it, so I had to look up for other solutions and spend about 3 / 4 hours reading online just to convince myself that finding any meaningful in the classical human way, is becoming pretty much impossible task. As the content of information on the Internet has grown tremendously over the last years, it seems the quality of posts and commited data is exponentially detereorating. So the only way to solve crashes of binaries is either to stick to a debugger such as gdb or simply try rebuild the .deb binary from scratch and see whether a recompile from source might makes a difference.

After even more digging up online, found out some Gentoo forums threads, where people described thethe issue was also connected to the failing reformime libc use bug, with an applied C patch, found threads on Ubuntu and Debian users complaining about mysterious errors with libc with maildrop and even a bug report that this is some kind of libc bug, related to the precompiled version of maildrop shipped by default deb based repos.

Hence, My approach to resolve it was to recompile maildrop from source code, which even though looking a tedious task came with plenty of dependencies, I had to install plenty of developlment libraries and tools, compilers etc. as well as the following libs.

# apt install pcre2-utils
# apt install libpcre2-dev
# apt install libidn11
# apt install libidn2-dev
# apt install libcourier-unicode-dev
# apt install libcourier-unicode4

Then had to download and install from source the latest available versions of courier-authlib and its dependencies courier-unicode and once having those two recompiled with

# links https://sourceforge.net/projects/courier/files/courier/1.3.12/courier-1.3.12.tar.bz2/download
# links https://sourceforge.net/projects/courier/files/maildrop/3.1.8/maildrop-3.1.8.tar.bz2/download
# links https://sourceforge.net/projects/courier/files/authlib/0.72.3/courier-authlib-0.72.3.tar.bz2/download
# links https://sourceforge.net/projects/courier/files/courier-unicode/2.3.1/courier-unicode-2.3.1.tar.bz2/download

# tar -jxvf courier-unicode-2.3.1.tar.bz2
# tar -jxvvf courier-authlib-0.72.3.tar.bz2
# tar -jxvvf maildrop-3.1.8.tar.bz2

# cd courier-unicode-2.0/
# ./configure && make && make install

# cd ..
# cd courier-authlib-0.72.3
​# ./configure && make && make install
# cd ..

# cd maildrop-3.1.8/
​# ./configure && make && make install

I've took the time to also preinstall a bunch of perl modules deb packages which rawly are the ones found in file, i've built with the binaries perl-modules-for-qmail-needed.txt

To reinstall the binaries, run a small shell loop:

# for i in $(cat perl-modules-for-qmail-needed.txt); do apt install –reinstall $i –yes; done


Have to say also identified an issue with /var/qmail/bin/qmail-scanner-queue.pl with qmail-inject failing after testing qmail-scanner-queue installation with:

# /downloads/qmail-scanner-2.11st/contrib/test_installation.sh -doit
 

# ./test_installation.sh -doit

Sending standard test message – no viruses… 1/4
qmail-inject: fatal: qq temporary problem (#4.3.0)
Bad error. qmail-inject died


Anyone who ever administrated Qmail Mail server knows pretty well, about the Terrible error:

qmail-inject: fatal: qq temporary problem (#4.3.0)


and that it could be mostly anything, thus anyways to find out what might be the cause I've continued to Debug.

# ldd qmail-inject
        linux-vdso.so.1 (0x00007ffc43f5a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcc2099b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcc20ba1000)


To succed debug the QMAIL issues with qmail I found as very useful something from another old post of mine on debugging qmail errors – Testing Qmail installation for problems: Common reasons for unworking qmail / How to debug Qmail mail server failing to delivery or send emails, the following log debug loop:

# for i in $(ls -d /var/log/qmail/*qmail*/); do tail -n 10 $i/current|tai64nlocal; sleep 5; done


Finally the last step resolve the qmail-inject error, was to modify /var/qmail/bin/qmail-scanner-queue.pl and exchange PATH of /usr/bin/reformime default shipped debian repository to new custom built /usr/local/bin/reformime.


After retesting the qmail-scanner installation all seemed fine onwards:
 

# /downloads/qmail-scanner-2.11st/contrib/test_installation.sh -doit

Sending standard test message – no viruses… 1/4
done!

Sending eicar test virus – should be caught by perlscanner module… 2/4
done!

Sending eicar test virus with altered filename – should only be caught by commercial anti-virus modules (if you have any)… 3/4
done!

Sending bad spam message for anti-spam testing – In case you are using SpamAssassin… 4/4


If you have enabled $sa_quarantine, $sa_delete or $sa_reject the
spam-message wont't arrive to the recipients. But if you have enabled
(good idea!) 'debug' you should check
/var/spool/qscan/qmail-queue.log (or where ever you have the log).


        Done!

Finished test. Now go and check Email sent to postmaster@mail.pc-freak.net and/or the log..

Thibs Qmail install qmr_inst_check script also reported my server qmail install scripts as in good state:
 

# /downloads/scripts/qmr_inst_check
! vpopmail database do not exist!

So Hip Hip Hooray my Qmails works again ! Me fixed it again ! 
if you need help with fixing your company Professional Mail QMAIL server or Postfix, contact me via the contact form. Enjoy

 

yum search file in all installable RPM, find out which rpm package provides binary file or missing library dependency on CentOS / RHEL / Fedora

Friday, August 23rd, 2024

images/centos-rhel-yum-clean-var-cache-yum

Sometimes if you have a missing library or a file you know should be available via an rpm but you're not sure which RPM you have to install you have to look up for library or binary file amongs all available installable r[ms on Redhat Linux / CentOS / Fedora or other RPM based distro.

It is really annoying especially, if you try to install an rpm binary and the package does not install due to missing dependency library. Having a missing dependency package could happen, if you use some custom internal prepared repository that is mirroring from original rpm repositories and the RPM Repositories are situated behind a DMZ firewall network (such scenarios are common for corporations and IT companies).
 
Finding out which file is provided by which package on Debian / Ubuntu and other deb based linux distributions is easy and done via the

# apt-file search filename

Thus if you're a system administrator coming from a Debian GNU / Linux sysadmin realm into the wonderful world of redhats, you will want to have an alternative to apt-file tool. You will be happy to find out that that this tedious task is easily done in RPM based Linux and is integrated straight into yum package manager too.

The command to search which rpm package provides a file is:

# yum whatprovides 'nc'

[root@rhel-linux ~]# yum whatprovides nc
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
2:nmap-ncat-6.40-19.el7.x86_64 : Nmap's Netcat replacement
Repo        : base
Matched from:
Provides    : nc

 

2:nmap-ncat-6.40-19.el7.x86_64 : Nmap's Netcat replacement
Repo        : @base
Matched from:
Provides    : nc

 

yum whatprovides search_file_name can be also invoked with its shortcut yum provides 'search_file_name'

[root@rhel-server ~]# yum provides '/bin/ls'
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
coreutils-8.22-24.el7.x86_64 : A set of basic GNU tools commonly used in shell scripts
Repo        : base
Matched from:
Filename    : /bin/ls

coreutils-8.22-24.el7_9.2.x86_64 : A set of basic GNU tools commonly used in shell scripts
Repo        : updates
Matched from:
Filename    : /bin/ls

 Here is another example:

[root@rhel-server ~]# yum -q provides '*lesspipe.sh*'
less-458-9.el7.x86_64 : A text file browser similar to more, but better
Repo        : base
Matched from:
Filename    : /usr/bin/lesspipe.sh

source-highlight-3.1.6-6.el7.i686 : Produces a document with syntax highlighting
Repo        : base
Matched from:
Filename    : /usr/bin/src-hilite-lesspipe.sh

source-highlight-3.1.6-6.el7.x86_64 : Produces a document with syntax highlighting
Repo        : base
Matched from:
Filename    : /usr/bin/src-hilite-lesspipe.sh

spirv-tools-2019.1-4.el7.x86_64 : API and commands for processing SPIR-V modules
Repo        : epel
Matched from:
Filename    : /usr/bin/spirv-lesspipe.sh

You can search for any file and if the RPm repository is defined under /etc/yum/repos.d/* and enabled, yum whatprovides command should be able to find it and tell you which RPM package you have to install to have the file installed Redhat way.

  • You can list all enabled RPM repositories with cmd:
     

[root@rhel-server ~]# yum repolist enabled
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
repo id                                                   repo name                                                                      status
3party                                                    Third party packages – x86_64                                                   2,631
base/7/x86_64                                             CentOS-7 – Base                                                                10,072
cr/7/x86_64                                               CentOS-7 – CR                                                                       0
epel/7/x86_64                                             EPEL packages for RedCent 7 – x86_64                                           13,791
extras/7/x86_64                                           CentOS-7 – Extras                                                                 526
updates/7/x86_64                                          CentOS-7 – Updates                                                              5,802
zabbix-6.0                                                Zabbix 6.0 repo                                                                   429
repolist: 33,251
 

  • To list disable RPM repositories:
     

# yum repolist disabled


To list all present available repositories that could be enabled and are set via the /etc/yum.repos.d/* configs

# yum repolist all

How to Update / Migrate zabbix-agent 5 to zabbix-agent2 6 on Redhat / CentOS / Fedora Linux

Friday, August 9th, 2024

Upgrade-zabbix-agent1-5-to-zabbix-agent2-6-on-RHEL-CentOS-Fedora-Linux-howto-logo

If you have servers reporting monitoring with Zabbix running still on Zabbix-Agent 1 version 5.0.X but already migrated the Zabbix-server to Zabbix 6, it is a good idea to update the Agent to Zabbix Agent 6 As sson as possible, as you know lacking behind in version makes updating harder and more complicated task.

Mine and I guess most system administrators experience points that Keeping at the same level of versioning on many applications historically has shown to reduce unexpected errors and bugs but nowadays, the rule of keeping local and remote application ( programs )  at the same version level is regularly broken.

Theoretically Zabbix-Agent (Client) and Zabbix (Server) has a compitability for a certain range of versions (Zabbix agents 2 from version 4.4 onwards are compatible with Zabbix 7.0; Zabbix agent 2 must not be newer than 7.0 – for more on zabbix agent – > server version compitability check here ) and having a slight version difference should not be really a problem but often you might have a third party proxies in between such as haproxy or zabbix-proxy or other network oddities and thus my personal opinion is that for interoperability it is better to keep the Zabbix Clients and Zabbix Servers across the DMZ-ed networks running at same version level.

Some would say I have an old fashion thinking as software and technology is moving forward, but as I see how programming code writing and even software is constantly degradating just a reflection of degradation of human element, I prefer to keep my old know how and always stick to same versioning whenever possible.

Some would wonder then why would I upgrade to Zabbix-agent2 ? , if have to keep the same versioning, the reason is zabbix-agent2 is written in GO Language and is much faster and supposably better piece of software than Zabbix Agent1 that is written in Python.

Moreover having Zabbix agent 2 instead of 1 gives also benefits as you can do a bit more with zabbix and on the other hand the machines are more ready for monitoring in terms of future. To know more about the Benefits of Zabbix Agent2 compared to Zabbix Agent 1 read the Agent vs Agent2 comparison on zabbix website.

 

With this little introduction, lets proceed with the exact steps to take to upgrade zabbix-agent1 to zabbix-agent2.

1. Check the current installed Zabbix-Agent version 

[user@monitored-server ~]$ rpm -qa |grep -i zabb
zabbix-get-5.0.42-1.el8.x86_64
zabbix-sender-5.0.42-1.el8.x86_64
zabbix-agent-5.0.42-1.el8.x86_64

[user@server ~]$ 

 

2. Create backup copy of current system working zabbix_agentd.conf
 

Before messing up with the working zabbix-agent as usual create the necessery backup to prevent later suprises

[user@monitored-server ~]$ cp -vrpf /etc/zabbix/zabbix_agentd.conf /etc/zabbix/zabbix_agentd.conf.bak-$(date '+%Y-%m-%d_%H-%M-%S')

3. Check current configured Zabbix repos

 

[user@monitored-server ~]$ vim /etc/yum.repos.d/zabbix.repo
 

[zabbix-4.0]
name = zabbix-4.0 – 8
baseurl = http://zabbix-repo-server.com/external/zabbix-4.0/8/$basearch
enabled = 0
gpgkey = http://zabbix-repo-server.com/external/zabbix-4.0/zabbix-official-repo.key
gpgcheck = 1

[zabbix-4.4]
name = zabbix-4.4 – 8
baseurl = http://zabbix-repo-server.com/external/zabbix-4.4/8/$basearch
enabled = 0
gpgkey = http://zabbix-repo-server.com/external/zabbix-4.4/zabbix-official-repo.key
gpgcheck = 1

[zabbix-5.0]
name = zabbix-5.0 – 8
baseurl = http://zabbix-repo-server.com/external/zabbix-5.0/8/$basearch
enabled = 1
gpgkey = http://zabbix-repo-server.com/external/zabbix-5.0/zabbix-official-repo.key
gpgcheck = 1

[zabbix-5.4]
name = zabbix-5.4 – 8
baseurl = http://zabbix-repo-server.com/external/zabbix-5.4/8/$basearch
enabled = 0
gpgkey = http://zabbix-repo-server.com/external/zabbix-5.4/zabbix-official-repo.key
gpgcheck = 1

[zabbix-6.0]
name = zabbix-6.0 – 8
baseurl = http://zabbix-repo-server.com/external/zabbix-6.0/8/$basearch
enabled = 0
gpgkey = http://zabbix-repo-server.com/external/zabbix-6.0/zabbix-official-repo.key
gpgcheck = 1


4. Modify repositories and include the Zabbix Agent6 yum repos 
 

[user@monitored-server ~]$ cp -rpf zabbix.repo zabbix.repo.5.0.rpmsave

As we want to keep only the 6.0 version, leave only the zabbix-6.0 section and enable the repo:
 

[user@monitored-server ~]$ vim /etc/yum.repos.d/zabbix.repo

[zabbix-6.0]
name = zabbix-6.0 – 8
baseurl = http://zabbix-repo-server.com/external/zabbix-6.0/8/$basearch
enabled = 1
gpgkey = http://zabbix-repo-server.com/external/zabbix-6.0/zabbix-official-repo.key
gpgcheck = 1


5. Update zabbix-agent to zabbix-agent2 and update zabbix-get zabbix-sender versions

To not disrupt reported monitoring for zabbix-agent, don't delete zabbix-agent1 but instead in pararallel install and configure
zabbix-agent2 and then once configuration is migrated from Agent 1 to 2, stop the old zabbix-agent and bring up the new one.

[user@monitored-server ~]$ yum check-update

[user@monitored-server ~]$ yum install zabbix-agent2 zabbix-get zabbix-sender -y

Note that if you want to have a precise version number of zabbix-agent that is lets say 6.0.31 to correspond to zabbix-server 6.0.31 (even though in the repositories newer RPM versions are available), run:
 

[user@monitored-server ~]$ yum upgrade zabbix-agent2-6.0.31-release1.el8

 

  • Check new zabbix_agent2 installed version 


# zabbix_agent2 -V
zabbix_agent2 (Zabbix) 6.0.31
Revision b6d93755a1b 17 June 2024, compilation time: {undefined} {undefined}, built with: go1.21.3
Plugin communication protocol version is 6.0.13

Copyright (C) 2024 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 1.1.1k  FIPS 25 Mar 2021
Running with OpenSSL 1.1.1k  FIPS 25 Mar 2021

We use the library Eclipse Paho (eclipse/paho.mqtt.golang), which is
distributed under the terms of the Eclipse Distribution License 1.0 (The 3-Clause BSD License)
available at https://www.eclipse.org/org/documents/edl-v10.php

We use the library go-modbus (goburrow/modbus), which is
distributed under the terms of the 3-Clause BSD License
available at https://github.com/goburrow/modbus/blob/master/LICENSE

 

6. Migrate old /etc/zabbix/zabbix_agentd.conf to /etc/zabbix/zabbix-agent2.conf

For readability to show the main configured variables for zabbix-agent without the tons of comments, to later include in agent2
 

[root@monitored-server ~]# cat /etc/zabbix/zabbix_agentd.conf | grep -v '\#' | sed '/^$/d' 
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
Server=10.50.37.8,127.0.0.1
ServerActive=10.50.37.8,127.0.0.1
Hostname=fqdn-of-monitored-host.domain.com
Timeout=20
Include=/etc/zabbix/zabbix_agentd.d/*.conf

The default zabbix-agent2 installed config would like similar to:

[root@monitored-server ~]# cat /etc/zabbix/zabbix_agent2.conf | grep -v '\#' | sed '/^$/d'
PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=0
Server=127.0.0.1
# Specify the location of the Zabbix server host.
ServerActive=127.0.0.1
Hostname=Zabbix server
Include=/etc/zabbix/zabbix_agent2.d/*.conf
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
Include=./zabbix_agent2.d/plugins.d/*.conf

The new migrate one, should be like:

[root@monitored-server ~]# vim /etc/zabbix/zabbix_agent2.conf
PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=10
Server=10.34.89.7,127.0.0.1
ServerActive=10.34.89.7,127.0.0.1
Hostname=lqgblu02f.ffm.de.int.atosorigin.com
Timeout=20
Include=/etc/zabbix/zabbix_agent2.d/*.conf
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf


7. Add few Optimization variables for better zabbix-server -> zabbix-proxy -> zabbix-server interactions 

If you have sometimes a network delays between zabbix server -> zabbix client and vice versa (depending on whether Zabbix agent is configured as Active or Passive mode), it is often useful 
to add those 2 variables:

# How often list of active checks is refreshed, in seconds
RefreshActiveChecks=60
# Refresh the active checks on start.ForceActiveChecksOnStart=1
ForceActiveChecksOnStart=1


Also it might be a good practice to add zabbix_agent2.log monitoring with the agent itself, if the log exceeds certain amount, instead of calling it via logrotate.
 

# Perform log file rotation at the 1 MB point for the specified filepath
LogFileSize=1

 

[root@monitored-server ~]# vim /etc/zabbix/zabbix_agent2.conf
PidFile=/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=10
Server=10.34.89.7,127.0.0.1
ServerActive=10.34.89.7,127.0.0.1
Hostname=lqgblu02f.ffm.de.int.atosorigin.com
RefreshActiveChecks=60
ForceActiveChecksOnStart=1
Timeout=20
Include=/etc/zabbix/zabbix_agent2.d/*.conf
PluginSocket=/run/zabbix/agent.plugin.sock
ControlSocket=/run/zabbix/agent.sock
Include=/etc/zabbix/zabbix_agent2.d/plugins.d/*.conf

 

8. Stop the old zabbix agent process and run the new one

# systemctl status –full zabbix-agent2
# systemctl stop zabbix-agent


Assuming that the configuratoin of zabbix-agent is correct, execute zabbix-agent2 via system control.and check its status
 

# systemctl start zabbix-agent2
# systemctl status –full zabbix-agent2


If no errors in the configuration, the zabbix_agent2 process should be up and running and the status of above systemctl cmd should report fine.
If you need concretics regarding exact Zabbix checks or whther current conigured Userparameter scripts errors, or any other warnings or errors
of zabbix_agent2 interacting to the server, check further the logs

[root@monitored-server ~]# tail -n 10 /var/log/zabbix/zabbix_agent2.log  
2024/08/06 17:26:52.998749 using plugin 'WebPage' (built-in) providing following interfaces: exporter, configurator
2024/08/06 17:26:52.998760 using plugin 'ZabbixAsync' (built-in) providing following interfaces: exporter
2024/08/06 17:26:52.998794 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2024/08/06 17:26:52.998804 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 100 exceeds limits
2024/08/06 17:26:52.998820 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2024/08/06 17:26:52.998993 Plugin communication protocol version is 6.0.13
2024/08/06 17:26:52.999018 Zabbix Agent2 hostname: [lqgblu02f.ffm.de.int.atosorigin.com]
2024/08/06 17:26:54.000667 [102] cannot connect to [127.0.0.1:10051]: dial tcp :0->127.0.0.1:10051: connect: connection refused
2024/08/06 17:26:54.000836 [102] active check configuration update from host [lqgblu02f.ffm.de.int.atosorigin.com] started to fail
2024/08/06 17:26:59.344837 Zabbix Agent 2 stopped. (6.0.31)

All Debian Linux package repository apt sources.list file for Debian versions 6, 7, 8, 9, 10, 11 and 12

Friday, May 31st, 2024

debian-package-management-repositories-for-all-distributions

If you have to administrate legacy Debian servers, that keeps hanging either for historical reasons or just because you didn't have time to upgrade it up to latest versions, machines that are hanging in the hangar or a mid office building Old server room, doing nothing but simply NAT (Network Address Translation), Proxying, serving  traffic via Squid / Haproxy / Apache / Varnish or Nginx server but you still want to have the possibility to extend the OS even though it is out of date / End of Life reached and out of support as well as perhaps full of security holes, but due to its unvisibility on the Internet hanging in a Demilitarized network the machine stayed on the Local (DMZ)-ed network and still for example you need to install simple things for administration reasons locally on the machine, for example nmap or netcat or some of the network tools for monitoring such as iftop or iptraf etc. you might find out unfortunately that this is not possible anymore, because the configured /etc/apt/sources.list repository mirror is no longer available at its URL. Thus to restore the functioning of apt and apt-get pkg management tools on Debian you need to correct the broken missing package mirrors due to resructurings on the network with a correct ones, originally provided by Debian or eventually if this doesn't work a possible Debian package archive URL. 

In this article, I'll simply provide such URLs you might use to correct your no longer functioning package manager due to package repositoriy unavailibility, below are the URLs (most of which that should be working as of year 2024). To resolve the issues edit and place the correct Debian version you're using.

1. Check the version of the Debian Linux

# cat /etc/debian_version


or use the universal way to check the linux OS, that should be working on almost all Linux distributions

# cat /etc/issue
Debian GNU/Linux 9 \n \l

2. Modify /etc/apt/sources.list and place URL according to Debian distro version

# vim /etc/apt/sources.list


3. Repositories URL list Original and Archived for .deb packages according to Debian distro release
Debian 6 (Wheezy)

Original repostiroes (Not Available and Not working anymore as of year 2024)

 

Old Archived .deb repository for 6 Squeeze

deb http://archive.debian.org/debian squeeze main
deb http://archive.debian.org/debian squeeze-lts main


​Debian 7 (Wheezy)

Original repostiroes (Not Available and Not working anymore as of year 2024)

Old Archived .deb repository for Jessie (still working as of 2024) :

deb http://archive.debian.org/debian wheezy main contrib non-free
deb http://archive.debian.org/debian-security wheezy/updates main

( Security updates are not provided anymore.)

NOTE:  If you get an error about keyrings, just install it
 

# apt-get install debian-archive-keyring


Debian 8 (Jessie)
Original .deb package repository with non-free included for Debian 8 "Jessie"

deb http://deb.debian.org/debian/ jessie main contrib non-free
deb http://ftp.debian.org/debian/ jessie-updates main contrib
deb http://security.debian.org/ jessie/updates main contrib non-free

Old Archived .deb repository for 8 Jessie (still working as of 2024):

deb http://archive.debian.org/debian/ jessie main non-free contrib
deb-src http://archive.debian.org/debian/ jessie main non-free contrib
deb http://archive.debian.org/debian-security/ jessie/updates main non-free contrib
deb-src http://archive.debian.org/debian-security/ jessie/updates main non-free contrib

 

# echo "Acquire::Check-Valid-Until false;" | tee -a /etc/apt/apt.conf.d/10-nocheckvalid

# apt-get update

# apt-get update && apt-get upgrade

 

 If you need backports, first be warned that these are archived and no longer being updated; they may have security bugs or other major issues. They are not supported in any way.

deb http://archive.debian.org/debian/ jessie-backports main


Debian 9 (Stretch)
Original .deb package repository with non-free included for Debian 9 "Stretch":

 

deb http://deb.debian.org/debian/ stretch main contrib non-free
deb http://deb.debian.org/debian/ stretch-updates main contrib non-free
deb http://security.debian.org/ stretch/updates main contrib non-free

Archived old repository .deb for Stretch :

deb http://archive.debian.org/debian/ stretch main contrib non-free
deb http://archive.debian.org/debian/ stretch-proposed-updates main contrib non-free
deb http://archive.debian.org/debian-security stretch/updates main contrib non-free


Debian 10 (Buster)
Origian repository URL:

deb http://deb.debian.org/debian/ buster main non-free contrib
deb http://deb.debian.org/debian/ buster-updates main non-free contrib
deb http://security.debian.org/ buster/updates main non-free contrib

 

Fixing unworking backports for Debian 10 Buster


Change the /etc/apt/sources.list URL with this one

deb http://archive.debian.org/debian buster-backports main contrib non-free


If you want to list packages installed via the backports repository only, that needs to be replaced with newer versions (if such available from the repository)

# apt list –installed | grep backports
# dpkg –list | grep bpo
# dpkg –list | grep -E '^ii.*bpo.*'

ii  libpopt0:amd64                        1.18-2                         amd64        lib for parsing cmdline parameters
ii  libuutil3linux                        2.0.3-9~bpo10+1                amd64        Solaris userland utility library for Linux
ii  libzfs4linux                          2.0.3-9~bpo10+1                amd64        OpenZFS filesystem library for Linux


Debian 11 (Bullseye)
Origianl repository address:

deb http://deb.debian.org/debian bullseye main contrib non-free
deb http://deb.debian.org/debian bullseye-updates main contrib non-free
deb http://security.debian.org/debian-security bullseye-security main contrib non-free

Debian 12 (Bookworm)
Original Repository :

 

deb http://deb.debian.org/debian bookworm main contrib non-free-firmware non-free
deb http://deb.debian.org/debian bookworm-updates main contrib non-free-firmware non-free
deb http://security.debian.org/debian-security bookworm-security main contrib non-free-firmware non-free

Add Backports to sources.list

deb http://deb.debian.org/debian bookworm-backports main


Thats all, hopefully that would help some sysadmin out there. Enjoy !

How to view WIFI Passwords for Profile from command line with netsh on Windows 10

Wednesday, May 29th, 2024

how-to-find-out-your-wifi-password-on-windows-10

The common way, if you have connected to a Wireless Network Access Point and saved the password in Windows is to view the password via Windows GUI interface, via menus following few easy steps:

1. Settings -> Network and Internet -> Network and Sharing Center

network-and-sharing-center
2. Click on (Wifi Network name) for which you need password and 
3. In View your active networks section

select-wifi
4. When the Wi-Fi network status window opens, click Wireless Properties

wireless-properties
5. Move to the Security Tab and check the checkbox, next to "Show Characters" to view the network password.

show-wifi-password-windows-10
 

Nevertheless as a system administrator you might have wondered, how you can easily review in plain text Saved Wireless Networks Wi-FI passwords, without using the Graphical Interface via a direct command line cmd.exe?
Such thing is helpful on maintaining multiple Windows 10 hosts, especially if you have a telnet or SSH remote administration enabled or you have a domain of PCs.
To do so open cmd.exe command prompt and run:

C:\Users> netsh

netsh>wlan show profile

Profiles on interface Wi-Fi:

Group policy profiles (read only)
———————————

User profiles
————-
All User Profile : WIFI_Pofile-name
All User Profile: Hotel stage 2
All User Profile: Home Wifi
All User Profile: HP_Custom

Now lets review the clear text password of the profile from netsh console:

netsh>wlan show profile "WIFI_Pofile-name" key=clear

Profile WIFI_Pofile-name on interface Wi-Fi:
===================================================

Applied: All User Profile

Profile information
——————-
Version : 1
Type : Wireless LAN
Name : WIFI_Pofile-name
Control options :
Connection mode : Connect automatically
Network broadcast : Connect only if this network is broadcasting
AutoSwitch : Do not switch to other networks
MAC Randomization : Disabled

Connectivity settings
———————
Number of SSIDs : 1
SSID name : "WIFI_Pofile-name"
Network type : Infrastructure
Radio type : [ Any Radio Type ]
Vendor extension : Not present

Security settings
—————–
Authentication : WPA2-Personal
Cipher : CCMP
Authentication : WPA2-Personal
Cipher : GCMP
Security key : Present
Key Content : Very-secret-password-for-WIFI-plain-text

TADADAM !

We see the password key text Saved WIFI Passwords plain text !

Note that sometimes, if you have a Hidden Wifi Network the command to use to reveal the plain text password with netsh would be:

C:\Users> netsh wlan show profile "name=SSID hidden WiFi Net" key=clear


This trick is very much used today by "hackers" e.g. script kiddies, who break up into others windows.
It is also useful if you want to have a quick way to review plain text passwords for WIFI accounts with organization, lets say if you're a security expert and doing some kind of periodic Security audits within a corporation on multiple Domain attached computers.

Thanks to Martin Petrov (Amridikon) for his trick as I've learned first time from his blog https://mpetrov.net, which is full of many computer geek goodies stuff.

Of course this approach can be easily scripted with a short PowerShell script:
 

netsh wlan show profile |
    Select-String '(?<=All User Profile\s+:\s).+' |
    ForEach-Object {
        $wlan = $_.Matches.Value
        $passw = netsh wlan show profile $wlan key=clear |
            Select-String '(?<=Key Content\s+:\s).+'

        [pscustomobject]@{
            Name     = $wlan
            Password = $passw.Matches.Value
        }
    }

 

If you need the script View-all-wifi-passwords-plaintext-windows10.ps1 to reuse it download it from here.
 

Windows-WiFi-PasswordRevealer-ScreenShot
There is also some freeware tools online which can help you reveal passwords, saving you any typing, that might be useful if you want to delegate the task to a non-sysadmin user, you can simply point him and ask him to install a GUI Win tool like Wifi Password revealer (that makes showing plain text passwords piece of cake) and let user reveal his passwords for himself, if needs the password to share it to a colleague 🙂
That's all folks, Happy hacking !

Haproxy Enable / Disable Application backend server configured to roundrobin in emergency case via haproxy socket command

Thursday, May 2nd, 2024

haproxy-stats-socket

Haproxy LB backend BACKEND_ROUNDROBIN are configured to roundrobin with check health check port  (check port 33333).
For example letsa say haproxy server is running with a haproxy_roundrobin.cfg like this one.

Under some circumstances however if check port TCP 33333 is UP, but behind 1 or more of Application that is providing the resources to customers misbehaves ,
(app-server1, app-server2, app-server3, app-server4) members , Load Balancer cannot know this, because traffic routing decision is made based on Echo port.

One example scenario when this can happen is if Application server has issue with connectivity towards Database hosts:
(db-host1, db-host2, db-host3, db-host4)

If this happens 25% of traffic might still get balanced to broken Application server. If such scenario happens during OnCall and this is identified as problem,
work around would be to temporary disable the misbehaving App servers member from the 4 configured roundrobin pairs in haproxyproduction.cfg :

For example if app-server3 App node is identified as failing and 25% via LB is lost, to resolve it until broken Application server node is fixed, you will have to temporary exclude it from the ring of roundrobin backend hosts.

1.  Check the status of haproxy backends

echo "show stat" | socat stdio /var/lib/haproxy/stats

As you can see the backend is disabled.

Another way to do it which will make your sessions to the server not directly cut but kept for some time is to put the server you want to exclude from haproxy roundrobin to "maintenace mode".

echo "set server bk_BACKEND_ROUNDROBIN/app-server3 state maint" | socat unix-connect:/var/lib/haproxy/stats stdio

Actually, there is even better and more advanced way to disable backend from a configured rounrobin pair of hosts, with putting the available connections in a long waiting queue in the proxy, and if the App host is inavailable for not too short, haproxy will just ask the remote client to keep the connection for longer and continue the session interaction to remote side and wait for the App server connectivity to go out of maintenance, this is done via "drain" option.

echo "set server bk_BACKEND_ROUNDROBIN/app-server3 state drain" | socat unix-connect:/var/lib/haproxy/stats stdio

 

  • This sets the backend in DRAIN mode. No new connections are accepted and existing connections are drained.

To get a better idea on what is drain state, here is excerpt from haproxy official documentation:

Force a server's administrative state to a new state. This can be useful to
disable load balancing and/or any traffic to a server. Setting the state to
"ready" puts the server in normal mode, and the command is the equivalent of
the "enable server" command. Setting the state to "maint" disables any traffic
to the server as well as any health checks. This is the equivalent of the
"disable server" command. Setting the mode to "drain" only removes the server
from load balancing but still allows it to be checked and to accept new
persistent connections. Changes are propagated to tracking servers if any.


2. Disable backend app-server3 from rounrobin 


 

echo "disable server BACKEND_ROUNDROBIN/app-server3" | socat unix-connect:/var/lib/haproxy/stats stdio

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
stats,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,282917,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Frontend_Name,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
Backend_Name,app-server4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,1,0,282917,0,,1,4,1,,0,,2,0,,0,L4OK,,12,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,app-server3,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,0,1,1,2,2,23,,1,4,2,,0,,2,0,,0,L4OK,,11,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,282917,0,,1,4,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,

Once it is confirmed from Application supprt colleagues, that machine is out of maintenance node and working properly again to reenable it:

3. Enable backend app-server3

echo "enable server bk_BACKEND_ROUNDROBIN/app-server3" | socat unix-connect:/var/lib/haproxy/stats stdio

4. Check backend situation again

echo "show stat" | socat stdio /var/lib/haproxy/stats
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
stats,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,282955,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,
Frontend_Name,FRONTEND,,,0,0,3000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
Backend_Name,app-server4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,1,0,282955,0,,1,4,1,,0,,2,0,,0,L4OK,,12,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,app-server3,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,0,1,1,2,3,58,,1,4,2,,0,,2,0,,0,L4OK,,11,,,,,,,0,,,,0,0,,,,,-1,,,0,0,0,0,
Backend_Name,BACKEND,0,0,0,0,300,0,0,0,0,0,,0,0,0,0,UP,1,1,1,,0,282955,0,,1,4,0,,0,,1,0,,0,,,,,,,,,,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,


You should see the backend enabled again.

NOTE:
If you happen to get some "permission denied" errors when you try to send haproxy commands via the configured haproxy status this might be related to the fact you have enabled the socket in read only mode, if that is so it means the haproxy cannot be written to and therefore you can only read info from it with status commands, but not send any write operations to haproxy via unix socket.

One example haproxy configuration that enables haproxy socket in read only looks like this in haproxy.cfg:
 

 stats socket /var/lib/haproxy/stats


To make the haproxy socket read / write mode, for root superuser and some other users belonging to admin group 'adm', you should set the haproxy.cfg to something like:

stats socket /var/lib/haproxy/stats-qa mode 0660 group adm level admin

or if no special users with a set admin group needed to have access to socket, use instead config like:

stats socket /var/lib/haproxy/stats-qa.sock mode 0600 level admin

Improve haproxy logging with custom log-format for better readiability

Friday, April 12th, 2024

Haproxy logging is a very big topic, worthy of many articles, but unfortunately not enough is written on the topic, perhaps for the reason haproxy is free software and most people who use it doesn't follow the philosophy of free software sharing but want to keep, the acquired knowledge on the topic for their own and if possible in the capitalist world most of us live to use it for a Load Balancer haproxy consultancy, consultancy fee or in their daily job as system administrators (web and middleware) or cloud specialist etc. 🙂

Having a good haproxy logging is very important as you need to debug issues with backend machines or some other devices throwing traffic to the HA Proxy.
Thus it is important to build a haproxy logging in a way that it provides most important information and the information is as simple as possible, so everyone can understand what is in without much effort and same time it contains enough debug information, to help you if you want to use the output logs with Graylog filters or process data with some monitoring advanced tool as Prometheus etc.

In our effort to optimize the way haproxy logs via a configured handler that sends the haproxy output to logging handler configured to log through rsyslog, we have done some experiments with logging arguments and came up with few variants, that we liked. In that article the idea is I share this set of logging  parameters with hope to help some other guy that starts with haproxy to build a good logging readable and easy to process with scripts log output from haproxy.

The criterias for a decent haproxy logging used are:

1. Log should be simple but not dumb
2. Should be concrete (and not too much complicated)
3. Should be easy to read for the novice and advanced sysadmin

Before starting, have to say that building the logging format seems tedious task but to make it fit your preference could take a lot of time, especially as logging parameters naming is hard to remember, thus the haproxy logging documentation log-format description table comes really handy:

Haproxy log-format paremeters ASCII table
 

 Please refer to the table for log-format defined variables :
 

+---+------+-----------------------------------------------+-------------+
| R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
+---+------+-----------------------------------------------+-------------+
|   | %o   | special variable, apply flags on all next var |             |
+---+------+-----------------------------------------------+-------------+
|   | %B   | bytes_read           (from server to client)  | numeric     |
| H | %CC  | captured_request_cookie                       | string      |
| H | %CS  | captured_response_cookie                      | string      |
|   | %H   | hostname                                      | string      |
| H | %HM  | HTTP method (ex: POST)                        | string      |
| H | %HP  | HTTP request URI without query string (path)  | string      |
| H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string      |
| H | %HU  | HTTP request URI (ex: /foo?bar=baz)           | string      |
| H | %HV  | HTTP version (ex: HTTP/1.0)                   | string      |
|   | %ID  | unique-id                                     | string      |
|   | %ST  | status_code                                   | numeric     |
|   | %T   | gmt_date_time                                 | date        |
|   | %Ta  | Active time of the request (from TR to end)   | numeric     |
|   | %Tc  | Tc                                            | numeric     |
|   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                 | numeric     |
|   | %Tl  | local_date_time                               | date        |
|   | %Th  | connection handshake time (SSL, PROXY proto)  | numeric     |
| H | %Ti  | idle time before the HTTP request             | numeric     |
| H | %Tq  | Th + Ti + TR                                  | numeric     |
| H | %TR  | time to receive the full request from 1st byte| numeric     |
| H | %Tr  | Tr (response time)                            | numeric     |
|   | %Ts  | timestamp                                     | numeric     |
|   | %Tt  | Tt                                            | numeric     |
|   | %Tw  | Tw                                            | numeric     |
|   | %U   | bytes_uploaded       (from client to server)  | numeric     |
|   | %ac  | actconn                                       | numeric     |
|   | %b   | backend_name                                  | string      |
|   | %bc  | beconn      (backend concurrent connections)  | numeric     |
|   | %bi  | backend_source_ip       (connecting address)  | IP          |
|   | %bp  | backend_source_port     (connecting address)  | numeric     |
|   | %bq  | backend_queue                                 | numeric     |
|   | %ci  | client_ip                 (accepted address)  | IP          |
|   | %cp  | client_port               (accepted address)  | numeric     |
|   | %f   | frontend_name                                 | string      |
|   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
|   | %fi  | frontend_ip              (accepting address)  | IP          |
|   | %fp  | frontend_port            (accepting address)  | numeric     |
|   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
|   | %lc  | frontend_log_counter                          | numeric     |
|   | %hr  | captured_request_headers default style        | string      |
|   | %hrl | captured_request_headers CLF style            | string list |
|   | %hs  | captured_response_headers default style       | string      |
|   | %hsl | captured_response_headers CLF style           | string list |
|   | %ms  | accept date milliseconds (left-padded with 0) | numeric     |
|   | %pid | PID                                           | numeric     |
| H | %r   | http_request                                  | string      |
|   | %rc  | retries                                       | numeric     |
|   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
|   | %s   | server_name                                   | string      |
|   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
|   | %si  | server_IP                   (target address)  | IP          |
|   | %sp  | server_port                 (target address)  | numeric     |
|   | %sq  | srv_queue                                     | numeric     |
| S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
| S | %sslv| ssl_version (ex: TLSv1)                       | string      |
|   | %t   | date_time      (with millisecond resolution)  | date        |
| H | %tr  | date_time of HTTP request                     | date        |
| H | %trg | gmt_date_time of start of HTTP request        | date        |
| H | %trl | local_date_time of start of HTTP request      | date        |
|   | %ts  | termination_state                             | string      |
| H | %tsc | termination_state with cookie status          | string      |
+---+------+-----------------------------------------------+-------------+
R = Restrictions : H = mode http only ; S = SSL only


Our custom log-format built in order to fulfill our needs is as this:

log-format %ci:%cp\ %H\ [%t]\ [%f\ %fi:%fp]\ [%b/%s\ %si:%sp]\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%sq/%bq


Once you place the log-format as a default for all haproxy frontend / backends or for a custom defined ones, the output you will get when tailing the log is:

# tail -f /var/log/haproxy.log

Apr  5 21:47:19  10.42.73.83:23262 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:23.879] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME/bk_appserv3 10.75.226.88:61310] 1/0/55250 55 sD 4/2/1/0/0/0
Apr  5 21:48:14  10.42.73.83:57506 haproxy-fqdn-hostname.com [05/Apr/2024:21:47:18.925] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv1 10.35.242.134:61310] 1/0/55236 55 sD 4/2/1/0/0/0
Apr  5 21:49:09  10.42.73.83:46520 haproxy-fqdn-hostname.com [05/Apr/2024:21:48:13.956] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv2 10.75.226.89:61310] 1/0/55209 55 sD 4/2/1/0/0/0


If you don't care about extra space and logs being filled with more naming, another variant of above log-format, that makes it even more readable even for most novice sys admin or programmer would look like this:

log-format [%t]\ %H\ [IN_IP]\ %ci:%cp\ [FT_NAME]\ %f:%fp\ [FT_IP]\ %fi:%fp\ [BK_NAME]\ [%b/%s:%sp]\ [BK_IP]\ %si:%sp\ [TIME_WAIT]\ {%Tw/%Tc/%Tt}\ [CONN_STATE]\ {%B\ %ts}\ [STATUS]\ [%ac/%fc/%bc/%sc/%sq/%bq]

Once you apply the config test the haproxy.cfg to make sure no syntax errors during copy / paste from this page

haproxy-serv:~# haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid


Next restart graceously haproxy 

haproxy-serv:~# /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)


Once you reload haproxy graceously without loosing the established connections in stead of restarting it completely via systemd sysctl restart haproxy:

 

2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.200.198.195:50714 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.012] [FrotnendProd 10.55.0.20:27800] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/1/0/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.189:54290 haproxy-fqdn-hostname.com
[05/Apr/2024:21:46:03.056] [FrotnendProd 10.55.0.20:27900] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/4/3/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.190:26778 haproxy-fqdn-hostname.com
[05/Apr/2024:21:46:03.134] [FrotnendProd 10.55.0.20:27900] [BackendProd/tsefas02s 10.35.242.134:27900] 1/-1/0 0 CC 4/4/3/0/0/0

Note that in that log localhost haproxy[pid] is written by rsyslog, you can filter it out by modifying rsyslogd configurations

The only problem with this log-format is not everyone wants to have to much repeating information pointer on which field is what, but I personally liked this one as well because using it even though occuping much more space, makes the log much easier to process with perl or python scripting for data visualize and very for programs that does data or even "big data" analysis.

DNS Monitoring: Check and Alert if DNS nameserver resolver of Linux machine is not properly resolving shell script. Monitor if /etc/resolv.conf DNS runs Okay

Thursday, March 14th, 2024

linux-monitor-check-dns-is-resolving-fine

If you happen to have issues occasionally with DNS resolvers and you want to keep up an eye on it and alert if DNS is not properly resolving Domains, because sometimes you seem to have issues due to network disconnects, disturbances (modifications), whatever and you want to have another mean to see whether a DNS was reachable or unreachable for a time, here is a little bash shell script that does the "trick".

Script work mechacnism is pretty straight forward as you can see we check what are the configured nameservers if they properly resolve and if they're properly resolving we write to log everything is okay, otherwise we write to the log DNS is not properly resolvable and send an ALERT email to preconfigured Email address.

Below is the check_dns_resolver.sh script:

 

#!/bin/bash
# Simple script to Monitor DNS set resolvers hosts for availability and trigger alarm  via preset email if any of the nameservers on the host cannot resolve
# Use a configured RESOLVE_HOST to try to resolve it via available configured nameservers in /etc/resolv.conf
# if machines are not reachable send notification email to a preconfigured email
# script returns OK 1 if working correctly or 0 if there is issue with resolving $RESOLVE_HOST on $SELF_HOSTNAME and mail on $ALERT_EMAIL
# output of script is to be kept inside DNS_status.log

ALERT_EMAIL='your.email.address@email-fqdn.com';
log=/var/log/dns_status.log;
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2));  

SELF_HOSTNAME=$(hostname –fqdn);
RESOLVE_HOST=$(hostname –fqdn);

for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $RESOLVE_HOST  $i); 

if [[ “$?” == ‘0’ ]]; then echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i on host $SELF_HOST OK 1" | tee -a $log; 
else 
echo "$(date "+%y.%m.%d %T")$RESOLVE_HOST $i on host $SELF_HOST NOT_OK 0" | tee -a $log; 

echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i DNS on host $SELF_HOST resolve ERROR" | mail -s "$RESOLVE_HOST /etc/resolv.conf $i DNS on host $SELF_HOST resolve ERROR";

fi

 done

Download check_dns_resolver.sh here set the script to run via a cron job every lets say 5 minutes, for example you can set a cronjob like this:
 

# crontab -u root -e
*/5 * * * *  check_dns_resolver.sh 2>&1 >/dev/null

 

Then Voila, check the log /var/log/dns_status.log if you happen to run inside a service downtime and check its output with the rest of infrastructure componets, network switch equipment, other connected services etc, that should keep you in-line to proof during eventual RCA (Root Cause Analysis) if complete high availability system gets down to proof your managed Linux servers was not the reason for the occuring service unavailability.

A simplified variant of the check_dns_resolver.sh can be easily integrated to do Monitoring with Zabbix userparameter script and DNS Check Template containing few Triggers, Items and Action if I have time some time in the future perhaps, I'll blog a short article on how to configure such DNS zabbix monitoring, the script zabbix variant of the DNS monitor script is like this:

[root@linux-server bin]# cat check_dns_resolver.sh 
#!/bin/bash
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2));  for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $(hostname –fqdn) $i); if [[ “$?” == ‘0’ ]]; then echo "$i OK 1"; else echo "$i NOT OK 0"; fi; done

[root@linux-server bin]#


Hope this article, will help someone to improve his Unix server Infrastucture monitoring.

Enjoy and Cheers !

How to count number of ESTABLISHED state TCP connections to a Windows server

Wednesday, March 13th, 2024

count-netstat-established-connections-on-windows-server-howto-windows-logo-debug-network-issues-windows

Even if you have the background of a Linux system administrator, sooner or later you will have have to deal with some Windows hosts, thus i'll blog in this article shortly on how the established TCP if it happens you will have to administarte a Windows hosts or help a windows sysadmin noobie 🙂

In Linux it is pretty easy to check the number of established conenctions, because of the wonderful command wc (word count). with a simple command like:
 

$ netstat -etna |wc -l


Then you will get the number of active TCP connections to the machine and based on that you can get an idea on how busy the server is.

But what if you have to deal with lets say a Microsoft Windows 2012 /2019 / 2020 or 2022 Server, assuming you logged in as Administrator and you see the machine is quite loaded and runs multiple Native Windows Administrator common services such as IIS / Active directory Failover Clustering, Proxy server etc.
How can you identify the established number of connections via a simple command in cmd.exe?

1.Count ESTABLISHED TCP connections from Windows Command Line

Here is the answer, simply use netstat native windows command and combine it with find, like that and use the /i (ignores the case of characters when searching the string) /c (count lines containing the string) options

C:\Windows\system32>netstat -p TCP -n|  find /i "ESTABLISHED" /c
1268

Voila, here are number of established connections, only 1268 that is relatively low.
However if you manage Windows servers, and you get some kind of hang ups as part of the monitoring, it is a good idea to setup a script based on this simple command for at least Windows Task Scheduler (the equivallent of Linux's crond service) to log for Peaks in Established connections to see whether Server crashes are not related to High Rise in established connections.
Even better if company uses Zabbix / Nagios, OpenNMS or other  old legacy monitoring stuff like Joschyd even as of today 2024 used in some big of the TOP IT companies such as SAP (they were still using it about 4 years ago for their SAP HANA Cloud), you can set the script to run and do a Monitoring template or Alerting rules to draw you graphs and Trigger Alerts if your connections hits a peak, then you at least might know your Windows server is under a "Hackers" Denial of Service attack or there is something happening on the network, like Cisco Network Infrastructure Switch flappings or whatever.

Perhaps an example script you can use if you decide to implement the little nestat established connection checks Monitoring in Zabbix is the one i've writen about in the previous article "Calculate established connection from IP address with shell script and log to zabbix graphic".

2. Few Useful netstat options for the Windows system admin
 

C:\Windows\System32> netstat -bona


netstat-useful-arguments-for-the-windows-system-administrator

Cmd.exe will lists executable files, local and external IP addresses and ports, and the state in list form. You immediately see which programs have created connections or are listening so that you can find offenders quickly.

b – displays the executable involved in  creating the connection.
o – displays the owning process ID.
n – displays address and port numbers.
a – displays all connections and listening ports.

As you can see in the screenshot, by using netstat -bona you get which process has binded to which local address and the Process ID PID of it, that is pretty useful in debugging stuff.

3. Use a Third Party GUI tool to debug more interactively connection issues

If you need to keep an eye in interactive mode, sometimes if there are issues CurrPorts tool can be of a great help

currports-windows-network-connections-diagnosis-cports

CurrPorts Tool own Description

CurrPorts is network monitoring software that displays the list of all currently opened TCP/IP and UDP ports on your local computer. For each port in the list, information about the process that opened the port is also displayed, including the process name, full path of the process, version information of the process (product name, file description, and so on), the time that the process was created, and the user that created it.
In addition, CurrPorts allows you to close unwanted TCP connections, kill the process that opened the ports, and save the TCP/UDP ports information to HTML file , XML file, or to tab-delimited text file.
CurrPorts also automatically mark with pink color suspicious TCP/UDP ports owned by unidentified applications (Applications without version information and icons).

Sum it up

What we learned is how to calculate number of established TCP connections from command line, useful for scripting, how you can use netstat to display the process ID and Process name that relates to a used Local / Remote TCP connections, and how eventually you can use this to connect it to some monitoring tool to periodically report High Peaks with TCP established connections (usually an indicator of servere system issues).