Generate and Add UUID for every existing Redhat / CentOS / RHEL network interface to configuration if missing howto


August 5th, 2023

linux-fix-missing-uid-on-redhat-centos-fedora-networking-logo

If you manage old Linux machines it might be after the update either due to update mess or because of old system administrators which manually included the UUID to the config forgot to include it in the present network configuration in /etc/sysconfig/networking-scripts/ifcfg-* Universally Unique IDentifier (UUID)128-bit label I used a small one liner after listing all the existing configured LAN interfaces reported from iproute2 network stack with ip command. As this might be useful to someone out there here is the simple command that returns a number of commands to later just copy paste to console once verified there are no duplicates of the UUID already in the present server configuration with grep.

In overall to correct the configs and reload the network with the proper UUIDs here is what I had to do:


# grep -rli UUID /etc/sysconfig/network-scripts/ifcfg-*

No output from the recursive grep means UUIDs are not present on any existing interface, so we can step further check all the existing machines network ifaces and generate the missing UUIDs with uuidgen command

# ip a s |grep -Ei ': <'|sed -e 's#:##g' |grep -v '\.' |awk '{ print $2 }'
ifcfg-venet0
ifcfg-eth0
ifcfg-eth1

ifcfg-eth2
ifcfg-eth3

I've stumbled on that case on some legacy Linux inherited from other people sysadmins and in order to place the correct 

# for i in $(ip a s |grep -Ei ': <'|sed -e 's#:##g' |grep -v '\.' |awk '{ print $2 }'); do echo "echo UUID=$(uuidgen $i)"" >> ifcfg-$i"; done|grep -v '\-lo' 
echo UUID=26819d24-9452-4431-a9ca-176d87492b75 >> ifcfg-venet0
echo UUID=3c7e8848-0232-436f-a52a-46db9a03eb33 >> ifcfg-eth0
echo UUID=1fc0454d-bf23-417d-b960-571fc04754d2 >> ifcfg-eth1
echo UUID=5793c1e5-4481-4f09-967e-2cceda85c35f >> ifcfg-eth2
echo UUID=65fdcaf6-d271-4845-a8f1-0ec478c375d1 >> ifcfg-eth3


As you can see I exclude the loopback interface -lo from the ouput as it is not necessery to have UUID for it.
That's all folks problem solved. Enjoy

How to monitor Postfix Mail server work correct with simple one liner Zabbix user parameter script / Simple way to capture and report SMTP machine issues Zabbix template


June 22nd, 2023

setup-zabbix-smtp-mail-monitoring-postfix-qmail-exim-with-easy-userparameter-script-and-template-zabbix-logo

In this article, I'm going to show you how to setup a very simple monitoring if a local running SMTP (Postfix / Qmail / Exim) is responding correctly on basic commands. The check would helpfully keep you in track to know whether your configured Linux server local MTA (Mail Transport Agent) is responding on requests on TCP / IP protocol Port 25, as well as a check for process existence of master (that is the main postfix) proccess, as well as the usual postfix spawned sub-processes qmgr (the postfix queue manager), tsl mgr (TLS session cache and PRNG manager), pickup (Postfix local mail pickup) – or email receiving process.

 

Normally a properly configured postfix installation on a Linux whatever you like distribution would look something like below:

#  ps -ef|grep -Ei 'master|postfix'|grep -v grep
root        1959       1  0 Jun21 ?        00:00:00 /usr/libexec/postfix/master -w
postfix     1961    1959  0 Jun21 ?        00:00:00 qmgr -l -t unix -u
postfix     4542    1959  0 Jun21 ?        00:00:00 tlsmgr -l -t unix -u
postfix  2910288    1959  0 11:28 ?        00:00:00 pickup -l -t unix -u

At times, during mail server restarts the amount of processes that are sub spawned by postfix, may very and if you a do a postfix restart

# systemctl restart postfix

The amout of spawned processes running as postfix username might decrease, and only qmgr might be available for second thus in the consequential shown Template the zabbix processes check to make sure the Postfix is properly operational on the Linux machine is made to check for the absolute minumum of 

1. master (postfix process) that runs with uid root
2. and one (postfix) username binded proccess 

If the amount of processes on the host is less than this minimum number and the netcat is unable to simulate a "half-mail" sent, the configured Postfix alarm Action (media and Email) will take place, and you will get immediately notified, that the monitored Mail server has issue!

The idea is to use a small one liner connection with netcat and half simulate a normal SMTP transaction just like you would normally do:

 

root@pcfrxen:/root # telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 This is Mail2 Pc-Freak.NET ESMTP
HELO localhost
250 This is Mail2 Pc-Freak.NET
MAIL FROM:<hipopo@pc-freak.net>
250 ok
RCPT TO:<hip0d@remote-smtp-server.com>

 

and then disconnect the connection.

1. Create new zabbix userparameter_smtp_check.conf file

The simple userparameter one liner script to do the task looks like this:

# vi /etc/zabbix/zabbix_agent.d/userparameter_smtp_check.conf

UserParameter=smtp.check,(if [[ $(echo -e “HELO localhost\n MAIL FROM: root@$HOSTNAME\n RCPT TO: report-email@your-desired-mail-server.com\n  QUIT\n” | /usr/bin/nc localhost 25 -w 5 2>&1 | grep -Ei ‘220\s.*\sESMTP\sPostfix|250\s\.*|250\s\.*\sOk|250\s\.*\sOk|221\.*\s\w’|wc -l) == ‘5’ ]]; then echo "SMTP OK 1"; else echo "SMTP NOK 0"; fi)

Set the proper permissions so either file is owned by zabbix:zabbix or it is been able to be read from all system users.
 

# chmod a+r /etc/zabbix/zabbix_agent.d/userparameter_smtp_check.conf

2. Create a new Template for the Mail server monitoring
 


 

Just like any other template name it with what fits you as you see, I've call it PROD SMTP Monitoring, as the template is prepared to specifically monitor In Production Linux machines, and a separate template is used to monitor the Quality Assurance (QAs) as well as PreProd (Pre Productions).

3. Create the followng Items and Depedent Item to process zabbix-agent received data from the Userparam script
 

Above is the list of basic Items and Dependent Item you will need to configure inside the SMTP Check zabbix Template.

The Items should have the following content and configurations:
 

/postfix-main-proc-service-item-zabbix-shot


*Name: postfix_main_proc.service
Type: Zabbix agent(active)
*Key: proc.num[master,root]
Type of Information: Numeric (unassigned)
*Update interval: 30s
Custom Intervals: Flexible
*History storage period: 90d
*Trend storage period: 365d
Show Value: as is
Applications: Postfix Checks
Populated host inventory field: -None-
Description: The item counts master daemon process that runs Postfix daemons on demand

Where the arguments pased to proc.num[] function are:
  master is the process that is being looked up for and root is the username with which the the postfix master daemon is running. If you need to adapt it for qmail or exim that shouldn't be a big deal you only have to in advance check the exact processes that are normally running on the machine
and configure a similar process check for it.

*Name: postfix_sub_procs.service_cnt
Type: Zabbix agent(active)
*Key: proc.num[,postfix]
Type of information: Numeric (unassigned)
Update Interval: 30s
*History Storage period: Storage Period 90d
*Trend storage period: Storage Period 365d
Description: The item counts master daemon processes that runs postfix daemons on demand.

Here the idea with this Item is to check the number of processes that are running with user / groupid that is postfix. Again for other SMPT different from postfix, just set it to whatever user / group 
you would like zabbix to look up for in Linux the process list. As you can see here the check for existing postfix mta process is done every 30 seconds (for more critical environments you can put it to less).

For simple zabbix use this Dependent Item is not necessery required. But as we would like to process more closely the output of the userparameter smtp script, you have to set it up.
If you want to write graphical representation by sending data to Grafana.

*Name: postfix availability check
Key: postfix_boolean_check[boolean]
Master Item: PROD SMTP Monitoring: postfix availability check
Type of Information: Numeric unassigned
*History storage period: Storage period 90d
*Trend storage period: 365d

Applications: Postfix Checks

Description: It returns boolean value of SMTP check
1 – True (SMTP is OK)
0 – False (SMTP does not responds)

Enabled: Tick

*Name: postfix availability check
*Key: smtp.check
Custom intervals: Flexible
*Update interval: 30 m
History sotrage period: Storage Period 90d
Applications: Postfix Checks
Populates host inventory field: -None-
Description: This check is testing if the SMTP relay is reachable, without actual sending an email
Enabled: Tick

4. Configure following Zabbix Triggers

 

Note: The severity levels you should have previosly set in Zabbix up to your desired ones.

Name: postfix master root process is not running
*Problem Expression: {PROD SMTP Monitoring:proc.num[master,root].last()}<1

OK event generation: Recovery expression
*Recovery Expression: {PROD SMTP Monitoring:proc.num[master,root].last()}>=1
Allow manual close: Tick

Description: The item counts master daemon process that runs Postfix daemon on demand.
Enabed: Tick

I would like to have an AUTO RESOLVE for any detected mail issues, if an issue gets resolved. That is useful especially if you don't have the time to put the Zabbix monitoring in Maintainance Mode during Operating system planned updates / system reboots or unexpected system reboots due to electricity power loss to the server colocated – Data Center / Rack . 


*Name: postfix master sub processes are not running
*Problem Expression: {P09 PROD SMTP Monitoring:proc.num[,postfix].last()}<1
PROBLEM event generation mode: Single
OK event closes: All problems

*Recovery Expression: {P09 PROD SMTP Monitoring:proc.num[,postfix].last()}>=1
Problem event generation mode: Single
OK event closes: All problems
Allow manual close: Tick
Enabled: Tick

Name: SMTP connectivity check
Severity: WARNING
*Expression: {PROD SMTP Monitoring:postfix_boolen_check[boolean].last()}=0
OK event generation: Expression
PROBLEM even generation mode: SIngle
OK event closes: All problems

Allow manual close: Tick
Enabled: Tick

5. Configure respective Zabbix Action

 

zabbix-configure-Actions-screenshotpng
 

As the service is tagged with 'pci service' tag we define the respective conditions and according to your preferences, add as many conditions as you need for the Zabbix Action to take place.

NOTE! :
Assuming that communication chain beween Zabbix Server -> Zabbix Proxy (if zabbix proxy is used) -> Zabbix Agent works correctly you should start receiving that from the userparameter script in Zabbix with the configured smtp.check userparam key every 30 minutes.

Note that this simple nc check will keep a trail records inside your /var/log/maillog for each netcat connection, so keep in mind that in /var/log/maillog on each host which has configured the SMTP Check zabbix template, you will have some records  similar to:

# tail -n 50 /var/log/maillog
2023-06-22T09:32:18.164128+02:00 lpgblu01f postfix/smtpd[2690485]: improper command pipelining after HELO from localhost[127.0.0.1]:  MAIL FROM: root@your-machine-fqdn-address.com\n RCPT TO: your-supposable-receive-addr@whatever-mail-address.com\n  QUIT\n
2023-06-22T09:32:18.208888+02:00 lpgblu01f postfix/smtpd[2690485]: 32EB02005B: client=localhost[127.0.0.1]
2023-06-22T09:32:18.209142+02:00 lpgblu01f postfix/smtpd[2690485]: disconnect from localhost[127.0.0.1] helo=1 mail=1 rcpt=1 quit=1 commands=4
2023-06-22T10:02:18.889440+02:00 lpgblu01f postfix/smtpd[2747269]: connect from localhost[127.0.0.1]
2023-06-22T10:02:18.889553+02:00 lpgblu01f postfix/smtpd[2747269]: improper command pipelining after HELO from localhost[127.0.0.1]:  MAIL FROM: root@your-machine-fqdn-address.com\n RCPT TO: your-supposable-receive-addr@whatever-mail-address.com\n  QUIT\n
2023-06-22T10:02:18.933933+02:00 lpgblu01f postfix/smtpd[2747269]: E3ED42005B: client=localhost[127.0.0.1]
2023-06-22T10:02:18.934227+02:00 lpgblu01f postfix/smtpd[2747269]: disconnect from localhost[127.0.0.1] helo=1 mail=1 rcpt=1 quit=1 commands=4
2023-06-22T10:32:26.143282+02:00 lpgblu01f postfix/smtpd[2804195]: connect from localhost[127.0.0.1]
2023-06-22T10:32:26.143439+02:00 lpgblu01f postfix/smtpd[2804195]: improper command pipelining after HELO from localhost[127.0.0.1]:  MAIL FROM: root@your-machine-fqdn-address.com\n RCPT TO: your-supposable-receive-addr@whatever-mail-address.com\n  QUIT\n
2023-06-22T10:32:26.186681+02:00 lpgblu01f postfix/smtpd[2804195]: 2D7F72005B: client=localhost[127.0.0.1]
2023-06-22T10:32:26.186958+02:00 lpgblu01f postfix/smtpd[2804195]: disconnect from localhost[127.0.0.1] helo=1 mail=1 rcpt=1 quit=1 commands=4
2023-06-22T11:02:26.924039+02:00 lpgblu01f postfix/smtpd[2860398]: connect from localhost[127.0.0.1]
2023-06-22T11:02:26.924160+02:00 lpgblu01f postfix/smtpd[2860398]: improper command pipelining after HELO from localhost[127.0.0.1]:  MAIL FROM: root@your-machine-fqdn-address.com\n RCPT TO: your-supposable-receive-addr@whatever-mail-address.com\n  QUIT\n
2023-06-22T11:02:26.963014+02:00 lpgblu01f postfix/smtpd[2860398]: EB08C2005B: client=localhost[127.0.0.1]
2023-06-22T11:02:26.963257+02:00 lpgblu01f postfix/smtpd[2860398]: disconnect from localhost[127.0.0.1] helo=1 mail=1 rcpt=1 quit=1 commands=4
2023-06-22T11:32:29.145553+02:00 lpgblu01f postfix/smtpd[2916905]: connect from localhost[127.0.0.1]
2023-06-22T11:32:29.145664+02:00 lpgblu01f postfix/smtpd[2916905]: improper command pipelining after HELO from localhost[127.0.0.1]:  MAIL FROM: root@your-machine-fqdn-address.com\n RCPT TO: your-supposable-receive-addr@whatever-mail-address.com\n  QUIT\n
2023-06-22T11:32:29.184539+02:00 lpgblu01f postfix/smtpd[2916905]: 2CF7D2005B: client=localhost[127.0.0.1]
2023-06-22T11:32:29.184729+02:00 lpgblu01f postfix/smtpd[2916905]: disconnect from localhost[127.0.0.1] helo=1 mail=1 rcpt=1 quit=1 commands=4

 

 

That's all folks use the :
Configuration -> Host (menu)

and assign the new SMTP check template to as many of the Linux hosts where you have setup the Userparameter script and Enjoy the new mail server monitoring at hand.

Unfreeze stucked “freezed” messages from Exim Mail server queue


June 16th, 2023

how-to-unfreeze-messages-from-exim-mail-server-smtp-logo-linux-and-some-other-useful-exim-commands.png

Messages are frozen when the mail server has determined it cannot do anything to deliver the message. (they can also be manually frozen).

Exim has option to set how long frozen messages are kept on the system.

On a Debian/Ubuntu based install the /etc/exim4/conf.d/main/02_exim4-config_options file has the option timeout_frozen_after = 7d. Which means messages frozen for 7 days will get expunged.

Frozen messages really shouldn't be a problem on most systems. They are often just spam messages that can't get properly delivered.

If you have to deal with freezed mails from the exim mail server, unlike postfix, where there is no "freeze" scheme but the messages just stuck in the queue and you might want to simply ask the mail server to resend failed to deliver messages once again through a simple:

# postqueue -f


With exim to ask the server to resend the freeze-d states messages there is another aproach:

It is for this reason that I am writing this post to share how you can ask the exim to resend the "frozen" messages, as on exim there is no so much straight forward way.

To find out what letters are stored in the exim queue run

# exim -bp

To unfreeze the messages a simple while loop can be written, which lists all frozen state messages and unfreezes these letters one by one in a cycle:

# exim -bp | grep -i frozen | awk '{print $3}' | while read LINE; do exim -Mt $LINE; done

Another approach to unfreeze the frozen multitude of messages which should be a bit quicker if you have to do it for a very large amount of frozen states mails is to use xargs command:

# mailq | grep frozen | awk '{print $3}' | xargs exim -v -M

Since we on exim topic in this article, for starters with Exim, here is few other useful exim queue commands, that might be beneficial if you have to deal with EXIM SMTP.

Attempting to send a mail with a specified ID

# /usr/sbin/exim -M email-id


Forcefully run another queue to execute

# /usr/sbin/exim -qf

We see the logs related to letter

# /usr/sbin/exim -Mvl messageID

To see the body of the letter

# /usr/sbin/exim -Mvb messageID

To see the beginning (header) of the letter only

# /usr/sbin/exim -Mvh messageID

Deletes the mail without sending any error messages

# /usr/sbin/exim -Mrm messageID

Shows the number of letters in the queue

# /usr/sbin/exim -bpr | grep "<" | wc -l

Shows the number of frozen mails in the queue

# /usr/sbin/exim -bpr | grep frozen | wc -l

Deletes all frozen letters

# /usr/sbin/exim -bpr | grep frozen | awk {'print $3'} | xargs exim -Mrm

To remove a message from the Exim queue

# exim -Mrm {message-id}

Remove all messages from the Exim queue

# exim -bp | awk '/^ *[0-9]+[mhd]/{print "exim -Mrm " $3}' | bash

Another way to do it:

# exim -bp | exiqgrep -i | xargs exim -Mrm


Fastest solution to delete all emails in exim queue (for less than 5 seconds) is
 

# cd /var/spool
# mv exim exim.old
# mkdir -p exim/input
# mkdir -p exim/msglog
# mkdir -p exim/db
# chown -R mail:mail exim
# /sbin/service exim restart

or if you have AV / AntiSpam integrated to mail server:

# cd /var/spool
# mv exim exim.old
# mkdir -p exim/db
# mkdir -p exim/input
# mkdir -p exim/msglog
# mkdir -p exim/scan
# chown -R mail:mail exim
# /sbin/service exim restart

Deletes the entire exim queue

# /usr/sbin/exim -qff

Install specific zabbix-agent version / Downgrade Zabbix Agent client to exact preferred old RPM version on CentOS / Fedora / RHEL Linux from repo


June 7th, 2023

zabbix-update-downgrade-on-centos-rhel-fedora-and-other-rpm-based-linux-zabbix-logo

 

In below article, I'll give you the short Update zabbix procedure to specific version release, if you need to have it running in tandem with rest of zabbix infra, as well as expain shortly how to downgrade zabbix version to a specific release number
to match your central zabbix-serveror central zabbix proxies.

The article is based on personal experience how to install / downgrade the specific zabbuix-agent  release on RPM based distros.
I know this is pretty trivial stuff but still, hope this might be useful to some sysadmin out there thus I decided to quickly blog it.

 

1. Prepare backup of zabbix_agentd.conf
 

cp -rpf /etc/zabbix/zabbix_agentd.conf /home/your-user/zabbix_agentd.conf.bak.$(date +"%b-%d-%Y")

 

2. Create zabbix repo source file in yum.repos.d directory

cd /etc/yum.repos.d/
vim zabbix.repo 

 

[zabbix-5.0]

name=Zabbix 5.0 repo

baseurl=http://zabixx-rpm-mirrors-site.com/centos/external/zabbix-5.0/8/x86_64/

enabled=1

gpgcheck=0

 

3. Update zabbix-agent to a specific defined version

yum search zabbix-agent –enablerepo zabbix-5.0

To update zabbix-agent for RHEL 7.*

# yum install zabbix-agent-5.0.34-1.el7.x86_64


For RHEL 8.*

# yum install zabbix-agent-5.0.34-1.el8.x86_64


4. Restart zabbix-agentd and check its status to make sure it works correctly
 

systemctl status zabbix-agentd
systemctl restart zabbix-agentd
# systemctl status zabbix-agentd


Go to zabbix-server WEB GUI interface and check that data is delivered as normally in Latest Data for the host fom recent time, to make sure host monitoring is continuing flawlessly as before change.

NB !: If yum use something like versionlock is enabled remove the versionlock for package and update then, otherwise it will (weirldly look) look like the package is missing.
I'm saying that because I've hit this issue and was wondering why i cannot install the zabbix-agent even though the version is listed, available and downloadable from the repository.


5. Downgrade agent-client to specific version (Install old version of Zabbix from Repo)
 

Sometimes by mistake you might have raised the Zabbix-agent version to be higher release than the zabbix-server's version and thus breach out the Zabbix documentation official recommendation to keep
up the zabbix-proxy, zabbix-server and zabbix-agent at the exactly same version major and minor version releases. 

If so, then you would want to decrease / downgrade the version, to match your Zabbix overall infrastructure exact version for each of Zabbix server -> Zabbix Proxy server -> Agent clients.

To downgrade the version, I prefer to create some backups, just in case for all /etc/zabbix/ configurations and userparameter scripts (from experience this is useful as sometimes some RPM binary update packages might cause /etc/zabbix/zabbix_agentd.conf file to get overwritten. To prevent from restoring zabbix_agentd.conf from your most recent backup hence, I prefer to just crease the zabbix config backups manually.
 

# cd /root

# mkdir -p /root/backup/zabbix-agent 

# tar -czvf zabbix_agent.tar.gz /etc/zabbix/

# tar -xzvf zabbix_agent.tar.gz 


Then list the available installable zabbix-agent versions
 

[root@sysadminshelp:~]# yum –showduplicates list zabbix-agent
Заредени плъгини: fastestmirror
Determining fastest mirrors
 * base: centos.uni-sofia.bg
 * epel: fedora.ipacct.com
 * extras: centos.uni-sofia.bg
 * remi: mirrors.uni-ruse.bg
 * remi-php74: mirrors.uni-ruse.bg
 * remi-safe: mirrors.uni-ruse.bg
 * updates: centos.uni-sofia.bg
Инсталирани пакети
zabbix-agent.x86_64                                                     5.0.30-1.el7                                                     @zabbix
Налични пакети
zabbix-agent.x86_64                                                     5.0.0-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.1-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.2-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.3-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.4-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.5-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.6-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.7-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.8-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.9-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.10-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.11-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.12-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.13-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.14-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.15-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.16-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.17-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.18-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.19-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.20-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.21-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.22-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.23-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.24-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.25-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.26-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.27-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.28-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.29-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.30-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.31-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.32-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.33-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.34-1.el7                                                     zabbix

 

Next lets install the most recent zabbix-versoin from the CentOS repo, which for me as of time of writting this article is 5.0.34.

# yum downgrade -y zabbix-agent-5.0.34-1.el7

# cp -rpf /root/backup/zabbix-agent/etc/zabbix/zabbix_agentd.conf /etc/zabbix/

# systemctl start zabbix-agent.service

# systemctl enable  zabbix-agent.service
 

# zabbix_agentd -V
zabbix_agentd (daemon) (Zabbix) 5.0.30
Revision 2c96c38fb4b 28 November 2022, compilation time: Nov 28 2022 11:27:43

Copyright (C) 2022 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 1.0.1e-fips 11 Feb 2013
Running with OpenSSL 1.0.1e-fips 11 Feb 2013

 

That's all folks you should be at your custom selected preferred version of zabbix-agent.
Enjoy ! 🙂

IPFilter firewall basics use for Adding / Removing and Cloning firewall rules


June 1st, 2023

ipfilter-bsd-solaris-unix-firewall-short-review-ofLinux_firewalls-BSD_Firewall_logo

Linux users have most definitely used Netfilter (the older from us might remember and have used ipchains) and rest
should know well or at least partially tried iptables or if you have digged into Linux firewalls more professionally, might have tried nftables
and the newer firewalld (firewall-cmd) that is the standard nowadays in CentOS / Fedora and RHEL (again an abstraction over iptables.).
On Debian firewall is organized around custom shell scripts that deal with iptables chains, or if on Ubuntu perhaps you have tried UFW (The Uncomplicated Firewall)
frontend program for managing firewalls again with iptables. For the lazy ones UFW even has another GUI frontend called Gufw (intended to be easy, intuitive,
graphical user interface for managing Uncomlicated firewall.

Different Linux distributions do use a different set of firewall mechanisms preconfigure but there are other firewall solutions on other Unixes such as ipfilter.
That historically were heavily used that is worthy mentioning and if you happen to pop-up working as a network guy inside some large corporations you might face it.

IPFilter (commonly referred to as ipf) is an open-source software package that provides firewall services and network address translation (NAT) for many Unix-like operating systems.
The author and software maintainer is Darren Reed. IPFilter supports both IPv4 and IPv6 protocols, and is a stateful firewall.
IPFilter is delivered with FreeBSD, NetBSD, Solaris 10 & 11, illumos, OpenIndiana and HP-UX.
It used to be a part of OpenBSD, but it was removed by Theo de Raadt in May 2001 due to problems with its license.
It was subsequently replaced in OpenBSD by PF, which was developed by OpenBSD's own developers.
DragonFly BSD removed its support for IPFilter in May 2011.

IPFilter can be installed as a runtime-loadable kernel module or directly incorporated into the operating system kernel, depending on the specifics of each kernel and user preferences.
The software's documentation recommends the module approach, if possible.

Here are some commands for displaying, changing and distributing IP filters with ipfilter.
It will be mostly useful, if you happen to have some obsolete OS infrastructure or OpenBSD.

The commands given below are to add / remove and activate rules on machine with ipfilter:

# ipfilter –clone
# ipfilter –save
# ipfilter –activate
# ipfilter -addrule
# ipfilter -delrule
# help ipfilter

1. Check ipfilter current config

# ipfilter –show
Name: default_ipv4, Type: ipv4, State: active
Rule    Source IP                               Protocol   Dest Port   Action
1     any                                            tcp       22     permit
2     any                                            tcp       23     permit
3     any                                            tcp       80     permit
4     any                                            tcp      443     permit
5     any                                            udp      161     permit
6     any                                            udp      123     permit
7     any                                            tcp      600 – 1023     permit
8     any                                            udp      600 – 1023     permit
Name: default_ipv6, Type: ipv6, State: active
Rule    Source IP                               Protocol   Dest Port   Action
1     any                                            tcp       22     permit
2     any                                            tcp       23     permit
3     any                                            tcp       80     permit
4     any                                            tcp      443     permit
5     any                                            udp      161     permit
6     any                                            udp      123     permit
7     any                                            tcp      600 – 1023     permit
8     any                                            udp      600 – 1023     permit
Name: default_ipv4_new, Type: ipv4, State: defined
Rule    Source IP                               Protocol   Dest Port   Action
1     any                                            tcp       22     permit
2     any                                            tcp       23     permit
3     any                                            tcp       80     permit
4     any                                            tcp      443     permit
5     any                                            udp      161     permit
6     any                                            udp      123     permit
7     any                                            tcp      600 – 1023     permit
8     any                                            udp      600 – 1023     permit

2. Clone and activate ipfilter configuration

# ipfilter –clone default_ipv4_new -from default_ipv4
# ipfilter –activate default_ipv4_new
# ipfilter –show
Name: default_ipv4, Type: ipv4, State: defined
Rule    Source IP                               Protocol   Dest Port   Action
1     any                                            tcp       22     permit
2     any                                            tcp       23     permit
3     any                                            tcp       80     permit
4     any                                            tcp      443     permit
5     any                                            udp      161     permit
6     any                                            udp      123     permit
7     any                                            tcp      600 – 1023     permit
8     any                                            udp      600 – 1023     permit
Name: default_ipv6, Type: ipv6, State: active
Rule    Source IP                               Protocol   Dest Port   Action
1     any                                            tcp       22     permit
2     any                                            tcp       23     permit
3     any                                            tcp       80     permit
4     any                                            tcp      443     permit
5     any                                            udp      161     permit
6     any                                            udp      123     permit
7     any                                            tcp      600 – 1023     permit
8     any                                            udp      600 – 1023     permit
Name: default_ipv4_neu, Type: ipv4, State: active
Rule    Source IP                               Protocol   Dest Port   Action
1     any                                            tcp       22     permit
2     any                                            tcp       23     permit
3     any                                            tcp       80     permit
4     any                                            tcp      443     permit
5     any                                            udp      161     permit
6     any                                            udp      123     permit
7     any                                            tcp      600 – 1023     permit
8     any                                            udp      600 – 1023     permit

3. Modify cloned configuration

Lets say we would like to delete the telnet port accept traffic rule  (port 23)

# ipfilter –delrule default_ipv4_new -rule 2

To permit the rule agian

# ipfilter –addrule default_ipv4_new -rule 2 -sip any -dp 23 -proto tcp -act permit

To save the rule

# ipfilter –save default_ipv4_new                          

Install btop on Debian Linux, btop an advanced htop like monitoring for Linux to beautify your console life


May 30th, 2023

btop-linux-monitoring-tool-screenshot-help-menu

I've accidently stubmled on btop a colorful and interactive ncurses like command line utility to provide you a bunch of information about CPU / memory / disks and processes with nice console graphic in the style of Cubic Player 🙂
Those who love htop and like their consoles to be full of shiny colors, will really appreciate those nice Linux monitoring tool.
To install btop on latest current stable Debian bullseyes, you will have to install it via backports, as the regular Debian repositories does not have the tool available out of the box.

To Add backports packages support for your Debian 11:

1. Edit /etc/apt/sources.list and include following repositories

 

# vim /etc/apt/sources.list

deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free


2. Update the known repos list to include it

 

# apt update


3. Install the btop deb package from backports

 

# apt-cache show btop|grep -A 20 -i descrip
Description-en: Modern and colorful command line resource monitor that shows usage and stats
 btop is a modern and colorful command line resource monitor that shows
 usage and stats for processor, memory, disks, network and processes.
 btop features:
  – Easy to use, with a game inspired menu system.
  – Full mouse support, all buttons with a highlighted key is clickable
  and mouse scroll works in process list and menu boxes.
  – Fast and responsive UI with UP, DOWN keys process selection.
  – Function for showing detailed stats for selected process.
  – Ability to filter processes.
  – Easy switching between sorting options.
  – Tree view of processes.
  – Send any signal to selected process.
  – UI menu for changing all config file options.
  – Auto scaling graph for network usage.
  – Shows IO activity and speeds for disks
  – Battery meter
  – Selectable symbols for the graphs
  – Custom presets
  – And more…
  btop is written in C++ and is continuation of bashtop and bpytop.
Description-md5: 73df6c70fe01f5bf05cca0e3031c1fe2
Multi-Arch: foreign
Homepage: https://github.com/aristocratos/btop
Section: utils
Priority: optional
Filename: pool/main/b/btop/btop_1.2.7-1~bpo11+1_amd64.deb
Size: 431500
SHA256: d79e35c420a2ac5dd88ee96305e1ea7997166d365bd2f30e14ef57b556aecb36


 

# apt install -t bullsye-backports btop –yes

Once I installed it, I can straight use it except on some of my Linux machines, which were having a strange encoding $LANG defined, those ones spitted some errors like:

root@freak:~# btop
ERROR: No UTF-8 locale detected!
Use –utf-force argument to force start if you're sure your terminal can handle it.

 


To work around it simply redefine LANG variable and rerun it
 

# export LANG=en_US.UTF8

# btop

 

btop-linux-monitoring-console-beautiful-colorful-tool-graphics-screenshot

btop-linux-monitoring-tool-screenshot-help-menu

How to log multiple haproxy / apache / mysql instance via haproxy log-tagging / Segregating log management for multiple HAProxy instances using rsyslog


May 23rd, 2023

rsyslog-logo-picture-use-programname-and-haproxy-log-tag-directives-together-to-log-as-many-process-streams-as-you-like

 

Introduction

This article provides a guide on refining haproxy  logging mechanism by leveraging the `programname` property in rsyslog, coupled with the `log-tag` directive in haproxy.
This approach will create a granular logging setup, separating logs according to their originating services and specific custom tags, enhancing overall log readability.

Though the article is written concretely for logging multiple log streams from haproxy this can be successfully applied
for any other Linux service to log as many concrete log-tagged data streams as you prefer.

Scope

The guide focuses on tailoring the logging mechanisms for two haproxy  instances named `haproxy` and `haproxyssl`, utilizing the `programname` property in rsyslog and the `log-tag` directive in haproxy for precise log management.

The haproxy and haproxyssl instances are two separate systemd config file prepared instances.
haproxy instance is simple haproxy proxying tcp traffic in non-encrypted form, whether haproxyssl is a special instance
prepared to tunnel the incoming http traffic in ssl form. Both instances of haproxy runs as a separate processes on the server.

Here is the systemd configuration of haproxy systemd service file:

# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target


As well as the systemd service configuration for haproxyssl:
 

# cat /usr/lib/systemd/system/haproxyssl.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy_ssl_prod.cfg" "PIDFILE=/run/haproxy_ssl_prod.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxyssl -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target

 

Step 1: Configuring HAProxy instances with `log-tag`
 

To distinguish between logs from two HAProxy instances, `log-tag` directive is used to add tags to logs. This tag is used to filter these logs in rsyslog.
Modify the HAProxy configuration file in `/etc/haproxy/haproxy.*.cfg`

HAProxy Instance 1 (haproxy)
 

#———————————————————————
# Global settings
#———————————————————————
global
      log          127.0.0.1 local6 debug
      log-tag      haproxy

HAProxy Instance 2 (haproxyssl)


#———————————————————————
# Global settings
#———————————————————————
global
    log          127.0.0.1 local5 debug
    log-tag      haproxyssl

 

Step 2: Implementing rsyslog configuration for haproxy logs
 

Next, create a new rsyslog configuration file, stored in /etc/rsyslog.d/. Ensure the new configuration file ends in `.conf`

HAProxy Instance 1 (haproxy)

Now add rsyslog rules to filters logs based on the `programname` and the custom log tag:
 

# vi /etc/rsyslog.d/55_haproxy.conf
if $programname == 'haproxy' then /var/log/haproxy.log
&stop

HAProxy Instance 2 (haproxyssl)
# vi /etc/rsyslog.d/51_haproxy_ssl.conf
if $programname == 'haproxy_ssl' then /var/log/haproxy_ssl.log
&stop


These rules filter logs that originate from haproxy  and contain the respective string haproxy   or haproxy_ssl , directing them to their respective log files. The `& stop` directive ensures that rsyslog stops processing the log once a match is found, preventing dublication.

Finally, restart both the haproxy and rsyslog services for the changes to take effect:

# systemctl restart haproxy
# systemctl restart haproxyssl
# systemctl restart rsyslog


Reading References

haproxy:   log-tag directive

rsyslog:    rsyslogd documentation

This is a guest article originally written by: Dimitar Paskalev, guest blogging with good interesting articles is always mostly welcome 

Perl Modules via HTTP Proxy installation, update and install perl CPAN modules behind a Firewall DMZ-ed networks


May 19th, 2023

If you have to maintain perl script written applications on Linux servers that are sitting behind a very paranoid set-up firewalls
and Local DMZ network, but you still need to maintain the servers and applications versions including perl CPAN (Comprehensive Perl Archive networking) module libraries, you could still do so via another Proxy machine Hub on the Local network, where you either have to manually download all the newest perl versions and CPAN module (libraries) or you can set it up that Proxy machine
to access only a specific Secured internet URLs for perl stuff.

Proxying perl downloads can be done via FTP connects, but as FTP is communicating in plain text and the protocol is known
for not behaving very well behind firewalls, it is a better idea to use for CPAN downloads HTTP or HTTPS protocol.

Normlly Perl is using FTP to download files from the internet. To enable Perl using also HTTP, please install the following RPM:

# yum install perl-libwww-perl

After figure out a CPAN-mirror from http://mirrors.cpan.org/search.cgi?country=Germany, we can start with the configuration from CPAN.

or debian package

# apt install libwww-perl

At the first run from /usr/bin/cpan the initial configuration will be started up which will be done automatically:

[username@linux-host ~]$ cpan

/home/linux-username/.cpan/CPAN/MyConfig.pm initialized.


CPAN is the world-wide archive of perl resources. It consists of about
100 sites that all replicate the same contents all around the globe.
Many countries have at least one CPAN site already. The resources
found on CPAN are easily accessible with the CPAN.pm module. If you
want to use CPAN.pm, you have to configure it properly.

If you do not want to enter a dialog now, you can answer 'no' to this
question and I'll try to autoconfigure. (Note: you can revisit this
dialog anytime later by typing 'o conf init' at the cpan prompt.)

Are you ready for manual configuration? [yes] no

— SNIP — SNAP — SNIP — SNAP — SNIP — SNAP —

commit: wrote /home/linux-username/.cpan/CPAN/MyConfig.pm
Terminal does not support AddHistory.

cpan shell — CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan> q
Terminal does not support GetHistory.
Lockfile removed.
[username@linux-host ~]$

After the initial configuration you have to run /usr/bin/cpan again, to configure the HTTP-proxy and an alternative HTTP-URL for the default FTP URL:
 

[username@linux-host ~]$ cpan
Terminal does not support AddHistory.

cpan shell — CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan> o conf http_proxy http://proxy-host-to-internet.com:8080
    http_proxy         http://proxy-host-to-internet.com:8080

cpan> o conf urllist push http://mirrors.zerg.biz/cpan/

cpan> o conf commit
commit: wrote /home/linux-username/.cpan/CPAN/MyConfig.pm

cpan> q
Terminal does not support GetHistory.
Lockfile removed.
[username@linux-host ~]$

From now CPAN will load it's files from the internet with the HTTP-proxy:
 

[username@linux-host ~]$ cpan
Terminal does not support AddHistory.

cpan shell — CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan> i  Example::DB::Oracle
CPAN: Storable loaded ok
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
  http://mirrors.zerg.biz/cpan/authors/01mailrc.txt.gz
Going to read /home/linux-host/.cpan/sources/authors/01mailrc.txt.gz
Fetching with LWP:
  http://mirrors.zerg.biz/cpan/modules/02packages.details.txt.gz
Going to read /home/linux-host/.cpan/sources/modules/02packages.details.txt.gz
  Database was generated on Thu, 07 Jan 2010 10:44:22 GMT

  There's a new CPAN.pm version (v1.9402) available!
  [Current version is v1.7602]
  You might want to try
    install Bundle::CPAN
    reload cpan
  without quitting the current session. It should be a seamless upgrade
  while we are running…

Fetching with LWP:
  http://mirrors.zerg.biz/cpan/modules/03modlist.data.gz
Going to read /home/linux-username/.cpan/sources/modules/03modlist.data.gz
Going to write /home/linux-username/.cpan/Metadata
Strange distribution name [Example::DB::Oracle]
Module id = Example::DB::Oracle
    CPAN_USERID  MSERGEANT (MSERGEANT <msergeant@cpan.org>)
    CPAN_VERSION undef
    CPAN_FILE    M/MS/MSERGEANT/DBIx-AnyDBD-2.01.tar.gz
    INST_FILE    (not installed)

cpan> q
Terminal does not support GetHistory.
Lockfile removed.

Now as the new proxy http URL http://proxy-host-to-internet.com:8080 is set on the machine, to upgrade the existing modules non interactively

# perl MCPAN -e upgrade

or do it the old fashioned way via the MCPAN perl shell:

# perl -MCPAN -e shell

Starting with version 2.29 of the cpan shell, a new download mechanism
is the default which exclusively uses cpan.org as the host to download
from. The configuration variable pushy_https can be used to (de)select
the new mechanism. Please read more about it and make your choice
between the old and the new mechanism by running

    o conf init pushy_https

Once you have done that and stored the config variable this dialog
will disappear.

cpan shell — CPAN exploration and modules installation (v2.29)
Enter 'h' for help.

cpan[1]> upgrade

That's all folks after a while if no errors are spit during the Perl modules update you'll be at the latest versions of CPAN and modules.
 

 

The Holy Martyrs of Novo Selo Monastery “Holy Trinity” Selo Bulgaria. The holy priest-martyrs and martyrs who suffered by the Turks on May 9, 1876, canonized on April 3, 2011


May 10th, 2023

Saints-of-Monastery-of-Novo-Selo-today-city-of-Apriltzi-monastery-of-Holy_Trinity

There are many saints who labored hard over the centuries, but there are few known that has done their feat in the Bulgarian lands. We Christian honor of the saints for centuries and whose memory is marked in the Church calendar of Saints, but there are some less known but not lesser in their confessory saints than the ancients. The Lord honors with eternal wreaths not only the sufferers of the early Christian times (during the early periods of persecution 1st and 2nd century) , but also those who are much closer to us in time and place and have endured persecutions and torments no less than the ancients.

We know about them from history, but it is not history that will join us to the sacrificial table, that will gather us in the church temple and that will strengthen us in faith. The signs of holiness are given to us from above, but our participation to accept it is also necessary, our "Let it be, Oh Lord!", another knowledge, so that we can pluck holiness from the depths of oblivion, be part of its path, tell about it and we bear witness to her miracles.

Today is another day of reaffirmation of holiness, deserved before God and known to people for a time.
Today, our thoughts and prayers are directed to the foothills of the Central Balkans (mountains) and to the Holy Trinity Monastery, illuminated by the martyrdom of the sisters, whose feet once touched the ground here and prayer chants rose day and night along with the fragrant smoke from the incense in the temple. And indeed, this monastery is like a candlestick placed on high, so that the light from it, which illuminated our land in the time of April's bloody harvest (on 20th of April 1976, The April Revolt against the Ottoman turks yoke has arised) never goes out and shines again, especially when our Church needs it.

Although it is not ancient, like others that glorify the Lord in the Slavic language over the centuries, the virgin monastery "Holy Trinity" was built around 1830 – when the Orthodox faith, like a vine, daily burned by non-believers, instead of withering – strengthened , and the thought of liberation more and more embraces souls and gives hope. For decades girls from the most awake and enlightened Orthodox families – not only from Novo Selo, but also from the neighboring places – put themselves under the protection of the One God the Holy Trinity. The inhabitants of Novo Selo ( Novoseltsi ) are famous with his zeal – both to the Church and to good morals.

Indeed, Novo Selo is a place worthy of raising and offering to the Most High the animate sacrifices of the true faith. To work for the pleasing to God, every wealthy family gave its contribution – and not only in form of money donations, lands and all kind of donations for the maintenance of the temples, the monastery, but also in the selection of worthy priests who would lay down their souls for their (flock) pasoms. And it is enough to mention only the two who received from the Savior a bright crown for their martyrdom, so that through their holy endurance and faith be be praised through them all the priests, guardians of the Christian faith who put their live for the herd during the many years of slavery that Bulgarians, Serbians, Greek, Romanians and many other Christians has suffered from the unfaithful.

With what words can we describe the zeal of priest Nikola Barbulov, a teacher and presbyter with a wise soul, who worked hard to successfully complete his studies in Bessarabia (nowadays Moldavia). He was fond of books, prosperous as a teacher, worthy as a priest, and educated many pupils; his spiritual children were the most active part of the Church body, among which many received priesthood as well.

Saint Pope Nicholas is similar to the great patriarch Euthymius, because he stayed to the end to guard his verbal flock; moreover, he was not a bloodless martyr, but a sufferer, put to death in terrible torments without renouncing his faith.


And with what words should we call the priest Georgi Dylgodreiski?

A warrior of Christ, as if a second Saint George, he fought not only against the thought, but also against the living enemy of the Cross. The new passion sufferer did not defend himself in the single combat with the Agarians (ottoman turks), he did not defend himself, he did not think about himself, but how to slow down the hordes of beasts and give the weak and helpless at least a little time to escape or hide from the ferocious infidels.

He guarded the House of God (the Church) and accepted circumcision as a grace, because he knew that through it he received eternal life.
We will not have enough words to praise the strength of those fragile nuns – fragile by nature, but strong in their faith.

The testimony of their life and death are the stories passed down from generation to generation in the new village families. There are not many memories, but the grace of their martyrdom is like an invisible light above the earth and, even without mentioning them, without praying to them, they are intercessors for those in trouble. There is no need for descriptions of their existence, because it is before the eyes of all who know the order in Orthodox monasteries. The sisters lived a celibate and angelic life – in deeds and prayers, in common breathing. Through their association they were a model and a guarantee of virtue to each one of them.
They prayed not only for their souls and for the salvation and well-being of their neighbors, but also for what was then in the heart of every Bulgarian – that God's destinies would be fulfilled and the Fatherland would once again become an independent Christian state.
Raised from childhood in piety and firm confession, entrusted by their relatives to God and the Mother of God, the brides of Christ humbly and daily wore the sweet yoke of Christ.

 

Every good deed of theirs – both prayer and manual work (following the example of the ancients saints of the path of ‘ora and labora’ – added oil to the lamps with which they would welcome the Bridegroom and enter with Him into the Kingdom of God.

The abbess of the monastery at that time was Susana, daughter of the mayor of the village, the fighter for faith and family Tsonko Somlev.

She worthily carries the burden of the board, making sure that the sisters reside according to the statute drawn up for them on Mount Athos by hieromonk Spyridon.
The ordeal of the priests and nuns whose martyrdom we celebrate today began when, on May 1, 1876, a mutiny signal was given on Mount Baban.

During this very harsh times for Bulgarian enslaved people in attempt to revolt against the unrighteess system of Ottoman empire not honoring the liberty and rights of people, the drunkenness of the breath of freedom raised the people to their feet, and the very next day the leaders announced the Novosel Republic (as a separate entity from the Ottoman Empire). During the several days of celebration, bells ring the valley, and prayer chants invite everyone to experience a moment of earthly joy, but also to call on the Lord of hosts to be their helper in the coming sorrows …

In the following days, the Christians tried to organize themselves and resist the Agarians, which hearing about the desire of people to self-govern themselves in a new tiny Christian country.

The unexpected cold and snow tormented the Chetniks (armed group part of Cheta a small battalion of armed liberation forces) of Tsanko Dyustabanov, and hunger weakened severely their strength.

And here the thirty nuns of the "Holy Trinity," adding to their prayers and service with the strength of their godly devotion, give no sleep to their eyes, nor slumber to their eyelids, but some alternate in the kitchen and bakery, and others took constant care for the under-shoeed and under-dressed boys, collecting everything that could serve them as clothing – socks knitted manually by them, scarfs, warm flannels…

And just according to the Gospel of Christ, anticipating the close meeting with the Bridegroom, they repeat the words of the Psalmist: "My heart is ready, God, my heart is ready…" (Ps. 56:8).

At that time, an army gathered around Sevlievo and Pleven, but not regular soldiers, to fight only against the armed chetniks and to keep the many innocent peaceful villants who officially did not took the guns but only supported and beseeched for the freedom of the darkness of harsh taxes and lack of rights as the ordinary muslims.

A blasphemous congregation of Abazis and Circassians is coming and multiplying around the monastery.

Adding to their natural demonic cruelty the orders of their leader, the thrice-cursed Deli Nejib Agha, they, seized with depravity and a desire to kill, slither like locusts, consigning to death and scorn all living things in their path..
Some of the residents, known for the approaching bashi-bazouk (irregular soldiers of Ottomans army rised in times of war), are hiding in the mountains, others are running in panic to the fields, and some of the houses in the new village are already engulfed in flames.

However, most of the nuns remain in the monastery.
In vain they hope that the hordes will not dare to desecrate the Holy presanctified heavenly place of God, the Holy Trinity monastery.

The priest Georgi Hristov, holed up in the upper floor of one of the monastery buildings, tries to slow down the enemy and give at least some of the fleeing time to escape.

There, once captured his body was cut down bit by bit by the enraged Circassians who burst through the monastery gate…

The last refuge of some of the nuns is the church building. One of them fails to enter with the others and is cut with a scythe (turkish half moon shaped sword) at the threshold of the temple.

We know about the last moments of the earthly life of the new martyrs from the shocking stories of four of the nuns who survived after having inflicted unbearable sufferings.

For the rest of their lives, they relive their humiliation and humiliation and vilify those of their sisters who took the martyrdom wreath.

The ungodly infidels shoot through the windows and hit several of the sisters.
Then they enter the church and start cutting whoever they want – both the living and the dead.
In front of the Church altar and inside it, Mother Abbess Susana, six other nuns and one laywoman died.

The abuses of the innocent victims, according to the testimonies, are inhumanly cruel. When they begin to strip everyone of their clothes and see that one of the sisters is still alive, the demonic minions blind and suffocate her, stuffing her eyes and mouth with mortar. The enraged and embittered instruments of the devil indulge in robbery and violence, not only in the church and the monastery, but everywhere in the settlement.

And so for the hearth (in glory) of the Bulgarian Orthodox true Christian faith, which was destroyed by the permission of the all-powerful providence. Ehat was said came true: "They shed their blood like water around Jerusalem and there was no one to bury them: they left the corpses of the slaves for food for the birds of the sky and the bodies of your reverends – for the earthly beasts."

But what is happening in Novo Selo is even more terrible – the Agarians not only desecrate the bodies of their victims. In the monastery church, they cut the icons into splinters, ransacked everything, and finally began to burn both the holy abode and the houses, so that there was nothing left to bury and mourn, and everywhere the abomination of desolation reigned.

But in the temple, the fire does not completely destroy the remains of the martyrs, so that later the dry bones can speak to everyone who looks at them with believing eyes and bows before their feat. And the testimony of our words is the ossuary of the "Holy Trinity" monastery rebuilt from the ashes, where one feels the invisible spiritual power and God's grace flowing from the remains – the holy relics – of the martyrs.

And let us from today on May 9, on the day of their suffering death, call them by their names, so that we also have their holy intercession prayers: you, newly martyred holy nuns
Susanna, Sophia, Elisaveta, Ephrosinia, Christina, Calista and Ekaterina,

and you, Susana Chorbadjieva, who during her lifetime was not able to join the sisters, but through your blood received a place in their image, as well as you, newly martyred priests Nicholai and Georgi, a couple of sympathizers and namesakes of the Glorious 9 Martyrs of Serdika (martyred near city of Sofia), together with all others who suffered for the faith and family, whose names now only the Lord knows, because they are all with Him in eternal and endless life, pray to the Holy Trinity, call on the Holy Mother of God and all the saints and give us strength to preserve your memory, so that you will be mentioned in future generations and forever. Amen.

* Novo selo – now a district of the town of Apriltsi Bulgaria.

Translated with minor inclusions from Official site of Bulgarian Orthodox Church

Origianl Bulgarian text Source : Holy Metropolis of Lovchan, Bulgaria
Note:
The Holy Novoselski Muchenici was canonized by the Bulgarian Orthodox Church, after following a canonization procedures and affirmation of the saintship of the martyrs of Novoselo decided to officially
canonize the saints together with the Martyrs Saints of Batak
.
Canonization of the Nove Selo Monastery saints, was officially announced with a Holy Liturgy in Sofia Capital, Church Saint Alexander Nevski  on April 3, 2011.