How to RPM update Hypervisors and Virtual Machines running Haproxy High Availability cluster on KVM, Virtuozzo without a downtime on RHEL / CentOS Linux


May 20th, 2022

virtuozzo-kvm-virtual-machines-and-hypervisor-update-manual-haproxy-logo


Here is the scenario, lets say you have on your daily task list two Hypervisor (HV) hosts running CentOS or RHEL Linux with KVM or Virutozzo technology and inside the HV hosts you have configured at least 2 pairs of virtual machines one residing on HV Host 1 and one residing on HV Host 2 and you need to constantly keep the hosts to the latest distribution major release security patchset.

The Virtual Machines has been running another set of Redhat Linux or CentOS configured to work in a High Availability Cluster running Haproxy / Apache / Postfix or any other kind of HA solution on top of corosync / keepalived or whatever application cluster scripts Free or Open Source technology that supports a switch between clustered Application nodes.

The logical question comes how to keep up the CentOS / RHEL Machines uptodate without interfering with the operations of the Applications running on the cluster?

Assuming that the 2 or more machines are configured to run in Active / Passive App member mode, e.g. one machine is Active at any time and the other is always Passive, a switch is possible between the Active and Passive node.

HAProxy--Load-Balancer-cluster-2-nodes-your-Servers

In this article I'll give a simple step by step tested example on how you I succeeded to update (for security reasons) up to the latest available Distribution major release patchset on one by one first the Clustered App on Virtual Machines 1 and VM2 on Linux Hypervisor Host 1. Then the App cluster VM 1 / VM 2 on Hypervisor Host 2.
And finally update the Hypervisor1 (after moving the Active resources from it to Hypervisor2) and updating the Hypervisor2 after moving the App running resources back on HV1.
I know the procedure is a bit monotonic but it tries to go through everything step by step to try to mitigate any possible problems. In case of failure of some rpm dependencies during yum / dnf tool updates you can always revert to backups so in anyways don't forget to have a fully functional backup of each of the HV hosts and the VMs somewhere on a separate machine before proceeding further, any possible failures due to following my aritcle literally is your responsibility 🙂

 

0. Check situation before the update on HVs / get VM IDs etc.

Check the virsion of each of the machines to be updated both Hypervisor and Hosted VMs, on each machine run:
 

# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)


The machine setup I'll be dealing with is as follows:
 

hypervisor-host1 -> hypervisor-host1.fqdn.com 
•    virt-mach-centos1
•    virt-machine-zabbix-proxy-centos (zabbix proxy)

hypervisor-host2 -> hypervisor-host2.fqdn.com
•    virt-mach-centos2
•    virt-machine-zabbix2-proxy-centos (zabbix proxy)

To check what is yours check out with virsh cmd –if on KVM or with prlctl if using Virutozzo, you should get something like:

[root@hypervisor-host2 ~]# virsh list
 Id Name State
—————————————————-
 1 vm-host1 running
 2 virt-mach-centos2 running

 # virsh list –all

[root@hypervisor-host1 ~]# virsh list
 Id Name State
—————————————————-
 1 vm-host2 running
 3 virt-mach-centos1 running

[root@hypervisor-host1 ~]# prlctl list
UUID                                    STATUS       IP_ADDR         T  NAME
{dc37c201-08c9-589d-aa20-9386d63ce3f3}  running      –               VM virt-mach-centos1
{76e8a5f8-caa8-5442-830e-aa4bfe8d42d9}  running      –               VM vm-host2
[root@hypervisor-host1 ~]#

If you have stopped VMs with Virtuozzo to list the stopped ones as well.
 

# prlctl list -a

[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# vzlist -a
                                CTID      NPROC STATUS    IP_ADDR         HOSTNAME
[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# prlctl list
UUID                                    STATUS       IP_ADDR         T  NAME
{92075803-a4ce-5ec0-a3d8-9ee83d85fc76}  running      –               VM virt-mach-centos2
{74a7bbe8-9245-5385-ac0d-d10299100789}  running      –               VM vm-host1

# prlctl list -a


If due to Virtuozzo version above command does not return you can manually check in the VM located folder, VM ID etc.
 

[root@hypervisor-host2 vmprivate]# ls
74a7bbe8-9245-4385-ac0d-d10299100789  92075803-a4ce-4ec0-a3d8-9ee83d85fc76
[root@hypervisor-host2 vmprivate]# pwd
/vz/vmprivate
[root@hypervisor-host2 vmprivate]#


[root@hypervisor-host1 ~]# ls -al /vz/vmprivate/
total 20
drwxr-x—. 5 root root 4096 Feb 14  2019 .
drwxr-xr-x. 7 root root 4096 Feb 13  2019 ..
drwxr-x–x. 4 root root 4096 Feb 18  2019 1c863dfc-1deb-493c-820f-3005a0457627
drwxr-x–x. 4 root root 4096 Feb 14  2019 76e8a5f8-caa8-4442-830e-aa4bfe8d42d9
drwxr-x–x. 4 root root 4096 Feb 14  2019 dc37c201-08c9-489d-aa20-9386d63ce3f3
[root@hypervisor-host1 ~]#


Before doing anything with the VMs, also don't forget to check the Hypervisor hosts has enough space, otherwise you'll get in big troubles !
 

[root@hypervisor-host2 vmprivate]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos_hypervisor-host2-root   20G  1.8G   17G  10% /
devtmpfs                          20G     0   20G   0% /dev
tmpfs                             20G     0   20G   0% /dev/shm
tmpfs                             20G  2.0G   18G  11% /run
tmpfs                             20G     0   20G   0% /sys/fs/cgroup
/dev/sda1                        992M  159M  766M  18% /boot
/dev/mapper/centos_hypervisor-host2-home  9.8G   37M  9.2G   1% /home
/dev/mapper/centos_hypervisor-host2-var   9.8G  355M  8.9G   4% /var
/dev/mapper/centos_hypervisor-host2-vz    755G   25G  692G   4% /vz

 

[root@hypervisor-host1 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  1.8G   45G   4% /
devtmpfs                  20G     0   20G   0% /dev
tmpfs                     20G     0   20G   0% /dev/shm
tmpfs                     20G  2.1G   18G  11% /run
tmpfs                     20G     0   20G   0% /sys/fs/cgroup
/dev/sda2                992M  153M  772M  17% /boot
/dev/mapper/centos-home  9.8G   37M  9.2G   1% /home
/dev/mapper/centos-var   9.8G  406M  8.9G   5% /var
/dev/mapper/centos-vz    689G   12G  643G   2% /vz

Another thing to do before proceeding with update is to check and tune if needed the amount of CentOS repositories used, before doing anything with yum.
 

[root@hypervisor-host2 yum.repos.d]# ls -al
total 68
drwxr-xr-x.   2 root root  4096 Oct  6 13:13 .
drwxr-xr-x. 110 root root 12288 Oct  7 11:13 ..
-rw-r–r–.   1 root root  4382 Mar 14  2019 CentOS7.repo
-rw-r–r–.   1 root root  1664 Sep  5  2019 CentOS-Base.repo
-rw-r–r–.   1 root root  1309 Sep  5  2019 CentOS-CR.repo
-rw-r–r–.   1 root root   649 Sep  5  2019 CentOS-Debuginfo.repo
-rw-r–r–.   1 root root   314 Sep  5  2019 CentOS-fasttrack.repo
-rw-r–r–.   1 root root   630 Sep  5  2019 CentOS-Media.repo
-rw-r–r–.   1 root root  1331 Sep  5  2019 CentOS-Sources.repo
-rw-r–r–.   1 root root  6639 Sep  5  2019 CentOS-Vault.repo
-rw-r–r–.   1 root root  1303 Mar 14  2019 factory.repo
-rw-r–r–.   1 root root   666 Sep  8 10:13 openvz.repo
[root@hypervisor-host2 yum.repos.d]#

 

[root@hypervisor-host1 yum.repos.d]# ls -al
total 68
drwxr-xr-x.   2 root root  4096 Oct  6 13:13 .
drwxr-xr-x. 112 root root 12288 Oct  7 11:09 ..
-rw-r–r–.   1 root root  1664 Sep  5  2019 CentOS-Base.repo
-rw-r–r–.   1 root root  1309 Sep  5  2019 CentOS-CR.repo
-rw-r–r–.   1 root root   649 Sep  5  2019 CentOS-Debuginfo.repo
-rw-r–r–.   1 root root   314 Sep  5  2019 CentOS-fasttrack.repo
-rw-r–r–.   1 root root   630 Sep  5  2019 CentOS-Media.repo
-rw-r–r–.   1 root root  1331 Sep  5  2019 CentOS-Sources.repo
-rw-r–r–.   1 root root  6639 Sep  5  2019 CentOS-Vault.repo
-rw-r–r–.   1 root root  1303 Mar 14  2019 factory.repo
-rw-r–r–.   1 root root   300 Mar 14  2019 obsoleted_tmpls.repo
-rw-r–r–.   1 root root   666 Sep  8 10:13 openvz.repo


1. Dump VM definition XMs (to have it in case if it gets wiped during update)

There is always a possibility that something will fail during the update and you might be unable to restore back to the old version of the Virtual Machine due to some config misconfigurations or whatever thus a very good idea, before proceeding to modify the working VMs is to use KVM's virsh and dump the exact set of XML configuration that makes the VM roll properly.

To do so:
Check a little bit up in the article how we have listed the IDs that are part of the directory containing the VM.
 

[root@hypervisor-host1 ]# virsh dumpxml (Id of VM virt-mach-centos1 ) > /root/virt-mach-centos1_config_bak.xml
[root@hypervisor-host2 ]# virsh dumpxml (Id of VM virt-mach-centos2) > /root/virt-mach-centos2_config_bak.xml

 


2. Set on standby virt-mach-centos1 (virt-mach-centos1)

As I'm upgrading two machines that are configured to run an haproxy corosync cluster, before proceeding to update the active host, we have to switch off
the proxied traffic from node1 to node2, – e.g. standby the active node, so the cluster can move up the traffic to other available node.
 

[root@virt-mach-centos1 ~]# pcs cluster standby virt-mach-centos1


3. Stop VM virt-mach-centos1 & backup on Hypervisor host (hypervisor-host1) for VM1

Another prevention step to make sure you don't get into damaged VM or broken haproxy cluster after the upgrade is to of course backup 

 

[root@hypervisor-host1 ]# prlctl backup virt-mach-centos1

or
 

[root@hypervisor-host1 ]# prlctl stop virt-mach-centos1
[root@hypervisor-host1 ]# cp -rpf /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3 /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3-bak
[root@hypervisor-host1 ]# tar -czvf virt-mach-centos1_vm_virt-mach-centos1.tar.gz /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3

[root@hypervisor-host1 ]# prlctl start virt-mach-centos1


4. Remove package version locks on all hosts

If you're using package locking to prevent some other colleague to not accidently upgrade the machine (if multiple sysadmins are managing the host), you might use the RPM package locking meachanism, if that is used check RPM packs that are locked and release the locking.

+ List actual list of locked packages

[root@hypervisor-host1 ]# yum versionlock list  
…
…..
0:libtalloc-2.1.16-1.el7.*
0:libedit-3.0-12.20121213cvs.el7.*
0:p11-kit-trust-0.23.5-3.el7.*
1:quota-nls-4.01-19.el7.*
0:perl-Exporter-5.68-3.el7.*
0:sudo-1.8.23-9.el7.*
0:libxslt-1.1.28-5.el7.*
versionlock list done                          

+ Clear the locking            

# yum versionlock clear                               


+ List actual list / == clear all entries
 

[root@virt-mach-centos2 ]# yum versionlock list; yum versionlock clear
[root@virt-mach-centos1 ]# yum versionlock list; yum versionlock clear
[root@hypervisor-host1 ~]# yum versionlock list; yum versionlock clear
[root@hypervisor-host2 ~]# yum versionlock list; yum versionlock clear


5. Do yum update virt-mach-centos1


For some clarity if something goes wrong, it is really a good idea to make a dump of the basic packages installed before the RPM package update is initiated,
The exact versoin of RHEL or CentOS as well as the list of locked packages, if locking is used.

Enter virt-mach-centos1 (ssh virt-mach-centos1) and run following cmds:
 

# cat /etc/redhat-release  > /root/logs/redhat-release-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


+ Only if needed!!
 

# yum versionlock clear
# yum versionlock list


Clear any previous RPM packages – careful with that as you might want to keep the old RPMs, if unsure comment out below line
 

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

Proceed with the update and monitor closely the output of commands and log out everything inside files using a small script that you should place under /root/status the script is given at the end of the aritcle.:
 

yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
yum check-update | wc -l
yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

6. Check if everything is running fine after upgrade

Reboot VM
 

# shutdown -r now


7. Stop VM virt-mach-centos2 & backup  on Hypervisor host (hypervisor-host2)

Same backup step as prior 

# prlctl backup virt-mach-centos2


or
 

# prlctl stop virt-mach-centos2
# cp -rpf /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76 /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76-bak
## tar -czvf virt-mach-centos2_vm_virt-mach-centos2.tar.gz /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76

# prctl start virt-mach-centos2


8. Do yum update on virt-mach-centos2

Log system state, before the update
 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list

 

+ Clean old install update / packages if required
 

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Initiate the update

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l 
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


9. Check if everything is running fine after upgrade
 

Reboot VM
 

# shutdown -r now

 

10. Stop VM vm-host2 & backup
 

# prlctl backup vm-host2


or

# prlctl stop vm-host2

Or copy the actual directory containig the Virtozzo VM (use the correct ID)
 

# cp -rpf /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9 /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9-bak
## tar -czvf vm-host2.tar.gz /vz/vmprivate/76e8a5f8-caa8-4442-830e-aa5bfe8d42d9

# prctl start vm-host2


11. Do yum update vm-host2
 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Clear only if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Do the rpm upgrade

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


12. Check if everything is running fine after upgrade
 

Reboot VM
 

# shutdown -r now


13. Do yum update hypervisor-host2

 

 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock   if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Update rpms
 

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


14. Stop VM vm-host1 & backup


Some as ealier
 

# prlctl backup vm-host1

or
 

# prlctl stop vm-host1

# cp -rpf /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789 /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789-bak
# tar -czvf vm-host1.tar.gz /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789

# prctl start vm-host1


15. Do yum update vm-host2
 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


16. Check if everything is running fine after upgrade

+ Reboot VM

# shutdown -r now


17. Do yum update hypervisor-host1

Same procedure for HV host 1 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock
 

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


18. Check if everything is running fine after upgrade

Reboot VM
 

# shutdown -r now


Check hypervisor-host1 all VMs run as expected 


19. Check if everything is running fine after upgrade

Reboot VM
 

# shutdown -r now


Check hypervisor-host2 all VMs run as expected afterwards


20. Check once more VMs and haproxy or any other contained services in VMs run as expected

Login to hosts and check processes and logs for errors etc.
 

21. Haproxy Unstandby virt-mach-centos1

Assuming that the virt-mach-centos1 and virt-mach-centos2 are running a Haproxy / corosync cluster you can try to standby node1 and check the result
hopefully all should be fine and traffic should come to host node2.

[root@virt-mach-centos1 ~]# pcs cluster unstandby virt-mach-centos1


Monitor logs and make sure HAproxy works fine on virt-mach-centos1


22. If necessery to redefine VMs (in case they disappear from virsh) or virtuosso is not working

[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos1_config_bak.xml
[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos2_config_bak.xml


23. Set versionlock to RPMs to prevent accident updates and check OS version release

[root@virt-mach-centos2 ]# yum versionlock \*
[root@virt-mach-centos1 ]# yum versionlock \*
[root@hypervisor-host1 ~]# yum versionlock \*
[root@hypervisor-host2 ~]# yum versionlock \*

[root@hypervisor-host2 ~]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)

Other useful hints

[root@hypervisor-host1 ~]# virsh console dc37c201-08c9-489d-aa20-9386d63ce3f3
Connected to domain virt-mach-centos1
..

! Compare packages count before the upgrade on each of the supposable identical VMs and HVs – if there is difference in package count review what kind of packages are different and try to make the machines to look as identical as possible  !

Packages to update on hypervisor-host1 Count: XXX
Packages to update on hypervisor-host2 Count: XXX
Packages to update virt-mach-centos1 Count: – 254
Packages to update virt-mach-centos2 Count: – 249

The /root/status script

+++

#!/bin/sh
echo  '=======================================================   '
echo  '= Systemctl list-unit-files –type=service | grep enabled '
echo  '=======================================================   '
systemctl list-unit-files –type=service | grep enabled

echo  '=======================================================   '
echo  '= systemctl | grep ".service" | grep "running"            '
echo  '=======================================================   '
systemctl | grep ".service" | grep "running"

echo  '=======================================================   '
echo  '= chkconfig –list                                        '
echo  '=======================================================   '
chkconfig –list

echo  '=======================================================   '
echo  '= netstat -tulpn                                          '
echo  '=======================================================   '
netstat -tulpn

echo  '=======================================================   '
echo  '= netstat -r                                              '
echo  '=======================================================   '
netstat -r


+++

That's all folks, once going through the article, after some 2 hours of efforts or so you should have an up2date machines.
Any problems faced or feedback is mostly welcome as this might help others who have the same setup.

Thanks for reading me 🙂

Tags: , , , , , , , , , , , , , , , ,

No Comments »

« Previous Post

How to monitor Haproxy Application server backends with Zabbix userparameter autodiscovery scripts


May 13th, 2022

zabbix-backend-monitoring-logo

Haproxy is doing quite a good job in High Availability tasks where traffic towards multiple backend servers has to be redirected based on the available one to sent data from the proxy to. 

Lets say haproxy is configured to proxy traffic for App backend machine1 and App backend machine2.

Usually in companies people configure a monitoring like with Icinga or Zabbix / Grafana to keep track on the Application server is always up and running. Sometimes however due to network problems (like burned Network Switch / router or firewall misconfiguration) or even an IP duplicate it might happen that Application server seems to be reporting reachable from some monotoring tool on it but unreachable from  Haproxy server -> App backend machine2 but reachable from App backend machine1. And even though haproxy will automatically switch on the traffic from backend machine2 to App machine1. It is a good idea to monitor and be aware that one of the backends is offline from the Haproxy host.
In this article I'll show you how this is possible by using 2 shell scripts and userparameter keys config through the autodiscovery zabbix legacy feature.
Assumably for the setup to work you will need to have as a minimum a Zabbix server installation of version 5.0 or higher.

1. Create the required  haproxy_discovery.sh  and haproxy_stats.sh scripts 

You will have to install the two scripts under some location for example we can put it for more clearness under /etc/zabbix/scripts

[root@haproxy-server1 ]# mkdir /etc/zabbix/scripts

[root@haproxy-server1 scripts]# vim haproxy_discovery.sh 
#!/bin/bash
#
# Get list of Frontends and Backends from HAPROXY
# Example: ./haproxy_discovery.sh [/var/lib/haproxy/stats] FRONTEND|BACKEND|SERVERS
# First argument is optional and should be used to set location of your HAPROXY socket
# Second argument is should be either FRONTEND, BACKEND or SERVERS, will default to FRONTEND if not set
#
# !! Make sure the user running this script has Read/Write permissions to that socket !!
#
## haproxy.cfg snippet
#  global
#  stats socket /var/lib/haproxy/stats  mode 666 level admin

HAPROXY_SOCK=""/var/run/haproxy/haproxy.sock
[ -n “$1” ] && echo $1 | grep -q ^/ && HAPROXY_SOCK="$(echo $1 | tr -d '\040\011\012\015')"

if [[ “$1” =~ (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?):[0-9]{1,5} ]];
then
    HAPROXY_STATS_IP="$1"
    QUERYING_METHOD="TCP"
fi

QUERYING_METHOD="${QUERYING_METHOD:-SOCKET}"

query_stats() {
    if [[ ${QUERYING_METHOD} == “SOCKET” ]]; then
        echo "show stat" | socat ${HAPROXY_SOCK} stdio 2>/dev/null
    elif [[ ${QUERYING_METHOD} == “TCP” ]]; then
        echo "show stat" | nc ${HAPROXY_STATS_IP//:/ } 2>/dev/null
    fi
}

get_stats() {
        echo "$(query_stats)" | grep -v "^#"
}

[ -n “$2” ] && shift 1
case $1 in
        B*) END="BACKEND" ;;
        F*) END="FRONTEND" ;;
        S*)
                for backend in $(get_stats | grep BACKEND | cut -d, -f1 | uniq); do
                        for server in $(get_stats | grep "^${backend}," | grep -v BACKEND | grep -v FRONTEND | cut -d, -f2); do
                                serverlist="$serverlist,\n"'\t\t{\n\t\t\t"{#BACKEND_NAME}":"'$backend'",\n\t\t\t"{#SERVER_NAME}":"'$server'"}'
                        done
                done
                echo -e '{\n\t"data":[\n’${serverlist#,}’]}'
                exit 0
        ;;
        *) END="FRONTEND" ;;
esac

for frontend in $(get_stats | grep "$END" | cut -d, -f1 | uniq); do
    felist="$felist,\n"'\t\t{\n\t\t\t"{#'${END}'_NAME}":"'$frontend'"}'
done
echo -e '{\n\t"data":[\n’${felist#,}’]}'

 

[root@haproxy-server1 scripts]# vim haproxy_stats.sh 
#!/bin/bash
set -o pipefail

if [[ “$1” = /* ]]
then
  HAPROXY_SOCKET="$1"
  shift 0
else
  if [[ “$1” =~ (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?):[0-9]{1,5} ]];
  then
    HAPROXY_STATS_IP="$1"
    QUERYING_METHOD="TCP"
    shift 1
  fi
fi

pxname="$1"
svname="$2"
stat="$3"

DEBUG=${DEBUG:-0}
HAPROXY_SOCKET="${HAPROXY_SOCKET:-/var/run/haproxy/haproxy.sock}"
QUERYING_METHOD="${QUERYING_METHOD:-SOCKET}"
CACHE_STATS_FILEPATH="${CACHE_STATS_FILEPATH:-/var/tmp/haproxy_stats.cache}"
CACHE_STATS_EXPIRATION="${CACHE_STATS_EXPIRATION:-1}" # in minutes
CACHE_INFO_FILEPATH="${CACHE_INFO_FILEPATH:-/var/tmp/haproxy_info.cache}" ## unused
CACHE_INFO_EXPIRATION="${CACHE_INFO_EXPIRATION:-1}" # in minutes ## unused
GET_STATS=${GET_STATS:-1} # when you update stats cache outsise of the script
SOCAT_BIN="$(which socat)"
NC_BIN="$(which nc)"
FLOCK_BIN="$(which flock)"
FLOCK_WAIT=15 # maximum number of seconds that "flock" waits for acquiring a lock
FLOCK_SUFFIX='.lock'
CUR_TIMESTAMP="$(date '+%s')"

debug() {
  [ “${DEBUG}” -eq 1 ] && echo "DEBUG: $@" >&2 || true
}

debug "SOCAT_BIN        => $SOCAT_BIN"
debug "NC_BIN           => $NC_BIN"
debug "FLOCK_BIN        => $FLOCK_BIN"
debug "FLOCK_WAIT       => $FLOCK_WAIT seconds"
debug "CACHE_FILEPATH   => $CACHE_FILEPATH"
debug "CACHE_EXPIRATION => $CACHE_EXPIRATION minutes"
debug "HAPROXY_SOCKET   => $HAPROXY_SOCKET"
debug "pxname   => $pxname"
debug "svname   => $svname"
debug "stat     => $stat"

# check if socat is available in path
if [ “$GET_STATS” -eq 1 ] && [[ $QUERYING_METHOD == “SOCKET” && -z “$SOCAT_BIN” ]] || [[ $QUERYING_METHOD == “TCP” &&  -z “$NC_BIN” ]]
then
  echo 'ERROR: cannot find socat binary'
  exit 126
fi

# if we are getting stats:
#   check if we can write to stats cache file, if it exists
#     or cache file path, if it does not exist
#   check if HAPROXY socket is writable
# if we are NOT getting stats:
#   check if we can read the stats cache file
if [ “$GET_STATS” -eq 1 ]
then
  if [ -e “$CACHE_FILEPATH” ] && [ ! -w “$CACHE_FILEPATH” ]
  then
    echo 'ERROR: stats cache file exists, but is not writable'
    exit 126
  elif [ ! -w ${CACHE_FILEPATH%/*} ]
  then
    echo 'ERROR: stats cache file path is not writable'
    exit 126
  fi
  if [[ $QUERYING_METHOD == “SOCKET” && ! -w $HAPROXY_SOCKET ]]
  then
    echo "ERROR: haproxy socket is not writable"
    exit 126
  fi
elif [ ! -r “$CACHE_FILEPATH” ]
then
  echo 'ERROR: cannot read stats cache file'
  exit 126
fi

# index:name:default
MAP="
1:pxname:@
2:svname:@
3:qcur:9999999999
4:qmax:0
5:scur:9999999999
6:smax:0
7:slim:0
8:stot:@
9:bin:9999999999
10:bout:9999999999
11:dreq:9999999999
12:dresp:9999999999
13:ereq:9999999999
14:econ:9999999999
15:eresp:9999999999
16:wretr:9999999999
17:wredis:9999999999
18:status:UNK
19:weight:9999999999
20:act:9999999999
21:bck:9999999999
22:chkfail:9999999999
23:chkdown:9999999999
24:lastchg:9999999999
25:downtime:0
26:qlimit:0
27:pid:@
28:iid:@
29:sid:@
30:throttle:9999999999
31:lbtot:9999999999
32:tracked:9999999999
33:type:9999999999
34:rate:9999999999
35:rate_lim:@
36:rate_max:@
37:check_status:@
38:check_code:@
39:check_duration:9999999999
40:hrsp_1xx:@
41:hrsp_2xx:@
42:hrsp_3xx:@
43:hrsp_4xx:@
44:hrsp_5xx:@
45:hrsp_other:@
46:hanafail:@
47:req_rate:9999999999
48:req_rate_max:@
49:req_tot:9999999999
50:cli_abrt:9999999999
51:srv_abrt:9999999999
52:comp_in:0
53:comp_out:0
54:comp_byp:0
55:comp_rsp:0
56:lastsess:9999999999
57:last_chk:@
58:last_agt:@
59:qtime:0
60:ctime:0
61:rtime:0
62:ttime:0
"

_STAT=$(echo -e "$MAP" | grep :${stat}:)
_INDEX=${_STAT%%:*}
_DEFAULT=${_STAT##*:}

debug "_STAT    => $_STAT"
debug "_INDEX   => $_INDEX"
debug "_DEFAULT => $_DEFAULT"

# check if requested stat is supported
if [ -z “${_STAT}” ]
then
  echo "ERROR: $stat is unsupported"
  exit 127
fi

# method to retrieve data from haproxy stats
# usage:
# query_stats "show stat"
query_stats() {
    if [[ ${QUERYING_METHOD} == “SOCKET” ]]; then
        echo $1 | socat ${HAPROXY_SOCKET} stdio 2>/dev/null
    elif [[ ${QUERYING_METHOD} == “TCP” ]]; then
        echo $1 | nc ${HAPROXY_STATS_IP//:/ } 2>/dev/null
    fi
}

# a generic cache management function, that relies on 'flock'
check_cache() {
  local cache_type="${1}"
  local cache_filepath="${2}"
  local cache_expiration="${3}"  
  local cache_filemtime
  cache_filemtime=$(stat -c '%Y' "${cache_filepath}" 2> /dev/null)
  if [ $((cache_filemtime+60*cache_expiration)) -ge ${CUR_TIMESTAMP} ]
  then
    debug "${cache_type} file found, results are at most ${cache_expiration} minutes stale.."
  elif "${FLOCK_BIN}" –exclusive –wait "${FLOCK_WAIT}" 200
  then
    cache_filemtime=$(stat -c '%Y' "${cache_filepath}" 2> /dev/null)
    if [ $((cache_filemtime+60*cache_expiration)) -ge ${CUR_TIMESTAMP} ]
    then
      debug "${cache_type} file found, results have just been updated by another process.."
    else
      debug "no ${cache_type} file found, querying haproxy"
      query_stats "show ${cache_type}" > "${cache_filepath}"
    fi
  fi 200> "${cache_filepath}${FLOCK_SUFFIX}"
}

# generate stats cache file if needed
get_stats() {
  check_cache 'stat' "${CACHE_STATS_FILEPATH}" ${CACHE_STATS_EXPIRATION}
}

# generate info cache file
## unused at the moment
get_info() {
  check_cache 'info' "${CACHE_INFO_FILEPATH}" ${CACHE_INFO_EXPIRATION}
}

# get requested stat from cache file using INDEX offset defined in MAP
# return default value if stat is ""
get() {
  # $1: pxname/svname
  local _res="$("${FLOCK_BIN}" –shared –wait "${FLOCK_WAIT}" "${CACHE_STATS_FILEPATH}${FLOCK_SUFFIX}" grep $1 "${CACHE_STATS_FILEPATH}")"
  if [ -z “${_res}” ]
  then
    echo "ERROR: bad $pxname/$svname"
    exit 127
  fi
  _res="$(echo $_res | cut -d, -f ${_INDEX})"
  if [ -z “${_res}” ] && [[ “${_DEFAULT}” != “@” ]]
  then
    echo "${_DEFAULT}"  
  else
    echo "${_res}"
  fi
}

# not sure why we'd need to split on backslash
# left commented out as an example to override default get() method
# status() {
#   get "^${pxname},${svnamem}," $stat | cut -d\  -f1
# }

# this allows for overriding default method of getting stats
# name a function by stat name for additional processing, custom returns, etc.
if type get_${stat} >/dev/null 2>&1
then
  debug "found custom query function"
  get_stats && get_${stat}
else
  debug "using default get() method"
  get_stats && get "^${pxname},${svname}," ${stat}
fi


! NB ! Substitute in the script /var/run/haproxy/haproxy.sock with your haproxy socket location

You can download the haproxy_stats.sh here and haproxy_discovery.sh here

2. Create the userparameter_haproxy_backend.conf

[root@haproxy-server1 zabbix_agentd.d]# cat userparameter_haproxy_backend.conf 
#
# Discovery Rule
#

# HAProxy Frontend, Backend and Server Discovery rules
UserParameter=haproxy.list.discovery[*],sudo /etc/zabbix/scripts/haproxy_discovery.sh SERVER
UserParameter=haproxy.stats[*],sudo /etc/zabbix/scripts/haproxy_stats.sh  $2 $3 $4

# support legacy way

UserParameter=haproxy.stat.downtime[*],sudo /etc/zabbix/scripts/haproxy_stats.sh  $2 $3 downtime

UserParameter=haproxy.stat.status[*],sudo /etc/zabbix/scripts/haproxy_stats.sh  $2 $3 status

UserParameter=haproxy.stat.last_chk[*],sudo /etc/zabbix/scripts/haproxy_stats.sh  $2 $3 last_chk

 

3. Create new simple template for the Application backend Monitoring and link it to monitored host

create-configuration-template-backend-monitoring

create-template-backend-monitoring-macros

 

Go to Configuration -> Hosts (find the host) and Link the template to it


4. Restart Zabbix-agent, in while check autodiscovery data is in Zabbix Server

[root@haproxy-server1 ]# systemctl restart zabbix-agent


Check in zabbix the userparameter data arrives, it should not be required to add any Items or Triggers as autodiscovery zabbix feature should automatically create in the server what is required for the data regarding backends to be in.

To view data arrives go to Zabbix config menus:

Configuration -> Hosts -> Hosts: (lookup for the haproxy-server1 hostname)

zabbix-discovery_rules-screenshot

The autodiscovery should have automatically created the following prototypes

zabbix-items-monitoring-prototypes
Now if you look inside Latest Data for the Host you should find some information like:

HAProxy Backend [backend1] (3 Items)
        
HAProxy Server [backend-name_APP/server1]: Connection Response
2022-05-13 14:15:04            History
        
HAProxy Server [backend-name/server2]: Downtime (hh:mm:ss)
2022-05-13 14:13:57    20:30:42        History
        
HAProxy Server [bk_name-APP/server1]: Status
2022-05-13 14:14:25    Up (1)        Graph
        ccnrlb01    HAProxy Backend [bk_CCNR_QA_ZVT] (3 Items)
        
HAProxy Server [bk_name-APP3/server1]: Connection Response
2022-05-13 14:15:05            History
        
HAProxy Server [bk_name-APP3/server1]: Downtime (hh:mm:ss)
2022-05-13 14:14:00    20:55:20        History
        
HAProxy Server [bk_name-APP3/server2]: Status
2022-05-13 14:15:08    Up (1)

To make alerting in case if a backend is down which usually you would like only left thing is to configure an Action to deliver alerts to some email address.

Tags: , , , , , , , , , , , , , ,

No Comments »

« Previous Post

Christ is Risen ! Truly He is Risen ! The origin of the tradition Paschal Greeting and Coloring of Eggs on Easter Holidays in the Church


April 27th, 2022

 

Christ-is-Risen-Truly-he-is-risen-and-the-christian-origin-of-red-eggs-worldwide-Christ-triumphant-icon

Christ is Risen ! Truly He is Risen !

Христос воскресе ! Воистину воскресе ! (Khristos voskrese! Voistinu voskrese!) – Church Slavonic Paschal Greeting

Χριστὸς ἀνέστη!  Ἀληθῶς ἀνέστη ! (Khristós anésti! – Alithós anésti!) – Greek Paschal Greeting

Christus Resurrexit ! Resurrexit Vere ! – Latin Paschal Greeting

The Easter Eggs are so famous today for the kids worldwide, even though the world does not put much accent on the feast of Pascha (Easter). All kind of colored eggs are to be find in stores, many christian countries both Western and Eastern all throughout the world have the tradition of coloring eggs for Easter. 
The tradition is also the same here in Orthodox Bulgaria, as we have the tradition to boil and color eggs in various colors. 
Usually the first egg is colored in Dark Red and once sanctified in the Church is put on the iconostasis (the prayer corner in the house) in front of Christ, Virgin Mary and saints icons and kept their until next year.

Miraculously this Egg usually does not start decaying or smelling as an ordinary egg will do if left out of the fridge for a month or so. This first egg in dedication and memory of Christ's resurrection is kept on the iconostasis until the next year's Pascha and then buried somewhere in a green clean place for sanctification of the land.

This is a good and well followed tradition for those strict about religion, but even those who did not strictly follow Christianity or orthodoxy do color eggs for the fun of kids and as an expression for joy of the Paschal feast. Both grown and kids then try out their forces whose egg is more powerful by knocking each other's eggs to test whose egg shell is more solid and can stand up the break. The egg that is "victorious" once people test their power that is stronger and withstand the "egg fight" is kept for another egg duel with another person.

According to old superstitious belief if you win over in an eggs fight this is interpteted as you will have a good health and well being for the upcoming period till next year's Pascha.
 

How and from wherein this Boiled Eggs coloring originated ?


The short answer is it is connected to one of Church's traditions about the poor apostle Equal Saint Mary ( Maria ) Magdalene, who have given as a gift to Roman emperor Tiberius an Egg with the All Famous Greeting dialog in the Orthodox Churches among people with person A saying:   Christ is Risen ! person B responding: Truly he is Risen! (Христос Воскресе Войстину Воскресе !)

From the time of the many appearances of the Savior Christ in flesh after his Glorious Resurrection described by the Holy Evangelists in the Gospels and the fervent sermon of St. Mary Magdalene (one of the of the so called Myrrh- Bearing Woman who were the first who have visited the tomb where Dead body of Christ was led and become witnesses of  the Resurrection). The surviving Bible's New Testament 4 Gospel books do not provide further details about the activities of St. Mary Magdalene and her life. The Church mouth-by -mouth tradition of her later life in several local Christian churches differ slightlyhowever everywhere they essentially report on the zealous co-apostolic activity of St. Mary Magdalene. And the differences between these traditions depend on which of the evangelical women these churches understand by the name of St. Mary Magdalene.

Some Western Christian churches, as well as the Church Fathers and learned theologians, unite in one or two personalities three evangelical women: the sinner who repented in the house of Simon the Pharisee, shed tears at the Savior Christ's feet, wiped them with her hair. and she anointed them with precious ointment, and Mary of Bithynia, sister of Lazarus of Bythynia ( resurrected in the fourth day after death by Chrsit and commemorated one day before Palm Sunday )' and Mary Magdalene, who was delivered from the Savior Christ by seven demons. But the Orthodox Church now, as before, recognizes those mentioned in the Gospels with different signs, three persons as different separate ones, and does not want to base historical information on arbitrary, plausible interpretations. Therefore, the tradition of the Orthodox Church states that after the Gospel appearances of the Risen Christ before His Ascension and after, St. Mary Magdalene resided with the Blessed Virgin and the Apostles and was an active helper of the first successes of spreading the Christian faith first in Jerusalem. But full of zeal, fervent faith, and zealous love for God's gospel, she then preached in other lands, proclaiming everywhere the heavenly grace, joy, and salvation of all who believed in the Savior of the world, the Risen Christ.

Saint-Mary-Magdalene-gifting-red-egg-to-emperor-Tiberius-Orthodox-icon-one-of-Myrrh-Bearing-Woman

While visiting Italy to preach, St. Mary Magdalene found an opportunity to appear before the then-reigning Emperor Tiberius I, and presented him, according to generally accepted Eastern custom, with an egg painted red and greated him with "Christ is risen!"

The modesty of the gift of Mary Magdalene did not surprise the emperor, because he knew the ancient custom of the East, also among the Jews, going for the first time to superiors, or on solemn occasions to acquaintances or patrons, to offer a gift of honor, with some known or special, symbolic meaning. Examples of this can be found in Jewish Old Testament history, as are the gifts presented by the rich Wise men (magis – today their relics are kept for veneration in Cathedral of Cologne Germany) to the born Jesus Christ in Bethlehem of Judea. Even the poor in such cases offered as a gift various fruits from their locality or eggs from birds. Thus, partly following this ancient custom and with the red color of the egg laid and with the hitherto unheard words "Christ is risen!" to arouse the curiosity of the suspicious Emperor Tiberius. The holy co-apostle Mary Magdalene, by explaining the significance of this gift, began her fervent sermon on the Resurrection truth and the teachings of the Savior Christ for salvation. With great inspiration and conviction she told the emperor about the life, miracles, crucifixion and resurrection of Jesus Christ according to his own prophecy. She gave a direct, simple-minded account of the extremely unjust, biased judgment of Jesus Christ by the embittered members of the Jerusalem Sanhedrin. governor of Judea Pilate of Pontus, in condemning Jesus Christ to crucifixion. She explained how all this incurred the wrath of the Roman emperor then and how Tiberius handed them over to a court in which Pilate was deprived of power and exiled to Gaul, in the city of Vienna, where, according to legend, tormented by remorse and despair, he killed himself. According to another legend, Pilate repented, turned to Christ in prayer, as a sign of which his head was accepted by an angel after being cut off.

According to Church tradition, the sisters of Lazarus Martha and Mary went to Italy with St. Mary Magdalene; and Pilate, learning of this and fearing the denunciation of his unlawful actions by the Christians, himself sent a message to the Emperor Tiberius about Jesus Christ, in which he testified to the virtuous life of Christ, the healing of all diseases and infirmities from Him, even for the resurrection of the dead and for His other great miracles. Pilate asserted that in examining the accusations of the Jews, he found no fault in Jesus Christ; he made great efforts to deliver Him from the hands of the troubled Jews, but failed to deliver Him and betrayed Jesus to their will because of the cries of the people and the rebellious accusation of the Jews against Pilate himself …

 

... as a witness, overwhelmed with fear, he told the emperor about everything that had happened to Jesus Christ, who became an object of faith as God …

After such testimonies from the Roman governor of Judea and the worshipers of the Savior Christ, Emperor Tiberius, according to legend, himself believed in the Savior Christ, proposed to include Jesus Christ in the image of the Roman gods, and even when the Roman Senate rejected this proposal, Tiberius by royal decree threatened to punish anyone who dared to grieve believers in Jesus Christ.

In this way, with the zealous, fearless sermon on the Savior Christ, St. Mary Magdalene, along with other devout Christians, persuaded the pagan governor of Judea to testify in writing about the universal event of Christ's Resurrection before the pagan world and persuaded the then Roman emperor of the Savior Christ, thus facilitating the spread of Christianity.

Anastasis-Hristos-Voskrese-beautiful-orthodox-Mosaic
Anastasis (Resurrection) Church Fresco


And the Christians of that time, learning about the significance and strength of the impression caused by the offering of a red egg by Mary Magdalene to Emperor Tiberius with the words: "Christ is risen!" then began to imitate her in this and as a remembrance of Christ's Resurrection they began to give each other red eggs and say: "Christ is risen! … He is risen indeed! …"

Thus, this custom gradually spread everywhere and became universal for Christians around the world. In it, the egg serves as a symbol of Christ's resurrection and the resurrection of the dead, and of our expected new-birth for eternal bliss in the future life, the pledge for which is Christ's Resurrection.

Just as a bird is born from an egg and begins to live an independent life after its release from the shell, and the vast circle of life is revealed to it, so we, at the second coming of Christ to earth , rejected from ourselves together with the earthly body all that is mortal on earth.

By the power of Christ's Resurrection we will be resurrected and resurrected to another, higher, eternal, immortal life.

And the red color of the Easter egg reminds us that the redemption of mankind and our future new life have been acquired through the shedding of the cross on the pure blood of the Savior Christ.

Thus, the red egg serves to remind us of one of the most important dogmas of the Divine revealed Christian faith.

 

After the crucifixion of Jesus by the Jews in terrible miracles took place in nature, many dead righteous people rose, with His resurrection on the third day.Pilate, as a witness overwhelmed with great fear, informed the Caesar of all things that had happened to Jesus Christ.

In Eastern Orthodox Tradition Christ is Risen ! Truly he is Risen Greeting is used to joyfully great each other all around the Orthodox countries in the first 3 days of easter, and can be used instead of normal Hello greeting ! for the upcoming week The Holy Easter Weak which is a week of great joy and even by a hello greating in the Church could be used for 40 days as a normal greeting.

It is worthy to close this article with the praisal words, read on the first day of Pascha  authored by one of the most important Church fathers and
compiler of most served Liturgy service throughout the yearly service calendar:

"Christ is risen, and you are overthrown!
Christ is risen, and the demons are fallen!
Christ is risen, and the angels rejoice!
Christ is risen, and life reigns!
Christ is risen, and not one dead remains in a tomb!
For Christ, being raised from the dead, has become the first-fruits of them that have slept."

Saint John Chrysostom

Tags: , , , , , , , , , , , , , , , , ,

No Comments »

« Previous Post

The Holy and Great Week of Passions of Christ in the Church – Day by day explained


April 19th, 2022

Christ-the-bridegroom-orthodox-passion-of-Christ-week

The Holy Great Week of Christ Passions

The last week of the earthly life of the Lord Jesus Christ is called the "Great" or "Passion Week", i.e. A week of suffering, a prelude to eternal life. The Lord's life was coming to an end. Having resurrected Lazarus on the Sabbath as a proof of the Mass coming Resurrection known in the Church as Lazarus Saturday as it is always celebrated Saturday in the Orthodox Church on which people gathered to solemnly welcome the Messiah Christ, and triumphantly entered Jerusalem on Palm Sunday. Following that the Savior Jesus Christ who prophecised his betrayel to the Cross for human sin, the Lord voluntarily walked step by step to His predestined inevitability.

Every day of the Passion Week is called Great and Holy for the reason this week is the most Holy and Sanctified week of the whole Calendar Church year. Each of the Seven days of it, the Church commemorates events of last week of Christ's life and suffering on earth before Resurrection and Ascension to Heaven through special services the way of Christ to Golgotha, the sufferings and His redemptive work on the Cross.

Worship during Holy Week

Lent services on the weekdays of Lent are characterized by their penitential singing. The royal doors (of the alter known as Dveri) remain closed as a symbol of man's separation from the Kingdom of God. Church vestments are dark, usually purple in the color of repentance.

Bulgariand-Church-Kings-doors-Carski_dveri_-_Sv._Spas_(Rashtak)_in_North_Macedonia

No Divine Liturgy is performed on weekdays, but so that the (ordinary chrsitians who go often and pray God) – so called faithful can support themselves in their ascetic effort of fasting by accepting Holy Communion, a Liturgy of the Presanctified Gifts is performed (a specific Liturgy prepared for the Purpose that is only served during great Lent). This service is very ancient, it is mentioned in the canons of the VII century, but it was established earlier for sure. Most likely the Liturgy of the Presanctified Gifts, practice to sanctify bread and wine in prior has later evolved in the Roman-Catholic Churchs errenous from Eastern Orthodox point of view – Eucharistic Adoration
– (a consacration host kept usually in the so called (monstrace). Traditionally, Presanctified Liturgy creator is considered to be Pope St. Gregory I the (Dialogus), Pope who governed the Western Church in (VI century) – some  theologians today claims it was developed at least partially or coauthored also by Saint Ambrose of Mediolan (Milan).

The pre-consecrated liturgy consists of a solemn Lenten Vespers (prelonged repentance songs) with elements from Psalms and readings from Holy Scripts regarding life and suffering of Christ, to which is added the part of "transfer" of the Holy Gifts from the Alter to the Upper place (the place where the proskomidia occurs) and walked in on the "Great Entrance" Liturgy part with the Sacraments placed in the Holy Chalice held by the priest in front of iconstansisa and back to the Alter of Sacrifice, however the consecration of the Gifts itself is not performed, the Eucharistic gifts are already sanctified and prepared on previous Sunday Saint Basil or Saint John Chrysostomos liturgy.  That is why the service is called the Presanctified Liturgy, i.e. of the pre-consecrated Gifts.

Usually This service takes place on Wednesdays and Fridays or at least on one of these days and on the 6th week of Maria of Egypt is served 3 times instead of 2 throughout the week to venerate the Most Holy Mother Mary of Egypt which from a Harlot turned a saint by immerse repentance, and cause of that become the patron saint for repentance and example for true repentance, that each and every Christian aims follow, every day of his life.

Following the 6th weeks of Fasting a period that the ancient Church placed for try out of ones self soul state and cleanance of passions comes saint Lazarus Saturday.

Lazarus Saturday is the only day of the year when Sunday service worship is integrated in Saturday.  Usually Sunday service is a service of higher importance than the other ones, a faithful gathering to share the unspeakable joy for the Resurrection of Christ and his triumph of Life over Death. 

Lazarus Saturday is the beginning of the Easter celebration. During the Liturgy of Lazarus, the Church glorifies Christ as "Resurrection and Life", who even before His sufferings and death, with the resurrection of Lazarus, confirmed the foreshadowing of the universal resurrection of mankind coming. It was because of the resurrection of Lazarus that Christ was glorified by the people as the long-awaited Messiah (no man ever was able to rise up a death rotting person from the Death after four days in grave) truly identifying him as the promised King of Israel and the fulfillment of long ages awaited Old Testament prophecies.

The very feast of the triumphal entrance of the Lord into Jerusalem (Palm Sunday) belongs to the twelve most importance Church feasts, known in the Church as "Feasts of the Lord". Christ immediate worship by all Jews on his entrance in Jerusalem  is directly connected with that of Lazarus Day on which he did the miracle of commanding Lazarus to wake up from Death,  returning life of a long dead Lazarus.

On the eve of the feast, the prophecies about the Messianic King from the Old Testament are read, along with the Gospel accounts of Christ's entry into Jerusalem, as another confirmation that Christ is the True Messiah.

In the morning, the willow twigs we hold in our hands throughout the Liturgy are blessed, thus showing that we welcome Jesus Christ as King and Savior, just like the Jews has received him in Jerusalem 21 centuries again in  year 0 A.D.

Extract Prot. Thomas HOPCO "Fundamentals of Orthodoxy" with short modifications from:
Church NewsPaper of Bulgarian Orthodox Church, Issue 7 of April 17, 1998

Holy Monday, Holy Tuesday and Holy Wednesday

se-jenih-griadet-v-polunoshti-molitva-here-is-the-bridgeroom-comes-church-slavonic-prayer

Church Slavonic (Old Bulgarian) notable singing during the first 3 days of the Holy Week sung in the Orthodox Church

Text translates as:

Behold, the Bridegroom comes at midnight,

And blessed is that servant whom He shall find watching,

And again, unworthy is the servant whom He shall find heedless.

Beware, therefore, O my soul, do not be weighed down with sleep,

Lest you be given up to death, and lest you be shut out of the Kingdom.

But rouse yourself crying: Holy, Holy, Holy, art Thou, O our God,

Through the Theotokos have mercy on us.

Troparion of Bridegroom Matins

During the first three days of Holy Week, the Church commemorates the Lord's last stay in Jerusalem. In these days the worship is very intense: there is a Midnight Office (Μεσονύκτικον, Mesonýktikon; Slavonic: Полунощница), The Hours matins, Psalms Book chapters, reading of the Gospel and Liturgy of the Presanctified Gifts. During the "lessons" given by, the four Gospels to the Gospel of John are read. 13, verses 30. 
 

Great and Holy Monday

On Holy Monday, the evangelists tell us how the Son of God entered the Jerusalem temple and found it full of merchants. Overwhelmed with holy wrath, He overthrew their tables and drove them out, because the temple is a house of prayer, not a marketplace. (Matt. 21: 12-13, Mark 11: 15-19; Luke 19: 45-46).

Great-and-holy-Monday-the-parable-of-the-fig-tree-icon

On Holy Monday, the Church celebrates St. Patriarch Joseph, the son of St. James the Patriarch and a type of Jesus Christ.

Saint_Joseph-the-Patriarch-sold-in-Egypt-by-his-brothers-a-prototype-for-Christ
Joseph The Magnificient

Joseph was sold by his brothers to merchants traveling to Egypt.

Joseph-the-brilliant-as-Second-after-Pharaoh-in-Egypt-the-all-comely-icon

There, in a foreign land, he went through many sufferings, but Pharaoh made him second in power and position in the whole kingdom (Gen. 41: 38-46). Like Joseph, the Lord Jesus Christ was betrayed by the Jews to the Gentiles, tortured, and suffered for human sins.

Great-and-Holy-Monday-Christ-extreme-humility-icon

The Icon of Christ the Bridegroom (Ο Νυμφίος)

The Church also invites us to reflect on the image of the barren fig tree, which withered after being cursed by the Lord (Mark 11: 12-14, 20-26, Matt. 21: 18-22). "Every tree that bringeth not forth good fruit is hewn down, and cast into the fire" (Matt. 3:10).

Great-and-holy-Monday-the-parable-of-the-fig-tree-icon
 

In the same way, we will be condemned if we do not live in prayerful communion with God, do not strive to improve our faith, do not fill ourselves with virtues, and do not bear spiritual fruit.

Great and Holy Tuesday

"Watch therefore: for ye know neither the day nor the hour wherein the Son of man cometh" (Matt. 25:13).

(Gospel reading: Matins 22: 15-23: 39; Liturgy Mat. 24: 36-26: 2).

Holy Tuesday is a day for teachings and final moral instructions:

The Lord Jesus Christ gives us an example of how to do good – not to give from our surplus for this purpose, but as a poor widow to set aside from our last material means.

Great-Holy-Tuesday-the-10-virgins-parable-orthodox-christian-icon

Speaking of the approaching days of struggle and trial, Christ tells of the ten wise virgins who were always ready to meet the Savior (Matt. 25: 1-13). It reminds us that we must "be vigilant and not be discouraged" and keep our lamps lit in anticipation of the Divine Bridegroom.

That is why on Holy Tuesday the Church sings:

Here comes the bridegroom at midnight,
and blessed is that servant whom he hath found awake,
and unworthy is he whom he finds careless.

Therefore beware, my soul, lest you sleep,
to be delivered to death and to remain outside the closed doors of the Kingdom,
but come to your senses and exclaim: Holy, holy, holy, O God,
have mercy on us for the sake of the Mother of God!

"The light of the body is the eye" (Matt. 6:22), says the Lord. the unsullied human heart and soul, and "the oil is alms or all our good deeds" (St. John Chrysostom).

Living virtuously, with the fear of God and trust in the Lord, we will be ready to meet the Savior and enter the marriage hall – the Kingdom of Heaven.

The church also reminds us of the parable of the talents (Matt. 25: 14-30) and invites us to work hard and improve the abilities God has given us.

Then follow prophecies about the fate of the city of Jerusalem for the last days of the Second Coming of the Lord
(Matt. 25: 31-46, Mark 13: 1-31, Luke 21: 5-38).

Great Holy Wednesday

On the day of Holy and Holy Wednesday we remember one of the last events before the Lord's saving sufferings for us: the precious ointment, which in his sincere repentance a sinner woman poured on the Savior's head (Matt. 26: 6-13, Mark 14: 3-9).

She managed to enter the house where Christ was, the woman carrying an alabaster vessel with precious very expensive ointment, she wanted to pay her enormous respects to Him. In a hurry (scared that someone from the people in surrounding Christ might stop here) in order for not to interfere with her good intentions, she broke the vessel that was helding a high amount of oilment, making it easier to spill the ointment on Christ.

Great-wednesday-the-sinful-harlot-woman-with-oilments-cleaning-up-feets-of-Christ-with-her-hair

The precious ointment cost three hundred dinars ! (Mark 14: 5), so some being sick of the passion of Judah the Iscariot (The Love for Money the works of the Flesh) resented it: "Why is this waste?", "The ointment could be sold and the money given to the poor."
And Christ answered them, "You always have the poor with you, but you do not always have Me," "she has done a good work for Me [by] deceiving to anoint My body for burial." Her zeal will be heard all over the world.
Like the prodigal son, the sinner realized her sins and "came to her senses."

Let us also come to our senses about our real spiritual condition and repent of our sins, so that with our repentant tears we may "anoint" the Lord like that repentant woman !

Jude-Betrays-Christ-selling-him-for-30-silver-coins

On the same day, we recall the decision of the Sanhedrin to condemn Jesus Christ. Then Judas Iscariot went to the Jewish leaders and agreed to hand him over for thirty pieces of silver (Matt. 26: 14-16, Mark 14: 10-11, Luke 22: 1-6).

We should well think:

Do we, who bear the name of Christ, not betray Christ through our ungodly deeds?

From that day on, the kneeling prayers do not cease, as one should understand we have done plenty of badness and has inflicted additional pains to Christ, who suffered for all great sins on the Cross.
 

Great Wednesday

Great and Holy Thursday – Remembrance of the Last Supper

On that day, the Lord Jesus Christ celebrated the Passover in the home of a resident of Jerusalem
(Matt. 26: 17-35, Mark 14: 12-31, Luke 22: 7-38, John 13: 1-17, 26).

Before supper He washed the feet of the apostles and said, "I did not come to serve, but to serve."
The Savior then instituted the sacrament of the Eucharist (Communion) by Himself partaking of the holy apostles.

By His great mercy, the Lord also gives us the opportunity to receive His true body and blood during the Holy Liturgy, so that by accepting Christ within us, we may strive to keep Him through the purity of our hearts.

The-Secret-Supper-Tajna-vecherya-Aton-Manuil-Panselinos-Protata

After bequeathing the new commandment to love all, Christ revealed to His disciples that He would be betrayed.
Bewildered, the students asked who would do this.

The-Betrayal-of-Juda-Orthodox-icon-heprodosia
Judas asked is it him that will betray ? 
Christ answered him so meekly that the others did not understand.
Judas got up, went out leaving the holy eucharistic supper.
And pupils, thought he was going shopping for required goods for the brothers because he was a treasurer (an accent how we should keep a good mind and try to think well about others all the time).

Next  great accent is Lord's Prayer.

The-Prayer-of-Christ-in-Gethsemane-garden-Great-Tarnovo-Museum-MOLENIE-Muzej_V-Turnovo

Christ Prayer in Gethsemane Garden – Bulgarian Icon museum Great Tarnovo

In the Garden of Gethsemane After supper Christ and the apostles went to the Garden of Gethsemane (Matt. 26: 36-46, Luke 22: 39-46, John 18: 1), where he prayed until the coming of the traitor.

Mockery-of-Christ-icon

Usually on Thursday evening the morning of Good Friday service is served, when the so-called Twelve Gospels are read, ie. the twelve passages of the Gospel that tell of Christ's sufferings.


Through them we witness the mockery, suffering, and crucifixion of Christ, through which He redeemed us.
"Here is the Lamb of God who took away our sins."

And again we wonder if we do not crucify Christ through our passions and sins.

Jesus_in_Golgotha_by_Theophanes_the_Cretan-orthodox-icon-fifteen-century

Jesus in Golgotha – Theophanes the Cretan

On this day, the priests take the Cross out of the altar, which symbolizes its carrying from Christ to Golgotha.

The Great annointing of the sick service is served so called "Велик Маслосвет" – during whose many prayers to saints healers are red to intercede for us following by 7 Act of Apostle readings and 7 Gospel Chapter Readings, wherever possible in large Cathedral Churches, this is served by 7 priests  every willing layman is anointed with oil 7 times after reading each of the 7 Gospels for restorating of Health of the sick as well as a special blessing in the manner of the ancient Church tradition.

Great and Holy Friday

The Way of the Cross and Golgotha ​​We remember the great sufferings of Jesus Christ, who freely agreed to be judged, flogged, spat upon, beaten with slaps, and shown before the people in a purple robe, with a cross in his hand and a crown of thorns on his head. 

Armed with a heavy cross from Pilate's praetorium (judgement place), Christ was led to Golgotha ​​on the crucifixion.

The-Crucifix-of-Christ-Razpiatie-Hristovo

Crucified between two robbers for desecration in terrible natural disturbances – an earthquake and an eclipse of the sun, he died, accepted death to save all mankind from death.

On this day, every Christian should follow complete fasting (eat nothing and drink nothing) and pray and sorrow deeply for the Lord.
According to church rules, even the sick should only eat bread (at best a very dry one) and drink a little bit of water. Joys of any kind of type should be abstained and all passions avoived and one should ask God for mercy for himself, his family and ask is merceful to everyone.

Great and Holy Saturday

The burial of Christ the Savior and His descent into hell are commemorated.
He died on the cross, blood and water flowed from His pierced ribs.
Joseph of Arimathea and Nicodemus, asking Pilate for permission, removed Him from the cross, anointed Him with perfume, wrapped Him in a new shroud, and laid Him in a new tomb carved into a rock in the Garden of Gethsemane.

The_Burial_Lamentations_by_Theophanes_the_Cretan-Stavronikita-monastery-mount-athos-wall-painting

Epitaphios (Lamentation of Christ) from Stavronikita monastery, Mount Athos – Theophanes the Cretan

Holy-Saturday-The-Resurrected-Christ-Empty-tomb-grave-icon

Myrrh-bearing women were present at His burial in the tomb, among whom, in tears with her grief-stricken heart, was His Mother the Holy Mother of God.

The church sings regarding this great events:

"In the grave with his body and in hell with his soul as God,
in heaven with the thief and on the throne with the Father and the Spirit You were, Christ,
Who fills everything. "

The Jews sealed the tomb and set up a guard.

Great secret! "Let the human creature silence !" – sings the Church instead of the Cherubim song on Holy Saturday.
The lord of life is in the grave, but he will soon be famous for the miracle of the resurrection.

On the Saturday morning after the liturgy, in some places it is customary for the priest to give flowers to the faithful as an expression of joyful anticipation of the Resurrection.

Holy Week in the statutes of the ancient churches

Initially, Easter was preceded by a two-three-day fast, which took place one week – the so-called.
Passion Week, or the Week of Christ's Suffering.

Subsequently, the 40-day fast was added to Lent, similar to the forty days during which Christ fasted in the wilderness. It was intended for the "announced", that is, for those who would be baptized on Easter.

For a long time during the practice of mass baptisms of the elderly, the sacrament was performed on Easter, when baptism was especially experienced as a participation in the voluntary death and resurrection of the Lord.
That is why the Easter Liturgy is extremely baptismal in nature.

After the sixth century, the baptism of children began to predominate, so the mass baptism of adults on Easter was gradually abandoned.
It was then that the meaning of Pentecost was changed – from a catechetical period, fasting became a period of repentance for members of the Church.

In the ninth century, Pentecost the word stems from the Greek Πεντηκοστή (Pentēkostē) meaning "fiftieth" was finally united with Holy Week, and so the duration of Lent increased.

The length of Lent varied, depending on how local churches viewed the inclusion of Holy Week at Pentecost and whether they considered Saturdays and Sundays, when canons forbid fasting, to be part of it.

In the Constantinople Statutes (followed by our Bulgarian Orthodox Church), Holy Week is not considered part of Pentecost, and Saturdays and Sundays are included in the Lent period, although they are not Lent days in the full sense of the word.

Thus, according to the Constantinople Statute, The Pentecost Lent had 6 weeks of 7 days, ie 42 days.
If Lazarus Saturday and Palm Sunday are excluded from it, the duration of Lent is exactly 40 days.

According to this statute, Lent begins on Maundy Monday from the first week of Lent and ends on Friday of the sixth week, that is, on the eve of Palm Sunday.

The troparions included in the Triodion (Постен Триод – The Church Service book with sung text used during the Lent, for this day speak of the "fulfillment of the soul-beneficial fortieth Pentecost" and the anticipation of the "holy week of the Passion."

The interpretation of the rule in the Apostolic Decrees (Church rules guidance book text from the end of the IV century) is similar, where it says:

"Perform this fast before Easter, beginning on the second day (that is, Monday) and ending on Friday. After these days, as completing the fasting, begin the holy week of Easter by fasting through it with fear and trembling."

It is no coincidence that the liturgies of Lazarus Sabbath and the Lord's Entrance into Jerusalem have baptismal elements.

According to another tradition, reflected in the 29th canon of the Sixth Ecumenical Council (681) – that is the year of Creation of Today's country of Bulgaria (which is the only country in Europe that did not change his name as of year 681), Holy Week was part of Pentecost, where it is called "the last week of Pentecost".

This other practice is preserved by the ancient churches, which separated from Orthodoxy after the Fourth Ecumenical Council in Chalcedon (451) – The Armenian, Coptic, Syriac Orthodox Church of Antioch, Ethiopian Church of Toledo, (perhaps the Jacobite Syrian Church) etc.

Even though this historic tradition was well preserved in those Churches and many of their church order or customs such as veneration for the icons, holy relics, the problem with them preventing them to be in  ull communion with Eastern Orthodox Church stems in their rejection to accept the V-th XI-th and XII Ecumenical Ecumenical Councils and their perseverance on monophysitism (literally translated as, one nature – a teaching that says Christ has only one Nature and one Will a Godly, they say they do not reject that Christ was also real man in flesh but they consider the Godly nature of Christ has consumed the manly, which makes up their wrong understanding that Christ on the Cross did not fully suffer with his manly nature, but both God and man has suffered on the Cross – a doctrine which according to the Church councils is a pure hearesy, we can also conclude by the one nature of Christ that the so called today Oriental Orthodox Churches teach, that Christ on the Cross did not bear all the sins of the world as a man but he received all the sins and turmoils and evils as God.

In contrast in Eastern Orthodox Churches we do consider the truth that Christ has two Natures manly and Godly as well as Two Wills.
Some of the upmentioned ancient Oriental Orthodox Churches keep up to the heresy of monothelitism and that is why they're not communion with us the Eastern Orthodox.

The two wills in Orthodoxy is known under the term dyothelitism or dythelitism (stems from Greek δυοθελητισμός "doctrine of two wills") is a particular Christological doctrine that teaches the existence of two wills (divine and human) in the person of Jesus Christ.
Specifically, dyothelitism correlates the distinctiveness of two wills with the existence of two specific natures (divine and human) in the person of Jesus Christ (dyophysitism).

The Catechism of the One Holy Orthodox Church is stated: "Similarly, at the Sixth ecumenical council, Constantinople III in 681, the Church confessed that Christ possesses two wills and two natural operations, divine and human. They are not opposed to each other, but co-operate in such a way that the Word made flesh willed humanly in obedience to his Father all that he had decided divinely with the Father and the Holy Spirit for our salvation. Christ's human will 'does not resist or oppose but rather submits to his divine and almighty will.'"

This position is in opposition to the Monothelitism position in the Christological debates. The debate concerning the Monothelite churches and the Catholic Church came to a conclusion at the Third Council of Constantinople in 681. The Council declared that in line with the declarations of the Council of Chalcedon in 451, which declared two natures in the one person of Jesus Christ, there are equally two "wills" or "modes of operation" in the one person of Jesus Christ as well.

Dyothelitism was championed by Maximus the Confessor against monothelitism, the doctrine of one will. 

According to their tradition, Saturdays and Sundays as "non-fasting days" are not included in the calculation of Pentecost, so these churches fast 8 weeks for 5 days, ie 40, but fasting for pre-Chalcedonians begins one week earlier (when we have The week where orthodox stop eating Milk and Diary – Сиропустна Неделя (Milk-quit Sunday).

According to some liturgists, the appearance of the preparatory "Milk-quit" week before the beginning of Lent is the result of the desire to combine the two traditions in the Church.

Important clarification to make here is we have different view from  upmention Ancient considered schismatic Churches. Cause these ones only accept Church father decision in ecumenical councils until the 4th and cause they reject authencity of the IV th, XI th and  XII th ecumenical councils and consider Christ has only one nature a Godly one, they don't reject the existence of Human nature completely, however they stand for that Godly nature of Christ completely succumbs the human one and therefore it turns out Christ suffered on the Cross only as God (that Eastern Orthodox Churches consider as heresy).

Our believe of the Eastern Orthodox Church  Jesus Christ has two natures and two wills a Manly and Godly and his desire to humilate the Will of the Father and the Holy Spiritut to fulfill the salvational plan was voluntery.

The Roman Catholic Church since ancient times, has included Holy Week of Pentecost. However, through several councils, she lifted the ban on fasting on the Sabbath (64 Apostolic Rule). Unfortunately fasting today in Western Roman Catholic Churches is trongly reduced and all in all officially the layman in that Church has to fast about 4 days in the whole year, where in practice most people usually fast only one day on the Good Friday.

This practice is sharply condemned in the 55th canon of the Sixth Ecumenical Council. That is why the Roman Catholic Church calculates Lent as follows: 6 weeks of 6 days of fasting makes 36 days. To them are added 4.
Therefore for Catholics, the Great Lent begins on Wednesday, the so-called. Clean Wednesday (which according to Church tradition is the day on which Judah decided to betray Christ promising the Sanhedrin to sell them Christ for 30 silver coins … )

What is the reason for Holy Week Fasting

In our Eastern Orthodox Church on Holy and Great Friday, is a very holy and sad day – considered the saddest day in the year, because we sorrow for the great unrighhtousness done to King and The master of Light and Universe and Son of God Christ, being betrayed, joked and beaten in a substitute for us (as we in reality deserve this disgraceful faith for our multitude of transgressions).

Therefore the Goal of following the whole 7 days of Passion week in a Steady fasting is to cleanse up the soul and body, increase our talents (the virtues), prepare to receive Christ in His Glorious Resurrection in our Souls through the Mystery of the Mysterious – the Holy Communion and most importantly win over our sinful passion's rooted in hatred,lust, gluttony, greed, sloth, wrath, envy, pride and all evil and most importantly commune with God with constant prayer and spiritual labors.

The constant prayer is attained in church laymans differently by reading of morning, evening private rules, canons, attendance of the many, many morning and evening services.
What is unique is the church services are constructed in a way that the morning services are served in the evenings where possible after Sunrise about 19:00 o'clock, and evening services are
served in the mornings together with the Hours and on Fridays united with a Liturgy of the Presanctified gifts.

In monasteries especially in Holy Mount Athos and some of the more ascetic ones, the frequent custom is often to use with a blessing of their elder the constant repetition in one self of the so called "Jesus Prayer";

Lord Jesus Christ have mercy on me the sinner!  Lord Jesus Christ have mercy on me the sinner! Lord Jesus Christ have mercy on me the sinner! 

The weapons of the spiritual war used are abstinence of food or at least reducing the food intake and more importantly, reduce the passions. The most important fasting of course is the spiritual.

But for the spiritual advancement a good leverage shown by the Holy Fathers is the Fleshly fasting given to be followed during this week.
Fasting according to church canons for this week, includes only eating if heath allows it of raw foods, vegetables and fruits, bread and plant foods without oil, the local custom not mandatory tradition in the Bulgarian Orthodox Church is to also not eat fat containing nuts, throughout the week with exceptions on Great Thursday the day of The Last Support, where oil is allowed because of the Greatness of the Feast.

The fast during Holy Week is especially strict – "without wine and oil", ie dry foods, as only on Holy Thursday, after Holy Communion, believers used for the spiritual holiday "oil", ie vegetable oil.
Holy Sabbath was treated with special care, as it was the only Sabbath that the canons decreed as a fast day.

Fasting on Holy Saturday lasts until midnight, until the Lord's Day, when the Lord's Resurrection is announced.
The Apostolic Decrees stipulate: "The Sabbath lasts until the roosters sing, the fast ends with the coming of the first day after the Sabbath, which is the Resurrection."

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

2 Comments »

« Previous Post

Webserver farm behind Load Balancer Proxy or how to preserve incoming internet IP to local net IP Apache webservers by adding additional haproxy header with remoteip


April 18th, 2022

logo-haproxy-apache-remoteip-configure-and-check-to-have-logged-real-ip-address-inside-apache-forwarded-from-load-balancer

Having a Proxy server for Load Balancing is a common solutions to assure High Availability of Web Application service behind a proxy.
You can have for example 1 Apache HTTPD webservers serving traffic Actively on one Location (i.e. one city or Country) and 3 configured in the F5 LB or haproxy to silently keep up and wait for incoming connections as an (Active Failure) Backup solution. 

Lets say the Webservers usually are set to have local class C IPs as 192.168.0.XXX or 10.10.10.XXX and living in isolated DMZed well firewalled LAN network and Haproxy is configured to receive traffic via a Internet IP 109.104.212.13 address and send the traffic in mode tcp via a NATTed connection (e.g. due to the network address translation the source IP of the incoming connections from Intenet clients appears as the NATTed IP 192.168.1.50.

The result is that all incoming connections from haproxy -> webservers will be logged in Webservers /var/log/apache2/access.log wrongly as incoming from source IP: 192.168.1.50, meaning all the information on the source Internet Real IP gets lost.

load-balancer-high-availailibility-haproxy-apache
 

How to pass Real (Internet) Source IPs from Haproxy "mode tcp" to Local LAN Webservers  ?
 

Usually the normal way to work around this with Apache Reverse Proxies configured is to use HTTP_X_FORWARDED_FOR variable in haproxy when using HTTP traffic application that is proxied (.e.g haproxy.cfg has mode http configured), you have to add to listen listener_name directive or frontend Frontend_of_proxy

option forwardfor
option http-server-close

However unfortunately, IP Header preservation with X_FORWADED_FOR  HTTP-Header is not possible when haproxy is configured to forward traffic using mode tcp.

Thus when you're forced to use mode tcp to completely pass any traffic incoming to Haproxy from itself to End side, the solution is to
 

  • Use mod_remoteip infamous module that is part of standard Apache installs both on apache2 installed from (.deb) package  or httpd rpm (on redhats / centos).

 

1. Configure Haproxies to send received connects as send-proxy traffic

 

The idea is very simple all the received requests from outside clients to Haproxy are to be send via the haproxy to the webserver in a PROXY protocol string, this is done via send-proxy

             send-proxy  – send a PROXY protocol string

Rawly my current /etc/haproxy/haproxy.cfg looks like this:
 

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        maxconn 99999
        nbproc          1
        nbthread 2
        cpu-map         1 0
        cpu-map         2 1


defaults
        log     global
       mode    tcp


        timeout connect 5000
        timeout connect 30s
        timeout server 10s

    timeout queue 5s
    timeout tunnel 2m
    timeout client-fin 1s
    timeout server-fin 1s

                option forwardfor

    retries                 15

 

 

frontend http-in
                mode tcp

                option tcplog
        log global

                option logasap
                option forwardfor
                bind 109.104.212.130:80
    fullconn 20000
default_backend http-websrv
backend http-websrv
        balance source
                maxconn 3000

stick match src
    stick-table type ip size 200k expire 30m
        stick on src


        server ha1server-1 192.168.0.205:80 check send-proxy weight 254 backup
        server ha1server-2 192.168.1.15:80 check send-proxy weight 255
        server ha1server-3 192.168.2.30:80 check send-proxy weight 252 backup
        server ha1server-4 192.168.1.198:80 check send-proxy weight 253 backup
                server ha1server-5 192.168.0.1:80 maxconn 3000 check send-proxy weight 251 backup

 

 

frontend https-in
                mode tcp

                option tcplog
                log global

                option logasap
                option forwardfor
        maxconn 99999
           bind 109.104.212.130:443
        default_backend https-websrv
                backend https-websrv
        balance source
                maxconn 3000
        stick on src
    stick-table type ip size 200k expire 30m


                server ha1server-1 192.168.0.205:443 maxconn 8000 check send-proxy weight 254 backup
                server ha1server-2 192.168.1.15:443 maxconn 10000 check send-proxy weight 255
        server ha1server-3 192.168.2.30:443 maxconn 8000 check send-proxy weight 252 backup
        server ha1server-4 192.168.1.198:443 maxconn 10000 check send-proxy weight 253 backup
                server ha1server-5 192.168.0.1:443 maxconn 3000 check send-proxy weight 251 backup

listen stats
    mode http
    option httplog
    option http-server-close
    maxconn 10
    stats enable
    stats show-legends
    stats refresh 5s
    stats realm Haproxy\ Statistics
    stats admin if TRUE

 

After preparing your haproxy.cfg and reloading haproxy in /var/log/haproxy.log you should have the Real Source IPs logged in:
 

root@webserver:~# tail -n 10 /var/log/haproxy.log
Apr 15 22:47:34 pcfr_hware_local_ip haproxy[2914]: 159.223.65.16:58735 [15/Apr/2022:22:47:34.586] https-in https-websrv/ha1server-2 1/0/+0 +0 — 7/7/7/7/0 0/0
Apr 15 22:47:34 pcfr_hware_local_ip haproxy[2914]: 20.113.133.8:56405 [15/Apr/2022:22:47:34.744] https-in https-websrv/ha1server-2 1/0/+0 +0 — 7/7/7/7/0 0/0
Apr 15 22:47:35 pcfr_hware_local_ip haproxy[2914]: 54.36.148.248:15653 [15/Apr/2022:22:47:35.057] https-in https-websrv/ha1server-2 1/0/+0 +0 — 7/7/7/7/0 0/0
Apr 15 22:47:35 pcfr_hware_local_ip haproxy[2914]: 185.191.171.35:26564 [15/Apr/2022:22:47:35.071] https-in https-websrv/ha1server-2 1/0/+0 +0 — 8/8/8/8/0 0/0
Apr 15 22:47:35 pcfr_hware_local_ip haproxy[2914]: 213.183.53.58:42984 [15/Apr/2022:22:47:35.669] https-in https-websrv/ha1server-2 1/0/+0 +0 — 6/6/6/6/0 0/0
Apr 15 22:47:35 pcfr_hware_local_ip haproxy[2914]: 159.223.65.16:54006 [15/Apr/2022:22:47:35.703] https-in https-websrv/ha1server-2 1/0/+0 +0 — 7/7/7/7/0 0/0
Apr 15 22:47:36 pcfr_hware_local_ip haproxy[2914]: 192.241.113.203:30877 [15/Apr/2022:22:47:36.651] https-in https-websrv/ha1server-2 1/0/+0 +0 — 4/4/4/4/0 0/0
Apr 15 22:47:36 pcfr_hware_local_ip haproxy[2914]: 185.191.171.9:6776 [15/Apr/2022:22:47:36.683] https-in https-websrv/ha1server-2 1/0/+0 +0 — 5/5/5/5/0 0/0
Apr 15 22:47:36 pcfr_hware_local_ip haproxy[2914]: 159.223.65.16:64310 [15/Apr/2022:22:47:36.797] https-in https-websrv/ha1server-2 1/0/+0 +0 — 6/6/6/6/0 0/0
Apr 15 22:47:36 pcfr_hware_local_ip haproxy[2914]: 185.191.171.3:23364 [15/Apr/2022:22:47:36.834] https-in https-websrv/ha1server-2 1/1/+1 +0 — 7/7/7/7/0 0/0

 

2. Enable remoteip proxy protocol on Webservers

Login to each Apache HTTPD and to enable remoteip module run:
 

# a2enmod remoteip


On Debians, the command should produce a right symlink to mods-enabled/ directory
 

# ls -al /etc/apache2/mods-enabled/*remote*
lrwxrwxrwx 1 root root 31 Mar 30  2021 /etc/apache2/mods-enabled/remoteip.load -> ../mods-available/remoteip.load

 

3. Modify remoteip.conf file and allow IPs of haproxies or F5s

 

Configure RemoteIPTrustedProxy for every Source IP of haproxy to allow it to send X-Forwarded-For header to Apache,

Here are few examples, from my apache working config on Debian 11.2 (Bullseye):
 

webserver:~# cat remoteip.conf
RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 192.168.0.1
RemoteIPTrustedProxy 192.168.0.205
RemoteIPTrustedProxy 192.168.1.15
RemoteIPTrustedProxy 192.168.0.198
RemoteIPTrustedProxy 192.168.2.33
RemoteIPTrustedProxy 192.168.2.30
RemoteIPTrustedProxy 192.168.0.215
#RemoteIPTrustedProxy 51.89.232.41

On RedHat / Fedora other RPM based Linux distrubutions, you can do the same by including inside httpd.conf or virtualhost configuration something like:
 

<IfModule remoteip_module>
      RemoteIPHeader X-Forwarded-For
      RemoteIPInternalProxy 192.168.0.0/16
      RemoteIPTrustedProxy 192.168.0.215/32
</IfModule>


4. Enable RemoteIP Proxy Protocol in apache2.conf / httpd.conf or Virtualhost custom config
 

Modify both haproxy / haproxies config as well as enable the RemoteIP module on Apache webservers (VirtualHosts if such used) and either in <VirtualHost> block or in main http config include:

RemoteIPProxyProtocol On


5. Change default configured Apache LogFormat

In Domain Vhost or apache2.conf / httpd.conf

Default logging Format will be something like:
 

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined


or
 

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

 

Once you find it in /etc/apache2/apache2.conf / httpd.conf or Vhost, you have to comment out this by adding shebang infont of sentence make it look as follows:
 

LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%a %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent


The Changed LogFormat instructs Apache to log the client IP as recorded by mod_remoteip (%a) rather than hostname (%h). For a full explanation of all the options check the official HTTP Server documentation page apache_mod_config on Custom Log Formats.

and reload each Apache server.

on Debian:

# apache2ctl -k reload

On CentOS

# systemctl restart httpd


6. Check proxy protocol is properly enabled on Apaches

 

remoteip module will enable Apache to expect a proxy connect header passed to it otherwise it will respond with Bad Request, because it will detect a plain HTML request instead of Proxy Protocol CONNECT, here is the usual telnet test to fetch the index.htm page.

root@webserver:~# telnet localhost 80
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1

HTTP/1.1 400 Bad Request
Date: Fri, 15 Apr 2022 19:04:51 GMT
Server: Apache/2.4.51 (Debian)
Content-Length: 312
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
<hr>
<address>Apache/2.4.51 (Debian) Server at grafana.pc-freak.net Port 80</address>
</body></html>
Connection closed by foreign host.

 

root@webserver:~# telnet localhost 80
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
HEAD / HTTP/1.1

HTTP/1.1 400 Bad Request
Date: Fri, 15 Apr 2022 19:05:07 GMT
Server: Apache/2.4.51 (Debian)
Connection: close
Content-Type: text/html; charset=iso-8859-1

Connection closed by foreign host.


To test it with telnet you can follow the Proxy CONNECT syntax and simulate you're connecting from a proxy server, like that:
 

root@webserver:~# telnet localhost 80
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
CONNECT localhost:80 HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Fri, 15 Apr 2022 19:13:38 GMT
Server: Apache/2.4.51 (Debian)
Location: https://zabbix.pc-freak.net
Cache-Control: max-age=900
Expires: Fri, 15 Apr 2022 19:28:38 GMT
Content-Length: 310
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://zabbix.pc-freak.net">here</a>.</p>
<hr>
<address>Apache/2.4.51 (Debian) Server at localhost Port 80</address>
</body></html>
Connection closed by foreign host.

You can test with curl simulating the proxy protocol CONNECT with:

root@webserver:~# curl –insecure –haproxy-protocol https://192.168.2.30

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta name="generator" content="pc-freak.net tidy">
<script src="https://ssl.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-2102595-3";
urchinTracker();
</script>
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-2102595-6");
pageTracker._trackPageview();
} catch(err) {}
</script>

 

      –haproxy-protocol
              (HTTP) Send a HAProxy PROXY protocol v1 header at the beginning of the connection. This is used by some load balancers and reverse proxies
              to indicate the client's true IP address and port.

              This option is primarily useful when sending test requests to a service that expects this header.

              Added in 7.60.0.


7. Check apache log if remote Real Internet Source IPs are properly logged
 

root@webserver:~# tail -n 10 /var/log/apache2/access.log

213.183.53.58 – – [15/Apr/2022:22:18:59 +0300] "GET /proxy/browse.php?u=https%3A%2F%2Fsteamcommunity.com%2Fmarket%2Fitemordershistogram%3Fcountry HTTP/1.1" 200 12701 "https://www.pc-freak.net" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0"
88.198.48.184 – – [15/Apr/2022:22:18:58 +0300] "GET /blog/iq-world-rank-country-smartest-nations/?cid=1330192 HTTP/1.1" 200 29574 "-" "Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)"
213.183.53.58 – – [15/Apr/2022:22:19:00 +0300] "GET /proxy/browse.php?u=https%3A%2F%2Fsteamcommunity.com%2Fmarket%2Fitemordershistogram%3Fcountry
HTTP/1.1" 200 9080 "https://www.pc-freak.net" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:98.0) Gecko/20100101 Firefox/98.0"
159.223.65.16 – – [15/Apr/2022:22:19:01 +0300] "POST //blog//xmlrpc.php HTTP/1.1" 200 5477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
159.223.65.16 – – [15/Apr/2022:22:19:02 +0300] "POST //blog//xmlrpc.php HTTP/1.1" 200 5477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36"
213.91.190.233 – – [15/Apr/2022:22:19:02 +0300] "POST /blog/wp-admin/admin-ajax.php HTTP/1.1" 200 1243 "https://www.pc-freak.net/blog/wp-admin/post.php?post=16754&action=edit" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0"
46.10.215.119 – – [15/Apr/2022:22:19:02 +0300] "GET /images/saint-Paul-and-Peter-holy-icon.jpg HTTP/1.1" 200 134501 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36 Edg/100.0.1185.39"
185.191.171.42 – – [15/Apr/2022:22:19:03 +0300] "GET /index.html.latest/tutorials/tutorials/penguins/vestnik/penguins/faith/vestnik/ HTTP/1.1" 200 11684 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"

116.179.37.243 – – [15/Apr/2022:22:19:50 +0300] "GET /blog/wp-content/cookieconsent.min.js HTTP/1.1" 200 7625 "https://www.pc-freak.net/blog/how-to-disable-nginx-static-requests-access-log-logging/" "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)"
116.179.37.237 – – [15/Apr/2022:22:19:50 +0300] "GET /blog/wp-content/plugins/google-analytics-dashboard-for-wp/assets/js/frontend-gtag.min.js?ver=7.5.0 HTTP/1.1" 200 8898 "https://www.pc-freak.net/blog/how-to-disable-nginx-static-requests-access-log-logging/" "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)"

 

You see from above output remote Source IPs in green are properly logged, so haproxy Cluster is correctly forwarding connections passing on in the Haproxy generated Initial header the Real IP of its remote connect IPs.


Sum it up, What was done?


HTTP_X_FORWARD_FOR is impossible to set, when haproxy is used on mode tcp and all traffic is sent as received from TCP IPv4 / IPv6 Network stack, e.g. modifying any HTTP sent traffic inside the headers is not possible as this might break up the data.

Thus Haproxy was configured to send all its received data by sending initial proxy header with the X_FORWARDED usual Source IP data, then remoteip Apache module was used to make Apache receive and understand haproxy sent Header which contains the original Source IP via the send-proxy functionality and example was given on how to test the remoteip on Webserver is working correctly.

Finally you've seen how to check configured haproxy and webserver are able to send and receive the End Client data with the originator real source IP correctly and those Internet IP is properly logged inside both haproxy and apaches.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , ,

No Comments »

« Previous Post

Install Zabbix Agent client on CentOS 9 Stream Linux, Disable Selinux and Firewalld on CentOS9 to make zabbix-agentd send data to server


April 14th, 2022

https://pc-freak.net/images/zabbix_agent_active_passive-zabbix-agent-centos-9-install-howto

Installing Zabbix is usually a trivial stuff, you either use the embedded distribution built packages if such are available this is for example defetch the right zabbix release repository  that configures the Zabbix official repo in the system, configure the Zabbix server or Proxy if such is used inside /etc/zabbix/zabbix_agentd.conf and start the client, i.e. I expected that it will be a simple and straight forward also on the freshly installed CentOS 9 Linux cause placing a zabbix-agent monitroing is a trivial stuff however installing came to error:

Key import failed (code 2). Failing package is: zabbix-agent-6.0.3-1.el8.x86_64

 

This is what I've done

1. Download and install zabbix-release-6.0-1.el8.noarch.rpm directly from zabbix

I've followed the official documentation from zabbix.com and ran:
 

[root@centos9 /root ]# rpm -Uvh https://repo.zabbix.com/zabbix/6.0/rhel/8/x86_64/zabbix-release-6.0-1.el8.noarch.rpm

…


2. Install  the zabbix-agent RPM package from the repositry

[root@centos9 rpm-gpg]# yum install zabbix-agent -y
Last metadata expiration check: 0:02:46 ago on Tue 12 Apr 2022 08:49:34 AM EDT.
Dependencies resolved.
=============================================
 Package                               Architecture                Version                              Repository                      Size
=============================================
Installing:
 zabbix-agent                          x86_64                      6.0.3-1.el8                          zabbix                         526 k
Installing dependencies:
 compat-openssl11                      x86_64                      1:1.1.1k-3.el9                       appstream                      1.5 M
 openldap-compat                       x86_64                      2.4.59-4.el9                         baseos                          14 k

Transaction Summary
==============================================
Install  3 PackagesTotal size: 2.0 M
Installed size: 6.1 M
Downloading Packages:
[SKIPPED] openldap-compat-2.4.59-4.el9.x86_64.rpm: Already downloaded
[SKIPPED] compat-openssl11-1.1.1k-3.el9.x86_64.rpm: Already downloaded
[SKIPPED] zabbix-agent-6.0.3-1.el8.x86_64.rpm: Already downloaded
Zabbix Official Repository – x86_64                                                                          1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0xA14FE591:
 Userid     : "Zabbix LLC <packager@zabbix.com>"
 Fingerprint: A184 8F53 52D0 22B9 471D 83D0 082A B56B A14F E591
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
Key import failed (code 2). Failing package is: zabbix-agent-6.0.3-1.el8.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by e
xecuting 'yum clean packages'.
Error: GPG check FAILED


3. Work around to skip GPG to install zabbix-agent 6 on CentOS 9

With Linux everything becomes more and more of a hack …
The logical thing to was to first,  check and it assure that the missing RPM GPG key is at place

[root@centos9 rpm-gpg]# ls -al  /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
-rw-r–r– 1 root root 1719 Feb 11 16:29 /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591

Strangely the key was in place.

Hence to have the key loaded I've tried to import the gpg key manually with gpg command:

[root@centos9 rpm-gpg]# gpg –import /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591


And attempted install again zabbix-agent once again:
 

[root@centos9 rpm-gpg]# yum install zabbix-agent -y
Last metadata expiration check: 0:02:46 ago on Tue 12 Apr 2022 08:49:34 AM EDT.
Dependencies resolved.
==============================================
 Package                               Architecture                Version                              Repository                      Size
==============================================
Installing:
 zabbix-agent                          x86_64                      6.0.3-1.el8                          zabbix                         526 k
Installing dependencies:
 compat-openssl11                      x86_64                      1:1.1.1k-3.el9                       appstream                      1.5 M
 openldap-compat                       x86_64                      2.4.59-4.el9                         baseos                          14 k

Transaction Summary
==============================================
Install  3 Packages

Total size: 2.0 M
Installed size: 6.1 M
Downloading Packages:
[SKIPPED] openldap-compat-2.4.59-4.el9.x86_64.rpm: Already downloaded
[SKIPPED] compat-openssl11-1.1.1k-3.el9.x86_64.rpm: Already downloaded
[SKIPPED] zabbix-agent-6.0.3-1.el8.x86_64.rpm: Already downloaded
Zabbix Official Repository – x86_64                                                                          1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0xA14FE591:
 Userid     : "Zabbix LLC <packager@zabbix.com>"
 Fingerprint: A184 8F53 52D0 22B9 471D 83D0 082A B56B A14F E591
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
Key import failed (code 2). Failing package is: zabbix-agent-6.0.3-1.el8.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: GPG check FAILED


Unfortunately that was not a go, so totally pissed off I've disabled the gpgcheck for packages completely as a very raw bad and unrecommended work-around to eventually install the zabbix-agentd like that.

Usually the RPM gpg key failures check on RPM packages could be could be workaround with in dnf, so I've tried that one without success.

[root@centos9 rpm-gpg]# dnf update –nogpgcheck
Total                                                                                                        181 kB/s | 526 kB     00:02
Zabbix Official Repository – x86_64                                                                          1.6 MB/s | 1.7 kB     00:00
Importing GPG key 0xA14FE591:
 Userid     : "Zabbix LLC <packager@zabbix.com>"
 Fingerprint: A184 8F53 52D0 22B9 471D 83D0 082A B56B A14F E591
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
Is this ok [y/N]: y
Key import failed (code 2). Failing package is: zabbix-agent-6.0.3-1.el8.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Further tried to use the –nogpgpcheck 
which according to its man page:


–nogpgpcheck 
Skip checking GPG signatures on packages (if RPM policy allows).


In yum the nogpgcheck option according to its man yum does exactly the same thing


[root@centos9 rpm-gpg]# yum install zabbix-agent –nogpgcheck -y
 

Dependencies resolved.
===============================================
 Package                             Architecture                  Version                               Repository                     Size
===============================================
Installing:
 zabbix-agent                        x86_64                        6.0.3-1.el8                           zabbix                        526 k

Transaction Summary
===============================================

Total size: 526 k
Installed size: 2.3 M
Is this ok [y/N]: y
Downloading Packages:
…
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                     1/1
  Running scriptlet: zabbix-agent-6.0.3-1.el8.x86_64                                                                                     1/2
  Reinstalling     : zabbix-agent-6.0.3-1.el8.x86_64                                                                                     1/2
  Running scriptlet: zabbix-agent-6.0.3-1.el8.x86_64                                                                                     1/2
  Running scriptlet: zabbix-agent-6.0.3-1.el8.x86_64                                                                                     2/2
  Cleanup          : zabbix-agent-6.0.3-1.el8.x86_64                                                                                     2/2
  Running scriptlet: zabbix-agent-6.0.3-1.el8.x86_64                                                                                     2/2
  Verifying        : zabbix-agent-6.0.3-1.el8.x86_64                                                                                     1/2
  Verifying        : zabbix-agent-6.0.3-1.el8.x86_64                                                                                     2/2

Installed:
  zabbix-agent-6.0.3-1.el8.x86_64

Complete!
[root@centos9 ~]#

Voila! zabbix-agentd on CentOS 9 Install succeeded!

Yes I know disabling a GPG check is not really secure and seems to be an ugly solution but since I'm cut of time in the moment and it is just for experimental install of zabbix-agent on CentOS
plus we already trusted the zabbix package repository anyways, I guess it doesn't much matter.

4. Configure Zabbix-agent on the machine

Once you choose how the zabbix-agent should sent the data to the zabbix-server (e.g. Active or Passive) mode the The minimum set of configuration you should
have at place should be something like mine:

[root@centos9 ~]# grep -v '\#' /etc/zabbix/zabbix_agentd.conf | sed /^$/d
PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
Server=192.168.1.70,127.0.0.1
ServerActive=192.168.1.70,127.0.0.1
Hostname=centos9
Include=/etc/zabbix/zabbix_agentd.d/*.conf

5. Start and Enable zabbix-agent client

To have it up and running

[root@centos9 ~]# systemct start zabbix-agent
[root@centos9 ~]# systemctl enable zabbix-agent

6. Disable SELinux to prevent it interfere with zabbix-agentd 

Other amazement was that even though I've now had configured Active check and a Server and correct configuration the Zabbix-Server could not reach the zabbix-agent for some weird reason.
I thought that it might be selinux and checked it and seems by default in the fresh installed CentOS 9 Linux selinux is already automatically set to enabled.

After stopping it i made sure, SeLinux would block for security reasons client connectivity to the zabbix-server until you either allow zabbix exception in SeLinux or until completely disable it.
 

[root@centos9 ~]# sestatus

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31

To temporarily change the mode from its default targeted to permissive mode 

[root@centos9 ~]# setenforce 0

[root@centos9 ~]# sestatus

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31


That would work for current session but won't take affect on next reboot, thus it is much better to disable selinux on next boot:

[root@centos9 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing – SELinux security policy is enforced.
#     permissive – SELinux prints warnings instead of enforcing.
#     disabled – No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of these three values:
#     targeted – Targeted processes are protected,
#     minimum – Modification of targeted policy. Only selected processes are protected. 
#     mls – Multi Level Security protection.
SELINUXTYPE=targeted

 

To disable selinux change:

SELINUXTYPE=disabled

[root@centos9 ~]# grep -v \# /etc/selinux/config

SELINUX=disabled
SELINUXTYPE=targeted


To make the OS disable selinux and test it is disabled you will have to reboot 

[root@centos9 ~]# reboot


Check its status again, it should be:

[root@centos9 ~]# sestatus
SELinux status:                 disabled


7. Enable zabbix-agent through firewall or disable firewalld service completely

By default CentOS 9 has the firewalld also enabled and either you have to enable zabbix to communicate to the remote server host.

To enable access for from and to zabbix-agentd in both Active / Passive mode:

#firewall settings:
[root@centos9 rpm-gpg]# firewall-cmd –permanent –add-port=10050/tcp
[root@centos9 rpm-gpg]# firewall-cmd –permanent –add-port=10051/tcp
[root@centos9 rpm-gpg]# firewall-cmd –reload
[root@centos9 rpm-gpg]# systemctl restart firewalld
[root@centos9 rpm-gpg]# systemctl restart zabbix-agent


If the machine is in a local DMZ-ed network with tightly configured firewall router in front of it, you could completely disable firewalld.

[root@centos9 rpm-gpg]# systemctl stop firewalld
[root@centos9 rpm-gpg]# systemctl disable firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

 

Next login to Zabbix-server web interface with administrator and from Configuration -> Hosts -> Create the centos9 hostname and add it a template of choice. The data from the added machine should shortly appear after another zabbix restart:

[root@centos9 rpm-gpg]#  systemctl restart zabbix-agentd


8. Tracking other oddities with the zabbix-agent through log

If anyways still zabbix have issues connectin to remote node, increase the debug log level section
 

[root@centos9 rpm-gpg]# vim /etc/zabbix/zabbix_agentd.conf
DebugLevel 5

### Option: DebugLevel
#       Specifies debug level:
#       0 – basic information about starting and stopping of Zabbix processes
#       1 – critical information
#       2 – error information
#       3 – warnings
#       4 – for debugging (produces lots of information)
#       5 – extended debugging (produces even more information)
#
# Mandatory: no
# Range: 0-5
# Default:
# DebugLevel=3

[root@centos9 rpm-gpg]# systemctl restart zabbix-agent

Keep in mind that debugging will be too verbose, so once you make the machine being seen in zabbix, don't forget to comment out the line and restart agent to turn it off.

9. Testing zabbix-agent, How to send an alert to specific item key

Usually when writting userparameter scripts, data collected from scripts is being sent to zabbix serveria via Item keys.
Thus one way to check the zabbix-agent -> zabbix server data send works fine is to send some simultaneous data via a key
Once zabbix-agent is configured on the machine 

In this case we will use something like ApplicationSupport-Item as an item.
 

[root@centos9 rpm-gpg]# /usr/bin/zabbix_sender -c "/etc/zabbix/zabbix_agentd.conf" -k "ApplicationSupport-Item" -o "here is the message"

Assuming you have created the newly prepared zabbix-agent host into Zabbix Server, you should be shortly able to see the data come in Latest data.

Tags: , , , , , , , , , , , , , , , , , , , ,

No Comments »

« Previous Post

How to remove GNOME environment and Xorg server on CentOS 7 / 8 / 9 Linux


April 13th, 2022

centos-linux-remove-gnome-gui-remove-howto-logo

If you have installed recent version of CentOS, you have noticed by default the Installator did setup Xserver and GNOME as Graphical Environment as well the surrounding GUI Administration tools. That's really not needed on "headless" monitorless Linux servers as this wastes up for nothing a very tiny amount of the machine CPU and RAM and Disk resource on keeping services up and running. Even worse a Graphical Environment on a Production server poses a security breach as their are much more services running on the OS that could be potentially hacked.

Removal of GUI across CentOS is similar but slightly differs. Hence in this article, I'll show how it can be removed on CentOS Linux 7 / 8 and 9. Removal of Graphics is usual operation for sysadmins thus there is plenty of info on the net,how this is done on CentOS 7 and COS 8 but unfortunately as of time of writting this article, couldn't find anything on the net on how to Remove GUI environment on CentOS 9.

The reason for this article is mostly for documentation purposes for myself

First list the available meta-package groups installed on the OS:

1. List machine installed package groups

 

yum-groupinstall-gnome

[root@centos ~]# yum grouplist
Last metadata expiration check: 3:55:48 ago on Mon 11 Apr 2022 03:26:06 AM EDT.
Available Environment Groups:
   Server
   Minimal Install
   Workstation
   KDE Plasma Workspaces
   Custom Operating System
   Virtualization Host
Installed Environment Groups:
   Server with GUI
Installed Groups:
   Container Management
   Headless Management
Available Groups:
   Legacy UNIX Compatibility
   Console Internet Tools
   Development Tools
   .NET Development
   Graphical Administration Tools
   Network Servers
   RPM Development Tools
   Scientific Support
   Security Tools
   Smart Card Support
   System Tools
   Fedora Packager


On CentOS 8 and CentOS 9 to list the installed package groups, you can use also:

[root@centos ~]# dnf grouplist
…
Installed Environment Groups:
   Server with GUI

2. Remove GNOME and Xorg GUIs on CentOS 7

[root@centos ~]# yum groupremove "Server with GUI" –skip-broken

[root@centos ~]# yum groupremove "GNOME Desktop" -y
…

3. Remove GNOME and X on CentOS 8

[root@centos ~]# dnf groupremove 'X Window System' 'GNOME' -y
…

4. Remove Graphical Environment on CentOS 9

 

centos9-linux-groupremove-command-screenshot

[root@centos ~]# yum groupremove GNOME 'Graphical Administration Tools' -y

 

Removing Groups:
 GNOME

Transaction Summary
====================================================
Remove  123 Packages

Freed space: 416 M
Is this ok [y/N]: y
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
 

…
  xorg-x11-drv-libinput-1.0.1-3.el9.x86_64
  xorg-x11-server-Xorg-1.20.11-10.el9.x86_64
  xorg-x11-server-Xwayland-21.1.3-2.el9.x86_64
  xorg-x11-server-common-1.20.11-10.el9.x86_64
  xorg-x11-server-utils-7.7-44.el9.x86_64
  xorg-x11-xauth-1:1.1-10.el9.x86_64
  xorg-x11-xinit-1.4.0-11.el9.x86_64

Complete!


Graphical Administration Tools – is a group of tools that 

Or alternatively you can do

[root@centos ~]# yum remove gnome* xorg* -y


5. Change the Graphical boot to text multiuser

[root@centos ~]# systemctl set-default multi-user.target


6. Install GNOME / X GUI on the CentOS 7 / 8 / 9

Sometimes GNOME Desktop environment and Xorg are missing on previously delpoyed installs but you need it back for some reason.For example it was earlier removed a year ago on the server as it was not needed, but the machine use type changes and now you need to have installed an Oracle Server / Oracle Client which usually depends on having at least a minimal working version of X environment ont the Linux.


To install back the GNOME and X back on the machine:

[root@centos ~]# yum groupistall "Server with GUI" –skip-broken

[root@centos9 network-scripts]# yum groupinstall "Server with GUI" –skip-broken
Last metadata expiration check: 0:09:26 ago on Mon 11 Apr 2022 07:43:11 AM EDT.
No match for group package "insights-client"
No match for group package "redhat-release"
No match for group package "redhat-release-eula"
Dependencies resolved.
===================================================
 Package                                       Arch       Version              Repository     Size
===================================================
Installing group/module packages:
 NetworkManager-wifi                           x86_64     1:1.37.2-1.el9       baseos         75 k
 cheese                                        x86_64     2:3.38.0-6.el9       appstream      96 k
 chrome-gnome-shell                            x86_64     10.1-14.el9          appstream      33 k
 eog                                           x86_64     40.3-2.el9           appstream     3.6 M
 evince                                        x86_64     40.4-4.el9           appstream     2.8 M
 evince-nautilus                               x86_64     40.4-4.el9           appstream      20 k
 gdm                                           x86_64     1:40.1-13.el9        appstream     894 k
 gnome-bluetooth                               x86_64     1:3.34.5-3.el9       appstream      44 k
 gnome-calculator                              x86_64     40.1-2.el9           appstream     1.4 M
 gnome-characters                              x86_64     40.0-3.el9           appstream     236 k
 gnome-classic-session                         noarch     40.6-1.el9           appstream      36 k
 gnome-color-manager                           x86_64     3.36.0-7.el9         appstream     1.1 M
 gnome-control-center                          x86_64     40.0-22.el9          appstream     5.7 M
 gnome-disk-utility                            x86_64     40.2-2.el9           appstream     1.1 M
 gnome-font-viewer                             x86_64     40.0-3.el9           appstream     233 k
 gnome-initial-setup                           x86_64     40.1-2.el9           appstream     1.1 M
 gnome-logs                                    x86_64     3.36.0-6.el9         appstream     416 k
…
Installing dependencies:
 cheese-libs                                   x86_64     2:3.38.0-6.el9       appstream     941 k
 clutter                                       x86_64     1.26.4-7.el9         appstream     1.1 M
 clutter-gst3                                  x86_64     3.0.27-7.el9         appstream      85 k
 clutter-gtk                                   x86_64     1.8.4-13.el9         appstream      47 k
 cogl                                          x86_64     1.22.8-5.el9         appstream     505 k
 colord-gtk                                    x86_64     0.2.0-7.el9          appstream      33 k
 dbus-daemon                                   x86_64     1:1.12.20-5.el9      appstream     202 k
 dbus-tools                                    x86_64     1:1.12.20-5.el9      baseos         52 k
 evince-previewer                              x86_64     40.4-4.el9           appstream      29 k

Installing weak dependencies:
 gnome-tour                                    x86_64     40.1-1.el9           appstream     722 k
 nm-connection-editor                          x86_64     1.26.0-1.el9         appstream     838 k
 p11-kit-server                                x86_64     0.24.1-2.el9         appstream     199 k
 pinentry-gnome3                               x86_64     1.1.1-8.el9          appstream      41 k
Installing Environment Groups:
 Server with GUI
Installing Groups:
 base-x
 Container Management
 core
 fonts
 GNOME
 guest-desktop-agents
 Hardware Monitoring Utilities
 hardware-support
 Headless Management
 Internet Browser
 multimedia
 networkmanager-submodules
 print-client
 Server product core
 standard

Transaction Summary
=======================================================
Install  114 Packages

Total download size: 96 M
Installed size: 429 M
Is this ok [y/N]: y
…

or yum groupinstall GNOME

[root@centos9 ~]# yum grouplist
Last metadata expiration check: 3:55:48 ago on Mon 11 Apr 2022 03:26:06 AM EDT.
Available Environment Groups:
…
Installed Environment Groups:
   Server with GUI
…

Next you should change the OS default run level to 5 to make CentOS automatically start the Xserver and gdm.

To see the list of available default Login targets do:
 


[root@centos ~]# find / -name "runlevel*.target"
/usr/lib/systemd/system/runlevel0.target
/usr/lib/systemd/system/runlevel1.target
/usr/lib/systemd/system/runlevel2.target
/usr/lib/systemd/system/runlevel3.target
/usr/lib/systemd/system/runlevel4.target
/usr/lib/systemd/system/runlevel5.target
/usr/lib/systemd/system/runlevel6.target

The meaning of each runlevel is as follows:

Run Level Target Units Description
0 runlevel0.target, poweroff.target Shut down and power off
1 runlevel1.target, rescue.target Set up a rescue shell
2,3,4 runlevel[234].target, multi- user.target Set up a nongraphical multi-user shell
5 runlevel5.target, graphical.target Set up a graphical multi-user shell
6 runlevel6.target, reboot.target Shut down and reboot the system


If this does not work you can try:

yum-groupinstall-gnome

[root@centos ~]#  yum -y groups install "GNOME Desktop"
…


7. To check the OS configured boot target
 

[root@centos ~]# systemctl get-default
multi-user.target


multi-user.target is a mode of operation that is text mode only with multiple logins supported on tty and remotely.

To change it to graphical

[root@centos ~]# systemctl set-default graphical.target


or simply link it yourself
 

[root@centos ~]# ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target

[root@centos ~]# reboot


If the X was not used so far ever, you will get a few graphial screens to accept the License Information and Finish the configuration,i .e.

1. Accept the license by clicking on the “LICENSE INFORMATION“.

2. Tick mark the “I accept the license agreement” and click on “Done“.

3. Click on “FINISH CONFIGURATION” to complete the setup.
And voila GDM (Graphical Login) Greater should shine up.
 

You could also go the manual route by adding an .xinitrc file in your home directory (instead of making the graphical login screen the default, as done above with the sudo systemctl set-default graphical.target command). To do this, issue the command:

[root@centos ~]# echo "exec gnome-session" >> ~/.xinitrc

 

Tags: , , , , , , , , , , , ,

No Comments »

« Previous Post

How to disable Windows pagefile.sys and hiberfil.sys to temporary or permamently save disk space if space is critically low


March 28th, 2022

howto-pagefile-hiberfil.sys-remove-reduce-increase-increase-size-windows-logo

Sometimes you have to work with Windows 7 / 8 / 10 PCs  etc. that has a very small partition C:\
drive or othertimes due to whatever the disk got filled up with time and has only few megabytes left
and this totally broke up the windows performance as Windows OS becomes terribly sluggish and even
simple things as opening Internet Browser (Chrome / Firefox / Opera ) or Windows Explorer stones the PC performance.

You might of course try to use something like Spacesniffer tool (a great tool to find lost data space on PC s short description on it is found in my previous article how to
delete temporary Internet Files and Folders to to speed up and free disk space ) or use CCleaner to clean up a bit the pc.
Sometimes this is not enough though or it is not possible to do at all the main
partition disk C:\ is anyhow too much low (only 30-50MB are available on HDD) or the Physical or Virtual Machine containing the OS is filled with important data
and you couldn't risk to remove anything including Internet Temporary files, browsing cookies … whatever.

Lets say you are the fate chosen guy as sysadmin to face this uneasy situation and have no easy
way to add disk space from another present free space partition or could not add a new SATA hard drive
SSD drive, what should you do?
 

The solution wipe off pagefile.sys and hiberfil.sys

Usually every Windows installation has a pagefile.sys and hiberfil.sys.

  • pagefile.sys – is the default file that is used as a swap file, immediately once the machine runs out of memory. For Unix / Linux users better understanding pagefile.sys is the equivallent of Linux's swap partition. Of course as the pagefile is in a file and not in separate partition the swapping in Windows is perhaps generally worse than in Linux.
  • hiberfil.sys – is used to store data from the machine on machine Hibernation (for those who use the feature)


Pagefile.sys which depending on the configured RAM memory on the OS could takes up up to 5 – 8 GB, there hanging around doing nothing but just occupying space. Thus a temporary workaround that could free you some space even though it will degrade performance and on servers and production machines this is not a good solution on just user machines, where you temporary need to free space any other important task you can free up space
by seriously reducing the preconfigured default size of pagefile.sys (which usually is 1.5 times the active memory on the OS – hence if you have 4GB you would have a 6Gigabytes of pagefile.sys).

Other possibility especially on laptop and movable devices running Win OS is to disable hiberfil.sys, read below how this is done.


The temporary solution here is to simply free space by either reducing the pagefile.sys or completely disabling it


1. Disable pagefile.sys on Windows XP, Windows 7 / 8 / 10 / 11


The GUI interface to disable pagefile across all NT based Windows OS-es is quite similar, the only difference is newer versions of Windows has slightly more options.


1.1 Disable pagefile on Windows XP


Quickest way is to find pagefile.sys settings from GUI menus

1. Computer (My Computer) – right click mouse
2. Properties (System Preperties will appear)
3. Advanced (tab) 
4. Settings
5. Advanced (tab)
6. Change button

windows-xp-pagefile-disable-screenshot

1.2 Disable pagefile on Windows 7

 

advanced-system-settings-control-panel-system-and-security-screenshot

windows-system-properties-screenshot-properties-advanced-change-Virtual-memory-pagefile-screenshot

system-properties-performance-options
 

Once applied you'll be required to reboot the PC

How-to-turn-off-Virtual-Memory-Paging_File-in-Windows-7-restart

 

1.3 Disable Increase / Decrease pagefile.sys on Windows 10 / Win 11
 

open-system-properties-advanced-win10

win10-performance-options-menu-screenshot

configure-virtual-memory-win-10-screenshot


1.4 Make Windows clear pagefile.sys on shutdown

On home PCs it might be useful thing to clear up ( nullify) pagefile.sys on shutdown, that could save you some disk space on every reboot, until file continuously grows to its configured Maximum.

Run

regedit

Modify registry key at location

  

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

windows-clean-up-pagefile-sys-file-on-shutdown-or-reboot-registry-editor-value-screenshot

You can apply the value also via a registry file you can get the Enable Clearpagefile at shutdown here .reg.
 

2. Manipulating pagefile.sys size and file delete from command line with wmic tool 

For scripting purposes you might want to use the wmic pagefile which can do increase / decrase or delete the file without GUI, that is very helpful if you have to admin a Windows Domain (Active Directory)
 

[hipo.WINDOWS-PC] ➤ wmic pagefile /?

PAGEFILE – Virtual memory file swapping management.

HINT: BNF for Alias usage.
(<alias> [WMIObject] | [] | [] ) [].

USAGE:

PAGEFILE ASSOC []
PAGEFILE CREATE <assign list>
PAGEFILE DELETE
PAGEFILE GET [] []
PAGEFILE LIST [] []

 

[hipo.WINDOWS-PC] ➤ wmic pagefile
AllocatedBaseSize  Caption          CurrentUsage  Description      InstallDate                Name             PeakUsage  Status  TempPageFile
4709               C:\pagefile.sys  499           C:\pagefile.sys  20200912061902.938000+180  C:\pagefile.sys  525                FALSE

 

[hipo.WINDOWS-PC] ➤ wmic pagefile list /format:list

AllocatedBaseSize=4709
CurrentUsage=499
Description=C:\pagefile.sys
InstallDate=20200912061902.938000+180
Name=C:\pagefile.sys
PeakUsage=525
Status=
TempPageFile=FALSE

wmic-pagefile-command-line-tool-for-windows-default-output-screenshot

 

  • To change the Initial Size or Maximum Size of Pagefile use:
     

➤ wmic pagefileset where name="C:\\pagefile.sys" set InitialSize=2048,MaximumSize=2048

  • To move the pagefile / change location of pagefile to less occupied disk drive partition (i.e. D:\ drive)

     

     

    Sometimes you might have multiple drives on the PC and some of them might be having multitudes of gigabytes while main drive C:\ could be fully occupied due to initial install bad drive organization, in that case a good work arount to save you space so you can work normally with the server is just to temporary or permanently move pagefile to another drive.

wmic pagefileset where name="D:\\pagefile.sys" set InitialSize=2048,MaximumSize=2048


!! CONSIDER !!! 

That if you have the option to move the pagefile.sys for best performance it is advicable to place the file inside another physical disk, preferrably a Solid State Drive one, SATA disks are too slow and reduced Input / Output disk operations will lead to degraded performance, if there is lack of memory (i.e. pagefile.sys is actively open read and wrote in).
 

  • To delete pagefile.sys 
     

➤ wmic pagefileset where name="C:\\pagefile.sys" delete

 

If for some reason you prefer to not use wmic but simple del command you can delete pagefile.sys also by:

Removing file default "Hidden" and "system" file attributes – set for security reasons as the file is a system file usually not touched by user. This will save you from "permission denied" errors:
 

➤ attrib -s -h %systemdrive%\pagefile.sys


Delete the file:
 

➤ del /a /q %systemdrive%\pagefile.sys


3. Disable hibernation on Windows 7 / 8 and Win 10 / 11

Disabling hibernation file hiberfil.sys can also free up some space, especially if the hibernation has been actively used before and the file is written with data. Of course, that is more common on notebooks.
Windows hibernation has significantly improved over time though i didn't have very pleasant experience in the past and I prefer to disable it just in case.
 

3.1 Disable Windows 7 / 8 / 10 / 11 hibernation from GUI 

Disable it through:

Control Panel -> All Control Panel Items -> Power Options -> Edit Plan Settings -> Change advanced power settings


 like shown in below screenshot:

Windqows-power-options-Advanced-settings-Allow-Hybrid-sleep-option-menu-screenshot

 

3.2 Disable Windows 7 / 8 / 10 / 11 hibernation from command line

Disable hibernation Is done in the same way through the powercfg.exe command, to disable it
if you're cut of disk space and you want to save space from it:

run as Administrator in Command Line Windows (cmd.exe)
 

powercfg.exe /hibernate off

If you later need to switch on hibernation
 

powercfg.exe /hibernate on


disable-hiberfile-windows-screenshot

3.3 Disable Windows hibernation on legacy Windows XP

On XP to disable hibernation open

1. Power Options Properties
2. Select Hibernate. 
3. Select Enable Hibernation to clear the checkbox and disable Hibernation mode. 
4. Select OK to apply the change.

Close the Power Options Properties box. 

enable-disable-hibernate-windows-xp-menu-screenshot

To sum it up

We have learned some basics on Windows swapping and hibernation and i've tried to give some insight on how thiese files if misconfigured could lead to degraded Win OS performance. In any case using SSD as of 2022 to store both files is a best practice for machines that has plenty of memory always try to completely disable / remove the files. It was shown how  to manage pagefile.sys and hiberfil.sys across Windows Operating Systems different versions both from GUI and via command line as well as how you can configure pagefile.sys to be cleared up on pc reboot.
 

Tags: , , , , , , , , , , , , , , , , , , ,

No Comments »

« Previous Post

Saint Luke of Odrin, a Glorious Bulgarian XVI century Christ martyr who publicly refuted Islam, feast day 23 of March


March 23rd, 2022

Saint_Lukas-of-Odrin-Adreanopolis-icon-Bulgarian-martyr-saint-of-ottoman-time

Holy Reverend Martyr Luka (the bulgarian name for Luke) was born in city of today's Edirne ( Odrin ) formerly known as Adreanopolis to pious Bulgarian parents – Atanas and Dominica. After the early death of his father, his mother gave him to a rich merchant from (Zagor) today's city Stara Zagora in Bulgaria, who treated him like a son.
The merchant once went to Russia, taking 13-year-old Luka with him.
On his return he stopped in Constantinople.

There Luca quarreled with a turk and started beating him. Then the enraged Turks captured him and wanted to kill him.
The frightened Luke shouted: "Let me go! I will convert to Islam!"

The turk immediately took him in and forced him to renounce Christ and convert to Islam, which the young boy did out of fear.
But after the fear passed, Luke became frightened by his action, lost his joy, and began to repent bitterly.

He reported himself to his batismal father (have to say at that time the importance of the God-father was very enormous much more than today). 
His father tried to release him with the help of the Russian ambassador.
But the evil turk, instead of freeing the boy, tied his hands and forcibly circumcised him.

After some time, Luca managed to escape from this Turkish home where he was forced to work as being counted a member of Islamic community and introduced himself to the Russian ambassador, who in attempt to save him from his slavery, sent him to the cities of Smyrna and Tire.
There Luke became seriously ill and, fearing death, called a Christian priest.
After hearing his confession, the priest advised him to retire to Mount Athos and repent there under the guidance of virtuous men.
After his recovery, he actually followed the advice of the clergyman and entered the Iviron  Athos Monastery, where he was restored to his Christian faith through the sacrament of Holy Anointing.

Saint_Loukas-Mytilene-confessor-Christian-martyr-who-denounced-Islam-neomartyr

Saint Luka of Odrin ( Adreanopolis ), the saint is well known in Greece as Saint Luke of Mytiliny (Mytilene a main city
in Lesbos Island Greece)

From there he moved to the Stavronikita Monastery, where he received a monastic vow.
But the enemy of human salvation did not give him peace.
Continuously Driven by temptations, he fled from one monastery to another. He went to the  Bulgarian "Zografski" ( Zographou ) monastery (from where he returned to the world, but failed to calm down, came to Mount Athos again) and then consequentlially to  Xyropotamou, Koutloumousiou  ( Kutlumush ), Saint John's Scyte, Grigoriou (Grigoriat), Saint. Anna Scyte.

Saint_Loukas of-Odrin-Adrianople-icon

Thinking about why he has no peace during all this time, but always moving from place to place, he came to the conclusion that this is a punishment from God for his denial of the Christian faith.

So he came up with the idea of ​​martyrdom (as a mean to redeem his sin) and shared it with some clergy: no one dared to recommend a martyrdom to him because of his youth.
But in the end, seeing his unwavering determination, the priest Ananias gave him a cell rule in preparation for martyrdom for Christ. Then the clergyman Visarion cut his hair in a Great Schema " μεγαλόσχημος" monastic vows and went with him to the island of Mytilene (today in Greece).

After partaking in the Holy Mysteries of Christ, St. Luke dressed as a Turk and went to the Qadi (Judge – qadis have been an institution in cities in Turkish empire pretty much like today's Courts).

He told allegorically how he was forcibly converted to Islam, openly renounced ungodly Islam and professed the Christian faith in the following way:

He went beforte the kadi and asked him in loud voice,

"Is it just for me to be deceived, a child like myself?"

The kadi asked,
"And who deceived you?"

Luka responded,

"Someone deceive me, giving me a badge of islam [meaning circumcision."

The kadi then asked to see the badge. But when Luke went to open his clothing, those
present in the courthouse finally understood what he meant and shouted for him to stop.

Luka then said,

"Being a young child of thirteen, I was deceived by you
and I came over to your religion, not being able to discern
the truth from falsehood. I therefore remained
with what is false and a lie for some time because I
understood your religion was not true, but false.
And he whom you call a prophet is not a prophet but a
deceiver and a mythmaker, and he has deceived all of
you and you believed him . Having therefore been
informed that your religion is darknes, I reject
it before you and I confess my former Christian faith
which is true light. I believe and worship a true God,
my Lord Jesus Christ who will come to judge all
the world, the living and the ded, and who will render to
each according to his works. If you do not believe in
HIm. as I do, you will all be damned."


The kadi further asked Luka where he is from and he said he has came to this place by ship from Russia as Russia has been considered the only country protectorof Orthodox Christians in the empire and
wanting to protect his relatives and the Church hierarchy in the Empire from from further beating or destructions of Churches and arrests of Orthodox Priests and clergy within the empire by the turks.

He further asked:
"Where are you staying?"

"Nowhere. I did not stop anywhere, but came straight here."

The kadi turned to the others in court and said,

"He is crazy. See if he recognized his own shoes."

Luka turned and immediately rushed to find his shoes and showed them
to the kadi and said,

"I am not crazy, as you say. Here are my shoes. I brought them in Constantinople."

The kadi said,
"I am sorry for you son.
If you do not listen to me,
you will suffer many torments, many of which
you have not even heard. So think well."

Luka replied:

"I have already though on all the torments which you
can inflict on me, and I have come. So, whatever you
plan to do to me, do it quickly, without delay, and be
assured that I am clean and blameless and I will not deny
my faith. I am an Orthodox Christian and I will die an
Orthodox Christian. I worship Christ and it is Christ
whom I desire. I confess him here where I once
denied Him in ignorance and I proclaim Him with true knowledge."

All-Bulgariasn-saints-Church-service-icon-Vsi_Bylgarski_svetii_sluzhba
All Bulgarian Saints icon, Christ in Glory surrounded by
Bulgarian saints – below Christ throne – Saint King Boris the Baptizer
and Saint Patriarch Euthimius, the monastery on the Bottom Rila Monastery
The first by importance and Largest Monastery in Bulgaria, Saint Loukas of Adreanopolis
is also commemorated on this date too

Neither the caresses and promises of the Muslims nor the threats of torture could shake the hard young man in his determination to die for Christ. The Turks beat him severely and threw him in prison, shackling his legs. At that time all the churches were praying for the martyr.
The Metropolitan of the island and the Elder Visarion managed beseech to send him to prison the Holy Mysteries of Christ for the last communion.
Finally, the judge saw Luke's unwavering firmness and sentenced him to death by hanging.
When the executioner hoped for the noose around his neck, he said,
"Confess Muhammad, our great prophet, and we will let you go!"

The Holy Martyr replied,

"I believe in my Lord Jesus Christ and I worship Him alone!"

St. Luke was hanged on March 23, 1802 at the age of 16.

The holy body of the martyr remained hanging on the gallows for three days, but during all this time it remained white and beautiful, emitting an unusual fragrance.
After three days, the Turks tied a large stone around his neck and his body was thrown into the depths of the sea, but instead of sinking, it remained with the stone on the water surface. When night came, the Christians found him on the shore and handed him over to the earth with due honors. The other clothes of the Venerable Martyr received healing power and by touching them the sick received healing.


Let by the Holy Prayers of Saint Luka of Odrin, God quickly grant all People worldwide more peace, love, hope, faith and wisdom that are so desperately needed today !

Text on the living of the saint  extracted from:
1.  Lives of the Saints. Ed. † Bishop Parthenius (Levkijski) of Lefkada, archim. Dr. Athanasius (Bonchev). Synodal Publishing House, Sofia, 1991
2. Witness for Christ: Orthodox Christian Neomartyrs of the Ottoman Period 1437 – 1860 by Nomikos Michael Vaporis p. 252-257
 

Tags: , , , , , , , , , , , , , , , , ,

No Comments »

« Previous Post