Archive for July, 2022

How to update expiring OpenSSL certificates without downtime on haproxy Pacemaker / Corosync PCS Cluster

Tuesday, July 19th, 2022

pcm-active-passive-scheme-corosync-pacemaker-openssl-renew-fix-certificate

Lets say you have a running PCS Haproxy cluster with 2 nodes and you have already a configuration in haproxy with a running VIP IP and this proxies
are tunneling traffic to a webserver such as Apache or directly to an Application and you end up in the situation where the configured certificates,
are about to expire soon. As you can guess having the cluster online makes replacing the old expiring SSL certificate with a new one relatively easy
task. But still there are a couple of steps to follow which seems easy but systemizing them and typing them down takes some time and effort.
In short you need to check the current certificates installed on the haproxy inside the Haproxy configuration files,
in my case the haproxy cluster was running 2 haproxy configs haproxyprod.cfg and haproxyqa.cfg and the certificates configured are places inside this
configuration.

Hence to do the certificate update, I had to follow few steps:

A. Find the old certificate key or generate a new one that will be used later together with the CSR (Certificate Request File) to generate the new Secure Socket Layer
certificate pair.
B. Either use the old .CSR (this is usually placed inside the old .CRT certificate file) or generate a new one
C. Copy those .CSR file to the Copy / Paste buffer and place it in the Website field on the step to fill in a CSR for the new certificate on the Domain registrer
such as NameCheap / GoDaddy / BlueHost / Entrust etc.
D. Registrar should then be able to generate files like the the new ServerCertificate.crt, Public Key Root Certificate Authority etc.
E. You should copy and store these files in some database for future perhaps inside some database such as .xdb
for example you can se the X – Certificate and Key management xca (google for xca download).
F. Copy this certificate and place it on the top of the old .crt file that is configured on the haproxies for each domain for which you have configured it on node2
G. standby node1 so the cluster sends the haproxy traffic to node2 (where you should already have the new configured certificate)
H. Prepare the .crt file used by haproxy by including the new ServerCertificate.crt content on top of the file on node1 as well
I. unstandby node1
J. Check in browser by accessing the URL the certificate is the new one based on the new expiry date that should be extended in future
K. Check the status of haproxy
L. If necessery check /var/log/haproxy.log on both clusters to check all works as expected

haserver_cluster_sample

Below are the overall commands to use to complete below jobs

Old extracted keys and crt files are located under /home/username/new-certs

1. Check certificate expiry start / end dates


[root@haproxy-serv01 certs]# openssl s_client -connect 10.40.18.88:443 2>/dev/null| openssl x509 -noout -enddate
notAfter=Aug 12 12:00:00 2022 GMT

2. Find Certificate location taken from /etc/haproxy/haproxyprod.cfg / /etc/haproxy/haproxyqa.cfg

# from Prod .cfg
   bind 10.40.18.88:443 ssl crt /etc/haproxy/certs/www.your-domain.com.crt ca-file /etc/haproxy/certs/ccnr-ca-prod.crt 
 

# from QA .cfg

    bind 10.50.18.87:443 ssl crt /etc/haproxy/certs/test.your-domain.com.crt ca-file /etc/haproxy/certs

3. Check  CRT cert expiry


# for haproxy-serv02 qa :443 listeners

[root@haproxy-serv01 certs]# openssl s_client -connect 10.50.18.87:443 2>/dev/null| openssl x509 -noout -enddate 
notAfter=Dec  9 13:24:00 2029 GMT

 

[root@haproxy-serv01 certs]# openssl x509 -enddate -noout -in /etc/haproxy/certs/www.your-domain.com.crt
notAfter=Aug 12 12:00:00 2022 GMT

[root@haproxy-serv01 certs]# openssl x509 -noout -dates -in /etc/haproxy/certs/www.your-domain.com.crt 
notBefore=May 13 00:00:00 2020 GMT
notAfter=Aug 12 12:00:00 2022 GMT


[root@haproxy-serv01 certs]# openssl x509 -noout -dates -in /etc/haproxy/certs/other-domain.your-domain.com.crt 
notBefore=Dec  6 13:52:00 2019 GMT
notAfter=Dec  9 13:52:00 2022 GMT

4. Check public website cert expiry in a Chrome / Firefox or Opera browser

In a Chrome browser go to updated URLs:

https://www.your-domain/login

https://test.your-domain/login

https://other-domain.your-domain/login

and check the certs

5. Login to one of haproxy nodes haproxy-serv02 or haproxy-serv01

Check what crm_mon (the cluster resource manager) reports of the consistancy of cluster and the belonging members
you should get some output similar to below:

[root@haproxy-serv01 certs]# crm_mon
Stack: corosync
Current DC: haproxy-serv01 (version 1.1.23-1.el7_9.1-9acf116022) – partition with quorum
Last updated: Fri Jul 15 16:39:17 2022
Last change: Thu Jul 14 17:36:17 2022 by root via cibadmin on haproxy-serv01

2 nodes configured
6 resource instances configured

Online: [ haproxy-serv01 haproxy-serv02 ]

Active resources:

 ccnrprodlbvip  (ocf::heartbeat:IPaddr2):       Started haproxy-serv01
 ccnrqalbvip    (ocf::heartbeat:IPaddr2):       Started haproxy-serv01
 Clone Set: haproxyqa-clone [haproxyqa]
     Started: [ haproxy-serv01 haproxy-serv02 ]
 Clone Set: haproxyprod-clone [haproxyprod]
     Started: [ haproxy-serv01 haproxy-serv02 ]


6. Create backup of existing certificates before proceeding to regenerate expiring
On both haproxy-serv01 / haproxy-serv02 run:

 

# cp -vrpf /etc/haproxy/certs/ /home/username/etc-haproxy-certs_bak_$(date +%d_%y_%m)/


7. Find the .key file etract it from latest version of file CCNR-Certificates-DB.xdb

Extract passes from XCA cert manager (if you're already using XCA if not take the certificate from keypass or wherever you have stored it.

+ For XCA cert manager ccnrlb pass
Find the location of the certificate inside the .xdb place etc.

+++++ www.your-domain.com.key file +++++

—–BEGIN PUBLIC KEY—–

—–END PUBLIC KEY—–


# Extracted from old file /etc/haproxy/certs/www.your-domain.com.crt
 

—–BEGIN RSA PRIVATE KEY—–

—–END RSA PRIVATE KEY—–


+++++

8. Renew Generate CSR out of RSA PRIV KEY and .CRT

[root@haproxy-serv01 certs]# openssl x509 -noout -fingerprint -sha256 -inform pem -in www.your-domain.com.crt
SHA256 Fingerprint=24:F2:04:F0:3D:00:17:84:BE:EC:BB:54:85:52:B7:AC:63:FD:E4:1E:17:6B:43:DF:19:EA:F4:99:L3:18:A6:CD

# for haproxy-serv01 prod :443 listeners

[root@haproxy-serv02 certs]# openssl x509 -x509toreq -in www.your-domain.com.crt -out www.your-domain.com.csr -signkey www.your-domain.com.key


9. Move (Standby) traffic from haproxy-serv01 to ccnrl0b2 to test cert works fine

[root@haproxy-serv01 certs]# pcs cluster standby haproxy-serv01


10. Proceed the same steps on haproxy-serv01 and if ok unstandby

[root@haproxy-serv01 certs]# pcs cluster unstandby haproxy-serv01


11. Check all is fine with openssl client with new certificate


Check Root-Chain certificates:

# openssl verify -verbose -x509_strict -CAfile /etc/haproxy/certs/ccnr-ca-prod.crt -CApath  /etc/haproxy/certs/other-domain.your-domain.com.crt{.pem?)
/etc/haproxy/certs/other-domain.your-domain.com.crt: OK

# openssl verify -verbose -x509_strict -CAfile /etc/haproxy/certs/thawte-ca.crt -CApath  /etc/haproxy/certs/www.your-domain.com.crt
/etc/haproxy/certs/www.your-domain.com.crt: OK

################# For other-domain.your-domain.com.crt ##############
Do the same

12. Check cert expiry on /etc/haproxy/certs/other-domain.your-domain.com.crt

# for haproxy-serv02 qa :15443 listeners
[root@haproxy-serv01 certs]# openssl s_client -connect 10.40.18.88:15443 2>/dev/null| openssl x509 -noout -enddate
notAfter=Dec  9 13:52:00 2022 GMT

[root@haproxy-serv01 certs]#  openssl x509 -enddate -noout -in /etc/haproxy/certs/other-domain.your-domain.com.crt 
notAfter=Dec  9 13:52:00 2022 GMT


Check also for 
+++++ other-domain.your-domain.com..key file +++++
 

—–BEGIN PUBLIC KEY—–

—–END PUBLIC KEY—–

 


# Extracted from /etc/haproxy/certs/other-domain.your-domain.com.crt
 

—–BEGIN RSA PRIVATE KEY—–

—–END RSA PRIVATE KEY—–


+++++

13. Standby haproxy-serv01 node 1

[root@haproxy-serv01 certs]# pcs cluster standby haproxy-serv01

14. Renew Generate CSR out of RSA PRIV KEY and .CRT for second domain other-domain.your-domain.com

# for haproxy-serv01 prod :443 renew listeners
[root@haproxy-serv02 certs]# openssl x509 -x509toreq -in other-domain.your-domain.com.crt  -out domain-certificate.com.csr -signkey domain-certificate.com.key


And repeat the same steps e.g. fill the CSR inside the domain registrer and get the certificate and move to the proxy, check the fingerprint if necessery
 

[root@haproxy-serv01 certs]# openssl x509 -noout -fingerprint -sha256 -inform pem -in other-domain.your-domain.com.crt
SHA256 Fingerprint=60:B5:F0:14:38:F0:1C:51:7D:FD:4D:C1:72:EA:ED:E7:74:CA:53:A9:00:C6:F1:EB:B9:5A:A6:86:73:0A:32:8D


15. Check private key's SHA256 checksum

# openssl pkey -in terminals-priv.KEY -pubout -outform pem | sha256sum
# openssl x509 -in other-domain.your-domain.com.crt -pubkey -noout -outform pem | sha256sum

# openssl pkey -in  www.your-domain.com.crt-priv-KEY -pubout -outform pem | sha256sum

# openssl x509 -in  www.your-domain.com.crt -pubkey -noout -outform pem | sha256sum


16. Check haproxy config is okay before reload cert


# haproxy -c -V -f /etc/haproxy/haproxyprod.cfg
Configuration file is valid


# haproxy -c -V -f /etc/haproxy/haproxyqa.cfg
Configuration file is valid

Good so next we can the output of status of certificate

17.Check old certificates are reachable via VIP IP address

Considering that the cluster VIP Address is lets say 10.40.18.88 and running one of the both nodes cluster to check it do something like:
 

# curl -vvI https://10.40.18.88:443|grep -Ei 'start date|expire date'


As output you should get the old certificate


18. Reload Haproxies for Prod and QA on node1 and node2

You can reload the haproxy clusters processes gracefully something similar to kill -HUP but without loosing most of the current established connections with below cmds:

Login on node1 (haproxy-serv01) do:

# /usr/sbin/haproxy -f /etc/haproxy/haproxyprod.cfg -D -p /var/run/haproxyprod.pid  -sf $(cat /var/run/haproxyprod.pid)
# /usr/sbin/haproxy -f /etc/haproxy/haproxyqa.cfg -D -p /var/run/haproxyqa.pid  -sf $(cat /var/run/haproxyqa.pid)

repeat the same commands on haproxy-serv02 host

19.Check new certificates online and the the haproxy logs

# curl -vvI https://10.50.18.88:443|grep -Ei 'start date|expire date'

*       start date: Jul 15 08:19:46 2022 GMT
*       expire date: Jul 15 08:19:46 2025 GMT


You should get the new certificates Issueing start date and expiry date.

On both nodes (if necessery) do:

# tail -f /var/log/haproxy.log

Saint Sergius of Radonezh The WonderWorker (Sergij Radonezhki) a quick helper saint in life problems, pupils, students in educational hardships, exams and Orphans. A short living of father of Russian monasticism

Wednesday, July 6th, 2022

Saint-Sergius-of-Radonezh-icon-Russian-saint-hermit

Saint Sergius of Radonezh the Wonderworker (born in 1314, passed to Christ in 1392) is an ancient saint with enormous importance for the whole Christiandome and the Eastern Orthodox World.
He is perhaps the greatest ascet of the Russian land and a spiritual star who shone over the whole world sharing the and increasing the faith of Christ to shine upon many by his holy prayers. Saint Sergij Radonezhki is sometimes called in Russia the Abbot of the Russian land, for the reason he become the initiator (creator) and first abbot of the most notable and biggest monastery of Russia the Monastery of Saint Sergij of Radonezh in the XIV century.

In the Church service sung books, he is for that reason glorified calling him "a bright beacon of the Russian land, shining through its miracles like a second sun".  Saint Sergius Radonezhki's monastery was established in Glory of the Most Holy Trinity and the great ascet following the earlier example and spiritual tradition bequeted of the Great ascet Saint John of Rila (Ioan Rilski)  who lived in IX century in (876 – c. 946)  who established in the Rila mountain, the most famous Rila Monastey in Bulgaria, saint Sergius started his spiritual endurance in Russia as an ascet and did not initially planned to create a monastery, but God who sees everything seing his great ascetism sent him monks, willing to learn true spiritual life and that is how the Radonezh Monastery was born. Monastery quickly become aprototype of a new, amazingly pure and strict monastic life in Holy Russia, centered near in Moscow and the monastery which become a Laurel (the slavonic word translated as Lavra), (meaning the inhibitants of the monastery were exceeding 1000 monks) since y. 1744, become known in worldwide as Holy Trinity-Sergius Lavra.

Saint-Sergius-of-Radonezh-Lavra-Sveti-Sergieva-Lara-Holy-Trinity-monastery

Just as the spiritual heart of Bulgaria is located in the Rila Holy Monastery (Rilski Manastir), where are the the holy incorruptable relics of most glorified saint Reverend John of Rila the Wonderworker, the Sergius Trinity lavra has emerged as most important spiritual center of Russia Kingdom and later Russian Empire. What was the spiritual significance for preventing the Orthodox Christian faith and shedding hope via the dark ages of Ottoman Turks slavery of Bulgaria,  Reverend  Sergiy Radonezhki was for Russia especially in the hard times when Russia was a small country and fought for their freedom and independence from the Tatars and other surrounding nations, who were constantly destroying parts of the then small kingdom of Rus. Thousands of pilgrims with reverence and gratitude have come and continue to come to the Trinity-Sergius Lavra for worship, near the monastery now is established the city of Sergiyev Posad (inhabited today with about 103 000 people).

Saint_Sergius-of-Radonezh-the-Wonderworker-Sveti-Sergij_Radonezhki-Chudotvoretz-aliased-abbot-of-Russia-land

The shroud of the holy relics of Saint Sergius of Radonezh XV century

Prayers to St. Sergius of Radonezh are famous to protect you from any life problems.  People pray to the saint to protect children from bad influences and from failures at school. The prayer to the saint who was a model of humility helps in achieving humility and subduing the pride of ourselves and others.

Icons-and-frescoes-of-the-Holy-Trinity-Church-Radonezh-monastery-author-famous-iconograph-st-Andrey-Rublev

The very famous Holy Trinity unique icon held in main Church of Radonezh monastery painted by most famous Russian iconographer Saint Andrey Rublev

The icons and frescoes in the Trinity Cathedral in St Radonezh Lavra are unique piece of medieval art and were the work of Reverend Andrei Rublev and Daniil Chernyov in 1425. The main church icon that was painted in "praise of Rev. Sergius" is the old testamental visit the Holy Trinity to Abraham and Sarra icon – which is among the most famous and unique work of Russian icon painting. 

The Church feast of the Saint Sergij

The memory of St. Sergius of Radonezh is honored 4 times a year:

1. October 8 – on the day of his presentation to God
2. June 5 – together with the feast of the Rostov-Yaroslavl saints
3. October 18 – on the day of the discovery of the relics of St. Sergius of Radonezh
(interestingly just 1 day after in the Church calendar is 19 of October is the main feast of Saint John of Rila)
4. and on July 19 – together with the Radonezh Saints Church.

Saint_Sergius-of-Radonezh-holy-relics-raque

Reliquary with the incorruptable Holy Relics of Saint Sergij of Radonezh, kept in the Saint-Sergieva-Lavra monastery

Saint-Sergij-Radonezhki-holy-relics-opened-relics-raque

Opened reliquarium coffin with the incorruptable relics of saint Serigius of Radonezh (the feasts are opened for veneration by pilgrims 4 times a year during the saint feast veneration)

saint-Sergij-Sergius-Radonezhki-ikona-the-appearance-of-the-mother-of-God-to-saint-Sergius

The Apparition of the Mother of God to St. Sergius 16th century icon

 

Short Living (Biography) of Saint Sergius Radonezhki

Sergius of Radonezh was one of the most famous political and historical figures of the 14th century.
He is the founder of the Trinity-Sergius Lavra, teacher and mentor of many Russian saints canonized by the Church.
According to ancient tradition, mainly from his student Epiphanes, Rev. Sergius was born in 1314 in the village of Varnitsa, Rostov in the family of the famous Rostov boyars Kiril and Maria (who are also canonized as saints) and in whose memory there are Churches consecrated in Russia and Belarus.

His parents named him Bartholomew at birth.
Although the family was noble, they lived very modestly and were very religious.
At the age of 7, Sergius and his brothers started going to school, and while his two brothers' education came relatiely easily, it did not for Bartholomew as for bartholomew learning was hard.
Suffering for his hardships to study even though he put great efforts to do so, as he did not have any other means to become proficient pupil like his brothers, wishing to change this, the boy fervently prayed to God to give him reason and strength to advance in learning.

According to the chronicles, the boy once he was looking for the lost horses of his father met an old schimonk elder who was praying to God. The boy asked the old man to pray for his success in school. The the unusual bright monk prayed and blessed him. Since then, the boy began to progress in his studies quickly.

In 1330, Bartholomew's parents moved to the village of Radonezh, near Moscow. Soon his parents died, and he, together with his older brother Stefan, left the inheritance to the youngest brother and took up a hermit life in the nearby forest, where they built a cell, and then built a small church, which was consecrated with the name of the Holy Trinity.

Soon Stefan decided to abandon the hermit life and went to the Moscow Epiphany Monastery, where he became abbot.
Bartholomew stayed and in 1337, at the age of 23, he was ordained a monk with the name Sergius.

The young ascetic spent more than a year in complete solitude. His life consisted of prayers, fasting and hard work.
Sergius became famous in the surrounding area and soon other monks began to come to him for advice and soon the brotherhood numbered 12 people. Each monk lived in a separate cell, and together they gathered for worship.
This is how the famous Sergius-Troitskaya Lavra was founded. In 1354, Sergius was ordained abbot.

As the biography testifies, Reverend Sergius of Radonezh performed miracles even in his earthly life:

"…
Pilgrims came, they saw the poverty of the desert, but they also saw the peace and grace among the brothers, and they brought this leaven into their native families like light, like salt. And the name of the God-pleaser became glorious throughout Russia and many came to him with faith. Once, by praying, the old man healed a seriously ill person, and another time he healed a mad nobleman who was not in his right mind, ranting and fighting, so that ten they couldn't hold the man back."

St. Sergius of Radonezh was honored with a vision of the Most Holy Mother of God, who appeared to him during a night service and said:
"Do not be afraid, My chosen one. I have come to visit you. Do not grieve, because your prayer for the students and the monastery has been heard; and your abode shall abound in all things; not only in your life, but also in your presentation before God. I will not leave this place, and will irrevocably supply all that is necessary, preserve and protect it with my covering."


Saint Sergius of Radonezh knew how to act with "quiet and meek words" even on the most hardened and cruel hearts and in this way reconciled even ruling country region principles and kings at war.
Thanks to him, all the princes united before the Kulikovo battle, thanks to whom Russia managed to establish itself as a leading world country in the mid-centuries recognizing the main role of the Moscow prince Dimitriy.
The Russian army received a blessing from the venerable Sergius of Radonezh before the upcoming battle with the Tatars.

He predicted (prophecised) the victory over the Tatars, and on September 8, 1380, on the feast of the Nativity of the Virgin, the Russian troops defeated the Tatars at Kulikovo field, marking the beginning of liberation from the Tatar yoke. In other words at the time when Bulgarian Empire had crashed and has been enslaved, the Trnov kingdom fall down in 1393 and Vidin's kingdom of Bulgaria in 1396, just few years before the opposite hapened in Russia. Already baptized by many Bulgarian clergymen and given the Old Bulgarian Church language so called in newer times the Church Slavonic Books, Russia together with its spiritual flourishment managed to liberate from the Tatars and increasing gruadally in influence, countryland and power.
Prince Dimitrii deeply respected his spiritual father – the Reverend Sergius of Radonezh, who was also the godfather of his children.

Sergius of Radonezh died on September 25, 1392, reaching a very old age.
He predicted his death 2 years before and appointed his successor – his student Reverend Nikon.
St. Sergius of Radonezh was buried in a monastery founded by him, and 30 years later his body and clothes were found incorruptible and odoring with heavenly odor.

This happened in 1422 during the construction of the new and enlargened "Life-giving Trinity" church.

Shortly before the construction began, Sergius of Radonezh appeared in a dream to a pious man, telling him to convey the following words to the brothers:
"Why do you leave me so long in the grave, buried in the ground with all this water, it is narrow here for my body. "

When they dug for the foundations, the imperishable relics and clothes of St. Sergius of Radonezh were found unharmed, and the grave was full of water.

During the consecration of the Trinity Church, the relics of the saint were transferred to it, where they lay even to this day.

Let by the Holy Prayers of Saint Sergius of Radonezh the world, find more peace, love, faith and brotherhood, which we desperately need in these day of the absurd brother shedful war in Ukraine. 

Holy Reverend Father Sergij of Radonezh, pray the Lord Jesus Christ to save our souls and grant repentance to us sinners and peace to rule again in our hearts !