Archive for October, 2021

26 October the Feast of Holy Great-Martyr Demetrius the Myroblyte known also as Demetrius of Thessaloniki

Tuesday, October 26th, 2021

Sveti Dimitar Solunski_kopie-ikona

Bulgarian icon of Saint Demetrius

Every 26 of October in the Bulgarian Orthodox Church and whole Bulgarian nation we honor deeply the memory of martyrdom of Saint Demetrius the Myroblyte (meaning 'the Myrrh-Gusher' or 'Myrrh-Streamer'; a term that stemmed from 3rd century – 306 y. the year of Maryrdom of this great saint. Saint Demetrius over the century has been one of the most venerated saints in the Eastern Orthodox Church and since the Christianization of Bulgaria his glory also spread quickly throughout the Bulgarian Empire lands.

Saint-Demetrius-and-Saint-George-icon-St-George-killing-Dragon-saint-Dimitar-killing-a-man

During the Middle Ages, he came to be revered as one of the most important Orthodox military saints, often paired with Saint George of Lydda and for that in many of the Orthodox Churches worldwide there are icons of the two saints painted together holding their warrior equipment spear, shield and sward .

saint_Demetrios_of_Thessaloniki_icon_on-graved-stone

St. Demetrius (Dimitar in Bulgarian) feast day is 26 October for Eastern Orthodox Christians, which falls on 8 November for those following the old calendar. In the Roman Catholic church he is most commonly called "Demetrius of Sermium" and his memorial falls on 8 October, which seem to coincide with my Birthday 🙂

Demetrius was born to pious Christian parents in Thessaloniki, the Eastern Roman Empire region Macedonia in 270 (Macedonia has been part of the Bulgarian kingdom and Empire for many centuries).

According to the hagiographies, Demetrius was a young man of senatorial family who became proconsul of the Thessalonica district. He was run through with spears in around 306 AD in Thessaloniki, during the Christian persecutions of Galerian, which matches his depiction in the 7th century mosaics.

Most historical scholars follow the hypothesis put forward by Bollandist Hippolyte Delehaye (1859–1941), that his veneration was transferred from Sirmium when Thessaloniki replaced it as the main military base in the area in 441/442 AD. His very large church in Thessaloniki, the Hagios Demetrios, dates from the mid-5th century. Thessaloniki remained a centre of his veneration, and he is the patron saint of the city.

After the growth of his veneration as saint, the city of Thessaloniki suffered repeated attacks and sieges from the Slavic peoples who moved into the Balkans, and Demetrius was credited with many miraculous interventions to defend the city. Hence later traditions about Demetrius regard him as a soldier in the Roman army, and he came to be regarded as an important military martyr. Unsurprisingly, he was extremely popular in the Middle Ages. Disputes between Bohemond I of Antioch and Alexios I Komnenos appear to have resulted in Demetrius being appropriated as patron saint of crusading.

saint-Dymitr_z_Salonik-sankt-Peterburg-icon

Saint Demetrius Russian Icon

Demetrius was also venerated as patron of agriculture, peasants and shepherds in the Greek countryside during the Middle Ages. 

Most scholars still believe that for four centuries after his death, Demetrius had no physical relics, and in their place an unusual empty shrine called the "ciborium" was built inside Hagios Demetrios. What were purported to be his remains subsequently appeared in Thessaloniki, but the local archbishop John, who compiled the first book of the Miracles ca. 610, was publicly dismissive of their authenticity. The relics were assumed to be genuine after they started emitting a liquid and strong-scented myrrh. This gave Demeterius the epithet Myroblyte.

Saint Demetrius used to be a mayor of Thessaloniki and had been very educated for his time, the Roman empire ordered him to find and imprison, torture and eventually kill all Christians in the city who refuse to follow the paganic Roman religion. Being a brave in heart and a being a Christian himself, he refused to follow the unrighteous emperor decree and even on the contrary started to put special efforts for the raising of the Christian faith in the city. 

Despite this position in the still-pagan empire, he remained fervent in faith and works for Christ, encouraging many Christians to endure persecution and even bringing many pagans to the faith.

When Maximian returned from one of his campaigns to Thessaloniki, which he had made his capital, he had pagan games and sacrifices celebrated for his triumph. Demetrios was denounced by pagans who were envious of his success, and he was thrown into prison. While in prison he was visited by a young Christian named Nestor, who asked him for a blessing to engage in single combat with the giant Lyaios (or Lyaeus), who was posing as the champion of paganism. Demetrios gave his blessing and Nestor, against all odds. Nestor succeded to slew his opponent in the arena contrary to any expectations as Lyaios used to kill many, many christians on the circus arena, as David had once defeated Goliath. Saint Demetrius blesses Nestor but warned him he will have to endure a martyrdom after his defeat of Nestor which occured shortly after the defeath of Lyaios, Nestor was captured and martyred for Christ. Being raged out by the killing of Lyaios, the Romans send trooops and killed with spears saint Demetrius while he was praying in the prison.

According to some (Greek) hagiographic legend, as retold by Dimitry of Rostov in particular, Demetrius appeared in 1207 in the camp of tsar Kaloyan of with a lance and so killing him. This scene, known as Чудо о погибели царя Калояна ("the miracle of the destruction of tsar Kaloyan") became a popular element in the iconography of Demetrius. He is shown on horseback piercing the king with his spear, paralleling the iconography (and often shown alongside) of Saint George and the Dragon.


The reason of High veneration of Saint Demetrius in Bulgaria today ?
 

The godly life he led, together with his military virtues and martyrdom, led the people of Thessaloniki to declare him their saint-warrior and patron. According to the beliefs of the local centuries, the saint defended Thessaloniki, performing miracle after miracle, but in August 1185 something unheard of happened. The second richest and most important city in the empire after Constantinople was captured by the Normans and subjected to unprecedented looting. The Church of St. Dimitar was burned and the relics of the saint were scattered. The medieval Greek, who was inclined to seek God's intervention everywhere, was spiritually broken. The Romans saw the fall of Dimitrov as a punishment for their sinfulness. It is clear to them that St. Dimitar left them.

Meanwhile, in the north, the memories of the old Bulgarian kingdom were more than alive, and it became increasingly difficult for the Bulgarians to tolerate the Roman rule. The moment for a mass uprising was ripe. According to Nikita Honiat, there were three key events at the beginning of the uprising. The first concerned the desire of the brothers Peter and Assen (prominent Bulgarian boyars) to be included in the proniat lists of the empire and to receive a small landed estate at the foot of the Balkan Mountains. To this end, most likely in the autumn of 1185, they appeared in person before Emperor Isaac II Angel in Kipsela, just as he was preparing to march against the Normans who had conquered Thessaloniki. The refusal to comply with their demands provoked sharp resentment in the younger brother Assen, who personally threatened the emperor with rebellion. This unheard of behavior of the young boyar was punished with a slap.

Medieval_Bulgarian_King_Asen_portrait

King Ivan Assen I (Tsar of Bulgaria 1187/1188–1196)

The second important event was the imposition of additional taxes on the livestock of the population on the occasion of the emperor's wedding to the Hungarian Princess Margaret. This led to the outbreak of strong and mass discontent among the population of Moesia. The two brothers knew very well what they were doing and used the mass discontent to make their threat a reality. However, the insults, material hardship and the presence of two brilliant leaders in the face of Assenevtsi were not enough for a revolt.

Bulgarian-icon-of-saint-Dimitar-the-Myrrh-Bringer-Ikona-na-sv.Dimitar-Mirotochivi

Bulgarian Medieval Icon of Saint Demetrius the Myrrh-Bringer

The Bulgarians also had to receive a "divine" guarantee for their work. They believed that the Lord should show them that they were chosen and worthy of their freedom, that they not only could, but should take up arms against the Byzantine Vasilevs. And the sign was not late. On October 26, 1185, Assenevtsi, together with a large crowd, gathered in Tarnovo to consecrate the newly built church "St. Dimitar. Meanwhile, a miraculous icon of the saint appeared in the city. It was alleged that she had left Thessaloniki, conquered by the Normans, and found her home in the new temple of the Bulgarians.

The religious consciousness of the medieval Bulgarian interpreted this as a refusal of St. Dimitar to defend the Romans and a sacred guarantee that the saint will protect the Bulgarians in their cause for freedom. And indeed the old church in Thessaloniki had collapsed and plundered, the Romans were punished, the Empire was humiliated.

 

Those gathered in the church began to shout and call for the rejection of the yoke and for the restoration of the glory of the old kings. In this atmosphere of patriotic enthusiasm, the older brother, Todor (named Peter), placed a golden tiara on his head, put on a red cloak, and put on the purple shoes that only the Byzantine Vasilevs could wear. Thus, after 167 years of interruption of the throne of the Bulgarian kings, a Bulgarian ascended again. The coronation of Peter as king and the beginning of the great uprising of the Bulgarians was one of those moments in history when all accounts end and only faith gives the people the courage to take the hand outstretched by the uncertainty of the future and follow the path indicated by her, not knowing where he was taking her.

Niketa Choniates writes: “With such (divine) prophecies the whole nation was won for the cause and all raised their swords. And because their rebellion was successful from the very beginning, the Bulgarians believed even more that God had approved their freedom. "

At first, Isaac II Angel was unable to respond to the uprising, as he had to deal with the Normans and the usurper of Cyprus, Isaac I Komnenos. It was not until December 1185 that Vasilevs sent his uncle Sevastocrator John against the rebels. However, no battle took place because the Sevastocrator was recalled on suspicion of rebellion. At the head of the second army was Caesar John VI Kantakouzenos, who went to Hemus, but was defeated in a night attack by Assen-evtsi. The Bulgarians took the lives of most of the Roman army, and its commander managed to escape by abandoning the entire convoy. A third army of the great Byzantine general Alexy Branas was also sent, but it turned against the emperor and marched to Constantinople instead of Tarnovo.

Byzantine_themes-in-Bulgaria-on_the_Balkans-map-11th-12th-century
The Byzantine themes (or districts) of Bulgaria and Paristrion 

Paristrion – (Greek: Παρίστριον, lit. 'beside the Ister'), or Paradounabon/Paradounabis (αραδούναβον / Παραδούναβις), which is preferred in official documents, was a Byzantine province covering the southern bank of the Lower Danube (Moesia Inferior) in the 11th and 12th centuries.

It was not until 1186 that the emperor personally led a large army and decided to deal with the Bulgarians once and for all. His campaign forced the brothers to retreat across the Danube to their Kuman allies, and Isaac II Angel plundered Moesia and returned to Constantinople. According to the story of Nikita Honiat, the emperor was so arrogant of his success that he met with ridicule the reminder of Vasily II the Bulgarian assassin that the Bulgarians would revolt and that one day they would be liberated.

At that time, Assen's personality became more and more prominent, and he became the real leader of the rebellion. In the summer of 1186, the Assenevs crossed the Danube again, conquered the plain and set their goal to bring the endeavor to a successful conclusion. Niketa Choniates says:

"And then they returned to their homeland Moesia; finding the land abandoned by the Roman armies, they took on even greater confidence, leading their Cuman auxiliary detachments as if they were legions of demons. They did not simply want to secure their possessions and establish control over Moesia; They wanted to devastate the Roman territories and unite the political power of Moesia and Bulgaria in one empire as before. "

Isaac Angel's second campaign was not long in coming. In the autumn of 1186 he set out again against the two brothers, passing through the fortress of Beroe and heading for Serdica (today's Sofia), from where he intended to cross Hemus and attack Tarnovo. The winter of 1186, however, blocked the passages and forced the emperor to abandon his endeavor for another year. With the arrival of the spring of 1187, the Romans crossed the mountain and besieged the Lovech fortress. However, the Bulgarian troops offered unprecedented resistance and after a three-month siege Isaac II Angel had to ask for peace.

Saint_Demetrius_Tarnovo_Church-Klearchos

The Church Saint Demetrius built by King Asen I in memoriam of great Miracles of Bulgarians victories over Byzantines
Church is located near the Tarnovo Fortress of Trapezica

The Church slavonic written sources tells how the brothers spread the word a patron saint of Thessaloniki – St. Demetrius, came to Tarnovo to help the Bulgarian people to be liberated…
 

Thus, most probably, the Lovech armistice was signed in front of the city walls, which de jure recognized the Bulgarian power north of the Balkans. The long road to freedom began on that distant St. Dimitrov's Day in 1185. he was finally walked away. St. Dimitar became the patron of the Asenevtsi dynasty and one of the most beloved Bulgarian saints, and the Bulgarians proved to the world that their pursuit of freedom is nothing but a great national feat, in which with a true understanding of the necessary and possible, with steady faith and unwavering energy in the design and implementation, the political and spiritual resurrection of the Second Bulgarian Kingdom was reached.


Saint_Demetrius_Bulgarian_icon-1824_Sveti_Dimitar

Saint Demetrius Bulgarian icon year 1824

St. Demetrius is depicted on horseback spearing a man, not because he ever a killed a man but because he blessed Nestor to win over the Gladiator Lyaeus. The Church decided to commemory the memory and bravery of Saint Nestor who also confessed Christ in his martyrdom every on the next day after the memory of st. Demetrius is celebrated. Saint Nestor even today is celebrated in the Church calendar on 27-th of October.

In Bulgaria the veneration of saint Demetrius was of high esteem especially in the Second Bulgarian Empire and many churches and monasteries has been built around the country (counting at few hundred temples and monasteries) with him being their patron.

Saint_Demetrius-Holy-relics-relics

Saint Demetrius Holy Relics in the St. Demetrius Church in Thessaloniki Greece (the white papers are names of people who ask for help from the saint)

Saint Demetrius is famous in Thessaloniki and highly venerated every year during his feast as he has been summoned by the Church to protect the city on multiple occasions which he did so far during pandemics such as the Black Death and during invasion of alien (non-Christian) nations.

It is mostly remarkable that every year during his feast day, a great miracle happens from the exact place where he was martyred (situated in the Church named after him), a myrrh with heavenly odor is streaming which is taken by believers for oilment and as a blessing carefully kept until the next year feast of the saint.
Because of the high amount of myrrh outflow a special pool was kept to keep the oilment sparring out of his holy relics.

As Saint Demetrius has helped multiple times to many of their saints as we know from history, especially in times of epidemies and pandemies like it is now let by his holy prayers those who venerate him and the people worldwide finds Healing and relief and an Enlightment and blessing from the light of Christ, just like Nestor found in his blessing !

Holy Martyr Demetrius of Thessaloniki pray the Lord for us the sinners !!!

How to move transfer binary files encoded with base64 on Linux with Copy Paste of text ASCII encoded string

Monday, October 25th, 2021

base64-encode-decode-binary-files-to-transfer-between-servers-base64-artistic-logo

If you have to work on servers in a protected environments that are accessed via multiple VPNs, Jump hosts or Web Citrix and you have no mean to copy binary files to your computer or from your computer because you have all kind of FTP / SFTP or whatever Data Copy clients disabled on remote jump host side or CITRIX server and you still are looking for a way to copy files between your PC and the Remote server Side.
Or for example if you have 2 or more servers that are in a special Demilitarized Network Zones ( DMZ ) and the machines does not have SFTP / FTP / WebServer or other kind of copy protocol service that can be used to copy files between the hosts and you still need to copy some files between the 2 or more machines in a slow but still functional way, then you might not know of one old school hackers trick you can employee to complete the copy of files between DMZ-ed Server Host A lets say with IP address (192.168.50.5) -> Server Host B (192.168.30.7). The way to complete the binary file copy is to Encode the binary on Server Host A and then, use cat  command to display the encoded string and copy whole encoded cat command output  to your (local PC buffer from where you access the remote side via SSH via the CITRIX or Jump host.). Then decode the encoded file with an encoding tool such as base64 or uuencode. In this article, I'll show how this is done with base64 and uuencode. Base64 binary is pretty standard in most Linux / Unix OS-es today on most Linux distributions it is part of the coreutils package.
The main use of base64 encoding to encode non-text Attachment files to Electronic Mail, but for our case it fits perfectly.
Keep in mind, that this hack to copy the binary from Machine A to Machine B of course depends on the Copy / Paste buffer being enabled both on remote Jump host or Citrix from where you reach the servers as well as your own PC laptop from where you access the remote side.

base64-character-encoding-string-table

Base64 Encoding and Decoding text strings legend

The file copy process to the highly secured PCI host goes like this:
 

1. On Server Host A encode with md5sum command

[root@serverA ~]:# md5sum -b /tmp/inputbinfile-to-encode
66c4d7b03ed6df9df5305ae535e40b7d *inputbinfile-to-encode

 

As you see one good location to encode the file would be /tmp as this is a temporary home or you can use alternatively your HOME dir

but you have to be quite careful to not run out of space if you produce it anywhere 🙂

 

2. Encode the binary file with base64 encoding

 [root@serverB ~]:# base64 -w0 inputbinfile-to-encode > outputbin-file.base64

The -w0 option is given to disable line wrapping. Line wrapping is perhaps not needed if you will copy paste the data.

base64-encoded-binary-file-text-string-linux-screenshot

Base64 Encoded string chunk with line wrapping

For a complete list of possible accepted arguments check here.

3. Cat the inputbinfile-to-encode just generated to display the text encoded file in your SecureCRT / Putty / SuperPutty etc. remote ssh access client

[root@serverA ~]:# cat /tmp/inputbinfile-to-encode
f0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAMGEAAAAAAABAAAAAAAAAACgXAgAAAAAAAAAAA
EAAOAALAEAAHQAcAAYAAAAEAAA ……………………………………………………………… cTD6lC+ViQfUCPn9bs

 

4. Select the cat-ted string and copy it to your PC Copy / Paste buffer


If the bin file is not few kilobytes, but few megabytes copying the file might be tricky as the string produced from cat command would be really long, so make sure the SSH client you're using is configured to have a large buffer to scroll up enough and be able to select the whole encoded string until the end of the cat command and copy it to Copy / Paste buffer.

 

5. On Server Host B paste the bas64 encoded binary inside a newly created file

Open with a text editor vim / mc or whatever is available

[root@serverB ~]:# vi inputbinfile-to-encode

Some very paranoid Linux / UNIX systems might not have even a normal text editor like 'vi' if you happen to need to copy files on such one a useful thing is to use a simple cat on the remote side to open a new File Descriptor buffer, like this:

[root@server2 ~]:# cat >> inputbinfile-to-encode <<'EOF'
Paste the string here

 

6. Decode the encoded binary with base64 cmd again

[root@serverB ~]:# base64 –decode outputbin-file.base64 > inputbinfile-to-encode

 

7. Set proper file permissions (the same as on Host A)

[root@serverB ~]:#  chmod +x inputbinfile-to-encode

 

8. Check again the binary file checksum on Host B is identical as on Host A

[root@serverB ~]:# md5sum -b inputbinfile-to-encode
66c4d7b03ed6df9df5305ae535e40b7d *inputbinfile-to-encode

As you can md5sum match on both sides so file should be OK.

 

9. Encoding and decoding files with uuencode


If you are lucky and you have uuencode installed (sharutils) package is present on remote machine to encode lets say an archived set of binary files in .tar.gz format do:

Prepare the archive of all the files you want to copy with tar on Host A:

[root@Machine1 ~]:#  tar -czvf /bin/whatever /usr/local/bin/htop /usr/local/bin/samhain /etc/hosts archived-binaries-and-configs.tar.gz

[root@Machine1 ~]:# uuencode archived-binaries-and-configs.tar.gz archived-binaries-and-configs.uu

Cat / Copy / paste the encoded content as usual to a file on Host B:

Then on Machine 2 decode:

[root@Machine2 ~]:# uuencode -c < archived-binaries-and-configs.tar.gz.uu

 

Conclusion


In this short method I've shown you a hack that is used often by script kiddies to copy over files between pwn3d machines, a method which however is very precious and useful for sysadmins like me who has to admin a paranoid secured servers that are placed in a very hard to access environments.

With the same method you can encode or decode not only binary file but also any standard input/output file content. base64 encoding is quite useful stuff to use also in bash scripts or perl where you want to have the script copy file in a plain text format . Datas are encoded and decoded to make the data transmission and storing process easier. You have to keep in mind always that Encoding and Decoding are not similar to encryption and decryption as encr. deprytion gives a special security layers to the encoded that. Encoded data can be easily revealed by decoding, so if you need to copy between the servers very sensitive data like SSL certificates Private RSA / DSA key, this command line utility tool better to be not used for sesitive data copying.

 

 

19 October Saint John of Rila Тhe Wonderworker – A little known Hermit saint of Great importance and fervent prayer intercessor for humanity

Wednesday, October 20th, 2021

Saint-Ioann_John_of_Rila_Rylsky_icon

I've already blogged a number of times articles of Saint John of Rila as he is The Greatest venerated Bulgarian Saint as of today his birthplace nearby today's Ruen Monastery as well as my inspiration from the Night Vigil prayer gatherings in the Monastery in his Memoriam in German. But I'm pretty sure this spiritual giant  is under estimated and little known to the public not only the non-bulgarian English speakers but even to Bulgarian and the saint deserves much greater veneration and honour than it has received through the ages for the fact he has been among the greatest prayer intercessors for the world until the beginning of ages.

For his immerse hermit achievements in a similar way as Saint Anthony the Great he has been granted All the innumerous Gifts of The Holy Spirit.
The humble hermit even in his life time he has been defined as "Angel in flesh" and a "Citizen of heaven".
The life of Saint John has been a X-th century continuation of the life of Saint Simeon the Stylites and Saint Simeon Stylites the Younger attempted to be repeated as far as possible by more contemporary saints such as Saint Siluan the Athonite.
 

He has been a teacher (even though he ran away from people) all his life. He has been the major establisher of Monasticism in Bulgaria, 

In 18 of August year 946 Anno Dommini saint John has departed this world. Dying with a painless way called in Church language with the term Dormition or Assumption – (meeling he did not felt any kind of fleshly pain as normally one goes through when he ends his life unsanctified). His Dormition  in a similar way as The Mother of God (Theotokos), an end of life who has been granted throughout human history only to few other saints.

Assumption_Dormition_of_Saint_John_of_Rila_by_Toma_Vishanov

Saint John who spend his hermit lifetime in Mountain of Rila was born circa y. 876 AD and passed away to Christ on August 18, 946 A.D.
Thirthy four years after his dormition, he appeared in a dream to his pupils and has commanded them to dig out his incorruptable body and move it to in Sredetz (today the city of Sofia capital of Bulgaria). In Sredetz the holy relics of the saint has been kept in honour for 200 years.

Saint John has been a contemporary to Saint King Boris I the Baptizer of Bulgaria, King Simeon the Great (Son of King Boris, who raised Bulgaria to Empire reaching to 3 Oceans, who has been one of the largest empires of All times) and King Petar (Peter) the First. His glory as a Wonderworker who has been desired to been seen by many to receive physical healing or spiritual advice has been mostly through the rule of King Petar I (927 – 969). In Medieval times the Glory of Saint John of Rila has been widely known throughout all the Christianized Lands (including Byzantine Empire) and the West in the Catholic Church. The implication of his importance as a medieval healer saint and a model for hermit life and saintship and image for truthful spiritual life is well seen as his Biography (Living) is seen as he has about 15 Known different livings saying varyious details about his lifetime written in different centuries in Old Bulgarian Language and in Middle centuries in the new transformed form of Old Bulgarian writting called Church Slavonic, where saint has been simply adressed by monk name Ioan (John).

According to his biography we found until the age of 25 Saint John has been a simple shepherd. Until the yough age in his heart John had the burning love in his heart and a desire to dedicate himself completely to God. When his parents has passed away, he gave away all his received inheritance to the poor and sick people and entered as a aprentice in the monastery of today Boboshevski Saint Dimitar under the Ruen Monastery in the Mountain of Vlahina.

There in he received a Theological education, studied the liturgical books, received a spiritual dignity and prepared himself for the great spiritual mission of hermitage and life long prayer for the world. Receiving the monks schima, he gave away completely to fasting and prayer, initially establishing himself in Vitosha Mountain (most likely in today Monastery of German Monastery St. John of Rila (convent of Mount Athos Monastery of Zograph) nearby Sofia

Saint_Ivan_Rilski_Zemen-ikona

St. John of Rila Wall Painting XIV century from the Church Zemen Monastery Saint John the Theologian

In German Monastery often times the ancient Christian tradition of Night Vigil is followed on hist feast Major Christian feasts even today.
Later he has been for a while in the small Village  "Dobarsko" near The Europe Famous Resort of Bansko.
A short after 20 km of Dobarsko he has established The Most Famous Monastery for the Balkans, the Rila Monastery, a pivotal place whose light has shined upon practically all parts of the land as it become a school for tens of thousands hundred monks throughout the last 20 centuries. Many manuscript works prepared in Rila Monastery has been later on transferred in Today Ukraine, Russia, Serbia, Greece and even the Western Catholic lands.
There is no exact place in history when Saint John was officially canonized as in Medival Times, saints did not go through an official canonization but has been testified by the multitudes of peoples who has been cured by their wonder-working prayers. By the prayers of Saint John many has been able to conceive childs, leppers get cured and could walk, eye diseases were healed, full health has been restored or the gift of faith has been giving, not to mention the multitudes of people who came to Christian faith thanks to the prayers of St. John.

The-Meeting-of-Saint-John-of-Rila-and-King-Peter-First-of-Bulgaria

King Petar I-st meets the Hermit Holy Father John of Rila (Wall Painting)

The story of the powerful medieval Bulgarian ruler Peter the First who travelled 450 Kilometers only to see the saint is well known. Travelling about 450 kilometers from the capital of Bulgaria Veliki Preslav to the Cave of Saint John of Rila only to meet him.

Cave-of-Saint-John-of-Rila-Peshterata-na-sv-Ioan-Rilski

The Cave of Saint John of Rila Cave exit

The monk however refused to meet him due to his immerse humility, where King Peter I-st was able to talk in a long distance via a valley with the saint, offering him gold and fruits. St. John refused to receive the king's gold, as he said he did not left the world and everyhing to it to attain gold or anything but to save his soul. He received the fruits not wanting to insult the king. Many people who deserved to learn a true monasticism and ascetism has quickly reached the saint and asked them to create a monastic brotherhood and wanting to be useful his poor brothers and having revelation of God that it is the will of God to establish the monsatery he accepted and become the first Abbot of Rila Monastery.

He passed to the Savior Christ about in year on 18 of August 946 AD being aged about 70 years old he has been buried in the Church porch in a stone tomb, preserved even today.

The-Testament-of-Saint-John-of-Rila-Given-to-his-pupils-monks
Saint John Giving his Written Testament to his Monks (The Testament is Famous for the quote "I beg you children to mostly fear from the snake of avarice which is the root of all evils".


Until his death he has departed from the brotherhood in complete solitude and wrote his famous "Testament".  The Testament of Saint John of Rila he left for his pupils on 25 of March on the feast of the Annunciation in year 941. The text of the Saint John Testament is preserved in a copy manuscripts dated from the XVIII and XIX century and contains a multitude of quotes from The Scriptures of Old and New Testament

Engravement-of-the-Living-of-Saint-John-of_Rila_monastery-icon

The Living of Saint John of Rila


Saint_John_of-Kronstadt-Ioann_Kronshtadtskij

 

A little known fact is that the Russian All Famous Saint Father John of Kronstadt (also a wonderworker) (Ivan Ilyich Sergiyev, born: 31 October [O.S. 19 October] 1829 + Passed to Christ: 2 January 1909 [O.S. 20 December 1908]) has received his Baptizmal name after Saint John of Rila as he has been born on his feast day and his parents following the Russian tradition to name the baby after the Saint honour in the Church Calendar gave the name John of St. John of Kronstadt. St. John of Kronstadt deeply venerated Saint John of Rila and asked his prayer intercession alway until his life time.
​In 1900, Father John established the women's monastery of Ioannovsky Convent, named for his name patron John of Rila, as a branch of the Sura Monastery of St. John the Theologian, where he was later also buried.

Recognition of Saint John of Rila Holiness shortly after his death


After his death in y. 946, st John has been buried nearby the established by him Holy Rila Convent, shortly after King Peter I-st ordered to be transferred to Sredetz (Sofia). Most likely this can be considered for original date of canonization of the saint. About y. 989 – 992 y. his holy relics has been transferred by King Samul and then Patriarch German-Gabvril (German-Gabriel) in Sredetz. This happened on 19 of October and since then on this date is venerated his bright memory.
Initially his holy relics has been kept for venerateion in the Church of "Saint Great Martyr Georgi the Glorybringer" and later on moved to Church of "Saint Apostle and Evangelist Luke". In the XII century in his honour the Church of "Saint Luka (Saint Luke)" was built near which a monastery has been established. The Byzantine writter Ioannis Skylitzes / John Skylitzes testifies that in Sredetz the Holy Relics of Saint John has miraculously cured emperor Manuel I Komnenos.

In 1183 during the consequential Hungarian-Byzantine war, the Hungarian King Bela III of Hungar conquered Sredetz and moved the holy relics of Saint John in his capitzal Estergom today the Komárom-Esztergom County. According to tradition the local Roman Catholic archibishop declared he has not been aware for the existence of a saint John of Rila and because of his words the saint has punished him with immediate numbness. Once he venerated the holy relics of the reliquary and begged for forgiveness, his speech was recovered. Amazed and worried about this miracle, in year 1187 hungarians transferred back the holy relics of the saint to Tarnovo? Bulgaria.

The-Return-of-Holy-Relics-of-Saint-John-of-Rila

The Return of the Holy Relics of St. John to Trnovo Bulgaria

On 1st of June the Church celebrates the return of the holy relics of saint John of Rila from Tarnovo to Rila monastery. In 1195 Bulgarian king Ivan Asen the First solemnly transferred the Holy relics once again to Tarnovo in the church still being built St. 40 Martyrs (dedicated to 40 Martyrs of Nikomidia). Until the church was completed the holy relics has been kept on Krastec Peak, now the place has a small chapel in honour of St. John of Rila The Wonderworker".

In 1469 year Bulgaria is already invaded by the Ottoman Turks, thus  after a permission from the Sultan, the monks of Rila convent brought the holy relics of the saint from Veliko Tarnovo to his established Rila Monastery, where holy relics are kept to this date and where anyone who needs a spiritual fortification, restorement of health, or suffering from unclean spirits (devils) could ask with faith and hope for healing for the saint and wait for the miracle.
I myself have had the chance to meet a man who has being to a Night Vigil to Rila monastery and testified that he and his life both being in his old age of fifties after not being able to have kids and trying out everything has asked the saint during a night vigil to pray for them to able to conceive a child and the saint heard their prayers and granted them a kid and she conceived right after that and brought to the earth after 9 months a very healthy and lively baby 🙂

Another modern miracle I'm personally aware of made by Saint John is the Gift of Faith in God which was granted to a Greek architect, I've personally met while being on a pilgrimage visit to Holy Mount Athos. The Greek shared that he had never been a believed in God and was a grown atheist, until he had a work duty to do some reconstruction restoration works in Bulgarian Rila monastery. Spending a few months as part of his business-trip duties, he has received the gift of faith. I've met the Greek (unfortunately no longer remember his name), nearby the monastery of Xenophontos.

 

The troparion of our rev. father John of Rila, tone 1

A foundation of the repentance, an example of the tenderness,
a model of the consolation and of the spiritual perfection,
oh Reverend Father, was your equal to the Angels life therefore
staying in prayers and fasts and in tears
pray to the Christ God for our souls.


Today the holy relics are miraculously still incorruptable and emit a specific nice odor  which is typical for many of other saints, a testify to the unbelievers for their saintship. The saints relics in a special reliquarium in the right corner behind the iconostasis icons. The right hand of the saint is kept in nun convent in Island of Tinos Greece and the story how it ended there is a worthy for a small pamphlet and I'll try to say few words on that in some of the next posts.

Let by the Holy Prayers of Saint John of Rila our Home Land Bulgaria and all Countries are granted more Peace, Love, Faith and Kindness and relief from the terrible diseases that are plaguing the world today !

Holy Father John of Rila Pray the mercyful Lord Jesus with All the Saints and The Most Holy Theotokos to save our Souls and grant mercies to us the sinners !

Saint protector of the Family and The receipt of Saint Petka Tarnovska by King Asen II story

Thursday, October 14th, 2021

Chapel-Trun-Sveta-Petka-Ikona-ot-16-17-vek-na-Sveta-Petka-v-skalniq-paraklis-na-Trun-231x300

In these hazy and confused times where the family as institution is failing and it is becoming more and more modern for people to live together without official Civil marriages count are steadily declining not to mention that the Church marriages no matter whether it is a Protestant, Roman Catholic is very rare thing. The attack against families is multi-vector one, it is descredited and being pranked on the TV in movies and serials, in the press. Many of the bad sides of marriage are exposed as well as the incapabilities of any traditional marriage to respond to the modern challenges of the world and hence many choose to not marry. But originally God created man to live in a family Adam and Eve were the first marital couple (even though they did not officially binded it on paper) in the municipality or the Church. Of course there situation was slightly different than today as they were the only couple in the beginning when God created man one would say. But even after that through the Ages Marriage and fidelity that stems out of it has been considered normal. However today normality is being pranked and abnormality is being enforced from all sides. 
As the topic of family does directly concern me as I have a family I thought therefore it is useful to mention again today about the feast of Saint Petka Epivatska (Epivates) famous as Saint Petka of Tarnovo who is considered in our Eastern Orthodox Christian countries as a protectress of family.

The reason why we venerate saint Petka here in Bulgaria is because here holy relics has stayed in Tarnovo for almost 2 Centuries and being in Bulgaria, they have worked many miracles, many of which were done over a family husbands and wifes who had their usual trials, like infertility, problems in family, quarrels etc.
For being famous for that miracles she has been invoked for centuries worldwide by believing Christians for help on their family trials.

Below is shortly the history of how St. Petka Epivatska, holy relics were transferred to Tarnovo and the multiple transfers of her relics until she finally choose to reside in Iash Romania.
 

+++

Posreshhane_na_moshhite_na_sv-petka_v_Trnovo-Bylgariq
The reception of Holy Relics of Saint Petka Paraskeva in medieval Capital of Bulgaia Tarnovo

In 1230 King Ivan Asen II the most powerful South-eastern European ruler demanded from the the Knights of the Crusaders to submit him her holy relics who are found still in Tracian city Kaliakratea ruled at that time by the Holy Latin Empire. King Ivan Asen II together with the patriach Joachim the first receives her holy relics with honor and settles her incorruptabilities into the newly creates Church in honour of herself St. Petka behind Tsarevets FortressSaint Petka became from that point considered as a protectress of the city, the throne and the country.
Her holy relics arrived from Kallikrateia in Tarnovo, the Capital of Second Bulgarian Empire in year 1230 AD, she has been thus called Paraskeva of Tarnovo and has been venerated as a protectress of the Tarnovo city the Bulgarian nation and the country. The attitude towards Saint Petka Tarnovska as a protectress of Bulgarian nation and contry is been clearly seen by the mention in the Bulgarian and International acts (documents) and manuscripts of that XII – XII century.

To learn more about Saint Petka  Parskeva Epivates of Thrakia feast day today 14 of October check my previous article here.

Let by the Holy Prayers of Saint Petka the Families be granted grace to endure the hardships of life! Saint Petka pray Christ for us!

Apache disable requests to not log to access.log Logfile through SetEnvIf and dontlog httpd variables

Monday, October 11th, 2021

apache-disable-certain-strings-from-logging-to-access-log-logo

Logging to Apache access.log is mostly useful as this is a great way to keep log on who visited your website and generate periodic statistics with tools such as Webalizer or Astats to keep track on your visitors and generate various statistics as well as see the number of new visitors as well most visited web pages (the pages which mostly are attracting your web visitors), once the log analysis tool generates its statistics, it can help you understand better which Web spiders visit your website the most (as spiders has a predefined) IP addresses, which can give you insight on various web spider site indexation statistics on Google, Yahoo, Bing etc. . Sometimes however either due to bugs in web spiders algorithms or inconsistencies in your website structure, some of the web pages gets double visited records inside the logs, this could happen for example if your website uses to include iframes.

Having web pages accessed once but logged to be accessed twice hence is erroneous and unwanted, and though that usually have to be fixed by the website programmers, if such approach is not easily doable in the moment and the website is running on critical production system, the double logging of request can be omitted thanks to a small Apache log hack with SetEnvIf Apache config directive. Even if there is no double logging inside Apache log happening it could be that some cron job or automated monitoring scripts or tool such as monit is making periodic requests to Apache and this is garbling your Log Statistics results.

In this short article hence I'll explain how to do remove certain strings to not get logged inside /var/log/httpd/access.log.

1. Check SetEnvIf is Loaded on the Webserver
 

On CentOS / RHEL Linux:

# /sbin/apachectl -M |grep -i setenvif
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
 setenvif_module (shared)


On Debian / Ubuntu Linux:

/usr/sbin/apache2ctl -M |grep -i setenvif
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-default.conf:1
 setenvif_module (shared)


2. Using SetEnvIf to omit certain string to get logged inside apache access.log


SetEnvIf could be used either in some certain domain VirtualHost configuration (if website is configured so), or it can be set as a global Apache rule from the /etc/httpd/conf/httpd.conf 

To use SetEnvIf  you have to place it inside a <Directory …></Directory> configuration block, if it has to be enabled only for a Certain Apache configured directory, otherwise you have to place it in the global apache config section.

To be able to use SetEnvIf, only in a certain directories and subdirectories via .htaccess, you will have defined in <Directory>

AllowOverride FileInfo


The general syntax to omit a certain Apache repeating string from keep logging with SetEnvIf is as follows:
 

SetEnvIf Request_URI "^/WebSiteStructureDirectory/ACCESS_LOG_STRING_TO_REMOVE$" dontlog


General syntax for SetEnvIf is as follows:

SetEnvIf attribute regex env-variable

SetEnvIf attribute regex [!]env-variable[=value] [[!]env-variable[=value]] …

Below is the overall possible attributes to pass as described in mod_setenvif official documentation.
 

  • Host
  • User-Agent
  • Referer
  • Accept-Language
  • Remote_Host: the hostname (if available) of the client making the request.
  • Remote_Addr: the IP address of the client making the request.
  • Server_Addr: the IP address of the server on which the request was received (only with versions later than 2.0.43).
  • Request_Method: the name of the method being used (GET, POST, etc.).
  • Request_Protocol: the name and version of the protocol with which the request was made (e.g., "HTTP/0.9", "HTTP/1.1", etc.).
  • Request_URI: the resource requested on the HTTP request line – generally the portion of the URL following the scheme and host portion without the query string.

Next locate inside the configuration the line:

CustomLog /var/log/apache2/access.log combined


To enable filtering of included strings, you'll have to append env=!dontlog to the end of line.

 

CustomLog /var/log/apache2/access.log combined env=!dontlog

 

You might be using something as cronolog for log rotation to prevent your WebServer logs to become too big in size and hard to manage, you can append env=!dontlog to it in same way.

If you haven't used cronolog is it is perhaps best to show you the package description.

server:~# apt-cache show cronolog|grep -i description -A10 -B5
Version: 1.6.2+rpk-2
Installed-Size: 63
Maintainer: Debian QA Group <packages@qa.debian.org>
Architecture: amd64
Depends: perl:any, libc6 (>= 2.4)
Description-en: Logfile rotator for web servers
 A simple program that reads log messages from its input and writes
 them to a set of output files, the names of which are constructed
 using template and the current date and time.  The template uses the
 same format specifiers as the Unix date command (which are the same
 as the standard C strftime library function).
 .
 It intended to be used in conjunction with a Web server, such as
 Apache, to split the access log into daily or monthly logs:
 .
   TransferLog "|/usr/bin/cronolog /var/log/apache/%Y/access.%Y.%m.%d.log"
 .
 A cronosplit script is also included, to convert existing
 traditionally-rotated logs into this rotation format.

Description-md5: 4d5734e5e38bc768dcbffccd2547922f
Homepage: http://www.cronolog.org/
Tag: admin::logging, devel::lang:perl, devel::library, implemented-in::c,
 implemented-in::perl, interface::commandline, role::devel-lib,
 role::program, scope::utility, suite::apache, use::organizing,
 works-with::logfile
Section: web
Priority: optional
Filename: pool/main/c/cronolog/cronolog_1.6.2+rpk-2_amd64.deb
Size: 27912
MD5sum: 215a86766cc8d4434cd52432fd4f8fe7

If you're using cronolog to daily rotate the access.log and you need to filter out the strings out of the logs, you might use something like in httpd.conf:

 

CustomLog "|/usr/bin/cronolog –symlink=/var/log/httpd/access.log /var/log/httpd/access.log_%Y_%m_%d" combined env=!dontlog


 

3. Disable Apache logging access.log from certain USERAGENT browser
 

You can do much more with SetEnvIf for example you might want to omit logging requests from a UserAgent (browser) to end up in /dev/null (nowhere), e.g. prevent any Website requests originating from Internet Explorer (MSIE) to not be logged.

SetEnvIf User_Agent "(MSIE)" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


4. Disable Apache logging from requests coming from certain FQDN (Fully Qualified Domain Name) localhost 127.0.0.1 or concrete IP / IPv6 address

SetEnvIf Remote_Host "dns.server.com$" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


Of course for this to work, your website should have a functioning DNS servers and Apache should be configured to be able to resolve remote IPs to back resolve to their respective DNS defined Hostnames.

SetEnvIf recognized also perl PCRE Regular Expressions, if you want to filter out of Apache access log requests incoming from multiple subdomains starting with a certain domain hostname.

 

SetEnvIf Remote_Host "^example" dontlog

– To not log anything coming from localhost.localdomain address ( 127.0.0.1 ) as well as from some concrete IP address :

SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog

SetEnvIf Remote_Addr "192\.168\.1\.180" dontlog

– To disable IPv6 requests that be coming at the log even though you don't happen to use IPv6 at all

SetEnvIf Request_Addr "::1" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog


– Note here it is obligatory to escape the dots '.'


5. Disable robots.txt Web Crawlers requests from being logged in access.log

SetEnvIf Request_URI "^/robots\.txt$" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog

Using SetEnvIfNoCase to read incoming useragent / Host / file requests case insensitve

The SetEnvIfNoCase is to be used if you want to threat incoming originators strings as case insensitive, this is useful to omit extraordinary regular expression SetEnvIf rules for lower upper case symbols.

SetEnvIFNoCase User-Agent "Slurp/cat" dontlog
SetEnvIFNoCase User-Agent "Ask Jeeves/Teoma" dontlog
SetEnvIFNoCase User-Agent "Googlebot" dontlog
SetEnvIFNoCase User-Agent "bingbot" dontlog
SetEnvIFNoCase Remote_Host "fastsearch.net$" dontlog

Omit from access.log logging some standard web files .css , .js .ico, .gif , .png and Referrals from own domain

Sometimes your own site scripts do refer to stuff on your own domain that just generates junks in the access.log to keep it off.

SetEnvIfNoCase Request_URI "\.(gif)|(jpg)|(png)|(css)|(js)|(ico)|(eot)$" dontlog

 

SetEnvIfNoCase Referer "www\.myowndomain\.com" dontlog

CustomLog /var/log/apache2/access.log combined env=!dontlog

 

6. Disable Apache requests in access.log and error.log completely


Sometimes at rare cases the produced Apache logs and error log is really big and you already have the requests logged in another F5 Load Balancer or Haproxy in front of Apache WebServer or alternatively the logging is not interesting at all as the Web Application served written in ( Perl / Python / Ruby ) does handle the logging itself. 
I've earlier described how this is done in a good amount of details in previous article Disable Apache access.log and error.log logging on Debian Linux and FreeBSD

To disable it you will have to comment out CustomLog or set it to together with ErrorLog to /dev/null in apache2.conf / httpd.conf (depending on the distro)
 

CustomLog /dev/null
ErrorLog /dev/null


7. Restart Apache WebServer to load settings
 

An important to mention is in case you have Webserver with multiple complex configurations and there is a specific log patterns to omit from logs it might be a very good idea to:

a. Create /etc/httpd/conf/dontlog.conf / etc/apache2/dontlog.conf
add inside all your custom dontlog configurations
b. Include dontlog.conf from /etc/httpd/conf/httpd.conf / /etc/apache2/apache2.conf

Finally to make the changes take affect, of course you will need to restart Apache webserver depending on the distro and if it is with systemd or System V:

For systemd RPM based distro:

systemctl restart httpd

or for Deb based Debian etc.

systemctl apache2 restart

On old System V scripts systems:

On RedHat / CentOS etc. restart Apache with:
 

/etc/init.d/httpd restart


On Deb based SystemV:
 

/etc/init.d/apache2 restart


What we learned ?
 

We have learned about SetEnvIf how it can be used to prevent certain requests strings getting logged into access.log through dontlog, how to completely stop certain browser based on a useragent from logging to the access.log as well as how to omit from logging certain requests incoming from certain IP addresses / IPv6 or FQDNs and how to stop robots.txt from being logged to httpd log.


Finally we have learned how to completely disable Apache logging if logging is handled by other external application.
 

How to Recover deleted /var/lib/dpkg directory on Debian / Ubuntu Linux server

Wednesday, October 6th, 2021

how-to-recover-accidently-deleted-var-lib-dpkg-directory-on-debian-gnu-linux

Sometimes you might do something stupid, in the hurry like running the wrong rm  command and ending up deleting /var/lib/dpkg on your Debian / Ubuntu system.

by either wrongly issuing the rm to a directory or mistyping rm -r /var/lib/dpkg.
I know this is pretty dumb but sometimes we're all dumb, if you do so and you try to do the regular

root@debian:/ # apt update && apt upgrade

or try to install some random package onwards you will end up with error message:

E: Could not open lock file /var/lib/dpkg/lock – open (2: No such file or directory)

Ending up with this error, does totally blocks your further system administration activities with both apt / aptitude / apt-get as well as with dpkg package management tool.

 

1. The /var/backups recovery directory

Thankfully, by Gods mercy some of Debian Linux system architects has foreseen such issues might occur and have integrated into it the automatic periodic creation of some important files into directory /var/backups/

Hence the next step is to check what kind of backups are available, there:

root@debian:/ # ls -al /var/backups/
total 19892
drwxr-xr-x  7 root root      4096 Sep 24 06:25 ./
drwxr-xr-x 22 root root      4096 Dec 21  2020 ../
-rw-r–r–  1 root root    245760 Aug 20 06:25 alternatives.tar.0
-rw-r–r–  1 root root     15910 Aug 14 06:25 alternatives.tar.1.gz
-rw-r–r–  1 root root     15914 May 29 06:25 alternatives.tar.2.gz
-rw-r–r–  1 root root     15783 Jan 29  2021 alternatives.tar.3.gz
-rw-r–r–  1 root root     15825 Nov 20  2020 alternatives.tar.4.gz
-rw-r–r–  1 root root     15778 Jul 16  2020 alternatives.tar.5.gz
-rw-r–r–  1 root root     15799 Jul  4  2020 alternatives.tar.6.gz
-rw-r–r–  1 root root     80417 Aug 19 14:48 apt.extended_states.0
-rw-r–r–  1 root root      8693 Apr 27 22:40 apt.extended_states.1.gz
-rw-r–r–  1 root root      8658 Apr 17 19:45 apt.extended_states.2.gz
-rw-r–r–  1 root root      8601 Apr 15 00:52 apt.extended_states.3.gz
-rw-r–r–  1 root root      8599 Apr  9 00:26 apt.extended_states.4.gz
-rw-r–r–  1 root root      8542 Mar 18  2021 apt.extended_states.5.gz
-rw-r–r–  1 root root      8549 Mar 18  2021 apt.extended_states.6.gz
-rw-r–r–  1 root root   9030483 Jul  4  2020 aptitude.pkgstates.0
-rw-r–r–  1 root root    628958 May  7  2019 aptitude.pkgstates.1.gz
-rw-r–r–  1 root root    534758 Oct 21  2017 aptitude.pkgstates.2.gz
-rw-r–r–  1 root root    503877 Oct 19  2017 aptitude.pkgstates.3.gz
-rw-r–r–  1 root root    423277 Oct 15  2017 aptitude.pkgstates.4.gz
-rw-r–r–  1 root root    420899 Oct 14  2017 aptitude.pkgstates.5.gz
-rw-r–r–  1 root root    229508 May  5  2015 aptitude.pkgstates.6.gz
-rw-r–r–  1 root root        11 Oct 14  2017 dpkg.arch.0
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.1.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.2.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.3.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.4.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.5.gz
-rw-r–r–  1 root root        43 Oct 14  2017 dpkg.arch.6.gz
-rw-r–r–  1 root root      1319 Apr 27 22:28 dpkg.diversions.0
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.1.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.2.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.3.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.4.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.5.gz
-rw-r–r–  1 root root       387 Apr 27 22:28 dpkg.diversions.6.gz
-rw-r–r–  1 root root       375 Aug 23  2018 dpkg.statoverride.0
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.1.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.2.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.3.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.4.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.5.gz
-rw-r–r–  1 root root       247 Aug 23  2018 dpkg.statoverride.6.gz
-rw-r–r–  1 root root   3363749 Sep 23 14:32 dpkg.status.0
-rw-r–r–  1 root root    763524 Aug 19 14:48 dpkg.status.1.gz
-rw-r–r–  1 root root    760198 Aug 17 19:41 dpkg.status.2.gz
-rw-r–r–  1 root root    760176 Aug 13 12:48 dpkg.status.3.gz
-rw-r–r–  1 root root    760105 Jul 16 15:25 dpkg.status.4.gz
-rw-r–r–  1 root root    759807 Jun 28 15:18 dpkg.status.5.gz
-rw-r–r–  1 root root    759554 May 28 16:22 dpkg.status.6.gz

drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T00:22:30.p1e5J8/
drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T00:24:02.dAUgDs/
drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T12:29:51.FX27WJ/
drwx——  2 root root      4096 Oct 15  2017 ejabberd-2017-10-15T21:18:26.bPQWlW/
drwx——  2 root root      4096 Jul 16  2019 ejabberd-2019-07-16T00:49:52.Gy3sus/
-rw——-  1 root root      2512 Oct 20  2020 group.bak
-rw——-  1 root shadow    1415 Oct 20  2020 gshadow.bak
-rw——-  1 root root      7395 May 11 22:56 passwd.bak
-rw——-  1 root shadow    7476 May 11 22:56 shadow.bak

Considering the situation the important files for us that could, help us restore our previous list of packages, we had installed on the Debian are files under /var/backups/dpkg.status*

Luckily debian based systems keeps backups of its important files that can be used later on for system recovery activities.
Below is a common structure of /var/lib/dpkg on a deb based system.

hipo@debian:/home/hipo$ ls -l /var/lib/dpkg/
total 11504
drwxr-xr-x 2 root root    4096 Aug 19 14:33 alternatives/
-rw-r–r– 1 root root      11 Oct 14  2017 arch
-rw-r–r– 1 root root 2199402 Oct 19  2017 available
-rw-r–r– 1 root root 2197483 Oct 19  2017 available-old
-rw-r–r– 1 root root       8 Sep  6  2012 cmethopt
-rw-r–r– 1 root root    1319 Apr 27 22:28 diversions
-rw-r–r– 1 root root    1266 Nov 18  2020 diversions-old
drwxr-xr-x 2 root root  606208 Sep 23 14:32 info/
-rw-r—– 1 root root       0 Sep 23 14:32 lock
-rw-r—– 1 root root       0 Mar 18  2021 lock-frontend
drwxr-xr-x 2 root root    4096 Sep 17  2012 parts/
-rw-r–r– 1 root root     375 Aug 23  2018 statoverride
-rw-r–r– 1 root root     337 Aug 13  2018 statoverride-old
-rw-r–r– 1 root root 3363749 Sep 23 14:32 status
-rw-r–r– 1 root root 3363788 Sep 23 14:32 status-old
drwxr-xr-x 2 root root    4096 Aug 19 14:48 triggers/
drwxr-xr-x 2 root root    4096 Sep 23 14:32 updates/

 

2. Recreate basic /var/lib/dpkg directory and files structures

As you can see, there are 5 directories and the status file and some other files. 
Hence the first step is to restore the lost directory structure.

hipo@debian: ~$ sudo mkdir -p /var/lib/dpkg/{alternatives,info,parts,triggers,updates}


3. Recover /var/lib/dpkg/status file

Further on recover the dpkg status file from backup

hipo@debian: ~$  sudo cp /var/backups/dpkg.status.0 /var/lib/dpkg/status


4. Check dpkg package installation works again and reinstall base-files

Next check if dpkg – debian package manager is now working, by simply trying to download dpkg*.deb reinstalling it.

root@debian:/root # apt-get download dpkg
# sudo dpkg -i dpkg*.deb

If you get no errors next step is to reinstall base-files which is important package on which dpkg depends.

root@debian:/root # apt-get download base-files

root@debian:/root # sudo dpkg -i base-files*.deb

 

5. Update deb system package list and db consistency

Onwards try to update system package list and check dpkg / apt database consistency.

root@debian:/root # dpkg –audit

root@debian:/root # sudo apt-get update

root@debian:/root # sudo apt-get check


The result should be more of the files in /var/lib/dpkg should appear, thus list the directory again and compare to the earlier given list of it, they should be similar.

root@debian:/root # ls -l /var/lib/dpkg

6. Reinstall completely from source code dpkg, if nothing else works
 

If some files are missing they should get created with a normal daily sysadmin package management tasks so no worries.

In case if after attempting to upgrade the system or install a package with apt, you get some nasty error like:
 

'/usr/local/var/lib/dpkg/status' for reading: No such file or directory


Then the next and final thing to try as a recovery is to download compile from a new and reinstall dpkg from source code!

 

root@debian:/ # wget https://launchpad.net/ubuntu/+archive/primary/+files/dpkg_1.16.1.2ubuntu7.2.tar.bz2
root@debian:/ # tar -xvf dpkg_1.16*

root@debian:/ # cd dpkg-1.16*

root@debian:/ # ./configure

root@debian:/ # make

root@debian:/ # make install


Hopefully you'll have gcc and development tools provided by build-essential .deb package otherwise you have to download and compile this ones as well 🙂
If this doesn't bring you back the installed packages you had priorly (hopefully not), then waste no more time and do a backup of the main things on the server, and reinstall it completely.

The moral out of this incident is always to implement always to your system a good back up system and regularly create backups of /var/lib/dpkg , /etc/ , /usr/local* and other important files on a remote backup server, to be able to easily recover if you do by mistake something whacky.
 

Hope that helped anyone. Cheers 🙂