Archive for the ‘Educational’ Category

Saint martyr Angel of Lerina – a Bulgarian saint confessor and the Day of Saint Archangel Michael and of all Angels Archangels and Heavenly Powers

Wednesday, November 9th, 2022

saint_archangel_Michaill_Joanikij_papa_Vitanov_1820_Trjavna_Bulgaria

Saint Archangel Michael (Church of Saint  Archangel Michael Tryavna, Bulgaria) iconographer Yoanikij Papavitanov

On 8th of November in the Bulgarian Orthodox Church, we celebrate the day to remember the gathering of Archangel Michael with all the Angels Archangels, Cherubims and heavenly powers that have kept loyal to the Holy Trinity God – The Father, The Son, and The Holy Spirit.
The same arch-angels and powers who could do what they want and were created in the beginning of time after God as a helper Spirits to God and man.

The same angels are also supporting the whole universe with their deeds of love. They sustain the waters, make the wind blow, the clouds to move and give rain, the earth to give its fruits, possess and give wisdom or transfer secret messages from God to man when sent.

Others do protect all Christians and people from the evils of the fallen-agels who choose to misobey the True God Christ and follow the master of the evil spirits whose place is in the burning Gehenah and whose time is running out. 

They help the woman in birth-pain (like my sister Stanimira whose time to give birth is approaching), the make the organism of man to function properly. Or give the physics to make the stars shine on heaven, the Planet and heavenly bodies to move. Each and every place and Country and Church has its own guardian angels. And they're of a Big multitude the Church fathers says a lot about the Angels and many is still unknown and will be revealed in the that everyone will stand on the Judgement day in front of God and sees the Heaven and Hell realities and will stumble in fear seeing the gloriness of the archangels and cherubs (burning out of Love for God and man) made in a likeness of the Holy God.

The orthodox Church sticks clearly to the teaching of so called saint Dyonisious the Areopagites (often called in the Theology Pseudo Dyonisious), who was one of the important apostles of Christ, Athenian judge at the Areopagus Court in Athens, who lived in the first century. A convert to Christianity, he is venerated as a saint by multiple denominations.

 

The writings of Saint Dionysius the Areopagite hold great significance for the Orthodox Church. Four books of his have survived to the present day:

 

On the Celestial Hierarchy, On the Ecclesiastical Hierarchy, On the Names of God, On Mystical Theology

In additional, there are ten letters to various people.

The book On the Celestial Hierarchies was written actually in one of the countries of Western Europe, where Saint Dionysius was preaching. In it he speaks of the Christian teaching about the angelic world. The angelic (or Celestial-Heavenly) hierarchy comprises the nine angelic Ranks:

  • Seraphim
  • Cherubim
  • Thrones
  • Dominions
  • Powers
  • Authorities
  • Principalities
  • Archangels
  • Angels

 

The account of the Synaxis of the Bodiless Powers of Heaven is located under November 8.

saint-Agatangel-Bitolski-Bulgarian-saint-icon

The purpose of the divinely-established Angelic Hierarchy is the ascent towards godliness through purification, enlightenment and perfection. The highest ranks are bearers of divine light and divine life for the lower ranks. And not only are the sentient, bodiless angelic hosts included in the spiritual light-bearing hierarchy, but also the human race, created anew and sanctified in the Church of Christ.

There is too much to be said about Angels, Archangels, through the years from ancient times, they can heal and help, and grant special powers to man and many, many more. There were innumerable heresies who have over-deified heavenly powers, especially gnostics and that is a well known fact. For those who want to read about Angels, and their hierarchy there is a lot ot read and learn, angels has helped the saints in their hardship in fight with evilness, there is really a lot about this for those who want to further learn. 

But what is less known is here in today's relatively small country of Bulgaria, we have a local saint Angel of Lerina who is born in Bulgarian family and stems from a Bulgarian village. As his endeveour and confession of his love for Christ and the Church was enormous he has suffered martyrdom for Christ in the 17th century during the times Bulgaria was enslaved by the Ottoman Turks. Thus as there is not much written about saint Angel Lerinski (Lerina), I dedicated this small article in glory of his memory. The article is also in memoriam of my grand-grand-grand Father who was also named Angel himself, perhaps in glory of Saint Angel of Lerina.

The Life of Saint Angel of Lerina


Saint-Angel-Agathangel-of-Lerina-orthodox-icon

Saint New Martyr Angel of Lerina (Bitolski) – picture source Wikipedia

All the sources about the holy martyr Angel of Lerinsky that we have reached cite the story of Saint Paisius of Hilendar as the main source for the life of the new martyr, called Angel or Agathangel. This is what St. Paisius writes about him in the History of Slavonic Bulgaria":
           "In 1750, in Bitol, where the Turkish and Macedonian Pasha sits, the Turks tortured and beheaded a young man, handsome in face and stature, for the Christian faith. Many forced and tormented him to renounce Christ, but he wisely and courageously denounced their godless faith. The Bishop of Bitola recorded many of his answers, described his sufferings in Greek. And God showed a great sign over his powers. His name was Angel from the village of Lerin. This holy martyr Angel shone in our time in the Bulgarian land."
          The Bulgarian Orthodox Church honors the holy new martyr on November 8, the feast of the holy Archangels. Probably the veneration of the saint in our church dates from the time when he was martyred, because his martyrdom was described by Saint Paisius of Hilendar immediately after it happened, since Saint Paisius was his contemporary.

          Greek information about the new martyr Angel Lerinski appeared only in recent years.

In the electronic version of "Οι Νεομάρτυρες της Булгариас" (New Martyrs Bulgarian) Αρχιμανδρίτου του Οικουμενικού Θρόνου Θωμά Ανδρέου Ιεροκήρυκος Ιεράς Μητροπόλεως Ελευθερουπόλεως (Archimandrite of the Ecumenical See Thomas Andreu, Preacher of the Eleftheroupolis Holy Metropolis), Kavala, 2011, p.88, we read :

         "Another case of a new martyr of Greek origin is that of Angel (or Agatangel) from today's Florina (in Bulgarian Lerin). The 2009 calendar of the Holy Metropolis of Florin, Prespa and Eordei honors this new martyr, who was martyred in the monastery of Pelagonia (now Bitola, Macedonia) on February 17, 1727*. The book "History of Slavonic Bulgaria" by Paisiy Hilendarski talks about the martyrdom of the new martyr ("his name was Angel or Agatangel and he was from the village of Florina")

Saint-Agatangel_Bitolski-Greek-icon

…..
          Little is known about the new martyr. We know that he was born in 1732 in Florina, in the sanjak (prefecture) of Bitola (Monastery). When he grew up, he became a tall and handsome young man. At the age of 18, the Turks tried to convert him to Islam, but Angel – although very young – did not succumb to the temptations and then bravely accepted martyrdom. In his book Paisius Hilendarski mentions that: "In 1750 in the monastery… the Turks tortured and slaughtered a handsome young man because of his Christian faith… His name was Angel and he was from the village of Florina" His testimony in the monastery was attended by the local Greek metropolitan, who described his courage and the intelligent and logical answers he gave in court. Due to the fact that he condemned the Muslim faith with particular wisdom and courage, he was beheaded when he was only 18 years old in 1750. The Bulgarian Orthodox Church honors his memory on November 8, during the Feast of the Archangels…."

          Additional information about the holy new martyr Angel Lerinsky can be found on one of Florina's sites (http://agiospanteleimonas-florina.blogspot.com/2010/06/blog-post_8186.html.)


          The Metropolitan of Florini, Prespa and Eordaia, Theoclitus, addresses the citizens on the occasion of the decision to start the veneration of the holy martyr Agathangel of Florina (June, 2010):

          "With special feelings of joy, emotion, holy contentment and reverence, I turn to you, the blessed children of the Greek Macedonian land, to become participants in the great spiritual joy experienced by our local Church for the first official celebration of the memory of the holy new martyr Agathangel in the seat of our metropolis Florina. It is already known to all of you. that the holy new martyr Agathangel, martyred in the Pelagonian monastery, originated from Florina, is our fellow citizen. At an early age he left Florina and went to Vutelion of Byzantium, to the monastery, seeking better living conditions. There, exercising the profession of shoemaker, he soon distinguished himself by his honesty and his diligence. …. But what distinguished him from the young people of his time was the pure and firm faith he had in Christ and in His "orthodox church". He loved Christ more than anything else in his life. No other love could "steal" the love that Agathangel had in his heart for Christ, he loved Him simply, purely, with all his heart, with all his strength, he loved Him as his poor parents and his blessed ancestors loved Him.
         

Along with the love for Christ, the saint had love for his homeland, conquered Macedonia. Almost four hundred years of slavery count the long-suffering "Greek Macedonians".
The Turkish conquerors treated them with cruelty. Sometimes with flattery, sometimes with threats, sometimes with violence, they try to make them change their faith. To deny Christ. To renounce the Orthodox faith and become Muslims.
And whoever renounces his faith renounces his homeland.

          Agathangel's heart was troubled by the fact that several of his fellow Roman Christians did not withstand the temptations or the violence, denying Christ and the country. His brave heart rebelled. He could not bear the Orthodox faith to be dishonored. For this, when during the three-day Bairam, which is celebrated after Ramadan, the forced conversion of the Orthodox increased, this young boy, not yet twenty years old, went to Constantinople, where he received a Sultan's firman, which forbade the forced conversion in the area of ​​Pelagonia.

On his return to the Monastery, the saint was arrested by those outraged by the Turkish Sultan's decision, and after cruel torture, he was beheaded on February 17, 1727.

           In a meeting we held in the Holy Metropolis, in which, in addition to the Metropolitan, the Honorable Prefect of Florini Mr. Ioannis Voskopoulas, the Mayor of Florini Mr. Stefanos Papanastasiou, the President of TEDC of N. Florinis, Mr. Dimitrios Iliadis, and the President of the monasteries of N. Florinis, Mr. Theodoros Vosdou, decided to jointly hold events in honor of the holy new martyr Agathangel…”

* 1727 is mentioned as the year of the martyrdom of St. Angel Lerinski in some sources, and sometimes it is mentioned together with 1750, in the same source. This discrepancy in the years of the martyrdom leaves the doubt that different martyrs are being talked about.


For this reason, we cannot say exactly whether the reliquary with the relics – the holy head of the new martyr Agathangel in the Kykkos monastery in Cyprus, which contains the same description of his life, but the date February 17, 1727 is indicated, refers to the same martyr, for which speaks Saint Paisius of Hilendar.

Saint Reverend Martyr Jacob Kosturski and his two disciples Deacon Jacob and Monk Dionysius † November 1, 1520

Tuesday, November 1st, 2022

Icon_of_Jacob_of_Kostur-and-his-2-desciples-Yakov-and-Dyonisios_19_Century

Holy Reverend Martyr Yakov ( Jacob) was born in one of the Bulgarian village of the Kostur region in ex-Bulgarian Macedonia in the distance year 1520.
His parents names were Martin and Parascheva.

He was an experienced cattle breeder and through hard work became quite rich, but his own brother, out of envy, slandered him to the Turkish authorities for hearing the false rumor he allegedly found a hidden fortune. Saint Yakov is a saint of the pleiad of great Bulgarian saint martyrs, sadly little known today in Bulgaria today.

Therefore, Jacob sold his sheep, left his native land and went to Constantinople, where he became even richer.
Once he heard a Turk praising Christianity and telling how his wife was cured of madness through the prayers of Saint Patriarch Niphont.
Yakov was very interested in this and went to the patriarch.

The conversation they had so touched his soul that he gave away all his wealth and retired to the Agios Oros Holy Mount Athos monk republic.

There he revived the deserted Iverion old skete "Saint John the Forerunner" and lived under the guidance of a certain elder Ignatius.

Jacob labored so diligently and reached such a high spiritual perfection that he was honored with heavenly revelations: he was shown the abodes of heaven and the dungeons of hell, he began to penetrate the secret feelings and thoughts of his visitors, e.g. received by God the gift of clairvoyance.

The Lord also honored him with the gift of miracles: at his prayer, spring water flowed into the renewed hermitage, which has been called "the agiasma of Saint Jacob" ever since; twice the jar of oil filled itself when run empty by his prayers;
With his prayer he healed a Vatopedian novice monk who was seized with rabies; once rain fell from heaven in a time of drought; another time watered thirsty travelers with the stream that came out of earth but his prayers springed waters.

Desiring to give himself up to complete solitude, he and six of his disciples retired to the inner part of Mount Athos, where he spent days and nights in complete silence except on Saturdays and Sundays.

Finally he wished to go into the world to preach piety. His disciples followed him.
Everywhere the people flocked to receive his blessing and listen to his sermon.
He also performed many miracles with his prayers: he healed the insane, handed over to Satan an unrepentant Christian sorcerer to weaken the flesh (1 Cor. 5:5).

But the local Bishop Akakios, out of envy, reported to the Turkish authorities that this old foreigner was gathering a lot of people around him and could cause some rebellion.
The monk foresaw the rising danger and spent the whole night in prayer and calmly waited for the divine liturgy to end.

Early in the morning, 18 Turkish soldiers surrounded the church and captured the elder Jacob with two of his disciples – deacon Jacob and monk Dionysius (some other Biographers speak about 6 desciples).

The Turkish bey (local place governor) investigated the reverend for a long time, sometimes with exhortations, sometimes with threats, sometimes with torture, but finding no guilt in him, threw him into prison for 40 days, until he reported him to the High Gate (highest authority in the Ottoman Turkish Empire) and received an order from there.

After some time, an order came from Sultan Selim to bring the saint and his disciples to Edirne (Odrin), so that he himself could judge them.

The Sultan strictly investigated them. Venerable Jacob calmly spoke only the truth.

The Sultan ordered that he and his disciples be beaten with whips.
Not a word, not a groan was heard from the reverend.
Then they tightened Yakov and desciples deacon Jacob and Dyonisios heads with screws.

The saint did not suffer from this torture, but one of Deacon Jacob's eyes popped out of orbit.
In the meantime, the Sultan learned that the old martyr was prophetically predicting the future and jokingly asked St. James: "How many more years will I live?"
The reverend replied, "Nine months!"

holy-martyrs-Iakovos_Kastorias_Kosturski-and-his-desciples-Jacob-and-Dyonisios

His prophecy came true.
In order to find a sufficient reason to kill the innocent, the embittered sultan sent one of his pashas (bashaw – higher rank in the Ottoman political and military system, typically granted to governors, generals, dignitaries, and others ) to them in prison to ask them questions about Christ and Muhammad.

Venerable Jacob boldly confessed that only Jesus Christ is the true God, which enraged the sultan and served as an occasion for new cruel tortures:

they tore apart the bodies of the martyrs, broke their jaws, cut thongs from their skins and then watered their wounds with vinegar and salt, they beat them cruelly with whips, burned their thighs with fire.

They were tortured in this way for 17 days. At last the Sultan sentenced them to be hanged (perhaps because he knew about Jude the Escariot and wanted to mock the saints).

Before the sentence was carried out, Venerable Jacob stood among his disciples and invited them to pray for the world and for the Church, thanking the Lord for honoring them to inherit the kingdom of heaven.

All three fell on the ground and worshiped God.

Then Venerable Jacob took out three hidden particles from the life-giving Mysteries of Christ and with them he gave communion to himself and his disciples.

Then he raised his hands and eyes to heaven and cried out with a loud voice:

"Lord, into Your hands I commit my spirit!" And immediately, to everyone's surprise, quietly died.

This happened on November 1st, year 1520 A.D.
Some Christians bought their honest holy relics and buried them with honors.
Over their tombs on Sundays and holidays light appeared and miracles took place.

Later, their holy relics were transferred to Mount Athos by christians, and took them three kilomers away from their in the village of Albani and laid them in three separate graves.

Over their graves on Sundays and holidays, light appeared and miracles took place!

Later from there to the area of ​​Galatista near Thessaloniki, where the rest of the monk's students settled in the monastery "St. Anastasia (Deliverer from Potions) Uzorushitelnica".

Some Christians bought their holy relics and took them three kilometers away from their homeland, in the village of Albani, and laid them in three separate graves. Over their graves on Sundays and holidays, light appeared and miracles took place. Later, their relics were transferred to Mount Athos, and from there to the village of Galatista, Thessaloniki. There was a small monastery "St. Athanasius Uzorushitelnica", which the rest of the disciples of the martyr Jacob revived. The memory of the martyrs is celebrated on November 1.

There is a special service for the Reverend Martyr Jacob and his disciples, which is served on the day of their memory in the Iverion skete "Saint John the Forerunner" on Mount Athos and in the monastery "Saint Anastasia".

The Biography compiled out of:

1. © Lives of the Saints. Synodal Publishing House, Sofia, 1991, edited by Parthenius, Bishop of Levkia and Archimandrite Dr. Athanasius (Bonchev).
2.
Hristo Temelski From the collection Saints and spiritual leaders from Macedonia

Dendritism in early ascetism, Saint John of Rila – a Great ascetic saint who practiced Tree living form of monasticism

Friday, October 21st, 2022

saint-John-of-Rila-with-Rila-monastery

“Dendrites hanging on a tree of life blooming in virtues, like a good fruits of the Spirit”

The monasticism is born in the late antiquity in the desert to Egypt, Syria, Palestine and Asia Minor, in a peculiar dialogue with the development of urban Christianity.
As we know, Christianity is a city religion, the structure of it is following example of administrative organization common for the age in the Roman Empire.

With the end of the anti-Christian persecution, church the christian communities in the city of the empire to strengthen, grow, and move on to onether new way of development – with all pluses and minuses for a new state. Freedom to confess (worship) gradually retreating locally to patronage of the empire, which with its whole repressive power stands on the side of the new religion, who has conquered for its four centuries  of existence with it’s evangelic radicalism, won the hearts of many thousands of Roman citizens. This puts the Christian communities in front of the unknown for them challenge, being a Christian now provides additional citizen advantages for better arrangement of lifely affairs. The wine of the apostoles preach to  unreservedly following of Christ is now inter-mixe with the common for a man conformity.

This comes to be a problem for many Christians and most importantly for those who do not want (or cannot) live with their Christian identity in conditions of the Imperial Church.
Hence, the end of the persecutions coincides with the stormy flourishing of monasticism and ascetism, in all regions of the empire from the end of the east to the external borders of the western parts. In the church life is born a single center, which is trying to remind the believers that the Kingdom of God is not from this world, but instead every christian is called to secretly raise his own seed of faith in his field (the good and clean heart) – in secret and independently from the worldly rulers.


Among the diversified forms of cenobitic monasticism and asceticism, present during the early Byzantine period, a typical one known for its radicalism of faith and asceticism of Syrian Christians. It is in the midst Syria where the phenomenon of monastic stylites “pillars” e (στυλίτες) they have a specific hermit type of life, in whose the monk lives on the top of open or closed “tower” or high stone, without coming to earth, in constant prayer, regardless of climatic conditions. Stylitism has received a wide spread between the Orthodox and Monophysitites to theEast, but not among the Nestorians (early christian heretics). Among the well known Stylites are Saint Simeon Stalpnik (Stylites) (lived in 5th century), Daniil the Stylites (5th century), Simeon The-Newest (Novi) and Alipiy the Stylites (6th century).

Having formalized the whole course in monasticism, so-called open monasticism. monasticism is openly (του ανοικτού μέσου), to whose branch a part are the Stylites and the Dendrites  δενδρίτες  (the monks who lives on a tree or inside a tree), monks who live without a shelter. „βοσκοί”. Other extreme forms of ascetism are the “recluse” (οι έγκλειστοι, κλωβίτες), they are part of the  the so-called. "closed type" monasticism (του κλειστού μέσου), the most notable saints representatives from this type are the elders Barsanuphious and elder Ioan (Elder).

Among the most unusual and rarely practiced forms of asceticism was dendriticism. These ascetics remain in the Romance languages ​​with their Greek name – dendrites – inhabiting trees. They lived inside, in the hollow, or in the branches of the tree, standing or sitting. Their feat is compared to that of the columnists, who also live in a small space on tree posts or pillars. The small space they occupied for varying periods of time—usually from one to several years—developed in them the virtue of manly patience. Dendrites "serve God in the trees", these wonderful creations of God, among whom a chosen one become the Holy Christ cross tree that served for the salvation and sanctification of man. With their "blessed solitude" they become the new "witnesses of conscience" after the end of the "witnesses of blood" who deserved and won eternal life in the persecutions of the Roman Empire.

Saint_David_Solunski-Dendronite
 

St. David of Thessalonica But as we said, the creation of a common monastic culture in the spacious borders of the Christian Empire brought this exotic way of life to the West as well. Pillars appear even next to the walls of the capital city – in the person of St. Daniel the Pillar, and Rev. David of Thessalonica (6th century) is among the most famous hermits-dendrites not only in the Balkans, but also in the entire Christian world. And although he spent three years on the almond tree in prayer and fasting, after which he continued his feat on a pillar, in Orthodox iconography he remains immortalized precisely on a tree. And the second hermit who went through this ascetic test was St. John of Rila, the founder of monasticism in the Bulgarian lands.

saint-John-of-Rila-heals-the-demonic-posessed-icon

But as mentioned, the creation of a common monastic culture within the spacious borders of the Christian Empire brought this exotic way of life to the West.

Pillars saints appear even next to the walls of the capital city – in the person of St. Daniel the Pillar, and Rev. David of Thessalonica (6th century) is among the most famous hermits-dendrites not only in the Balkans, but also in the entire Christian world. And although he spent three years on the almond tree in prayer and fasting, after which he continued his feat on a pillar, in Orthodox iconography he remains immortalized depicted precisely on a tree. And the second hermit who went through this ascetic test was St. John of Rila, the founder of monasticism in the Bulgarian lands.

The way of thinking of people in ancient times was very different from the modern way.
All ascetic Byzantine literature testifies to the desire to see in every detail of everyday life a lesson, a symbol, a sign of divine providence in the life of every person.
In this sense, the feat of the ascetics-dendrites is rich in Christian symbols and metaphors. The hymnography of the Church in their glorification highlights two main elements – of the tree of life, to which they become partakers with their feat, and of the freedom of complete surrender in God's hands, inherent in the "birds of the sky", who do not care for their sustenance, but rely on God's mercy.

This is what the first symbol looks like – on the tree not as an ordinary residence, but as an image of the Cross of Christ.

"The dendrites hanging on the tree of life flourish in virtues as good fruits of the Spirit", and in the service of St. David of Thessalonica – the most famous monk in the Balkans who experienced this way of asceticism, we also read:

"Like a light bird he climbed on the tree and made a hut, freezing in winter and burning in summer. Thus he obtained the golden wings of dispassion and perfection, which lifted them to the heavens."

The-Mother-of-God-Theotokos-the-Tree-of-saints

The tree is undoubtedly one of the most common symbols in Christian literature, dating back to the early church. Tertullian compared Christians to evergreen trees.
Origen compares Christ to the tree of life. For Didymus the Blind (4th century), the tree is the vine Christ, whose branches are the righteous men who bear true fruit. This imagery enters the language of church writers through the Gospel texts. The tree is a symbol that Christ himself used many times in his parables – I am the Vine of life… The fig tree that does not bear fruit withers and becomes useless. According to church teaching, "everything that died through the tree of the knowledge of good and evil will live again through the tree of the Cross and the water of baptism that springs from the Tree of the Cross of Christ." Pseudo-Ambrosius.

Clement of Alexandria says that the tree of the living Christ grows in the paradise of the Church, in the waters of renewal and gives as its fruit the teaching and the evangelical way of life.

All these symbols and metaphors of the tree as a symbol of the fall, death and salvation of man pass through the Gospel and patristic texts in the life of the ascetics and more specifically of those of them who choose the solitude of the tree as a place of their witness and feat.

During the period before the Christianization of the Bulgarian people, the Orthodox Church accumulated serious ascetic experience, represented in the monastic movement in its various forms. Different ascetics preferred different forms of asceticism, and each of them developed new virtues and spiritual gifts in the ascetics.

This priceless centuries-old experience grafted into the young Bulgarian Church and its first hermit, St. John of Rila.

The Church of Christ in Bulgaria found in him an ascetic who, within the framework of his life, managed to go through all the forms of asceticism that Christian monasticism had known up to that point – he created a new monastery, where he studied, then became a hermit, lived in a cave, in the open rock, in the hollow of a tree, and finally receives the charisma of spiritual fatherhood and gathers disciple monastic brotherhood.

Already at the very beginning of the Long Life of St. John of Rila, we see the comparison of St. John with the popular Christian symbol of the fruit-bearing tree: "and it bore fruit, indeed a hundredfold, as a tree planted by springs of water."

As in the lives of the other dendrite monks, St. John did not immediately proceed to this difficult feat – he began his monastic journey "in a monastery for the sake of learning" and only then withdrew to the wilderness of the mountain, where he lived in a hut of branches.

Saint John then spent 12 years in a narrow and dark cave! (the cave is nowadays located about 50 minutes walk from the Rila monastery in the mountain of Rila)

 The biographer of Saint John St. Euthymius informs us that only after that he moved "at a considerable distance" from the cave and settled in the hollow of a huge oak, like the oak of Abraham. It is obvious the desire of Patriarch Euthymius to make a connection between this unusual feat for this region of the Christian world and the hospitality of Abraham, who met the Holy Trinity under the Oak.

In the hollow of the tree St. John ate chickpeas. This is, by the way, the first mention of growing chickpeas on the Bulgarian lands – the chickpea is an unpretentious, but still heat-loving plant, which is grown mainly in the southern Bulgarian lands and Dobrudja today.
In the life, the sprouting of chickpeas around the oak of St. John of Rila is compared to the miracle of manna from heaven. The behavior of the shepherds who secretly collect pods of chickpeas from the saint and their unusual joy shows (they loot it) that this food was indeed atypical of the region where St. John traveled and it successfully growing on this mountain coldly place is one of the innumerable miracles of the saints which started even in his lifetime.

After that, exactly what is the logic of the ascetics-dendrites in the other popular stories about them – the sick begin to flock around the saint, his living invariably mention "possessed by demons" who come to heal to him.

The life of prayerful patience and extreme asceticism of these strange hermits living in trees reminded them of Christ's words that "this kind does not come out except by prayer and fasting."
Life conveys to us the prayer of the saint, with which he frees the possessed – from the text we see that it does not have an exorcistic character as in most of the other saint livings, that is, St. John does not directly forbid the demons, but calls on God's omnipotence and thus reveals his deep, extreme humility, with which he receives daring before God.
It is noteworthy to mention that Reverend David of Thessalonica also received the charisma to cast out demons after spending three years in the branches of an almond tree.

saint-David-the-Tree-liver-hermit-saint-icon

The duration of life on the tree for different ascetics is different – for Saint David it is three years, in the life of Saint John of Rila the time is not specified (most likely because the saint asked God to hide this part of his life again for humility).

Three is a symbolic number, corresponding to the request of the prophet David to receive from God goodness, knowledge and prudence. According to Susan Ashbrook Harvey, tree life "seems to have had the character of a temporary discipline in the ascetic practice, in which different places and modes of asceticism were changed." Unlike dendrites, those who live on a pillar spend many more years there, this way of asceticism also becomes a social service.

While life in a tree is usually a transition to some other asceticism, which means that as living conditions it was much more difficult. This is how Reverend David's life is described by his biographer: Further on, the author describes the sufferings of David in the (ἔν) tree – he was tormented by cold, by heat, by winds, but his angel-like face did not change, but looked like blooming rose. Some of his disciples went to the tree and begged him to come down and help them—lest the spiritual wolf prey on the flock while the shepherd was gone. David, however, was steadfast in his decision to remain on the tree. "I will not come down from the tree for three years, then our Lord Jesus Christ will show me that he has accepted my prayer."

Three years later, an angel appeared to him and told him that God had heard his prayer and ordered him to come down from the tree and build himself a cell, because another mission (οἰκονομία) awaited him.
It is interesting that at this point David turns to the official church authority for a kind of sanction of the vision ‒ he sends his disciples to tell what happened to the bishop of Thessalonica, Dorotheus and ask him whether this vision is really from God.

Researchers of monastic culture in the Byzantine era note the greater remoteness, isolation of the ascetics-dendrites from those who live on pillars. Entire temporary settlements of pilgrims, sick people, people from different tribes and countries arise around the latter, which often make noise and disturb the ascetic life of the saint. The most famous example is Rev. Simeon The Stylite ( Stulpnik ) the founder of hermit Stylitism. Around his pillar there was always a crowd that was not always meek, obedient and pious. There are often half-wild Arabs among her, sometimes 200, 300, 1000 people to come.
They often made a noise and continued their tribal quarrels at the foot of the pillar.

However, the dendrites immediately leave the place of their feat if people begin to gather around them – be they pilgrims or disciples. We see this in the life of Revend John as well:
"And the valiant Ivan, as soon as he learned what was happening, got up and went away from there, because he was afraid, and even more, he hated human glory."

We see this clearly expressed in the lives of two brothers in Syria – Rev. Maro (Saint Maron passed on to Christ 410 AD) and Rev. Abraham.
The first was a dendrite, and the second a stylite.
John of Ephesus in the Lives of the Eastern Saints tells of the Reverend Maro(n) Dendrite that he lived in a hollow tree near his brother Abraham, a stylite in his monastery.
Unlike the monks, St. Maro did not communicate with visitors, the door of his tree was closed, and he lived in silence until someone sought healing.
When his brother died, Maro left his imprisonment in the tree and moved to his brother's place and then began to communicate more with people.
But while living in the tree, Maro received no visitors.

saint_Maron-the-Syriac-hermit
Like Reverend Maron in Syria, Saint John of Rila leaves the tree of his asceticism as soon as people begin to gather around him, and switches to another form of asceticism – very close to stair climbing, namely on a high and difficult-to-access rock (today known as the Rock of Saint John a common place for pilgrimage).

But even here, tempted and physically injured by the demons, the ascetic does not remain hidden from the people. It was during this period that he attracted the attention of St. King Peter, who tried to establish contact with him. Of course, the high rock on which St. John of Rila lived for seven and a half years provided much harsher living conditions than the steeple, which is usually near a populated place. But as a philosophy of the ascetic feat, in both cases it is about something in common – maximally narrowing the free space for movement and directing all energy upwards, in the power of prayer and constant unity with God.

The common moments in the lives of the two most famous Dendrite monks in the Balkans – Revend David of Thessaloniki and St. John of Rila, who labored three centuries are similar.

Both begin their monastic journey with "discipleship in a monastery" before heading for the desert, that is, moving away from human society.
For both of them, the life in the oak, respectively in the almond branches is a period of extreme asceticism, which greatly impresses the surrounding population, who begin to flock to them. Their unusual feat inspires in people a desire to live near them and even to imitate them – in their lives we see a number of people who seek their help – starting with students (one of Reverend David's students also became a "dendrite" as his mentor and settled in the hollow of a large tree). There glory quickly reaches the local bishop and all the clergy, as well as the rulers of the city – as mentioned in the life of the Thessalonian ascetic, and to Saint King (Tsar) Peter and the Bulgarian boyars as mentioned in the biographical life of Saint John of Rila .

The biographers of both monks include the stories of the healing of demoniacs precisely while they were living in their unprotected "homes" from the natural elements – i.e. the trees. Finally, for all their desire to remain hidden in the wilderness of their solitude, they attract not only the sick, the afflicted, and the disciples, but the attention of the powerful of the day.
But while St. David came from the East, from Mesopotamia, St. John was local and did not have a great Eastern ascetic teacher as he was local citizen born in Bulgaria, in our lands.
His way of asceticism is undoubtedly influenced by the general ascetic patterns of the age, but it is also unique – it is a testimony to the general internal logic of Christian asceticism, regardless of which parts of the Christian world it is practiced.

Paradoxically, the brightest monastic examples in the Balkans became precisely these two monks, struggling in these harsh, atypical for the western parts of the empire, conditions – dendriticism, stylitism, living in a narrow cave and a high cliff.

Until the 9th century, that is, throughout the early Byzantine period, in today's Greek lands, the cult, the respect for Rev. David of Thessalonica (born c. 450 – 540) was comparable only to that of Saint Great-Martyr (Demetrius) Dimitar of Thessalonica and St. Achilles, bishop of Larissa.

Great Respect and recognition as a saint for him was already alive in the first half of the 9th  century at the same time when saint John’s greatnes shined upon the world, as we can see from the life of St. Gregory the Decapolitan, who sent one of his monks to worship at the saint's grave in a monastery founded by him near Thessaloniki. St. David the Dendrite monastery was an attractive pilgrimage center in the Balkan lands of the empire until the 11th century, when the relics of the saint were brought by the Crusaders to Italy.

Without the spiritual presence of its founder, its monastery declined and disappeared, and its relics returned to Thessaloniki only in the 20th century and were laid in the church of "St. Theodora".

The abode of the Rila desert dweller has a different destiny – it remains as a living spiritual center throughout the centuries, in the heart of the Rila desert, and its founder, already a resident of the Heavenly Jerusalem, invariably remains a faithful and reliable breviary for his kindred in the flesh the Bulgarians.

Report presented at the international conference dedicated to the 500th anniversary of the transfer of the relics of St. John of Rila from Tarnovo to the monastery he founded, organized in 2019 at the Rila Monastery. It was published in the eponymous collection of conference reports under the title: "Dendrite Monks in the Balkans".

Article originally posted in Bulgarian by Zlatina Ivanova on 19.10.2022 – Translated with minor modifications by me (Georgi D. Georgiev a.k.a. hip0)

Saint martyr Trendafil of Starazagora (Starozagorski), little known Bulgarian saint martyred in XVI century

Monday, August 8th, 2022

In memory of the holy martyr Triandafil of Stara Zagora ( Starozagorski )

saint-Nikodimos-agiogarithes-sveti-Nikodim-Svetogorec

According to Venerable Nicodemus Agiorite, "Synaxaristis", Constantinople, 1845, and "Neon Martyrologion", Athens, 1961.

Holy Martyr Triandafil was very young Bulgarian, about 18 years old, a native of Stara Zagora, and he was a sailor.

svetogorskata-ikona-na-sv.Triandafil-ot-hrama-sv.Georgi-v-Zagora-saint-Trendafil

Suffered as a martyr for his Christian faith in Constantinople on August 8.

For the year of his death, Venerable Nicodemus Hagioritis in his work "Synaxarium" published in (Tsarigrad / Constantinople 1845) indicates year 1570 as a year of his martyrdom, and in his other work "Neon Martyrologion" (newly published in Athens in 1961) as a year of martyrdom 1680.

His memory was celebrated on his day (August 8) every year and the tradition continues today in the Bulgarian Orthodox Church as saint Trendafil is one of the constellation of Bulgarian saints.

Nicodemus Hagiorite reports that his life was written by the biographer Ioannis Cariophilis, but it seems that he did not have the original of the living at hand to include in his works.

Ikona-Sveti-Trendafil-ot-hrama-na-sveteca-v-Zagora.

As Bulgaria has been under a Spiritual and Physical slavery both the country and the Church has been under yoke, the country under the yoke of ottoman turks and the Church under the yoke of Greek slavery, only two verses are preserved in honor of the martyr (as the Greeks custom do), those two verses are given by Venerable Nicodemus in his "Synaxarion":

The name Trendafil literally translated is the flower Rosa multiflora.

Thus the glorification verses in the sinaxarion says:

Triandafil  appeared as the new trendafil (Rosa multiflora),
reddened by the flow of his blood.

So far, nothing else is known about this martyr of ours.

© Living of of the saints, translated into Bulgarian from the Church Slavonic text of Cheti-minei ("Chety-Minei") of St. Demetrius of Rostov.

Living of unknown saint Saint Sophronius of Sofia, known also as Saint Soprhonius of the Bulgaria / Sofronij of the Balkans

Saturday, May 28th, 2022

Sveti-Sofronij-Bylgarski-SofijskiSaint-Sophronius-of-Sofia-Bulgaria-of-the-Balkans
 

Biography of Staint Sofronij / Sophronius of Bulgaria Sofia, Known also as Saint Soprhonius of the Balkans

The parish priest of the village of Penkyovtsi (Sofia, Bulgaria region) Stefan (Te fled to Sofia with his wife due to Turkish violence, from hence he later fled to Wallachia region to the great Wallachian voivode (Duke) Radul. His wife died there and he became a monk named Sophronius (not to be mistaken with the very famous Bulgarian saint Saint Sofronius of Vratza / Sofronij Vrachanski).

After the death of Duke Radul, he returned from the Danube river to his homeland and settled in a monastery near city of Ruse (probably he lived in the cave monastery of Saint Joachim I Patriarch of Tarnovo and the Venerable Demetrius Besarabovsky, that even today is the biggest monastery nearby the city of Ruse).

There he struggled with common sanctification practices as fasting, prayer, work and alms for the poor. The devil did not tolerate his monastic great achievements and set a monastic servant against him, who struck him on the head with an ax and killed him.

Three years later, Sophronius appeared to the people living in the monastery, who obeyed his suggestion (obviously they got inspired to  dug up his grave which is a common ancient Christian practice for notable christians who might have been saints) and look up if his relics are incorruptable and found his relics incorruptible and fragrant (emitting a heavenly odor), as they have been inspired by God's Spirit to do.
People with great joy placed the holy relics in a coffin for common (universal) worship of all the Christians.

We do not know the exact dates he was  born or has been killed, because of the devilish envy, as at that time Bulgaria has been under the Ottoman turks and the founding of his holy relics has happened years after his martyrdom. But most probably the Venerable Sofronij  lived in the second half of the fifteenth century and the beginning of the sixteenth century, by the second decade of which he must have suffered. This information about him is told to us by the Bulgarian writer and priest Father (the bulgarian word for which is Pop – stems from the greek word Papas (Father) Peyu. The same Pop Peyu has been also the author of the life of Saint George of Sofia the New not to be mistaken with Saint George the Newest from Sofia  who maryrdom suffered in year 1515 because of his unwillingless to accept the false Islamic faith.

© Lives of the Saints. Synodal Publishing House, Sofia, 1991, edited by Parthenius, Bishop of Lefkada and Archimandrite Dr. Athanasius (Bonchev) with minor inclusions of the article author Georgi Georgiev

Let by his holy prayers the Bulgarian homeland and especially the suffering in Ukraine and all people everywhere, finds more Peace, Love, Hope, Faith and Goodness 

Christ is Risen ! Truly He is Risen ! The origin of the tradition Paschal Greeting and Coloring of Eggs on Easter Holidays in the Church

Wednesday, April 27th, 2022

 

Christ-is-Risen-Truly-he-is-risen-and-the-christian-origin-of-red-eggs-worldwide-Christ-triumphant-icon

Christ is Risen ! Truly He is Risen !

Христос воскресе ! Воистину воскресе ! (Khristos voskrese! Voistinu voskrese!) – Church Slavonic Paschal Greeting

Χριστὸς ἀνέστη!  Ἀληθῶς ἀνέστη ! (Khristós anésti! – Alithós anésti!) – Greek Paschal Greeting

Christus Resurrexit ! Resurrexit Vere ! – Latin Paschal Greeting

The Easter Eggs are so famous today for the kids worldwide, even though the world does not put much accent on the feast of Pascha (Easter). All kind of colored eggs are to be find in stores, many christian countries both Western and Eastern all throughout the world have the tradition of coloring eggs for Easter. 
The tradition is also the same here in Orthodox Bulgaria, as we have the tradition to boil and color eggs in various colors. 
Usually the first egg is colored in Dark Red and once sanctified in the Church is put on the iconostasis (the prayer corner in the house) in front of Christ, Virgin Mary and saints icons and kept their until next year.

Miraculously this Egg usually does not start decaying or smelling as an ordinary egg will do if left out of the fridge for a month or so. This first egg in dedication and memory of Christ's resurrection is kept on the iconostasis until the next year's Pascha and then buried somewhere in a green clean place for sanctification of the land.

This is a good and well followed tradition for those strict about religion, but even those who did not strictly follow Christianity or orthodoxy do color eggs for the fun of kids and as an expression for joy of the Paschal feast. Both grown and kids then try out their forces whose egg is more powerful by knocking each other's eggs to test whose egg shell is more solid and can stand up the break. The egg that is "victorious" once people test their power that is stronger and withstand the "egg fight" is kept for another egg duel with another person.

According to old superstitious belief if you win over in an eggs fight this is interpteted as you will have a good health and well being for the upcoming period till next year's Pascha.
 

How and from wherein this Boiled Eggs coloring originated ?


The short answer is it is connected to one of Church's traditions about the poor apostle Equal Saint Mary ( Maria ) Magdalene, who have given as a gift to Roman emperor Tiberius an Egg with the All Famous Greeting dialog in the Orthodox Churches among people with person A saying:   Christ is Risen ! person B responding: Truly he is Risen! (Христос Воскресе Войстину Воскресе !)

From the time of the many appearances of the Savior Christ in flesh after his Glorious Resurrection described by the Holy Evangelists in the Gospels and the fervent sermon of St. Mary Magdalene (one of the of the so called Myrrh- Bearing Woman who were the first who have visited the tomb where Dead body of Christ was led and become witnesses of  the Resurrection). The surviving Bible's New Testament 4 Gospel books do not provide further details about the activities of St. Mary Magdalene and her life. The Church mouth-by -mouth tradition of her later life in several local Christian churches differ slightlyhowever everywhere they essentially report on the zealous co-apostolic activity of St. Mary Magdalene. And the differences between these traditions depend on which of the evangelical women these churches understand by the name of St. Mary Magdalene.

Some Western Christian churches, as well as the Church Fathers and learned theologians, unite in one or two personalities three evangelical women: the sinner who repented in the house of Simon the Pharisee, shed tears at the Savior Christ's feet, wiped them with her hair. and she anointed them with precious ointment, and Mary of Bithynia, sister of Lazarus of Bythynia ( resurrected in the fourth day after death by Chrsit and commemorated one day before Palm Sunday )' and Mary Magdalene, who was delivered from the Savior Christ by seven demons. But the Orthodox Church now, as before, recognizes those mentioned in the Gospels with different signs, three persons as different separate ones, and does not want to base historical information on arbitrary, plausible interpretations. Therefore, the tradition of the Orthodox Church states that after the Gospel appearances of the Risen Christ before His Ascension and after, St. Mary Magdalene resided with the Blessed Virgin and the Apostles and was an active helper of the first successes of spreading the Christian faith first in Jerusalem. But full of zeal, fervent faith, and zealous love for God's gospel, she then preached in other lands, proclaiming everywhere the heavenly grace, joy, and salvation of all who believed in the Savior of the world, the Risen Christ.

Saint-Mary-Magdalene-gifting-red-egg-to-emperor-Tiberius-Orthodox-icon-one-of-Myrrh-Bearing-Woman

While visiting Italy to preach, St. Mary Magdalene found an opportunity to appear before the then-reigning Emperor Tiberius I, and presented him, according to generally accepted Eastern custom, with an egg painted red and greated him with "Christ is risen!"

The modesty of the gift of Mary Magdalene did not surprise the emperor, because he knew the ancient custom of the East, also among the Jews, going for the first time to superiors, or on solemn occasions to acquaintances or patrons, to offer a gift of honor, with some known or special, symbolic meaning. Examples of this can be found in Jewish Old Testament history, as are the gifts presented by the rich Wise men (magis – today their relics are kept for veneration in Cathedral of Cologne Germany) to the born Jesus Christ in Bethlehem of Judea. Even the poor in such cases offered as a gift various fruits from their locality or eggs from birds. Thus, partly following this ancient custom and with the red color of the egg laid and with the hitherto unheard words "Christ is risen!" to arouse the curiosity of the suspicious Emperor Tiberius. The holy co-apostle Mary Magdalene, by explaining the significance of this gift, began her fervent sermon on the Resurrection truth and the teachings of the Savior Christ for salvation. With great inspiration and conviction she told the emperor about the life, miracles, crucifixion and resurrection of Jesus Christ according to his own prophecy. She gave a direct, simple-minded account of the extremely unjust, biased judgment of Jesus Christ by the embittered members of the Jerusalem Sanhedrin. governor of Judea Pilate of Pontus, in condemning Jesus Christ to crucifixion. She explained how all this incurred the wrath of the Roman emperor then and how Tiberius handed them over to a court in which Pilate was deprived of power and exiled to Gaul, in the city of Vienna, where, according to legend, tormented by remorse and despair, he killed himself. According to another legend, Pilate repented, turned to Christ in prayer, as a sign of which his head was accepted by an angel after being cut off.

According to Church tradition, the sisters of Lazarus Martha and Mary went to Italy with St. Mary Magdalene; and Pilate, learning of this and fearing the denunciation of his unlawful actions by the Christians, himself sent a message to the Emperor Tiberius about Jesus Christ, in which he testified to the virtuous life of Christ, the healing of all diseases and infirmities from Him, even for the resurrection of the dead and for His other great miracles. Pilate asserted that in examining the accusations of the Jews, he found no fault in Jesus Christ; he made great efforts to deliver Him from the hands of the troubled Jews, but failed to deliver Him and betrayed Jesus to their will because of the cries of the people and the rebellious accusation of the Jews against Pilate himself …

 

... as a witness, overwhelmed with fear, he told the emperor about everything that had happened to Jesus Christ, who became an object of faith as God …

After such testimonies from the Roman governor of Judea and the worshipers of the Savior Christ, Emperor Tiberius, according to legend, himself believed in the Savior Christ, proposed to include Jesus Christ in the image of the Roman gods, and even when the Roman Senate rejected this proposal, Tiberius by royal decree threatened to punish anyone who dared to grieve believers in Jesus Christ.

In this way, with the zealous, fearless sermon on the Savior Christ, St. Mary Magdalene, along with other devout Christians, persuaded the pagan governor of Judea to testify in writing about the universal event of Christ's Resurrection before the pagan world and persuaded the then Roman emperor of the Savior Christ, thus facilitating the spread of Christianity.

Anastasis-Hristos-Voskrese-beautiful-orthodox-Mosaic
Anastasis (Resurrection) Church Fresco


And the Christians of that time, learning about the significance and strength of the impression caused by the offering of a red egg by Mary Magdalene to Emperor Tiberius with the words: "Christ is risen!" then began to imitate her in this and as a remembrance of Christ's Resurrection they began to give each other red eggs and say: "Christ is risen! … He is risen indeed! …"

Thus, this custom gradually spread everywhere and became universal for Christians around the world. In it, the egg serves as a symbol of Christ's resurrection and the resurrection of the dead, and of our expected new-birth for eternal bliss in the future life, the pledge for which is Christ's Resurrection.

Just as a bird is born from an egg and begins to live an independent life after its release from the shell, and the vast circle of life is revealed to it, so we, at the second coming of Christ to earth , rejected from ourselves together with the earthly body all that is mortal on earth.

By the power of Christ's Resurrection we will be resurrected and resurrected to another, higher, eternal, immortal life.

And the red color of the Easter egg reminds us that the redemption of mankind and our future new life have been acquired through the shedding of the cross on the pure blood of the Savior Christ.

Thus, the red egg serves to remind us of one of the most important dogmas of the Divine revealed Christian faith.

 

After the crucifixion of Jesus by the Jews in terrible miracles took place in nature, many dead righteous people rose, with His resurrection on the third day.Pilate, as a witness overwhelmed with great fear, informed the Caesar of all things that had happened to Jesus Christ.

In Eastern Orthodox Tradition Christ is Risen ! Truly he is Risen Greeting is used to joyfully great each other all around the Orthodox countries in the first 3 days of easter, and can be used instead of normal Hello greeting ! for the upcoming week The Holy Easter Weak which is a week of great joy and even by a hello greating in the Church could be used for 40 days as a normal greeting.

It is worthy to close this article with the praisal words, read on the first day of Pascha  authored by one of the most important Church fathers and
compiler of most served Liturgy service throughout the yearly service calendar:

"Christ is risen, and you are overthrown!
Christ is risen, and the demons are fallen!
Christ is risen, and the angels rejoice!
Christ is risen, and life reigns!
Christ is risen, and not one dead remains in a tomb!
For Christ, being raised from the dead, has become the first-fruits of them that have slept."

Saint John Chrysostom

The Holy and Great Week of Passions of Christ in the Church – Day by day explained

Tuesday, April 19th, 2022

Christ-the-bridegroom-orthodox-passion-of-Christ-week

The Holy Great Week of Christ Passions

The last week of the earthly life of the Lord Jesus Christ is called the "Great" or "Passion Week", i.e. A week of suffering, a prelude to eternal life. The Lord's life was coming to an end. Having resurrected Lazarus on the Sabbath as a proof of the Mass coming Resurrection known in the Church as Lazarus Saturday as it is always celebrated Saturday in the Orthodox Church on which people gathered to solemnly welcome the Messiah Christ, and triumphantly entered Jerusalem on Palm Sunday. Following that the Savior Jesus Christ who prophecised his betrayel to the Cross for human sin, the Lord voluntarily walked step by step to His predestined inevitability.

Every day of the Passion Week is called Great and Holy for the reason this week is the most Holy and Sanctified week of the whole Calendar Church year. Each of the Seven days of it, the Church commemorates events of last week of Christ's life and suffering on earth before Resurrection and Ascension to Heaven through special services the way of Christ to Golgotha, the sufferings and His redemptive work on the Cross.

Worship during Holy Week

Lent services on the weekdays of Lent are characterized by their penitential singing. The royal doors (of the alter known as Dveri) remain closed as a symbol of man's separation from the Kingdom of God. Church vestments are dark, usually purple in the color of repentance.

Bulgariand-Church-Kings-doors-Carski_dveri_-_Sv._Spas_(Rashtak)_in_North_Macedonia

No Divine Liturgy is performed on weekdays, but so that the (ordinary chrsitians who go often and pray God) – so called faithful can support themselves in their ascetic effort of fasting by accepting Holy Communion, a Liturgy of the Presanctified Gifts is performed (a specific Liturgy prepared for the Purpose that is only served during great Lent). This service is very ancient, it is mentioned in the canons of the VII century, but it was established earlier for sure. Most likely the Liturgy of the Presanctified Gifts, practice to sanctify bread and wine in prior has later evolved in the Roman-Catholic Churchs errenous from Eastern Orthodox point of view – Eucharistic Adoration
– (a consacration host kept usually in the so called (monstrace). Traditionally, Presanctified Liturgy creator is considered to be Pope St. Gregory I the (Dialogus), Pope who governed the Western Church in (VI century) – some  theologians today claims it was developed at least partially or coauthored also by Saint Ambrose of Mediolan (Milan).

The pre-consecrated liturgy consists of a solemn Lenten Vespers (prelonged repentance songs) with elements from Psalms and readings from Holy Scripts regarding life and suffering of Christ, to which is added the part of "transfer" of the Holy Gifts from the Alter to the Upper place (the place where the proskomidia occurs) and walked in on the "Great Entrance" Liturgy part with the Sacraments placed in the Holy Chalice held by the priest in front of iconstansisa and back to the Alter of Sacrifice, however the consecration of the Gifts itself is not performed, the Eucharistic gifts are already sanctified and prepared on previous Sunday Saint Basil or Saint John Chrysostomos liturgy.  That is why the service is called the Presanctified Liturgy, i.e. of the pre-consecrated Gifts.

Usually This service takes place on Wednesdays and Fridays or at least on one of these days and on the 6th week of Maria of Egypt is served 3 times instead of 2 throughout the week to venerate the Most Holy Mother Mary of Egypt which from a Harlot turned a saint by immerse repentance, and cause of that become the patron saint for repentance and example for true repentance, that each and every Christian aims follow, every day of his life.

Following the 6th weeks of Fasting a period that the ancient Church placed for try out of ones self soul state and cleanance of passions comes saint Lazarus Saturday.

Lazarus Saturday is the only day of the year when Sunday service worship is integrated in Saturday.  Usually Sunday service is a service of higher importance than the other ones, a faithful gathering to share the unspeakable joy for the Resurrection of Christ and his triumph of Life over Death. 

Lazarus Saturday is the beginning of the Easter celebration. During the Liturgy of Lazarus, the Church glorifies Christ as "Resurrection and Life", who even before His sufferings and death, with the resurrection of Lazarus, confirmed the foreshadowing of the universal resurrection of mankind coming. It was because of the resurrection of Lazarus that Christ was glorified by the people as the long-awaited Messiah (no man ever was able to rise up a death rotting person from the Death after four days in grave) truly identifying him as the promised King of Israel and the fulfillment of long ages awaited Old Testament prophecies.

The very feast of the triumphal entrance of the Lord into Jerusalem (Palm Sunday) belongs to the twelve most importance Church feasts, known in the Church as "Feasts of the Lord". Christ immediate worship by all Jews on his entrance in Jerusalem  is directly connected with that of Lazarus Day on which he did the miracle of commanding Lazarus to wake up from Death,  returning life of a long dead Lazarus.

On the eve of the feast, the prophecies about the Messianic King from the Old Testament are read, along with the Gospel accounts of Christ's entry into Jerusalem, as another confirmation that Christ is the True Messiah.

In the morning, the willow twigs we hold in our hands throughout the Liturgy are blessed, thus showing that we welcome Jesus Christ as King and Savior, just like the Jews has received him in Jerusalem 21 centuries again in  year 0 A.D.

Extract Prot. Thomas HOPCO "Fundamentals of Orthodoxy" with short modifications from:
Church NewsPaper of Bulgarian Orthodox Church, Issue 7 of April 17, 1998

Holy Monday, Holy Tuesday and Holy Wednesday

se-jenih-griadet-v-polunoshti-molitva-here-is-the-bridgeroom-comes-church-slavonic-prayer

Church Slavonic (Old Bulgarian) notable singing during the first 3 days of the Holy Week sung in the Orthodox Church

Text translates as:

Behold, the Bridegroom comes at midnight,

And blessed is that servant whom He shall find watching,

And again, unworthy is the servant whom He shall find heedless.

Beware, therefore, O my soul, do not be weighed down with sleep,

Lest you be given up to death, and lest you be shut out of the Kingdom.

But rouse yourself crying: Holy, Holy, Holy, art Thou, O our God,

Through the Theotokos have mercy on us.

Troparion of Bridegroom Matins

During the first three days of Holy Week, the Church commemorates the Lord's last stay in Jerusalem. In these days the worship is very intense: there is a Midnight Office (Μεσονύκτικον, Mesonýktikon; Slavonic: Полунощница), The Hours matins, Psalms Book chapters, reading of the Gospel and Liturgy of the Presanctified Gifts. During the "lessons" given by, the four Gospels to the Gospel of John are read. 13, verses 30
 

Great and Holy Monday

On Holy Monday, the evangelists tell us how the Son of God entered the Jerusalem temple and found it full of merchants. Overwhelmed with holy wrath, He overthrew their tables and drove them out, because the temple is a house of prayer, not a marketplace. (Matt. 21: 12-13, Mark 11: 15-19; Luke 19: 45-46).

Great-and-holy-Monday-the-parable-of-the-fig-tree-icon

On Holy Monday, the Church celebrates St. Patriarch Joseph, the son of St. James the Patriarch and a type of Jesus Christ.

Saint_Joseph-the-Patriarch-sold-in-Egypt-by-his-brothers-a-prototype-for-Christ
Joseph The Magnificient

Joseph was sold by his brothers to merchants traveling to Egypt.

Joseph-the-brilliant-as-Second-after-Pharaoh-in-Egypt-the-all-comely-icon

There, in a foreign land, he went through many sufferings, but Pharaoh made him second in power and position in the whole kingdom (Gen. 41: 38-46). Like Joseph, the Lord Jesus Christ was betrayed by the Jews to the Gentiles, tortured, and suffered for human sins.

Great-and-Holy-Monday-Christ-extreme-humility-icon

The Icon of Christ the Bridegroom (Ο Νυμφίος)

The Church also invites us to reflect on the image of the barren fig tree, which withered after being cursed by the Lord (Mark 11: 12-14, 20-26, Matt. 21: 18-22). "Every tree that bringeth not forth good fruit is hewn down, and cast into the fire" (Matt. 3:10).

Great-and-holy-Monday-the-parable-of-the-fig-tree-icon
 

In the same way, we will be condemned if we do not live in prayerful communion with God, do not strive to improve our faith, do not fill ourselves with virtues, and do not bear spiritual fruit.

Great and Holy Tuesday

"Watch therefore: for ye know neither the day nor the hour wherein the Son of man cometh" (Matt. 25:13).

(Gospel reading: Matins 22: 15-23: 39; Liturgy Mat. 24: 36-26: 2).

Holy Tuesday is a day for teachings and final moral instructions:

The Lord Jesus Christ gives us an example of how to do good – not to give from our surplus for this purpose, but as a poor widow to set aside from our last material means.

Great-Holy-Tuesday-the-10-virgins-parable-orthodox-christian-icon

Speaking of the approaching days of struggle and trial, Christ tells of the ten wise virgins who were always ready to meet the Savior (Matt. 25: 1-13). It reminds us that we must "be vigilant and not be discouraged" and keep our lamps lit in anticipation of the Divine Bridegroom.

That is why on Holy Tuesday the Church sings:

Here comes the bridegroom at midnight,
and blessed is that servant whom he hath found awake,
and unworthy is he whom he finds careless.

Therefore beware, my soul, lest you sleep,
to be delivered to death and to remain outside the closed doors of the Kingdom,
but come to your senses and exclaim: Holy, holy, holy, O God,
have mercy on us for the sake of the Mother of God!

"The light of the body is the eye" (Matt. 6:22), says the Lord. the unsullied human heart and soul, and "the oil is alms or all our good deeds" (St. John Chrysostom).

Living virtuously, with the fear of God and trust in the Lord, we will be ready to meet the Savior and enter the marriage hall – the Kingdom of Heaven.

The church also reminds us of the parable of the talents (Matt. 25: 14-30) and invites us to work hard and improve the abilities God has given us.

Then follow prophecies about the fate of the city of Jerusalem for the last days of the Second Coming of the Lord
(Matt. 25: 31-46, Mark 13: 1-31, Luke 21: 5-38).

Great Holy Wednesday

On the day of Holy and Holy Wednesday we remember one of the last events before the Lord's saving sufferings for us: the precious ointment, which in his sincere repentance a sinner woman poured on the Savior's head (Matt. 26: 6-13, Mark 14: 3-9).

She managed to enter the house where Christ was, the woman carrying an alabaster vessel with precious very expensive ointment, she wanted to pay her enormous respects to Him. In a hurry (scared that someone from the people in surrounding Christ might stop here) in order for not to interfere with her good intentions, she broke the vessel that was helding a high amount of oilment, making it easier to spill the ointment on Christ.

Great-wednesday-the-sinful-harlot-woman-with-oilments-cleaning-up-feets-of-Christ-with-her-hair

The precious ointment cost three hundred dinars ! (Mark 14: 5), so some being sick of the passion of Judah the Iscariot (The Love for Money the works of the Flesh) resented it: "Why is this waste?", "The ointment could be sold and the money given to the poor."
And Christ answered them, "You always have the poor with you, but you do not always have Me," "she has done a good work for Me [by] deceiving to anoint My body for burial." Her zeal will be heard all over the world.
Like the prodigal son, the sinner realized her sins and "came to her senses."

Let us also come to our senses about our real spiritual condition and repent of our sins, so that with our repentant tears we may "anoint" the Lord like that repentant woman !

Jude-Betrays-Christ-selling-him-for-30-silver-coins

On the same day, we recall the decision of the Sanhedrin to condemn Jesus Christ. Then Judas Iscariot went to the Jewish leaders and agreed to hand him over for thirty pieces of silver (Matt. 26: 14-16, Mark 14: 10-11, Luke 22: 1-6).

We should well think:

Do we, who bear the name of Christ, not betray Christ through our ungodly deeds?

From that day on, the kneeling prayers do not cease, as one should understand we have done plenty of badness and has inflicted additional pains to Christ, who suffered for all great sins on the Cross.
 

Great Wednesday

Great and Holy Thursday – Remembrance of the Last Supper

On that day, the Lord Jesus Christ celebrated the Passover in the home of a resident of Jerusalem
(Matt. 26: 17-35, Mark 14: 12-31, Luke 22: 7-38, John 13: 1-17, 26).

Before supper He washed the feet of the apostles and said, "I did not come to serve, but to serve."
The Savior then instituted the sacrament of the Eucharist (Communion) by Himself partaking of the holy apostles.

By His great mercy, the Lord also gives us the opportunity to receive His true body and blood during the Holy Liturgy, so that by accepting Christ within us, we may strive to keep Him through the purity of our hearts.

The-Secret-Supper-Tajna-vecherya-Aton-Manuil-Panselinos-Protata

After bequeathing the new commandment to love all, Christ revealed to His disciples that He would be betrayed.
Bewildered, the students asked who would do this.

The-Betrayal-of-Juda-Orthodox-icon-heprodosia
Judas asked is it him that will betray ? 
Christ answered him so meekly that the others did not understand.
Judas got up, went out leaving the holy eucharistic supper.
And pupils, thought he was going shopping for required goods for the brothers because he was a treasurer (an accent how we should keep a good mind and try to think well about others all the time).

Next  great accent is Lord's Prayer.

The-Prayer-of-Christ-in-Gethsemane-garden-Great-Tarnovo-Museum-MOLENIE-Muzej_V-Turnovo

Christ Prayer in Gethsemane Garden – Bulgarian Icon museum Great Tarnovo

In the Garden of Gethsemane After supper Christ and the apostles went to the Garden of Gethsemane (Matt. 26: 36-46, Luke 22: 39-46, John 18: 1), where he prayed until the coming of the traitor.

Mockery-of-Christ-icon

Usually on Thursday evening the morning of Good Friday service is served, when the so-called Twelve Gospels are read, ie. the twelve passages of the Gospel that tell of Christ's sufferings.


Through them we witness the mockery, suffering, and crucifixion of Christ, through which He redeemed us.
"Here is the Lamb of God who took away our sins."

And again we wonder if we do not crucify Christ through our passions and sins.

Jesus_in_Golgotha_by_Theophanes_the_Cretan-orthodox-icon-fifteen-century

Jesus in Golgotha – Theophanes the Cretan

On this day, the priests take the Cross out of the altar, which symbolizes its carrying from Christ to Golgotha.

The Great annointing of the sick service is served so called "Велик Маслосвет" – during whose many prayers to saints healers are red to intercede for us following by 7 Act of Apostle readings and 7 Gospel Chapter Readings, wherever possible in large Cathedral Churches, this is served by 7 priests  every willing layman is anointed with oil 7 times after reading each of the 7 Gospels for restorating of Health of the sick as well as a special blessing in the manner of the ancient Church tradition.

Great and Holy Friday

The Way of the Cross and Golgotha ​​We remember the great sufferings of Jesus Christ, who freely agreed to be judged, flogged, spat upon, beaten with slaps, and shown before the people in a purple robe, with a cross in his hand and a crown of thorns on his head. 

Armed with a heavy cross from Pilate's praetorium (judgement place), Christ was led to Golgotha ​​on the crucifixion.

The-Crucifix-of-Christ-Razpiatie-Hristovo

Crucified between two robbers for desecration in terrible natural disturbances – an earthquake and an eclipse of the sun, he died, accepted death to save all mankind from death.

On this day, every Christian should follow complete fasting (eat nothing and drink nothing) and pray and sorrow deeply for the Lord.
According to church rules, even the sick should only eat bread (at best a very dry one) and drink a little bit of water. Joys of any kind of type should be abstained and all passions avoived and one should ask God for mercy for himself, his family and ask is merceful to everyone.

Great and Holy Saturday

The burial of Christ the Savior and His descent into hell are commemorated.
He died on the cross, blood and water flowed from His pierced ribs.
Joseph of Arimathea and Nicodemus, asking Pilate for permission, removed Him from the cross, anointed Him with perfume, wrapped Him in a new shroud, and laid Him in a new tomb carved into a rock in the Garden of Gethsemane.

The_Burial_Lamentations_by_Theophanes_the_Cretan-Stavronikita-monastery-mount-athos-wall-painting

Epitaphios (Lamentation of Christ) from Stavronikita monastery, Mount Athos – Theophanes the Cretan

Holy-Saturday-The-Resurrected-Christ-Empty-tomb-grave-icon

Myrrh-bearing women were present at His burial in the tomb, among whom, in tears with her grief-stricken heart, was His Mother the Holy Mother of God.

The church sings regarding this great events:

"In the grave with his body and in hell with his soul as God,
in heaven with the thief and on the throne with the Father and the Spirit You were, Christ,
Who fills everything. "

The Jews sealed the tomb and set up a guard.

Great secret! "Let the human creature silence !" – sings the Church instead of the Cherubim song on Holy Saturday.
The lord of life is in the grave, but he will soon be famous for the miracle of the resurrection.

On the Saturday morning after the liturgy, in some places it is customary for the priest to give flowers to the faithful as an expression of joyful anticipation of the Resurrection.

Holy Week in the statutes of the ancient churches

Initially, Easter was preceded by a two-three-day fast, which took place one week – the so-called.
Passion Week, or the Week of Christ's Suffering.

Subsequently, the 40-day fast was added to Lent, similar to the forty days during which Christ fasted in the wilderness. It was intended for the "announced", that is, for those who would be baptized on Easter.

For a long time during the practice of mass baptisms of the elderly, the sacrament was performed on Easter, when baptism was especially experienced as a participation in the voluntary death and resurrection of the Lord.
That is why the Easter Liturgy is extremely baptismal in nature.

After the sixth century, the baptism of children began to predominate, so the mass baptism of adults on Easter was gradually abandoned.
It was then that the meaning of Pentecost was changed – from a catechetical period, fasting became a period of repentance for members of the Church.

In the ninth century, Pentecost the word stems from the Greek Πεντηκοστή (Pentēkostē) meaning "fiftieth" was finally united with Holy Week, and so the duration of Lent increased.

The length of Lent varied, depending on how local churches viewed the inclusion of Holy Week at Pentecost and whether they considered Saturdays and Sundays, when canons forbid fasting, to be part of it.

In the Constantinople Statutes (followed by our Bulgarian Orthodox Church), Holy Week is not considered part of Pentecost, and Saturdays and Sundays are included in the Lent period, although they are not Lent days in the full sense of the word.

Thus, according to the Constantinople Statute, The Pentecost Lent had 6 weeks of 7 days, ie 42 days.
If Lazarus Saturday and Palm Sunday are excluded from it, the duration of Lent is exactly 40 days.

According to this statute, Lent begins on Maundy Monday from the first week of Lent and ends on Friday of the sixth week, that is, on the eve of Palm Sunday.

The troparions included in the Triodion (Постен Триод – The Church Service book with sung text used during the Lent, for this day speak of the "fulfillment of the soul-beneficial fortieth Pentecost" and the anticipation of the "holy week of the Passion."

The interpretation of the rule in the Apostolic Decrees (Church rules guidance book text from the end of the IV century) is similar, where it says:

"Perform this fast before Easter, beginning on the second day (that is, Monday) and ending on Friday. After these days, as completing the fasting, begin the holy week of Easter by fasting through it with fear and trembling."

It is no coincidence that the liturgies of Lazarus Sabbath and the Lord's Entrance into Jerusalem have baptismal elements.

According to another tradition, reflected in the 29th canon of the Sixth Ecumenical Council (681) – that is the year of Creation of Today's country of Bulgaria (which is the only country in Europe that did not change his name as of year 681), Holy Week was part of Pentecost, where it is called "the last week of Pentecost".

This other practice is preserved by the ancient churches, which separated from Orthodoxy after the Fourth Ecumenical Council in Chalcedon (451) – The Armenian, Coptic, Syriac Orthodox Church of Antioch, Ethiopian Church of Toledo, (perhaps the Jacobite Syrian Church) etc.

Even though this historic tradition was well preserved in those Churches and many of their church order or customs such as veneration for the icons, holy relics, the problem with them preventing them to be in  ull communion with Eastern Orthodox Church stems in their rejection to accept the V-th XI-th and XII Ecumenical Ecumenical Councils and their perseverance on monophysitism (literally translated as, one nature – a teaching that says Christ has only one Nature and one Will a Godly, they say they do not reject that Christ was also real man in flesh but they consider the Godly nature of Christ has consumed the manly, which makes up their wrong understanding that Christ on the Cross did not fully suffer with his manly nature, but both God and man has suffered on the Cross – a doctrine which according to the Church councils is a pure hearesy, we can also conclude by the one nature of Christ that the so called today Oriental Orthodox Churches teach, that Christ on the Cross did not bear all the sins of the world as a man but he received all the sins and turmoils and evils as God.

In contrast in Eastern Orthodox Churches we do consider the truth that Christ has two Natures manly and Godly as well as Two Wills.
Some of the upmentioned ancient Oriental Orthodox Churches keep up to the heresy of monothelitism and that is why they're not communion with us the Eastern Orthodox.

The two wills in Orthodoxy is known under the term dyothelitism or dythelitism (stems from Greek δυοθελητισμός "doctrine of two wills") is a particular Christological doctrine that teaches the existence of two wills (divine and human) in the person of Jesus Christ.
Specifically, dyothelitism correlates the distinctiveness of two wills with the existence of two specific natures (divine and human) in the person of Jesus Christ (dyophysitism).

The Catechism of the One Holy Orthodox Church is stated: "Similarly, at the Sixth ecumenical council, Constantinople III in 681, the Church confessed that Christ possesses two wills and two natural operations, divine and human. They are not opposed to each other, but co-operate in such a way that the Word made flesh willed humanly in obedience to his Father all that he had decided divinely with the Father and the Holy Spirit for our salvation. Christ's human will 'does not resist or oppose but rather submits to his divine and almighty will.'"

This position is in opposition to the Monothelitism position in the Christological debates. The debate concerning the Monothelite churches and the Catholic Church came to a conclusion at the Third Council of Constantinople in 681. The Council declared that in line with the declarations of the Council of Chalcedon in 451, which declared two natures in the one person of Jesus Christ, there are equally two "wills" or "modes of operation" in the one person of Jesus Christ as well.

Dyothelitism was championed by Maximus the Confessor against monothelitism, the doctrine of one will. 

According to their tradition, Saturdays and Sundays as "non-fasting days" are not included in the calculation of Pentecost, so these churches fast 8 weeks for 5 days, ie 40, but fasting for pre-Chalcedonians begins one week earlier (when we have The week where orthodox stop eating Milk and Diary – Сиропустна Неделя (Milk-quit Sunday).

According to some liturgists, the appearance of the preparatory "Milk-quit" week before the beginning of Lent is the result of the desire to combine the two traditions in the Church.

Important clarification to make here is we have different view from  upmention Ancient considered schismatic Churches. Cause these ones only accept Church father decision in ecumenical councils until the 4th and cause they reject authencity of the IV th, XI th and  XII th ecumenical councils and consider Christ has only one nature a Godly one, they don't reject the existence of Human nature completely, however they stand for that Godly nature of Christ completely succumbs the human one and therefore it turns out Christ suffered on the Cross only as God (that Eastern Orthodox Churches consider as heresy).

Our believe of the Eastern Orthodox Church  Jesus Christ has two natures and two wills a Manly and Godly and his desire to humilate the Will of the Father and the Holy Spiritut to fulfill the salvational plan was voluntery.

The Roman Catholic Church since ancient times, has included Holy Week of Pentecost. However, through several councils, she lifted the ban on fasting on the Sabbath (64 Apostolic Rule). Unfortunately fasting today in Western Roman Catholic Churches is trongly reduced and all in all officially the layman in that Church has to fast about 4 days in the whole year, where in practice most people usually fast only one day on the Good Friday.

This practice is sharply condemned in the 55th canon of the Sixth Ecumenical Council. That is why the Roman Catholic Church calculates Lent as follows: 6 weeks of 6 days of fasting makes 36 days. To them are added 4.
Therefore for Catholics, the Great Lent begins on Wednesday, the so-called. Clean Wednesday (which according to Church tradition is the day on which Judah decided to betray Christ promising the Sanhedrin to sell them Christ for 30 silver coins … )

What is the reason for Holy Week Fasting

In our Eastern Orthodox Church on Holy and Great Friday, is a very holy and sad day – considered the saddest day in the year, because we sorrow for the great unrighhtousness done to King and The master of Light and Universe and Son of God Christ, being betrayed, joked and beaten in a substitute for us (as we in reality deserve this disgraceful faith for our multitude of transgressions).

Therefore the Goal of following the whole 7 days of Passion week in a Steady fasting is to cleanse up the soul and body, increase our talents (the virtues), prepare to receive Christ in His Glorious Resurrection in our Souls through the Mystery of the Mysterious – the Holy Communion and most importantly win over our sinful passion's rooted in hatred,lust, gluttony, greed, sloth, wrath, envy, pride and all evil and most importantly commune with God with constant prayer and spiritual labors.

The constant prayer is attained in church laymans differently by reading of morning, evening private rules, canons, attendance of the many, many morning and evening services.
What is unique is the church services are constructed in a way that the morning services are served in the evenings where possible after Sunrise about 19:00 o'clock, and evening services are
served in the mornings together with the Hours and on Fridays united with a Liturgy of the Presanctified gifts.

In monasteries especially in Holy Mount Athos and some of the more ascetic ones, the frequent custom is often to use with a blessing of their elder the constant repetition in one self of the so called "Jesus Prayer";

Lord Jesus Christ have mercy on me the sinner!  Lord Jesus Christ have mercy on me the sinner! Lord Jesus Christ have mercy on me the sinner! 

The weapons of the spiritual war used are abstinence of food or at least reducing the food intake and more importantly, reduce the passions. The most important fasting of course is the spiritual.

But for the spiritual advancement a good leverage shown by the Holy Fathers is the Fleshly fasting given to be followed during this week.
Fasting according to church canons for this week, includes only eating if heath allows it of raw foods, vegetables and fruits, bread and plant foods without oil, the local custom not mandatory tradition in the Bulgarian Orthodox Church is to also not eat fat containing nuts, throughout the week with exceptions on Great Thursday the day of The Last Support, where oil is allowed because of the Greatness of the Feast.

The fast during Holy Week is especially strict – "without wine and oil", ie dry foods, as only on Holy Thursday, after Holy Communion, believers used for the spiritual holiday "oil", ie vegetable oil.
Holy Sabbath was treated with special care, as it was the only Sabbath that the canons decreed as a fast day.

Fasting on Holy Saturday lasts until midnight, until the Lord's Day, when the Lord's Resurrection is announced.
The Apostolic Decrees stipulate: "The Sabbath lasts until the roosters sing, the fast ends with the coming of the first day after the Sabbath, which is the Resurrection."

Create Linux High Availability Load Balancer Cluster with Keepalived and Haproxy on Linux

Tuesday, March 15th, 2022

keepalived-logo-linux

Configuring a Linux HA (High Availibiltiy) for an Application with Haproxy is already used across many Websites on the Internet and serious corporations that has a crucial infrastructure has long time
adopted and used keepalived to provide High Availability Application level Clustering.
Usually companies choose to use HA Clusters with Haproxy with Pacemaker and Corosync cluster tools.
However one common used alternative solution if you don't have the oportunity to bring up a High availability cluster with Pacemaker / Corosync / pcs (Pacemaker Configuration System) due to fact machines you need to configure the cluster on are not Physical but VMWare Virtual Machines which couldn't not have configured a separate Admin Lans and Heartbeat Lan as we usually do on a Pacemaker Cluster due to the fact the 5 Ethernet LAN Card Interfaces of the VMWare Hypervisor hosts are configured as a BOND (e.g. all the incoming traffic to the VMWare vSphere  HV is received on one Virtual Bond interface).

I assume you have 2 separate vSphere Hypervisor Physical Machines in separate Racks and separate switches hosting the two VMs.
For the article, I'll call the two brand new brought Virtual Machines with some installation automation software such as Terraform or Ansible – vm-server1 and vm-server2 which would have configured some recent version of Linux.

In that scenario to have a High Avaiability for the VMs on Application level and assure at least one of the two is available at a time if one gets broken due toe malfunction of the HV, a Network connectivity issue, or because the VM OS has crashed.
Then one relatively easily solution is to use keepalived and configurea single High Availability Virtual IP (VIP) Address, i.e. 10.10.10.1, which would float among two VMs using keepalived so at a time at least one of the two VMs would be reachable on the Network.

haproxy_keepalived-vip-ip-diagram-linux

Having a VIP IP is quite a common solution in corporate world, as it makes it pretty easy to add F5 Load Balancer in front of the keepalived cluster setup to have a 3 Level of security isolation, which usually consists of:

1. Physical (access to the hardware or Virtualization hosts)
2. System Access (The mechanism to access the system login credetials users / passes, proxies, entry servers leading to DMZ-ed network)
3. Application Level (access to different programs behind L2 and data based on the specific identity of the individual user,
special Secondary UserID,  Factor authentication, biometrics etc.)

 

1. Install keepalived and haproxy on machines

Depending on the type of Linux OS:

On both machines
 

[root@server1:~]# yum install -y keepalived haproxy

If you have to install keepalived / haproxy on Debian / Ubuntu and other Deb based Linux distros

[root@server1:~]# apt install keepalived haproxy –yes

2. Configure haproxy (haproxy.cfg) on both server1 and server2

 

Create some /etc/haproxy/haproxy.cfg configuration

 

[root@server1:~]vim /etc/haproxy/haproxy.cfg

#———————————————————————
# Global settings
#———————————————————————
global
    log          127.0.0.1 local6 debug
    chroot       /var/lib/haproxy
    pidfile      /run/haproxy.pid
    stats socket /var/lib/haproxy/haproxy.sock mode 0600 level admin 
    maxconn      4000
    user         haproxy
    group        haproxy
    daemon
    #debug
    #quiet

#———————————————————————
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#———————————————————————
defaults
    mode        tcp
    log         global
#    option      dontlognull
#    option      httpclose
#    option      httplog
#    option      forwardfor
    option      redispatch
    option      log-health-checks
    timeout connect 10000 # default 10 second time out if a backend is not found
    timeout client 300000
    timeout server 300000
    maxconn     60000
    retries     3

#———————————————————————
# round robin balancing between the various backends
#———————————————————————

listen FRONTEND_APPNAME1
        bind 10.10.10.1:15000
        mode tcp
        option tcplog
#        #log global
        log-format [%t]\ %ci:%cp\ %bi:%bp\ %b/%s:%sp\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
        balance roundrobin
        timeout client 350000
        timeout server 350000
        timeout connect 35000
        server app-server1 10.10.10.55:30000 weight 1 check port 68888
        server app-server2 10.10.10.55:30000 weight 2 check port 68888

listen FRONTEND_APPNAME2
        bind 10.10.10.1:15000
        mode tcp
        option tcplog
        #log global
        log-format [%t]\ %ci:%cp\ %bi:%bp\ %b/%s:%sp\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
        balance roundrobin
        timeout client 350000
        timeout server 350000
        timeout connect 35000
        server app-server1 10.10.10.55:30000 weight 5
        server app-server2 10.10.10.55:30000 weight 5 

 

You can get a copy of above haproxy.cfg configuration here.
Once configured roll it on.

[root@server1:~]#  systemctl start haproxy
 
[root@server1:~]# ps -ef|grep -i hapro
root      285047       1  0 Mar07 ?        00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy   285050  285047  0 Mar07 ?        00:00:26 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

Bring up the haproxy also on server2 machine, by placing same configuration and starting up the proxy.
 

[root@server1:~]vim /etc/haproxy/haproxy.cfg


 

3. Configure keepalived on both servers

We'll be configuring 2 nodes with keepalived even though if necessery this can be easily extended and you can add more nodes.
First we make a copy of the original or existing server configuration keepalived.conf (just in case we need it later on or if you already had something other configured manually by someone – that could be so on inherited servers by other sysadmin)
 

[root@server1:~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.orig
[root@server2:~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.orig

a. Configure keepalived to serve as a MASTER Node

 

[root@server1:~]# vim /etc/keepalived/keepalived.conf

Master Node
global_defs {
  router_id server1-fqdn # The hostname of this host.
  
  enable_script_security
  # Synchro of the state of the connections between the LBs on the eth0 interface
   lvs_sync_daemon eth0
 
notification_email {
        linuxadmin@notify-domain.com     # Email address for notifications 
    }
 notification_email_from keepalived@server1-fqdn        # The from address for the notifications
    smtp_server 127.0.0.1                       # SMTP server address
    smtp_connect_timeout 15
}

vrrp_script haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
  user root
}

vrrp_instance LB_VIP_QA {
  virtual_router_id 50
  advert_int 1
  priority 51

  state MASTER
  interface eth0
  smtp_alert          # Enable Notifications Via Email
  
  authentication {
              auth_type PASS
              auth_pass testp141

    }
### Commented because running on VM on VMWare
##    unicast_src_ip 10.44.192.134 # Private IP address of master
##    unicast_peer {
##        10.44.192.135           # Private IP address of the backup haproxy
##   }

#        }
# master node with higher priority preferred node for Virtual IP if both keepalived up
###  priority 51
###  state MASTER
###  interface eth0
  virtual_ipaddress {
     10.10.10.1 dev eth0 # The virtual IP address that will be shared between MASTER and BACKUP
  }
  track_script {
      haproxy
  }
}

 

 To dowload a copy of the Master keepalived.conf configuration click here

Below are few interesting configuration variables, worthy to mention few words on, most of them are obvious by their names but for more clarity I'll also give a list here with short description of each:

 

  • vrrp_instance – defines an individual instance of the VRRP protocol running on an interface.
  • state – defines the initial state that the instance should start in (i.e. MASTER / SLAVE )state –
  • interface – defines the interface that VRRP runs on.
  • virtual_router_id – should be unique value per Keepalived Node (otherwise slave master won't function properly)
  • priority – the advertised priority, the higher the priority the more important the respective configured keepalived node is.
  • advert_int – specifies the frequency that advertisements are sent at (1 second, in this case).
  • authentication – specifies the information necessary for servers participating in VRRP to authenticate with each other. In this case, a simple password is defined.
    only the first eight (8) characters will be used as described in  to note is Important thing
    man keepalived.conf – keepalived.conf variables documentation !!! Nota Bene !!! – Password set on each node should match for nodes to be able to authenticate !
  • virtual_ipaddress – defines the IP addresses (there can be multiple) that VRRP is responsible for.
  • notification_email – the notification email to which Alerts will be send in case if keepalived on 1 node is stopped (e.g. the MASTER node switches from host 1 to 2)
  • notification_email_from – email address sender from where email will originte
    ! NB ! In order for notification_email to be working you need to have configured MTA or Mail Relay (set to local MTA) to another SMTP – e.g. have configured something like Postfix, Qmail or Postfix

b. Configure keepalived to serve as a SLAVE Node

[root@server1:~]vim /etc/keepalived/keepalived.conf
 

#Slave keepalived
global_defs {
  router_id server2-fqdn # The hostname of this host!

  enable_script_security
  # Synchro of the state of the connections between the LBs on the eth0 interface
  lvs_sync_daemon eth0
 
notification_email {
        linuxadmin@notify-host.com     # Email address for notifications
    }
 notification_email_from keepalived@server2-fqdn        # The from address for the notifications
    smtp_server 127.0.0.1                       # SMTP server address
    smtp_connect_timeout 15
}

vrrp_script haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
  user root
}

vrrp_instance LB_VIP_QA {
  virtual_router_id 50
  advert_int 1
  priority 50

  state BACKUP
  interface eth0
  smtp_alert          # Enable Notifications Via Email

authentication {
              auth_type PASS
              auth_pass testp141
}
### Commented because running on VM on VMWare    
##    unicast_src_ip 10.10.192.135 # Private IP address of master
##    unicast_peer {
##        10.10.192.134         # Private IP address of the backup haproxy
##   }

###  priority 50
###  state BACKUP
###  interface eth0
  virtual_ipaddress {
     10.10.10.1 dev eth0 # The virtual IP address that will be shared betwee MASTER and BACKUP.
  }
  track_script {
    haproxy
  }
}

 

Download the keepalived.conf slave config here

 

c. Set required sysctl parameters for haproxy to work as expected
 

[root@server1:~]vim /etc/sysctl.conf
#Haproxy config
# haproxy
net.core.somaxconn=65535
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3

4. Test Keepalived keepalived.conf configuration syntax is OK

 

[root@server1:~]keepalived –config-test
(/etc/keepalived/keepalived.conf: Line 7) Unknown keyword 'lvs_sync_daemon_interface'
(/etc/keepalived/keepalived.conf: Line 21) Unable to set default user for vrrp script haproxy – removing
(/etc/keepalived/keepalived.conf: Line 31) (LB_VIP_QA) Specifying lvs_sync_daemon_interface against a vrrp is deprecated.
(/etc/keepalived/keepalived.conf: Line 31)              Please use global lvs_sync_daemon
(/etc/keepalived/keepalived.conf: Line 35) Truncating auth_pass to 8 characters
(/etc/keepalived/keepalived.conf: Line 50) (LB_VIP_QA) track script haproxy not found, ignoring…

I've experienced this error because first time I've configured keepalived, I did not mention the user with which the vrrp script haproxy should run,
in prior versions of keepalived, leaving the field empty did automatically assumed you have the user with which the vrrp script runs to be set to root
as of RHELs keepalived-2.1.5-6.el8.x86_64, i've been using however this is no longer so and thus in prior configuration as you can see I've
set the user in respective section to root.
The error Unknown keyword 'lvs_sync_daemon_interface'
is also easily fixable by just substituting the lvs_sync_daemon_interface and lvs_sync_daemon and reloading
keepalived etc.

Once keepalived is started and you can see the process on both machines running in process list.

[root@server1:~]ps -ef |grep -i keepalived
root     1190884       1  0 18:50 ?        00:00:00 /usr/sbin/keepalived -D
root     1190885 1190884  0 18:50 ?        00:00:00 /usr/sbin/keepalived -D

Next step is to check the keepalived statuses as well as /var/log/keepalived.log

If everything is configured as expected on both keepalived on first node you should see one is master and one is slave either in the status or the log

[root@server1:~]#systemctl restart keepalived

 

[root@server1:~]systemctl status keepalived|grep -i state
Mar 14 18:59:02 server1-fqdn Keepalived_vrrp[1192003]: (LB_VIP_QA) Entering MASTER STATE

[root@server1:~]systemctl status keepalived

● keepalived.service – LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Mon 2022-03-14 18:15:51 CET; 32min ago
  Process: 1187587 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1187589 (code=exited, status=0/SUCCESS)

Mar 14 18:15:04 server1lb-fqdn Keepalived_vrrp[1187590]: Sending gratuitous ARP on eth0 for 10.44.192.142
Mar 14 18:15:50 server1lb-fqdn systemd[1]: Stopping LVS and VRRP High Availability Monitor…
Mar 14 18:15:50 server1lb-fqdn Keepalived[1187589]: Stopping
Mar 14 18:15:50 server1lb-fqdn Keepalived_vrrp[1187590]: (LB_VIP_QA) sent 0 priority
Mar 14 18:15:50 server1lb-fqdn Keepalived_vrrp[1187590]: (LB_VIP_QA) removing VIPs.
Mar 14 18:15:51 server1lb-fqdn Keepalived_vrrp[1187590]: Stopped – used 0.002007 user time, 0.016303 system time
Mar 14 18:15:51 server1lb-fqdn Keepalived[1187589]: CPU usage (self/children) user: 0.000000/0.038715 system: 0.001061/0.166434
Mar 14 18:15:51 server1lb-fqdn Keepalived[1187589]: Stopped Keepalived v2.1.5 (07/13,2020)
Mar 14 18:15:51 server1lb-fqdn systemd[1]: keepalived.service: Succeeded.
Mar 14 18:15:51 server1lb-fqdn systemd[1]: Stopped LVS and VRRP High Availability Monitor

[root@server2:~]systemctl status keepalived|grep -i state
Mar 14 18:59:02 server2-fqdn Keepalived_vrrp[297368]: (LB_VIP_QA) Entering BACKUP STATE

[root@server1:~]# grep -i state /var/log/keepalived.log
Mar 14 18:59:02 server1lb-fqdn Keepalived_vrrp[297368]: (LB_VIP_QA) Entering MASTER STATE
 

a. Fix Keepalived SECURITY VIOLATION – scripts are being executed but script_security not enabled.
 

When configurating keepalived for a first time we have faced the following strange error inside keepalived status inside keepalived.log 
 

Feb 23 14:28:41 server1 Keepalived_vrrp[945478]: SECURITY VIOLATION – scripts are being executed but script_security not enabled.

 

To fix keepalived SECURITY VIOLATION error:

Add to /etc/keepalived/keepalived.conf on the keepalived node hosts
inside 

global_defs {}

After chunk
 

enable_script_security

include

# Synchro of the state of the connections between the LBs on the eth0 interface
  lvs_sync_daemon_interface eth0

 

5. Prepare rsyslog configuration and Inlcude additional keepalived options
to force keepalived log into /var/log/keepalived.log

To force keepalived log into /var/log/keepalived.log on RHEL 8 / CentOS and other Redhat Package Manager (RPM) Linux distributions

[root@server1:~]# vim /etc/rsyslog.d/48_keepalived.conf

#2022/02/02: HAProxy logs to local6, save the messages
local7.*                                                /var/log/keepalived.log
if ($programname == 'Keepalived') then -/var/log/keepalived.log
if ($programname == 'Keepalived_vrrp') then -/var/log/keepalived.log
& stop

[root@server:~]# touch /var/log/keepalived.log

Reload rsyslog to load new config
 

[root@server:~]# systemctl restart rsyslog
[root@server:~]# systemctl status rsyslog

 

rsyslog.service – System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/rsyslog.service.d
           └─rsyslog-service.conf
   Active: active (running) since Mon 2022-03-07 13:34:38 CET; 1 weeks 0 days ago
     Docs: man:rsyslogd(8)

           https://www.rsyslog.com/doc/
 Main PID: 269574 (rsyslogd)
    Tasks: 6 (limit: 100914)
   Memory: 5.1M
   CGroup: /system.slice/rsyslog.service
           └─269574 /usr/sbin/rsyslogd -n

Mar 15 08:15:16 server1lb-fqdn rsyslogd[269574]: — MARK —
Mar 15 08:35:16 server1lb-fqdn rsyslogd[269574]: — MARK —
Mar 15 08:55:16 server1lb-fqdn rsyslogd[269574]: — MARK —

 

If once keepalived is loaded but you still have no log written inside /var/log/keepalived.log

[root@server1:~]# vim /etc/sysconfig/keepalived
 KEEPALIVED_OPTIONS="-D -S 7"

[root@server2:~]# vim /etc/sysconfig/keepalived
 KEEPALIVED_OPTIONS="-D -S 7"

[root@server1:~]# systemctl restart keepalived.service
[root@server1:~]#  systemctl status keepalived

● keepalived.service – LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-02-24 12:12:20 CET; 2 weeks 4 days ago
 Main PID: 1030501 (keepalived)
    Tasks: 2 (limit: 100914)
   Memory: 1.8M
   CGroup: /system.slice/keepalived.service
           ├─1030501 /usr/sbin/keepalived -D
           └─1030502 /usr/sbin/keepalived -D

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[root@server2:~]# systemctl restart keepalived.service
[root@server2:~]# systemctl status keepalived

6. Monitoring VRRP traffic of the two keepaliveds with tcpdump
 

Once both keepalived are up and running a good thing is to check the VRRP protocol traffic keeps fluently on both machines.
Keepalived VRRP keeps communicating over the TCP / IP Port 112 thus you can simply snoop TCP tracffic on its protocol.
 

[root@server1:~]# tcpdump proto 112

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:08:07.356187 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:08.356297 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:09.356408 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:10.356511 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:11.356655 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20

[root@server2:~]# tcpdump proto 112

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
​listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:08:07.356187 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:08.356297 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:09.356408 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:10.356511 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:11.356655 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20

As you can see the VRRP traffic on the network is originating only from server1lb-fqdn, this is so because host server1lb-fqdn is the keepalived configured master node.

It is possible to spoof the password configured to authenticate between two nodes, thus if you're bringing up keepalived service cluster make sure your security is tight at best the machines should be in a special local LAN DMZ, do not configure DMZ on the internet !!! 🙂 Or if you eventually decide to configure keepalived in between remote hosts, make sure you somehow use encrypted VPN or SSH tunnels to tunnel the VRRP traffic.

[root@server1:~]tcpdump proto 112 -vv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:36:25.530772 IP (tos 0xc0, ttl 255, id 59838, offset 0, flags [none], proto VRRP (112), length 40)
    server1lb-fqdn > vrrp.mcast.net: vrrp server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20, addrs: VIPIP_QA auth "testp431"
11:36:26.530874 IP (tos 0xc0, ttl 255, id 59839, offset 0, flags [none], proto VRRP (112), length 40)
    server1lb-fqdn > vrrp.mcast.net: vrrp server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20, addrs: VIPIP_QA auth "testp431"

Lets also check what floating IP is configured on the machines:

[root@server1:~]# ip -brief address show
lo               UNKNOWN        127.0.0.1/8 
eth0             UP             10.10.10.5/26 10.10.10.1/32 

The 10.10.10.5 IP is the main IP set on LAN interface eth0, 10.10.10.1 is the floating IP which as you can see is currently set by keepalived to listen on first node.

[root@server2:~]# ip -brief address show |grep -i 10.10.10.1

An empty output is returned as floating IP is currently configured on server1

To double assure ourselves the IP is assigned on correct machine, lets ping it and check the IP assigned MAC  currently belongs to which machine.
 

[root@server2:~]# ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.526 ms
^C
— 10.10.10.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.526/0.526/0.526/0.000 ms

[root@server2:~]# arp -an |grep -i 10.44.192.142
? (10.10.10.1) at 00:48:54:91:83:7d [ether] on eth0
[root@server2:~]# ip a s|grep -i 00:48:54:91:83:7d
[root@server2:~]# 

As you can see from below output MAC is not found in configured IPs on server2.
 

[root@server1-fqdn:~]# /sbin/ip a s|grep -i 00:48:54:91:83:7d -B1 -A1
 eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:48:54:91:83:7d brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/26 brd 10.10.1.191 scope global noprefixroute eth0

Pretty much expected MAC is on keepalived node server1.

 

7. Testing keepalived on server1 and server2 maachines VIP floating IP really works
 

To test the overall configuration just created, you should stop keeaplived on the Master node and in meantime keep an eye on Slave node (server2), whether it can figure out the Master node is gone and switch its
state BACKUP to save MASTER. By changing the secondary (Slave) keepalived to master the floating IP: 10.10.10.1 will be brought up by the scripts on server2.

Lets assume that something went wrong with server1 VM host, for example the machine crashed due to service overload, DDoS or simply a kernel bug or whatever reason.
To simulate that we simply have to stop keepalived, then the broadcasted information on VRRP TCP/IP proto port 112 will be no longer available and keepalived on node server2, once
unable to communicate to server1 should chnage itself to state MASTER.

[root@server1:~]# systemctl stop keepalived
[root@server1:~]# systemctl status keepalived

● keepalived.service – LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2022-03-15 12:11:33 CET; 3s ago
  Process: 1192001 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1192002 (code=exited, status=0/SUCCESS)

Mar 14 18:59:07 server1lb-fqdn Keepalived_vrrp[1192003]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:32 server1lb-fqdn systemd[1]: Stopping LVS and VRRP High Availability Monitor…
Mar 15 12:11:32 server1lb-fqdn Keepalived[1192002]: Stopping
Mar 15 12:11:32 server1lb-fqdn Keepalived_vrrp[1192003]: (LB_VIP_QA) sent 0 priority
Mar 15 12:11:32 server1lb-fqdn Keepalived_vrrp[1192003]: (LB_VIP_QA) removing VIPs.
Mar 15 12:11:33 server1lb-fqdn Keepalived_vrrp[1192003]: Stopped – used 2.145252 user time, 15.513454 system time
Mar 15 12:11:33 server1lb-fqdn Keepalived[1192002]: CPU usage (self/children) user: 0.000000/44.555362 system: 0.001151/170.118126
Mar 15 12:11:33 server1lb-fqdn Keepalived[1192002]: Stopped Keepalived v2.1.5 (07/13,2020)
Mar 15 12:11:33 server1lb-fqdn systemd[1]: keepalived.service: Succeeded.
Mar 15 12:11:33 server1lb-fqdn systemd[1]: Stopped LVS and VRRP High Availability Monitor.

 

On keepalived off, you will get also a notification Email on the Receipt Email configured from keepalived.conf from the working keepalived node with a simple message like:

=> VRRP Instance is no longer owning VRRP VIPs <=

Once keepalived is back up you will get another notification like:

=> VRRP Instance is now owning VRRP VIPs <=

[root@server2:~]# systemctl status keepalived
● keepalived.service – LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-03-14 18:13:52 CET; 17h ago
  Process: 297366 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 297367 (keepalived)
    Tasks: 2 (limit: 100914)
   Memory: 2.1M
   CGroup: /system.slice/keepalived.service
           ├─297367 /usr/sbin/keepalived -D -S 7
           └─297368 /usr/sbin/keepalived -D -S 7

Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: Remote SMTP server [127.0.0.1]:25 connected.
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: SMTP alert successfully sent.
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: (LB_VIP_QA) Sending/queueing gratuitous ARPs on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1

[root@server2:~]#  ip addr show|grep -i 10.10.10.1
    inet 10.10.10.1/32 scope global eth0
    

As you see the VIP is now set on server2, just like expected – that's OK, everything works as expected. If the IP did not move double check the keepalived.conf on both nodes for errors or misconfigurations.

To recover the initial order of things so server1 is MASTER and server2 SLAVE host, we just have to switch on the keepalived on server1 machine.

[root@server1:~]# systemctl start keepalived

The automatic change of server1 to MASTER node and respective move of the VIP IP is done because of the higher priority (of importance we previously configured on server1 in keepalived.conf).
 

What we learned?
 

So what we learned in  this article?
We have seen how to easily install and configure a High Availability Load balancer with Keepalived with single floating VIP IP address with 1 MASTER and 1 SLAVE host and a Haproxy example config with few frontends / App backends. We have seen how the config can be tested for potential errors and how we can monitor whether the VRRP2 network traffic flows between nodes and how to potentially debug it further if necessery.
Further on rawly explained some of the keepalived configurations but as keepalived can do pretty much more,for anyone seriously willing to deal with keepalived on a daily basis or just fine tune some already existing ones, you better read closely its manual page "man keepalived.conf" as well as the official Redhat Linux documentation page on setting up a Linux cluster with Keepalived (Be prepare for a small nightmare as the documentation of it seems to be a bit chaotic, and even I would say partly missing or opening questions on what does the developers did meant – not strange considering the havoc that is pretty much as everywhere these days.)

Finally once keepalived hosts are prepared, it was shown how to test the keepalived application cluster and Floating IP does move between nodes in case if one of the 2 keepalived nodes is inaccessible.

The same logic can be repeated multiple times and if necessery you can set multiple VIPs to expand the HA reachable IPs solution.

high-availability-with-two-vips-example-diagram

The presented idea is with haproxy forward Proxy server to proxy requests towards Application backend (servince machines), however if you need to set another set of server on the flow to  process HTML / XHTML / PHP / Perl / Python  programming code, with some common Webserver setup ( Nginx / Apache / Tomcat / JBOSS) and enable SSL Secure certificate with lets say Letsencrypt, this can be relatively easily done. If you want to implement letsencrypt and a webserver check this redundant SSL Load Balancing with haproxy & keepalived article.

That's all folks, hope you enjoyed.
If you need to configure keepalived Cluster or a consultancy write your query here 🙂

Saint Prophor Pchinski Saint Jochichim of Osogovo and Saint Gabriel of Lesnovo the three little known Bulgarian spritual followers of Saint John of Rila

Saturday, January 15th, 2022

Biography of St. Prohor Pshinski

Saint_Prohor_Pchinski-face-icon

St. Prohor Pshinski. Mural from the 15th century in the church "St. Archangel Michael" in Saparevo near Kyustendil. Source: bartol, bartol.blog.bg

 St. Prohor Pshinski. Mural from the 15th century in the church "St. Archangel Michael" in Saparevo near Kyustendil. Source: bartol, bartol.blog.bg The Rev.

Prohor Pshinski was a Bulgarian by birth from pious parents in the Ovce Pole region of northern Macedonia. It was given by God to childless parents for their prayers and sucked in along with his mother's milk and her high piety. When he grew up, his parents insisted on marrying him. But he once heard in the temple the words of the Savior: "He that loveth father or mother more than me is not worthy of me," (Matt. 10:37). the town of Vranje in Yugoslavia. He lived in a cave where water springs for 32 years. Once a frightened deer ran to him, chased by a hunter who soon appeared.

Saint_Prohor_Pchinski_Fresco-Byzantine-Empire-icon
Prophor Pchinski (Pshinksi) Byzantine Empire Icon

At his first meeting, the monk called the hunter by name and predicted that he would soon become a Byzantine emperor. At that time Bulgaria was under Byzantine slavery (1018 – 1186). And indeed, after some time this man reigned under the name of Roman Diogenes (1067-1071). Astonished by the fulfillment of this prophecy, he discovered the incorruptible relics of the late Prohor Pshinski and built a large temple in the name of the saint on the site of his asceticism. His holy relics were laid there and a monastic fraternity gathered.

This monastery still exists. The Venerable Prohor Pshinski died on September 14, but due to the great feast of the Exaltation of the Holy Cross, the celebration of his memory was postponed to January 15. His monastery celebrates his memory on September 19.

© Lives of the Saints. Synodal Publishing House, Sofia, 1991, edited by Parthenius, Bishop of Lefkada (Levkijski) and Archimandrite Dr. Athanasius (Bonchev).

Saint_Prohor_Pshinski-XV-century-wall-painting-icon-Kyustendil-Bulgaria

Saint Prohor Pchinski (Pshinski) Wall Painting icon XV century Kyustendil Bulgaria

The prologue biography of St. Prohor Pshinski is an original ancient Bulgarian writting, known in a single transcript in the New Prologue from the beginning of the 14th century (GIM, Uvar. 70). Under the date of October 19, the Venerable Hermit Saint is mentioned next to St. Ivan Rilski. Both texts are published by Kl. Ivanova (1977). D. Chesmedjiev (2009) points out that the cult of St. Prohor Pshinski is poorly fixed in the written tradition. All the details about the saint are known from his prologue. St. Prohor lived in the 11th century, working in the Kozyak mountain (near the village of Staro Nagorichino). After his death, his relics were transferred to the church he founded, called Pshinya, where he was healed.

Saint Prophor Pchinski in Modern Theology

His cult probably originated during the Byzantine rule. His memory is celebrated on September 13 and October 19. In the New Prologue, in addition to biographies of St. Ivan Rilski and St. Prohor Pshinski, there are also biographies of St. Achilles of Larissa, St. Simeon of Serbia and others. 30 years ago Kl. Ivanova (Ivanova 1977: 59) has suggested that the manuscript was compiled in the Pshin Monastery itself, but the spelling and language features of the collection are Serbian. According to the latest research by the same researcher, Nora's prologue originates from Jerusalem and was created in the Serbian monastery "St. Archangel Michael ”(Ivanova 2008: 68–70).

 

Biography of Saint Joachim of Osogovo

Images of St. Ivan Rilski and St. Joachim of Sarandapor from the Poganovo Monastery, end of the 15th century. It is not known where he came from, from his life it is known that he came from the west, according to Ivan Snegarov, perhaps from Zeta, in the Osogovo Mountains, where he sought monastic asylum.

An unknown boyar from the village of Gradets, not far from Kriva Palanka, today in northern Macedonia, shows him the place he was looking for monastic solitude – a cave by the Sarandapor River, today's Kriva River. Here St. Joachim spends his life as a hermit, in fasting and prayer, and the local Bulgarians revere him as a holy man. 

Osogovo Monastery.

Kutugenski-Manastir-Sveti-Joachim-Ioakim-Osogovski
Osogovo Monastery Saint Joachim Icon

Joachim Osogowski died on August 16, 1105.On this date the Bulgarian Orthodox Church commemorates him.

Years later, the widowed priest Theodore of the Sheep Field settled in the place of Joachim's hermitage, adopting the monastic name Theophanes. After his saint appears, he discovers his miraculous relics, which are laid in the church built in his memory. In the 12th century, the cult of St. Joachim became so popular in northwestern Macedonia that a monastery of the same name was built around the temple. 

Joachim-of-Osogovo-known-also-as-Sarandopolski-Poganovo_Ivan_Rilski

Images of St. Ivan Rilski and St. Joachim of Sarandapor from the Poganovo Monastery, end of the XV-th century ( the three most famous spiritual pupils of Saint John of Rila )

In the monastery, similar to the life of St. Ivan Rilski, a life of St. Joachim was created, known from later transcripts, as well as a service of the saint. Around the middle of the 14th century the cult of St. Joachim spread to the eastern Bulgarian lands, and at the end of the 14th century it was transferred to Russia.

Biography of Saint Gavriil (Gabriel) Lesnovski

Saint-Gabriel-of-Lesnovo-320px-Archangels_Chapel_in_Rila_Monastery_Gabriel_of_Lesnovo_-_year-1845

Saint Gabriel of Lesnovo fresco from Archangels Chapel Rila Monastery, Bulgaria

Saint_Gabriel-of-Lesnovo-icon-St-Alexander-Nevski
St. Gavriil Lesnowski.
Detail of a mosaic on one of the doors of the Patriarchal Cathedral "St. Alexander Nevsky" in Sofia St. Gavriil Lesnowski. Detail of a mosaic on one of the doors of the Patriarchal Cathedral "St. Alexander Nevsky" in Sofia

Reverend Gavriil Lesnovski is one of the three great followers of the Rila desert dweller St. Ivan Rilski. He lived in the XI – XII century. He was born in the village of Osiche, Palaneshko (Macedonia). He came from rich Bulgarian parents and received a good education. When he was old, his parents betrothed him to a good-looking bride. Soon, however, his fiancée died. Then he entered a monastery and became a monk. With the inheritance he received from his parents, Gabriel built a monastery with a church named after St. Archangel Michael in the Lesnovo Mountains, northern Macedonia, near the present town of Kratovo.

The Venerable Father gathered monks, appointed an abbot, and he himself secluded himself in the mountains of desert life and silence, doing so for 30 years. He then returned to his monastery and died on January 15. Thirty years after the death of the Venerable Gabriel, a Russian monk named Joseph in the town of Sredets (Sofia) – at the suggestion of the saint – went to the place where the Venerable struggled, found his grave, found his incorruptible relics, laid them in a coffin and provided for prayer worship to believers.

For many years the holy relics of the Reverend Gabriel rested in the Lesnovo Monastery and performed many miracles. Probably in the thirteenth century one of the Bulgarian kings of the Assen dynasty brought them to his capital Tarnovo and laid them in the church "Holy Apostles" in Trapezitsa. According to the Reverend's prologue of 1330, "they have lain there until now and give healing." After the Ottoman invasion of Bulgaria, traces of the holy relics of the Venerable Gavriil Lesnovski are lost. The monastery he founded was later named after him and became an important literary center. © Lives of the Saints. Synodal Publishing House, Sofia, 1991, edited by Parthenius, Bishop of Lefkada (Levkijski) and Archimandrite Dr. Athanasius (Bonchev).

Saint-Gabriel_Lesnovski-Saint-_Joachom_Osogovski_and-saint_Prohor-Pchinski-Saint_Alexander_Cathedral_SofiaSaint Gabriel of Lesnovo, Saint Joachim of Osogovo and Saint Prohor Pchinski mosaic saint Alexander Nevski Cathedral Church, Sofia, Bulgaria

The Saints Prohor Pchinski, Joachim of Osogovo together with Saint Gabriel of Lesnovo according to Bulgarian Orthodox Church tradition are considered to be 3 of the many pupil monks of Saint John of Rila who spread the light of Holy Eastern Orthodox Christian faith in whole Bulgarian lands and from there towards Russia and far west Serbia, Croatia, Hungary who historically has been orthodox and later converted to Roman Catholicism.

Install and configure rkhunter for improved security on a PCI DSS Linux / BSD servers with no access to Internet

Wednesday, November 10th, 2021

install-and-configure-rkhunter-with-tightened-security-variables-rkhunter-logo

rkhunter or Rootkit Hunter scans systems for known and unknown rootkits. The tool is not new and most system administrators that has to mantain some good security servers perhaps already use it in their daily sysadmin tasks.

It does this by comparing SHA-1 Hashes of important files with known good ones in online databases, searching for default directories (of rootkits), wrong permissions, hidden files, suspicious strings in kernel modules, commmon backdoors, sniffers and exploits as well as other special tests mostly for Linux and FreeBSD though a ports for other UNIX operating systems like Solaris etc. are perhaps available. rkhunter is notable due to its inclusion in popular mainstream FOSS operating systems (CentOS, Fedora,Debian, Ubuntu etc.).

Even though rkhunter is not rapidly improved over the last 3 years (its last Official version release was on 20th of Febuary 2018), it is a good tool that helps to strengthen even further security and it is often a requirement for Unix servers systems that should follow the PCI DSS Standards (Payment Card Industry Data Security Standards).

Configuring rkhunter is a pretty straight forward if you don't have too much requirements but I decided to write this article for the reason there are fwe interesting options that you might want to adopt in configuration to whitelist any files that are reported as Warnings, as well as how to set a configuration that sets a stricter security checks than the installation defaults. 

1. Install rkhunter .deb / .rpm package depending on the Linux distro or BSD

  • If you have to place it on a Redhat based distro CentOS / Redhat / Fedora

[root@Centos ~]# yum install -y rkhunter

 

  • On Debian distros the package name is equevallent to install there exec usual:

root@debian:~# apt install –yes rkhunter

  • On FreeBSD / NetBSD or other BSD forks you can install it from the BSD "World" ports system or install it from a precompiled binary.

freebsd# pkg install rkhunter

One important note to make here is to have a fully functional Alarming from rkhunter, you will have to have a fully functional configured postfix / exim / qmail whatever mail server to relay via official SMTP so you the Warning Alarm emails be able to reach your preferred Alarm email address. If you haven't installed postfix for example and configure it you might do.

– On Deb based distros 

[root@Centos ~]#yum install postfix


– On RPM based distros

root@debian:~# apt-get install –yes postfix


and as minimum, further on configure some functional Email Relay server within /etc/postfix/main.cf
 

# vi /etc/postfix/main.cf
relayhost = [relay.smtp-server.com]

2. Prepare rkhunter.conf initial configuration


Depending on what kind of files are present on the filesystem it could be for some reasons some standard package binaries has to be excluded for verification, because they possess unusual permissions because of manual sys admin monification this is done with the rkhunter variable PKGMGR_NO_VRFY.

If remote logging is configured on the system via something like rsyslog you will want to specificly tell it to rkhunter so this check as a possible security issue is skipped via ALLOW_SYSLOG_REMOTE_LOGGING=1. 

In case if remote root login via SSH protocol is disabled via /etc/ssh/sshd_config
PermitRootLogin no variable, the variable to include is ALLOW_SSH_ROOT_USER=no

It is useful to also increase the hashing check algorithm for security default one SHA256 you might want to change to SHA512, this is done via rkhunter.conf var HASH_CMD=SHA512

Triggering new email Warnings has to be configured so you receive, new mails at a preconfigured mailbox of your choice via variable
MAIL-ON-WARNING=SetMailAddress

 

# vi /etc/rkhunter.conf

PKGMGR_NO_VRFY=/usr/bin/su

PKGMGR_NO_VRFY=/usr/bin/passwd

ALLOW_SYSLOG_REMOTE_LOGGING=1

# Needed for corosync/pacemaker since update 19.11.2020

ALLOWDEVFILE=/dev/shm/qb-*/qb-*

# enabled ssh root access skip

ALLOW_SSH_ROOT_USER=no

HASH_CMD=SHA512

# Email address to sent alert in case of Warnings

MAIL-ON-WARNING=Your-Customer@Your-Email-Server-Destination-Address.com

MAIL-ON-WARNING=Your-Second-Peronsl-Email-Address@SMTP-Server.com

DISABLE_TESTS=os_specific


Optionally if you're using something specific such as corosync / pacemaker High Availability cluster or some specific software that is creating /dev/ files identified as potential Risks you might want to add more rkhunter.conf options like:
 

# Allow PCS/Pacemaker/Corosync
ALLOWDEVFILE=/dev/shm/qb-attrd-*
ALLOWDEVFILE=/dev/shm/qb-cfg-*
ALLOWDEVFILE=/dev/shm/qb-cib_rw-*
ALLOWDEVFILE=/dev/shm/qb-cib_shm-*
ALLOWDEVFILE=/dev/shm/qb-corosync-*
ALLOWDEVFILE=/dev/shm/qb-cpg-*
ALLOWDEVFILE=/dev/shm/qb-lrmd-*
ALLOWDEVFILE=/dev/shm/qb-pengine-*
ALLOWDEVFILE=/dev/shm/qb-quorum-*
ALLOWDEVFILE=/dev/shm/qb-stonith-*
ALLOWDEVFILE=/dev/shm/pulse-shm-*
ALLOWDEVFILE=/dev/md/md-device-map
# Needed for corosync/pacemaker since update 19.11.2020
ALLOWDEVFILE=/dev/shm/qb-*/qb-*

# tomboy creates this one
ALLOWDEVFILE="/dev/shm/mono.*"
# created by libv4l
ALLOWDEVFILE="/dev/shm/libv4l-*"
# created by spice video
ALLOWDEVFILE="/dev/shm/spice.*"
# created by mdadm
ALLOWDEVFILE="/dev/md/autorebuild.pid"
# 389 Directory Server
ALLOWDEVFILE=/dev/shm/sem.slapd-*.stats
# squid proxy
ALLOWDEVFILE=/dev/shm/squid-cf*
# squid ssl cache
ALLOWDEVFILE=/dev/shm/squid-ssl_session_cache.shm
# Allow podman
ALLOWDEVFILE=/dev/shm/libpod*lock*

 

3. Set the proper mirror database URL location to internal network repository

 

Usually  file /var/lib/rkhunter/db/mirrors.dat does contain Internet server address where latest version of mirrors.dat could be fetched, below is how it looks by default on Debian 10 Linux.

root@debian:/var/lib/rkhunter/db# cat mirrors.dat 
Version:2007060601
mirror=http://rkhunter.sourceforge.net
mirror=http://rkhunter.sourceforge.net

As you can guess a machine that doesn't have access to the Internet neither directly, neither via some kind of secure proxy because it is in a Paranoic Demilitarized Zone (DMZ) Network with many firewalls. What you can do then is setup another Mirror server (Apache / Nginx) within the local PCI secured LAN that gets regularly the database from official database on http://rkhunter.sourceforge.net/ (by installing and running rkhunter –update command on the Mirror WebServer and copying data under some directory structure on the remote local LAN accessible server, to keep the DB uptodate you might want to setup a cron to periodically copy latest available rkhunter database towards the http://mirror-url/path-folder/)

# vi /var/lib/rkhunter/db/mirrors.dat

local=http://rkhunter-url-mirror-server-url.com/rkhunter/1.4/


A mirror copy of entire db files from Debian 10.8 ( Buster ) ready for download are here.

Update entire file property db and check for rkhunter db updates

 

# rkhunter –update && rkhunter –propupdate

[ Rootkit Hunter version 1.4.6 ]

Checking rkhunter data files…
  Checking file mirrors.dat                                  [ Skipped ]
  Checking file programs_bad.dat                             [ No update ]
  Checking file backdoorports.dat                            [ No update ]
  Checking file suspscan.dat                                 [ No update ]
  Checking file i18n/cn                                      [ No update ]
  Checking file i18n/de                                      [ No update ]
  Checking file i18n/en                                      [ No update ]
  Checking file i18n/tr                                      [ No update ]
  Checking file i18n/tr.utf8                                 [ No update ]
  Checking file i18n/zh                                      [ No update ]
  Checking file i18n/zh.utf8                                 [ No update ]
  Checking file i18n/ja                                      [ No update ]

 

rkhunter-update-propupdate-screenshot-centos-linux


4. Initiate a first time check and see whether something is not triggering Warnings

# rkhunter –check

rkhunter-checking-for-rootkits-linux-screenshot

As you might have to run the rkhunter multiple times, there is annoying Press Enter prompt, between checks. The idea of it is that you're able to inspect what went on but since usually, inspecting /var/log/rkhunter/rkhunter.log is much more easier, I prefer to skip this with –skip-keypress option.

# rkhunter –check  –skip-keypress


5. Whitelist additional files and dev triggering false warnings alerts


You have to keep in mind many files which are considered to not be officially PCI compatible and potentially dangerous such as lynx browser curl, telnet etc. might trigger Warning, after checking them thoroughfully with some AntiVirus software such as Clamav and checking the MD5 checksum compared to a clean installed .deb / .rpm package on another RootKit, Virus, Spyware etc. Clean system (be it virtual machine or a Testing / Staging) machine you might want to simply whitelist the files which are incorrectly detected as dangerous for the system security.

Again this can be achieved with

PKGMGR_NO_VRFY=

Some Cluster softwares that are preparing their own /dev/ temporary files such as Pacemaker / Corosync might also trigger alarms, so you might want to suppress this as well with ALLOWDEVFILE

ALLOWDEVFILE=/dev/shm/qb-*/qb-*


If Warnings are found check what is the issue and if necessery white list files due to incorrect permissions in /etc/rkhunter.conf .

rkhunter-warnings-found-screenshot

Re-run the check until all appears clean as in below screenshot.

rkhunter-clean-report-linux-screenshot

Fixing Checking for a system logging configuration file [ Warning ]

If you happen to get some message like, message appears when rkhunter -C is done on legacy CentOS release 6.10 (Final) servers:

[13:45:29] Checking for a system logging configuration file [ Warning ]
[13:45:29] Warning: The 'systemd-journald' daemon is running, but no configuration file can be found.
[13:45:29] Checking if syslog remote logging is allowed [ Allowed ]

To fix it, you will have to disable SYSLOG_CONFIG_FILE at all.
 

SYSLOG_CONFIG_FILE=NONE