Archive for August, 2021

Fix Out of inodes on Postfix Linux Mail Cluster. How to clean up filesystem running out of Inodes, Filesystem inodes on partition is 100% full

Wednesday, August 25th, 2021

Inode_Entry_inode-table-content

Recently we have faced a strange issue with with one of our Clustered Postfix Mail servers (the cluster is with 2 nodes that each has configured Postfix daemon mail servers (running on an OpenVZ virtualized environment).
A heartbeat that checks liveability of clusters and switches nodes in case of one of the two gets broken due to some reason), pretty much a standard SMTP cluster.

So far so good but since the cluster is a kind of abondoned and is pretty much legacy nowadays and used just for some Monitoring emails from different scripts and systems on servers, it was not really checked thoroughfully for years and logically out of sudden the alarming email content sent via the cluster stopped working.

The normal sysadmin job here  was to analyze what is going on with the cluster and fix it ASAP. After some very basic analyzing we catched the problem is caused by a  "inodes full" (100% of available inodes were occupied) problem, e.g. file system run out of inodes on both machines perhaps due to a pengine heartbeat process  bug  leading to producing a high number of .bz2 pengine recovery archive files stored in /var/lib/pengine>

Below are the few steps taken to analyze and fix the problem.
 

1. Finding out about the the system run out of inodes problem


After logging on to system and not finding something immediately is wrong with inodes, all I can see from crm_mon is cluster was broken.
A plenty of emails were left inside the postfix mail queue visible with a standard command

[root@smtp1: ~ ]# postqueue -p

It took me a while to find ot the problem is with inodes because a simple df -h  was showing systems have enough space but still cluster quorum was not complete.
A bit of further investigation led me to a  simple df -i reporting the number of inodes on the local filesystems on both our SMTP1 and SMTP2 got all occupied.

[root@smtp1: ~ ]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/simfs            500000   500000  0   100% /
none                   65536      61   65475    1% /dev

As you can see the number of inodes on the Virual Machine are unfortunately depleted

Next step was to check directories occupying most inodes, as this is the place from where files could be temporary moved to a remote server filesystem or moved to another partition with space on a server locally attached drives.
Below command gives an ordered list with directories locally under the mail root filesystem / and its respective occupied number files / inodes,
the more files under a directory the more inodes are being occupied by the files on the filesystem.

 

run-out-if-inodes-what-is-inode-find-out-which-filesystem-or-directory-eating-up-all-your-system-inodes-linux_inode_diagram.gif
1.1 Getting which directory consumes most of the inodes on the systems

 

[root@smtp1: ~ ]# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
….
…..

…….
    586 /usr/lib64/python2.4
    664 /usr/lib64
    671 /usr/share/man/man8
    860 /usr/bin
   1006 /usr/share/man/man1
   1124 /usr/share/man/man3p
   1246 /var/lib/Pegasus/prev_repository_2009-03-10-1236698426.308128000.rpmsave/root#cimv2/classes
   1246 /var/lib/Pegasus/prev_repository_2009-05-18-1242636104.524113000.rpmsave/root#cimv2/classes
   1246 /var/lib/Pegasus/prev_repository_2009-11-06-1257494054.380244000.rpmsave/root#cimv2/classes
   1246 /var/lib/Pegasus/prev_repository_2010-08-04-1280907760.750543000.rpmsave/root#cimv2/classes
   1381 /var/lib/Pegasus/prev_repository_2010-11-15-1289811714.398469000.rpmsave/root#cimv2/classes
   1381 /var/lib/Pegasus/prev_repository_2012-03-19-1332151633.572875000.rpmsave/root#cimv2/classes
   1398 /var/lib/Pegasus/repository/root#cimv2/classes
   1696 /usr/share/man/man3
   400816 /var/lib/pengine

Note, the above command orders the files from bottom to top order and obviosuly the bottleneck directory that is over-eating Filesystem inodes with an exceeding amount of files is
/var/lib/pengine
 

2. Backup old multitude of files just in case of something goes wrong with the cluster after some files are wiped out


The next logical step of course is to check what is going on inside /var/lib/pengine just to find a very ,very large amount of pe-input-*NUMBER*.bz2 files were suddenly produced.

 

[root@smtp1: ~ ]# ls -1 pe-input*.bz2 | wc -l
 400816


The files are produced by the pengine process which is one of the processes that is controlling the heartbeat cluster state, presumably it is done by running process:

[root@smtp1: ~ ]# ps -ef|grep -i pengine
24        5649  5521  0 Aug10 ?        00:00:26 /usr/lib64/heartbeat/pengine


Hence in order to fix the issue, to prevent some inconsistencies in the cluster due to the file deletion,  copied the whole directory to another mounted parition (you can mount it remotely with sshfs for example) or use a local one if you have one:

[root@smtp1: ~ ]# cp -rpf /var/lib/pengine /mnt/attached_storage


and proceeded to clean up some old multitde of files that are older than 2 years of times (720 days):


3. Clean  up /var/lib/pengine files that are older than two years with short loop and find command

 


First I made a list with all the files to be removed in external text file and quickly reviewed it by lessing it like so

[root@smtp1: ~ ]#  cd /var/lib/pengine
[root@smtp1: ~ ]# find . -type f -mtime +720|grep -v pe-error.last | grep -v pe-input.last |grep -v pe-warn.last -fprint /home/myuser/pengine_older_than_720days.txt
[root@smtp1: ~ ]# less /home/myuser/pengine_older_than_720days.txt


Once reviewing commands I've used below command to delete the files you can run below command do delete all older than 2 years that are different from pe-error.last / pe-input.last / pre-warn.last which might be needed for proper cluster operation.

[root@smtp1: ~ ]#  for i in $(find . -type f -mtime +720 -exec echo '{}' \;|grep -v pe-error.last | grep -v pe-input.last |grep -v pe-warn.last); do echo $i; done


Another approach to the situation is to simply review all the files inside /var/lib/pengine and delete files based on year of creation, for example to delete all files in /var/lib/pengine from 2010, you can run something like:
 

[root@smtp1: ~ ]# for i in $(ls -al|grep -i ' 2010 ' | awk '{ print $9 }' |grep -v 'pe-warn.last'); do rm -f $i; done


4. Monitor real time inodes freeing

While doing the clerance of old unnecessery pengine heartbeat archives you can open another ssh console to the server and view how the inodes gets freed up with a command like:

 

# check if inodes is not being rapidly decreased

[root@csmtp1: ~ ]# watch 'df -i'


5. Restart basic Linux services producing pid files and logs etc. to make then workable (some services might not be notified the inodes on the Hard drive are freed up)

Because the hard drive on the system was full some services started to misbehaving and /var/log logging was impacted so I had to also restart them in our case this is the heartbeat itself
that  checks clusters nodes availability as well as the logging daemon service rsyslog

 

# restart rsyslog and heartbeat services
[root@csmtp1: ~ ]# /etc/init.d/heartbeat restart
[root@csmtp1: ~ ]# /etc/init.d/rsyslog restart

The systems had been a data integrity legacy service samhain so I had to restart this service as well to reforce the /var/log/samhain log file to again continusly start writting data to HDD.

# Restart samhain service init script 
[root@csmtp1: ~ ]# /etc/init.d/samhain restart


6. Check up enough inodes are freed up with df

[root@smtp1 log]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 410531 19469 91% /
none 65536 61 65475 1% /dev


I had to repeat the same process on the second Postfix cluster node smtp2, and after all the steps like below check the status of smtp2 node and the postfix queue, following same procedure made the second smtp2 cluster member as expected 🙂

 

7. Check the cluster node quorum is complete, e.g. postfix cluster is operating normally

 

# Test if email cluster is ok with pacemaker resource cluster manager – lt-crm_mon
 

[root@csmtp1: ~ ]# crm_mon -1
============
Last updated: Tue Aug 10 18:10:48 2021
Stack: Heartbeat
Current DC: smtp2.fqdn.com (bfb3d029-89a8-41f6-a9f0-52d377cacd83) – partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============

Online: [ smtp2.fqdn.com smtp1.fqdn.com ]

failover-ip (ocf::heartbeat:IPaddr2): Started csmtp1.ikossvan.de
Clone Set: postfix_clone
Started: [ smtp2.fqdn.com smtp1fqdn.com ]
Clone Set: pingd_clone
Started: [ smtp2.fqdn.com smtp1.fqdn.com ]
Clone Set: mailto_clone
Started: [ smtp2.fqdn.com smtp1.fqdn.com ]

 

8.  Force resend a few hundred thousands of emails left in the email queue


After some inodes gets freed up due to the file deletion, i've reforced a couple of times the queued mail servers to be immediately resent to remote mail destinations with cmd:

 

# force emails in queue to be resend with postfix

[root@smtp1: ~ ]# sendmail -q


– It was useful to watch in real time how the queued emails are quickly decreased (queued mails are successfully sent to destination addresses) with:

 

# Monitor  the decereasing size of the email queue
[root@smtp1: ~ ]# watch 'postqueue -p|grep -i '@'|wc -l'

The Dormition of Virgin Mary the Mother of the Lord Jesus Christ, the Eastern Orthodox Church view and few words on feast origin

Tuesday, August 17th, 2021

Dormition-of-Mother-of-God-Koimesis_Greek_Mosaic_at_Chora_Church_Constantinople

Dormition of the Mother of God is a feast highly venerated in both Eastern Orthodox Churches as well as Oriental Orthodox Churches (The Coptic Church and the Armenian Apostolic Church which venerated the feast not on a fixed date), whether the rest of Orthodox world do celebrate on 15 of August the Dormition of the Mother of God. The so called old calendar Churches do celebrate the feast on 28 of August as the old church calendar coincides with this date on the public national calendar the countries use nowadays, whether some of the Churches such as our Bulgarian Orthodox Church, the Romanian Orthodox Church as well the Greek orthodox Church do celebarete on 15 of August.

Dormition_de_la_Vierge-stone-icon

 

The feast is one of the major Church feasts also in Western Roman Cathic church world known as the Assumption of Mary. The way the feast is venerated varyies, but it is important to say this feast is "The Summer Easter of the Church". In the ancient times the feast has been preceded in the Church by two fastings one for the Transfiguration of God Lord Jesus Christ that was venerated on 6th of August and one week fast preceding the Dormition or the Sleeping like death called in Church Slavonic Uspenie (falling asleep) that the Mother of God experineced before her righteous Soul passed to the Lord Jesus Christ. The painless death of the Mother of God has occured according to the Byzantine author Hypollitos of Thebes who lived in VIII centuryin 41 A.D., 11 years after Jesus's Crucifix and Glorious Resurrection.

Dormition_of_the_Mother_of_God_Bulgarian-icon-from_year-1893

Bulgarian icon from year 1893

The Dormition term expresses the Church belief that Virgin Mary on her death bed has been in a completely peaceful state of the Soul that and did not suffered the terrible pains of death and division of soul and body that each of us the sinners faces because of her extraordinary righteousness and as a blessing of Christ for the ever Virgin, because of her humility and life lived in service of God, and poor and every of her near sides and for her unceasing prayer for the world that she attained even during her earthly life time. The belief of Dormition is not a Bible found doctrine but a Church tradiiton and due to that protestant do not consider it as of a high value but we orthodox firmly believe this was not mention in the bible, because the Mother of God herself unwillingness for glory because she found unworthy compared to the great miracles life, suffering and salvationory plan Jesus completed. The dormition of Mother of God is found in many of Apocryphal writtings which never become official part of the Church canon.

In the language of the scripture, death is often called a "sleeping" or "falling asleep" (Greek κοίμησις; whence κοιμητήριον > coemetērium > cemetery, "a place of sleeping"). A prominent example of this is the name of this feast; another is the Dormition of Anna, Mary's mother.

The first Christian centuries may be silent, but "The Dormition/Assumption of Mary" appears in a Greek document (attributed to John the Theologian) edited by Tischendorf published in The Ante-Nicene Fathers, dated by Tischendorf as no later than the 4th century. Then there is the apocryphal literature such as the Protoevangelium of James, regarding the end of the Virgin Mary's life, though it is asserted, without surviving documentation, that the feast of the Dormition was being observed in Jerusalem shortly after the Council of Ephesus.

Gracanica_Monastery_Serbia-Dormition-of-The-Mother-of-God-Virgin_Mary

Dormition of the Mother of God, fresco from Gračanica, c. 1321. (See also:Palaiologian Renaissance)

Before the 4th-5th century Dormition was not celebrated among the Christians as a holy day and earlier it was perhaps not celebrated because the Church did not have time to celebrate too much of special feasts, as there was the era of heretics and gnostics from 1st –  2nd centuries which give birth to multiple heresies and the era of persecution that ended just about the 4th century. Only then the Church posessing freedom in the world could review its belief and clearly define the faith as earlier both Bishops (the apostle successors as the layman) was mostly busy with each one keeping their own faith instead of clearly documenting or giving separate feasts, until the 4th century many of the Church feasts of the martyr saints were still not recognized and known in the "Universal" Church around the world, because travelling was not so common as nowadays and each local established church by apostles strives to keep their faith in Christ and not clear up Church dogmas (this gave raise of course to the VII ecumenical councels who followed from 3rd to 8 centuries.).
 

Epiphanius of Salamis who become bishop of Salamis toda's Cypros (circa 310/20 – 403), a Jew by birth, born in Phoenicia, converted to Christianity in adulthood and lived as a monk for over 20 years in Palestine from 335–340 to 362, writes in "Panarion" (his book on Herisology) in "Contra antidicomarianitas" about the death of the Virgin Mary the following:

 

"If any think am mistaken, moreover, let them search through the scriptures any neither find Mary's death, nor whether or not she died, nor whether or not she was buried—even though John surely travelled throughout Asia. And yet, nowhere does he say that he took the holy Virgin with him. Scripture simply kept silence because of the overwhelming wonder, not to throw men's minds into consternation. For I dare not say—though I have my suspicions, I keep silent. Perhaps, just as her death is not to be found, so I may have found some traces of the holy and blessed Virgin. …The holy virgin may have died and been buried—her falling asleep was with honour, her death in purity, her crown in virginity. Or she may have been put to death—as the scripture says, 'And a sword shall pierce through her soul'—her fame is among the martyrs and her holy body, by which light rose on the world, [rests] amid blessings. Or she may have remained alive, for God is not incapable of doing whatever he wills. No one knows her end. But we must not honour the saints to excess; we must honour their Master. It is time for the error of those who have gone astray to cease."


Saint Ambrosius of Milan ( Mediolan ) in 4th century says:

"Neither the letter of Scripture nor Tradition does not teach us that Mary had left this life as a consequence of suffering from bodily ulcers." pointing to her sufferless sleep like death.

Holy_Mother-of-God-Arbanasi-Monastery-three-handed-Troeruchica-Miracle-Making-icon-Bulgaria-Great_Tarnovo

The famous icon Holy Mother of God Three-handed (Troeruchica) Arbanasi Monastery of the Dormition of Mother of God near medieval capital of Bulgaria Veliko Tarnovo

Our Orthodox Church specifically holds,teaching that Mary died a natural death, like any human being; that her soul was received by Christ upon death; and that her body was resurrect on the third day after her repose, at which time she was taken up, bodily only, into heaven when the apostles, miraculously transported from the ends of the earth, found her tomb to be empty. The specific belief of us Orthodox is expressed in their liturgical texts used at the feast of the Dormition.

Mount-Olives-Marys-burial-tomb-entrane-to-tombThe rock-cut Tomb of Virgin Mary and its entrance, its front side covered in icons; eastern apse of the crypt


The holy body that was taken in Heaven after the Virgin's resurrection 3 days after her Dormition has been originally placed after the burial procession in 41 A.D. in the Church of the Sepulchre of Saint Mary, also known as the Tomb of the Virgin Mary (Hebrew: קבר מרים‎; Greek: Τάφος της Παναγίας) it is located (identified) by historians in the Kidron Valley – at the foot of Mount of Olives, in Jerusalem.

The-stone-bench-on-which-Virgin-Marys-most-pure-body-was-laid-out-Jerusalem-mary-sarcophagusThe stone bench on which the most pure body of the Virgin Mary (Theotokos) has been laid out

 

Let by the Holy Prayers of The Virgin Mary Theotokos, we find grace and God grants mercy upon anyone in torture, in fear in bitterness and sorrows, in weakness and sickness, to all broken-hearted, to all the leppers and mind sick and who lay on the death beds be granted with a painless death similar to hers.

Let the Mother of God keep and protect the Holy Land of Bulgaria and All The Orthodox Lands and the rest of lands, villages, cities and all inhabitants on earth !!! AMEN

 

Linux: Howto Fix “N: Repository ‘http://deb.debian.org/debian buster InRelease’ changed its ‘Version’ value from ‘10.9’ to ‘10.10’” error to resolve apt-get release update issue

Friday, August 13th, 2021

Linux's surprises and disorganization is continuously growing day by day and I start to realize it is becoming mostly impossible to support easily this piece of hackware bundled together.
Usually so far during the last 5 – 7 years, I rarely had any general issues with using:

 apt-get update && apt-get upgrade && apt-get dist-upgrade 

to raise a server's working stable Debian Linux version packages e.g. version X.Y to verzion X.Z (for example up the release from Debian Jessie from 8.1 to 8.2). 

Today I just tried to follow this well known and established procedure that, of course nowdays is better to be done with the newer "apt" command instead with the legacy "apt-get"
And the set of 

 

# apt-get update && apt-get upgrade && apt-get dist-upgrade

 

has triggered below shitty error:
 

root@zabbix:~# apt-get update && apt-get upgrade
Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://security.debian.org buster/updates/non-free Sources [688 B]
Get:4 http://repo.zabbix.com/zabbix/5.0/debian buster InRelease [7096 B]
Get:5 http://security.debian.org buster/updates/main Sources [198 kB]
Get:6 http://security.debian.org buster/updates/main amd64 Packages [300 kB]
Get:7 http://security.debian.org buster/updates/main Translation-en [157 kB]
Get:8 http://security.debian.org buster/updates/non-free amd64 Packages [556 B]
Get:9 http://deb.debian.org/debian buster/main Sources [7836 kB]
Get:10 http://repo.zabbix.com/zabbix/5.0/debian buster/main Sources [1192 B]
Get:11 http://repo.zabbix.com/zabbix/5.0/debian buster/main amd64 Packages [4785 B]
Get:12 http://deb.debian.org/debian buster/non-free Sources [85.7 kB]
Get:13 http://deb.debian.org/debian buster/contrib Sources [42.5 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:15 http://deb.debian.org/debian buster/main Translation-en [5968 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 Contents (deb) [37.3 MB]
Get:17 http://deb.debian.org/debian buster/contrib amd64 Packages [50.1 kB]
Get:18 http://deb.debian.org/debian buster/non-free amd64 Packages [87.7 kB]
Get:19 http://deb.debian.org/debian buster/non-free Translation-en [88.9 kB]
Get:20 http://deb.debian.org/debian buster/non-free amd64 Contents (deb) [861 kB]
Fetched 61.1 MB in 22s (2774 kB/s)
Reading package lists… Done
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'


As I used to realize nowdays, as Linux started originally as 'Hackers' operating system, its legacy is just one big hack and everything from simple maintenance up to the higher and more sophisticated things requires a workaround 'hack''.

 

This time the hack to resolve error:
 

N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'


is up to running cmd:
 

debian-server:~# apt-get update –allow-releaseinfo-change
Поп:1 http://ftp.de.debian.org/debian buster-backports InRelease
Поп:2 http://ftp.debian.org/debian stable InRelease
Поп:3 http://security.debian.org stable/updates InRelease
Изт:5 https://packages.sury.org/php buster InRelease [6837 B]
Изт:6 https://download.docker.com/linux/debian stretch InRelease [44,8 kB]
Изт:7 https://packages.sury.org/php buster/main amd64 Packages [317 kB]
Игн:4 https://attic.owncloud.org/download/repositories/production/Debian_10  InRelease
Изт:8 https://download.owncloud.org/download/repositories/production/Debian_10  Release [964 B]
Изт:9 https://packages.sury.org/php buster/main i386 Packages [314 kB]
Изт:10 https://download.owncloud.org/download/repositories/production/Debian_10  Release.gpg [481 B]
Грш:10 https://download.owncloud.org/download/repositories/production/Debian_10  Release.gpg
  Следните подписи са невалидни: DDA2C105C4B73A6649AD2BBD47AE7F72479BC94B
Грш:11 https://ookla.bintray.com/debian generic InRelease
  403  Forbidden [IP: 52.39.193.126 443]
Четене на списъците с пакети… Готово
N: Repository 'https://packages.sury.org/php buster InRelease' changed its 'Suite' value from '' to 'buster'
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://download.owncloud.org/download/repositories/production/Debian_10  Release: 

apt-get-update-allow-releaseinfo-change-debian-linux-screenshot

Onwards to upgrade the system up to the latest .deb packages, as usual run:

# apt-get -y update && apt-get upgrade -y

 

and updates should be applied as usual with some prompts on whether you prefer to keep or replace existing service configuration and some information on some general changes that might affect your installed services. In a few minutes and few prompts hopefully your Debian OS should be up to the latest stable.