Posts Tagged ‘finding’

How to test RAM Memory for errors in Linux / UNIX OS servers. Find broken memory RAM banks

Friday, December 3rd, 2021



1. Testing the memory with motherboard integrated tools

Memory testing has been integral part of Computers for the last 50 years. In the dawn of computers those older perhaps remember memory testing was part of the computer initialization boot. And this memory testing was delaying the boot with some seconds and the user could see the memory numbers being counted up to the amount of memory. With the increased memory modern computers started to have and the annoyance to wait for a memory check program to check the computer hardware memory on modern computers this check has been mitigated or completely removed on some hardware.
Thus under some circumstances sysadmins or advanced computer users might need to check the memory, especially if there is some suspicion for memory damages or if for example a home PC starts crashing with Blue screens of Death on Windows without reason or simply the PC or some old arcane Linux / UNIX servers gets restarted every now and then for now apparent reason. When such circumstances occur it is an idea to start debugging the hardware issue with a simple memory check.

There are multiple ways to test installed memory banks on a server laptop or local home PC both integrated and using external programs.
On servers that is usually easily done from ILO or IPMI or IDRAC access (usually web) interface of the vendor, on laptops and home usage from BIOS or UEFI (Unified Extensible Firmware Interface) acces interface on system boot that is possible as well.


An old but gold TIP, more younger people might not know is the


Prolonged SHIFT key press which once held with the user instructs the machine to initiate a memory test before the computer starts reading what is written in the boot loader.

So before anything else from below article it might be a good idea to just try HOLD SHIFT for 15-20 seconds after a complete Shut and ON from the POWER button.

If this test does not triggered or it is triggered and you end up with some corrupted memory but you're not sure which exact Memory bank is really crashing and want to know more on what memory Bank and segments are breaking up you might want to do a more thorough testing. In below article I'll try to explain shortly how this can be done.

2. Test the memory using a boot USB Flash Drive / DVD / CD 

Say hello to memtest86+. It is a Linux GRUB boot loader bootable utility that tests physical memory by writing various patterns to it and reading them back. Since memtest86+ runs directly off the hardware it does not require any operating system support for execution. Perhaps it is important to mention that memtest86 (is PassMark memtest86)and memtest86+ (An Advanced Memory diagnostic tool) are different tools, the first is freeware and second one is FOSS software.

To use it all you'll need is some version of Linux. If you don't already have some burned in somewhere at your closet, you might want to burn one.
For Linux / Mac users this is as downloading a Linux distribution ISO file and burning it with

# dd if=/path/to/iso of=/dev/sdbX bs=80M status=progress

Windows users can burn a Live USB with whatever Linux distro or download and burn the latest versionof memtest86+ from  on Windows Desktop with some proggie like lets say UnetBootIn.

2.1. Run memtest86+ on Ubuntu

Many Linux distributions such as Ubuntu 20.0 comes together with memtest86+, which can be easily invoked from GRUB / GRUB2 Kernel boot loader.
Ubuntu has a separate menu pointer for a Memtest.


Other distributions RPM based distributions such as CentOS, Fedora Linux, Redhat things differ.

2.2. memtest86+ on Fedora

Fedora used to have the memtest86+ menu at the GRUB boot selection prompt, but for some reason removed it and in newest Fedora releases as of time such as Fedora 35 memtest86+ is preinstalled and available but not visible, to start on  already and to start a memtest memory test tool:

  •   Boot a Fedora installation or Rescue CD / USB. At the prompt, type "memtest86".

boot: memtest86

2.3 memtest86+ on RHEL Linux

The memtest86+tool is available as an RPM package from Red Hat Network (RHN) as well as a boot option from the Red Hat Enterprise Linux rescue disk.
And nowadays Red Hat Enterprise Linux ships by default with the tool.

Prior redhat (now legacy) releases such as on RHEL 5.0 it has to be installed and configure it with below 3 commands.

[root@rhel ~]# yum install memtest86+
[root@rhel ~]# memtest-setup
[root@rhel ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

    Again as with CentOS to boot memtest86+ from the rescue disk, you will need to boot your system from CD 1 of the Red Hat Enterprise Linux installation media, and type the following at the boot prompt (before the Linux kernel is started):

boot: memtest86

memtest86+ testing 5 memory slots

As you see all on above screenshot the Memory banks are listed as Slots. There are a number of Tests to be completed until
it can be said for sure memory does not have any faulty cells. 

Pass: 0
Errors: 0 

Indicates no errors, so in the end if memtest86 does not find anything this values should stay at zero.
memtest86+ is also usable to detecting issues with temperature of CPU. Just recently I've tested a PC thinking that some memory has defects but it turned out the issue on the Computer was at the CPU's temperature which was topping up at 80 – 82 Celsius.

If you're unfortunate and happen to get some corrupted memory segments you will get some red fields with the memory addresses found to have corrupted on Read / Write test operations:


2.4. Install and use memtest and memtest86+ on Debian / Mint Linux

You can install either memtest86+ or just for the fun put both of them and play around with both of them as they have a .deb package provided out of debian non-free /etc/apt/sources.list repositories.

root@jeremiah:/home/hipo# apt-cache show memtest86 memtest86+
Package: memtest86
Version: 4.3.7-3
Installed-Size: 302
Maintainer: Yann Dirson <>
Architecture: amd64
Depends: debconf (>= 0.5) | debconf-2.0
Recommends: memtest86+
Suggests: hwtools, memtester, kernel-patch-badram, grub2 (>= 1.96+20090523-1) | grub (>= 0.95+cvs20040624), mtools
Description-en: thorough real-mode memory tester
 Memtest86 scans your RAM for errors.
 This tester runs independently of any OS – it is run at computer
 boot-up, so that it can test *all* of your memory.  You may want to
 look at `memtester', which allows testing your memory within Linux,
 but this one won't be able to test your whole RAM.
 It can output a list of bad RAM regions usable by the BadRAM kernel
 patch, so that you can still use you old RAM with one or two bad bits.
 This is the last DFSG-compliant version of this software, upstream
 has opted for a proprietary development model starting with 5.0.  You
 may want to consider using memtest86+, which has been forked from an
 earlier version of memtest86, and provides a different set of
 features.  It is available in the memtest86+ package.
 A convenience script is also provided to make a grub-legacy-based
 floppy or image.

Description-md5: 0ad381a54d59a7d7f012972f613d7759
Section: misc
Priority: optional
Filename: pool/main/m/memtest86/memtest86_4.3.7-3_amd64.deb
Size: 45470
MD5sum: 8dd2a4c52910498d711fbf6b5753bca9
SHA256: 09178eca21f8fd562806ccaa759d0261a2d3bb23190aaebc8cd99071d431aeb6

Package: memtest86+
Version: 5.01-3
Installed-Size: 2391
Maintainer: Yann Dirson <>
Architecture: amd64
Depends: debconf (>= 0.5) | debconf-2.0
Suggests: hwtools, memtester, kernel-patch-badram, memtest86, grub-pc | grub-legacy, mtools
Description-en: thorough real-mode memory tester
 Memtest86+ scans your RAM for errors.
 This tester runs independently of any OS – it is run at computer
 boot-up, so that it can test *all* of your memory.  You may want to
 look at `memtester', which allows to test your memory within Linux,
 but this one won't be able to test your whole RAM.
 It can output a list of bad RAM regions usable by the BadRAM kernel
 patch, so that you can still use your old RAM with one or two bad bits.
 Memtest86+ is based on memtest86 3.0, and adds support for recent
 hardware, as well as a number of general-purpose improvements,
 including many patches to memtest86 available from various sources.
 Both memtest86 and memtest86+ are being worked on in parallel.
Description-md5: aa685f84801773ef97fdaba8eb26436a

Tag: admin::benchmarking, admin::boot, hardware::storage:floppy,
 interface::text-mode, role::program, scope::utility, use::checking
Section: misc
Priority: optional
Filename: pool/main/m/memtest86+/memtest86+_5.01-3_amd64.deb
Size: 75142
MD5sum: 4f06523532ddfca0222ba6c55a80c433
SHA256: ad42816e0b17e882713cc6f699b988e73e580e38876cebe975891f5904828005


root@jeremiah:/home/hipo# apt-get install –yes memtest86+

root@jeremiah:/home/hipo# apt-get install –yes memtest86

Reading package lists… Done
Building dependency tree       
Reading state information… Done
Suggested packages:
  hwtools kernel-patch-badram grub2 | grub
The following NEW packages will be installed:
0 upgraded, 1 newly installed, 0 to remove and 21 not upgraded.
Need to get 45.5 kB of archives.
After this operation, 309 kB of additional disk space will be used.
Get:1 buster/main amd64 memtest86 amd64 4.3.7-3 [45.5 kB]
Fetched 45.5 kB in 0s (181 kB/s)     
Preconfiguring packages …
Selecting previously unselected package memtest86.
(Reading database … 519985 files and directories currently installed.)
Preparing to unpack …/memtest86_4.3.7-3_amd64.deb …
Unpacking memtest86 (4.3.7-3) …
Setting up memtest86 (4.3.7-3) …
Generating grub configuration file …
Found background image: saint-John-of-Rila-grub.jpg
Found linux image: /boot/vmlinuz-4.19.0-18-amd64
Found initrd image: /boot/initrd.img-4.19.0-18-amd64
Found linux image: /boot/vmlinuz-4.19.0-17-amd64
Found initrd image: /boot/initrd.img-4.19.0-17-amd64
Found linux image: /boot/vmlinuz-4.19.0-8-amd64
Found initrd image: /boot/initrd.img-4.19.0-8-amd64
Found linux image: /boot/vmlinuz-4.19.0-6-amd64
Found initrd image: /boot/initrd.img-4.19.0-6-amd64
Found linux image: /boot/vmlinuz-4.19.0-5-amd64
Found initrd image: /boot/initrd.img-4.19.0-5-amd64
Found linux image: /boot/vmlinuz-4.9.0-8-amd64
Found initrd image: /boot/initrd.img-4.9.0-8-amd64
Found memtest86 image: /boot/memtest86.bin
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
File descriptor 3 (pipe:[66049]) leaked on lvs invocation. Parent PID 22581: /bin/sh
Processing triggers for man-db (2.8.5-2) …


After this both memory testers memtest86+ and memtest86 will appear next to the option of booting a different version kernels and the Advanced recovery kernels, that you usually get in the GRUB boot prompt.

2.5. Use memtest embedded tool on any Linux by adding a kernel variable


2.4.1. Reboot your computer

# reboot

2.4.2. At the GRUB boot screen (with UEFI, press Esc).

2.4.3 For 4 passes add temporarily the memtest=4 kernel parameter.

memtest=        [KNL,X86,ARM,PPC,RISCV] Enable memtest
                Format: <integer>
                default : 0 <disable>
                Specifies the number of memtest passes to be
                performed. Each pass selects another test
                pattern from a given set of patterns. Memtest
                fills the memory with this pattern, validates
                memory contents and reserves bad memory
                regions that are detected.

3. Install and use memtester Linux tool

At some condition, memory is the one of the suspcious part, or you just want have a quick test. memtester  is an effective userspace tester for stress-testing the memory subsystem.  It is very effective at finding intermittent and non-deterministic faults.

The advantage of memtester "live system check tool is", you can check your system for errors while it's still running. No need for a restart, just run that application, the downside is that some segments of memory cannot be thoroughfully tested as you already have much preloaded data in it to have the Operating Sytstem running, thus always when possible try to stick to rule to test the memory using memtest86+  from OS Boot Loader, after a clean Machine restart in order to clean up whole memory heap.

Anyhow for a general memory test on a Critical Legacy Server  (if you lets say don't have access to Remote Console Board, or don't trust the ILO / IPMI Hardware reported integrity statistics), running memtester from already booted is still a good idea.

3.1. Install memtester on any Linux distribution from source

# tar zxvf memtester-4.2.2.tar.gz
# cd memtester-4.2.2
# make && make install

3.2 Install on RPM based distros


On Fedora memtester is available from repositories however on many other RPM based distros it is not so you have to install it from source.

[root@fedora ]# yum install -y memtester


3.3. Install memtester on Deb based Linux distributions from source

To install it on Debian / Ubuntu / Mint etc. , open a terminal and type:

root@linux:/ #  apt install –yes memtester

The general run syntax is:

memtester [-p PHYSADDR] [ITERATIONS]

You can hence use it like so:

hipo@linux:/ $ sudo memtester 1024 5

This should allocate 1024MB of memory, and repeat the test 5 times. The more repeats you run the better, but as a memtester run places a great overall load on the system you either don't increment the runs too much or at least run it with  lowered process importance e.g. by nicing the PID:

hipo@linux:/ $ nice -n 15 sudo memtester 1024 5


  • If you have more RAM like 4GB or 8GB, it is upto you how much memory you want to allocate for testing.
  • As your operating system, current running process might take some amount of RAM, Please check available Free RAM and assign that too memtester.
  • If you are using a 32 Bit System, you cant test more than 4 GB even though you have more RAM( 32 bit systems doesnt support more than 3.5 GB RAM as you all know).
  • If your system is very busy and you still assigned higher than available amount of RAM, then the test might get your system into a deadlock, leads to system to halt, be aware of this.
  • Run the memtester as root user, so that memtester process can malloc the memory, once its gets hold on that memory it will try to apply lock. if specified memory is not available, it will try to reduce required RAM automatically and try to lock it with mlock.
  • if you run it as a regular user, it cant auto reduce the required amount of RAM, so it cant lock it, so it tries to get hold on that specified memory and starts exhausting all system resources.

If you have 8 Gigas of RAM plugged into the PC motherboard you have to multiple 1024*8 this is easily done with bc (An arbitrary precision calculator language) tool:

root@linux:/ # bc -l
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 

 for example you should run:

root@linux:/ # memtester 8192 5

memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 8192MB (2083520512 bytes)
got  8192MB (2083520512 bytes), trying mlock …Loop 1/1:
  Stuck Address       : ok        
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok        
  Block Sequential    : ok        
  Checkerboard        : ok        
  Bit Spread          : ok        
  Bit Flip            : ok        
  Walking Ones        : ok        
  Walking Zeroes      : ok        
  8-bit Writes        : ok
  16-bit Writes       : ok



4. Shell Script to test server memory for corruptions

If for some reason the machine you want to run a memory test doesn't have connection to the external network such as the internet and therefore you cannot configure a package repository server and install memtester, the other approach is to use a simple memory test script such as

# Downloaded from
echo "ByteOnSite Memory Test"
cpus=`cat /proc/cpuinfo | grep processor | wc -l`
if [ $cpus -lt 6 ]; then
threads=$(($cpus / 2))
echo "Detected $cpus CPUs, using $threads threads.."
memory=`free | grep 'Mem:' | awk {'print $2'}`
memoryper=$(($memory / $threads))
echo "Detected ${memory}K of RAM ($memoryper per thread).."
freespace=`df -B1024 . | tail -n1 | awk {'print $4'}`
if [ $freespace -le $memory ]; then
echo You do not have enough free space on the current partition. Minimum: $memory bytes
exit 1
echo "Clearing RAM Cache.."
sync; echo 3 > /proc/sys/vm/drop_cachesfile
echo > dump.memtest.img
echo "Writing to dump file (dump.memtest.img).."
for i in `seq 1 $threads`;
# 1044 is used in place of 1024 to ensure full RAM usage (2% over allocation)
dd if=/dev/urandom bs=$memoryper count=1044 >> dump.memtest.img 2>/dev/null &
echo $i
for pid in "${pids[@]}"
wait $pid

echo "Reading and analyzing dump file…"
echo "Pass 1.."
md51=`md5sum dump.memtest.img | awk {'print $1'}`
echo "Pass 2.."
md52=`md5sum dump.memtest.img | awk {'print $1'}`
echo "Pass 3.."
md53=`md5sum dump.memtest.img | awk {'print $1'}`
if [ “$md51” != “$md52” ]; then
elif [ “$md51” != “$md53” ]; then
elif [ “$md52” != “$md53” ]; then
if [ $fail -eq 0 ]; then
echo "Memory test PASSED."
echo "Memory test FAILED. Bad memory detected."
rm -f dump.memtest.img
exit $fail

Nota Bene !: Again consider the restults might not always be 100% trustable if possible restart the server and test with memtest86+

Consider also its important to make sure prior to script run,  you''ll have enough disk space to produce the dump.memtest.img file – file is created as a test bed for the memory tests and if not scaled properly you might end up with a full ( / ) root directory!


4.1 Other memory test script with dd and md5sum checksum

I found this solution on the well known sysadmin site nixCraft, I think it makes sense and quicker.

First find out memory site using free command.

# free
             total       used       free     shared    buffers     cached
Mem:      32867436   32574160     293276          0      16652   31194340
-/+ buffers/cache:    1363168   31504268
Swap:            0          0          0

It shows that this server has 32GB memory,

# dd if=/dev/urandom bs=32867436 count=1050 of=/home/memtest

free reports by k and use 1050 is to make sure file memtest is bigger than physical memory.  To get better performance, use proper bs size, for example 2048 or 4096, depends on your local disk i/o,  the rule is to make bs * count > 32 GB.

# md5sum /home/memtest; md5sum /home/memtest; md5sum /home/memtest

If you see md5sum mismatch in different run, you have faulty memory guaranteed.
The theory is simple, the file /home/memtest will cache data in memory by filling up all available memory during read operation. Using md5sum command you are reading same data from memory.

5. Other ways to test memory / do a machine stress test

Other good tools you might want to check for memory testing is mprime – 

  •  (mprime can also be used to stress test your CPU)

Alternatively, use the package stress-ng to run all kind of stress tests (including memory test) on your machine.
Perhaps there are other interesting tools for a diagnosis of memory if you know other ones I miss, let me know in the comment section.

26 October the Feast of Holy Great-Martyr Demetrius the Myroblyte known also as Demetrius of Thessaloniki

Tuesday, October 26th, 2021

Sveti Dimitar Solunski_kopie-ikona

Bulgarian icon of Saint Demetrius

Every 26 of October in the Bulgarian Orthodox Church and whole Bulgarian nation we honor deeply the memory of martyrdom of Saint Demetrius the Myroblyte (meaning 'the Myrrh-Gusher' or 'Myrrh-Streamer'; a term that stemmed from 3rd century – 306 y. the year of Maryrdom of this great saint. Saint Demetrius over the century has been one of the most venerated saints in the Eastern Orthodox Church and since the Christianization of Bulgaria his glory also spread quickly throughout the Bulgarian Empire lands.


During the Middle Ages, he came to be revered as one of the most important Orthodox military saints, often paired with Saint George of Lydda and for that in many of the Orthodox Churches worldwide there are icons of the two saints painted together holding their warrior equipment spear, shield and sward .


St. Demetrius (Dimitar in Bulgarian) feast day is 26 October for Eastern Orthodox Christians, which falls on 8 November for those following the old calendar. In the Roman Catholic church he is most commonly called "Demetrius of Sermium" and his memorial falls on 8 October, which seem to coincide with my Birthday 🙂

Demetrius was born to pious Christian parents in Thessaloniki, the Eastern Roman Empire region Macedonia in 270 (Macedonia has been part of the Bulgarian kingdom and Empire for many centuries).

According to the hagiographies, Demetrius was a young man of senatorial family who became proconsul of the Thessalonica district. He was run through with spears in around 306 AD in Thessaloniki, during the Christian persecutions of Galerian, which matches his depiction in the 7th century mosaics.

Most historical scholars follow the hypothesis put forward by Bollandist Hippolyte Delehaye (1859–1941), that his veneration was transferred from Sirmium when Thessaloniki replaced it as the main military base in the area in 441/442 AD. His very large church in Thessaloniki, the Hagios Demetrios, dates from the mid-5th century. Thessaloniki remained a centre of his veneration, and he is the patron saint of the city.

After the growth of his veneration as saint, the city of Thessaloniki suffered repeated attacks and sieges from the Slavic peoples who moved into the Balkans, and Demetrius was credited with many miraculous interventions to defend the city. Hence later traditions about Demetrius regard him as a soldier in the Roman army, and he came to be regarded as an important military martyr. Unsurprisingly, he was extremely popular in the Middle Ages. Disputes between Bohemond I of Antioch and Alexios I Komnenos appear to have resulted in Demetrius being appropriated as patron saint of crusading.


Saint Demetrius Russian Icon

Demetrius was also venerated as patron of agriculture, peasants and shepherds in the Greek countryside during the Middle Ages. 

Most scholars still believe that for four centuries after his death, Demetrius had no physical relics, and in their place an unusual empty shrine called the "ciborium" was built inside Hagios Demetrios. What were purported to be his remains subsequently appeared in Thessaloniki, but the local archbishop John, who compiled the first book of the Miracles ca. 610, was publicly dismissive of their authenticity. The relics were assumed to be genuine after they started emitting a liquid and strong-scented myrrh. This gave Demeterius the epithet Myroblyte.

Saint Demetrius used to be a mayor of Thessaloniki and had been very educated for his time, the Roman empire ordered him to find and imprison, torture and eventually kill all Christians in the city who refuse to follow the paganic Roman religion. Being a brave in heart and a being a Christian himself, he refused to follow the unrighteous emperor decree and even on the contrary started to put special efforts for the raising of the Christian faith in the city. 

Despite this position in the still-pagan empire, he remained fervent in faith and works for Christ, encouraging many Christians to endure persecution and even bringing many pagans to the faith.

When Maximian returned from one of his campaigns to Thessaloniki, which he had made his capital, he had pagan games and sacrifices celebrated for his triumph. Demetrios was denounced by pagans who were envious of his success, and he was thrown into prison. While in prison he was visited by a young Christian named Nestor, who asked him for a blessing to engage in single combat with the giant Lyaios (or Lyaeus), who was posing as the champion of paganism. Demetrios gave his blessing and Nestor, against all odds. Nestor succeded to slew his opponent in the arena contrary to any expectations as Lyaios used to kill many, many christians on the circus arena, as David had once defeated Goliath. Saint Demetrius blesses Nestor but warned him he will have to endure a martyrdom after his defeat of Nestor which occured shortly after the defeath of Lyaios, Nestor was captured and martyred for Christ. Being raged out by the killing of Lyaios, the Romans send trooops and killed with spears saint Demetrius while he was praying in the prison.

According to some (Greek) hagiographic legend, as retold by Dimitry of Rostov in particular, Demetrius appeared in 1207 in the camp of tsar Kaloyan of with a lance and so killing him. This scene, known as Чудо о погибели царя Калояна ("the miracle of the destruction of tsar Kaloyan") became a popular element in the iconography of Demetrius. He is shown on horseback piercing the king with his spear, paralleling the iconography (and often shown alongside) of Saint George and the Dragon.

The reason of High veneration of Saint Demetrius in Bulgaria today ?

The godly life he led, together with his military virtues and martyrdom, led the people of Thessaloniki to declare him their saint-warrior and patron. According to the beliefs of the local centuries, the saint defended Thessaloniki, performing miracle after miracle, but in August 1185 something unheard of happened. The second richest and most important city in the empire after Constantinople was captured by the Normans and subjected to unprecedented looting. The Church of St. Dimitar was burned and the relics of the saint were scattered. The medieval Greek, who was inclined to seek God's intervention everywhere, was spiritually broken. The Romans saw the fall of Dimitrov as a punishment for their sinfulness. It is clear to them that St. Dimitar left them.

Meanwhile, in the north, the memories of the old Bulgarian kingdom were more than alive, and it became increasingly difficult for the Bulgarians to tolerate the Roman rule. The moment for a mass uprising was ripe. According to Nikita Honiat, there were three key events at the beginning of the uprising. The first concerned the desire of the brothers Peter and Assen (prominent Bulgarian boyars) to be included in the proniat lists of the empire and to receive a small landed estate at the foot of the Balkan Mountains. To this end, most likely in the autumn of 1185, they appeared in person before Emperor Isaac II Angel in Kipsela, just as he was preparing to march against the Normans who had conquered Thessaloniki. The refusal to comply with their demands provoked sharp resentment in the younger brother Assen, who personally threatened the emperor with rebellion. This unheard of behavior of the young boyar was punished with a slap.


King Ivan Assen I (Tsar of Bulgaria 1187/1188–1196)

The second important event was the imposition of additional taxes on the livestock of the population on the occasion of the emperor's wedding to the Hungarian Princess Margaret. This led to the outbreak of strong and mass discontent among the population of Moesia. The two brothers knew very well what they were doing and used the mass discontent to make their threat a reality. However, the insults, material hardship and the presence of two brilliant leaders in the face of Assenevtsi were not enough for a revolt.


Bulgarian Medieval Icon of Saint Demetrius the Myrrh-Bringer

The Bulgarians also had to receive a "divine" guarantee for their work. They believed that the Lord should show them that they were chosen and worthy of their freedom, that they not only could, but should take up arms against the Byzantine Vasilevs. And the sign was not late. On October 26, 1185, Assenevtsi, together with a large crowd, gathered in Tarnovo to consecrate the newly built church "St. Dimitar. Meanwhile, a miraculous icon of the saint appeared in the city. It was alleged that she had left Thessaloniki, conquered by the Normans, and found her home in the new temple of the Bulgarians.

The religious consciousness of the medieval Bulgarian interpreted this as a refusal of St. Dimitar to defend the Romans and a sacred guarantee that the saint will protect the Bulgarians in their cause for freedom. And indeed the old church in Thessaloniki had collapsed and plundered, the Romans were punished, the Empire was humiliated.


Those gathered in the church began to shout and call for the rejection of the yoke and for the restoration of the glory of the old kings. In this atmosphere of patriotic enthusiasm, the older brother, Todor (named Peter), placed a golden tiara on his head, put on a red cloak, and put on the purple shoes that only the Byzantine Vasilevs could wear. Thus, after 167 years of interruption of the throne of the Bulgarian kings, a Bulgarian ascended again. The coronation of Peter as king and the beginning of the great uprising of the Bulgarians was one of those moments in history when all accounts end and only faith gives the people the courage to take the hand outstretched by the uncertainty of the future and follow the path indicated by her, not knowing where he was taking her.

Niketa Choniates writes: “With such (divine) prophecies the whole nation was won for the cause and all raised their swords. And because their rebellion was successful from the very beginning, the Bulgarians believed even more that God had approved their freedom. "

At first, Isaac II Angel was unable to respond to the uprising, as he had to deal with the Normans and the usurper of Cyprus, Isaac I Komnenos. It was not until December 1185 that Vasilevs sent his uncle Sevastocrator John against the rebels. However, no battle took place because the Sevastocrator was recalled on suspicion of rebellion. At the head of the second army was Caesar John VI Kantakouzenos, who went to Hemus, but was defeated in a night attack by Assen-evtsi. The Bulgarians took the lives of most of the Roman army, and its commander managed to escape by abandoning the entire convoy. A third army of the great Byzantine general Alexy Branas was also sent, but it turned against the emperor and marched to Constantinople instead of Tarnovo.

The Byzantine themes (or districts) of Bulgaria and Paristrion 

Paristrion – (Greek: Παρίστριον, lit. 'beside the Ister'), or Paradounabon/Paradounabis (αραδούναβον / Παραδούναβις), which is preferred in official documents, was a Byzantine province covering the southern bank of the Lower Danube (Moesia Inferior) in the 11th and 12th centuries.

It was not until 1186 that the emperor personally led a large army and decided to deal with the Bulgarians once and for all. His campaign forced the brothers to retreat across the Danube to their Kuman allies, and Isaac II Angel plundered Moesia and returned to Constantinople. According to the story of Nikita Honiat, the emperor was so arrogant of his success that he met with ridicule the reminder of Vasily II the Bulgarian assassin that the Bulgarians would revolt and that one day they would be liberated.

At that time, Assen's personality became more and more prominent, and he became the real leader of the rebellion. In the summer of 1186, the Assenevs crossed the Danube again, conquered the plain and set their goal to bring the endeavor to a successful conclusion. Niketa Choniates says:

"And then they returned to their homeland Moesia; finding the land abandoned by the Roman armies, they took on even greater confidence, leading their Cuman auxiliary detachments as if they were legions of demons. They did not simply want to secure their possessions and establish control over Moesia; They wanted to devastate the Roman territories and unite the political power of Moesia and Bulgaria in one empire as before. "

Isaac Angel's second campaign was not long in coming. In the autumn of 1186 he set out again against the two brothers, passing through the fortress of Beroe and heading for Serdica (today's Sofia), from where he intended to cross Hemus and attack Tarnovo. The winter of 1186, however, blocked the passages and forced the emperor to abandon his endeavor for another year. With the arrival of the spring of 1187, the Romans crossed the mountain and besieged the Lovech fortress. However, the Bulgarian troops offered unprecedented resistance and after a three-month siege Isaac II Angel had to ask for peace.


The Church Saint Demetrius built by King Asen I in memoriam of great Miracles of Bulgarians victories over Byzantines
Church is located near the Tarnovo Fortress of Trapezica

The Church slavonic written sources tells how the brothers spread the word a patron saint of Thessaloniki – St. Demetrius, came to Tarnovo to help the Bulgarian people to be liberated…

Thus, most probably, the Lovech armistice was signed in front of the city walls, which de jure recognized the Bulgarian power north of the Balkans. The long road to freedom began on that distant St. Dimitrov's Day in 1185. he was finally walked away. St. Dimitar became the patron of the Asenevtsi dynasty and one of the most beloved Bulgarian saints, and the Bulgarians proved to the world that their pursuit of freedom is nothing but a great national feat, in which with a true understanding of the necessary and possible, with steady faith and unwavering energy in the design and implementation, the political and spiritual resurrection of the Second Bulgarian Kingdom was reached.


Saint Demetrius Bulgarian icon year 1824

St. Demetrius is depicted on horseback spearing a man, not because he ever a killed a man but because he blessed Nestor to win over the Gladiator Lyaeus. The Church decided to commemory the memory and bravery of Saint Nestor who also confessed Christ in his martyrdom every on the next day after the memory of st. Demetrius is celebrated. Saint Nestor even today is celebrated in the Church calendar on 27-th of October.

In Bulgaria the veneration of saint Demetrius was of high esteem especially in the Second Bulgarian Empire and many churches and monasteries has been built around the country (counting at few hundred temples and monasteries) with him being their patron.


Saint Demetrius Holy Relics in the St. Demetrius Church in Thessaloniki Greece (the white papers are names of people who ask for help from the saint)

Saint Demetrius is famous in Thessaloniki and highly venerated every year during his feast as he has been summoned by the Church to protect the city on multiple occasions which he did so far during pandemics such as the Black Death and during invasion of alien (non-Christian) nations.

It is mostly remarkable that every year during his feast day, a great miracle happens from the exact place where he was martyred (situated in the Church named after him), a myrrh with heavenly odor is streaming which is taken by believers for oilment and as a blessing carefully kept until the next year feast of the saint.
Because of the high amount of myrrh outflow a special pool was kept to keep the oilment sparring out of his holy relics.

As Saint Demetrius has helped multiple times to many of their saints as we know from history, especially in times of epidemies and pandemies like it is now let by his holy prayers those who venerate him and the people worldwide finds Healing and relief and an Enlightment and blessing from the light of Christ, just like Nestor found in his blessing !

Holy Martyr Demetrius of Thessaloniki pray the Lord for us the sinners !!!

Getting Console and Graphical hardware system information on Linux with cpuinfo, neofetch, CPU-X (CPU-Z Unix alternative), I-nex and inxi

Tuesday, September 17th, 2019


Earlier I've wrote extensive article on how to get hardware information on Linux using tools such as dmidecode, hardinfo, lshw, hwinfo, x86info and biosdecode but there are few other hardware reporting tools for Linux worthy to mention that has been there for historical reasons such as cpuinfo as we as some new shiny ones such as neofetch (a terminal / console hardware report tool as well the CPU-X and I-Nex  which is Linux equivalent to the all known almost standard for Windows hardware detection CPU-Z worthy to say few words about.

1. cpuinfo


Perhaps the most basic tool to give you a brief information about your Processor type (model) number of Cores and Logical Processors is cpuinfo

I remember cpuinfo has been there since the very beginning on almost all Linux distributions's repository, nowadays its popularity of the days when the kings on the Linux OS server scenes were Slackware, Caldera OpenLinux and Redhat 6.0 Linux and Debian 3.0  declined but still for scripting purposes it is handy small proggie.

To install and run it in Debian  / Ubuntu / Mint Linux etc.:


aptitude install -y cpuinfo





2. neofetch


The next one worthy to install and check is neofetch (a cross-platform and easy-to-use system information
 command line script that collects your Linux system information and display it on the terminal next to an image, it could be your distributions logo or any ascii art of your choice.)

The cool thing about neofetch is besides being able to identify the System server / desktop hardware parameters, it gives some basic info about number of packages installed on the system, memory free and in use, used kernel and exact type of System (be it Dell PowerEdge Model XX, IBM eSeries Model / HP Proliant Model etc.


neofetch info generated on my home used Lenovo Thikpad T420

neofetch info from running current machine

neofetch even supports Mac OS X and Windows OS ! 🙂

To install neofetch on Mac OS X:

/usr/bin/ruby -e "$(curl -fsSL"

or via Mac ported packages using brew

brew install neofetch


neofetch is even installable on Windows OS that has the scoop command line installer tool installer manager with below PowerShell code in cmd.exe (Command line):

powershell Set-ExecutionPolicy RemoteSigned -scope CurrentUser
iex (new-object net.webclient).downloadstring('')
scoop install git
scoop install neofetch


By the way Scoop was quite a finding for me and it is pretty handy to install plenty of useful command line Linux / UNIX tools, such as curl, wget, git etc. in the same easy straight forward way as a standard yum or apt-get on Windows (without explicitly installing things as GnuWin and CygWin).

3. CPU-X graphical user interface hardware report Linux GUI alternative to Windows CPU-Z

The packages for CPU-X are a bit outdated and even though there are rpm packages for Fedora, OpenSuSE and .deb package for Debian for Debian, Ubuntu and ArchLinux (pacman), there is no up to date version for Debian 10 and the package builds distributed for different Linux distros are a bit outdated.

Thus to install CPU-X on any Linux distribution it is perhaps best to use the portable version (static binary) of CPU-X.
It is currently available on

To install latest portable version of CPU-X


mkdir CPU-X
cd CPU-X

tar -zxvvf CPU-X_v3.2.4_portable.tar.gz
-rwxr-xr-x yohan/users 4563032 2019-01-13 22:15 CPU-X_v3.2.4_portable.bsd64
-rwxr-xr-x yohan/users 5484968 2019-01-13 22:15 CPU-X_v3.2.4_portable.linux64


cp -rpf CPU-X_v3.2.4_portable.linux64 /usr/local/bin/
ln -sf /usr/local/bin/CPU-X_v3.2.4_portable.linux64 /usr/local/bin/cpu-x

Next run as superuser (root)

hipo@jeremiah:~$ su -c 'cpu-x'


As seen from below screenshots cpu-x reports a lot of concrete specific hardware data on:

  • Processor
  • Motherboard
  • Memory
  • System
  • Graphic card
  • Performance







CPU-X can be installed also on FreeBSD very easily by just installing from BSD port tree sysutils/cpu-x/
It is also said to work on other *BSDs, NetBSD, OpenBSD Unixes but I guess this will require a manual compilation based on FreeBSD's port Makefile.

4. I-Nex another GUI alternative to CPU-Z for UNIX / Linux

I-Nex is even more useful for general hardware reporting as it reports many hardware specifications not reported by CPU-X such as Battery type and Model Name  (if the hardware report is on a laptop), info on USB devices slots or plugged USB devices brand and specifications, the available Network devices on the system (MAC Addresses) of each of it, Installed and used drivers on Hard Disk (ATA / SATA / SCSI / SSD), HW Sector size, Logical Block size, HDD Sectors count and other specific Hard Drive data as well as information on available Audio (Sound Blaster) devices (HDA-Intel), used Codecs, loaded kernel ALSA driver, Video card used and most importantly indicators on Processor reported CPU (temperature).


To install I-nex

Go to or any of the mirror links where it resides and install the respective package, in my case, I was doing the installation on Debian Linux, so fetched current latest amd64 package which as of moment of writting this article is i-nex_7.6.0-0-bzr977-20161012-ubuntu16.10.1_amd64.deb , next installed it with dpkg

dpkg -i i-nex_7.6.0-0-bzr977-20161012-ubuntu16.10.1_amd64.deb


As the package was depending on some other .deb packages, which failed to install to install the missing ones I had to further run

apt –fix-broken install




I-Nex thermal indicators about CPU temperature on a Linux Desktop notebook








There are other Hardware identification report tools such as CUDA-Z that are useful to check if you have Nvidia Video Card hardware Installed on the PC to check the status of CUDA enabled GPUs, useful if working with nVidia Geforce, Quadro, Tesla cards and ION chipsets.

If you use it however be aware that CUDA-Z is not compatible with 3rd-party linux drivers for NVidia so make sure you have the current official Nvidia version.


5. Inxi full featured system information script


Inxi is a 10000 lines mega bash script that fetches hardware details from multiple different sources in /proc /sys and from commands on the system, and generates a beautiful looking console report that non technical users can read easily.



inxi -Fx




Each of the pointed above tools has different method of collection of Hardware information from various resources e.g. – kernel loaded modules, dmesg, files like /proc/meminfo /proc/version /proc/scsi/scsi /proc/partitions.
Hence some of the tools are likely to report more info than otheres, so in case if some information you need regarding the system plugged in hardware is missing you can perhaps obtain it from another program. Most Linux distribution desktop provided GNOME package are including Hardinfo gui tool, but in many cases above mentioned tools are likely to add even more on info on what is inside your PC Box.
If you're aware of others tools that are useful not mentioned here please share it.

What is inode and how to find out which directory is eating up all your filesystem inodes on Linux, Increase inode count on a ext3 ext4 and ufs filesystems

Tuesday, August 20th, 2019


If you're a system administrator of multiple Linux servers used for Web serving delivery / Mail server sysadmin, Database admin or any High amount of Drives Data Storage used for backup servers infra, Data Repository administrator such as Linux hosted Samba / CIFS shares, etc. or using some Linux Hosting Provider to host your website or any other UNIX like Infrastructure servers that demands a storage of high number of files under a Directory  you might end up with the common filesystem inode depletion issues ( Maximum Inode number for a filesystem is predefined, limited and depending on the filesystem configured size).

In case a directory stored files end up exceding the amount of possible addressable inodes could prevent any data to be further assiged and stored on the Filesystem.

When a device runs out of inodes, new files cannot be created on the device, even though there may be plenty free space available and the first time it happened to me very long time ago I was completely puzzled how this is possible as I was not aware of Inodes existence  …

Reaching maximum inodes number (e.g. inode depletion), often happens on Busy Mail servers (receivng tons of SPAM email messages) or Content Delivery Network (CDN – Website Image caching servers) which contain many small files on EXT3 or EXT4 Journalled filesystems. File systems (such as Btrfs, JFS or XFS) escape this limitation with extents or dynamic inode allocation, which can 'grow' the file system or increase the number of inodes.


Hence ending being out of inodes could cause various oddities on how stored data behaves or communicated to other connected microservices and could lead to random application disruptions and odd results costing you many hours of various debugging to find the root cause of inodes (index nodes) being out of order.

In below article, I will try to give an overall explanation on what is an I-Node on a filesystem, how inodes of FS unit could be seen, how to diagnose a possible inode poblem – e.g.  see the maximum amount of inodes available per filesystem and how to prepare (format) a new filesystem with incrsed set of maximum inodes.


What are filesystem i-nodes?


This is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory.
The data structure described in the inodes might vary slightly depending on the filesystem but usually on EXT3 / EXT4 Linux filesystems each inode stores the index to block that contains attributes and disk block location(s) of the object's data.
– Yes for those who are not aware on how a filesystem is structured on *nix it does allocate all stored data in logical separeted structures called data blocks. Each file stored on a local filesystem has a file descriptor, there are virtual unit structures file tables and each of the inodes that are a reference number has a own data structure (inode table).

Inodes / "Index" are slightly unusual on file system structure that stored the access information of files as a flat array on the disk, with all the hierarchical directory information living aside from this as explained by Unix creator and pioneer- Dennis Ritchie (passed away few years ago).


Simplified explanation on file descriptors, file table and inode, table on a common Linux filesystem

Here is another description on what is I-node, given by Ken Thompson (another Unix pioneer and father of Unix) and Denis Ritchie, described in their paper published in 1978:

"    As mentioned in Section 3.2 above, a directory entry contains only a name for the associated file and a pointer to the file itself. This pointer is an integer called the i-number (for index number) of the file. When the file is accessed, its i-number is used as an index into a system table (the i-list) stored in a known part of the device on which the directory resides. The entry found thereby (the file's i-node) contains the description of the file:…
    — The UNIX Time-Sharing System, The Bell System Technical Journal, 1978  "


What is typical content of inode and how I-nodes play with rest of Filesystem units?

The inode is just a reference index to a data block (unit) that contains File-system object attributes. It may include metadata information such as (times of last change, access, modification), as well as owner and permission data.


On a Linux / Unix filesystem, directories are lists of names assigned to inodes. A directory contains an entry for itself, its parent, and each of its children.



Structure of inode table-on Linux Filesystem diagram (picture source

  • Information about files(data) are sometimes called metadata. So you can even say it in another way, "An inode is metadata of the data."
  •  Inode : Its a complex data-structure that contains all the necessary information to specify a file. It includes the memory layout of the file on disk, file permissions, access time, number of different links to the file etc.
  •  Global File table : It contains information that is global to the kernel e.g. the byte offset in the file where the user's next read/write will start and the access rights allowed to the opening process.
  • Process file descriptor table : maintained by the kernel, that in turn indexes into a system-wide table of files opened by all processes, called the file table .

The inode number indexes a table of inodes in a known location on the device. From the inode number, the kernel's file system driver can access the inode contents, including the location of the file – thus allowing access to the file.

  •     Inodes do not contain its hardlink names, only other file metadata.
  •     Unix directories are lists of association structures, each of which contains one filename and one inode number.
  •     The file system driver must search a directory looking for a particular filename and then convert the filename to the correct corresponding inode number.

The operating system kernel's in-memory representation of this data is called struct inode in Linux. Systems derived from BSD use the term vnode, with the v of vnode referring to the kernel's virtual file system layer.

But enough technical specifics, lets get into some practical experience on managing Filesystem inodes.

Listing inodes on a Fileystem

Lets say we wan to to list an inode number reference ID for the Linux kernel (files):


root@linux: # ls -i /boot/vmlinuz-*
 3055760 /boot/vmlinuz-3.2.0-4-amd64   26091901 /boot/vmlinuz-4.9.0-7-amd64
 3055719 /boot/vmlinuz-4.19.0-5-amd64  26095807 /boot/vmlinuz-4.9.0-8-amd64

To list an inode of all files in the kernel specific boot directory /boot:


root@linux: # ls -id /boot/
26091521 /boot/

Listing inodes for all files stored in a directory is also done by adding the -i ls command flag:

Note the the '-1' flag was added to to show files in 1 column without info for ownership permissions


root@linux:/# ls -1i /boot/
26091782 config-3.2.0-4-amd64
 3055716 config-4.19.0-5-amd64
26091900 config-4.9.0-7-amd64
26095806 config-4.9.0-8-amd64
26091525 grub/
 3055848 initrd.img-3.2.0-4-amd64
 3055644 initrd.img-4.19.0-5-amd64
26091902 initrd.img-4.9.0-7-amd64
 3055657 initrd.img-4.9.0-8-amd64
 3055760 vmlinuz-3.2.0-4-amd64
 3055719 vmlinuz-4.19.0-5-amd64
26091901 vmlinuz-4.9.0-7-amd64
26095807 vmlinuz-4.9.0-8-amd64


To get more information about Linux directory, file, such as blocks used by file-unit, Last Access, Modify and Change times, current External Symbolic or Static links for filesystem object:

root@linux:/ # stat /etc/
  File: /etc/
  Size: 16384         Blocks: 32         IO Block: 4096   catalog
Device: 801h/2049d    Inode: 6365185     Links: 231
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-08-20 06:29:39.946498435 +0300
Modify: 2019-08-14 13:53:51.382564330 +0300
Change: 2019-08-14 13:53:51.382564330 +0300
 Birth: –


Within a POSIX system (Linux-es) and *BSD are more or less such, a file has the following attributes[9] which may be retrieved by the stat system call:

   – Device ID (this identifies the device containing the file; that is, the scope of uniqueness of the serial number).
    File serial numbers.
    – The file mode which determines the file type and how the file's owner, its group, and others can access the file.
    – A link count telling how many hard links point to the inode.
    – The User ID of the file's owner.
    – The Group ID of the file.
    – The device ID of the file if it is a device file.
    – The size of the file in bytes.
    – Timestamps telling when the inode itself was last modified (ctime, inode change time), the file content last modified (mtime, modification time), and last accessed (atime, access time).
    – The preferred I/O block size.
    – The number of blocks allocated to this file.


Getting more extensive information on a mounted filesystem

Most Linuxes have the tune2fs installed by default (in debian Linux this is through e2fsprogs) package, with it one can get a very good indepth information on a mounted filesystem, lets say about the ( / ) root FS.

root@linux:~# tune2fs -l /dev/sda1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          abe6f5b9-42cb-48b6-ae0a-5dda350bc322
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              30162944
Block count:              120648960
Reserved block count:     6032448
Free blocks:              13830683
Free inodes:              26575654
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      995
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Thu Sep  6 21:44:22 2012
Last mount time:          Sat Jul 20 11:33:38 2019
Last write time:          Sat Jul 20 11:33:28 2019
Mount count:              6
Maximum mount count:      22
Last checked:             Fri May 10 18:32:27 2019
Check interval:           15552000 (6 months)
Next check after:         Wed Nov  6 17:32:27 2019
Lifetime writes:          338 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       21554129
Default directory hash:   half_md4
Directory Hash Seed:      d54c5a90-bc2d-4e22-8889-568d3fd8d54f
Journal backup:           inode blocks

Important note to make here is file's inode number stays the same when it is moved to another directory on the same device, or when the disk is defragmented which may change its physical location. This also implies that completely conforming inode behavior is impossible to implement with many non-Unix file systems, such as FAT and its descendants, which don't have a way of storing this invariance when both a file's directory entry and its data are moved around. Also one inode could point to a file and a copy of the file or even a file and a symlink could point to the same inode, below is example:

$ ls -l -i /usr/bin/perl*
266327 -rwxr-xr-x 2 root root 10376 Mar 18  2013 /usr/bin/perl
266327 -rwxr-xr-x 2 root root 10376 Mar 18  2013 /usr/bin/perl5.14.2

A good to know is inodes are always unique values, so you can't have the same inode number duplicated. If a directory is damaged, only the names of the things are lost and the inodes become the so called “orphan”, e.g.  inodes without names but luckily this is recoverable. As the theory behind inodes is quite complicated and is complicated to explain here, I warmly recommend you read Ian Dallen's Unix / Linux / Filesystems – directories inodes hardlinks tutorial – which is among the best academic Tutorials explaining various specifics about inodes online.


How to Get inodes per mounted filesystem


root@linux:/home/hipo# df -i
Filesystem       Inodes  IUsed   IFree IUse% Mounted on


dev             2041439     481   2040958   1% /dev
tmpfs            2046359     976   2045383   1% /run
tmpfs            2046359       4   2046355   1% /dev/shm
tmpfs            2046359       6   2046353   1% /run/lock
tmpfs            2046359      17   2046342   1% /sys/fs/cgroup
/dev/sdb5        1221600    2562   1219038   1% /usr/var/lib/mysql
/dev/sdb6        6111232  747460   5363772  13% /var/www/htdocs
/dev/sdc1      122093568 3083005 119010563   3% /mnt/backups
tmpfs            2046359      13   2046346   1% /run/user/1000

As you see in above output Inodes reported for each of mounted filesystems has a specific number. In above output IFree on every mounted FS locally on Physical installed OS Linux is good.

Here is an example on how to recognize a depleted Inodes on a OpenXen Virtual Machine with attached Virtual Hard disks.

linux:~# df -i
Filesystem         Inodes     IUsed      IFree     IUse%   Mounted on
/dev/xvda         2080768    2080768     0      100%    /
tmpfs             92187      3          92184   1%     /lib/init/rw
varrun            92187      38          92149   1%    /var/run
varlock            92187      4          92183   1%    /var/lock
udev              92187     4404        87783   5%    /dev
tmpfs             92187       1         92186   1%    /dev/shm


Finding files with a certain inode

At some cases if you want to check all the copy files of a certain file that have the same i-node pointer it is useful to find them all by their shared inode this is possible with simple find (below example is for /usr/bin/perl binary sharing same inode as perl5.28.1:


ls -i /usr/bin/perl
23798851 /usr/bin/perl*


 find /usr/bin -inum 435308 -print


Find directory that has a large number of files in it?

To get an overall number of inodes allocated by a certain directory, lets say /usr /var


root@linux:/var# du -s –inodes /usr /var
566931    /usr
56020    /var/

To get a list of directories use by inode for a directory with its main contained sub-directories sorted from 1 till highest number use:

du -s –inodes * 2>/dev/null |sort -g


Usually running out of inodes means there is a directory / fs mounts that has too many (small files) that are depleting the max count of possible inodes.

The most simple way to list directories and number of files in them on the server root directory is with a small bash shell loop like so:

for i in /*; do echo $i; find $i |wc -l; done

Another way to identify the exact directory that is most likely the bottleneck for the inode depletion in a sorted by file count, human readable form:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

This will dump a list of every directory on the root (/) filesystem prefixed with the number of files (and subdirectories) in that directory. Thus the directory with the largest number of files will be at the bottom.


The -xdev switch is used to instruct find to narrow it's search to only the device where you're initiating the search (any other sub-mounted NAS / NFS filesystems from a different device will be omited).


Print top 10 subdirectories with Highest Inode Usage


Once identifed the largest number of files directories that is perhaps the issue, to further get a list of Top subdirectories in it with highest amount of inodes used, use below cmd:


for i in `ls -1A`; do echo "`find $i | sort -u | wc -l` $i"; done | sort -rn | head -10


To list more than 10 of the top inodes used dirs change the head -10 to whatever num needed.

N.B. ! Be very cautious when running above 2 find commands on a very large filesystems as it will be I/O Excessive and in filesystems that has some failing blocks this could create further problems.

To omit putting a high I/O load on a production filesystem, it is possible to also use du + very complex regular expression:

cd /backup
du –inodes -S | sort -rh | sed -n         '1,50{/^.\{71\}/s/^\(.\{30\}\).*\(.\{37\}\)$/\1…\2/;p}'

Results returned are from top to bottom.


How to Increase the amount of Inodes count on a new created volume EXT4 filesystem

Some FS-es XFS, JFS do have an auto-increase inode feature in case if their is physical space, whether otheres such as reiserfs does not have inodes at all but still have a field reported when queried for errors. But the classical Linux ext3 / ext4 does not have a way to increase the inode number on a live filesystem. Instead the way to do it there is to prepare a brand new filesystem on a Disk / NAS / attached storage.

The number of inodes at format-time of the block storage can be as high as 4 billion inodes. Before you create the new FS, you have to partition the new the block storage as ext4 with lets say parted command (or nullify the content of an with dd to clean up any previous existing data on a volume if there was already existing data:

parted /dev/sda

dd if=/dev/zero of=/dev/path/to/volume

  then format it with this additional parameter:


mkfs.ext4 -N 3000000000 /dev/path/to/volume


Here in above example the newly created filesystem of EXT4 type will be created with 3 Billion inodes !, for setting a higher number on older ext3 filesystem max inode count mkfs.ext3 could be used instead.

Bear in mind that 3 Billion number is a too high number and if you plan to have some large number of files / directories / links structures just raise it up to your pre-planning requirements for FS. In most cases it will be rarely anyone that want to have this number higher than 1 or 2 billion of inodes.

On FreeBSD / NetBSD / OpenBSD setting inode maximum number for a UFS / UFS2 (which is current default FreeBSD FS), this could be done via newfs filesystem creation command after the disk has been labeled with disklabel:


freebsd# newfs -i 1024 /dev/ada0s1d


Increase the Max Count of Inodes for a /tmp filesystem


Sometimes on some machines it is necessery to have ability to store very high number of small files (e.g. have a very large number of inodes) on a temporary filesystem kept in memory. For example some web applications served by Web Server Apache + PHP, Nginx + Perl-FastCGI are written in a bad manner so they kept tons of temporary files in /tmp, leading to issues with exceeded amount of inodes.
If that's the case to temporary work around you can increase the count of Inodes for /tmp to a very high number like 2 billions using:


mount -o remount,nr_inodes=<bignum> /tmp

To make the change permanent on next boot if needed don't forget to put the nr_inodes=whatever_bignum as a mount option for the temporary fs to /etc/fstab

Eventually, if you face this issues it is best to immediately track which application produced the mess and ask the developer to fix his messed up programs architecture.




It was explained on the very common issue of having maximum amount of inodes on a filesystem depleted and the unpleasent consequences of inability to create new files on living FS.
Then a general overview was given on what is inode on a Linux / Unix filesystem, what is typical content of inode, how inode addressing is handled on a FS. Further was explained how to get basic information about available inodes on a filesystem, how to get a filename/s based on inode number (with find), the well known way to determine inode number of a directory or file (with ls) and get more extensive information on a FS on inodes with tune2fs.
Also was explained how to identify directories containing multitudes of files in order to determine a sub-directories that is consuming most of the inodes on a filesystem. Finally it was explained very raughly how to prepare an ext4 filesystem from scratch with predefined number to inodes to much higher than the usual defaults by mkfs.ext3 / mkfs.ext4 and *bsds newfs as well as how to raise the number of inodes of /tmp tmpfs temporary RAM filesystem.

Finding nearest package software repository in Debian GNU / Linux

Thursday, September 6th, 2012

finding debian Ubuntu package repository icon

I’m about to chage the good old computeres until this very moment this blog and few other website were running on. Right now, I’m installing the brand new machine Lenovo ThinkCentre Edge great and hopefully powerful enough machine to take care for the periodic occuring high traffic loads which break up webserver or SQL server. Well anyways, I just installed latest Debian GNU / Linux on this brand new piece of iron. During install I couldn’t connect the PC to network so Debian install was unable to determine, the nearest Debian package repository, hence after completing install and anually configuring Debian network . Because during install the system had no connection with the Internet, no proper package repository definitions were present in /etc/apt/sources.list, hence I had to find the nearest package software repository. Normally one can check in Debian official WorldWide Mirror sites full address list and determine by some rationalization with ping or / and a manual package download which repo is quickest. There is thanksfully a better automated way one can determine the closest deb Debian / Ubuntu located repository with netselect-apt.

Here is apt-cache description:

debian:~# apt-cache search netselect-apt
netselect-apt - speed tester for choosing a fast Debian mirror

Using the tool is trivial, just install, run it and it does all 4 u 🙂

1. Install netselect-apt

debian:~# apt-get install --yes netselect-apt

2. Run it

debian:~# netselect-apt
130/debian/); keeping only under first name.
netselect: 2 (2 active) nameserver request(s)...

Duplicate address (,; keeping only under first name.
netselect: 1 (1 active) nameserver request(s)...
Running netselect to choose 1 out of 383 addresses.
The fastest server seems to be:

Writing sources.list.
sources.list exists, moving to sources.list.1346964774

As you can see from output, the tool finds the quickest download deb repository and generate /etc/apt/sources.list file in current directory, where it is run in, in this exact case it creates it in root user home dir – e.g. in /root/ directory. Once the repo address is found you can copy paste it with some text editor to /etc/apt/sources.list or move it over /etc/apt/sources.list;

debian:~# cp -rpf /etc/apt/sources.list /etc/apt/sources.list.$(date +%d_%m_%Y|sed -e 's/^ *//')
debian:~# mv /root/sources.list /etc/apt/sources.list

Just in case as I always make first copy of original sources.list, this is not necessery but IMHO a generally good sysadmin habit 🙂

Besides netselect-apt, which automatically choose between all available list of software repo servers, there is also netselect tool. netselect does basically the same the only difference is one has to manually pass by as arguments deb package repositories and the tool then does tests and returns which is the overall quickest deb download source.

netselect is definitely useful if you have started few own mirror of repositories and want to determine which is the best among them.

Here is how netselect is used:

# netselect -vv \ 2792 ms 23 hops 100% ok ( 1/ 1) [ 9213] 9999 ms 30 hops 0% ok 94 ms 8 hops 100% ok (10/10) [ 169] 46 ms 15 hops 100% ok (10/10) [ 115] 9999 ms 30 hops 0% ok

According to above output, the “best reachable” (quickest) repository is the one to which are the least miliseconds –

Thanks to- -for the nice Debian package box picture – all copyrights belong to respective authors and licensing.
Cheers ! 🙂