Posts Tagged ‘number’

Theophany (Jesus’s baptizm in Jordan) feast in Bulgarian Orthodox Church on 6th of January

Saturday, January 7th, 2012

Baptism of Christ Theophany Old Orthodox icon

On 6th of January in our Bulgarian Orthodox Church just like in the most Christian realm, we celebrate the great feast of Theophany / Epiphany (Baptizm of our Saviour Jesus Christ in Jordan).

What exactly we celebrate is the Baptizm of our Saviour Jesus Christ in Jordan by st. John the Baptist (John the Forerunner as we call him in the church). This day is very important for us as Christians and this is why the holy fathers in the church has ordered this feast to be among the 12 most important feasts in our Church, the so called (Lords feasts).

On Theophany's day it is a rule in orthodox Church that the Great Blessing of Water is performed. The Blessing of the water and the preceding holy water from the priests blessing is taken by Church layman and we bring a little of this water to our homes to bless through that our homes.

In our Church belief the Holy Water from the Theophany feast is considered the most powerful in spiritual sense holy water as this water is the same water with which our sinless Saviour and lamb (Son) of God Jesus Christ was baptized in Jordan.

The reason why we call the feast Theophany is because God in his essence of Holy Trinity appeared clearly to mankind for a first time in Human history. Our Holy Trinity (3 essence God in one indistructable and insaparatable God – one God as God said for himself in the beginning of writtings) has revealed himself in front of all the people gathered along John the Baptist in Jordan waiting to be baptized in his three essences:

1. God the Father spoke from heaven manifesting and testifying about Jesus Christ being his beloved and only son and saviour of mankdin
,br />2. God the Son (Jesus Christ), has physically appeared to receive the baptizm to fulfill all righteousness and (the old testemential prophecies) and to begin his 3 years mission on earth.

3. God Holy Spirit) has descended from heaven on Jesus Christ in a publicly observable form of a dove

Theophany's feast is called by some english speaking orthodox christians Epiphany, but this is a term less used in orthodox christendom and much more spread in Roman Catholic one.

The Gospel readings in the church tell of the Lord's baptism by John in the Jordan River. The epistle reading of the Divine Liturgy tells of the consequences of the Lord's appearing which is the divine epiphany.

After the end of the st. Basil the Great Holy Liturgy served, the Great Blessing of Water is performed by one or more priests (depending on the number of present priests). The meaning of the blessing of the waters meaning is to show that mankind and all of God's creation, were created to be blessed and filled with the sanctifying of God's presence.

A very local unique bulgarian tradition on this number is that if the great blessings of water is performed by a priest near a river or a sea shore the cross be thrown in the water in order to bless the waters. Pulling out the holy cross from the Sea Bulgarian local Tradition on Theophany feast

Then a number of brave man jump in and do a race swimming aiming to pull out the crucifixion of the water. It is believed that the one who could pull out the cross will get God's great blessings through the upcoming church year.A person who pulled out the cross on Theophany Bulgarian tradition

Let us pray trust and hope on God to also appear to us who seek him, and show us his Holy Trinity wholeness mercies just like he did himself to the people waiting for Baptizm from John the Baptist by the holy prayes of the Theotokos and his holy saints and all heavinly hosts. Amen

How to Prevent Server inaccessibility by using a secondary SSH Server access port

Monday, December 12th, 2011

One of the Debian servers’s SSH daemon suddenly become inaccessible today. While trying to ssh I experienced the following error:

$ ssh root@my-server.net -v
OpenSSH_5.8p1 Debian-2, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to mx.soccerfame.com [83.170.104.169] port 22.
debug1: Connection established.
debug1: identity file /home/hipo/.ssh/id_rsa type -1
debug1: identity file /home/hipo/.ssh/id_rsa-cert type -1
debug1: identity file /home/hipo/.ssh/id_dsa type -1
debug1: identity file /home/hipo/.ssh/id_dsa-cert type -1
...
Connection closed by remote host

Interestingly only the SSH server and sometimes the mail server was failing to respond and therefore any mean to access the server was lost. Anyways some of the services on the server for example Nginx continued working just fine.
Some time ago while still working for design.bgweb development company, I’ve experienced some similar errors with SSH servers, so I already had a clue, on a way to work around the issue and to secure myself against the situation to loose access to remote server because the secure shell daemon has broken up.

My work around is actually very simple, I run a secondary sshd (different sshd instance) listening on a different port number.

To do so I invoke the sshd daemon on port 2207 like so:

debian:~# /usr/sbin/sshd -p 2207
debian:~#

Besides that to ensure my sshd -p 2207 will be running on next boot I add:

/usr/sbin/sshd -p 2207

to /etc/rc.local (before the script end line exit 0 ). I do set the sshd -p 2207 to run via /etc/rc.local on purpose instead of directly adding a Port 2207 line in /etc/ssh/sshd_config. The reason, why I’m not using /etc/ssh/sshd_config is that I’m not sure if using the sshd config to set a secondary port does run the port under a different sshd parent. If using the config doesn’t run the separate ssh port under a different server parent this will mean that once the main parent hangs, the secondary port will become inaccessible as well.

John McCarthy Creator and Father of Modern Artificial Intelligence and Lisp programming language creator passed away at 84

Wednesday, October 26th, 2011

John McCarthy Creator of Lisp programming language and Invetor of modern Artificial Intelligence

Yesterday night, one more Computer Genius – John McCarthy has passed away at the age of 84.
John McCarthy is mostly famous for the creation of Lisp Programming language, which was probably the most used programming language in the short past. There are plenty of corporate old iron hardwares which still run programs written in Lisp. Lisp is the language in which Richard Stallman has created his so famous EMACS text editor for GNU.

Computer Technology students, should have studied certainly Lisp in the form of Lisp Scheme.
Lisp is the the second oldest high level programming language only to be predeceded by Fortran .
Lisp gave birth to the so called Macro programming languages
and was invented by McCarthy in 1958, while he was in Massachusetts MIT university.
What is so important about Lisp is that it is de-facto the first language in the world which was written to be suitable for AI (Artificial Intelligence) researches. There is plenty of interesting information about Lisp as well as a number of forks and variations circulating for almost all the existing major operating systems nowdays.

Besides LISP creation McCarthy was in the first team who did a the first Remote Computer Chess game. The game played was among USSR and US scientists, where the moves were transferred by telegraph.
In 1972 MCCarthy was awarded with the Turing Award – (Today probably the most prestigious award for incredible technology achievements in the world).
McCarth’s home website had a lot of great papers on programming languages, mathematical theory of computation and most importantly philosophical words and notes on Artificial Intelligence
His site has a lot of his essays as well as his personal views on the world and predictions (foreseen probabilities by him) on the world future.
McCarthy had even written a short Sci-Fi story (The Robot and The Baby), the story aim was to explore the question, whether robots should have simulated emotions.John McCarthy AI later days life picture

John McCarthy is among the brightest computer genius who ever live on this planet as well as a true “icon” for a computer hacker. The news for his death is quite shocking especially after the sudden death of the creator of C programming Language and UNIX Denis Ritchie , and a week earlier the pass of Steve Jobs
It seems like no coincidence, that the brightest computer minds are departuring this life, probably God is taking them one by one just like he gave them the gifts to invent and revolutionize the technology we use today.
Surely McCarthy has left a huge landmark on technology and his name will be in the books for the generations to come.

How to Secure Apache on FreeBSD against Range header DoS vulnerability (affecting Apache 1.3/2.x)

Tuesday, August 30th, 2011

How to Secure Apache webserver on FreeBSD and CentOS against Range: header Denial of Service attack

Recently has become publicly known for the serious hole found in all Apache webserver versions 1.3.x and 2.0.x and 2.2.x. The info is to be found inside the security CVE-2011-3192 https://issues.apache.org/bugzilla/show_bug.cgi?id=51714

Apache remote denial of service is already publicly cirtuculating, since about a week and is probably to be used even more heavily in the 3 months to come. The exploit can be obtained from exploit-db.com a mirror copy of #Apache httpd Remote Denial of Service (memory exhaustion) is for download here

The DoS script is known in the wild under the name killapache.pl
killapache.pl PoC depends on perl ForkManager and thus in order to be properly run on FreeBSD, its necessery to install p5-Parallel-ForkManager bsd port :


freebsd# cd /usr/ports/devel/p5-Parallel-ForkManager
freebsd# make install && make install clean
...

Here is an example of the exploit running against an Apache webserver host.


freebsd# perl httpd_dos.pl www.targethost.com 50
host seems vuln
ATTACKING www.targethost.com [using 50 forks]
:pPpPpppPpPPppPpppPp
ATTACKING www.targethost.com [using 50 forks]
:pPpPpppPpPPppPpppPp
...

In about 30 seconds to 1 minute time the DoS attack with only 50 simultaneous connections is capable of overloading any vulnerable Apache server.

It causes the webserver to consume all the machine memory and memory swap and consequently makes the server to crash in most cases.
During the Denial of Service attack is in action access the websites hosted on the webserver becomes either hell slow or completely absent.

The DoS attack is quite a shock as it is based on an Apache range problem which started in year 2007.

Today, Debian has issued a new versions of Apache deb package for Debian 5 Lenny and Debian 6, the new packages are said to have fixed the issue.

I assume that Ubuntu and most of the rest Debian distrubtions will have the apache’s range header DoS patched versions either today or in the coming few days.
Therefore work around the issue on debian based servers can easily be done with the usual apt-get update && apt-get upgrade

On other Linux systems as well as FreeBSD there are work arounds pointed out, which can be implemented to close temporary the Apache DoS hole.

1. Limiting large number of range requests

The first suggested solution is to limit the lenght of range header requests Apache can serve. To implement this work raround its necessery to put at the end of httpd.conf config:


# Drop the Range header when more than 5 ranges.
# CVE-2011-3192
SetEnvIf Range (?:,.*?){5,5} bad-range=1
RequestHeader unset Range env=bad-range
# We always drop Request-Range; as this is a legacy
# dating back to MSIE3 and Netscape 2 and 3.
RequestHeader unset Request-Range
# optional logging.
CustomLog logs/range-CVE-2011-3192.log common env=bad-range
CustomLog logs/range-CVE-2011-3192.log common env=bad-req-range

2. Reject Range requests for more than 5 ranges in Range: header

Once again to implement this work around paste in Apache config file:

This DoS solution is not recommended (in my view), as it uses mod_rewrite to implement th efix and might be additionally another open window for DoS attack as mod_rewrite is generally CPU consuming.


# Reject request when more than 5 ranges in the Range: header.
# CVE-2011-3192
#
RewriteEngine on
RewriteCond %{HTTP:range} !(bytes=[^,]+(,[^,]+){0,4}$|^$)
# RewriteCond %{HTTP:request-range} !(bytes=[^,]+(?:,[^,]+){0,4}$|^$)
RewriteRule .* - [F]

# We always drop Request-Range; as this is a legacy
# dating back to MSIE3 and Netscape 2 and 3.
RequestHeader unset Request-Range

3. Limit the size of Range request fields to few hundreds
To do so put in httpd.conf:


LimitRequestFieldSize 200

4. Dis-allow completely Range headers: via mod_headers Apache module

In httpd.conf put:


RequestHeader unset Range
RequestHeader unset Request-Range

This work around could create problems on some websites, which are made in a way that the Request-Range is used.

5. Deploy a tiny Apache module to count the number of Range Requests and drop connections in case of high number of Range: requests

This solution in my view is the best one, I’ve tested it and I can confirm on FreeBSD works like a charm.
To secure FreeBSD host Apache, against the Range Request: DoS using mod_rangecnt, one can literally follow the methodology explained in mod_rangecnt.c header:


freebsd# wget http://people.apache.org/~dirkx/mod_rangecnt.c
..
# compile the mod_rangecnt module
freebsd# /usr/local/sbin/apxs -c mod_rangecnt.c
...
# install mod_rangecnt module to Apache
freebsd# /usr/local/sbin/apxs -i -a mod_rangecnt.la
...

Finally to load the newly installed mod_rangecnt, Apache restart is required:


freebsd# /usr/local/etc/rc.d/apache2 restart
...

I’ve tested the module on i386 FreeBSD install, so I can’t confirm this steps works fine on 64 bit FreeBSD install, I would be glad if I can hear from someone if mod_rangecnt is properly compiled and installed fine also on 6 bit BSD arch.

Deploying the mod_rangecnt.c Range: Header to prevent against the Apache DoS on 64 bit x86_amd64 CentOS 5.6 Final is also done without any pitfalls.


[root@centos ~]# uname -a;
Linux centos 2.6.18-194.11.3.el5 #1 SMP Mon Aug 30 16:19:16 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@centos ~]# /usr/sbin/apxs -c mod_rangecnt.c
...
/usr/lib64/apr-1/build/libtool --silent --mode=link gcc -o mod_rangecnt.la -rpath /usr/lib64/httpd/modules -module -avoid-version mod_rangecnt.lo
[root@centos ~]# /usr/sbin/apxs -i -a mod_rangecnt.la
...
Libraries have been installed in:
/usr/lib64/httpd/modules
...
[root@centos ~]# /etc/init.d/httpd configtest
Syntax OK
[root@centos ~]# /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

After applying the mod_rangecnt patch if all is fine the memory exhaustion perl DoS script‘s output should be like so:


freebsd# perl httpd_dos.pl www.patched-apache-host.com 50
Host does not seem vulnerable

All of the above pointed work-arounds are only a temporary solution to these Grave Apache DoS byterange vulnerability , a few days after the original vulnerability emerged and some of the up-pointed work arounds were pointed. There was information, that still, there are ways that the vulnerability can be exploited.
Hopefully in the coming few weeks Apache dev team should be ready with rock solid work around to the severe problem.

In 2 years duration these is the second serious Apache Denial of Service vulnerability after before a one and a half year the so called Slowloris Denial of Service attack was capable to DoS most of the Apache installations on the Net.

Slowloris, has never received the publicity of the Range Header DoS as it was not that critical as the mod_range, however this is a good indicator that the code quality of Apache is slowly decreasing and might need a serious security evaluation.

How to reboot remotely Linux server if reboot, shutdown and init commands are not working (/sbin/reboot: Input/output error) – Reboot Linux in emergency using MagicSysRQ kernel sysctl variable

Saturday, July 23rd, 2011

SysRQ an alternative way to restart unrestartable Linux server

I’ve been in a situation today, where one Linux server’s hard drive SCSI driver or the physical drive is starting to break off where in dmesg kernel log, I can see a lot of errors like:

[178071.998440] sd 0:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
[178071.998440] end_request: I/O error, dev sda, sector 89615868

I tried a number of things to remount the hdd which was throwing out errors in read only mode, but almost all commands I typed on the server were either shown as missng or returning an error:
Input/output error

Just ot give you an idea what I mean, here is a paste from the shell:

linux-server:/# vim /etc/fstab
-bash: vim: command not found
linux-server:/# vi /etc/fstab
-bash: vi: command not found
linux-server:/# mcedit /etc/fstab
-bash: /usr/bin/mcedit: Input/output error
linux-server:/# fdisk -l
-bash: /sbin/fdisk: Input/output error

After I’ve tried all kind of things to try to diagnose the server and all seemed failing, I thought next a reboot might help as on server boot the filesystems will get checked with fsck and fsck might be able to fix (at least temporary) the mess.

I went on and tried to restart the system, and guess what? I got:

/sbin/reboot init Input/output error

I hoped that at least /sbin/shutdown or /sbin/init commands might work out and since I couldn’t use the reboot command I tried this two as well just to get once again:

linux-server:/# shutdown -r now
bash: /sbin/shutdown: Input/output error
linux-server:/# init 6
bash: /sbin/init: Input/output error

You see now the situation was not pinky, it seemed there was no way to reboot the system …
Moreover the server is located in remote Data Center and I the tech support there is conducting assigned task with the speed of a turtle.
The server had no remote reboot, web front end or anything and thefore I needed desperately a way to be able to restart the machine.

A bit of research on the issue has led me to other people who experienced the /sbin/reboot init Input/output error error mostly caused by servers with failing hard drives as well as due to HDD control driver bugs in the Linux kernel.

As I was looking for another alternative way to reboot my Linux machine in hope this would help. I came across a blog post Rebooting the Magic Wayhttp://www.linuxjournal.com/content/rebooting-magic-way

As it was suggested in Cory’s blog a nice alternative way to restart a Linux machine without using reboot, shutdown or init cmds is through a reboot with the Magic SysRQ key combination

The only condition for the Magic SysRQ key to work is to have enabled the SysRQ – CONFIG_MAGIC_SYSRQ in Kernel compile time.
As of today luckily SysRQ Magic key is compiled and enabled by default in almost all modern day Linux distributions in this numbers Debian, Fedora and their derivative distributions.

To use the sysrq kernel capabilities as a mean to restart the server, it’s necessery first to activate the sysrq through sysctl, like so:

linux-server:~# sysctl -w kernel.sysrq=1
kernel.sysrq = 1

I found enabling the kernel.sysrq = 1 permanently in the kernel is also quite a good idea, to achieve that I used:

echo 'kernel.sysrq = 1' >> /etc/sysctl.conf

Next it’s wise to use the sync command to sync any opened files on the server as well stopping as much of the server active running services (MySQL, Apache etc.).

linux-server:~# sync

Now to reboot the Linux server, I used the /proc Linux virtual filesystem by issuing:

linux-server:~# echo b > /proc/sysrq-trigger

Using the echo b > /proc/sysrq-trigger simulates a keyboard key press which does invoke the Magic SysRQ kernel capabilities and hence instructs the kernel to immediately reboot the system.
However one should be careful with using the sysrq-trigger because it’s not a complete substitute for /sbin/reboot or /sbin/shutdown -r commands.
One major difference between the standard way to reboot via /sbin/reboot is that reboot kills all the running processes on the Linux machine and attempts to unmount all filesystems, before it proceeds to sending the kernel reboot instruction.

Using echo b > /proc/sysrq-trigger, however neither tries to umount mounted filesystems nor tries to kill all processes and sync the filesystem, so on a heavy loaded (SQL data critical) server, its use might create enormous problems and lead to severe data loss!

SO BEWARE be sure you know what you’re doing before you proceed using /proc/sysrq-trigger as a way to reboot ;).

20th of July St. Elijah (Elias) feast in Bulgarian Orthodox Church / st. prophet Elias short Living

Thursday, July 21st, 2011

Saint Prophet Elijah Bulgarian Orthodox Christian icon

It’s again 20th of July and we in the Bulgarian Orthodox Church, celebrate the Holy memory of st. Prophet Elijah (Elias

St. Elijah is among the greatest Old Testamental prophets. He lived before the birth of our saviour Jesus Christ in a time when apostacy from true faith was severe.

In Elias’s time (few centuries B.C.), there used to be also a severe hunger on the land.
Elias was among the little number of Jews who did continued to honour the True God creator of Heaven and of Earth instead of turning to false religious teachings inspired by demons, like it’s the Baal’s cult which was widely spread among Jews back in the time.

Because of his faith in the true God, st. Prophet Elijah was hunted by the king’s wife Jezabel and he was forced to escape from Bersheeba and went alone in the wilderness and sat down in despondency under a juniper tree.

As he slept, an angel touched him, and said unto him, “Arise and eat; because the journey is too great for thee.” He arose and found a cake and a cruse of water. Having partaken of the provision, he went forward on his way for forty days to Horeb, where he took residence in a cave. Here God appeared to him and said, “What dost thou here, Elijah?” In answer to Elijah’s despondent words God manifests to him his glory, and then directs him to return to Damascus and anoint Hazael king over the Arameans (Syria), Jehu king over Israel, and Elisha to be prophet in his room (1 Kings 19:13-21; compare 2 Kings 8:7-15; 9:1-10).

Some six years after this he prophecised and warned by God’s commandment Ahab and Jezebel of the violent deaths they would die (1 Kings 21:19-24; 22:38)

The reason for the great hunger at this times was the fact that no rain has fallen down to the land for two years, as Elias prayed to God that no rain will be given to the people of Israel until they turn back to their true God.

One of the most notable moments of st. Elias’s earhtly living was the great miracle he performed by God’s providence which proofed that only the God of Israel is the true God of heaven and earth.

This happened while he was still in desolation on mount Carmel, it came to pass that Elijah met Obadiah, one of king Ahab’s officers (the king at that time of Israel).
Obadiah was he sent out at this time to seek for pasturage for the cattle (as the hunger on the land was so severe that even pastures for animals was obscure).
Elias’s by God’s word bade him go and tell his master that Elijah was there. King Ahab came forth and met Elias and accused him to be the “Trouble of Israel” as, he thought that his lands troubles are because Elias did not turns to Baal as the the king himself and most of the Jews at that time.
It was then proposed by Elias, that sacrifices should be publicly offered (to determine whose God is in control of taking out the hunger – e.g. Elias’s God of Israel or Baal to whom has many falsely turned.
, for the purpose of determining whether Baal or the Israelite’s God is the one’s master of the Universe.
The competition was accepted by Ahab and many of the prophets of Baal gathered together to “beg” his God for a miracle.

Baal’s false manu prophets gathered around the sacrifice place and started Begging Baal to send fire and burn the sacrificial altar.
Nomatter how hard they screamed, nothing happened so eventually they started cutting their bodies pouring blood in a hope to catch baal’s attention and make him have mercy on them.

As long time passed on and no miracle happened Elias told baal’s worshippers to step back from the sacrificial calf on the built altar and ordered some of the king’s servants to spill water over the calf (three times).
The water brought was so much that the whole altar trench was over-filled with water.

Then saint Elias begged to God: “Oh Lord, God of Abraham, Isaac and Jacob, please hear me! Let oh God this people recognize that You Lord is God and please oh Lord turn the people’s hearts towards you.”, he was still praying when fire came down from heaven and burnt the stones, the water in and the trench itself.

When all the gathered people around saw this great miracle of God, they bowed down and confessed the One and only God.
By Elias’s order the Baal’s false priests has been killed.
After this great miracle, Elias said to Ahab to came back home, before the rain has followed him.
The saint prophet climbed to the mountain top and prayed for a very long time.

After the holy man of God completed his prayers a powerful wind started blowing and the sky has filled with big clouds and a pouring rain come down to the overdried earth.
The king’s wife Jezabel, still kept her stubborness and continued severely hunting for st. Elias and even gave a vow to kill him because all her prophets were killed.

But God again was with st. Elias and saved him by warning him to run away in the wilderness.

Elias was in a big sorrow, here because many of the people still continued to honour the false god baal even though all the great heavenly signs.

God however by a great miracles, has once again shown the prophet he should be of a good spirit always and always to trust in God as he even send his Angel to give Elijah food in the desert.

Elias was among the only 7000 of Israelish man who does rejected to venerate the false god Baal.

By God’s providence st. Elias had a pupil prophet called Elijah.
The time now drew near when he was to be taken up into heaven (2 Kings 2:1-12). He went down to Gilgal, where there was a school of prophets, and where his successor Elisha, whom he had anointed some years before, resided.
Elisha was distraught by the thought of his master’s leaving him, and refused to be parted from him. The two went on and came to Bethel and Jericho, and crossed the Jordan, the waters of which were “divided hither and thither” when smitten with Elijah’s mantle.
Upon arriving at the borders of Gilead, which Elijah had left many years before, it “came to pass as they still went on and talked” they were suddenly separated by a chariot and horses of fire; and “Elijah went up by a whirlwind into heaven,” Elisha receiving his mantle, which fell from Elijah as he ascended.
Elijah’s chosen successor was the prophet Elisha; Elijah designated Elisha as such by leaving his mantle with him (2 Kings 2:13-15), so that his wish for “a double portion” of the older prophet’s spirit (2:9)

Later on in New Testamential time it was again st. prophet Elias, who appearance in glory on Mount Tabor and talked with our Saviour Christ at the Son of God’s Glorious Transfiguration.
An interesting fact is that Elias and Enoch are the only two people who according to the Holy writtings were taken by God straight to heaven. No other mortal man has been taken that way in flesh.

According to Orthodox Church’s tradition, Elias and Enoch will be brought back to earth again by God in the last days of the anti-christ, when they will again testify about our Lord Jesus Christ’s being the truth God and will also expose the anti-christ who will falsely pretend as being the true god of Israel and a true saviour of mankind.

Again according to this church history this two prophets at this last days, will be given an enormous power from God, even a power to stop the rain to come down to earth and the power to command the rivers to stop.

Oh Holy Prophet Elias, let God have mercy on us the sinners by your Holy prayers and save us, give us grace and always show us His great mercies.
Amen!

How to enable AUTO fsck (ext3, ext4, reiserfs, LVM filesystems) checking on Linux boot through /etc/fstab

Tuesday, July 12th, 2011

How to auto FSCK manual fsck screenshot

Are you an administrator of servers and it happens a server is DOWN.
You request the Data Center to reboot, however suddenly the server fails to boot properly and you have to request for IPKVM or some web java interface to directly access the server physical terminal …

This is a very normal admin scenario and many people who have worked in the field of remote system administrators (like me), should have experienced that bad times multiple times.

Sadly enough only a insignifant number of administrators try to do their best to reduce this down times to resolve client stuff downtime but prefer spending time playing the ztype! game or watching some porn website 😉

Anyways there are plenty of things like Server Auto Reboot on Crash with software Watchdog etc., that we as sysadmins can do to reduce server downtimes and most of the manual human interactions on server boot time.

In that manner of thougts a very common thing when setting up a new Linux server that many server admins forget or don’t know is to enable all the server partition filesystems to be auto fscked during server boot time.

By not enabling the auto filesystem check options in Linux the server filesystems did not automatically scan and fix hard drive partitions for fs innode inconsistencies.
Even though the filesystems are tuned to automatically get checked on every 38 system reboots, still if some kind of filesystem errors are found that require a manual confirmation the boot process is interrupted and the admin ends up with a server which is not reachable remotely via ssh !

For the remote system administrator, this times are a terrible times of waitings, prayers and hopes that the server hardware is fine 😉 as well as being on hold to get a KVM to get into the server manually and enter the necessery input to fsck prompt.

Many of this bad times can be completely avoided with a very simple fix through /etc/fstab by enabling all server partitions containing any filesystem to be automatically checked and fixed in case if inconsistencies or errors are found by fsck.ext3, fsck.ext4, fsck.reiserfs etc. commands.

A very typical default /etc/fstab file you will find on many servers should look something like:

/dev/sda8 / ext3 errors=remount-ro 0 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda1 /home ext3 defaults 0 0

Notice the line:
/dev/sda1 /home ext3 defaults 0 0

The first column in the example contains the device name, the second one its mount point, third its filesystem type, fourth the mount options, fifth (a number) dump options, and sixth (another number) filesystem check options. Let’s take a closer look at this stuff.

The ones which are interesting to enable auto fsck checking and error resolving is provided usually by the last sixth variable (filesystem check option) which in the above example equals 0 .

When the filesystem check option equals 0 this means the auto fsck and repair for the respective filesystem is disabled.
Some time in the past the dump backup option (5th option in the example) was also used but as far as I can understand today it’s not that important in modern GNU/Linux distributions.

Now having the above sample crontab in order to enable the fsck file checking on Linux boot for /dev/sda1 , we will need to modify the above line’s filesystem check option be 2, e.g. the line would afterwards look like:

/dev/sda1 /home ext3 defaults 0 2

Setting the 2 as an option for filesystem check is necessery for every filesystem which is not mounted as a root filesystem /

In above example /etc/fstab you already see that auto filesystem fsck is enabled for root partition:

/dev/sda8 / ext3 errors=remount-ro 0 1
(notice the 1 in the end of the line)

Finally a modified version of the default sample /etc/fstab which will check the extra /dev/sda1 /home partition would look like so:

/dev/sda8 / ext3 errors=remount-ro 0 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda1 /home ext3 defaults 0 2

Making sure all Linux server partitions has the auto filesystem check option enabled is something absoultely necessery!
Enabling the auto fsck on servers always makes me sleep calmer 😉
Hope it helps your too. 🙂