Posts Tagged ‘event’

Migration of audit messages from snoopy to auditd

Tuesday, April 20th, 2010

his article may be out of date and may be deleted in the future.

This article explains the migration from the previous service "Snoopy" to "Auditd". Only commands that are executed as a user with root rights should be recorded here.

 

Uninstall/disable snoopy
 

Configuration of auditd

Files needed
Auditd start/stop script

/etc/init.d/auditd

Rules for monitoring by auditd

/etc/audit/audit.rules

Auditd plugin for syslog service

/etc/audisp/plugins.d/syslog.conf

Edit the /etc/audit/audit.rules file
Auditd can be specifically configured to capture and exclude messages. The following list is helpful for excluding certain event entries ("msgtype"):

* 1000 – 1099 are for commanding the audit system
* 1100 – 1199 user space trusted application messages
* 1200 – 1299 messages internal to the audit daemon
* 1300 – 1399 audit event messages
* 1400 – 1499 kernel SE Linux use
* 1500 – 1599 AppArmor events
* 1600 – 1699 kernel crypto events
* 1700 – 1799 kernel abnormal records
* 1800 – 1999 future kernel use (maybe integrity labels and related events)
* 2001 – 2099 unused (kernel)
* 2100 – 2199 user space anomaly records
* 2200 – 2299 user space actions taken in response to anomalies
* 2300 – 2399 user space generated LSPP events
* 2400 – 2499 user space crypto events
* 2500 – 2999 future user space (maybe integrity labels and related events)

Adding the rules

In order for auditd to record the desired events, rules must be defined.

List of rules set up
Below is a list and explanation of the rules set up:

-a exclude,always -F msgtype>=2400 -F msgtype<=2499
-a exclude,always -F msgtype=PATH
-a exclude,always -F msgtype=CWD
-a exclude,always -F msgtype=EOE
-a exit,always -F arch=b64 -F auid!=0 -F auid!=4294967295 -S execve
-a exit,always -F arch=b32 -F auid!=0 -F auid!=4294967295 -S execve

The first rule excludes crypto events in user space – these include, for example, messages about a user logging in.
The second through fourth rules remove the information not necessary for monitoring before it is logged.
The fifth and sixth rules capture the commands entered by users moving within an interactive shell. Services etc. executed by the system are therefore not recorded.
It should be noted here that a separate rule must be created for systems that contain both 32- and 64-bit commands and libraries.

Rule syntax

In general, it makes sense to keep the number of existing rules low in order to reduce the load. Therefore, if possible, several rule fields (-F option) should be combined in one rule. Since Auditd obviously has a problem with multiple event entries that are defined in plain text, these have been created in individual rules. The syntax description of the individual rules is given in the next listing:

-a contains the instructions
The action value "exclude" and the list value "always" are specified for rules that should not lead to any log entry
The action values ​​"exit" and "always" have been specified for rules that should lead to a log entry
"exit" stands for a log entry after the command has been executed
-F defines a rules field
Depending on the application, the rules defined here filter by event entry ("msgtype"), architecture ("arch") and login UID ("auid").
-S stands for the syscall. In the rules that should lead to a log entry, the value "execve" is monitored – i.e. when commands are executed.

Redirect to syslog

Within the file /etc/audisp/plugins.d/syslog.conf the value

active = no
on

active = yes
set.

restart auditd with the command

/etc/init.d/auditd restart
the settings are accepted.

Additional information

The following man pages can be consulted for more information:

auditctl
audit.rules
auditd
auditd.conf

Pentecost the Birthday of the Church and the receival of the Gifts of Holy Spirit of Faith, Hope, Love for the Mankind

Saturday, June 19th, 2021

For one more year it is Pentecost 50 days after the celebrated Easter (Resurrection of Christ), we celebrate the feast that marked the birth of the Church as a Body of Christ that is binding all its members us the ordinary people who are baptized in the Name of The Father, The Son and The Holy Spirit.

Pentecost is a day of mystery that turned human history, as the same Spirit who was in Christ and has been in the Father in the Holy Trinity has descended from Heaven sent as a helper to each and every believer in Christ to strenghten and guide him in his narrow path to the Kingdom of Heaven and the Eternal salvation promised by the Savior Christ.

There is many books written about Pentecost but no book or intellectual thought is capable to transpond one's experience of receiving the Holy Spirit in the Soul and Heat of man. This glorious event is experienced by every Christian during baptism and the following Mystery of Chrismation when the Priest baptizes a new member in the name of God and is experienced, by us in the mystery of repentence when crying for our bad deeds and transgressions of the law of God, we cry in sorrow to God asking for forgiveness and renewal. The descent of the Holy Spirit over us is supernatural event that put the beginning of the contemporary understanding in our contemporary civillization of supernaturality and the realm of spirits as we understand them today where the Spirit of God is over all things.

On Pentecost it is the Holy Spirit who descended towards the Holy Apostles giving them the Super natural powers to Heal The Sick, Prophecise, speak in all human tongues, chase away evil spirits and do multitude of unseen wonderworks in human history that are continuing even to this day in the Church. The Holy Spirit has gifted all Church members with the gifts of the Spirit of Faith, Hope and Love, Endurance, Manhood, and all the virtues of man that were in Christ not because of a human effort but for the Love of God for man.
The abiding of the Holy Spirit in man is a never ending Heaven and a bliss granted for man for free. Saint Seraphim of Sarov has well described in his Conversation with Motovilov what is the experience to have the God the Holy Spirit in One's self.

The_Descent-of-the_Holy_Spirit-Soshestvie-Svetago-Duha

A bit more on the facts around the Descent of the Holy Spirit to the Apostles is in my prior article here , though this is just a very basic attempt to transfer the meaning of the feast as definitely all the books on the earth and all the human knowledge put together is like a drop in the universe compared to Holy Spirit itself.

The Spirit of Truth which the world could not comprehend was sent by Jesus Christ first to the apostles and the desciples and then to each and every Christian member of the One Holy Eastern Orthodox Church throughout human history until the end of ages.

Pentecost_Rabula_Gospels-6th-century
One of the most ancient icons of the Pentecost Syriac Rabbula Gospels 6th century

Happy Feast to All Christians ! Happy Birthday Church !

The Church feast of St. Tsar (King) Boris Mihail (Michael) I – Baptizer of Bulgaria – 2nd of May a Triumph for Bulgarian Autocephalous Church and Church Slavonic

Monday, May 2nd, 2011

saint Tsar Boris the baptizer of all Bulgaria

Today on Second of May every year in the Bulgarian Orthodox Church calendar we commemorate the memory of Saint King Boris (Michael) who is a baptizer of Bulgarian nation and among the greatest saint of Bulgaria, perhaps second in saintship after St. John of Rila.

St. King Boris was the ruler of the First bulgarian Empire (852-889) the actual title as we know it from the chronicles is not king but archont which literally means ruler, but as the main chronicles who came to us are Byzantine we have to take in consideration that Byzantine Chronicles aimed to discredit the Bulgarian rulers because of feeling inferior every non-Byzantines who in Byzantine terms were considered barbarians.

He had a notable correspondence with Pope Nicolas II head of the Western Church before the Great Schism.
Saint Boris I is famous with with 115 Questions (which are fully preserved). He  beseech the pope to sent an apostoles of faith who will teach the new nation to faith in the same time he lead a correspondence with the Byzantine emperor and the Eastern Church.

The pope sent 2 missionaries cardinal Formiosa of Portuens and Bishop Paul of Papulon. King Boris requested for Cardinal Formiosa to become a future archibishop of Bulgaria, but the pope Nicolas II being afraid of loosing ground of a choice of future Bulgarian archibishop rejected king Boris request it is interesting fact that the same cardinal Formiosa become the next Pope of the Roman Western Church in period (891 – 896).

Taking in consideration the papal's desire to dominate over him and the more freedoms given by the Byzantine Church along with the fact that Saint Cyril and Methodius's pupils came to him with a ready Church Slavonic Glagolic version of the Holy Bible and a new Church language (and considering the fact that the seven pupils of St. Cyril and Methodius were being chased away from Great Moravia (Saint Clement of Ohrid (Kliment Ohridski) and Naum Ohrdiski with Sava, Angelarius and Gorazd) and seeing the fallacy of the filioque addition to the Creed of Faith finally King Boris received baptism by the hands of Patriarch Photius.

He received holy baptismal in year 864, receiving the Christian name Michael (Mihail) receiving his Christian name from his godfather, Emperor Michael III.

According to Church tradition the reason to Baptize in Christian faith was his amazement of an icon of the Judgement Day he saw in one of his visits to Constantinople.

The Church tradition also says saint Boris I's sister have lived for a long time in Constantinople, where she received baptism and once returning to the Bulgaria she bring the light of faith here too.

Besides that some of his family has already earlier converted to Christianity and some earlier Bulgarian Khans such as Trivelius (Tervel) who is considered to saved Europe from Muslim Invasion earlier were already Christians.

King_Boris I bapttish of Byzantine Emperor Michael III

During his holy reign he has established mass Christianization of Bulgaria, where the traditional ancient pagan traditions and belief in fake gods like Tangra were abolished completely.

St. Tsar Boris has secured the Bulgarian Church an autocephalousy, he also received the saints Cyril and Methodius, when they were banished from Great Moravia.

Our saint king has secured a refuge for st. Cyril and Methodius and provided them with assistance to develop the Slavonic alphabet and literature.

After he abdicated in 889, his eldest son and successor tried to restore the old pagan religion but was deposed by Boris I. During the Council of Preslav which followed that event, the Byzantine clergy was replaced with Bulgarian and the Greek language was replaced with Old Bulgarian as an official language of the Church and the state.

Bulgaria_under_rule_of_Boris_I-st-the-baptizer-king-of-Bulgaria

In 889 Boris abdicated the throne and became a monk. His son and successor Vladimir attempted a pagan reaction (to return the Bulgarians to the old belief in Tanra), which brought Boris out of his monastic life in 893 out of zeal for Christ and a fatherhood for the future of Bulgarian nation.

Vladimir was defeated and blinded by a miracle of God as Boris had less soldiers than his pagan son but in the name of the Lord Jesus Christ he showed victorious.

Once defeating his unbelieving son, Boris gathered the Council of Preslav in which the Old Greek language in Church was replaced with Church Slavonic and most imporantly on the council King Boris on the Church assembly placed his third son, Simeon I of Bulgaria on the throne, threatening him with the same fate if he too apostatized.

Simeon I was reluctant to become a secular ruler as according to some of the written sources he was preparing to become a great spiritual leader a monk and perhaps a next archibishop or even patriarch in the newly established autocephalous Bulgarian Church but following the monk rule of humility he decided and accepted the throne but throughout his life he kept his great zeal for monasticism and enlightenment. His ruling become a golden age for the Church Slavonic he financed seriously and worked hardly to prepare educated monks and to translate a major works of the Holy Fathers such as Shestodnev (The Six Days of Creation), Simeonov Sbornik (Simeon's Collection) and many of the key works of saint Athanasius the Great, Saint Basyl the Great, Saint John of Damascus etc.

According to some historians it was King Simeon who later give birth to the second by size Monastic Community in Byzantine Emperire the Stone Monasteries of (Meteora).

Boris returned to his monastery, emerging once again in c. 895 to help Simeon fight the Magyars, who had invaded Bulgaria in alliance with the Byzantines. After the passing of this crisis, Boris resumed monastic life and led a holy life until he pass out in year 907.

Here is the daily troparion assigned by the Bulgarian Orthodox Church sang at the churches today (the text is translation from Bulgarian):

St. Boris-Michael, prince of Bulgaria, Troparion

Full of the fear of God, and enlightened by holy baptism, thou becamest a habitation of the Holy Spirit,
O right-believing King Boris; and having established the Orthodox Faith in the land of Bulgaria,
and set aside the scepter of kingship,
thou madest thine abode in the wil derness,
didst flourish in ascetic struggles, and found grace before the Lord.
And now, standing before the throne of the Most High, pray thou, that He grant unto us who entreat thee salva tion for our souls.

Free Software in Balkans 2010 (A free software conference on the Balkanies is approaching)

Tuesday, August 24th, 2010

Richard Stallman on a Free Software Conference

Today when I was reviewing my daily visited websites I come across an interesting news.
This kind of meeting is quite a news because it is the first in line where Free software users and developmers fromall around the balkanies will meet to discuss,test,share and continue the free software code and ideals.
The first Free Software conference in the Balkanies is about to be hosted in Vlora University Albania .

The conference Free Software in Balkans (FreeSB2010) is an annual meeting of the free software users, developpers and supporters in the Balkan countries. It will travel from country to country, year to year to different locations. The Conference will gather professionals, academics and enthusiasts who share the vision that software should be free and open for the community to develop and customize to its needs, and that knowledge is a communal property and free and open to everyone.

This kind of event will build up the social network between free software fans and developers and will further help in the general spread of free software on the balkanies.
This kind of meeting are already for a long time a tradition in many states in America, so having it in the balkanies is quite a development.

You can read a bit more about the exact focus of Free Software in Balkans 2010 conference here

Pitily right now there is no published scheduled list of presentations which are about to be given in the conference, but I guess the conference schedule would be out in a few days time.
I’m not yet sure if I’ll have the time and opportunity to attend the conference, however I do hope that somebody of my balkan readers will got interested into the “fabulous” Free Software event and will support the event’s initiative by visiting it 🙂

The Celebration of the feast of the temple Holy Trinity

Thursday, June 25th, 2009

Just few weeks ago it was the feast of our temple here in Dobrich the holy triniy. For this event a lot of priests either from Dobrich and the district as well as the Metropolit Kiril of Varna. Father Veliko has insisted to ordain me with a church rank. The church rank I received grants me oportunity to enter the holy altar and help in the church services (the holy liturgies). My sister has captured a couple of snapshots both video and pictures the pictures I’ve upload and can be seen

Restart hung Mac OS application – How to kill programs in Mac OS – alternative of Windows CTRL + ALT + DEL

Friday, May 23rd, 2014

mac-kill-hung-application-killing-freezed-application-on-mac-osx
If you happen to have the rare case of having a hung MAC OS X application and you're coming from a Linux / Windows background you will be certainly wonderhing how to kill Mac OS X hung application.
In Mac OS the 3 golden buttons to kill crashed application are:

 

COMMAND + OPTION + ESCAPE

Command + Option + Escape

while pressed simultaneously is the Mac Computer equivalent of Windows CTRL + ALT + DEL

mac-os-force-kill-hung-application-force-program-exit-mac-OSX

Holding together COMMAND  + OPTION + ESCAPE on MAC OS brings up the Force Quit Window showing and letting you choose between the list of open applications. To close freezed MAC application, choose it and Press the Force Quit Button this will kill immediately that application.  

To directly end application without invoking the choose Force Quit Window menu, to force a hanging app quit right click on its icon in Dock (CTRL + Click) and choose "Force Quit” from context menu.

A little bit more on why applications hung in MAC OS. Each application in MAC OS has its event queue. Event queue is created on initial application launch, event queue is buffer that accepts input from system (could be user input from kbd or mouse, messages passed from other programs etc.). Program is hanging when system detects queued events are not being used.

macOSX-EventQueue-reason-for-application-hung-in-macs
Other reasons for Mac OS hanging program is whether you're attaching detaching new hardware peripherals (i.e. problems caused by improper mount / unmounts), same hang issues are often observed on BSD and Linux. Sometimes just re-connecting (mouse, external hdd etc.) resolves it.
Program hungs due to buggy software are much rarer in Macs just like in IPhones and Ipads due to fact mac applications are very well tested until published in appstore.

Issues with program hungs in Mac sometimes happen after "sleep mode" during "system wake" function – closing, opening macbook. If a crashed program is of critical importance and you don't want to "Force Quit" with COMMAND + OPTION + ESC. Try send PC to sleep mode for a minute or 2 by pressing together OPTION + COMMAND + EJECT.

An alternative approach to solve hanging app issue is to Force-quit Finder and Dock to try that, launch Terminal

And type there:

# killall Dock

Other useful to know Mac OS keyboard combination is COMMAND + OPTION + POWERHold together Command and Option and after a while press Power – This is a shortcut to instruct your Mac PC to reboot.

Happy Sysadmin day! – Last Friday of July – The day of the system administrator

Friday, July 25th, 2014

Its-SysAdminDay-SAD-day-the-day-of-the-system-administrtator-cake-hooray

Every Last Friday of July is System Administrator's day. This relatively new technocrat tradition started as suggested by the system administrator Ted Kekatos in 2000.

SysAdmin Day or (SAD) 🙂 as it is also famous is a day to appreciate the achieviments of anyone seriously involved in IT industry. Now is time to say SAD day is not a day for programmers to celebrate as there is separate Programmer's day / Day of the Programmer feast or (DOP / DOPE ) 🙂 that is celebrated on the 256th (hexadecimal 100th, or the 28th) day of each year (September 13 during common years and on September 12 in leap years).

System Administrator Appreaciation Day – (SAAD) was historically inspired by Hewlett Packard magazine advertisement in which a system administrator is presented with flowers and fruit-baskets by grateful co-workers as thanks for installing new printers.

 

DOPE's started as proposed by Valentin Balt and Michael Cherviakov, employees of Parallel Technologies (a software company). In 2002, they tried to gather signatures for a petition to the government of Russia to recognize the day as the official programmers' day.


The modern usual way to celebrate Sysadm day is to get your system administrator a day. If you're at poisition to administrate a couple of hundreds of Windows hosts, you can send a prank message to your colleagues reminding them to buy you a beer or mass mail / set a Outlook Meeting reminder that system administrators – doesn't drink flowers and chocolate 🙂

 

A Day In The Life Of A SAN Administrator

Since 2000, a lot of UNIX / Linux sysadmins and  geeks tend to organize gatherings and Free Software celebration meetings on the SAD day.

SysAdminDay has inspired plenty of other pranky novels and music to glorify the remarkable event 🙂

Here are few to cheer you up.

System Administrator's Day song

Sysadmin day (Karaoke) song


Here are also some PoC for some weird ways to celebrate the day of the system administrator 🙂

System Administrator's Day in Russia Remix
 


HAPPY SYSTEM ADMINISTRATOR DAY – HOORAY ! 🙂

Secure Apache webserver against basic Denial of Service attacks with mod_evasive on Debian Linux

Wednesday, September 7th, 2011

Secure Apache against basic Denial of Service attacks with mod evasive, how webserver DDoS works

One good module that helps in mitigating, very basic Denial of Service attacks against Apache 1.3.x 2.0.x and 2.2.x webserver is mod_evasive

I’ve noticed however many Apache administrators out there does forget to install it on new Apache installations or even some of them haven’t heard about of it.
Therefore I wrote this small article to create some more awareness of the existence of the anti DoS module and hopefully thorugh it help some of my readers to strengthen their server security.

Here is a description on what exactly mod-evasive module does:

debian:~# apt-cache show libapache2-mod-evasive | grep -i description -A 7

Description: evasive module to minimize HTTP DoS or brute force attacks
mod_evasive is an evasive maneuvers module for Apache to provide some
protection in the event of an HTTP DoS or DDoS attack or brute force attack.
.
It is also designed to be a detection tool, and can be easily configured to
talk to ipchains, firewalls, routers, and etcetera.
.
This module only works on Apache 2.x servers

How does mod-evasive anti DoS module works?

Detection is performed by creating an internal dynamic hash table of IP Addresses and URIs, and denying any single IP address which matches the criterias:

  • Requesting the same page more than number of times per second
  • Making more than N (number) of concurrent requests on the same child per second
  • Making requests to Apache during the IP is temporarily blacklisted (in a blocking list – IP blacklist is removed after a time period))

These anti DDoS and DoS attack protection decreases the possibility that Apache gets DoSed by ana amateur DoS attack, however it still opens doors for attacks who has a large bot-nets of zoombie hosts (let’s say 10000) which will simultaneously request a page from the Apache server. The result in a scenario with a infected botnet running a DoS tool in most of the cases will be a quick exhaustion of system resources available (bandwidth, server memory and processor consumption).
Thus mod-evasive just grants a DoS and DDoS security only on a basic, level where someone tries to DoS a webserver with only possessing access to few hosts.
mod-evasive however in many cases mesaure to protect against DoS and does a great job if combined with Apache mod-security module discussed in one of my previous blog posts – Tightening PHP Security on Debian with Apache 2.2 with ModSecurity2
1. Install mod-evasive

Installing mod-evasive on Debian Lenny, Squeeze and even Wheezy is done in identical way straight using apt-get:

deiban:~# apt-get install libapache2-mod-evasive
...

2. Enable mod-evasive in Apache

debian:~# ln -sf /etc/apache2/mods-available/mod-evasive.load /etc/apache2/mods-enabled/mod-evasive.load

3. Configure the way mod-evasive deals with potential DoS attacks

Open /etc/apache2/apache2.conf, go down to the end of the file and paste inside, below three mod-evasive configuration directives:

<IfModule mod_evasive20.c>
DOSHashTableSize 3097DOS
PageCount 30
DOSSiteCount 40
DOSPageInterval 2
DOSSiteInterval 1
DOSBlockingPeriod 120
#DOSEmailNotify hipo@mymailserver.com
</IfModule>

In case of the above configuration criterias are matched, mod-evasive instructs Apache to return a 403 (Forbidden by default) error page which will conserve bandwidth and system resources in case of DoS attack attempt, especially if the DoS attack targets multiple requests to let’s say a large downloadable file or a PHP,Perl,Python script which does a lot of computation and thus consumes large portion of server CPU time.

The meaning of the above three mod-evasive config vars are as follows:

DOSHashTableSize 3097 – Increasing the DoSHashTableSize will increase performance of mod-evasive but will consume more server memory, on a busy webserver this value however should be increased
DOSPageCount 30 – Add IP in evasive temporary blacklist if a request for any IP that hits the same page 30 consequential times.
DOSSiteCount 40 – Add IP to be be blacklisted if 40 requests are made to a one and the same URL location in 1 second time
DOSBlockingPeriod 120 – Instructs the time in seconds for which an IP will get blacklisted (e.g. will get returned the 403 foribden page), this settings instructs mod-evasive to block every intruder which matches DOSPageCount 30 or DOSSiteCount 40 for 2 minutes time.
DOSPageInterval 2 – Interval of 2 seconds for which DOSPageCount can be reached.
DOSSiteInterval 1 – Interval of 1 second in which if DOSSiteCount of 40 is matched the matched IP will be blacklisted for configured period of time.

mod-evasive also supports IP whitelisting with its option DOSWhitelist , handy in cases if for example, you should allow access to a single webpage from office env consisting of hundred computers behind a NAT.
Another handy configuration option is the module capability to notify, if a DoS is originating from a number of IP addresses using the option DOSEmailNotify
Using the DOSSystemCommand in relation with iptables, could be configured to filter out any IP addresses which are found to be matching the configured mod-evasive rules.
The module also supports custom logging, if you want to keep track on IPs which are found to be trying a DoS attack against the server place in above shown configuration DOSLogDir “/var/log/apache2/evasive” and create the /var/log/apache2/evasive directory, with:
debian:~# mkdir /var/log/apache2/evasive

I decided not to log mod-evasive DoS IP matches as this will just add some extra load on the server, however in debugging some mistakenly blacklisted IPs logging is sure a must.

4. Restart Apache to load up mod-evasive debian:~# /etc/init.d/apache2 restart
...

Finally a very good reading which sheds more light on how exactly mod-evasive works and some extra module configuration options are located in the documentation bundled with the deb package to read it, issue:

debian:~# zless /usr/share/doc/libapache2-mod-evasive/README.gz

How to check MASTER / SLAVE MySQL nodes status – Check MySQL Replication Status

Thursday, April 19th, 2012

I'm doing replication for one server. Its not the first time I do configure replication between two MySQL database nodes, however since I haven't done it for a few years, my "know how" has mostly vanished so I had some troubles in setting it up. Once I followed some steps to configure replication I had to check if the two MASTER / Slave MySQL db nodes communicate properly. Hence I decided to drop a short post on that just in case if someone has to do the same or if I myself forget how I did it so I can check later on:

1. Check if MASTER MySQL server node is configured properly

The standard way to check a MySQL master node status info is with:
 

mysql> show master status;
+——————+———-+———————————————————+——————+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+——————+———-+———————————————————+——————+
| mysql-bin.000007 | 106 | database1,database2,database3 | |
+——————+———-+———————————————————+——————+
1 row in set (0.00 sec)

By putting \G some extra status info is provided:
 

mysql> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.000007
Position: 106
Binlog_Do_DB: database1,database2,database3
Binlog_Ignore_DB:
1 row in set (0.00 sec)

ERROR:
No query specified

2. Check if Slave MySQL node is configured properly

To check status of the slave the cmd is:
 

mysql> show slave status;

The command returns an output like:
 

mysql> show slave status;+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+| Slave_IO_State | Master_Host | Master_User | Master_Port | Connect_Retry | Master_Log_File | Read_Master_Log_Pos | Relay_Log_File | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error |+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+| Waiting for master to send event | HOST_NAME.COM | slave_user | 3306 | 10 | mysql-bin.000007 | 106 | mysqld-relay-bin.000002 | 251 | mysql-bin.000007 | Yes | Yes | database1,database2,database3 | | | | | | 0 | | 0 | 106 | 407 | None | | 0 | No | | | | | | 0 | No | 0 | | 0 | |+———————————-+————-+————-+————-+—————+——————+———————+————————-+—————+———————–+——————+——————-+——————————————————-+———————+——————–+————————+————————-+—————————–+————+————+————–+———————+—————–+—————–+—————-+—————+——————–+——————–+——————–+—————–+——————-+—————-+———————–+——————————-+—————+—————+—————-+—————-+

As you can see the output is not too readable, as there are too many columns and data to be displayed and this doesn't fit neither a text console nor a graphical terminal emulator.

To get more readable (more verbose) status for the SQL SLAVE, its better to use command:
 

mysql> show slave status\G;

Here is a sample returned output:
 

mysql> show slave status\G;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: HOST_NAME.COM Master_User: slave_user Master_Port: 3306 Connect_Retry: 10 Master_Log_File: mysql-bin.000007 Read_Master_Log_Pos: 106 Relay_Log_File: mysqld-relay-bin.000002 Relay_Log_Pos: 251 Relay_Master_Log_File: mysql-bin.000007 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: database1,database2,database3 Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 106 Relay_Log_Space: 407 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)ERROR: No query specified

If show master status or shwo slave status commands didn't reveal replication issue, one needs to stare at the mysql log for more info.

How to fix “ERROR 1577 (HY000) at line 1: Cannot proceed because system tables used by Event Scheduler were found damaged at server start”

Saturday, May 12th, 2012

After migrating databases data from FreeBSD MySQL 5.0.83 server to a Debian Squeeze Linux MySQL version 5.1.61, below is a mysql –version issued on both the FreeBSD and the Debian servers

freebsd# mysql --version
mysql Ver 14.12 Distrib 5.0.83, for portbld-freebsd7.2 (i386) using 5.2

debian:~# mysql --version
mysql Ver 14.14 Distrib 5.1.61, for debian-linux-gnu (i486) using readline 6.1

The data SQL dump from the FreeBSD server was dumped with following command arguments:

freebsd# mysqldump --opt --allow-keywords --add-drop-table --all-databases -u root -p > complete_db_dump.sql

Then I used sftp to transfer complete_db_dump.sql dump to the a brand new installed latest Debian Squeeze 6.0.2. The Debian server was installed using a "clean Debian install" without graphical environment with CD downloaded from debian.org's site.

On the Debian machine I imported the dump with command:

debian:~# mysq -u root -p < complete_db_dump.sql

Right After the dump was imported I re-started SQL server which was previously installed with:

debian:~# apt-get install mysql-server
The error I got after restarting the mysql server:

debian:~# #/etc/init.d/mysql restart

was:

ERROR 1577 (HY000) at line 1: Cannot proceed because system tables used by Event Scheduler were found damaged at server start
ERROR 1547 (HY000) at line 1: Column count of mysql.proc is wrong. Expected 20, found 16. The table is probably corrupted

This error cost me a lot of nerves and searching in google to solve. It took me like half an hour of serious googling ,until I finally found the FIX!!!:

debian:~# mysql_upgrade -u root -h localhost -p --verbose --force
Enter password:
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/var/run/mysqld/mysqld.sock' '--host=localhost'
Running 'mysqlcheck' with connection arguments: '--port=3306' '--socket=/var/run/mysqld/mysqld.sock' '--host=localhost'
bible.holy_bible OK
bible.holybible OK
bible.quotes_meta OK

Afterwards finally I had to restart the mysql server once again in order to finally get rid of the shitty:

ERROR 1547 (HY000) at line 1: Column count of mysql.proc is wrong. Expected 20, found 16. The table is probably corrupted error!

debian:~# /etc/init.d/mysql restart
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..

This solved the insane Column count of mysql.proc is wrong. Expected 20, found 16 once and for all!

Before I came with this fix I tried all kind of forum suggested fixes like:

debian:~# mysql_upgrade -u root -p
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
This installation of MySQL is already upgraded to 5.1.61, use --force if you still need to run mysql_upgrade

debian:~# mysql_upgrade -p
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
This installation of MySQL is already upgraded to 5.1.61, use --force if you still need to run mysql_upgrade

And few more, none of them worked the only one that worked was:

debian:~# #mysql_upgrade -u root -h localhost -p --verbose --force

I have to say big thanks to Mats Lindth wonderful blog post which provided me with the solution.

It seems, since Oracle bought the Community edition of MySQL thinks with this database server are getting more and more messy and backwards incompatible day by day.
Lately, I'm experiencing too much hassles with MySQL version incompitabilities. Maybe I should think for migrating permanently to Postgre …

By the way the ERROR 1547 (HY000) at line 1: Column count of mysql.proc is wrong. is most probably caused of some kind of password hashing incompitability between the password hashing between the BSD and Debian SQL versions, as mysql -u root -p < dump.sql, does override default stored user passwords in the mysql database tables… Such password, hashing issues were common in prior MySQL 4 to MySQL 5 migrations I've done, however since MySQL 5+ is already storing its password strings encrypted with md5 encryption I wonder why on earth this mess happens ….