Posts Tagged ‘inittab’

Fix “init: Id “ad” respawning too fast: disabled for 5 minutes” – Reload /etc/inittab changes in memory apply without rebooting Linux server

Thursday, April 15th, 2021

inittab-logo-reload-inittab-without-reboot

During my daily sysadmin tasks I've been contacted by a colleague, reporting issues with missing logs in rsyslog on a very old Redhat Server release 5.11.
Exact version is:

root@linux-server:~# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.11 (Tikanga)

After checking the logs, I have confirmed his finding that in reality since about more than a year logs were not produced and al I could find multiple messages in /var/log/messages reading like:

init: Id "ad" respawning too fast: disabled for 5 minutes
init: Id "ad" respawning too fast: disabled for 5 minutes
init: Id "ad" respawning too fast: disabled for 5 minutes
init: Id "ad" respawning too fast: disabled for 5 minutes
init: Id "ad" respawning too fast: disabled for 5 minutes
init: Id "ad" respawning too fast: disabled for 5 minutes

I've checked the status of rsyslog which seemed to be fine

root@linux-server:~# /etc/init.d/rsyslog status
rsyslogd (pid  13709) is running…

The redhat version on the system was

root@linux-server:~# rpm -qa |grep -i rsyslog
rsyslog-3.22.1-7.el5

 

root@linux-server:~# tail -n 16 /var/log/messages
Apr 15 17:21:25 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:26:26 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:31:27 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:36:28 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:41:29 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:46:30 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:51:31 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 17:56:32 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 18:01:33 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 18:06:34 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 18:11:35 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 18:16:38 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 18:21:39 linux-server init: Id "ad" respawning too fast: disabled for 5 minutes

 

root@linux-server:~# /etc/init.d/rsyslog status
rsyslogd (pid  13709) is running…

Since the system is so old and I've seen this message and experienced this "respawning too fast: disabled for 5 minutes" myself in the past on some old Redhat 6.0 before RHEL was born as well as on Slackware Linux. The /etc/inittab which is nowadays obsoleted in newer Linux distributions was used to keep respawing a processes which have the chance to die out for some reason. 

For those unfamiliar with inittab there is a short extract from man inittab to get idea what it is.

 

NAME
       inittab  –  format of the inittab file used by the sysv-compatible init
       process

DESCRIPTION
       The inittab file describes which processes are started  at  bootup  and
       during  normal  operation  (e.g. /etc/init.d/boot, /etc/init.d/rc, get-
       tys…).  Init(8) distinguishes multiple runlevels, each of  which  can
       have  its  own  set of processes that are started.  Valid runlevels are
       0-6 plus A, B, and C for ondemand entries.  An  entry  in  the  inittab
       file has the following format:

              id:runlevels:action:process
 

So for example the use of /etc/inittab was very handy to configure a separate TTY12 (physical console) in the text environment of Linux to log all your messages. Another good use if you had a bash / perl / python script that you wanted to respawn (resurrect itself if it does out) on OS level without adding additional software like Dan Bernstein's all famous daemontools inittab was the right thing to use. It is a pity nowadays inittab is obsoleted in modern Linux OSes but the most likely reason to remove it is if you put some broken script that overeats CPU or memory if it runs multiple times you can easily get into a hung system.

Thus the logical thing to do is to check /etc/inittab content for any strange issues with less /etc/inittab and near the end of file found the problematic process which was triggering a never ending error messages to rsyslog and the module to protect from such messages in rsyslog by values $SystemLogRateLimitInterval and $SystemLogRateLimitBurst

# configure rsyslog rate limiting
# Rate-limiting
$SystemLogRateLimitInterval 5
$SystemLogRateLimitBurst 50000

The problem causing respawning too fast: disabled for 5 minutes

Was an old version of TivSM IBM Tivoli Service Manager /opt/tivoli/tsm/client/ba/bin/dsmc, set in the past in /etc/inittab it seems some colleague after updating to a more recent version has either changed the location of dsmc binary either the architecture of old tsm itself required a record in /etc/inittab in case if for some reasons or bugs the dsmc during backup creation was dying.

root@linux-server:~# tail -8 /etc/inittab
6:2345:respawn:/sbin/mingetty tty6

# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon

#ad:2345:respawn:/opt/tivoli/tsm/client/ba/bin/dsmc sched >/dev/null 2>&1

root@linux-server:~# rpm -qa |grep -i tivsm
TIVsm-API-5.3.4-0
TIVsm-stagent-5.3.4-0
TIVsm-BA-5.3.4-0
TIVsm-API64-5.3.4-0


The logical thing to do was to check whether this binary exist at all here is the result:

root@linux-server:~$ ls -al /opt/tivoli/tsm/client/ba/bin/dsmc
ls: /opt/tivoli/tsm/client/ba/bin/dsmc: No such file or directory

Obviously someone decided to comment out the inittab support for /opt/tivoli/tsm/client/ba/bin/dsmc as the binary was not present and the dsmc backup was executed via a separate one time cron job or the service itself was configured to run continue, but forgot to reread its configuration so in the kernel memory inittab was still having the instruction to loop over the dsmc binary, since the Linux machine was not rebooted ages (1472 days) or 4.8 years time.

root@linux-server:~#  uname -a; echo; uptime
Linux linux-server2.6.18-419.el5 #1 SMP Wed Feb 22 22:40:57 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

 19:04:34 up 1472 days,  5:20,  1 user,  load average: 0.12, 0.07, 0.06


So what really happens is <b>inittab</b> is trying to kind of re-run all the time dsmc process in a similar way like it would in a bash never ending loop;


while [ 1 ]; do 
/opt/tivoli/tsm/client/ba/bin/dsmc sched
done

Since the $PATH location to the binary returns 'No such file or directory' message this message floods up the rsyslog every second which triggers the LimitBurst protection of rsyslog causing rsyslog to disable completely logging for 5 minutes. The next 5 minutes when the time expires for blocking out logging due to reached limit burst.
dsmc binary sends again few ten thousand of messages for few seconds which are already waiting in a queue of rsyslog and the LimitBurst anti DDoS protection activates again. The reason for the LimitBurst is simply because if it logging is not disabled quickly the repeating message is going to fill the hard drive of the system and noone will be able to login. So rsyslog activated the good protection.

It seems noone from support colleagues, never ever noticed this init: Id "ad" respawning too fast: disabled for 5 minutes in /var/log/messages. So since the syslog was continuesly blocked by overflow of non-sense messages, systems  normal logging was interruped and respectively prevented any other meaningful error messages and warnings from the system to get properly logged  and perhaps flooed the remote rsyslog logging servers @logging-servers:514 in /etc/rsyslog.conf


Fix to respawning too fast: disabled for 5 minutes

Very simply make /etc/inittab get reloaded in memory with:

root@linux-server:~# /sbin/init q

or with the linked telnet, which was so much used by us sys admins in the past

root@linux-server:~# /sbin/telinit q

To make the rsyslog suspension disabled of course we need to restart it again.

root@linux-server:~# /etc/init.d/rsyslog restart

root@linux-server:~# /etc/init.d/rsyslog status
rsyslogd (pid  13710) is running…

And Voila logs from services are being delivered normally via configured stuff in /etc/rsyslog.conf, to make sure this is so:

root@linux-server:~# tail -8 /var/log/messages
Apr 15 14:36:29 linux-serverinit: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 14:41:37 linux-serverinit: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 14:51:22 linux-serverinit: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 14:56:30 linux-serverinit: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 15:01:38 linux-serverinit: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 15:06:45 linux-serverinit: Id "ad" respawning too fast: disabled for 5 minutes
Apr 15 18:21:49 linux-server init: Re-reading inittab
Apr 15 18:21:54 linux-server kernel: imklog 3.22.1, log source = /proc/kmsg started.
Apr 15 18:21:54 linux-server rsyslogd: [origin software=”rsyslogd” swVersion=”3.22.1″ x-pid=”13709″ x-info=”http://www.rsyslog.com”] (re)start
Apr 15 18:41:54 linux-server rsyslogd: — MARK —
Apr 15 19:01:54 linux-server rsyslogd: — MARK —
Apr 15 19:21:54 linux-server rsyslogd: — MARK —
Apr 15 19:41:54 linux-server rsyslogd: — MARK —
Apr 15 20:01:54 linux-server rsyslogd: — MARK —

How to set a crontab to execute commands on a seconds time interval on GNU / Linux and FreeBSD

Sunday, October 30th, 2011

crontab-execute-cron-jobs-every-second-on-linux-cron-logo
Have you ever been in need to execute some commands scheduled via a crontab, every let’s say 5 seconds?, naturally this is not possible with crontab, however adding a small shell script to loop and execute a command or commands every 5 seconds and setting it up to execute once in a minute through crontab makes this possible.
Here is an example shell script that does execute commands every 5 seconds:

#!/bin/bash
command1_to_exec='/bin/ls';
command2_to_exec='/bin/pwd';
for i in $(echo 1 2 3 4 5 6 7 8 9 10 11); do
sleep 5;
$command1_to_exec; $command2_to_exec;
done

This script will issue a sleep every 5 seconds and execute the two commands defined as $command1_to_exec and $command2_to_exec

Copy paste the script to a file or fetch exec_every_5_secs_cmds.sh from here

The script can easily be modified to execute on any seconds interval delay, the record to put on cron to use with this script should look something like:

# echo '* * * * * /path/to/exec_every_5_secs_cmds.sh' | crontab -

Where of course /path/to/exec_every_5_secs_cmds.sh needs to be modified to a proper script name and path location.

Another way to do the on a number of seconds program / command schedule without using cron at all is setting up an endless loop to run/refresh via /etc/inittab with a number of predefined commands inside. An example endless loop script to run via inittab would look something like:

while [ 1 ]; do
/bin/ls
sleep 5;
done

To run the above sample never ending script using inittab, one needs to add to the end of inittab, some line like:

mine:234:respawn:/path/to/script_name.sh

A quick way to add the line from consone would be with echo:

echo 'mine:234:respawn:/path/to/script' >> /etc/inittab

Of course the proper paths, should be put in:

Then to load up the newly added inittab line, inittab needs to be reloaded with cmd:

# init q

I've also red, some other methods suggested to run programs on a periodic seconds basis using just cron, what I found in stackoverflow.com's  as a thread proposed as a solution is:

* * * * * /foo/bar/your_script
* * * * * sleep 15; /foo/bar/your_script
* * * * * sleep 30; /foo/bar/your_script
* * * * * sleep 45; /foo/bar/your_script

One guy, even suggested a shorted way with cron:

0/15 * * * * * /path/to/my/script

How to make screenshot in /dev/tty console on GNU / Linux – Taking picture JPEG / PNG snapshot of text console in systems without graphical environment

Monday, April 30th, 2012

I'm used to making picture screenshots in GNOME desktop environment. As I've said in my prior posts, I'm starting to return to my old habits of using console ttys for regular daily jobs in order to increase my work efficiency. In that manner of thoughts sometimes I need to take a screenshot of what I'm seeing in my physical (TTY consoles) to be able to later reuse this. I did some experimenting and this is how this article got born.

In this post, I will shortly explain how a picture of a command running in console or terminal in GNU / Linux can be made

Before proceeding to the core of the article, I will say few words on ttys as I believe they might be helpful someone.
The abbreviation of tty comes after TeleTYpewritter phrase and is dating back somewhere near the 1960s. The TTY was invented to help people with impaired eyesight or hearing to use a telephone like typing interface.

In Unix / Linux / BSD ttys are the physical consoles, where one logs in (typing in his user/password). There are physical ttys and virtual vtys in today *nixes. Today ttys, are used everywhere in a modern Unixes or Unix like operating system with or without graphical environments.
Various Linux distributions have different number of physical consoles (TTYs) (terminals connected to standard output) and this depends mostly on the distro major contributors, developers or surrounding OS community philosophy.
Most modern Linux distributions have at least 5 to 7 physical ttys. Some Linux distributions like Debian for instance as of time of writting this, had 7 active by default physical consoles.
Adding 3 more ttys in Debian / Ubuntu Linux is done by adding the following lines in /etc/inittab:
 

7:23:respawn:/sbin/getty 38400 tty7
8:23:respawn:/sbin/getty 38400 tty8
9:23:respawn:/sbin/getty 38400 tty9

On some Linux distributions like Fedora version 9 and newer ones, new ttys can no longer be added via /etc/inittab,as the RedHat guys changed it for some weird reason, but I guess this is too broad issue to discuss ….

In graphical environments ttys are called methaphorically "virtual". For instance in gnome-terminal or while connecting to a remote SSH server, a common tty naming would be /dev/pts/8 etc.

tty command in Linux and BSDs can be used to learn which tty, one is operating in.

Here is output from my tty command, issued on 3rd TTY (ALT+F3) on my notebook:
 

noah:~# tty
/dev/tty3

A tty cmd output from mlterm GUI terminal is like so:
 

hipo@noah:~$ tty/dev/pts/9

Now as mentioned few basic things on ttys I will proceed further to explain how I managed to:

a) Take screenshot of a plain text tty screen into .txt file format
b) take a (picture) JPG / PNG screenshot of my Linux TTY consoles content

1. Take screenshot of plain text tty screen into a plain (ASCII) .txt file:

To take a screenshot of tty1, tty2 and tty3 text consoles in a txt plain text format, cat + a standard UNIX redirect is all necessery:
 

noah:~# cat /dev/vcs1 > /home/hipo/tty1_text_screenshot.txt
noah:~# cat /dev/vcs2 > /home/hipo/tty2_text_screenshot.txt
noah:~# cat /dev/vcs3 > /home/hipo/tty3_text_screenshot.txt

This will dump the text content of the console into the respective files, if however you try to dump an ncurses library like text interactive interfaces you will end up with a bunch of unreadable mess.
In order to read the produced text 'shots' onwards less command can be used …
 

noah:~# less /home/hipo/tty1_text_screenshot.txt
noah:~# less /home/hipo/tty2_text_screenshot.txt
noah:~# less /home/hipo/tty3_text_screenshot.txt

2. Take picture JPG / PNG snapshot of Linux TTY console content

To take a screenshot of my notebook tty consoles I had to first install a "third party program" snapscreenshot . There is no deb / rpm package available as of time of writting this post for the 4 major desktop linux distributions Ubuntu, Debian, Fedora and Slackware.
Hence to install snapscreenshot,I had to manually download the latest program tar ball source and compile e.g.:
 

noah:~# cd /usr/local/src
noah:/usr/local/src# wget -q http://bisqwit.iki.fi/src/arch/snapscreenshot-1.0.14.3.tar.bz2
noah:/usr/local/src# tar -jxvvvf snapscreenshot-1.0.14.3.tar.bz2

noah:/usr/local/src# cd snapscreenshot-1.0.14.3
noah:/usr/local/src/snapscreenshot-1.0.14# ./configure && make && make install
Configuring…
Fine. Done. make.
make: Nothing to be done for `all'.
if [ ! "/usr/local/bin" = "" ]; then mkdir –parents /usr/local/bin 2>/dev/null; mkdir /usr/local/bin 2>/dev/null; \
for s in snapscreenshot ""; do if [ ! "$s" = "" ]; then \
install -c -s -o bin -g bin -m 755 "$s" /usr/local/bin/"$s";fi;\
done; \
fi; \
if [ ! "/usr/local/man" = "" ]; then mkdir –parents /usr/local/man 2>/dev/null; mkdir /usr/local/man 2>/dev/null; \
for s in snapscreenshot.1 ""; do if [ ! "$s" = "" ]; then \
install -m 644 "$s" /usr/local/man/man"`echo "$s"|sed 's/.*\.//'`"/"$s";fi;\
done; \
fi

By default snapscreenshot command is made to take screenshot in a tga image format, this format is readable by most picture viewing programs available today, however it is not too common and not so standartized for the web as the JPEG and PNG.
Therefore to make the text console tty snapshot taken in PNG or JPEG one needs to use ImageMagick's convert tool. The convert example is also shown in snapscreenshot manual page Example section.

To take a .png image format screenshot of lets say Midnight Commander interactive console file manager running in console tty1, I used the command:
 

noah:/home/hipo# snapscreenshot -c1 -x1 > ~/console-screenshot.tga && convert ~/console-screenshot.tga console-screenshot.png

Linux text console tty mc screenshot with snapscreenshot terminal / console snapshotting program

Note that you need to have read/write permissions to the /dev/vcs* otherwise the snapscreenshot will be unable to read the tty and produce an error:
 

hipo@noah:~/Desktop$ snapscreenshot -c2 -x1 > snap.tga && convert snap.tga snap.pngGeometry will be: 1x2Reading font…/dev/console: Permission denied

To take simultaneous picture screenshot of everything contained in all text consoles, ranging from tty1 to tty5, issue:
 

noah:/home/hipo# snapscreenshot -c5 -x1 > ~/console-screenshot.tga && convert ~/console-screenshot.tga console-screenshot.png

Here is a resized 480×320 pixels version of the original screenshot the command produces:

All text Consoles tty1 to tty5 merged screenshot png image with snapscreenshot taken on Debian GNU / Linux

Storing a picture shot of the text (console) screen in JPEG (JPG) format is done analogously just the convert command output extension has to be changed to jpeg i.e.:
 

noah:/home/hipo# snapscreenshot -c5 -x1 > ~/console-screenshot.tga && convert ~/console-screenshot.tga console-screenshot.jpeg

I've also written a tiny wrapper shell script, to facilitate myself picture picture taking as I didn't like to type each time I want to take a screenshot of a tty the above long line.

Here is the wrapper script I wrote:
 

#!/bin/sh
### Config
# .tga produced file name
output_f_name='console-screenshot.tga';
# gets current date
cur_date=$(date +%d_%m_%Y|sed -e 's/^ *//');
# png output f name
png_f_name="console-screenshot-$cur_date.png";
### END Config
snapscreenshot -c$arg1 -x1 > $output_f_name && convert $output_f_name $png_f_name;
echo "Output png screenshot from tty1 console produced in";
echo "$PWD/$png_f_name";
/bin/rm -f $output_f_name;

You can also download my console-screenshot.sh snapscreenshot wrapper script here

The script is quite simplistic to use, it takes just one argument which is the number of the tty you would like to screenshot.
To use my script download it in /usr/local/bin and set it executable flag:
 

noah:~# cd /usr/local/bin
noah:/usr/local/bin# wget -q https://www.pc-freak.net/~bshscr/console-screenshot.sh
noah:/usr/local/bin# chmod +x console-screenshot.sh

Onwards to use the script to snapshot console terminal (tty1) type:
 

noan:~# console-screenshot.sh

I've made also mirror of latest version of snapscreenshot-1.0.14.3.tar.bz2 here just in case this nice little program disappears from the net in future times.

 

How to solve qmail /usr/local/bin/tcpserver: libc.so.6: failed to map segment from shared object: Cannot allocate memory

Saturday, April 30th, 2011

If you’re building (compiling) a new qmail server on some Linux host and after properly installing the qmail binaries and daemontools, suddenly you notice in readproctitle service errors: or somewhere in in qmail logs for instance in/var/log/qmail/current the error:

/usr/local/bin/tcpserver: error while loading shared libraries:
libc.so.6: failed to map segment from shared object: Cannot allocate memory

then you have hit a bug caused by insufficient memory assigned for tcpserver in your /var/qmail/supervise/qmail-smtpd/run daemontools qmail-smtpd initialize script:

This kind of issue is quite common especially on hardware architectures that are 64 bit and on Linux installations that are amd65 (x86_64) e.g. run 64 bit version of Linux.

It relates to the 64 bit architecture different memory distribution and thus as I said to solve requires increase in memory softlimit specified in the run script an example good qmail-smtpd run script configuration which fixed the libc.so.6: failed to map segment from shared object: Cannot allocate memory I use currently is as follows:

#!/bin/shQMAILDUID=`id -u vpopmail`NOFILESGID=`id -g vpopmail`MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`# softlimit changed from 8000000exec /usr/local/bin/softlimit -m 32000000 /usr/local/bin/tcpserver -v -H -R -l 0 -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD"
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp
/var/qmail/bin/qmail-smtpd
/home/vpopmail/bin/vchkpw /bin/true 2>&1

The default value which was for softlimit was:

exec /usr/local/bin/softlimit -m 8000000

A good softlimit raise up values which in most cases were solving the issue for me are:

exec /usr/local/bin/softlimit -m 3000000

or exec /usr/local/bin/softlimit -m 4000000

The above example run configuration fixed the issue on a amd64 debian 5.0 lenny install, the server hardware was:

CPU: Intel(R) Core(TM)2 Duo CPU @ 2.93GHz
System Memory: 4GB
HDD Disk space: 240GB

The softlimit configuration which I had to setup on another server with system parameters:

Intel(R) Core(TM) i7 CPU (8 CPUS) @ 2.80GHz
System Memory: 8GB
HDD Disk Space: 1.4Terabytes

is as follows:

#!/bin/sh
QMAILDUID=`id -u vpopmail`
NOFILESGID=`id -g vpopmail`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
exec /usr/bin/softlimit -m 64000000
/usr/local/bin/tcpserver -v -H -R -l 0
-x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD"
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp
/var/qmail/bin/qmail-smtpd
/home/vpopmail/bin/vchkpw /bin/true 2>&1

If none of the two configurations pointed out in the post works, for you just try to manually set up the exec /usr/bin/softlimit -m to some high value.

To assure that the newly set value is not producing the same error you will have to, reload completely the daemontools proc monitor system.
To do so open /etc/inittab comment out the line:

SV:123456:respawn:/command/svscanboot
to
#SV:123456:respawn:/command/svscanboot

Save again /etc/inittab and issue te cmd:

linux:~# init q

Now again open /etc/inittab and uncomment the commented line:

#SV:123456:respawn:/command/svscanboot to
SV:123456:respawn:/command/svscanboot

Lastly reload the inittab script once again with command:

linux:~# init q

To check if the error has disappeared check the readproctitle process, like so:

linux:~# ps ax|grep -i readproctitle

The command output should produce something like:

3070 ? S 0:00 readproctitle service errors: .......................................

Hope that helps.