How to reboot remotely Linux server if reboot,
shutdown and init commands are not working (/sbin/reboot:
Input/output error) - Reboot Linux in emergency using MagicSysRQ
kernel sysctl variable
I've been in a situation today, where one Linux server's hard drive
SCSI driver or the physical drive is starting to break off where in
dmesg kernel log, I can see a lot of errors like:
[178071.998440] sd 0:0:0:0: [sda] Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
[178071.998440] end_request: I/O error, dev sda, sector
89615868
I tried a number of things to remount the hdd which was throwing
out errors in read only mode, but almost all commands I typed on
the server were either shown as missng or returning an error:
Input/output error
Just ot give you an idea what I mean, here is a paste from the
shell:
linux-server:/# vim /etc/fstab
-bash: vim: command not found
linux-server:/# vi /etc/fstab
-bash: vi: command not found
linux-server:/# mcedit /etc/fstab
-bash: /usr/bin/mcedit: Input/output error
linux-server:/# fdisk -l
-bash: /sbin/fdisk: Input/output error
After I've tried all kind of things to try to diagnose the server
and all seemed failing, I thought next a reboot might help as on
server boot the filesystems will get checked with
fsck and
fsck might be able to fix (at least temporary) the mess.
I went on and tried to restart the system, and guess what? I
got:
/sbin/reboot init Input/output error
I hoped that at least
/sbin/shutdown or
/sbin/init
commands might work out and since I couldn't use the reboot command
I tried this two as well just to get once again:
linux-server:/# shutdown -r now
bash: /sbin/shutdown: Input/output error
linux-server:/# init 6
bash: /sbin/init: Input/output error
You see now the situation was not pinky, it seemed there was no way
to reboot the system ...
Moreover the server is located in remote Data Center and I the tech
support there is conducting assigned task with the speed of a
turtle.
The server had no remote reboot, web front end or anything and
thefore I needed desperately a way to be able to restart the
machine.
A bit of research on the issue has led me to other people who
experienced the
/sbin/reboot init Input/output error error
mostly caused by servers with failing hard drives as well as due to
HDD control driver bugs in the Linux kernel.
As I was looking for another alternative way to reboot my Linux
machine in hope this would help. I came across a blog post
Rebooting the Magic Way -
http://www.linuxjournal.com/content/rebooting-magic-way
As it was suggested in Cory's blog a
nice alternative way to
restart a Linux machine without using
reboot, shutdown or
init cmds is through
a reboot with the Magic SysRQ key combination
The only condition for the
Magic SysRQ key to work is to
have enabled the
SysRQ - CONFIG_MAGIC_SYSRQ in Kernel
compile time.
As of today luckily
SysRQ Magic key is compiled and enabled
by default in almost all modern day Linux distributions in this
numbers Debian, Fedora and their derivative distributions.
To use the
sysrq kernel capabilities as a mean to restart
the server, it's necessery first to activate the sysrq through
sysctl, like so:
linux-server:~# sysctl -w kernel.sysrq=1 kernel.sysrq =
1
I found enabling the
kernel.sysrq = 1 permanently in the
kernel is also quite a good idea, to achieve that I used:
echo 'kernel.sysrq = 1' >>
/etc/sysctl.conf
Next it's wise to use the
sync command to sync any opened
files on the server as well stopping as much of the server active
running services (MySQL, Apache etc.).
linux-server:~# sync
Now to reboot the Linux server, I used the
/proc Linux
virtual filesystem by issuing:
linux-server:~# echo b >
/proc/sysrq-trigger
Using the
echo b > /proc/sysrq-trigger simulates a
keyboard key press which does invoke the Magic SysRQ kernel
capabilities and hence instructs the kernel to immediately reboot
the system.
However one should be careful with using the
sysrq-trigger
because it's not a complete substitute for
/sbin/reboot or
/sbin/shutdown -r commands.
One major difference between the standard way to reboot via
/sbin/reboot is that
reboot kills all the running
processes on the Linux machine and attempts to unmount all
filesystems, before it proceeds to sending the kernel reboot
instruction.
Using
echo b > /proc/sysrq-trigger, however neither tries
to umount mounted filesystems nor tries to kill all processes and
sync the filesystem, so on a heavy loaded (SQL data critical)
server, its use might create enormous problems and lead to severe
data loss!
SO BEWARE be sure you know what you're doing before you proceed
using
/proc/sysrq-trigger as a way to reboot ;).