Being a remote system administrator for many years from now, I’ve experienced many times sudden Linux crashes, mostly caused by system hardware overloads.
This is a common thing especially on busy Web Servers with (Apache/MySQL). Situations like this are really messy and many times in order to bringt the server back online, one has to either physically go to the Data Center or contact the Technical support to request a server reboot.
In this terrible times, data loss might occur if the Server reset button is used to cold reboot it.
Happily in 99% of the cases the data loss which might occur could be prevented with Linux’s kernel capability to support the Magic SysRQ key !
Having the Magic SysRQ key functionality being supported in Linux in past times wasn’t that common, thanksfully these days this has changed andlmost every Linux distrubution supports this handy feature.
Now you might wonder what is the magic with those Magic SysRQ key ?
Let me explain, Magic SysRQ is a kernel level functionality which supports even completely crashed Linux systems with the horrifying:
Kernel Panic
message to be properly shutdown.
Using Magic SysRQ instead of the mostly used indiced cold reboots is really advantageous, as all the opened files by programs on the crashed server which hanged will be properly saved and closed and thus possible data loss caused by the sudden server crash will be minimized.
One other good thing about the Magic SysRQ Key keyboard combination is that initiating the Magic SysRQ could be made blindly (no need for server monitor or display to present any error messages on the failed server).
Now to check if the magic sysrq is enabled on a server one should issue the command:
Here are some Magic SysRQ keyboard combinations one can use in case of server failure:
ALT+SYSRQ+M to dump memory info;
ALT+SYSRQ+P to dump processes states;
ALT+SYSRQ+S to sync disks;
ALT+SYSRQ+U to unmount all mounted filesystems;
ALT+SYSRQ+E to terminate processes;
ALT+SYSRQ+I to kill all processes
ALT+SYSRQ+U to try to unmount once again;
ALT+SYSRQ+B to reboot.
I would skip to explain what each of the keyboard combinations will do as I believe the above description explains it well.
One classics of combinations one might want to issue on a failed Linux server supporting the Magic SysRQ would be:
ALT+SYSRQ+R
ALT+SYSRQ+E
ALT+SYSRQ+I
ALT+SYSRQ+S
ALT+SYSRQ+U
ALT+SYSRQ+B
The ALT+SYSRQ+REISUB key combination is very popular among system administrators.
Also Magic SysRQ is heavily used by kernel developers as it includes many handy debugging options.
If you try the Magic SysRQ key on older servers and you realize it doesn’t react you will have to recompile the linux kernel and enable the CONFIG_MAGIC_SYSRQ kernel time compilation option.
The Magic SysRQ can also be initiated remotely on a properly running server 😉 by initiating:
server:~# echo b > /proc/sysrq-trigger
This command would be useful if you want to just have fun and test that magic sysrq works on your system for sure 😉
To sum it up using the Magic SysRQ will guarantee your locked up, server a Safe Reboot and will hopefully save you a lot of time for backups recovery.
More helpful Articles
Tags: apache mysql, Button, capability, center, crash, data, description, distrubution, EALT, error messages, feature, filesystems, handy feature, info, kernel level, key, keyboard combination, keyboard combinations, level, level functionality, memory info, need, RALT, Reboot, reset, reset button, Safely, server crash, server failure, server reset, servers, sync, system administrator, system hardware, Technical, technical support, terrible times, time, UALT, web servers
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0
You did not mention how to test if it is enabled
>Now to check if the magic sysrq is enabled
View CommentView Comment>on a server one should issue the command:
Mozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+ Debian/squeeze (2.30.6-1) Epiphany/2.30.6
Hello,
You can use:
sysctl -a |grep kernel.sysrq
You should get something like: kernel.sysrq = 1
Hope this helps
best,
View CommentView CommentGeorgi
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0
thanks.
One thing that is vague for me is that how someone can send magic sysrq from keyboard without sitting in front of server? Since all server activities are done remotely and normally we use putty to connect remotely. In case of a kernel crash, network is down and you can not ssh to a crashed server.
View CommentView CommentMozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+ Debian/squeeze (2.30.6-1) Epiphany/2.30.6
Hi if you want a server to auto restart on crash, perhaps you would like to check my other article – (how to Automatically Restart GNU/Linux on kernel panic) – https://www.pc-freak.net/blog/how-to-automatically-reboot-restart-debian-gnu-lenny-linux-on-kernel-panic-some-general-cpu-overload-or-system-crash-2/
Also one other way is to use IPKVM / IKVM interface to restart the server if hanged remotely 🙂
Best!
View CommentView CommentGeorgi
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0
Hi
View CommentView CommentOne thing that is vague for me is that I remember upon a kernel panic, the num lock led is no longer working. Mean that the keyboard is also not responsive. So I wonder how that magic key work?