Posts Tagged ‘Anyhow’

Play Midis on Linux / Make Linux MIDI Ready for the Future – Enable embedded MIDI music to play in a Browser, Play MIDIs with VLC and howto enjoy Midis in Text Console

Wednesday, October 4th, 2017

how-to-play-midi-on-gnu-linux-in-graphic-environment-console-and-browser-midi-synthesizer-and-linux-tux-together

 

Play Midis on Linux or Make Linux MIDI Ready for the Future – Enable embedded MIDI music to play in a Browser, Play MIDIs with VLC and howto enjoy Midis in Text Console HOWTO

 

Playing MIDI has been quite a lot of fun historically,

if you grow up in the days when personal computers were still young and the Sound Blaster was a luxury, before the raise of Mp3 music format, you have certainly enjoyed the beeping of PC Speaker and later on during 386 and 486 / 586 computers the enjoyment of playing tracked music such as S3M and MOD,

in that good days playing MIDI music was the only alternative for PC maniacs who doesn't own a CD Drive (which itself) was another luxury and even thouse who had a CD ROM device, were mainly playing music in CD audio format (.CDA).
Anyhow MIDI was a cheap and a CPU unintensive way to listen to equivalent of favourite popular Audio Songs and for those who still remember many of the songs were recreated in MIDI format, just with a number of synthesized instruments without any voice (as MIDI is usually).

The same was true also for the good old days of raise of Mobile Phones, when polyphonic was a standard as CPU power was low MIDI was a perfect substitute for the CPU heavy Encoded MP3s / OGG and other formats that required a modern for that time Intel CPU running in 50+ Mhz usually 100 / 166Mhz was perfect for the days to play Mp3 but still even on that PCs we listened to Midi songs.

Therefore if you're one of those people like me who still enjoy to play some Midi Music in the year 2017 and feel a bit like Back into the Future movie and a Free Software fan and user, especially if you're a novice GNU  / Linux Free Software user, you will be unpleasently surprised that most today's default Linux distributions doesn't have an easy way to play Midi music format out of the box right after install.

Hence below article aims to give you an understanding on

How you can play Midi Music on GNU / Linux Operating System

First, lets Prepare to load necessery Linux kernel modules to make sure MIDI can be played by soundcard:

In /etc/modules make sure you have the following list of modules loaded:
 

linux-desktop:~# cat /etc/modules
3c59x
snd-emu10k1
snd-pcm-oss
snd-mixer-oss
snd-seq-oss

!Note the modules are working as of time of writting and in time can change to some other modules, depending on how the development of ALSA (Advanced Linux Sound Architecture) goes, and if the developers decide to rename the upmentioned modules

If you just have added the modules to /etc/modules with vim / nano to reload modules into the Linux kernel run:

 

linux-desktop:~# modprobe -a


Secondly, Installing a whole bunch of MIDI music related program tools can be achieved in Debian by installing the multimedia-midi package, e.g.:

 

linux-desktop:~# apt-get install –yes multimedia-midi

 

1. Playing Midi in Graphical environment with a double click using VLC


How to make MIDI easy listanable in Linux graphical environment like GNOME / KDE / XFCE desktop ?

 

If you want to make Midi music execution sa easy as  just clicking on the .MIDI file format on Linux you can do that with a midi extension available for VLC (Video Lan Client) Universal Multi Platform Media Player player

To install it on Debian Ubuntu GNU / Linux
 

# apt-get install –yes vlc-plugin-fluidsynth

 

Необходимо е да се изтеглят 6754 B архиви.
След тази операция ще бъде използвано 35,8 kB допълнително дисково пространство.
Изт:1 http://deb.debian.org/debian stretch/main amd64 vlc-plugin-fluidsynth amd64 2.2.6-1~deb9u1 [6754 B]
Изтеглени 6754 B за 0с (33,6 kB/сек)           
Selecting previously unselected package vlc-plugin-fluidsynth:amd64.
(Reading database … 382976 files and directories currently installed.)
Preparing to unpack …/vlc-plugin-fluidsynth_2.2.6-1~deb9u1_amd64.deb …
Unpacking vlc-plugin-fluidsynth:amd64 (2.2.6-1~deb9u1) …
Setting up vlc-plugin-fluidsynth:amd64 (2.2.6-1~deb9u1) …
Processing triggers for libvlc-bin:amd64 (2.2.6-1~deb9u1) …


Besides making your MIDI play on the GUI environment easy as a a point and click VLC will also be able to play MIDIs on GNU / Linux from your favourite browser (nomatter Firefox / Chrome or Opera), even though the player would play in a new PopUP Window it is easy to select once MIDI file from a random website for example – here is a directory listing of Webserver with Doom II Soundtrack in MIDI format , click over any file from list and Choose option for VLC to always remember that MIDI files has to be opened with VLC player.
 


2. Enable Firefox / IceWeasel browser to Support Website embedded MIDI files

 

 

So VLC could make you listen the downloadable MIDIs from Web pages but,
 

What if you have stumbled on an old website which was configured with very OLD HTML Code to play some nice music (or even different MIDI songs) for each part of the website (for each webpage) and you want to have the Websites created with embedded MIDIs to automatically play on Linux oncce you visit the site?


Sadly default support in Browser for MIDI across all GNU / Linux, I've used so far never worked out of the box, not that still anyone is developing modern websites with MIDIs, but still for the sake of backward compitability and for sake of interactivity it is worthy to enable embedded MIDI support in Linux

But with a couple of tunings as usual GNU / Linux can do almost everything, so here is how to enable embedded browser support for Midi on Linux (That should work with minor modifications not only on Debian / Ubuntu / ArchLinux but also on Fedoras, CentOS etc.
If you try it on any of this distributions, please drop a short comment and tell me in few lines how you made embedded midi worked on that distros.

 

apt-get install –yes timidity mozplugger

Next do restart firefox

Sometimes in order to work you might need to delete /home/[YOUR_USERNAME]/.mozilla/pluginreg.dat and restart firefox again, e.g. make a backup and give it a try:

 

cp -rpf /home/hipo/.mozilla/pluginreg.dat /home/hipo/.mozilla/pluginreg.dat.bak
rm -f /home/hipo/.mozilla/pluginreg.dat

 

Another good tip as talking for embedding MIDI support is to embed XPDF to render PDF pages inside the Browser, by default this is done by GNOME's Evince PDF reader but as it is sometimes buggy and might crash it is generally a good idea to switch to xpdf instead, if for some reason PDF is not directly displaying in browser or suddenly stopped working after some distro uipgrade, you might want to do below as well:
 

apt-get install xpdf

vim /etc/mozpluggerrc

Fin d and Comment out the line starting with:

It should look like this afterwards:

 Repeat Swallow ….
 

text/x-pdf: pdf: PDF file
#      repeat swallow(documentShell) fill: acroread -geometry +9000+9000 +useFrontEndProgram "$file"
        repeat noisy swallow(Xpdf) fill: xpdf -g +9000+9000 "$file"
        repeat noisy swallow(gv) fill: gv –safer –quiet –antialias -geometry +9000+9000 "$file"


 

3. Play Midi music in Linux text console / terminal


There is a console tool that historically has been like the Linux standard for playing midis over the years as I remember, its called timidity

 


To install timidity on .Deb based Linux:
 

linux-desktop:~$ su root
Password:
linux-desktop:~# apt-get install –yes timidity

Необходимо е да се изтеглят 0 B/580 kB архиви.
След тази операция ще бъде използвано 0 B допълнително дисково пространство.
(Reading database … 382981 files and directories currently installed.)
Preparing to unpack …/timidity_2.13.2-40.5_amd64.deb …
Unpacking timidity (2.13.2-40.5) over (2.13.2-40.5) …
Processing triggers for menu (2.1.47+b1) …
Processing triggers for man-db (2.7.6.1-2) …
Setting up timidity (2.13.2-40.5) …
Processing triggers for menu (2.1.47+b1) …

 

To test your new MIDI Synthesizer tool and make the enjoyment full you can download Doom 2 extracted MIDI Soundtrack from here
 

Once you have downloaded above Metal MIDI DOOM old school arcade soundtrack and untarred it into your home directory be it ~/doom-midis

A remark to make here is timidity is quite CPU intensive, but on modern Dual and Quad-Core PC Notebooks, the CPU load is not of a big concern.

To test and play with timidity:
 

linux-desktop~$ timidity ~/mp3/midis/*


timidity-playing-doom-midi-bunny-song-on-debian-stretch-gnome-terminal-screenshot
 

hipo@jericho:~/mp3/midis$ aplaymidi -l
 Port    Client name                      Port name
 14:0    Midi Through                     Midi Through Port-0
128:0    TiMidity                         TiMidity port 0
128:1    TiMidity                         TiMidity port 1
128:2    TiMidity                         TiMidity port 2
128:3    TiMidity                         TiMidity port 3

 


We have also the playmidi  (simple midi text console terminal player), which historically was working quite decent and I use it to in the past on my RedHat 6.0 and RedHat 7.0 to listen to my .MID format files but unfortunately as of time of writting something is wrong with it, so when I try to play MIDIs with it instead of timidity I get this erro:

 

$ playmidi *.mid
Playmidi 2.4 Copyright (C) 1994-1997 Nathan I. Laredo, AWE32 by Takashi Iwai
This is free software with ABSOLUTELY NO WARRANTY.
For details please see the file COPYING.
open /dev/sequencer: No such file or directory

Even though I tried hard to resolve that error by loading various midi related MIDI modules and following a lot of the suggestions online on how to  make /dev/sequencer work again it was all no luck.
 

Some people back in the distant year 2005, reported the problem was solved by simply loading snd-seq

But as of time of writting:

 

# modprobe snd-seq

 

Some people said in archlinux's Forum

/dev/sequencer sequencer: No such file or directory

 

is solved by loading snd-seq-oss kernel module, but on my Debian Linux 9.1 Stretch, this ain't work as well :

 

root@jericho:/home/hipo/mp3/midis# modprobe snd-seq-oss
modprobe: FATAL: Module snd-seq-oss not found in directory /lib/modules/4.9.0-3-amd64
root@jericho:/home/hipo/mp3/midis# uname -a;
Linux jericho 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux


Another invention of mine was to try to also link /dev/snd/seq to /dev/sequencer but this produced no positive result either:

 

# ln -sf /dev/snd/seq /dev/sequencer
# ls -al /dev/sequencer
lrwxrwxrwx 1 root root 12 окт  4 16:48 /dev/sequencer -> /dev/snd/seq


Note that after lining in that way I got following error with my attempt to play MIDIs with playmidi

# playmidi *.mid
Playmidi 2.4 Copyright (C) 1994-1997 Nathan I. Laredo, AWE32 by Takashi Iwai
This is free software with ABSOLUTELY NO WARRANTY.
For details please see the file COPYING.
there is no soundcard


Anyhow on some other Linux distributions (especially with Older Kernel versions), some of the above 3 suggested Fix might work perfectly fine so if you have some time give it a try please and drop me  a comment on how it went, you will help the GNU / Linux community out there that way.

Well never mind the bollocks, so

Now back to where I started timidity even though it will play fine it will not give any indication on the lenght of the midi song (precious information such as how much time is left until the end is over).

Hence if you prefer a player that gives you an indicator on how much is left towards the end length of each of the played MIDI file you can give a try to wildmidi:

 

linux-desktop:~$ apt-cache show wildmidi|grep -i description -A 2

Description-en: software MIDI player
 Minimal MIDI player implementation based on the wildmidi library that
 can either dump to WAV or playback over ALSA. It is intended to

Description-md5: b4b34070ae88e73e3289b751230cfc89
Homepage: http://www.mindwerks.net/projects/wildmidi/
Tag: implemented-in::c, role::program, sound::midi, sound::player,

Description: software MIDI player
Description-md5: 4673a7051f104675c73eb344bb045607
Homepage: http://wildmidi.sourceforge.net/
Bugs: https://bugs.launchpad.net/ubuntu/+filebug


If yet not installed install it after becoming admin user:

 

linux-desktop:~$ su root
Password:

linux-desktop:~# apt-get install –yes wildmidi


wildmidi is much less CPU intensive (it uses gstreamer to play (Gstreamer – open source multimedia framework)

And next give it a try by running:

 

linux-desktop:~$ wildmidi ~/mp3/midis/*

 

wildmidi-midi-lenght-status-text-console-player-for-linux-ubuntu-debian-fedora-suse

 

 

4. Editting MIDI files with Free Software and Proprietary MIDI Editor Programs

 


If you want a professional software that can play Midi in a fuzzy interactive GUI way and have some extra possibilities to edit MIDIs and other format give a try to Muse Sequencer:
 

 

linux-desktop:~$ sudo apt-get install –yes muse

The following NEW packages will be installed:
  muse
0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
Need to get 5814 kB of archives.
After this operation, 21.0 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 muse amd64 2.1.2-3+b1 [5814 kB]
Fetched 5814 kB in 2s (2205 kB/s)                             
    are supported and installed on your system.
Preconfiguring packages …
Selecting previously unselected package muse.
(Reading database … 382981 files and directories currently installed.)
Preparing to unpack …/muse_2.1.2-3+b1_amd64.deb …
Unpacking muse (2.1.2-3+b1) …
Processing triggers for mime-support (3.60) …
Processing triggers for desktop-file-utils (0.23-1) …
Processing triggers for doc-base (0.10.7) …
Processing 1 added doc-base file…
Registering documents with scrollkeeper…
Processing triggers for man-db (2.7.6.1-2) …
Processing triggers for shared-mime-info (1.8-1) …
Unknown media type in type 'all/all'
Unknown media type in type 'all/allfiles'
Processing triggers for gnome-menus (3.13.3-9) …
Setting up muse (2.1.2-3+b1) …
Processing triggers for hicolor-icon-theme (0.15-1) …


 

Below is short description what Muse can do for you:

 

MusE is a MIDI/audio sequencer with recording and editing capabilities.
 Some Highlights:
 

  * Standard midifile (smf) import-/export.
  * Organizes songs in tracks and parts which you can arrange with
    the part editor.
  * MIDI editors: pianoroll, drum, list, controller.
  * Score editor with high quality postscript printer output.
  * Realtime: editing while playing.
  * Unlimited number of open editors.
  * Unlimited undo/redo.
  * Realtime and step-recording.
  * Multiple MIDI devices.
  * Unlimited number of tracks.
  * Sync to external devices: MTC/MMC, Midi Clock, Master/Slave.
  * Audio tracks, LADSPA host for master effects.
  * Multithreaded.
  * Uses raw MIDI devices.
  * XML project file.
  * Project file contains complete app state (session data).
  * Application spanning Cut/Paste Drag/Drop.

 

linux-desktop~:$ muse

muse-advanced-midi-editor-free-software-for-linux

 

Below is another non-free program that you might, try if MusE doesn't fit your needs (is not rich enough for editting capabilities is bitwig (though I don't recommend since it is not free software)

bitwig – Bitwig Studio is a multi-platform music-creation system for production, performance and DJing, with a focus on flexible editing tools and a super-fast workflow.
 


bitwig-midi-and-audio-non-free-software-advanced-useful-sound-editor-for-linx


 

5. Some examples for Text editing and MIDI Conversion to CSV and ABC file formats There is pretty much more

For the MIDI Extremists who or people that create MIDIs and want to learn how a MIDI is made (the content of it etc.), I suggest you take a look at these 3 command line MIDI editing / conversion tools
 

  • midi2abc – A little tool to create MIDI formats to ABC format
  • midi2csv – Conver tour Favourite MIDI files to CSV for educational purposes so see what Channels, Tracks and Time Intervals is a MIDI song mad
  • midicopy – Copy selected, track, channel, time interval of MIDI file to another MIDI file3

 

Well, that's all folks now enjoy your MIDIs and don't forget to donate, as I'm jobless at the moment and the only profit I make is just a few bucks out of advertisement on this blog.
 

Share this on

BB – A must see ASCII Art Audio / Video portable demo for Linux, FreeBSD, UNIX and DOS

Thursday, May 24th, 2012

bb Audio Visual ASCII art Linux FreeBSD demonstration old school demo logo

I know and I have enjoyed BB – Portable Demo for already a decade.
I'm sure many newbies to the Free And Open Source (FOSS) realm don't know or heard of bb's existence as nowdays ASCII art is not so well known among youngsters. Hence this short post aims to raise some awareness of the existence of this already OLD but GOLD – awesome! text console / terminal demonstation BB 🙂

bb is pretty much in the spirit of Oldschool Assembly DOS demo scene dominating the geeks dome in the late 80's and yearly 90's.

Historically bb used to be one of the main stunning things one could show to a fellow GNU / Linux new comer.

For the year 2000, seeing all this awesome ASCII video demo running on free Operating System like GNU / Linux was a big think.
The fact that such an advanced ASCII art was distributed freely for an OS which used to exist since only (6 / 7 years) was really outstanding of its time.

BB text ascii art Linux demo entry screen characters matrix

I still remember how much I was amazed seeing a plain ascii video stream was possible only Linux. Moreover the minimal requirements of bb were quite low for its time – it worked on mostly all PCs one can find at the time.

BB's minimum requirements to work with no chops is just an old 486/66 DX2 CPU Mhz with few megas of memory (32MB of memory was more than enough to run it)

BB text sacii art Linux demo entry screen char matrix

A very unique feature of bb was it was the first Linux demo that succesfully run simultaneously playing on two monitor screens as one can read on the project website.
Unfortunately I didn't owned two monitors back in the day so never ever had the opportunity to see it running on two screens.
Anyhow I've seen it runnign somewhere on some of the Linux install fests visited some years ago…

The demo was developed by 4 man group ppl – the AA group the same digital artists are also the guys behind the AA Project.

AA Lib mascot logo :)

The main aim of AA-lib was to make possible (Doom, Second Reality, X windows) to run rendered in plain ASCII art text.

The project succeeded in a lot of his goals already as there is already existent such an ascii art ports of large games like QUAKE! Be sure to check this awesome project too AAquake ascii quake page is here
, as well as video and pictures could be viewed under a plain console Linux tty or in terminal (via SSH 🙂 )

Thanks to AA-Lib even text mode doom exists.

bb as well as aa-lib has ports for most modern Linux distros in that number one can easily get rpm or deb packages for most of distros.
On Slackware Linux you should compile it from source. Though compilation should be a straightfoward process, not that i tried it myself but I remember a close friend of mine (a great Slackware devotee) who was the one to show me the demo for a first time on his Slackware box.

1. Installing bb on Debian Linux

Debian Linux users like me are privileged as for already many years a Debian package of bb is maintaned thanks to Uwe Herman

Hence for anyone willing to enjoy bb install it by running:

debian:~# apt-get --yes install bb
....
ho@debian:~$ bb

If you're running a X server the aa-lib will immediately run with its X server compiled support:

Running BB Music Screesnhot

2. Installing BB demo on FreeBSD

On FreeBSD, bb demo has a port to install it run:

freebsd# cd /usr/ports/misc/bb freebsd# make install clean ...

Here is good time to say that even though in most of the machines, I've tested the demo I had on some of the hosts problems with sound due to buggy sound drivers.
As of time of writting hopefully on most machines there will be no troubles as most of the Linux sb drivers are better supported by ALSA.

Everyone interested in both Free Software and ASCII art knows well how big in significance is the AA-lib project for the historical development and attraction for new hackers to the Linux dome.
In that sense AAlib head developer Jan HubickaBy the way Jan Hubicka is also the author of another Linux tool called xaos. Xaos is a tool to deal with some kind of advanced higher mathematics stuff called fractals.

XAOS Screenshot Debian Squeeze Linux

Unfortunately I don't know a bit for fractal maths and what the purpose of the tool is but as you can see on the shot it looks nice running 🙂

Here are also, lot of the major BB parts in shots:

Running bb music screen screenshot Linux Debian 6 Squeeze

BB AScii fire Linux shot

bb demo ascii art fractals

BB demo ascii art back head and description of the dev

bb demo ascii zebra Linux screenshot

bb demo cannon gun shot

BB demo ring screenshot

BB demo spots Debian shot

BB developer head shot 2

BB developer profile shot

bb game ascii invaders demo

Linux extremist BB demo

BB demo zoomed text ascii art text

BB Demo thanks for watching screen

For those on MS-Windows OS platform, here is the demo 🙂

BB ASCII Demo standard size running in Linux (With sound)

Enjoy ! 🙂

Share this on

How to convert file content encoded in windows-cp1251 charset to UTF-8 (with iconv) to be delivered properly encoded to browsing end clients

Wednesday, May 16th, 2012

windows-cp1251 bulgarian to UTF-8 / Encoding Communication Decoding Communication Funny Picture

I have a bunch of old html files all encoded in the historically obsolete Windows-cp1251. Windows-CP1251 used to be common used 7 years ago and therefore still big portions of the web content in Bulgarian / Russian Cyrillic is still transferred to the end users in this encoding.

This was just before the "UTF-8 revolution", where massively people started using UTF-8,
Well it was clear the specific national country text encoding standards will quickly be moved by to UTF-8 – Universal Encoding format which abbreviation stands for (Unicode Transformation Format).

Though UTF-8 was clear to be "the future", many web developers mostly because of their incompetency or using an old sources of learning how to writen in HTML continued to use windows-cp1251 in HTMLs. I'm even convinced, there are still developers out there who are writting websites for Bulgarian / Russian / Macedonian customers using obsolete encodings …

The smarter developers of those accustomed to windows-cp1251, KOI-8R etc. etc., were using the meta tag to specify the type of charset of the web page content with:

<meta http-equiv="content-type" content="text/html;charset=windows-cp1251">

or

<meta http-equiv="content-type" content="text/html;charset=koi-8r">

Anyhow, still many devs even didn't placed the windows-cp1251 in the head of the HTML …

The result for the system administrator is always a mess – a lot of webpages that are showing like unreadable signs and tons of unhappy customers.
As always the system administrator is considered responsible, for the programmer mistakes :). So instead of programmers fix their bad cooking, the admin has to fix it all!

One quick work around me as admin has applied to failing to display pages in Cyrillic using the Windows-cp1251 character encoding was to force windows-cp1251 as a default encoding for the whole virtualhost or Apache directory with Apache directives like:

<VirtualHost *:80>
ServerAdmin some_user@some_host.com
DocumentRoot /var/www/html
AddDefaultCharset windows-cp1251
ServerName the_host_name.com
ServerAlias www.the_host_name.com
....
....
<Directory>
AddDefaultCharset windows-cp1251
>/Directory>
</VirtualHost>

Though this mostly would, work there are some occasions, where only a particular html files from all the content served by Apache is encoded in windows-cp1251, if most of the content is already written in UTF-8, this could be a big issues as you cannot just change the UTF-8 globally to windows-cp1251, just because few pages are written in archaic encoding….
Since most of the content is displayed to the client by Apache (as prior explained) just fine, only particular htmls lets's ay single.html, single2.html etc. etc. are displayed with some question marks or some non-human readable "hieroglyphs".

Below is a screenshot from two pages returned to my browser in wrongly set htmls charset:

Improper Windows CP1251 encoding with Apache set to serve UTF-8 encoding questiomarks

Improper Windows CP1251 delivered page in UTF-8 browser view

Apache returns cp1251 in some non-UTF8 wrong encoding (webserver improperly served cyrillic encoding)

Improperly served encoding CP1251 delivered by Apache in non-utf-8 encoding

When this kind of issues occur, the only solution is to simply login to the server and use iconv command to convert all files returning unreadable content from whatever the non UTF-8 encoding is lets say in my case Bulgarian typeset of cp1251 to UTF-8

Here is how the iconv command to convert between windows-cp1251 to utf-8 the two sample files named single1.html and single2.html

server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single1.html > single1.html.utf8
server:/web# mv single1.html single1.html.bak;
server:/web# mv single1.html.utf8 single1.html
server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single2.html > single2.html.utf8
server:/web# mv single2.html single2.html.bak;
server:/web# mv single2.html.utf8 single2.html

I always, make copies of the original cp1251 encoded files (as you see mv single1.html single1.html.bak), because if something goes wrong with convertion I can easily revert back.

If there are 10 files with consequential numbers naming they can be converted using a short for loop, like so:

server:/web# for i $(seq 1 10); do
/usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single$i.html > single$i.html.utf8;mv single$i.html single$i.html.bak
mv single$i.html.utf8 single$i.html
done

Just as earlier mentioned if single1.html, single2.html … has in the html <head>:

<meta http-equiv="Content-Type" content="text/html; charset=windows-1251">

You should open, each of the files in question and wipe out the line either by hand or use sed to wipe it in one loop if it has to be done for lets say 10 files named (single{1..10})

server:/web# for i in $(seq 1 10); do
sed '/<meta http-equiv="Content-Type" content="text\/html; charset=windows-1251>/d' single$i.txt > single$i.txt.new;
mv single$i.txt single$i.txt.bak;
mv single$i.txt.new single$i.txt

Well now,

Share this on

Auto restart Apache on High server load (bash shell script) – Fixing Apache server temporal overload issues

Saturday, March 24th, 2012

auto-restart-apache-on-high-load-bash-shell-script-fixing-apache-temporal-overload-issues

I've written a tiny script to check and restart, Apache if the server encounters, extremely high load avarage like for instance more than (>25). Below is an example of a server reaching a very high load avarage:;

server~:# uptime
13:46:59 up 2 days, 18:54, 1 user, load average: 58.09, 59.08, 60.05
load average: 0.09, 0.08, 0.08

Sometimes high load avarage is not a problem, as the server might have a very powerful hardware. A high load numbers is not always an indicator for a serious problems. Some 16 CPU dual core (2.18 Ghz) machine with 16GB of ram could probably work normally with a high load avarage like in the example. Anyhow as most servers are not so powerful having such a high load avarage, makes the machine hardly do its job routine.

In my specific, case one of our Debian Linux servers is periodically reaching to a very high load level numbers. When this happens the Apache webserver is often incapable to serve its incoming requests and starts lagging for clients. The only work-around is to stop the Apache server for a couple of seconds (10 or 20 seconds) and then start it again once the load avarage has dropped to less than "3".

If this temporary fix is not applied on time, the server load gets increased exponentially until all the server services (ssh, ftp … whatever) stop responding normally to requests and the server completely hangs …

Often this server overloads, are occuring at night time so I'm not logged in on the server and one such unexpected overload makes the server unreachable for hours.
To get around the sudden high periodic load avarage server increase, I've written a tiny bash script to monitor, the server load avarage and initiate an Apache server stop and start with a few seconds delay in between.

#!/bin/sh
# script to check server for extremely high load and restart Apache if the condition is matched
check=`cat /proc/loadavg | sed 's/\./ /' | awk '{print $1}'`
# define max load avarage when script is triggered
max_load='25'
# log file
high_load_log='/var/log/apache_high_load_restart.log';
# location of inidex.php to overwrite with temporary message
index_php_loc='/home/site/www/index.php';
# location to Apache init script
apache_init='/etc/init.d/apache2';
#
site_maintenance_msg="Site Maintenance in progress - We will be back online in a minute";
if [ $check -gt "$max_load" ]; then>
#25 is load average on 5 minutes
cp -rpf $index_php_loc $index_php_loc.bak_ap
echo "$site_maintenance_msg" > $index_php_loc
sleep 15;
if [ $check -gt "$max_load" ]; then
$apache_init stop
sleep 5;
$apache_init restart
echo "$(date) : Apache Restart due to excessive load | $check |" >> $high_load_log;
cp -rpf $index_php_loc.bak_ap $index_php_loc
fi
fi

The idea of the script is partially based on a forum thread – Auto Restart Apache on High Loadhttp://www.webhostingtalk.com/showthread.php?t=971304Here is a link to my restart_apache_on_high_load.sh script

The script is written in a way that it makes two "if" condition check ups, to assure 100% there is a constant high load avarage and not just a temporal 5 seconds load avarage jump. Once the first if is matched, the script first tries to reduce the server load by overwritting a the index.php, index.html script of the website with a one stating the server is ongoing a maintenance operations.
Temporary stopping the index page, often reduces the load in 10 seconds of time, so the second if case is not necessery at all. Sometimes, however this first "if" condition cannot decrease enough the load and the server load continues to stay too high, then the script second if comes to play and makes apache to be completely stopped via Apache init script do 2 secs delay and launch the apache server again.

The script also logs about, the load avarage encountered, while the server was overloaded and Apache webserver was restarted, so later I can check what time the server overload occured.
To make the script periodically run, I've scheduled the script to launch every 5 minutes as a cron job with the following cron:

# restart Apache if load is higher than 25
*/5 * * * * /usr/sbin/restart_apache_on_high_load.sh >/dev/null 2>&1

I have also another system which is running FreeBSD 7_2, which is having the same overload server problems as with the Linux host.
Copying the auto restart apache on high load script on FreeBSD didn't work out of the box. So I rewrote a little chunk of the script to make it running on the FreeBSD host. Hence, if you would like to auto restart Apache or any other service on FreeBSD server get /usr/sbin/restart_apache_on_high_load_freebsd.sh my script and set it on cron on your BSD.

This script is just a temporary work around, however as its obvious that the frequency of the high overload will be rising with time and we will need to buy new server hardware to solve permanently the issues, anyways, until this happens the script does a great job 🙂

I'm aware there is also alternative way to auto restart Apache webserver on high server loads through using monit utility for monitoring services on a Unix system. However as I didn't wanted to bother to run extra services in the background I decided to rather use the up presented script.

Interesting info to know is Apache module mod_overload exists – which can be used for checking load average. Using this module once load avarage is over a certain number apache can stop in its preforked processes current serving request, I've never tested it myself so I don't know how usable it is. As of time of writting it is in early stage version 0.2.2
If someone, have tried it and is happy with it on a busy hosting servers, please share with me if it is stable enough?

Share this on