Posts Tagged ‘format’

Play Midis on Linux / Make Linux MIDI Ready for the Future – Enable embedded MIDI music to play in a Browser, Play MIDIs with VLC and howto enjoy Midis in Text Console

Wednesday, October 4th, 2017

how-to-play-midi-on-gnu-linux-in-graphic-environment-console-and-browser-midi-synthesizer-and-linux-tux-together

 

Play Midis on Linux or Make Linux MIDI Ready for the Future – Enable embedded MIDI music to play in a Browser, Play MIDIs with VLC and howto enjoy Midis in Text Console HOWTO

 

Playing MIDI has been quite a lot of fun historically,

if you grow up in the days when personal computers were still young and the Sound Blaster was a luxury, before the raise of Mp3 music format, you have certainly enjoyed the beeping of PC Speaker and later on during 386 and 486 / 586 computers the enjoyment of playing tracked music such as S3M and MOD,

in that good days playing MIDI music was the only alternative for PC maniacs who doesn't own a CD Drive (which itself) was another luxury and even thouse who had a CD ROM device, were mainly playing music in CD audio format (.CDA).
Anyhow MIDI was a cheap and a CPU unintensive way to listen to equivalent of favourite popular Audio Songs and for those who still remember many of the songs were recreated in MIDI format, just with a number of synthesized instruments without any voice (as MIDI is usually).

The same was true also for the good old days of raise of Mobile Phones, when polyphonic was a standard as CPU power was low MIDI was a perfect substitute for the CPU heavy Encoded MP3s / OGG and other formats that required a modern for that time Intel CPU running in 50+ Mhz usually 100 / 166Mhz was perfect for the days to play Mp3 but still even on that PCs we listened to Midi songs.

Therefore if you're one of those people like me who still enjoy to play some Midi Music in the year 2017 and feel a bit like Back into the Future movie and a Free Software fan and user, especially if you're a novice GNU  / Linux Free Software user, you will be unpleasently surprised that most today's default Linux distributions doesn't have an easy way to play Midi music format out of the box right after install.

Hence below article aims to give you an understanding on

How you can play Midi Music on GNU / Linux Operating System

First, lets Prepare to load necessery Linux kernel modules to make sure MIDI can be played by soundcard:

In /etc/modules make sure you have the following list of modules loaded:
 

linux-desktop:~# cat /etc/modules
3c59x
snd-emu10k1
snd-pcm-oss
snd-mixer-oss
snd-seq-oss

!Note the modules are working as of time of writting and in time can change to some other modules, depending on how the development of ALSA (Advanced Linux Sound Architecture) goes, and if the developers decide to rename the upmentioned modules

If you just have added the modules to /etc/modules with vim / nano to reload modules into the Linux kernel run:

 

linux-desktop:~# modprobe -a


Secondly, Installing a whole bunch of MIDI music related program tools can be achieved in Debian by installing the multimedia-midi package, e.g.:

 

linux-desktop:~# apt-get install –yes multimedia-midi

 

1. Playing Midi in Graphical environment with a double click using VLC


How to make MIDI easy listanable in Linux graphical environment like GNOME / KDE / XFCE desktop ?

 

If you want to make Midi music execution sa easy as  just clicking on the .MIDI file format on Linux you can do that with a midi extension available for VLC (Video Lan Client) Universal Multi Platform Media Player player

To install it on Debian Ubuntu GNU / Linux
 

# apt-get install –yes vlc-plugin-fluidsynth

 

Необходимо е да се изтеглят 6754 B архиви.
След тази операция ще бъде използвано 35,8 kB допълнително дисково пространство.
Изт:1 http://deb.debian.org/debian stretch/main amd64 vlc-plugin-fluidsynth amd64 2.2.6-1~deb9u1 [6754 B]
Изтеглени 6754 B за 0с (33,6 kB/сек)           
Selecting previously unselected package vlc-plugin-fluidsynth:amd64.
(Reading database … 382976 files and directories currently installed.)
Preparing to unpack …/vlc-plugin-fluidsynth_2.2.6-1~deb9u1_amd64.deb …
Unpacking vlc-plugin-fluidsynth:amd64 (2.2.6-1~deb9u1) …
Setting up vlc-plugin-fluidsynth:amd64 (2.2.6-1~deb9u1) …
Processing triggers for libvlc-bin:amd64 (2.2.6-1~deb9u1) …


Besides making your MIDI play on the GUI environment easy as a a point and click VLC will also be able to play MIDIs on GNU / Linux from your favourite browser (nomatter Firefox / Chrome or Opera), even though the player would play in a new PopUP Window it is easy to select once MIDI file from a random website for example – here is a directory listing of Webserver with Doom II Soundtrack in MIDI format , click over any file from list and Choose option for VLC to always remember that MIDI files has to be opened with VLC player.
 


2. Enable Firefox / IceWeasel browser to Support Website embedded MIDI files

 

 

So VLC could make you listen the downloadable MIDIs from Web pages but,
 

What if you have stumbled on an old website which was configured with very OLD HTML Code to play some nice music (or even different MIDI songs) for each part of the website (for each webpage) and you want to have the Websites created with embedded MIDIs to automatically play on Linux oncce you visit the site?


Sadly default support in Browser for MIDI across all GNU / Linux, I've used so far never worked out of the box, not that still anyone is developing modern websites with MIDIs, but still for the sake of backward compitability and for sake of interactivity it is worthy to enable embedded MIDI support in Linux

But with a couple of tunings as usual GNU / Linux can do almost everything, so here is how to enable embedded browser support for Midi on Linux (That should work with minor modifications not only on Debian / Ubuntu / ArchLinux but also on Fedoras, CentOS etc.
If you try it on any of this distributions, please drop a short comment and tell me in few lines how you made embedded midi worked on that distros.

 

apt-get install –yes timidity mozplugger

Next do restart firefox

Sometimes in order to work you might need to delete /home/[YOUR_USERNAME]/.mozilla/pluginreg.dat and restart firefox again, e.g. make a backup and give it a try:

 

cp -rpf /home/hipo/.mozilla/pluginreg.dat /home/hipo/.mozilla/pluginreg.dat.bak
rm -f /home/hipo/.mozilla/pluginreg.dat

 

Another good tip as talking for embedding MIDI support is to embed XPDF to render PDF pages inside the Browser, by default this is done by GNOME's Evince PDF reader but as it is sometimes buggy and might crash it is generally a good idea to switch to xpdf instead, if for some reason PDF is not directly displaying in browser or suddenly stopped working after some distro uipgrade, you might want to do below as well:
 

apt-get install xpdf

vim /etc/mozpluggerrc

Fin d and Comment out the line starting with:

It should look like this afterwards:

 Repeat Swallow ….
 

text/x-pdf: pdf: PDF file
#      repeat swallow(documentShell) fill: acroread -geometry +9000+9000 +useFrontEndProgram "$file"
        repeat noisy swallow(Xpdf) fill: xpdf -g +9000+9000 "$file"
        repeat noisy swallow(gv) fill: gv –safer –quiet –antialias -geometry +9000+9000 "$file"


 

3. Play Midi music in Linux text console / terminal


There is a console tool that historically has been like the Linux standard for playing midis over the years as I remember, its called timidity

 


To install timidity on .Deb based Linux:
 

linux-desktop:~$ su root
Password:
linux-desktop:~# apt-get install –yes timidity

Необходимо е да се изтеглят 0 B/580 kB архиви.
След тази операция ще бъде използвано 0 B допълнително дисково пространство.
(Reading database … 382981 files and directories currently installed.)
Preparing to unpack …/timidity_2.13.2-40.5_amd64.deb …
Unpacking timidity (2.13.2-40.5) over (2.13.2-40.5) …
Processing triggers for menu (2.1.47+b1) …
Processing triggers for man-db (2.7.6.1-2) …
Setting up timidity (2.13.2-40.5) …
Processing triggers for menu (2.1.47+b1) …

 

To test your new MIDI Synthesizer tool and make the enjoyment full you can download Doom 2 extracted MIDI Soundtrack from here
 

Once you have downloaded above Metal MIDI DOOM old school arcade soundtrack and untarred it into your home directory be it ~/doom-midis

A remark to make here is timidity is quite CPU intensive, but on modern Dual and Quad-Core PC Notebooks, the CPU load is not of a big concern.

To test and play with timidity:
 

linux-desktop~$ timidity ~/mp3/midis/*


timidity-playing-doom-midi-bunny-song-on-debian-stretch-gnome-terminal-screenshot
 

hipo@jericho:~/mp3/midis$ aplaymidi -l
 Port    Client name                      Port name
 14:0    Midi Through                     Midi Through Port-0
128:0    TiMidity                         TiMidity port 0
128:1    TiMidity                         TiMidity port 1
128:2    TiMidity                         TiMidity port 2
128:3    TiMidity                         TiMidity port 3

 


We have also the playmidi  (simple midi text console terminal player), which historically was working quite decent and I use it to in the past on my RedHat 6.0 and RedHat 7.0 to listen to my .MID format files but unfortunately as of time of writting something is wrong with it, so when I try to play MIDIs with it instead of timidity I get this erro:

 

$ playmidi *.mid
Playmidi 2.4 Copyright (C) 1994-1997 Nathan I. Laredo, AWE32 by Takashi Iwai
This is free software with ABSOLUTELY NO WARRANTY.
For details please see the file COPYING.
open /dev/sequencer: No such file or directory

Even though I tried hard to resolve that error by loading various midi related MIDI modules and following a lot of the suggestions online on how to  make /dev/sequencer work again it was all no luck.
 

Some people back in the distant year 2005, reported the problem was solved by simply loading snd-seq

But as of time of writting:

 

# modprobe snd-seq

 

Some people said in archlinux's Forum

/dev/sequencer sequencer: No such file or directory

 

is solved by loading snd-seq-oss kernel module, but on my Debian Linux 9.1 Stretch, this ain't work as well :

 

root@jericho:/home/hipo/mp3/midis# modprobe snd-seq-oss
modprobe: FATAL: Module snd-seq-oss not found in directory /lib/modules/4.9.0-3-amd64
root@jericho:/home/hipo/mp3/midis# uname -a;
Linux jericho 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux


Another invention of mine was to try to also link /dev/snd/seq to /dev/sequencer but this produced no positive result either:

 

# ln -sf /dev/snd/seq /dev/sequencer
# ls -al /dev/sequencer
lrwxrwxrwx 1 root root 12 окт  4 16:48 /dev/sequencer -> /dev/snd/seq


Note that after lining in that way I got following error with my attempt to play MIDIs with playmidi

# playmidi *.mid
Playmidi 2.4 Copyright (C) 1994-1997 Nathan I. Laredo, AWE32 by Takashi Iwai
This is free software with ABSOLUTELY NO WARRANTY.
For details please see the file COPYING.
there is no soundcard


Anyhow on some other Linux distributions (especially with Older Kernel versions), some of the above 3 suggested Fix might work perfectly fine so if you have some time give it a try please and drop me  a comment on how it went, you will help the GNU / Linux community out there that way.

Well never mind the bollocks, so

Now back to where I started timidity even though it will play fine it will not give any indication on the lenght of the midi song (precious information such as how much time is left until the end is over).

Hence if you prefer a player that gives you an indicator on how much is left towards the end length of each of the played MIDI file you can give a try to wildmidi:

 

linux-desktop:~$ apt-cache show wildmidi|grep -i description -A 2

Description-en: software MIDI player
 Minimal MIDI player implementation based on the wildmidi library that
 can either dump to WAV or playback over ALSA. It is intended to

Description-md5: b4b34070ae88e73e3289b751230cfc89
Homepage: http://www.mindwerks.net/projects/wildmidi/
Tag: implemented-in::c, role::program, sound::midi, sound::player,

Description: software MIDI player
Description-md5: 4673a7051f104675c73eb344bb045607
Homepage: http://wildmidi.sourceforge.net/
Bugs: https://bugs.launchpad.net/ubuntu/+filebug


If yet not installed install it after becoming admin user:

 

linux-desktop:~$ su root
Password:

linux-desktop:~# apt-get install –yes wildmidi


wildmidi is much less CPU intensive (it uses gstreamer to play (Gstreamer – open source multimedia framework)

And next give it a try by running:

 

linux-desktop:~$ wildmidi ~/mp3/midis/*

 

wildmidi-midi-lenght-status-text-console-player-for-linux-ubuntu-debian-fedora-suse

 

 

4. Editting MIDI files with Free Software and Proprietary MIDI Editor Programs

 


If you want a professional software that can play Midi in a fuzzy interactive GUI way and have some extra possibilities to edit MIDIs and other format give a try to Muse Sequencer:
 

 

linux-desktop:~$ sudo apt-get install –yes muse

The following NEW packages will be installed:
  muse
0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
Need to get 5814 kB of archives.
After this operation, 21.0 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 muse amd64 2.1.2-3+b1 [5814 kB]
Fetched 5814 kB in 2s (2205 kB/s)                             
    are supported and installed on your system.
Preconfiguring packages …
Selecting previously unselected package muse.
(Reading database … 382981 files and directories currently installed.)
Preparing to unpack …/muse_2.1.2-3+b1_amd64.deb …
Unpacking muse (2.1.2-3+b1) …
Processing triggers for mime-support (3.60) …
Processing triggers for desktop-file-utils (0.23-1) …
Processing triggers for doc-base (0.10.7) …
Processing 1 added doc-base file…
Registering documents with scrollkeeper…
Processing triggers for man-db (2.7.6.1-2) …
Processing triggers for shared-mime-info (1.8-1) …
Unknown media type in type 'all/all'
Unknown media type in type 'all/allfiles'
Processing triggers for gnome-menus (3.13.3-9) …
Setting up muse (2.1.2-3+b1) …
Processing triggers for hicolor-icon-theme (0.15-1) …


 

Below is short description what Muse can do for you:

 

MusE is a MIDI/audio sequencer with recording and editing capabilities.
 Some Highlights:
 

  * Standard midifile (smf) import-/export.
  * Organizes songs in tracks and parts which you can arrange with
    the part editor.
  * MIDI editors: pianoroll, drum, list, controller.
  * Score editor with high quality postscript printer output.
  * Realtime: editing while playing.
  * Unlimited number of open editors.
  * Unlimited undo/redo.
  * Realtime and step-recording.
  * Multiple MIDI devices.
  * Unlimited number of tracks.
  * Sync to external devices: MTC/MMC, Midi Clock, Master/Slave.
  * Audio tracks, LADSPA host for master effects.
  * Multithreaded.
  * Uses raw MIDI devices.
  * XML project file.
  * Project file contains complete app state (session data).
  * Application spanning Cut/Paste Drag/Drop.

 

linux-desktop~:$ muse

muse-advanced-midi-editor-free-software-for-linux

 

Below is another non-free program that you might, try if MusE doesn't fit your needs (is not rich enough for editting capabilities is bitwig (though I don't recommend since it is not free software)

bitwig – Bitwig Studio is a multi-platform music-creation system for production, performance and DJing, with a focus on flexible editing tools and a super-fast workflow.
 


bitwig-midi-and-audio-non-free-software-advanced-useful-sound-editor-for-linx


 

5. Some examples for Text editing and MIDI Conversion to CSV and ABC file formats There is pretty much more

For the MIDI Extremists who or people that create MIDIs and want to learn how a MIDI is made (the content of it etc.), I suggest you take a look at these 3 command line MIDI editing / conversion tools
 

  • midi2abc – A little tool to create MIDI formats to ABC format
  • midi2csv – Conver tour Favourite MIDI files to CSV for educational purposes so see what Channels, Tracks and Time Intervals is a MIDI song mad
  • midicopy – Copy selected, track, channel, time interval of MIDI file to another MIDI file3

 

Well, that's all folks now enjoy your MIDIs and don't forget to donate, as I'm jobless at the moment and the only profit I make is just a few bucks out of advertisement on this blog.
 

How to convert OGG Vorbis .ogg to MP3 on GNU / Linux and FreeBSD

Friday, July 27th, 2012

I’ve used K3B just recently to RIP an Audio CD with music to MP3. K3b has done a great job ripping the tracks, the only problem was By default k3b RIPs songs in OGG Vorbis (.ogg) and not mp3. I personally prefer OGG Vorbis as it is a free freedom respecting audio format, however the problem was the .ogg-s cannot be read on many of the audio players and it could be a problem reading the RIPped oggs on Windows. I’ve done the RIP not for myself but for a Belarusian gfriend of mine and she is completely computer illiterate and if I pass her the songs in .OGG, there is no chance she succed in listening the oggs. I’ve seen later k3b has an option to choose to convert directly to MP3 Using linux mp3 lame library this however is time consuming and I have to wait another 10 minutes or so for the songs to be ripped to shorten the time I decided to directly convert the existing .ogg files to .mp3 on my (Debian Linux). There are probably many ways to convert .ogg to mp3 on linux and likely many GUI frontends (like SoundConverter) to use in graphic env.

SoundConverter Debian GNU Linux graphic GUI environment program for convertion of ogg to mp3 and mp3 to ogg, convert multiple sound formats on GNU / Linux.

I however am a console freak so I preferred doing it from terminal. I’ve done quick research on the net and figured out the good old ffmpeg is capable of converting .oggs to .mp3s. To convert all mp3s just ripped in the separate directory I had to run ffmpeg in a tiny bash loop.

A short bash shell script 1 liner combined with ffmpeg does it, e.g.;

for f in *.ogg; do ffmpeg -i "$f" "`basename "$f" .ogg`.mp3"; done.....

The loop example is in bash so in order to make the code work on FreeBSD it is necessery it is run in a bash shell and not in BSDs so common csh or tcsh.

Well, that’s all oggs are in mp3; Hip-hip Hooray 😉

How to add support for DJVU file format on M$ Windows, Mac, GNU / Linux and FreeBSD

Thursday, June 14th, 2012

Windjview Format paper Clipper logo / support DjView on Windows and Linux

By default there is no way to see what is inside a DJVU formatted document on both Windows and Linux OS platforms. It was just a few months ago I saw on one computer I had to fix up the DJVU format. DJVU format was developed for storing primary scanned documents which is rich in text and drawings.
Many old and ancient documents for example Church books in latin and some older stuff is only to be found online in DJVU format.
The main advantage of DJVU over lets say PDF which is also good for storing text and visual data is that DJVU's data encoding makes the files much more smaller in size, while still the quality of the scanned document is well readable for human eye.

DJVU is a file format alternative to PDF which we all know has been set itself to be one of the major standard formats for distributing electronic documents.

Besides old books there are plenty of old magazines, rare reports, tech reports newspapers from 1st and 2nd World War etc in DJVU.
A typical DJVU document takes a size of only lets say 50 to 100 KBytes of size just for comparison most a typical PDF encoded document is approximately sized 500 KiloBytes.

1.% Reading DJVU's on M$ Windoze and Mac-s (WinDjView)

The program reader for DJVU files in Windows is WinDjView WinDjView official download site is here

WinDjView is licensed under GPLv2 is a free software licensed under GPL2.

WinDjView works fine on all popular Windows versions (7, Vista, 2003, XP, 2000, ME, 98, NT4).

WinDjView with opened old document Sol manual ,,,,

I've made a mirror copy of WinDjView for download here (just in case something happens with the present release and someone needs it in future).

For Mac users there is also a port of WinDjView called MacDjView ;;;,

2.% Reading DJVU files on GNU / Linux

The library capable of rendering DJVUs in both Linux and Windows is djviewlibre again free software (A small note to make here is WinDjView also uses djviewlibre to render DJVU file content).

The program that is capable of viewing DJVU files in Linux is called djview4 I have so far tested it only with Debian GNU / Linux.

To add support to a desktop Debian GNU / Linux rel. (6.0.2) Squeeze, had to install following debs ;;;

debian:~# apt-get install --yes djview4 djvulibre-bin djviewlibre-desktop libdjviewlibre-text pdf2djvu
........
...

pdf2djvu is not really necessery to install but I installed it since I think it is a good idea to have a PDF to DJVU converter on the system in case I somedays need it ;;;

djview4 is based on KDE's QT library, so unfortunately users like me who use GNOME for a desktop environment will have the QT library installed as a requirement of above apt-get ;;;

Here is Djview4 screenshot with one opened old times Bulgarian magazine called Computer – for you

DJVU Pravetz Computer for you old school Bulgarian Pravetz magazine

Though the magazine opens fine, every now and then I got some spit errors whether scrolling the pages, but it could be due to improperly encoded DJVU file and not due to the reader. Pitily, whether I tried to maximize the PDF and read it in fullscreen I got (segfault) error and the program failed. Anyways at least I can read the magazine in non-fullscreen mode ;;; ,,,,

3.% Reading DJVU's on FreeBSD and (other BSDs)

Desktop FreeBSD users and other BSD OS enthusiasts could also use djview4 to view DJVUs as there is a BSD port in the ports tree.
To use it on BSD I had to install port /usr/ports/graphics/djview4:

freebsd# cd /usr/ports/graphics/djview4
freebsd# make install clean
,,,,...

For G / Linux users who has to do stuff with DJVU files, there are two other programs which might be useful:
 

  • a) djvusmooth – graphical editor for DjVu
  • b) gscan2pdf – A GUI to produce PDFs or DjVus from scanned documents

DJVUSmooth Debian GNU / Linux opened prog

I tried djusmooth to edit the same PDF magazine which I prior opened but I got an Unhandled exception: IOError, as you can in below shot:

DJVUSmooth Unhandled Exception IOError

This is probably normal since djvusmooth is in its very early stage of development – current version is 0.2.7-1

Unfortunately I don't have a scanner at home so I can't test if gscan2pdf produces proper DJVUs from scans, anyways I installed it to at least check the program interface which on a first glimpse looks simplistic:

gscan2pdf 0.9.31 Debian Linux Squeeze screenshot
To sum it up obviously DJVU seems like a great alternative to PDF, however its support for Free Software OSes is still lacking behind.
The Current windows DJVU works way better, though hopefully this will change soon.

How to convert file content encoded in windows-cp1251 charset to UTF-8 (with iconv) to be delivered properly encoded to browsing end clients

Wednesday, May 16th, 2012

windows-cp1251 bulgarian to UTF-8 / Encoding Communication Decoding Communication Funny Picture

I have a bunch of old html files all encoded in the historically obsolete Windows-cp1251. Windows-CP1251 used to be common used 7 years ago and therefore still big portions of the web content in Bulgarian / Russian Cyrillic is still transferred to the end users in this encoding.

This was just before the "UTF-8 revolution", where massively people started using UTF-8,
Well it was clear the specific national country text encoding standards will quickly be moved by to UTF-8 – Universal Encoding format which abbreviation stands for (Unicode Transformation Format).

Though UTF-8 was clear to be "the future", many web developers mostly because of their incompetency or using an old sources of learning how to writen in HTML continued to use windows-cp1251 in HTMLs. I'm even convinced, there are still developers out there who are writting websites for Bulgarian / Russian / Macedonian customers using obsolete encodings …

The smarter developers of those accustomed to windows-cp1251, KOI-8R etc. etc., were using the meta tag to specify the type of charset of the web page content with:

<meta http-equiv="content-type" content="text/html;charset=windows-cp1251">

or

<meta http-equiv="content-type" content="text/html;charset=koi-8r">

Anyhow, still many devs even didn't placed the windows-cp1251 in the head of the HTML …

The result for the system administrator is always a mess – a lot of webpages that are showing like unreadable signs and tons of unhappy customers.
As always the system administrator is considered responsible, for the programmer mistakes :). So instead of programmers fix their bad cooking, the admin has to fix it all!

One quick work around me as admin has applied to failing to display pages in Cyrillic using the Windows-cp1251 character encoding was to force windows-cp1251 as a default encoding for the whole virtualhost or Apache directory with Apache directives like:

<VirtualHost *:80>
ServerAdmin some_user@some_host.com
DocumentRoot /var/www/html
AddDefaultCharset windows-cp1251
ServerName the_host_name.com
ServerAlias www.the_host_name.com
....
....
<Directory>
AddDefaultCharset windows-cp1251
>/Directory>
</VirtualHost>

Though this mostly would, work there are some occasions, where only a particular html files from all the content served by Apache is encoded in windows-cp1251, if most of the content is already written in UTF-8, this could be a big issues as you cannot just change the UTF-8 globally to windows-cp1251, just because few pages are written in archaic encoding….
Since most of the content is displayed to the client by Apache (as prior explained) just fine, only particular htmls lets's ay single.html, single2.html etc. etc. are displayed with some question marks or some non-human readable "hieroglyphs".

Below is a screenshot from two pages returned to my browser in wrongly set htmls charset:

Improper Windows CP1251 encoding with Apache set to serve UTF-8 encoding questiomarks

Improper Windows CP1251 delivered page in UTF-8 browser view

Apache returns cp1251 in some non-UTF8 wrong encoding (webserver improperly served cyrillic encoding)

Improperly served encoding CP1251 delivered by Apache in non-utf-8 encoding

When this kind of issues occur, the only solution is to simply login to the server and use iconv command to convert all files returning unreadable content from whatever the non UTF-8 encoding is lets say in my case Bulgarian typeset of cp1251 to UTF-8

Here is how the iconv command to convert between windows-cp1251 to utf-8 the two sample files named single1.html and single2.html

server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single1.html > single1.html.utf8
server:/web# mv single1.html single1.html.bak;
server:/web# mv single1.html.utf8 single1.html
server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single2.html > single2.html.utf8
server:/web# mv single2.html single2.html.bak;
server:/web# mv single2.html.utf8 single2.html

I always, make copies of the original cp1251 encoded files (as you see mv single1.html single1.html.bak), because if something goes wrong with convertion I can easily revert back.

If there are 10 files with consequential numbers naming they can be converted using a short for loop, like so:

server:/web# for i $(seq 1 10); do
/usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single$i.html > single$i.html.utf8;mv single$i.html single$i.html.bak
mv single$i.html.utf8 single$i.html
done

Just as earlier mentioned if single1.html, single2.html … has in the html <head>:

<meta http-equiv="Content-Type" content="text/html; charset=windows-1251">

You should open, each of the files in question and wipe out the line either by hand or use sed to wipe it in one loop if it has to be done for lets say 10 files named (single{1..10})

server:/web# for i in $(seq 1 10); do
sed '/<meta http-equiv="Content-Type" content="text\/html; charset=windows-1251>/d' single$i.txt > single$i.txt.new;
mv single$i.txt single$i.txt.bak;
mv single$i.txt.new single$i.txt

Well now,

How to convert FLV to AVI and AVI to FLV Videos on Linux and BSD with avidemux and ffmpeg – Simple video editting with LiVES

Tuesday, May 8th, 2012

I'm starting to learn some video editing, as I need it sometimes for building client websites.
As a Linux user I needed to have some kind of software for amateur video editing.
For Microsoft Windows OS, there are tons of video editor programs both free and proprietary (paid).
Windows users can for instance use the free software program VirtualDub (licensed under GPL license) to easily cut movie scenes from a video.

Unfortunately VirtualDub didn't have a Linux or BSD version so in my case I had to look for another soft.

VirtualDub running on Microsoft Windows XP Screenshot (Biomassa)

I consulted a friend of mine who recommended a video editor program called LiVES.

If you haven't done any video editing previously on Linux (like my case was), you will certainly be happy to try LiVES

Debian GNU / Linux LiVES video editor logo bootscreen shot

LiVES can extract only sound from videos, cut selected parts (frames) from videos and do plenty of other nice stuff. It is just great piece of software for anyone, who needs to do simply (newbie) video editting.

With LiVES even an amateur video editor like me could, immediately learn how to chop a movie scenes

Screenshot opened video for editting with LiVES Linux movie editor  Debian Squeeze Linux shot

To master the basics and edit one video in FLV format it took me about 1 hour of time, as in the beginning it was confusing to get confortable with the program scenes selector.

One downside of LiVES it failure to open a FLV file I wanted to edit.
In order to be able to edit the flv movie hence I first had to convert the FLV to AVI or MPEG, as this two (video multimedia formats) are supported by LiVES video editor.

After completing my video scenes chopping to the AVI file I had to convert back to FLV.

In order to complete the convertion between FLV to AVI format on my Debian Linux, I used a program called avidemux

Avidemux has a nice GUI interface and also like Lives has support for video editting, though I have never succesfully done any video edits with it.

Avidemux IMHO is user (completely intuitive). To convert the FLV to AVI, all I had to do was simply open the file FLV file, press (CTRL+S) select my FLV video file format and select the output file extension format to be AVI.

Further on, used LiVES to cut my desired parts from my video of choice. Once the cuts were complete I saved the new cutted version of video to AVI.
Then I needed the video again in FLV to upload it in Joomla, so used ffmpegcommand line tool to do the AVI to FLV file converstion, like so:

hipo@noah:~$ /usr/bin/ffmpeg -i my_media_file.avi my_video_file.flv

Hope this article helps someone aiming to do basic video editting on Linux with LiVES and just like needed FLV to AVI and AVI to FLV convertions.

How to solve “Incorrect key file for table ‘/tmp/#sql_9315.MYI’; try to repair it” mysql start up error

Saturday, April 28th, 2012

When a server hard disk scape gets filled its common that Apache returns empty (no content) pages…
This just happened in one server I administer. To restore the normal server operation I freed some space by deleting old obsolete backups.
Actually the whole reasons for this mess was an enormous backup files, which on the last monthly backup overfilled the disk empty space.

Though, I freed about 400GB of space on the the root filesystem and on a first glimpse the system had plenty of free hard drive space, still restarting the MySQL server refused to start up properly and spit error:

Incorrect key file for table '/tmp/#sql_9315.MYI'; try to repair it" mysql start up error

Besides that there have been corrupted (crashed) tables, which reported next to above error.
Checking in /tmp/#sql_9315.MYI, I couldn't see any MYI – (MyISAM) format file. A quick google look up revealed that this error is caused by not enough disk space. This was puzzling as I can see both /var and / partitions had plenty of space so this shouldn't be a problem. Also manally creating the file /tmp/#sql_9315.MYI with:

server:~# touch /tmp/#sql_9315.MYI

Didn't help it, though the file created fine. Anyways a bit of a closer examination I've noticed a /tmp filesystem mounted besides with the other file system mounts ????
You can guess my great amazement to find this 1 Megabyte only /tmp filesystem hanging on the server mounted on the server.

I didn't mounted this 1 Megabyte filesystem, so it was either an intruder or some kind of "weird" bug…
I digged in Googling to see, if I can find more on the error and found actually the whole mess with this 1 mb mounted /tmp partition is caused by, just recently introduced Debian init script /etc/init.d/mountoverflowtmp.
It seems this script was introduced in Debian newer releases. mountoverflowtmp is some kind of emergency script, which is triggered in case if the root filesystem/ space gets filled.
The script has only two options:

# /etc/init.d/mountoverflowtmp
Usage: mountoverflowtmp [start|stop]

Once started what it does it remounts the /tmp to be 1 megabyte in size and stops its execution like it never run. Well maybe, the developers had something in mind with introducing this script I will not argue. What I should complain though is the script design is completely broken. Once the script gets "activated" and does its job. This 1MB mount stays like this, even if hard disk space is freed on the root partition – / ….

Hence to cope with this unhandy situation, once I had freed disk space on the root partition for some reason mountoverflowtmp stop option was not working,
So I had to initiate "hard" unmount:

server:~# mount -l /tmp

Also as I had a bunch of crashed tables and to fix them, also issued on each of the broken tables reported on /etc/init.d/mysql start start-up.

server:~# mysql -u root -p
mysql> use Database_Name;
mysql> repair table Table_Name extended;
....

Then to finally solve the stupid Incorrect key file for table '/tmp/#sql_XXYYZZ33444.MYI'; try to repair it error, I had to restart once again the SQL server:

Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..
root@server:/etc/init.d#

Tadadadadam!, SQL now loads and works back as before!

Convert single PDF pages to multiple SVG files on Debian Linux with pdf2svg

Sunday, February 26th, 2012

In my last article, I've explained How to create PNG, JPG, GIF pictures from one single PDF document
Convertion of PDF to images is useful, however as PNG and JPEG graphic formats are raster graphics the image quality gets crappy if the picture is zoomed to lets say 300%.
This means convertion to PNG / GIF etc. is not a good practice especially if image quality is targetted.

I myself am not a quality freak but it was interesting to find out if it is possible to convert the PDF pages to SVG (Scalable Vector Graphics) graphics format.

Converting PDF to SVG is very easy as for GNU / Linux there is a command line tool called pdf2svg
pdf2svg's official page is here

The traditional source way compile and install is described on the homepage. For Debian users pdf2svg has already existing a deb package.

To install pdf2svg on Debian use:

debian:~# apt-get install --yes pdf2svg
...

Once installed usage of pdf2svg to convert PDF to multiple SVG files is analogous to imagemagick's convert .
To convert the 44 pages Projects.pdf to multiple SVG pages – (each PDF page to a separate SVG file) issue:

debian:~/project-pdf-to-images$ for i in $(seq 1 44); do \
pdf2svg Projects.pdf Projects-$i.SVG $i; \
done

This little loop tells each page number from the 44 PDF document to be stored in separate SVG vector graphics file:

debian:~/project-pdf-to-images$ ls -1 *.svg|wc -l
44

For BSD users and in particular FreeBSD ones png2svg has a bsd port in:

/usr/ports/graphics/pdf2svg

Installing on BSD is possible directly via the port and convertion of PDF to SVG on FreeBSD, should be working in the same manner. The only requirement is that bash shell is used for the above little bash loop, as by default FreeBSD runs the csh. 
On FreeBSD launch /usr/local/bin/bash, before following the Linux instructions if you're not already in bash.

Now the output SVG files are perfect for editting with Inkscape or Scribus and the picture quality is way superior to old rasterized (JPEG, PNG) images

How to make screenshots on Slackware Linux with XFCE graphical environment

Thursday, February 23rd, 2012

1. Install the slackware binary package xfce4-screenshooter.

For the latest Slackware Linux release which as of time of writting is 13.37 xfce4-screenshooter-1.7.9-i486-3sl.txz can be download from here

Install of xfce4-screenshooter-1.7.9-i486-3sl.txz is done with slackware's usual installpkg package manager command:

bash-4.1# /sbin/installpkg xfce4-screenshooter-1.7.9-i486-3sl.txz

By the way, I haven't used slackware for a long time so in the mean time since Slackware 13, the default slackware packages format .tgz is now substituted with the newer .txz (better compressed .txz). The old .tgz was simply a tar archive with DEFLATE gzip. The newer .txz packages bundled with newer slackware releases are using the LZMA2 (XZ) chain algorithm for compression. LZMA implies higher compression than even bzip2 and this is the reason why Patrick Volkerding – the one man army man behind Slackware decided to use it.
The reason Vollerding choose using .txz is slackware network distribution will load up less the networks and will take less time for downloading extra slackware packages via the internet. The .txz also reduces slackware main CD size so more packages can be contained in the same 700MB sized slack install CD.

Anyways now back to the installation of xfce-screenshooter.

Once installed to runit use the Xfce menus:

Xfce Menu -> Accesories -> Screenshot

Next you will see the xfce-screenshooter program to pop-up:

To take a snapshot of the screen use:

Entire Screen -> Save

XFCE screenshooter Slackware Linux take a screenshot dialog

XFCE screenshooter Slackware Linux action Save