Posts Tagged ‘programmer’

How to convert file content encoded in windows-cp1251 charset to UTF-8 (with iconv) to be delivered properly encoded to browsing end clients

Wednesday, May 16th, 2012

windows-cp1251 bulgarian to UTF-8 / Encoding Communication Decoding Communication Funny Picture

I have a bunch of old html files all encoded in the historically obsolete Windows-cp1251. Windows-CP1251 used to be common used 7 years ago and therefore still big portions of the web content in Bulgarian / Russian Cyrillic is still transferred to the end users in this encoding.

This was just before the "UTF-8 revolution", where massively people started using UTF-8,
Well it was clear the specific national country text encoding standards will quickly be moved by to UTF-8 – Universal Encoding format which abbreviation stands for (Unicode Transformation Format).

Though UTF-8 was clear to be "the future", many web developers mostly because of their incompetency or using an old sources of learning how to writen in HTML continued to use windows-cp1251 in HTMLs. I'm even convinced, there are still developers out there who are writting websites for Bulgarian / Russian / Macedonian customers using obsolete encodings …

The smarter developers of those accustomed to windows-cp1251, KOI-8R etc. etc., were using the meta tag to specify the type of charset of the web page content with:

<meta http-equiv="content-type" content="text/html;charset=windows-cp1251">

or

<meta http-equiv="content-type" content="text/html;charset=koi-8r">

Anyhow, still many devs even didn't placed the windows-cp1251 in the head of the HTML …

The result for the system administrator is always a mess – a lot of webpages that are showing like unreadable signs and tons of unhappy customers.
As always the system administrator is considered responsible, for the programmer mistakes :). So instead of programmers fix their bad cooking, the admin has to fix it all!

One quick work around me as admin has applied to failing to display pages in Cyrillic using the Windows-cp1251 character encoding was to force windows-cp1251 as a default encoding for the whole virtualhost or Apache directory with Apache directives like:

<VirtualHost *:80>
ServerAdmin some_user@some_host.com
DocumentRoot /var/www/html
AddDefaultCharset windows-cp1251
ServerName the_host_name.com
ServerAlias www.the_host_name.com
....
....
<Directory>
AddDefaultCharset windows-cp1251
>/Directory>
</VirtualHost>

Though this mostly would, work there are some occasions, where only a particular html files from all the content served by Apache is encoded in windows-cp1251, if most of the content is already written in UTF-8, this could be a big issues as you cannot just change the UTF-8 globally to windows-cp1251, just because few pages are written in archaic encoding….
Since most of the content is displayed to the client by Apache (as prior explained) just fine, only particular htmls lets's ay single.html, single2.html etc. etc. are displayed with some question marks or some non-human readable "hieroglyphs".

Below is a screenshot from two pages returned to my browser in wrongly set htmls charset:

Improper Windows CP1251 encoding with Apache set to serve UTF-8 encoding questiomarks

Improper Windows CP1251 delivered page in UTF-8 browser view

Apache returns cp1251 in some non-UTF8 wrong encoding (webserver improperly served cyrillic encoding)

Improperly served encoding CP1251 delivered by Apache in non-utf-8 encoding

When this kind of issues occur, the only solution is to simply login to the server and use iconv command to convert all files returning unreadable content from whatever the non UTF-8 encoding is lets say in my case Bulgarian typeset of cp1251 to UTF-8

Here is how the iconv command to convert between windows-cp1251 to utf-8 the two sample files named single1.html and single2.html

server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single1.html > single1.html.utf8
server:/web# mv single1.html single1.html.bak;
server:/web# mv single1.html.utf8 single1.html
server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single2.html > single2.html.utf8
server:/web# mv single2.html single2.html.bak;
server:/web# mv single2.html.utf8 single2.html

I always, make copies of the original cp1251 encoded files (as you see mv single1.html single1.html.bak), because if something goes wrong with convertion I can easily revert back.

If there are 10 files with consequential numbers naming they can be converted using a short for loop, like so:

server:/web# for i $(seq 1 10); do
/usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single$i.html > single$i.html.utf8;mv single$i.html single$i.html.bak
mv single$i.html.utf8 single$i.html
done

Just as earlier mentioned if single1.html, single2.html … has in the html <head>:

<meta http-equiv="Content-Type" content="text/html; charset=windows-1251">

You should open, each of the files in question and wipe out the line either by hand or use sed to wipe it in one loop if it has to be done for lets say 10 files named (single{1..10})

server:/web# for i in $(seq 1 10); do
sed '/<meta http-equiv="Content-Type" content="text\/html; charset=windows-1251>/d' single$i.txt > single$i.txt.new;
mv single$i.txt single$i.txt.bak;
mv single$i.txt.new single$i.txt

Well now,

Fixing / Resolving Fullscreen Adobe Flash issues in Debian Linux

Monday, April 11th, 2011

Adobe Flash Player ugly Logo!

If you’re experiencing problems with maximising flash (let’s say youtube) videos on your Debian or Ubuntu or any other debian derivative.
You’re not the only one! I myself has often experienced the same annoying issue.

The flash fullscreen failures or slownesses are caused by flash player’s attempts to use directly your machine hardware, as Linux kernel is rather different than Windows and the guys from Macromedia are creating always a way more buggy port of flash for unix than it’s windows versions, it’s quite normal that the flash player is unable to properly address the computer hardware on Linux.

As i’m not programmer and I couldn’t exactly explain the cause for the fullscreen flash player mishaps, I’ll skip this and right give you the two command lines solution:

debian:~# mkdir /etc/adobe
debian:~# echo "OverrideGPUValidation = 1" >> /etc/adobe/mms.cfg

This should fix it for, you now just restart your Icedove (Firefox), Epiphany Opera or whatever browser you’re used to and launch some random video in youtube to test the solution, hopefully it should be okay 😉 But you never know with flash let’s just hope that very soon the open flash alternative gnash will be production ready and at last we the free software users will be freed from the evil “slavery” of adobe’s non-free flash player!
Though this tip is tested on Debian based Linux distributions it should most likely work same in all kind of other Linuxes.

The tip should also probably have effect in FreeBSD, though the location of the adobe directory and mms.cfg should probably be /usr/local/etc/adobe, I’ll be glad to hear from some FreeBSD user if including the OverrideGPUValidation = 1 flash option to mms.cfg like below:

# mkdir /usr/local/etc/adobe
# echo "OverrideGPUValidation = 1" >> /usr/local/etc/adobe/mms.cfg

would have an impact on any flash player fullscreen issues on FreeBSD and other BSD direvative OSes that run the linux-flash port.

Nokia 9300i Communicator with final firmware Symbian version 6.27 still contains an old telephone hanging bug

Sunday, September 5th, 2010

Earlier I’ve written an article tagged How to reset to factory defaults (hard reset) hanged Nokia 9300i device / How to format your Nokia 9300i

Therein I explained how I ended up with an unbootable telephone after improperly shutting down my Nokia directly removing the battery instead of pressing the Nokia front turn off button first and then remove the battery.

Two months ago in order to get rid of the annoying nokia hanging on boot up bug I decided to flush my firwmare.
The firmware was a flushed by a friend of me who is a Mobile software specialist he used a special so called double cable and a special software programmer device which is required to flush up a certain range of mobile phones models.

Since the firmware update my Nokia was equipped with the latest firmware version provided by Nokia corp before Nokia dropped the support for the phone.
The latest Nokia firmware version available which was installed on my phone was 6.27.(0)

For two months the Nokia worked just fine, however just recently once again after a couple of different SIM card changes with the phone, the phone hanged the same time as described in my earlier written article How to reset to factory defaults (hard reset) hanged Nokia 9300i device / How to format your Nokia 9300i

This pissed me off and I once again had to follow the necessary described steps about formatting the Nokia 9300i Communicator mobile device as it’s explained in the up-mentioned article.

The format went fine nevertheless the newer phone firmware version.
The format completed quite quickly once again it took in between 30 seconds and a minute to complete.

The good news after the format of my hanged Nokia phone was that it kept the newer firmware version, even after reverting to facture defaults with the firmware format.
Now thanksfully still my Symbian OS version is still 6.27(0)

It’s rather strange that even the newest avaialable firmware patch for the Nokia 9300i doesn’t fix such a vital boot up mobile hang error.

Hope this info is helpful to somebody out there.

How to fix problems with encoding not showing umlauts in after import of sql data to MySQL

Thursday, October 1st, 2009

I’m restoring some websites from backups this days. One of the swiss websites had a serious problem with umlauts not showing up.
This happened right after I’ve used an old dump from a MySQL Server running version 4.x, the imported data was to MySQL server version 5. The problem consisted in that everywhere an umault was placed the shown content was ü.

You can imagine how annoying and ugly that looked, the whole text was crappy.
After some googling with a help of one of my colleagues (a programmer). I was pointed to this nice article Mysql Latin1 Utf8 Conversion .
What happens is that for some reason the dump I’ve made had latin1 character-set even though the data inside was in utf8.
Thus importing the dump would try to import the data as latin1 and make a crap out of it. The fix is as simple as substituting latin8 to utf8 in your mysql dump file and then reimporting it again.
In my case the browser displayed by default the website characters in iso8859 instead of utf8, so I had to specificly to change the browser encoding to UTF8 to realize all is okay.
Then it was necessery to modify all the templates to use UTF8 instead of the wrong character encoding. I have no clue how does it happened that the same umlaut encoding on the old server, what I suspect is there was something with the Apache’s default character encoding probably I have it set there by default set to utf8.
Well so far so good, let’s see how much trashy stuff I have to deal with today.
END—–

Durankulak’s beach

Monday, August 27th, 2007

In Saturday we was on a Cisco course with Nomen, Niki (A friend programmer),and the Other Niki (Mitko’s brother). The course went smoothly up to some pointafter that Niki (the programmer) has received a call with an awful news…After the course I Nomen and Nomen’s brother went to the chineese restaurant and we ate rice with vegatables and spaghetti with vegetables and meat. An hour later we was at Nomen’s home we had to make the cisco’s Chapter One and Chapter two test. And Luckily I got 100% right answers on both of the two (I have to be honest that I used cheating and tricks on the tests so I probably deserve less.After that Nomen gave suggestion to go to Carapec or somewhere on the Beach on the Bulgarian Coast. At the end we ended beaching the next morning on Durankulak’s Camping beach after we setupped a fire for the night and we slept in tents. There was a lot of problems during the whole trip ( I won’t go into details) but Thanks to God Almighty all has ended well in the end. Talking about God, I’m smoking cigarettes again and I have to stop (I hope God would help again). Also something I have to note is I’m a sinner but God is faithful although I sin badly The Lord is gracious to me still. PRAISE THE LORD!!! HalleluJah! :]END—–

Troubled Day

Thursday, April 5th, 2007

It was a day full of waiting. The Admin/tech support personel in sofia is a real pain in the ass. I waited half a day for a simple debian base install. The guy installed debian over already installed freebsd on a server we sent destined for colocation in evolink. The machine is great dual 3ghz Xeon with 3 x 70 gb SCSI discs. In 4:00 o’clock I received a mail with username and password of the server, but the system was unaccessible for 1 more hour. I got really nervous I received tons of calls from the boss, our project Manager, Vladi the PHP programmer. I logged to the server in 5:00 and configured apache with mysql upto 10:00 o’clock then did a little walk with nomen in the central park and drinked one beer per capita. I had to meet Static and Amri in 6:00 o’clock but I was busy configuring the new rack so I missed a great oportunity to have fun with old friends. ORDER has to come back to Bulgaria, today or even he maybe is already in Varna. I’m quite impatient to see him! In the afternoon I went to some spiritual downfalls but now Thanks to God I’m alive and well. The pozvanete site already points to the new rack so I hope the machine would be able to serve it’s goal perfect. This will be made clear in the morning. Soon I’m going to bed. Glory be to God for fulfilling me with his love and sustaining my life and keeping me from evil! END—–

Day After Day

Thursday, February 22nd, 2007

I forgot I had German in the morning so I get to sleep at 3:00 a.m. In the morning my mother wake me up to ask my did I have lectures. I said yes in 10:00 then I rembembered about the German. I tried as fast as possible to stand up wash my teeths and go to the college. After the German we had some Introduction to Management classes. One of our programmers Pesho called me and said we have a problem with few of the scripts setupped to be run on crontab on one of the servers, then also the other programmer Milen called and said he had a problem with some other script which loops one of the webservers … having problems is really unpleasant. At least I’m not so sleepy as I should be for the small quantity of time I’ve slept in the night.We went to a coffee with Narf and Nomen in the lunch break after that Management again. And now I’m doing some stuff on the servers. I’m little tired now and maybe I’ll sleep. Praise the Lord for I’m spiritually okay most of the time today. Thanks God :]END—–