Posts Tagged ‘Detect’

How to Install and Use Kibana for Log Visualization

Wednesday, February 18th, 2026

/images/kibana-logo how to install it on linux
I saw Kibana in my professional career and I find it a very interesting tool for sysadmins, so I thought it might be helpful to someone out there to write a small article on how to install and use to to visualize data inside some elasticsearch software.

Kibana is an open-source data visualization and exploration tool used to analyze large volumes of data, especially logs. It is part of the ELK Stack (Elasticsearch, Logstash, Kibana), and is commonly used for centralized log management, security monitoring, and observability.

Kibana is often used in the so-called ELK pipeline for log file collection, analysis and visualization:

  • Elasticsearch is for searching, analyzing, and storing your data
  • Logstash (and Beats) is for collecting and transforming data, from any source, in any format
  • Kibana is a portal for visualizing the data and to navigate within the elastic stack
     

In this article, you'll learn how to:

  • Install Kibana
  • Connect it to Elasticsearch
  • Visualize log data
  • Use its basic features

Prerequisites

Before installing Kibana, make sure you have the following:

  • A Linux server running (Ubuntu / Debian / CentOS / RHEL)
  • Elasticsearch installed and running
  • Root or sudo access

Install Kibana

I. On Debian/Ubuntu
 

  1. Import the Elastic GPG key:

# wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

  1. Add the repository:

# echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list

  1. Update and install:


# apt update

# apt install kibana

II. On RHEL/CentOS Linux

  1. Create repo file:

# tee /etc/yum.repos.d/elastic.repo <<EOF

[elastic-8.x]

name=Elastic repository for 8.x packages

baseurl=https://artifacts.elastic.co/packages/8.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

EOF

  1. Install Kibana:

# yum install kibana

2. Configure Kibana

The configuration file is located at:

/etc/kibana/kibana.yml

Edit the file:

# vim /etc/kibana/kibana.yml

Update or add the following:
 

# Server settings
server.port: 5601
server.host: "0.0.0.0"

# Elasticsearch connection
elasticsearch.hosts: [“http://localhost:9200”]

# Logging
logging.level: info

# Security (only if Elasticsearch security is enabled)
# elasticsearch.username: "kibana_system"
# elasticsearch.password: "your_password_here"

Optional: Set basic auth or SSL settings if needed.

 

3. Start and Enable Kibana

# systemctl enable kibana

# systemctl start kibana

Check status:

# systemctl status kibana
…

 

4. Access Kibana Web Interface

Open your browser and go to:

http://<your-server-ip>:5601

You’ll be welcomed with the Kibana dashboard.

5. Import and Visualize Logs

Option A: Use Filebeat to Send Logs

Install Filebeat on the server with logs and configure it to send data to Elasticsearch. Kibana will then be able to visualize it.

# apt install filebeat

# filebeat modules enable system

# filebeat setup

# systemctl start filebeat

Option B: Ingest Logs via Logstash or Elasticsearch API

If you already have data in Elasticsearch, Kibana will automatically detect indices.
 

6. Create Index Pattern

  1. In Kibana, go to Stack Management -> Index Patterns
  2. Click Create Index Pattern
  3. Enter the name (e.g., filebeat-*)
  4. Select the timestamp field (usually @timestamp)
  5. Save

Now Kibana knows how to query and visualize your data.

7. Create Visualizations and Dashboards

  1. Go to Visualize -> Create visualization
  2. Choose a type (bar, pie, line, etc.)
  3. Select an index pattern
  4. Configure metrics and buckets

You can then save visualizations and add them to dashboards.

8. Secure Kibana

  • Configure TLS/SSL for Kibana / ElasticSearch (such as Logstash)
  • Use additional Elastic Security features like RBAC (Role Based Access Control, SSO (Single Sign On)
  • Secure Kibana with a reverse proxy (e.g., Nginx + Basic Auth or Apache / Haproxy infront)

Example Nginx config simple snippet:

location / {

  proxy_pass http://localhost:5601;

  auth_basic "Restricted";

  auth_basic_user_file /etc/nginx/.htpasswd;

}

 

What is Kibana used for and what it can do for you?

Use Case

Description

Log Monitoring

Visualize system and application logs in real time

Security Analytics

Detect anomalies, failed logins, suspicious activity

DevOps Dashboards

Track uptime, error rates, and system performance

SIEM

Use Elastic Security for threat detection

 

Once Kibana is installed on a server, you typically use it to visualize and explore data stored in Elasticsearch. Here’s a practical guide with sample usage scenarios:

Access Kibana

After installation, Kibana usually runs on port 5601 by default.

http://<your-server-ip>:5601

  • Open this URL in a browser.
  • You should see the Kibana dashboard.

Connect to Elasticsearch

Kibana automatically connects to your Elasticsearch instance if installed locally.
You can verify the connection:

GET /_cluster/health

  • Go to Dev Tools → Console in Kibana.
  • Run the above query to check cluster status.

Visualize Data

Kibana allows multiple types of visualizations:

  • Bar/line chart: trends over time.
  • Pie chart: distribution of values.
  • Data table: top IP addresses or most visited URLs.
  • Maps: geolocation of IP addresses.

Create Dashboards

  • Combine multiple visualizations in a Dashboard.
  • Useful for monitoring logs, metrics, or application performance.
  • Example: Create a dashboard with:

     

    • Requests per URL (bar chart)
    • Requests over time (line chart)
    • Top client IPs (data table)
    • Errors by type (pie chart)

 Search & Query Logs

  • Use Discover to search logs interactively.
  • Example KQL query:

status:500 AND url:"/login"

This finds all failed login requests.

Set Alerts (Optional)

  • Kibana’s Alerts and Actions can trigger notifications (email, Slack, etc.) when certain thresholds are crossed.
  • Example: alert if error responses exceed 100 in 5 minutes.

Once Kibana is installed on a server, you typically use it to visualize and explore data stored in Elasticsearch. Here’s a practical guide with sample usage scenarios:

Access Kibana

After installation, Kibana usually runs on port 5601 by default.

http://<your-server-ip>:5601

  • Open this URL in a browser.
  • You should see the Kibana dashboard.

Connect to Elasticsearch

Kibana automatically connects to your Elasticsearch instance if installed locally.
You can verify the connection:

GET /_cluster/health

  • Go to Dev Tools → Console in Kibana.
  • Run the above query to check cluster status.

Visualize Data

Kibana allows multiple types of visualizations:

  • Bar/line chart: trends over time.
  • Pie chart: distribution of values.
  • Data table: top IP addresses or most visited URLs.
  • Maps: geolocation of IP addresses.

Create Dashboards

  • Combine multiple visualizations in a Dashboard.
  • Useful for monitoring logs, metrics, or application performance.
  • Example: Create a dashboard with:
     

    • Requests per URL (bar chart)
    • Requests over time (line chart)
    • Top client IPs (data table)
    • Errors by type (pie chart)

 Search & Query Logs

  • Use Discover to search logs interactively.
  • Example KQL query:

status:500 AND url:"/login"

This finds all failed login requests.

Set Alerts (Optional)

  • Kibana’s Alerts and Actions can trigger notifications (email, Slack, etc.) when certain thresholds are crossed.
  • Example: alert if error responses exceed 100 in 5 minutes.

Once Kibana is installed on a server, you typically use it to visualize and explore data stored in Elasticsearch. Here’s a practical guide with sample usage scenarios:

Access Kibana

After installation, Kibana usually runs on port 5601 by default.

http://your-server-ip:5601

  • Open this URL in a browser.
  • You should see the Kibana dashboard.

Connect to Elasticsearch

Kibana automatically connects to your Elasticsearch instance if installed locally.
You can verify the connection:

GET /_cluster/health

  • Go to Dev Tools → Console in Kibana.
  • Run the above query to check cluster status.

Visualize Data

Kibana allows multiple types of visualizations:

  • Bar/line chart: trends over time.
  • Pie chart: distribution of values.
  • Data table: top IP addresses or most visited URLs.
  • Maps: geolocation of IP addresses.

Create Dashboards

  • Combine multiple visualizations in a Dashboard.
  • Useful for monitoring logs, metrics, or application performance.
  • Example: Create a dashboard with:

    • Requests per URL (bar chart)
    • Requests over time (line chart)
    • Top client IPs (data table)
    • Errors by type (pie chart)

 Search & Query Logs

  • Use Discover to search logs interactively.
  • Example KQL query:

status:500 AND url:"/login"

This finds all failed login requests.

Set Alerts (Optional)

  • Kibana’s Alerts and Actions can trigger notifications (email, Slack, etc.) when certain thresholds are crossed.
  • Example: alert if error responses exceed 100 in 5 minutes.

kibana-sample-dashboard-screenshot

Sample Kibana dashboard
 

kibana-geo-kibana-web-traffic-by-location

Kibana with connected servers to find out Geo Location
 

Summary closing words (what we did)

Step

Action

 1

Install Kibana from Elastic repo

2

Configure to connect to Elasticsearch

3

Start and enable the service

4

Access it via http://<ip>:5601

5

Ingest log data

6

Define index pattern

7

Create dashboards and visualizations

The idea of this article was just to introduce you to the existence of Elasticsearch / kibana and filebeat and logstack and not to give you a fully fine tuned install guide. The usual way to deploy Kibana on multiple servers of course is using a dockerized container version of it. There is plenty to learned on how to use kibana to do a monitoring of your machines. But most simple use is to directly access the locally visible kibana on a server and check the status of processes on the host instead of logging via SSH. Kibana can do pretty much


Some further useful Reading Resources

 

Tags: , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Monitoring, Various | No Comments »

Using GeoIP on Linux: Country-Based Filtering, Logging, and Traffic Control

Friday, January 16th, 2026

geoip-on-linux-country-based-filtering-logging-traffic-control-logo

GeoIP is one of those technologies that quietly sits in the background of many systems, yet it can be extremely powerful when used correctly. Whether you want to block traffic from specific countries, analyze access logs, or add geographic context to security events, GeoIP can be a valuable addition to your Linux toolbox.

In this article, we’ll go deeper and look at real GeoIP usage examples for:

  • Log analysis
  • Firewalls
  • Apache HTTP Server
  • HAProxy

All examples are based on typical GNU/Linux server environments.

What Is GeoIP? 

GeoIP maps an IP address to geographic data such as:

  • Country
  • City
  • ASN / ISP (depending on database)

Most modern systems use MaxMind GeoLite2 databases (

.mmdb

format).

Keep in Mind ! :
GeoIP data is approximate. VPNs, proxies, mobile networks, and CGNAT reduce accuracy. GeoIP should be treated as a heuristic, not a guarantee.

1. Installing GeoIP Databases on Linux deb based distro

On Debian / Ubuntu:

#

apt install geoipupdate

Configure

/etc/GeoIP.conf

with your MaxMind license key and run:  

# geoipupdate

Databases are usually stored in:

/usr/share/GeoIP/

2. GeoIP for Log Analysis (to get idea of where does your traffic origins from)

GeoIP with Apache HTTP Server

Apache can use GeoIP in two main ways:

  1. To do IP origin Logging 

  2. Do Access control based on IP origin

An altenartive GeoIP common use is to post-processing logs to find out attempts to breach your security.

Lets say you want to

Find top attacking countries against your SSHd service.

# grep "Failed password" /var/log/auth.log | \
awk '{print $(NF-3)}' | \
while read ip; do geoiplookup $ip; done |
\ sort | uniq -c | sort -nr


This command will provide you a visibility on attack sources georaphical Country origin

3. Installing Apache GeoIP Module

For legacy GeoIP (older systems):

# apt install libapache2-mod-geoip

For modern systems, GeoIP2 is preferred:

# apt install libapache2-mod-geoip2

Enable the module:

# a2enmod geoip2
# systemctl reload apache2

4. Configure GeoIP Logging in Apache (basic config)

Add country code to access logs:

LogFormat "%h %l %u %t \"%r\" %>s %b %{GEOIP_COUNTRY_CODE}e" geoip
CustomLog /var/log/apache2/access.log geoip

This allows you to analyze traffic by country later without blocking users.

5. Country-Based Filter Blocking in Apache based on IP origin

Example: allow only selected countries:

<IfModule mod_geoip2.c>
SetEnvIf GEOIP_COUNTRY_CODE ^(BG|DE)$ AllowCountry
Deny from all
Allow from env=AllowCountry
</IfModule>

Use this carefully. Blocking at the web server layer is better than firewall-level blocking, but still risky if you have global users.

6. Apply GeoIP to Apache Virtual Host

You can apply GeoIP rules per site:

<VirtualHost *:80>
ServerName example.com
<IfModule mod_geoip2.c>
SetEnvIf GEOIP_COUNTRY_CODE CN BlockCountry >
Deny from env=BlockCountry
</IfModule>
</VirtualHost>

This is useful when only specific applications need filtering.

Firewall vs Application Layer GeoIP (Pros and Cons)

Layer

Pros

Cons

Firewall

Early blocking

Hard to debug

Apache

Flexible per-site rules

App overhead

HAProxy

Centralized control

Requires careful config

Logs only

Safest

No blocking

7. Apply GeoIP to HAProxy

HAProxy is an excellent place to apply GeoIP logic because:

  • It sits in front of applications
  • ​Rules are fast and explicit
  • Logging is centralized

a. Preparing GeoIP Filtering to HAProxy service

HAProxy supports GeoIP2 via Lua or native ACLs using

.mmdb

Example directory:

/usr/share/GeoIP/GeoLite2-Country.mmdb

b. GeoIP-Based Access Control Lists ( ACLs ) in HAProxy

Basic country-based blocking:

frontend http_in
bind *:80

acl from_china src -m geoip CN
acl from_russia src -m geoip RU

http-request deny if from_china
http-request deny if from_russia

default_backend web_servers

This blocks traffic early, before it hits Apache or nginx.

c. GeoIP-Based Routing across different haproxy backends

Instead of blocking, you can route traffic differently:


acl eu_users src -m geoip DE FR NL
use_backend eu_backend if eu_users
default_backend global_backend

This is useful for:

  • Geo-based load balancing
  • Regional content
  • Legal compliance separation

d. GeoIP Logging config for HAProxy

Add country code to logs:

log-format "%ci:%cp [%t] %ft %b %s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC"

(%CC = country code)

This makes traffic analysis extremely efficient.

Keep in Mind !

Use HAProxy or web server level for enforcement, and firewall GeoIP only when absolutely necessary.

8. Fail2ban + GeoIP: Smarter Bans, Better Context

Fail2ban is excellent at reacting to abusive behavior, but by default it only sees IP addresses, not where they come from. Adding GeoIP allows you to:

  • Tag bans with country information
  • Apply different ban policies per region
  • Detect unusual behavior patterns

a. GeoIP-Enriched Fail2ban Logs

Fail2ban itself doesn’t natively evaluate GeoIP rules, but you can enrich logs post-ban.

Example action script (

/etc/fail2ban/action.d/geoip-notify.conf

):

 


[Definition]
actionban = echo "Banned from $(geoiplookup | cut -d: -f2)" >> /var/log/fail2ban-geoip.log
Enable it in a jail:
[sshd]
enabled = true
action = iptables[name=SSH] geoip-notify

Enable it in a jail:

[sshd]

enabled = true action = iptables[name=SSH] geoip-notify

Resulting log entry:

Banned 185.220.101.1 from Germany

This provides visibility without changing ban logic — a safe first step.


b. Use GeoIP-Aware Ban Policies 

You can also adjust ban times based on country.

Example strategy:

  • Short ban for local country
  • Longer ban for known high-noise regions

This is usually implemented via multiple jails and post-processing scripts rather than direct GeoIP matching inside Fail2ban.

Best practice:
Let Fail2ban do behavior detection — let GeoIP provide context, not decisions.

9. GeoIP with nftables (Linux Modern Firewall Layer)

iptables +

xt_geoip

is considered legacy. On modern systems, nftables is the preferred approach.

a. Using GeoIP Sets in nftables

nftables does not natively include GeoIP, but you can integrate GeoIP via generated IP sets.

Workflow:

  1. Convert GeoIP country IP ranges into nftables sets

  2. Load them dynamically

Example set definition:


table inet filter {
set geo_block {
type ipv4_addr
flags interval
}
}

Populate the set using a script:

nft add element inet filter geo_block { 1.2.3.0/24, 5.6.0.0/16 }

Then apply it:


chain input {
type filter hook input priority 0;
ip saddr @geo_block drop
}

b. Automating GeoIP ->  nftables

Typical automation pipeline:

GeoLite2 → country CSV → IP ranges → nftables set

Run this daily via cron.

Warning:

  • Large country sets = memory usage
  • Firewall reloads must be atomic
  • Test on non-production systems first

10. GeoIP Dashboard: Turning Logs into Insight

Blocking is optional — insight is mandatory.

a. Simple GeoIP Log Dashboard (CLI-Based)

Apache example:

# awk '{print $NF}' /var/log/apache2/access.log | \
sort | uniq -c | sort -nr

Where $NF contains country code.

Sample Result:

1243 US

987 DE

422 FR

310 CN

This already tells a story.

b. Visual Dashboard with ELK / Grafana

For larger environments:

HAProxy / Apache -> JSON logs Enrich logs with GeoIP

Send to:

  • ELK Stack
  • Loki + Grafana
  • Graylog

Metrics you want:

  • Requests per country
  • Errors per country
  • Bans per country
  • Login failures per country

This helps distinguish:

  • Marketing traffic
  • Legit users
  • Background Internet noise

11.  Create a Layered GeoIP Strategy

A sane, production-ready model using GeoIP would include something like:

  1. Logging first
    Apache / HAProxy logs with country codes

  2. Behavior detection
    Fail2ban reacts to abuse

  3. Traffic shaping
    HAProxy routes or rate-limits

  4. Firewall last
    nftables drops only obvious garbage

GeoIP is strongest when it supports decisions, not when it makes them alone.

12. Best Practices to consider

  • Prefer visibility over blocking
  • Avoid blanket country bans
  • Always log before denying

Combine GeoIP with:

  • Fail2ban
  • Rate limits
  • CAPTCHA or MFA
  • Keep GeoIP databases (regularly) updated
  • Test rules with real IPs before deploying

13. Common Mistakes to Avoid

Blocking entire continents Using GeoIP as authentication Applying firewall GeoIP without logs Forgetting database updates Assuming GeoIP accuracy

Close up

GeoIP is not a silver bullet against vampire attacks – but when used thoughtfully, it becomes a powerful signal enhancer and can give you a much broader understanding on what is going on inside your network traffic.

Whether you’re using it to filter out segment of evil intruders based on logs, routing traffic intelligently, or filtering obvious abusea, GeoIP fits naturally into a layered security model and is used across corporations and middle and even small sized businesses nowadays.

Used conservatively, GeoIP follows the classic Unix philosophy:

Small datasets, Simple rules, Real-world effectiveness, combined with rest of tools it gives info and ways to protect better your networks and server infra.

Tags: , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Everyday Life | No Comments »

How to check if your Linux WebServer is under a DoS attack

Friday, July 22nd, 2011

There are few commands I usually use to track if my server is possibly under a Denial of Service attack or under Distributed Denial of Service

Sys Admins who still have not experienced the terrible times of being under a DoS attack are happy people for sure …

1. How to Detect a TCP/IP Denial of Service Attack This are the commands I use to find out if a loaded Linux server is under a heavy DoS attack, one of the most essential one is of course netstat.
To check if a server is under a DoS attack with netstat, it’s common to use:

linux:~# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n|wc -l

If the output of below command returns a result like 2000 or 3000 connections!, then obviously it’s very likely the server is under a DoS attack.

To check all the IPS currently connected to the Apache Webserver and get a very brief statistics on the number of times each of the IPs connected to my server, I use the cmd:

linux:~# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
221 80.143.207.107 233 145.53.103.70 540 82.176.164.36

As you could see from the above command output the IP 80.143.207.107 is either connected 221 times to the server or is in state of connecting or disconnecting to the node.

Another possible way to check, if a Linux or BSD server is under a Distributed DoS is with the list open files command lsof
Here is how lsof can be used to list the approximate number of ESTABLISHED connections to port 80.

linux:~# lsof -i TCP:80
litespeed 241931 nobody 17u IPv4 18372655 TCP server.www.pc-freak.net:http (LISTEN)
litespeed 241931 nobody 25u IPv4 18372659 TCP 85.17.159.89:http (LISTEN)
litespeed 241931 nobody 30u IPv4 29149647 TCP server.www.pc-freak.net:http->83.101.6.41:54565 (ESTABLISHED)
litespeed 241931 nobody 33u IPv4 18372647 TCP 85.17.159.93:http (LISTEN)
litespeed 241931 nobody 34u IPv4 29137514 TCP server.www.pc-freak.net:http->83.101.6.41:50885 (ESTABLISHED)
litespeed 241931 nobody 35u IPv4 29137831 TCP server.www.pc-freak.net:http->83.101.6.41:52312 (ESTABLISHED)
litespeed 241931 nobody 37w IPv4 29132085 TCP server.www.pc-freak.net:http->83.101.6.41:50000 (ESTABLISHED)

Another way to get an approximate number of established connections to let’s say Apache or LiteSpeed webserver with lsof can be achieved like so:

linux:~# lsof -i TCP:80 |wc -l
2100

I find it handy to keep track of above lsof command output every few secs with gnu watch , like so:

linux:~# watch "lsof -i TCP:80"

2. How to Detect if a Linux server is under an ICMP SMURF attack

ICMP attack is still heavily used, even though it’s already old fashioned and there are plenty of other Denial of Service attack types, one of the quickest way to find out if a server is under an ICMP attack is through the command:

server:~# while :; do netstat -s| grep -i icmp | egrep 'received|sent' ; sleep 1; done
120026 ICMP messages received
1769507 ICMP messages sent
120026 ICMP messages received
1769507 ICMP messages sent

As you can see the above one liner in a loop would check for sent and recieved ICMP packets every few seconds, if there are big difference between in the output returned every few secs by above command, then obviously the server is under an ICMP attack and needs to hardened.

3. How to detect a SYN flood with netstat

linux:~# netstat -nap | grep SYN | wc -l
1032

1032 SYNs per second is quite a high number and except if the server is not serving let’s say 5000 user requests per second, therefore as the above output reveals it’s very likely the server is under attack, if however I get results like 100/200 SYNs, then obviously there is no SYN flood targetting the machine 😉

Another two netstat command application, which helps determining if a server is under a Denial of Service attacks are:

server:~# netstat -tuna |wc -l
10012

and

server:~# netstat -tun |wc -l
9606

Of course there also some other ways to check the count the IPs who sent SYN to the webserver, for example:

server:~# netstat -n | grep :80 | grep SYN |wc -l

In many cases of course the top or htop can be useful to find, if many processes of a certain type are hanging around.

4. Checking if UDP Denial of Service is targetting the server

server:~# netstat -nap | grep 'udp' | awk '{print $5}' | cut -d: -f1 | sort |uniq -c |sort -n

The above command will list information concerning possible UDP DoS.

The command can easily be accustomed also to check for both possible TCP and UDP denial of service, like so:

server:~# netstat -nap | grep 'tcp|udp' | awk '{print $5}' | cut -d: -f1 | sort |uniq -c |sort -n
104 109.161.198.86
115 112.197.147.216
129 212.10.160.148
227 201.13.27.137
3148 91.121.85.220

If after getting an IP that has too many connections to the server and is almost certainly a DoS host you would like to filter this IP.

You can use the /sbin/route command to filter it out, using route will probably be a better choice instead of iptables, as iptables would load up the CPU more than simply cutting the route to the server.

Here is how I remove hosts to not be able to route packets to my server:

route add 110.92.0.55 reject

The above command would null route the access of IP 110.92.0.55 to my server.

Later on to look up for a null routed IP to my host, I use:

route -n |grep -i 110.92.0.55

Well hopefully this should be enough to give a brief overview on how, one can dig in his server and find if he is under a Distributed Denial of Service, hope it’s helpful to somebody out there.
Cheers 😉

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | 5 Comments »

Howto to detect file encoding and convert default encoding of given files from one encoding to another on GNU/Linux and FreeBSD

Sunday, February 7th, 2010

I wanted to convert an html document character encoding to UTF-8, to achieve that of
course it was first needed to determine what kind of character encoding was used in
creation time of the file.
First thing I tried was:

hipo@noah:~/Desktop/test$ file File-Whole.htm
File-Whole.htm: HTML document text

as you can see that’s shit cause for some reason mime encoding is not printed by the file
command.
Next what I tried was:
hipo@noah:~/Desktop/test$ file --mime File-Whole.htm1File-Whole.htm1: text/html; charset=unknown-8bit

Here you see that character encoding is reported as charset=unknown-8bit which
ain’t cool at all and is of no use and prompts an error if I try it in iconv
Here is why I needed concretely to determine what kind of character set my file uses to later
be able to convert it using iconv .
To achieve my goal after consulting with Mr. Google , I found
out about enca — detect and convert encoding of text files
It’s obviously my lucky day because good guys from Debian has packaged enca so, everything came to the point of
apt-getting it.
# apt-get install enca

On FreeBSD enca port is available, so installing it cames simply to installing it from port tree.
Here is how:
pcfreak# cd /usr/ports/converters/enca;pcfreak# make install clean

Now I tried launching enca directly without any program parameters, but I was unlucky:

hipo@noah:~/Desktop/test$ enca file-Whole.htm
enca: Cannot determine (or understand) your language preferences.
Please use `-L language', or `-L none' if your language is not supported
(only a few multibyte encodings can be recognized then).
Run `enca --list languages' to get a list of supported languages.

I gave it another try, following prescribed usage parameters though I first checked my possibility
as a languages I can pass by to enca’s -L parameter.
Preliminary knowing that my text contains text in Bulgarian language, it wasn’t such a big deal
for me to determine the required language:

hipo@noah:~/Desktop/test$ enca -L bulgarian File-Whole.htm
transformation format 8 bits; CP1251

Knowing my character set all left for me was to do do the convert to UTF-8 to make text,
much more accessible.

hipo@noah:~/Desktop/test$ iconv --from-code=unknown-8bit --to=UTF-8 File-Whole.htm > File-Whole.htm.new
hipo@noah:~/Desktop/test$ mv File-Whole.htm.new File-Whole.htm

Well here we are conversion mission accomplished 🙂

Tags: , , , ,
Posted in FreeBSD, Linux and FreeBSD Desktop, System Administration | 6 Comments »