Posts Tagged ‘action’

How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages

Wednesday, February 19th, 2020

haproxy-logging-basics-how-to-log-to-separate-file-prevent-duplicate-messages-haproxy-haproxy-weblogo-squares
haproxy  logging can be managed in different form the most straight forward way is to directly use /dev/log either you can configure it to use some log management service as syslog or rsyslogd for that.

If you don't use rsyslog yet to install it: 

# apt install -y rsyslog

Then to activate logging via rsyslogd we can should add either to /etc/rsyslogd.conf or create a separte file and include it via /etc/rsyslogd.conf with following content:
 

Enable haproxy logging from rsyslogd


Log haproxy messages to separate log file you can use some of the usual syslog local0 to local7 locally used descriptors inside the conf (be aware that if you try to use some wrong value like local8, local9 as a logging facility you will get with empty haproxy.log, even though the permissions of /var/log/haproxy.log are readable and owned by haproxy user.

When logging to a local Syslog service, writing to a UNIX socket can be faster than targeting the TCP loopback address. Generally, on Linux systems, a UNIX socket listening for Syslog messages is available at /dev/log because this is where the syslog() function of the GNU C library is sending messages by default. To address UNIX socket in haproxy.cfg use:

log /dev/log local2 


If you want to log into separate log each of multiple running haproxy instances with different haproxy*.cfg add to /etc/rsyslog.conf lines like:

local2.* -/var/log/haproxylog2.log
local3.* -/var/log/haproxylog3.log


One important note to make here is since rsyslogd is used for haproxy logging you need to have enabled in rsyslogd imudp and have a UDP port listener on the machine.

E.g. somewhere in rsyslog.conf or via rsyslog include file from /etc/rsyslog.d/*.conf needs to have defined following lines:

$ModLoad imudp
$UDPServerRun 514


I prefer to use external /etc/rsyslog.d/20-haproxy.conf include file that is loaded and enabled rsyslogd via /etc/rsyslog.conf:

# vim /etc/rsyslog.d/20-haproxy.conf

$ModLoad imudp
$UDPServerRun 514​
local2.* -/var/log/haproxy2.log


It is also possible to produce different haproxy log output based on the severiy to differentiate between important and less important messages, to do so you'll need to rsyslog.conf something like:
 

# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log

 

Prevent Haproxy duplicate messages to appear in /var/log/messages

If you use local2 and some default rsyslog configuration then you will end up with the messages coming from haproxy towards local2 facility producing doubled simultaneous records to both your pre-defined /var/log/haproxy.log and /var/log/messages on Proxy servers that receive few thousands of simultanous connections per second.
This is a problem since doubling the log will produce too much data and on systems with smaller /var/ partition you will quickly run out of space + this haproxy requests logging to /var/log/messages makes the file quite unreadable for normal system events which are so important to track clearly what is happening on the server daily.

To prevent the haproxy duplicate messages you need to define somewhere in rsyslogd usually /etc/rsyslog.conf local2.none near line of facilities configured to log to file:

*.info;mail.none;authpriv.none;cron.none;local2.none     /var/log/messages

This configuration should work but is more rarely used as most people prefer to have haproxy log being written not directly to /dev/log which is used by other services such as syslogd / rsyslogd.

To use /dev/log to output logs from haproxy configuration in global section use config like:
 

global
        log /dev/log local2 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

The log global directive basically says, use the log line that was set in the global section for whole config till end of file. Putting a log global directive into the defaults section is equivalent to putting it into all of the subsequent proxy sections.

Using global logging rules is the most common HAProxy setup, but you can put them directly into a frontend section instead. It can be useful to have a different logging configuration as a one-off. For example, you might want to point to a different target Syslog server, use a different logging facility, or capture different severity levels depending on the use case of the backend application. 

Insetad of using /dev/log interface that is on many distributions heavily used by systemd to store / manage and distribute logs,  many haproxy server sysadmins nowdays prefer to use rsyslogd as a default logging facility that will manage haproxy logs.
Admins prefer to use some kind of mediator service to manage log writting such as rsyslogd or syslog, the reason behind might vary but perhaps most important reason is  by using rsyslogd it is possible to write logs simultaneously locally on disk and also forward logs  to a remote Logging server  running rsyslogd service.

Logging is defined in /etc/haproxy/haproxy.cfg or the respective configuration through global section but could be also configured to do a separate logging based on each of the defined Frontend Backends or default section. 
A sample exceprt from this section looks something like:

#———————————————————————
# Global settings
#———————————————————————
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#———————————————————————
defaults
    mode                    tcp
    log                     global
    option                  tcplog
    #option                  dontlognull
    #option http-server-close
    #option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 7
    #timeout http-request    10s
    timeout queue           10m
    timeout connect         30s
    timeout client          20m
    timeout server          10m
    #timeout http-keep-alive 10s
    timeout check           30s
    maxconn                 3000

 

 

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

 

#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_DKV_PROD_WLPFO
    mode tcp
    bind 192.168.233.5:30000-31050
    option tcplog
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
    default_backend Default_Bakend_Name


#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_DKV_PROD_WLPFO
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
    stick on src
    # (5) Server List
    # (5.1) Backend
    server Backend_Server1 10.10.10.1 check port 18088
    server Backend_Server2 10.10.10.2 check port 18088 backup


The log directive in above config instructs HAProxy to send logs to the Syslog server listening at 127.0.0.1:514. Messages are sent with facility local2, which is one of the standard, user-defined Syslog facilities. It’s also the facility that our rsyslog configuration is expecting. You can add more than one log statement to send output to multiple Syslog servers.

Once rsyslog and haproxy logging is configured as a minumum you need to restart rsyslog (assuming that haproxy config is already properly loaded):

# systemctl restart rsyslogd.service

To make sure rsyslog reloaded successfully:

systemctl status rsyslogd.service


Restarting HAproxy

If the rsyslogd logging to 127.0.0.1 port 514 was recently added a HAProxy restart should also be run, you can do it with:
 

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)


Or to restart use systemctl script (if haproxy is not used in a cluster with corosync / heartbeat).

# systemctl restart haproxy.service

You can control how much information is logged by adding a Syslog level by

    log         127.0.0.1 local2 info


The accepted values are the standard syslog security level severity:

Value Severity Keyword Deprecated keywords Description Condition
0 Emergency emerg panic System is unusable A panic condition.
1 Alert alert   Action must be taken immediately A condition that should be corrected immediately, such as a corrupted system database.
2 Critical crit   Critical conditions Hard device errors.
3 Error err error Error conditions  
4 Warning warning warn Warning conditions  
5 Notice notice   Normal but significant conditions Conditions that are not error conditions, but that may require special handling.
6 Informational info   Informational messages  
7 Debug debug   Debug-level messages Messages that contain information normally of use only when debugging a program.

 

Logging only errors / timeouts / retries and errors is done with option:

Note that if the rsyslog is configured to listen on different port for some weird reason you should not forget to set the proper listen port, e.g.:
 

  log         127.0.0.1:514 local2 info

option dontlog-normal

in defaults or frontend section.

You most likely want to enable this only during certain times, such as when performing benchmarking tests.

(or log-format-sd for structured-data syslog) directive in your defaults or frontend
 

Haproxy Logging shortly explained


The type of logging you’ll see is determined by the proxy mode that you set within HAProxy. HAProxy can operate either as a Layer 4 (TCP) proxy or as Layer 7 (HTTP) proxy. TCP mode is the default. In this mode, a full-duplex connection is established between clients and servers, and no layer 7 examination will be performed. When in TCP mode, which is set by adding mode tcp, you should also add option tcplog. With this option, the log format defaults to a structure that provides useful information like Layer 4 connection details, timers, byte count and so on.

Below is example of configured logging with some explanations:

Log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

haproxy-logged-fields-explained
Example of Log-Format configuration as shown above outputted of haproxy config:

Log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

haproxy_http_log_format-explained1

To understand meaning of this abbreviations you'll have to closely read  haproxy-log-format.txt. More in depth info is to be found in HTTP Log format documentation


haproxy_logging-explained

Logging HTTP request headers

HTTP request header can be logged via:
 

 http-request capture

frontend website
    bind :80
    http-request capture req.hdr(Host) len 10
    http-request capture req.hdr(User-Agent) len 100
    default_backend webservers


The log will show headers between curly braces and separated by pipe symbols. Here you can see the Host and User-Agent headers for a request:

192.168.150.1:57190 [20/Dec/2018:22:20:00.899] website~ webservers/server1 0/0/1/0/1 200 462 – – —- 1/1/0/0/0 0/0 {mywebsite.com|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.80 } "GET / HTTP/1.1"

 

Haproxy Stats Monitoring Web interface


Haproxy is having a simplistic stats interface which if enabled produces some general useful information like in above screenshot, through which
you can get a very basic in browser statistics and track potential issues with the proxied traffic for all configured backends / frontends incoming outgoing
network packets configured nodes
 experienced downtimes etc.

haproxy-statistics-report-picture

The basic configuration to make the stats interface accessible would be like pointed in above config for example to enable network listener on address
 

https://192.168.0.5:8080/stats


with hproxyuser / password config would be:

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

 

 

Sessions states and disconnect errors on new application setup

Both TCP and HTTP logs include a termination state code that tells you the way in which the TCP or HTTP session ended. It’s a two-character code. The first character reports the first event that caused the session to terminate, while the second reports the TCP or HTTP session state when it was closed.

Here are some essential termination codes to track in for in the log:
 

Here are some termination code examples most commonly to see on TCP connection establishment errors:

Two-character code    Meaning
—    Normal termination on both sides.
cD    The client did not send nor acknowledge any data and eventually timeout client expired.
SC    The server explicitly refused the TCP connection.
PC    The proxy refused to establish a connection to the server because the process’ socket limit was reached while attempting to connect.


To get all non-properly exited codes the easiest way is to just grep for anything that is different from a termination code –, like that:

tail -f /var/log/haproxy.log | grep -v ' — '


This should output in real time every TCP connection that is exiting improperly.

There’s a wide variety of reasons a connection may have been closed. Detailed information about all possible termination codes can be found in the HAProxy documentation.
To get better understanding a very useful reading to haproxy Debug errors with  is in haproxy-logging.txt in that small file are collected all the cryptic error messages codes you might find in your logs when you're first time configuring the Haproxy frontend / backend and the backend application behind.

Another useful analyze tool which can be used to analyze Layer 7 HTTP traffic is halog for more on it just google around.

Currency converter Free Desktop Software for Microsoft Windows

Saturday, December 8th, 2012

Business or not a business entity, in the mostly globalized world it is almost inevitable to go on at least monthly basis without a need for some kind of currency convertion. Of course there are plenty of websites allowing Free Money Convertion services out there. However as I'm not a big fan of the Software as a Service (SAS). I don't like other people to be able to sniff what kind of money, amounts I'm intending to convert as well as I don't like google or other search engine to profile me how frequently I'm converting or intending to convert money. Thus today I did a quick research what kind of Free and  Open Source Software FOSS is available to do the money convertion operation custom on my own webserver or my desktop PC. Though It was not exactly what I was looking for I found a Windows Desktop Software -CConverter which is capable of convertions between mostly if not all Currencies around the world. I'm not a m$ Windows user myself, but I was glad to know a Free Software exists for the task. CConverter is definitely a piece of soft useful for Businesses and People. In future if I do my own business it would be nice to know of this little handy soft existence and I will put it in action (to save costs) and add confidentiality to my business money transaction / convertions. Below is the two screenshots of CConverter I found on the project's sourceforge website:

Desktop Currency converter program cconverter windows desktop currency converter program

cconverter money desktop currency converter m$  windows software in action $$$

How to convert Adobe PDF file format to Microsoft Word DOC on MS Windows 2000 / XP / Vista / 7

Tuesday, July 24th, 2012

How to convert PDF to DOC on Microsoft Windows XP, MS Windows 7, Win Vista convert PDF to MS DOC 2003, ABBYY Covert Page
I had to convert Adobe PDF file to Microsoft Word ( .doc) file on Microsoft Windows OS for a friend. There is plenty of software available to convert PDF to DOC on Windows, as well as few web-site services claiming to convert correcly PDF to DOC. Converting PDF to DOC is easy and can be done with Open Office, however the reverse process is a real pain in the ass. I tried a dozen of free web serviecs to convert an ancient Latin writting PDF to DOC but none of them couldn’t properly convert it. Failing with the web services as a tool to convert, I’ve turned to seeking a tool that will do the trick. After trying few PDF to DOC converters which failed to produce a properly structed edittable DOC from the PDF file, I’ve come across ABBYY PDF Transformer 2.0. Abbyy PDF Transformer finally did it …

I’ve tried hard to look for a free software good PDF to DOC converter alternative for Windows but it seems as of time of writing this post there is no GPLed free software that does properly convert PDFs to MS WORD DOC ….

Using Abbyy PDF Transformer 2.0 is a piece of cake all I had to do is select the PDF file (pressing Open PDF) and then click on Convert (in right bottom corner). Below is a shot of Abby PDF transformer in action.

How to convert PDF to DOC on Microsoft Windows XP, MS Windows 7, Win Vista convert PDF to MS DOC 2003, abby pdf converter in action

Color Trick Microsoft and Google use to keep their users loyal and happy unwalfully

Tuesday, June 12th, 2012

Color mind influence has been longly researched. It is researched and there are some findingings on how we people react on colors. This researches are not much known and most of them are not put on the internet (??) One model claiming to have explained how colors influence is called HBDIHerrmann Brain Dominance Instruments.

In the picture beginning of this post, I have presented a quick "Personal Profile" of HBDI on how one think in order to determine in "which colors" one tends to think more ___

In short HBDI model claims to explain how people think in another model.
My personal view of it is it is like most science nowdays more based on faith than on a clearly conducted scientific research and facts. We know pretty well many people tried to explain how brain operates and many people give models to explain it however none of the models could grasp in completeness the complexity of human brain. Hence Businessman people who use this model in their daily life and they push it to us has put the model in action not that they know it is working but rather they believe it does .., Saying this few words as introduction I will contninue onwards to explain you about HBDI as in the business world it is considered as a "Strategic Asset" for a company success. Hence the use of richest companies of the model has a serious impact on us the common people and unknowing (uninformed) computer users.

Some of the companies who integrated the HBDI to their models we all know are of course not strangely Microsoft and Google
;;;

Below I present you a picture showing the HBDI The Whole Brain Model |||

HBDI The whole brain model

Next I show you Microsoft Windows OS worldly "infamous" flag |||

Microsoft Windows OS Flag

You can see for yourself the basic color from HBDI WHOLE brain model are integrated in the Microsoft flag, only the order of colors present and the color gamma is different;;;;

The basic colors in HBDI model to explain how human brain works is separated in 4 segments as you can see from above screenshot. There are a number of tests one can do to determine what is his exact HBDI profile, and in abstract terms in which kind of colors he prefers to think.

There are a whole "army" of people involved into this sect like philosophy (I call it philosophy as surely every model that tries to explain everything is doomed to fail it is the nature in which God created the universe so complex and he put us be part of it and not controllers of it that any Universal model trying to explain it has never succeeded so far. The HBDI has some fruits for the only reason it is believed to work well by the people with money.

As you see in the colors HBDI claims there are 4 segments corresponding to four basic colors

  • BLUE
  • YELLOW
  • GREEN
  • RED

Each of the colors is an indicator on how the person tends to think the BLUE people as HBDI practicioners (believers) calls them are —

Analytical, Fact Based, Logal, Quantitative

The YELLOW oriented people are claimed to be —

Holistic, Intuitive, Integrating, Synthesizing

The GREEN ones in model terms are interested in —

Organizing, Sequentiality, Planning, Detailizing facts

Finally the RED Ones are said to be —

Interpersonal, Feeling based, Kinesthetic, Emotional

Now as you can understand this model though it looks like promising is based on a philosophy which rejects the existing of spirit realm God Angels or good or evil. It claims everything we're are or we want to be can be achieved following the HBDI to develop your own brain.

This model as every human made model however does reject the fact that besides internal factors and brains we're put into external environment most of which we cannot control and therefore even if we try our best to have certain goals and complete them the external uncontrolled facts can be a reason to stop us to complete our goals.

Now back to my point, that Google, Microsoft and probably many other products and physical goods are heavily using the HBDI color scheme ;;;
Here is the Google Inc. Logo the color trait of HBDI is there:

Google Search Engine Logo and HBDI 4 colors embedded

For those doubting that Google Inc. and Microsoft Inc. are along the false believers of HBDI color scheme brain ideology I present below the Logo of Google Web Browser =- Google Chrome

Google Web Chrome Browser Logo 4 colors HBDI microsoft flag

It is evident 4 colors used as a main ones in the HBDI tool are present in Google Chrome just like in Microsoft Windows logo flag, the only difference is in the order of colors.
Also it is interesting the name Chrome that Google Chrome took is most likely taken from Aldous Huxley's – Brave New World (A book depicting a short future highly conditioned society) , the book story line goes around a society programmed to do the things they do.

I assume it is very likely that Google's founders Sergey Brin or some of their subordinate working for Google are very much into the idea of conditioning people just like in the book and this is most likely the reason they choose the Chrome as a title for Google's browser ,,,

The 4 Colors from HBDI yellow, green, blue, red are embedded also in the google .ico file (the little icon showing in browser URL bar), below is a screenshot of a tab where google is opened showing the .ico image:

Google Icon 4 colors Linux Debian Epiphany Browser tab screenshot

Do you remember the good old Windows XP start button, have you noticed the Windows flag embedded in it, if not let me show you;;;

Microsoft Windows XP Start Button and HBDI 4 colors scheme

But wait the Windows flag placed on the left bottom of Ms Windows-es is not only on XP it is also on Windows 3.11 cover, Winblows 98, Vista, Windows 2003, Windows 2007 and actually all the M$ operating systems ever produced since the very early days M$ become a top OS producer :::

Windows 3.11 Operating system logo flag

Microsoft Windows 95 4 colors flag and blue sky

Here is also the 4 colored (a bit like Nazi like looking) flag on M$ Win-doze 7 |||

MS Windows 7 start Menu m$ windows well known flag

Also the Microsoft Flag is positioned on the bottom left screen on purpose. It is well known fact that most of the world (except Arabic) are used to read the text from Left to Right

, therefore it is natural for our eyesight to look for the text on the left side. I just wonder why they placed the START on the bottom and not on top. It is natural we read text and books from the most top to the most bottom ,,,.,

Even Apple Computers nowdays Macs has most likely used the HBDI as the main 4 colors and some gamma from rainbow colors are present on their Classical Apple Computer logo

Old Apple Computer/s logo colors of rainbow 4 hbdi colors are there

Makes me wonder if Jobs employed the HBDI model in his company. Well what is the reason for people loving so much this rainbow colors combination. If we think for a second outside of HBDI's brainwashing ideology for what each color would stand for. Well it is simple is comes from our young years most of the people between age 2 and 50 years has been more or less exposed to the so colorful Kids Cartoons, which are all so colorfully painted. Since our very early age we've placed in us a love for colorfulness outlook (well again not all of us for example I prefer less colors, I'm sure there are plenty of people who don't like the heavy colors we see in almost everywhere around us).

The problem with this 4 colors use on purpose and all this unnatural color placing everywhere is that it is unnatural and not in good synergy with our surrounding natural environment. Therefore I personally think using a colorful color paintings on everywhere in both computer programs and the physical world plays us a bad joke and is one of the reasons so many people are on the virge to get crazy nowdays and many have already had already cracked out.

It is my firm believe more and more people should be educated on the harm of HBDI and the fact that, we're forced to 'live it' unwilfully every day by using even as "simple things" as computers and daily technology or buying food in the super market ,,,

How to show country flag, web browser type and Operating System in WordPress Comments

Wednesday, February 15th, 2012

!!! IMPORTANT UPDATE COMMENT INFO DETECTOR IS NO LONGER SUPPORTED (IS OBSOLETE) AND THE COUNTRY FLAGS AND OPERATING SYSTEM WILL BE NOT SHOWING INSTEAD,

!!!! TO MAKE THE COUNTRY FLAGS AND OS WP FUNCTIONALITY WORK AGAIN YOU WILL NEED TO INSTALL WP-USERAGENT !!!

I've come across a nice WordPress plugin that displays country flag, operating system and web browser used in each of posted comments blog comments.
Its really nice plugin, since it adds some transperancy and colorfulness to each of blog comments 😉
here is a screenshot of my blog with Comments Info Detector "in action":

Example of Comments Info Detector in Action on wordpress blog comments

Comments Info Detector as of time of writting is at stable ver 1.0.5.
The plugin installation and configuration is very easy as with most other WP plugins. To install the plugin;

1. Download and unzip Comments Info Detector

linux:/var/www/blog:# cd wp-content/plugins
linux:/var/www/blog/wp-content/plugins:# wget http://downloads.wordpress.org/plugin/comment-info-detector.zip
...
linux:/var/www/blog/wp-content/plugins:# unzip comment-info-detector.zip
...

Just for the sake of preservation of history, I've made a mirror of comments-info-detector 1.0.5 wp plugin for download here
2. Activate Comment-Info-Detector

To enable the plugin Navigate to;
Plugins -> Inactive -> Comment Info Detector (Activate)

After having enabled the plugin as a last 3rd step it has to be configured.

3. Configure comment-info-detector wp plugin

By default the plugin is disabled. To change it to enabled (configure it) by navigating to:

Settings -> Comments Info Detector

Next a a page will appear with variout fields and web forms, where stuff can be changed. Here almost all of it should be left as it is the only change should be in the drop down menus near the end of the page:

Display Country Flags Automatically (Change No to Yes)
Display Web Browsers and OS Automatically (Change No to Yes

Comments Info Detector WordPress plugin configuration Screenshot

After the two menus are set to "Yes" and pressing on Save Changes the plugin is enabled it will immediately start showing information inside each comment the GeoIP country location flag of the person who commented as well as OS type and Web Browser 🙂

Bulgaria silently signs for ACTA / Why ACTA, SOPA and PIPA are bad for our freedom

Thursday, February 2nd, 2012

r freedomYesterday silently with zero publicity, Bulgarian representatives ratified the ACTA (Trade agreement for fighting counterfeit.)
The name sounds really good, but it has not much to do with what ACTA is about, when applied to digital medias and data sharing.
The ACTA legislation has been ratified in Tokyo last week, where 22 of the European Union membership countries signed in favour of these "malicious" treaty.

The basic idea of ACTA looks tempting as it gives more freedoms to copyright holders, however if you look closely you will understand actually this copyright infringement clauses are not so in favour of us the users but mostly in favour of multinational corporations.
For all those who have not heard about ACTA and SOPA in short this is anti freedom of speech treaty, which if put in action could lead to serious filtering of the internet.
The ACTA 's controversial treaty has already raised an outcry from dozens of computer literated individuals who daily use the internet. Unfortunately, ACTA is less known among non-tech guys … and hence most people on the internet have no about its existence.

If ACTA is ratified and set to be valid as a legislation to Bulgaria, this could lead to total Internet censorship in BG (more or less like it is in china now).
ACTA legislation will make sharing files via torrents and other P2P community file sharing networks a criminal activity.
Another effect of ACTA is that practically free software which reads a proprietary formats like DVD becomes illegal in Europe (like it is currently in America) and I will become guilty for just reading the non-free format..
As a result of ACTA our ISP (Internet Service Providers) will be forced to log and keep all traffic flowing through their (Routering servers). Filters on a local ISP level that will be censoring free speech could also become totally lawful…
Already there are plenty of ANTI-ACTA and ANTI-SOPA propaganda website which are trying to bring some more awareness to the public for the issue… Once an individual is suspected, to fraudulent activity or anything that breaks what is in ACTA is he is presumed to be guilty of crome …
Just watch the two videos below and you will see how terrible the consequence could be if this legislation is integrated with todays Bulgarian government laws. If you're hearing for ACTA for a first time and you live in a country which has still not rafitied ACTA as a local country legislation, make sure you spread the word and let all your friends about the bad impact of this anti-human legislation. We have to really stand up and protest to retain our digital freedom !

The Internet can be censored if Protect IP ACT (PIPA) and Stop Online Piracy Act (SOPA) are put in action !

ANTI-ACTA – Hmmm But What can you do??
 

Cause and solution for Qmail sent error “Requested action aborted: error in processing Server replied: 451 qq temporary problem (#4.3.0)”

Friday, October 28th, 2011

One of the qmail servers I manage today has started returning strange errors in Squirrel webmail and via POP3/IMAP connections with Thunderbird.

What was rather strange is if the email doesn’t contain a link to a webpage or and attachment, e.g. mail consists of just plain text the mail was sent properly, if not however it failed to sent with an error message of:

Requested action aborted: error in processing Server replied: 451 qq temporary problem (#4.3.0)

After looking up in the logs and some quick search in Google, I come across some online threads reporting that the whole issues are caused by malfunction of the qmail-scanner.pl (script checking mail for viruses).

After a close examination on what is happening I found out /usr/sbin/clamd was not running at all?!
Then I remembered a bit earlier I applied some updates on the server with apt-get update && apt-get upgrade , some of the packages which were updated were exactly clamav-daemon and clamav-freshclam .
Hence, the reason for the error:

451 qq temporary problem (#4.3.0)

was pretty obvious qmail-scanner.pl which is using the clamd daemon to check incoming and outgoing mail for viruses failed to respond, so any mail which contained any content which needed to go through clamd for a check and returned back to qmail-scanner.pl did not make it and therefore qmail returned the weird error message.
Apparently for some reason apparently the earlier update of clamav-daemon failed to properly restart, the init script /etc/init.d/clamav-daemon .

Following fix was very simple all I had to do is launch clamav-daemon again:

linux:~# /etc/inid.d/clamav-daemon restart

Afterwards the error is gone and all mails worked just fine 😉

How to find out all programs bandwidth use with (nethogs) top like utility on Linux

Friday, September 30th, 2011

Just run across across a super nice top like, program for system administrators, its called nethogs and is definitely entering my “l337” admin outfit next to tools like iftop, nettop, ettercap, darkstat htop, iotop etc.

nethogs is ultra easy to use, to get immediately in console statistics about running processes UPLOAD and DOWNLOAD bandwidth consumption just run it:

linux:~# nethogs

Nethogs screenshot on Linux Server with Nginx
Nethogs running on Debian GNU/Linux serving static web content with Nginx

If you need to check what program is using what amount of network bandwidth, you will definitely love this tool. Having information of bandwidth consumption is also viewable partially with iftop, however iftop is unable to track the bandwidth consumption to each process using the network thus it seems nethogs is unique at what it does.

Nethogs supports IPv4 and IPv6 as well as supports network traffic over ppp. The tool is available via package repositories for Debian GNU/Lenny 5 and Debian Squeeze 6.

To install Nethogs on CentOS and Fedora distributions, you will have to install it from source. On CentOS 5.7, latest nethogs which as of time of writting this article is 0.8.0 compiles and installs fine with make && make install commands.

In the manner of thoughts of network bandwidth monitoring, another very handy tool to add extra understanding on what kind of traffic is crossing over a Linux server is jnettop
jnettop shows which hosts/ports is taking up the most network traffic.
It is available for install via apt in Debian 5/6).

Here is a screenshot on jnettop in action:

Jnettop check network traffic in console

To install jnettop on latest Fedoras / CentOS / Slackware Linux it has to be download and compiled from source via jnettop’s official wiki page
I’ve tested jnettop install from source on CentOS release 5.7 and it seems to compile just fine using the usual compile commands:

[root@prizebg jnettop-0.13.0]# ./configure
...
[root@prizebg jnettop-0.13.0]# make
...
[root@prizebg jnettop-0.13.0]# make install

If you need to have an idea on the network traffic passing by your Linux server distringuished by tcp/udp/icmp network protocols and services like ssh / ftp / apache, then you will definitely want to take a look at nettop (if of course not familiar with it yet).
Nettop is not provided as a deb package in Debian and Ubuntu, where it is included as rpm for CentOS and presumably Fedora?
Here is a screenshot on nettop network utility in action:

Nettop server traffic division by protocol screenshot
FreeBSD users should be happy to find out that jnettop and nettop are part of the ports tree and the two can be installed straight, however nethogs would not work on FreeBSD, I searched for a utility capable of what Nethogs can, but couldn’t find such.
It seems the only way on FreeBSD to track bandwidth back and from originating process is using a combination of iftop and sockstat utilities. Probably there are other tools which people use to track network traffic to the processes running on a hos and do general network monitoringt, if anyone knows some good tools, please share with me.

How to find out which processes are causing a hard disk I/O overhead in GNU/Linux

Wednesday, September 28th, 2011

iotop monitor hard disk io bottlenecks linux
To find out which programs are causing the most read/write overhead on a Linux server one can use iotop

Here is the description of iotop – simple top-like I/O monitor, taken from its manpage.

iotop does precisely the same as the classic linux top but for hard disk IN/OUT operations.

To check the overhead caused by some daemon on the system or some random processes launching iotop without any arguments is enough;

debian:~# iotop

The main overview of iostat statistics, are the:

Total DISK READ: xx.xx MB/s | Total DISK WRITE: xx.xx K/s
If launching iotop, shows a huge numbers and the server is facing performance drop downs, its a symptom for hdd i/o overheads.
iotop is available for Debian and Ubuntu as a standard package part of the distros repositories. On RHEL based Linuxes unfortunately, its not available as RPM.

While talking about keeping an eye on hard disk utilization and disk i/o’s as bottleneck and a possible pitfall to cause a server performance down, it’s worthy to mention about another really great tool, which I use on every single server I administrate. For all those unfamiliar I’m talking about dstat

dstat is a – versatile tool for generating system resource statistics as the description on top of the manual states. dstat is great for people who want to have iostat, vmstat and ifstat in one single program.
dstat is nowdays available on most Linux distributions ready to be installed from the respective distro package manager. I’ve used it and I can confirm tt is installable via a deb/rpm package on Fedora, CentOS, Debian and Ubuntu linuces.

Here is how the tool in action looks like:

dstat Linux hdd load stats screenshot

The most interesting things from all the dstat cmd output are read, writ and recv, send , they give a good general overview on hard drive performance and if tracked can reveal if the hdd disk/writes are a bottleneck to create server performance issues.
Another handy tool in tracking hdd i/o problems is iostat its a tool however more suitable for the hard core admins as the tool statistics output is not easily readable.

In case if you need to periodically grasp data about disks read/write operations you will definitely want to look at collectl i/o benchmarking tool .Unfortunately collect is not included as a packaget for most linux distributions except in Fedora. Besides its capabilities to report on servers disk usage, collect is also capable to show brief stats on cpu, network.

Collectl looks really promosing and even seems to be in active development the latest tool release is from May 2011. It even supports NVidia’s GPU monitoring 😉 In short what collectl does is very similar to sysstat which by the way also has some possibilities to track disk reads in time.  collectl’s website praises the tool, much and says that in most machines the extra load the tool would add to a system to generate reports on cpu, disk and disk io is < 0.1%.  I couldn’t find any data online on how much sysstat (sar) extra loads a system. It will be interesting if some of someone concluded some testing and can tell which of the two puts less load on a system.