Posts Tagged ‘server instances’

How to log multiple haproxy server instance processes on single server in seperate files with rsyslog filters

Monday, February 3rd, 2025

haproxy-log-frontend-backend-and-transferred-connections-in-separate-log-files-on-linux-server-logo

Lets say you want to have 2 separates instances of haproxy and log the output to separate files, how this can be achived?

In this article, i'll tell in few easy steps how to enable multiple haproxy server instances created on the same Linux server / VPS or docker container to run and log its served content in separate log files without using separate file logging handlers "local"s.
The task might be helpful for people who are involved with DevOps and has to route separate proxy traffic on same linux machine.
 

Lets say you have the following haproxy process instances running with separate haproxy configs:
 

1. haproxy
2. haproxy_worker2
3. haproxy_worker3
 

List of processes on the Linux host would looks like that.

[root@linux-server rsyslog.d]# ps -ef|grep -i hap
root     1151275 1147138  0 11:58 pts/2    00:00:00 grep –color=auto -i hap
root     1835200       1  0 Jan30 ?        00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  1835203 1835200  0 Jan30 ?        00:10:41 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
root     1835216       1  0 Jan30 ?        00:00:00 /usr/sbin/haproxy_worker2 -Ws -f /etc/haproxy/haproxy_worker2.cfg -p /run/haproxy_worker2.pid
haproxy  1835219 1835216  0 Jan30 ?        00:02:46 /usr/sbin/haproxy_worker2 -Ws -f /etc/haproxy/haproxy_worker2.cfg -p /run/haproxy_worker2.pid
root     1835216       1  0 Jan30 ?        00:00:00 /usr/sbin/haproxy_worker3 -Ws -f /etc/haproxy/haproxy_worker3.cfg -p /run/haproxy_worker3.pid
haproxy  1835219 1835216  0 Jan30 ?        00:02:46 /usr/sbin/haproxy_worker3 -Ws -f /etc/haproxy/haproxy_worker3.cfg -p /run/haproxy_worker3.pid

Question is how to log the 3 haproxies passed through configured connection IP and frontend / backend outputs to separate files

 /var/log/haproxy.log , /var/log/haproxy_worker2.log and /var/log/haproxy_worker3.log


To achieve the task, you will need to set-up 3 rsyslog config files name it according to your preferences and make sure no other rsyslog
file with haproxy related configuration does not mess up with the configs (e.g. is not having a config start number NUMBER_file.conf prior to the below created files.

Then create lets say 49_haproxy.conf and 50_haproxy_worker2.conf and 51_haproxy_worker3.conf

[root@linux-server rsyslog.d]# cat 48_haproxy.conf
#$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
#2022/02/02: HAProxy logs to local6, save the messages
# Template to include only the timestamp in HAProxy logs
template(name="HaproxyTimestampOnly" type="string" string="%timegenerated% %msg:::drop-last-lf%\n")
local6.*                /var/log/haproxy.log;HaproxyTimestampOnly
# Apply the template to HAProxy prod port mapping logs
#if $programname startswith 'haproxy[' then /var/log/haproxy.log;HaproxyTimestampOnly
& stop

[root@linux-server rsyslog.d]# cat 50_haproxy_worker2.conf
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
# Template to include only the timestamp in HAProxy logs
template(name="HaproxyTimestampOnly" type="string" string="%timegenerated% %msg:::drop-last-lf%\n")

# Apply the template to HAProxy prod port mapping logs
if $programname startswith 'haproxy_worker2' then /var/log/haproxy_worker2.log;HaproxyTimestampOnly

 

[root@linux-server rsyslog.d]# cat 51_haproxy_worker3.conf
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
# Template to include only the timestamp in HAProxy logs
template(name="HaproxyTimestampOnly" type="string" string="%timegenerated% %msg:::drop-last-lf%\n")

# Apply the template to HAProxy prod port mapping logs
if $programname startswith 'haproxy_worker3' then /var/log/haproxy_worker3.log;HaproxyTimestampOnly

Those rsyslog configs permissions has to be as follows:

[root@linux-server home]# ls -al /etc/rsyslog.d/48_haproxy.conf
-rw-r–r– 1 root root 488 Jan 30 12:44 /etc/rsyslog.d/48_haproxy.conf
[root@linux-server home]# ls -al /etc/rsyslog.d/50_haproxy_worker2.conf
-rw-r–r– 1 root root 379 Jan 30 12:45 /etc/rsyslog.d/50_haproxy_worker2.conf
[root@linux-server home]# ls -al /etc/rsyslog.d/51_haproxy_worker2.conf
-rw-r–r– 1 root root 379 Jan 30 12:45 /etc/rsyslog.d/51_haproxy_worker2.conf

 

The permissions for files to log the haproxy has to be as so:

[root@linux-server home]# ls -al /var/log/haproxy.log
-rw-r—– 1 haproxy haproxy 5014349 Feb  3 12:11 /var/log/haproxy.log
[root@linux-server home]# ls -al /var/log/haproxy_worker2.log
-rw-r—– 1 root root 728139 Feb  3 12:11 /var/log/haproxy_worker2.log
[root@linux-server home]# ls -al /var/log/haproxy_worker3.log
-rw-r—– 1 root root 728139 Feb  3 12:11 /var/log/haproxy_worker3.log

To make the changes take affect restart consequentially rsyslog first and then the 3 haproxy instances:

[root@linux-server home]# systemctl restart rsyslog
[root@linux-server home]# systemctl restart haproxy
[root@linux-server home]# systemctl restart haproxy2
[root@linux-server home]# systemctl restart haproxy3

Go on and check the logs that everything comes in from the haproxys running the same server into the separate files:

[root@linux-server home]# tail -f /var/log/haproxy.log /var/log/haproxy_worker2.log /var/log/haproxy_worker3.log
…

Hope this has helped someone out there looking to solve on how to log multiple haproxy instances on the same servers into separate files.

That's all folks. Enjoy!

Tags: , , , , , , , , ,
Posted in Everyday Life, Haproxy, Linux | No Comments »

How to enable output compression (gzipfile content compression) in nginx webserver

Friday, April 8th, 2011

I have recently installed and configured a Debian Linux server with nginx
. Since then I’ve been testing around different ways to optimize the nginx performance.

In my nginx quest, one of the most crucial settings which dramatically improved the end client performance was enabling the so called output compression which in Apache based servers is also known as content gzip compression .
In Apache webservers the content gzip compression is provided by a server module called mod_deflate .

The output compression nginx settings saves a lot of bandwidth and though it adds up a bit more load to the server, the plain text files like html, xml, js and css’s download time reduces drasticly as they’re streamed to the browser in gzip compressed format.
This little improvement in download speed also does impact the overall end user browser experience and therefore improves the browsing speed experience with websites.

If you have already had experience nginx you already know it is a bit fastidious and you have to be very careful with it’s configuration, however thanksfully enabling the gzip compression was actually rather easier than I thought.

Here is what I added in my nginx config to enable output compression:

## Compression
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 9;
gzip_http_version 1.1;
gzip_min_length 0;
gzip_vary on;

Important note here is that need to add this code in the nginx configuration block starting with:

http {
....
## Compression
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 9;
gzip_http_version 1.1;
gzip_min_length 0;
gzip_vary on;

In order to load the gzip output compression as a next step you need to restart the nginx server, either by it’s init script if you use one or by killing the old nginx server instances and starting up the nginx server binary again:
I personally use an init script, so restarting nginx for me is done via the cmd:

debian:~# /etc/init.d/nginx restart
Restarting nginx: nginx.

Now to test if the output gzip compression is enabled for nginx, you can simply use telnet

hipo@linux:~$ telnet your-nginx-webserver-domain.com 80
Escape character is '^]'.

After the Escape character is set ‘^]’ appears on your screen type in the blank space:

HEAD / HTTP/1.0

and press enter twice.
The output which should follow should look like:


HTTP/1.1 200 OK
Server: nginx
Date: Fri, 08 Apr 2011 12:04:43 GMT
Content-Type: text/html
Content-Length: 13
Last-Modified: Tue, 22 Mar 2011 15:04:26 GMT
Connection: close
Vary: Accept-Encoding
Expires: Fri, 15 Apr 2011 12:04:43 GMT
Cache-Control: max-age=604800
Accept-Ranges: bytes

The whole transaction with telnet command issued and the nginx webserver output should look like so:

hipo@linux:~$ telnet your-nginx-webserver-domain.com 80
Trying xxx.xxx.xxx.xxx...
Connected to your-nginx-webserver-domain.com
.Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 08 Apr 2011 12:04:43 GMT
Content-Type: text/html
Content-Length: 13
Last-Modified: Tue, 22 Mar 2011 15:04:26 GMT
Connection: close
Vary: Accept-Encoding
Expires: Fri, 15 Apr 2011 12:04:43 GMT
Cache-Control: max-age=604800
Accept-Ranges: bytes

The important message in the returned output which confirms your nginx output compression is properly configured is:

Vary: Accept-Encoding

If this message is returned by your nginx server, this means your nginx now will distribute it’s content to it’s clients in compressed format and apart from the browsing boost a lot of server and client bandwitdth will be saved.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration, Web and CMS | 3 Comments »