Posts Tagged ‘apache mysql’

How to log multiple haproxy / apache / mysql instance via haproxy log-tagging / Segregating log management for multiple HAProxy instances using rsyslog

Tuesday, May 23rd, 2023

rsyslog-logo-picture-use-programname-and-haproxy-log-tag-directives-together-to-log-as-many-process-streams-as-you-like

 

Introduction

This article provides a guide on refining haproxy  logging mechanism by leveraging the `programname` property in rsyslog, coupled with the `log-tag` directive in haproxy.
This approach will create a granular logging setup, separating logs according to their originating services and specific custom tags, enhancing overall log readability.

Though the article is written concretely for logging multiple log streams from haproxy this can be successfully applied
for any other Linux service to log as many concrete log-tagged data streams as you prefer.

Scope

The guide focuses on tailoring the logging mechanisms for two haproxy  instances named `haproxy` and `haproxyssl`, utilizing the `programname` property in rsyslog and the `log-tag` directive in haproxy for precise log management.

The haproxy and haproxyssl instances are two separate systemd config file prepared instances.
haproxy instance is simple haproxy proxying tcp traffic in non-encrypted form, whether haproxyssl is a special instance
prepared to tunnel the incoming http traffic in ssl form. Both instances of haproxy runs as a separate processes on the server.

Here is the systemd configuration of haproxy systemd service file:

# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target


As well as the systemd service configuration for haproxyssl:
 

# cat /usr/lib/systemd/system/haproxyssl.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy_ssl_prod.cfg" "PIDFILE=/run/haproxy_ssl_prod.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxyssl -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target

 

Step 1: Configuring HAProxy instances with `log-tag`
 

To distinguish between logs from two HAProxy instances, `log-tag` directive is used to add tags to logs. This tag is used to filter these logs in rsyslog.
Modify the HAProxy configuration file in `/etc/haproxy/haproxy.*.cfg`

HAProxy Instance 1 (haproxy)
 

#———————————————————————
# Global settings
#———————————————————————
global
      log          127.0.0.1 local6 debug
      log-tag      haproxy

HAProxy Instance 2 (haproxyssl)


#———————————————————————
# Global settings
#———————————————————————
global
    log          127.0.0.1 local5 debug
    log-tag      haproxyssl

 

Step 2: Implementing rsyslog configuration for haproxy logs
 

Next, create a new rsyslog configuration file, stored in /etc/rsyslog.d/. Ensure the new configuration file ends in `.conf`

HAProxy Instance 1 (haproxy)

Now add rsyslog rules to filters logs based on the `programname` and the custom log tag:
 

# vi /etc/rsyslog.d/55_haproxy.conf
if $programname == 'haproxy' then /var/log/haproxy.log
&stop

HAProxy Instance 2 (haproxyssl)
# vi /etc/rsyslog.d/51_haproxy_ssl.conf
if $programname == 'haproxy_ssl' then /var/log/haproxy_ssl.log
&stop


These rules filter logs that originate from haproxy  and contain the respective string haproxy   or haproxy_ssl , directing them to their respective log files. The `& stop` directive ensures that rsyslog stops processing the log once a match is found, preventing dublication.

Finally, restart both the haproxy and rsyslog services for the changes to take effect:

# systemctl restart haproxy
# systemctl restart haproxyssl
# systemctl restart rsyslog


Reading References

haproxy:   log-tag directive

rsyslog:    rsyslogd documentation

This is a guest article originally written by: Dimitar Paskalev, guest blogging with good interesting articles is always mostly welcome 

How to harden Linux Security and imprpove network efficiency on Kernel sysctl Level to Stop SYN flood

Friday, July 8th, 2011

Power up Linux and protect against DDoS with sysctl var optimization

Some long time ago I’ve written an article Optimizing Linux tcp/ip networking

In the article I’ve examined a number of Linux kernel sysctl variables, which significantly improve the way TCP/IP networking is handled by a non router Linux based servers.

As the time progresses I’ve been continuing to read materials on blogs and internet sites on various tips and anti Denial of Service rules which one could apply on newly installed hosting (Apache/MySql/Qmail/Proxy) server to improve webserver responce times and tighten the overall security level.

In my quest for sysctl 😉 I found a few more handy sysctl variables apart from the old ones I incorporate on every Linux server I adminstrate.
The sysctl variables improves the overall network handling efficiency and protects about common SYN/ACK Denial of service attacks.

Here are the extra sysctl variables I started incorporating just recently:

############ IPv4 Sysctl Settings ################
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
##################################################
#
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routers
net.ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignments
net.ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interface
net.ipv6.conf.default.max_addresses = 1
#####################################################

 

To use this settings paste the above sysctl variables in /etc/sysctl.conf and ask sysctl command to read and apply the newly added conf settings:

server:~# sysctl -p
...

Hopefully you should not get errors while applying the sysctl settings, if you get some errors, it’s possible some of the variable is differently named (depending on the Linux kernel version) or the Linux distribution on which sysctl’s are implemented.

For some convenience I’ve created unified sysctl variables /etc/sysct.conf containing the newly variables I started implementing to servers with the ones I already exlpained in my previous post Optimizing Linux TCP/IP Networking

Here is the optimized / hardened sysctl.conf file for download

I use this exact sysctl.conf these days on both Linux hosting / VPS / Mail servers etc. as well as on my personal notebook 😉

Here is also the the complete content of above’s sysctl.conf file, just in case if somebody wants to directly copy/paste it in his /etc/sysctl.conf

# Sysctl kernel variables to improve network performance and protect against common Denial of Service attacks
# It's possible that not all of the variables are working on all Linux distributions, test to make sure
# Some of the variables might need a slight modification to match server hardware, however in most cases it should be fine
# variables list compiled by hip0
### https://www.pc-freak.net
#### date 08.07.2011
############ IPv4 Sysctl Kernel Settings ################
net.ipv4.ip_forward = 0
# ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0
# ( same as above )
net.ipv4.conf.default.accept_source_route = 0
# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
# ( - || - )net.ipv4.tcp_fin_timeout = 40
# ( Decrease FIN timeout ) - Useful on busy/high load server
net.ipv4.tcp_keepalive_time = 4000
# ( keepalive tcp timeout )
net.core.rmem_default = 786426
# Receive memory stack size ( a good idea to increase it if your server receives big files )
##net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608
#( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
# ( maximum amount of option memory buffers )
# tcp reordering, increase max buckets, increase the amount of backlost
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
##net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
###################################################
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routersnet.
ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignmentsnet.
ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interfacenet.
ipv6.conf.default.max_addresses = 1
#####################################################
# Reboot if kernel panic
kernel.panic = 20

These sysctl settings will tweaken the Linux kernel default network settings performance and you will notice the improvements in website responsiveness immediately in some cases implementing this kernel level goodies will make the server perform better and the system load might decrease even 😉

This optimizations on a kernel level are not only handy for servers, their implementation on Linux Desktop should also have a positive influence on the way the network behaves and could improve significantly the responce times of opening pages in Firefox/Opera/Epiphany Torrent downloads etc.

Hope this kernel tweakenings are helpful to someone.
Cheers 😉

How to fix php “Fatal error: Class ‘SimpleXMLElement’ not found” and “Fatal error: Class ‘JLoader’ not found” on FreeBSD

Tuesday, June 21st, 2011

One of the contact forms running on a FreeBSD server configured to work on top of Apache+MySQL suddenly stopped working.

The errors that appeared on the webpage during a page request to the form url was:

Fatal error: Class 'SimpleXMLElement' not found in /var/www/joomla/plugins/system/plugin_googlemap2_helper.php on line 2176 Fatal error: Class 'JLoader' not found in /var/www/joomla/plugins/libraries/loader.php on line 161

As you see in the output the website which was causing the issues was running a Joomla version 1.5.23 Stable configured with RSForm!ver 1.5.x (as a contact form solution) and Google Maps version 2.13b plugins.

The Google Map from Google Maps plugin and the RSform were configured to appear on one physical configured article in Joomla and seemed to work just until now. However yesterday suddenly the error messages:
Fatal error: Class ‘SimpleXMLElement’ not found
Fatal error: Class ‘JLoader’ not found

came out of nothing, it’s really strange as I don’t remember doing any changes to either Joomla or the PHP installation on this server.
There is one more guy who has access to the Joomla installation which I suspect might have changed something in the Joomla, but this scenario is not very likely.

Anyways as the problem was there I had to fix it up. Obviously as the error message Fatal error: Class ‘SimpleXMLElement’ not found reported the server php simplexml was missing!

Just to assure myself the php simplexml extension is not present on the server I used the classical method of setting up a php file with phpinfo(); in it to check all the installed php extensions on the server.

Finally to solve the issue I had to install the module from ports php5-simplexml , e.g.:

freebsd# cd /usr/ports/textproc/php5-simplexml
freebsd# make install clean

Afterwards to make the new settings take place I did restart of my Apache server:

freebsd# /usr/local/etc/rc.d/apache2 restart
Syntax OK
Stopping apache2.
Waiting for PIDS: 63883.
Performing sanity check on apache2 configuration:
Syntax OK
Starting apache2.

Now my Joomla contact form is back to normal 😉

If someone has any idea why this error occured without any php or server modifications, and how comes that all worked fine beforehand even though I did not have the simplexml module instlaled on the server o_O, I would be enormously greatful.

Reboot Safely a frozen (hanged up) Linux Server with Magic SysRQ Key combination

Monday, June 20th, 2011

Linux Magic SysRQ Key picture

Being a remote system administrator for many years from now, I’ve experienced many times sudden Linux crashes, mostly caused by system hardware overloads.

This is a common thing especially on busy Web Servers with (Apache/MySQL). Situations like this are really messy and many times in order to bringt the server back online, one has to either physically go to the Data Center or contact the Technical support to request a server reboot.

In this terrible times, data loss might occur if the Server reset button is used to cold reboot it.

Happily in 99% of the cases the data loss which might occur could be prevented with Linux’s kernel capability to support the Magic SysRQ key !

Having the Magic SysRQ key functionality being supported in Linux in past times wasn’t that common, thanksfully these days this has changed andlmost every Linux distrubution supports this handy feature.

Now you might wonder what is the magic with those Magic SysRQ key ?

Let me explain, Magic SysRQ is a kernel level functionality which supports even completely crashed Linux systems with the horrifying:

Kernel Panic
message to be properly shutdown.

Using Magic SysRQ instead of the mostly used indiced cold reboots is really advantageous, as all the opened files by programs on the crashed server which hanged will be properly saved and closed and thus possible data loss caused by the sudden server crash will be minimized.

One other good thing about the Magic SysRQ Key keyboard combination is that initiating the Magic SysRQ could be made blindly (no need for server monitor or display to present any error messages on the failed server).

Now to check if the magic sysrq is enabled on a server one should issue the command:

Here are some Magic SysRQ keyboard combinations one can use in case of server failure:

ALT+SYSRQ+M to dump memory info;
ALT+SYSRQ+P to dump processes states;
ALT+SYSRQ+S to sync disks;
ALT+SYSRQ+U to unmount all mounted filesystems;
ALT+SYSRQ+E to terminate processes;
ALT+SYSRQ+I to kill all processes
ALT+SYSRQ+U to try to unmount once again;
ALT+SYSRQ+B to reboot.

I would skip to explain what each of the keyboard combinations will do as I believe the above description explains it well.

One classics of combinations one might want to issue on a failed Linux server supporting the Magic SysRQ would be:

ALT+SYSRQ+R
ALT+SYSRQ+E
ALT+SYSRQ+I
ALT+SYSRQ+S
ALT+SYSRQ+U
ALT+SYSRQ+B

The ALT+SYSRQ+REISUB key combination is very popular among system administrators.
Also Magic SysRQ is heavily used by kernel developers as it includes many handy debugging options.

If you try the Magic SysRQ key on older servers and you realize it doesn’t react you will have to recompile the linux kernel and enable the CONFIG_MAGIC_SYSRQ kernel time compilation option.

The Magic SysRQ can also be initiated remotely on a properly running server 😉 by initiating:

server:~# echo b > /proc/sysrq-trigger

This command would be useful if you want to just have fun and test that magic sysrq works on your system for sure 😉

To sum it up using the Magic SysRQ will guarantee your locked up, server a Safe Reboot and will hopefully save you a lot of time for backups recovery.