Configuring a Linux HA (High Availibiltiy) for an Application with Haproxy is already used across many Websites on the Internet and serious corporations that has a crucial infrastructure has long time
adopted and used keepalived to provide High Availability Application level Clustering.
Usually companies choose to use HA Clusters with Haproxy with Pacemaker and Corosync cluster tools.
However one common used alternative solution if you don't have the oportunity to bring up a High availability cluster with Pacemaker / Corosync / pcs (Pacemaker Configuration System) due to fact machines you need to configure the cluster on are not Physical but VMWare Virtual Machines which couldn't not have configured a separate Admin Lans and Heartbeat Lan as we usually do on a Pacemaker Cluster due to the fact the 5 Ethernet LAN Card Interfaces of the VMWare Hypervisor hosts are configured as a BOND (e.g. all the incoming traffic to the VMWare vSphere HV is received on one Virtual Bond interface).
I assume you have 2 separate vSphere Hypervisor Physical Machines in separate Racks and separate switches hosting the two VMs.
For the article, I'll call the two brand new brought Virtual Machines with some installation automation software such as Terraform or Ansible – vm-server1 and vm-server2 which would have configured some recent version of Linux.
In that scenario to have a High Avaiability for the VMs on Application level and assure at least one of the two is available at a time if one gets broken due toe malfunction of the HV, a Network connectivity issue, or because the VM OS has crashed.
Then one relatively easily solution is to use keepalived and configurea single High Availability Virtual IP (VIP) Address, i.e. 10.10.10.1, which would float among two VMs using keepalived so at a time at least one of the two VMs would be reachable on the Network.
Having a VIP IP is quite a common solution in corporate world, as it makes it pretty easy to add F5 Load Balancer in front of the keepalived cluster setup to have a 3 Level of security isolation, which usually consists of:
1. Physical (access to the hardware or Virtualization hosts)
2. System Access (The mechanism to access the system login credetials users / passes, proxies, entry servers leading to DMZ-ed network)
3. Application Level (access to different programs behind L2 and data based on the specific identity of the individual user,
special Secondary UserID, Factor authentication, biometrics etc.)
1. Install keepalived and haproxy on machines
Depending on the type of Linux OS:
On both machines
[root@server1:~]# yum install -y keepalived haproxy
…
If you have to install keepalived / haproxy on Debian / Ubuntu and other Deb based Linux distros
[root@server1:~]# apt install keepalived haproxy –yes
…
2. Configure haproxy (haproxy.cfg) on both server1 and server2
Create some /etc/haproxy/haproxy.cfg configuration
[root@server1:~]# vim /etc/haproxy/haproxy.cfg
#———————————————————————
# Global settings
#———————————————————————
global
log 127.0.0.1 local6 debug
chroot /var/lib/haproxy
pidfile /run/haproxy.pid
stats socket /var/lib/haproxy/haproxy.sock mode 0600 level admin
maxconn 4000
user haproxy
group haproxy
daemon
#debug
#quiet
#———————————————————————
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#———————————————————————
defaults
mode tcp
log global
# option dontlognull
# option httpclose
# option httplog
# option forwardfor
option redispatch
option log-health-checks
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3
#———————————————————————
# round robin balancing between the various backends
#———————————————————————
listen FRONTEND_APPNAME1
bind 10.10.10.1:15000
mode tcp
option tcplog
# #log global
log-format [%t]\ %ci:%cp\ %bi:%bp\ %b/%s:%sp\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
balance roundrobin
timeout client 350000
timeout server 350000
timeout connect 35000
server app-server1 10.10.10.55:30000 weight 1 check port 68888
server app-server2 10.10.10.55:30000 weight 2 check port 68888
listen FRONTEND_APPNAME2
bind 10.10.10.1:15000
mode tcp
option tcplog
#log global
log-format [%t]\ %ci:%cp\ %bi:%bp\ %b/%s:%sp\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
balance roundrobin
timeout client 350000
timeout server 350000
timeout connect 35000
server app-server1 10.10.10.55:30000 weight 5
server app-server2 10.10.10.55:30000 weight 5
You can get a copy of above haproxy.cfg configuration here.
Once configured roll it on.
[root@server1:~]# systemctl start haproxy
[root@server1:~]# ps -ef|grep -i hapro
root 285047 1 0 Mar07 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 285050 285047 0 Mar07 ? 00:00:26 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
Bring up the haproxy also on server2 machine, by placing same configuration and starting up the proxy.
[root@server1:~]# vim /etc/haproxy/haproxy.cfg
…
…
3. Configure keepalived on both servers
We'll be configuring 2 nodes with keepalived even though if necessery this can be easily extended and you can add more nodes.
First we make a copy of the original or existing server configuration keepalived.conf (just in case we need it later on or if you already had something other configured manually by someone – that could be so on inherited servers by other sysadmin)
[root@server1:~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.orig
[root@server2:~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.orig
a. Configure keepalived to serve as a MASTER Node
[root@server1:~]# vim /etc/keepalived/keepalived.conf
Master Node
global_defs {
router_id server1-fqdn # The hostname of this host.
enable_script_security
# Synchro of the state of the connections between the LBs on the eth0 interface
lvs_sync_daemon eth0
notification_email {
linuxadmin@notify-domain.com # Email address for notifications
}
notification_email_from keepalived@server1-fqdn # The from address for the notifications
smtp_server 127.0.0.1 # SMTP server address
smtp_connect_timeout 15
}
vrrp_script haproxy {
script "killall -0 haproxy"
interval 2
weight 2
user root
}
vrrp_instance LB_VIP_QA {
virtual_router_id 50
advert_int 1
priority 51
state MASTER
interface eth0
smtp_alert # Enable Notifications Via Email
authentication {
auth_type PASS
auth_pass testp141
}
### Commented because running on VM on VMWare
## unicast_src_ip 10.44.192.134 # Private IP address of master
## unicast_peer {
## 10.44.192.135 # Private IP address of the backup haproxy
## }
# }
# master node with higher priority preferred node for Virtual IP if both keepalived up
### priority 51
### state MASTER
### interface eth0
virtual_ipaddress {
10.10.10.1 dev eth0 # The virtual IP address that will be shared between MASTER and BACKUP
}
track_script {
haproxy
}
}
To dowload a copy of the Master keepalived.conf configuration click here
Below are few interesting configuration variables, worthy to mention few words on, most of them are obvious by their names but for more clarity I'll also give a list here with short description of each:
- vrrp_instance – defines an individual instance of the VRRP protocol running on an interface.
- state – defines the initial state that the instance should start in (i.e. MASTER / SLAVE )state –
- interface – defines the interface that VRRP runs on.
- virtual_router_id – should be unique value per Keepalived Node (otherwise slave master won't function properly)
- priority – the advertised priority, the higher the priority the more important the respective configured keepalived node is.
- advert_int – specifies the frequency that advertisements are sent at (1 second, in this case).
- authentication – specifies the information necessary for servers participating in VRRP to authenticate with each other. In this case, a simple password is defined.
only the first eight (8) characters will be used as described in to note is Important thing
man keepalived.conf – keepalived.conf variables documentation !!! Nota Bene !!! – Password set on each node should match for nodes to be able to authenticate ! - virtual_ipaddress – defines the IP addresses (there can be multiple) that VRRP is responsible for.
- notification_email – the notification email to which Alerts will be send in case if keepalived on 1 node is stopped (e.g. the MASTER node switches from host 1 to 2)
- notification_email_from – email address sender from where email will originte
! NB ! In order for notification_email to be working you need to have configured MTA or Mail Relay (set to local MTA) to another SMTP – e.g. have configured something like Postfix, Qmail or Postfix
b. Configure keepalived to serve as a SLAVE Node
[root@server1:~]# vim /etc/keepalived/keepalived.conf
#Slave keepalived
global_defs {
router_id server2-fqdn # The hostname of this host!
enable_script_security
# Synchro of the state of the connections between the LBs on the eth0 interface
lvs_sync_daemon eth0
notification_email {
linuxadmin@notify-host.com # Email address for notifications
}
notification_email_from keepalived@server2-fqdn # The from address for the notifications
smtp_server 127.0.0.1 # SMTP server address
smtp_connect_timeout 15
}
vrrp_script haproxy {
script "killall -0 haproxy"
interval 2
weight 2
user root
}
vrrp_instance LB_VIP_QA {
virtual_router_id 50
advert_int 1
priority 50
state BACKUP
interface eth0
smtp_alert # Enable Notifications Via Email
authentication {
auth_type PASS
auth_pass testp141
}
### Commented because running on VM on VMWare
## unicast_src_ip 10.10.192.135 # Private IP address of master
## unicast_peer {
## 10.10.192.134 # Private IP address of the backup haproxy
## }
### priority 50
### state BACKUP
### interface eth0
virtual_ipaddress {
10.10.10.1 dev eth0 # The virtual IP address that will be shared betwee MASTER and BACKUP.
}
track_script {
haproxy
}
}
Download the keepalived.conf slave config here
c. Set required sysctl parameters for haproxy to work as expected
[root@server1:~]# vim /etc/sysctl.conf
#Haproxy config
# haproxy
net.core.somaxconn=65535
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
4. Test Keepalived keepalived.conf configuration syntax is OK
[root@server1:~]# keepalived –config-test
(/etc/keepalived/keepalived.conf: Line 7) Unknown keyword 'lvs_sync_daemon_interface'
(/etc/keepalived/keepalived.conf: Line 21) Unable to set default user for vrrp script haproxy – removing
(/etc/keepalived/keepalived.conf: Line 31) (LB_VIP_QA) Specifying lvs_sync_daemon_interface against a vrrp is deprecated.
(/etc/keepalived/keepalived.conf: Line 31) Please use global lvs_sync_daemon
(/etc/keepalived/keepalived.conf: Line 35) Truncating auth_pass to 8 characters
(/etc/keepalived/keepalived.conf: Line 50) (LB_VIP_QA) track script haproxy not found, ignoring…
I've experienced this error because first time I've configured keepalived, I did not mention the user with which the vrrp script haproxy should run,
in prior versions of keepalived, leaving the field empty did automatically assumed you have the user with which the vrrp script runs to be set to root
as of RHELs keepalived-2.1.5-6.el8.x86_64, i've been using however this is no longer so and thus in prior configuration as you can see I've
set the user in respective section to root.
The error Unknown keyword 'lvs_sync_daemon_interface' is also easily fixable by just substituting the lvs_sync_daemon_interface and lvs_sync_daemon and reloading
keepalived etc.
Once keepalived is started and you can see the process on both machines running in process list.
[root@server1:~]# ps -ef |grep -i keepalived
root 1190884 1 0 18:50 ? 00:00:00 /usr/sbin/keepalived -D
root 1190885 1190884 0 18:50 ? 00:00:00 /usr/sbin/keepalived -D
Next step is to check the keepalived statuses as well as /var/log/keepalived.log
If everything is configured as expected on both keepalived on first node you should see one is master and one is slave either in the status or the log
[root@server1:~]#systemctl restart keepalived
[root@server1:~]# systemctl status keepalived|grep -i state
Mar 14 18:59:02 server1-fqdn Keepalived_vrrp[1192003]: (LB_VIP_QA) Entering MASTER STATE
[root@server1:~]# systemctl status keepalived
● keepalived.service – LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2022-03-14 18:15:51 CET; 32min ago
Process: 1187587 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1187589 (code=exited, status=0/SUCCESS)
Mar 14 18:15:04 server1lb-fqdn Keepalived_vrrp[1187590]: Sending gratuitous ARP on eth0 for 10.44.192.142
Mar 14 18:15:50 server1lb-fqdn systemd[1]: Stopping LVS and VRRP High Availability Monitor…
Mar 14 18:15:50 server1lb-fqdn Keepalived[1187589]: Stopping
Mar 14 18:15:50 server1lb-fqdn Keepalived_vrrp[1187590]: (LB_VIP_QA) sent 0 priority
Mar 14 18:15:50 server1lb-fqdn Keepalived_vrrp[1187590]: (LB_VIP_QA) removing VIPs.
Mar 14 18:15:51 server1lb-fqdn Keepalived_vrrp[1187590]: Stopped – used 0.002007 user time, 0.016303 system time
Mar 14 18:15:51 server1lb-fqdn Keepalived[1187589]: CPU usage (self/children) user: 0.000000/0.038715 system: 0.001061/0.166434
Mar 14 18:15:51 server1lb-fqdn Keepalived[1187589]: Stopped Keepalived v2.1.5 (07/13,2020)
Mar 14 18:15:51 server1lb-fqdn systemd[1]: keepalived.service: Succeeded.
Mar 14 18:15:51 server1lb-fqdn systemd[1]: Stopped LVS and VRRP High Availability Monitor
[root@server2:~]# systemctl status keepalived|grep -i state
Mar 14 18:59:02 server2-fqdn Keepalived_vrrp[297368]: (LB_VIP_QA) Entering BACKUP STATE
[root@server1:~]# grep -i state /var/log/keepalived.log
Mar 14 18:59:02 server1lb-fqdn Keepalived_vrrp[297368]: (LB_VIP_QA) Entering MASTER STATE
a. Fix Keepalived SECURITY VIOLATION – scripts are being executed but script_security not enabled.
When configurating keepalived for a first time we have faced the following strange error inside keepalived status inside keepalived.log
Feb 23 14:28:41 server1 Keepalived_vrrp[945478]: SECURITY VIOLATION – scripts are being executed but script_security not enabled.
To fix keepalived SECURITY VIOLATION error:
Add to /etc/keepalived/keepalived.conf on the keepalived node hosts
inside
global_defs {}
After chunk
enable_script_security
include
# Synchro of the state of the connections between the LBs on the eth0 interface
lvs_sync_daemon_interface eth0
5. Prepare rsyslog configuration and Inlcude additional keepalived options
to force keepalived log into /var/log/keepalived.log
To force keepalived log into /var/log/keepalived.log on RHEL 8 / CentOS and other Redhat Package Manager (RPM) Linux distributions
[root@server1:~]# vim /etc/rsyslog.d/48_keepalived.conf
#2022/02/02: HAProxy logs to local6, save the messages
local7.* /var/log/keepalived.log
if ($programname == 'Keepalived') then -/var/log/keepalived.log
if ($programname == 'Keepalived_vrrp') then -/var/log/keepalived.log
& stop
[root@server:~]# touch /var/log/keepalived.log
Reload rsyslog to load new config
[root@server:~]# systemctl restart rsyslog
[root@server:~]# systemctl status rsyslog
● rsyslog.service – System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/rsyslog.service.d
└─rsyslog-service.conf
Active: active (running) since Mon 2022-03-07 13:34:38 CET; 1 weeks 0 days ago
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 269574 (rsyslogd)
Tasks: 6 (limit: 100914)
Memory: 5.1M
CGroup: /system.slice/rsyslog.service
└─269574 /usr/sbin/rsyslogd -n
Mar 15 08:15:16 server1lb-fqdn rsyslogd[269574]: — MARK —
Mar 15 08:35:16 server1lb-fqdn rsyslogd[269574]: — MARK —
Mar 15 08:55:16 server1lb-fqdn rsyslogd[269574]: — MARK —
If once keepalived is loaded but you still have no log written inside /var/log/keepalived.log
[root@server1:~]# vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 7"
[root@server2:~]# vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 7"
[root@server1:~]# systemctl restart keepalived.service
[root@server1:~]# systemctl status keepalived
● keepalived.service – LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2022-02-24 12:12:20 CET; 2 weeks 4 days ago
Main PID: 1030501 (keepalived)
Tasks: 2 (limit: 100914)
Memory: 1.8M
CGroup: /system.slice/keepalived.service
├─1030501 /usr/sbin/keepalived -D
└─1030502 /usr/sbin/keepalived -D
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[root@server2:~]# systemctl restart keepalived.service
[root@server2:~]# systemctl status keepalived
…
6. Monitoring VRRP traffic of the two keepaliveds with tcpdump
Once both keepalived are up and running a good thing is to check the VRRP protocol traffic keeps fluently on both machines.
Keepalived VRRP keeps communicating over the TCP / IP Port 112 thus you can simply snoop TCP tracffic on its protocol.
[root@server1:~]# tcpdump proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:08:07.356187 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:08.356297 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:09.356408 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:10.356511 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:11.356655 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
[root@server2:~]# tcpdump proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:08:07.356187 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:08.356297 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:09.356408 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:10.356511 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
11:08:11.356655 IP server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20
As you can see the VRRP traffic on the network is originating only from server1lb-fqdn, this is so because host server1lb-fqdn is the keepalived configured master node.
It is possible to spoof the password configured to authenticate between two nodes, thus if you're bringing up keepalived service cluster make sure your security is tight at best the machines should be in a special local LAN DMZ, do not configure DMZ on the internet !!! 🙂 Or if you eventually decide to configure keepalived in between remote hosts, make sure you somehow use encrypted VPN or SSH tunnels to tunnel the VRRP traffic.
[root@server1:~]# tcpdump proto 112 -vv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:36:25.530772 IP (tos 0xc0, ttl 255, id 59838, offset 0, flags [none], proto VRRP (112), length 40)
server1lb-fqdn > vrrp.mcast.net: vrrp server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20, addrs: VIPIP_QA auth "testp431"
11:36:26.530874 IP (tos 0xc0, ttl 255, id 59839, offset 0, flags [none], proto VRRP (112), length 40)
server1lb-fqdn > vrrp.mcast.net: vrrp server1lb-fqdn > vrrp.mcast.net: VRRPv2, Advertisement, vrid 50, prio 53, authtype simple, intvl 1s, length 20, addrs: VIPIP_QA auth "testp431"
Lets also check what floating IP is configured on the machines:
[root@server1:~]# ip -brief address show
lo UNKNOWN 127.0.0.1/8
eth0 UP 10.10.10.5/26 10.10.10.1/32
The 10.10.10.5 IP is the main IP set on LAN interface eth0, 10.10.10.1 is the floating IP which as you can see is currently set by keepalived to listen on first node.
[root@server2:~]# ip -brief address show |grep -i 10.10.10.1
An empty output is returned as floating IP is currently configured on server1
To double assure ourselves the IP is assigned on correct machine, lets ping it and check the IP assigned MAC currently belongs to which machine.
[root@server2:~]# ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.526 ms
^C
— 10.10.10.1 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.526/0.526/0.526/0.000 ms
[root@server2:~]# arp -an |grep -i 10.44.192.142
? (10.10.10.1) at 00:48:54:91:83:7d [ether] on eth0
[root@server2:~]# ip a s|grep -i 00:48:54:91:83:7d
[root@server2:~]#
As you can see from below output MAC is not found in configured IPs on server2.
[root@server1-fqdn:~]# /sbin/ip a s|grep -i 00:48:54:91:83:7d -B1 -A1
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:48:54:91:83:7d brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/26 brd 10.10.1.191 scope global noprefixroute eth0
Pretty much expected MAC is on keepalived node server1.
7. Testing keepalived on server1 and server2 maachines VIP floating IP really works
To test the overall configuration just created, you should stop keeaplived on the Master node and in meantime keep an eye on Slave node (server2), whether it can figure out the Master node is gone and switch its
state BACKUP to save MASTER. By changing the secondary (Slave) keepalived to master the floating IP: 10.10.10.1 will be brought up by the scripts on server2.
Lets assume that something went wrong with server1 VM host, for example the machine crashed due to service overload, DDoS or simply a kernel bug or whatever reason.
To simulate that we simply have to stop keepalived, then the broadcasted information on VRRP TCP/IP proto port 112 will be no longer available and keepalived on node server2, once
unable to communicate to server1 should chnage itself to state MASTER.
[root@server1:~]# systemctl stop keepalived
[root@server1:~]# systemctl status keepalived
● keepalived.service – LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Tue 2022-03-15 12:11:33 CET; 3s ago
Process: 1192001 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 1192002 (code=exited, status=0/SUCCESS)
Mar 14 18:59:07 server1lb-fqdn Keepalived_vrrp[1192003]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:32 server1lb-fqdn systemd[1]: Stopping LVS and VRRP High Availability Monitor…
Mar 15 12:11:32 server1lb-fqdn Keepalived[1192002]: Stopping
Mar 15 12:11:32 server1lb-fqdn Keepalived_vrrp[1192003]: (LB_VIP_QA) sent 0 priority
Mar 15 12:11:32 server1lb-fqdn Keepalived_vrrp[1192003]: (LB_VIP_QA) removing VIPs.
Mar 15 12:11:33 server1lb-fqdn Keepalived_vrrp[1192003]: Stopped – used 2.145252 user time, 15.513454 system time
Mar 15 12:11:33 server1lb-fqdn Keepalived[1192002]: CPU usage (self/children) user: 0.000000/44.555362 system: 0.001151/170.118126
Mar 15 12:11:33 server1lb-fqdn Keepalived[1192002]: Stopped Keepalived v2.1.5 (07/13,2020)
Mar 15 12:11:33 server1lb-fqdn systemd[1]: keepalived.service: Succeeded.
Mar 15 12:11:33 server1lb-fqdn systemd[1]: Stopped LVS and VRRP High Availability Monitor.
On keepalived off, you will get also a notification Email on the Receipt Email configured from keepalived.conf from the working keepalived node with a simple message like:
=> VRRP Instance is no longer owning VRRP VIPs <=
Once keepalived is back up you will get another notification like:
=> VRRP Instance is now owning VRRP VIPs <=
[root@server2:~]# systemctl status keepalived
● keepalived.service – LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-03-14 18:13:52 CET; 17h ago
Process: 297366 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 297367 (keepalived)
Tasks: 2 (limit: 100914)
Memory: 2.1M
CGroup: /system.slice/keepalived.service
├─297367 /usr/sbin/keepalived -D -S 7
└─297368 /usr/sbin/keepalived -D -S 7
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: Remote SMTP server [127.0.0.1]:25 connected.
Mar 15 12:11:33 server2lb-fqdn Keepalived_vrrp[297368]: SMTP alert successfully sent.
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: (LB_VIP_QA) Sending/queueing gratuitous ARPs on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
Mar 15 12:11:38 server2lb-fqdn Keepalived_vrrp[297368]: Sending gratuitous ARP on eth0 for 10.10.10.1
[root@server2:~]# ip addr show|grep -i 10.10.10.1
inet 10.10.10.1/32 scope global eth0
As you see the VIP is now set on server2, just like expected – that's OK, everything works as expected. If the IP did not move double check the keepalived.conf on both nodes for errors or misconfigurations.
To recover the initial order of things so server1 is MASTER and server2 SLAVE host, we just have to switch on the keepalived on server1 machine.
[root@server1:~]# systemctl start keepalived
The automatic change of server1 to MASTER node and respective move of the VIP IP is done because of the higher priority (of importance we previously configured on server1 in keepalived.conf).
What we learned?
So what we learned in this article?
We have seen how to easily install and configure a High Availability Load balancer with Keepalived with single floating VIP IP address with 1 MASTER and 1 SLAVE host and a Haproxy example config with few frontends / App backends. We have seen how the config can be tested for potential errors and how we can monitor whether the VRRP2 network traffic flows between nodes and how to potentially debug it further if necessery.
Further on rawly explained some of the keepalived configurations but as keepalived can do pretty much more,for anyone seriously willing to deal with keepalived on a daily basis or just fine tune some already existing ones, you better read closely its manual page "man keepalived.conf" as well as the official Redhat Linux documentation page on setting up a Linux cluster with Keepalived (Be prepare for a small nightmare as the documentation of it seems to be a bit chaotic, and even I would say partly missing or opening questions on what does the developers did meant – not strange considering the havoc that is pretty much as everywhere these days.)
Finally once keepalived hosts are prepared, it was shown how to test the keepalived application cluster and Floating IP does move between nodes in case if one of the 2 keepalived nodes is inaccessible.
The same logic can be repeated multiple times and if necessery you can set multiple VIPs to expand the HA reachable IPs solution.
The presented idea is with haproxy forward Proxy server to proxy requests towards Application backend (servince machines), however if you need to set another set of server on the flow to process HTML / XHTML / PHP / Perl / Python programming code, with some common Webserver setup ( Nginx / Apache / Tomcat / JBOSS) and enable SSL Secure certificate with lets say Letsencrypt, this can be relatively easily done. If you want to implement letsencrypt and a webserver check this redundant SSL Load Balancing with haproxy & keepalived article.
That's all folks, hope you enjoyed.
If you need to configure keepalived Cluster or a consultancy write your query here 🙂
Tags:
Ansible,
application cluster,
backup,
Below,
cluster config,
common,
configured,
Configuring,
CPU,
Easy,
everything,
haproxy,
howto,
incoming traffic,
inside,
installation,
keepalived,
linux?,
long time,
Mar,
master node,
network,
nightmare,
Sending,
server2,
servers,
Set,
Slave,
smtp,
something,
Stopping,
test keepalived,
var,
vip address,
virtual machines,
Warning Journal 
Baby boomers and Generation X, Y, Z – Generational Marketing and 4 Common personality stereotype traits of people born over the last 60 years
Saturday, August 18th, 2018Those who are employed in the realm of Social or Internet Marketing definitely have to know the existence of at least 4 different conditional stereotypes, these are Baby Boomers and Generation X, Generation Y and Generation Z (Millenials).
According to Socielogist Karl Mannheim (who is among the founding fathers of classical socielogy) – "All members of a generation share a similar collective experience" or in other words people are categorized in generations depending on when they were born.
As stereotypes they're generalization of people born in different periods of time and sharing same or similar traits.
Because of the age and the conditions they grew up and as they share those general spirit of time and age, they tend to be more or less behaving in a similar ways in how they think save / spend money or share some common approach to life choices and attitude towards life and worldview.
But before proceeding to the 4 main cohert provisional stereotypes, its worthy to mention how these four common trait generations came to existence with a little bit of pre-history.
The pre WW I and WW II world situation and the First and Second World War played a pivotal role in forming the social conditions necessery for the development of the baby boomers.
* The depression Era people
Born in period: 1912 – 1921 who came at full maturity around 1930-1939 right in the beginning of WW I (all of whom are already deceased) as of 2018 as a cause of the war uncertainty and the havoc and the war conditions were very conservative, compulsive savers, tried their best to maintain a low debt. They had the mindset (responsibility) to leave some kind of legacy to their children. They were very patriotic, oriented towards work before pleasure, had a great respect for authority and had a strong sense of moral obligation. For all this character traits of this people undoubtfully a key role played the strong belief in God mostly all people had at the time.
The next in line conditional stereotype of people that came to earth are the:
* The World War II Generation
Born in year period: 1922 to 1927 who came to a mature age exactly at the terrible years of Second World War.
People of that time were either fighters for or against the Axis Powers or the Central Powers with the common shared goal to fight against the enemy (of course there are multiple of people who were just trying to survive and not taking a side in this meaningless war).
The current amount of people living are estimated to few million of deathbed elders worldwide.
As above conditional generations types mentioned are of importance for historical reasons and most of the people belonging to those depression pre WWI and WW II era are dead or just a few millions an overall in un less-consuming age (excluding the medicine consumption which is higher compared to youngsters).
I'll further proceed further with the Baby Boomers, GEN X, Y, Zs who are de-facto the still active members participating to society and economy more or less.
So what are these 4 Stereotypes of Generations that and why are so important for the modern marketers or business manager?
1. BABY BOOMERS also called for a short (Boomers)
These are people who have been defined by a birth year range (period) from early to mid 1940s until 1960 and 1964.
In Europe and North America, boomers are widely identified with privilege, as many grew up in a time of widespread government subsidies in post-war housing and education, and increasing affluence.
As a group, baby boomers were considered the wealthiest, most active, and most physically fit generation up to the era in which they arrived, and were amongst the first to grow up genuinely expecting the world to improve with time. They were also the generation that received peak levels of income; they could therefore reap the benefits of abundant levels of food, apparel, retirement programs, and sometimes even "midlife crisis" products. The increased consumerism for this generation has been regularly criticized as excessive (and that's for a good reason).
One feature of the boomers was that they have tended to think of themselves as a special generation, very different from those that had come before or that has come afterward. In the 1960s, as the relatively large numbers of young people became teenagers and young adults, they, and those around them, created a very specific rhetoric around their cohort, and the changes they were bringing about. This rhetoric had an important impact in the self perceptions of the boomers, as well as their tendency to define the world in terms of generations, which was a relatively new phenomenon. The baby boom has been described variously as a "shockwave" and with a methapors such as as "the pig in the python".
2. Generation X / GEN X
Generation X is considered the people born in the following birth year period 1960 forward in time until 1980s. A specific feature in the 60s-80s period was the shifting societal values, perhaps the spring of this generation was also connected to the increasing role and spread of communism in the world.
Sometimes this generation was referred as the "latchkey generation".
The term generation X itself was popularized largely by Douglas Coupland in his novel 1991 novel Generation X Tales for an Accelerated Culture
A very common trait for Generation X was the reduced adult supervision over kids when compared to previous generations a result of increasing divorce rates and the increased role of one parent children upbringing (in most cases that was the mother) which had to be actively involved as a workforce and lacked physically the time to spend enough time with its children and the increased use of childcare options in one parent families.
They were dubbed the "MTV" (Music Television) generation – that was a hit and most popular music TV in the early 1990s.
The kids representing generation X were described as slackers, cynical and disaffected.
The cultural influences dominating the tastes and feelings of the teen masses of that generation was musical genres such as punk music, heavy metal music, grunge and hip-hop and indie films (independent films) produced outside of the major film studio system.
According to many researches in midtime those generation are described as active, happy and achieving a work-life balance kind of lifestyle.
People belonging to Generation X are described as people with Enterpreneural tendencies.
Just to name a few of the celebrities and successful people who belong to this generation, that's Google's founder Sergey Brinn & Larry Page (born in 1973), Richard Stallman (founder of Free Software movement) as well movie and film producer celebrities such as Georgi Clooney, Lenny Kravitz, Quantin Tarantino, Kevin Smith, David Fincher etc.
According to United Kingdom survey study of 2500+ workers conducted by Workfront, GEN X are found to be among the hardest working employees in today's workforce. They are also ranked high by fellow workers for having a strong work ethics (about 59.5%), being helpful (55.4%) and very skilled (54.5%) of respondents as well marked as the best troubleshooters / problem solvers (41.6%) claimed so.
According to research conducted by Viacom, gen x they have a high desire for flexibility and fulfillment at work.
3. Generation Y (Millenials) – GEN Y
Following Generation X came on earth Genreation Y the birth period dated for this kids were years are stretchy year period that this generation is described are years 1980s – 1990s to yearly 2000s where birth period range of those ppl ends.
This kids are descendants of the GEN X and second wave Baby Boomers.
In the public this generation is referred as "echo boomers".
The Millenials characteristics are different based on the region of birth, they're famous for the increased familiarity with communication, media and digital technologies.
There upbringing was marked by increase in liberal approach to politics.
The Great recession crisis of the 2000s played a major impact on this generation because it has caused historical high levels of un-employment among youngsters and led to a possible long term economic and social damage to this generation.
Gen Y according are less brand loyal and the speed of the Internet has led the cohort to be similarly flexible and changing in its fashion, style consciousness and where
and how it is communicated with.
As I am born in 1983 me and my generation belongs to Generation Y and even though Bulgaria before 1991 was a Communist regime country, I should agree that I and many of my friends share a very similar behavior and way of thinking to the GEN Y stereotype described, but as I was born in a times of transition and Bulgaria as a Soviet Union Satellite at the time has been lacking behind in fashion and international culture due to the communist regime, me and my generation seem to be sharing a lot of common stereotype characteristics with Generation X such as the punk-rock, metal, hip-hop culture MTV culture and partly because of the GEN X like overall view on life.
Among most famous representative successful people of the Millenials generation are Mark Zuckerberg (Facebook founder), Prince William (the second in line to the British throne), Kim Jong Un (the leader dictator of North Korea) etc.
4. Generation Z ( GEN Z) / iGeneration / Generation Sensible (Post Millenials)
Following Millenials generation is GEN Z, demographers and researchers typically set as a starting birth date period of those generation 1990s and mid 2000s. As of time of writting there is still no clear consensus regarding ending birth years.
This is the so called Internet Generation because this generation used the internet and Smart Mobile Phone technology since a very young age, they are very confortable with technology (kinda of wired) and addicted to social media such as Facebook / Twitter / Instagram etc. Because of the level of digital communication, many people of this generation are more introvert oriented and often have problems expressing themselves freely in groups. Also they tend to lack the physical communication and more digitally community oriented, even though this depends much also on the specific personality and in some cases it is exactly the opposite.
* Summary
As a Marketer, Human Resources hiring personal specialist, a CEO or some kind of project / business manager it is a good idea to be aware of these 4 common stereotypes. However as this are stereotypes (and a theory) as everything theoritized the data is slighly biased and untrue. The marketer practice shows that whoever conducts a marketing and bases his sales on this theoritizing should consider this to be just one aspect of the marketing campaign those who are trying to sell, stuff ideas or ideology to any of those generation should be careful not to count 100% on the common traits found among the above 4 major groups and consider the individuality of person everyone has and just experiment a little bit to see what works and what doesn't.
Also it should be mentioned these diversification of stereotypes are mostly valid for the US citizens and Westerners but doesn't fully fit to ex-communist countries or countries of the Soviet union, those countries have a slightly different personality traits of person born in any of the year periods defined, same is more or less true for the poor parts of Africa and India, Vietnam, China and mostly all of the coomunist countries ex and current. It should be said that countries who belonged to the Soviet Union many of which are current Russian Federation Republics have a personality traits that are often mixture of the 4 stereotypes and even have a lot of the traits that were typical for the WW I and WW II generations, which makes dealing with this people a very weird experience.
Nomatter the standard error that should always considered when basing a marketing research hypothesis on Generational Marketing (using generational segmentation in marketing best potential customer targets), having a general insight and taking in consideration those stereotypes could seriously help in both marketing as well as HR specific fields like Change Management.
If you're a marketer, I recommend you take a quick look also on following very educative article Generational Makarketing and how to target each of the GEN X, Y, Z and Baby Boomers and what works best for each of them.
Nomatter what just like all Theories, the theory of Boomers and the Generation segmantation is not completely true, but it gives a good soil for reasoning as well definitely helps for people involved in sociology and business.
Comments and feedback on the article are mostly welcome as the topic is very broad and there is much more to be said …
Hope the article was interesting to you ….
What was your Generation like?
Tags: about, Achieving, africa, against, age, ALL, amount, birth, Comments, common, generations, key role, last, MTV, periods, personality, python, time, times, years
Posted in Business Management, Curious Facts, Economy, Educational, Politics, Psychology | No Comments »