Posts Tagged ‘enable’

Enable zabbix agent to work with SeLinux enabled on CentOS 7 Linux

Wednesday, October 19th, 2022

If you have the task to install and use zabbix-agent or zabbix-proxy to report to zabbix-server on CentOS 7 with enabled SeLinux services for security reasons and you have no mean to disable the selinux which is a common step to take under this circumstances, you will have to add the zabbix services to be exluded as permissive in selinux. In below article I'll show you how this is done in few easy steps.

zabbix-agent-service-selinux-linux-real-time-operating-sytems

 

1. Check the system sestatus

[root@linux zabbix]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing

Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28


2. Enable zabbix to be permissive in selinux

To be able to set zabbix to be in permissive mode as well as for further troubleshooting if you have to enable other  linux services inside selinux you have to install below RPM packs.

[root@linux zabbix]# yum install setroubleshoot.x86_64 setools.x86_64 setools-console.x86_64 policycoreutils-python.x86_64

Set the zabbix permissive exclude rule in SeLINUX

[root@linux zabbix]# semanage permissive –add zabbix_t

Re-run the zabbix proxy (if you have a local zabbix-proxy running and the zabbix-agent)

[root@linux zabbix]# systemctl start zabbix-proxy.service

[root@linux zabbix]# systemctl start zabbix-agent.service

[root@linux zabbix]# systemctl status zabbix-agent
● zabbix-agent.service – Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-10-18 09:30:16 CEST; 1 day 7h ago
 Main PID: 962952 (zabbix_agentd)
    Tasks: 6 (limit: 100884)
   Memory: 5.1M
   CGroup: /system.slice/zabbix-agent.service
           ├─962952 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─962955 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─962956 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─962957 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─962958 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─962959 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Oct 18 09:30:16 linux systemd[1]: Starting Zabbix Agent…
Oct 18 09:30:16 linux systemd[1]: Started Zabbix Agent.

3. Check inside audit logs all is OK

To make sure zabbix is really enabled to be omitted by selinux rules check audit.log

[root@linux zabbix]# grep zabbix_proxy /var/log/audit/audit.log

That's all folks, Enjoy ! 🙂

CentOS 8 / Redhat 8 insert additional guests additions to VM to enable Fullscreen, Copy / Paste and Shared Folder from host OS

Monday, January 10th, 2022

virtualbox-guest-additions-install-on-centos-8.3-linux-oracle-logo

My experience with enabling virtualbox additions guest tools on many of the separate Linux distributions throughout time is pretty bad as it always is a pain in the ass to enable fully functional full screen and copy paste for Virtualbox…
 
For those who installed it for a first time vbox guest addition tools for Virtualbox are additional software components added so the Emulated Operating system
could allow better screen resolution and better mouse integration support.

So far I've installed virtualbox additions tools to CentOS 7 and Debian Linux various releases and faced complications there as well.
Few days ago my colleague Georgi Stoyanov have installed CentOS 8.3 with current version of VirtualBox 6.1 (vesrsion from beginning of 2022) and he has also shared had issues with enabling the CentOS 8.3 Linux to work with guestadditions but eventually found a resolution.

Thus he has shared with me the solution and I share it with you, so hopefully someone else could enable Guesttools on his CentOS 8.3 with less digging online.
The error received is:

# ./VBoxLinuxAdditions.run

Trying to install Guest Additions in RHEL 8.3.

VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules. This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup
VirtualBox Guest Additions: or
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Building the modules for kernel
4.18.0-193.el8.x86_64.

VirtualBox Guest Additions: Look at /var/log/vboxadd-setup.log to find out what
went wrong
ValueError: File context for /opt/VBoxGuestAdditions-6.0.20/other/mount.vboxsf already defined
VirtualBox Guest Additions: Running kernel modules will not be replaced until
the system is restarted
Press Return to close this window…

No idea what to do next. Been trying for sometime.


To enable guestaddtions in CentOS 8.3, e.g. get arount the error you have to:


1. Install all necessery dependncies RPMs required by GuestAddition tools

 

# dnf install tar bzip2 kernel-devel-$(uname -r) kernel-headers perl gcc make elfutils-libelf-devel

# dnf -y install gcc automake make kernel-headers dkms bzip2 libxcrypt-compat kernel-devel perl

2.  Run below semanage and restorecon commands

 

# semanage fcontext -d /opt/VBoxGuestAdditions-/other/mount.vboxsf
# restorecon /opt/VBoxGuestAdditions-/other/mount.vboxsf

 

3.  Insert Virtualbox guest additions ISO and Run it

 

centos-insert-guest-additions-linux-virtualbox-screenshot
 

Devices -> Insert Guest Additions CD Image

 

Click Run button to exec Vbox_GAs_6.0.18 script or run it manually

Run-Guest-Additions-screenshot-virtualbox-centos-8

or mount it manually with mount command and execute the VBoxLinuxAdditions.run to do so:

 

$ cd /run/media/`whoami`/VB*
$ su
# ./VBoxLinuxAdditions.run
Installing additional modules …
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
VirtualBox Guest Additions: Running kernel modules will not be replaced until the system is restarted
VirtualBox Guest Additions: Starting.

 

4. Reboot the VM
 

# reboot

5. Check and Confirm Virtualbox guest additions are properly installed and running
 

# lsmod | grep vbox

 

6. Enable Copy / Paste from to Virttual Machine e.g. Shared Clipboard / Shared Folder etc.

 

Share-Clipboard-in-Virtualbox-screenshot-centos-8

 

The three options most useful besides the support for FullScreen OS emulation by Virtualbox to enable right after
guesttools is on are:


1. Devices -> Shared Clipboard -> Bidirectional
2. Devices -> Drag and Drop -> Bidirectional
3. Devices -> Shared Folders -> Shared Folder Settings

 

Adding proxy to yum repository on Redhat / Fedora / CentOS and other RPM based Linux distributions, Listing and enabling new RPM repositories

Tuesday, September 7th, 2021

yum-add-proxy-host-for-redhat-linux-centos-list-rpm-repositories-enable-disable-repositories

Sometimes if you work in a company that is following PCI standards with very tight security you might need to use a custom company prepared RPM repositories that are accessible only via a specific custom maintained repositories or alternatively you might need the proxy node  to access an external internet repository from the DMZ-ed firewalled zone where the servers lays .
Hence to still be able to maintain the RPM based servers up2date to the latest security patches and install software with yumone very useful feature of yum package manager is to use a proxy host through which you will reach your Redhat Package Manager files  files.

1. The http_proxy and https_proxy shell variables 

To set  a proxy host you need to define there the IP / Hostname or the Fully Qualified Domain Name (FQDN).

By default "http_proxy and https_proxy are empty. As you can guess https_proxy is used if you have a Secure Socket Layer (SSL) certificate for encrypting the communication channel (e.g. you have https:// URL).

[root@rhel: ~]# echo $http_proxy
[root@rhel: ~]#

2. Setting passwordless or password protected proxy host via http_proxy, https_proxy variables

There is a one time very straight forward to configure proxying of traffic via a specific remote configured server with server bourne again  shell (BASH)'s understood variables:
 

a.) Set password free open proxy to shell environment.

[root@centos: ~]# export https_proxy="https://remote-proxy-server:8080"


Now use yum as usual to update the available installabe package list or simply upgrade to the latest packages with lets say:

[root@rhel: ~]# yum check-update && yum update

b.) Configuring password protected proxy for yum

If your proxy is password protected for even tigher security you can provide the password on the command line as well.

[root@centos: ~]# export http_proxy="http://username:pAssW0rd@server:port/"

Note that if you have some special characters you will have to pass the string inside single quotes or escape them to make sure the password will properly handled to server, before trying out the proxy with yum, echo the variable.

[root@centos: ~]# export http_proxy='http://username:p@s#w:E@192.168.0.1:3128/'
  [root@centos: ~]# echo $http_proxy
http://username:p@s#w:E@server:port/

Then do whatever with yum:

[root@centos: ~]# yum check-update && yum search sharutils


If something is wrong and proxy is not properly connected try to reach for the repository manually with curl or wget

[root@centos: ~]# curl -ilk http://download.fedoraproject.org/pub/epel/7/SRPMS/ /epel/7/SRPMS/
HTTP/1.1 302 Found
Date: Tue, 07 Sep 2021 16:49:59 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Location: http://mirror.telepoint.bg/epel/7/SRPMS/
Content-Type: text/plain
Content-Length: 0
AppTime: D=2264
X-Fedora-ProxyServer: proxy01.iad2.fedoraproject.org
X-Fedora-RequestID: YTeYOE3mQPHH_rxD0sdlGAAAA80
X-Cache: MISS from pcfreak
X-Cache-Lookup: MISS from pcfreak:3128
Via: 1.1 pcfreak (squid/4.6)
Connection: keep-alive


Or if you need, you can test the user, password protected proxy with wget as so:

[root@centos: ~]# wget –proxy-user=USERNAME –proxy-password=PASSWORD http://your-proxy-domain.com/optional-rpms/


If you have lynx installed on the machine you can do the remote proxy successful authentication check with it with less typing:

[root@centos: ~]# lynx -pauth=USER:PASSWORD http://proxy-domain.com/optional-rpm/

 

3. Making yum proxy connection permanent via /etc/yum.conf

 

Perhaps the easiest and quickest way to add the http_proxy / https_proxy configured is to store it to automatically load on each server ssh login in your admin user (root) in /root/.bashrc or /root/.bash_profile or in the global /etc/profile or /etc/profile.d/custom.sh etc.

However if you don't want to have hacks and have more cleanness on the systems, the recommended "Redhat way" so to say is to store the configuration inside /etc/yum.conf

To do it via /etc/yum.conf you have to have some records there like:

# The proxy server – proxy server:port number 
proxy=http://mycache.mydomain.com:3128 
# The account details for yum connections 
proxy_username=yum-user 
proxy_password=qwerty-secret-pass

4. Listing RPM repositories and their state

As I had to install sharutils RPM package to the server which contains the file /bin/uuencode (that is provided on CentOS 7.9 Linux from Repo: base/7/x86_64 I had to check whether the repository was installed on the server.

To get a list of all yum repositories avaiable 

[root@centos:/etc/yum.repos.d]# yum repolist all
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.telecoms.bg
 * epel: mirrors.netix.net
 * extras: centos.telecoms.bg
 * remi: mirrors.netix.net
 * remi-php74: mirrors.netix.net
 * remi-safe: mirrors.netix.net
 * updates: centos.telecoms.bg
repo id                                repo name                                                                         status
base/7/x86_64                          CentOS-7 – Base                                                                   enabled: 10,072
base-debuginfo/x86_64                  CentOS-7 – Debuginfo                                                              disabled
base-source/7                          CentOS-7 – Base Sources                                                           disabled
c7-media                               CentOS-7 – Media                                                                  disabled
centos-kernel/7/x86_64                 CentOS LTS Kernels for x86_64                                                     disabled
centos-kernel-experimental/7/x86_64    CentOS Experimental Kernels for x86_64                                            disabled
centosplus/7/x86_64                    CentOS-7 – Plus                                                                   disabled
centosplus-source/7                    CentOS-7 – Plus Sources                                                           disabled
cr/7/x86_64                            CentOS-7 – cr                                                                     disabled
epel/x86_64                            Extra Packages for Enterprise Linux 7 – x86_64                                    enabled: 13,667
epel-debuginfo/x86_64                  Extra Packages for Enterprise Linux 7 – x86_64 – Debug                            disabled
epel-source/x86_64                     Extra Packages for Enterprise Linux 7 – x86_64 – Source                           disabled
epel-testing/x86_64                    Extra Packages for Enterprise Linux 7 – Testing – x86_64                          disabled
epel-testing-debuginfo/x86_64          Extra Packages for Enterprise Linux 7 – Testing – x86_64 – Debug                  disabled
epel-testing-source/x86_64             Extra Packages for Enterprise Linux 7 – Testing – x86_64 – Source                 disabled
extras/7/x86_64                        CentOS-7 – Extras                                                                 enabled:    500
extras-source/7                        CentOS-7 – Extras Sources                                                         disabled
fasttrack/7/x86_64                     CentOS-7 – fasttrack                                                              disabled
remi                                   Remi's RPM repository for Enterprise Linux 7 – x86_64                             enabled:  7,229
remi-debuginfo/x86_64                  Remi's RPM repository for Enterprise Linux 7 – x86_64 – debuginfo                 disabled
remi-glpi91                            Remi's GLPI 9.1 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-glpi92                            Remi's GLPI 9.2 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-glpi93                            Remi's GLPI 9.3 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-glpi94                            Remi's GLPI 9.4 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-modular                           Remi's Modular repository for Enterprise Linux 7 – x86_64                         disabled
remi-modular-test                      Remi's Modular testing repository for Enterprise Linux 7 – x86_64                 disabled
remi-php54                             Remi's PHP 5.4 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php55                             Remi's PHP 5.5 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php55-debuginfo/x86_64            Remi's PHP 5.5 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
!remi-php56                            Remi's PHP 5.6 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php56-debuginfo/x86_64            Remi's PHP 5.6 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php70                             Remi's PHP 7.0 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php70-debuginfo/x86_64            Remi's PHP 7.0 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php70-test                        Remi's PHP 7.0 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php70-test-debuginfo/x86_64       Remi's PHP 7.0 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php71                             Remi's PHP 7.1 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php71-debuginfo/x86_64            Remi's PHP 7.1 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php71-test                        Remi's PHP 7.1 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php71-test-debuginfo/x86_64       Remi's PHP 7.1 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
!remi-php72                            Remi's PHP 7.2 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php72-debuginfo/x86_64            Remi's PHP 7.2 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php72-test                        Remi's PHP 7.2 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php72-test-debuginfo/x86_64       Remi's PHP 7.2 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php73                             Remi's PHP 7.3 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php73-debuginfo/x86_64            Remi's PHP 7.3 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php73-test                        Remi's PHP 7.3 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php73-test-debuginfo/x86_64       Remi's PHP 7.3 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php74                             Remi's PHP 7.4 RPM repository for Enterprise Linux 7 – x86_64                     enabled:    423
remi-php74-debuginfo/x86_64            Remi's PHP 7.4 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php74-test                        Remi's PHP 7.4 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php74-test-debuginfo/x86_64       Remi's PHP 7.4 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php80                             Remi's PHP 8.0 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php80-debuginfo/x86_64            Remi's PHP 8.0 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php80-test                        Remi's PHP 8.0 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php80-test-debuginfo/x86_64       Remi's PHP 8.0 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-safe                              Safe Remi's RPM repository for Enterprise Linux 7 – x86_64                        enabled:  4,549
remi-safe-debuginfo/x86_64             Remi's RPM repository for Enterprise Linux 7 – x86_64 – debuginfo                 disabled
remi-test                              Remi's test RPM repository for Enterprise Linux 7 – x86_64                        disabled
remi-test-debuginfo/x86_64             Remi's test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo            disabled
updates/7/x86_64                       CentOS-7 – Updates                                                                enabled:  2,741
updates-source/7                       CentOS-7 – Updates Sources                                                        disabled
zabbix/x86_64                          Zabbix Official Repository – x86_64                                               enabled:    178
zabbix-debuginfo/x86_64                Zabbix Official Repository debuginfo – x86_64                                     disabled
zabbix-frontend/x86_64                 Zabbix Official Repository frontend – x86_64                                      disabled
zabbix-non-supported/x86_64            Zabbix Official Repository non-supported – x86_64                                 enabled:      5
repolist: 39,364

[root@centos:/etc/yum.repos.d]# yum repolist all|grep -i 'base/7/x86_64'
base/7/x86_64                       CentOS-7 – Base              enabled: 10,072

 

As you can see in CentOS 7 sharutils is enabled from default repositories, however this is not the case on Redhat 7.9, hence to install sharutils there you can one time enable RPM repository to install sharutils 

[root@centos:/etc/yum.repos.d]# yum –enablerepo=rhel-7-server-optional-rpms install sharutils

To install zabbix-agent on the same Redhat server, without caring that I need precisely  know the RPM repository that is providing zabbix agent that in that was (Repo: 3party/7Server/x86_64)  I had to:

[root@centos:/etc/yum.repos.d]# yum –enablerepo \* install zabbix-agent zabbix-sender


Permanently enabling repositories of course is possible via editting or creating fresh new file configuration manually on CentOS / Fedora under directory /etc/yum.repos.d/
On Redhat Enterprise Linux  servers it is easier to use the subscription-manager command instead, like this:
 

[root@rhel:/root]# subscription-manager repos –disable=epel/7Server/x86_64

[root@rhel:/root]# subscription-manager repos –enable=rhel-6-server-optional-rpms

How to enable HaProxy logging to a separate log /var/log/haproxy.log / prevent HAProxy duplicate messages to appear in /var/log/messages

Wednesday, February 19th, 2020

haproxy-logging-basics-how-to-log-to-separate-file-prevent-duplicate-messages-haproxy-haproxy-weblogo-squares
haproxy  logging can be managed in different form the most straight forward way is to directly use /dev/log either you can configure it to use some log management service as syslog or rsyslogd for that.

If you don't use rsyslog yet to install it: 

# apt install -y rsyslog

Then to activate logging via rsyslogd we can should add either to /etc/rsyslogd.conf or create a separte file and include it via /etc/rsyslogd.conf with following content:
 

Enable haproxy logging from rsyslogd


Log haproxy messages to separate log file you can use some of the usual syslog local0 to local7 locally used descriptors inside the conf (be aware that if you try to use some wrong value like local8, local9 as a logging facility you will get with empty haproxy.log, even though the permissions of /var/log/haproxy.log are readable and owned by haproxy user.

When logging to a local Syslog service, writing to a UNIX socket can be faster than targeting the TCP loopback address. Generally, on Linux systems, a UNIX socket listening for Syslog messages is available at /dev/log because this is where the syslog() function of the GNU C library is sending messages by default. To address UNIX socket in haproxy.cfg use:

log /dev/log local2 


If you want to log into separate log each of multiple running haproxy instances with different haproxy*.cfg add to /etc/rsyslog.conf lines like:

local2.* -/var/log/haproxylog2.log
local3.* -/var/log/haproxylog3.log


One important note to make here is since rsyslogd is used for haproxy logging you need to have enabled in rsyslogd imudp and have a UDP port listener on the machine.

E.g. somewhere in rsyslog.conf or via rsyslog include file from /etc/rsyslog.d/*.conf needs to have defined following lines:

$ModLoad imudp
$UDPServerRun 514


I prefer to use external /etc/rsyslog.d/20-haproxy.conf include file that is loaded and enabled rsyslogd via /etc/rsyslog.conf:

# vim /etc/rsyslog.d/20-haproxy.conf

$ModLoad imudp
$UDPServerRun 514​
local2.* -/var/log/haproxy2.log


It is also possible to produce different haproxy log output based on the severiy to differentiate between important and less important messages, to do so you'll need to rsyslog.conf something like:
 

# Creating separate log files based on the severity
local0.* /var/log/haproxy-traffic.log
local0.notice /var/log/haproxy-admin.log

 

Prevent Haproxy duplicate messages to appear in /var/log/messages

If you use local2 and some default rsyslog configuration then you will end up with the messages coming from haproxy towards local2 facility producing doubled simultaneous records to both your pre-defined /var/log/haproxy.log and /var/log/messages on Proxy servers that receive few thousands of simultanous connections per second.
This is a problem since doubling the log will produce too much data and on systems with smaller /var/ partition you will quickly run out of space + this haproxy requests logging to /var/log/messages makes the file quite unreadable for normal system events which are so important to track clearly what is happening on the server daily.

To prevent the haproxy duplicate messages you need to define somewhere in rsyslogd usually /etc/rsyslog.conf local2.none near line of facilities configured to log to file:

*.info;mail.none;authpriv.none;cron.none;local2.none     /var/log/messages

This configuration should work but is more rarely used as most people prefer to have haproxy log being written not directly to /dev/log which is used by other services such as syslogd / rsyslogd.

To use /dev/log to output logs from haproxy configuration in global section use config like:
 

global
        log /dev/log local2 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

The log global directive basically says, use the log line that was set in the global section for whole config till end of file. Putting a log global directive into the defaults section is equivalent to putting it into all of the subsequent proxy sections.

Using global logging rules is the most common HAProxy setup, but you can put them directly into a frontend section instead. It can be useful to have a different logging configuration as a one-off. For example, you might want to point to a different target Syslog server, use a different logging facility, or capture different severity levels depending on the use case of the backend application. 

Insetad of using /dev/log interface that is on many distributions heavily used by systemd to store / manage and distribute logs,  many haproxy server sysadmins nowdays prefer to use rsyslogd as a default logging facility that will manage haproxy logs.
Admins prefer to use some kind of mediator service to manage log writting such as rsyslogd or syslog, the reason behind might vary but perhaps most important reason is  by using rsyslogd it is possible to write logs simultaneously locally on disk and also forward logs  to a remote Logging server  running rsyslogd service.

Logging is defined in /etc/haproxy/haproxy.cfg or the respective configuration through global section but could be also configured to do a separate logging based on each of the defined Frontend Backends or default section. 
A sample exceprt from this section looks something like:

#———————————————————————
# Global settings
#———————————————————————
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#———————————————————————
defaults
    mode                    tcp
    log                     global
    option                  tcplog
    #option                  dontlognull
    #option http-server-close
    #option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 7
    #timeout http-request    10s
    timeout queue           10m
    timeout connect         30s
    timeout client          20m
    timeout server          10m
    #timeout http-keep-alive 10s
    timeout check           30s
    maxconn                 3000

 

 

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

 

#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_DKV_PROD_WLPFO
    mode tcp
    bind 192.168.233.5:30000-31050
    option tcplog
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
    default_backend Default_Bakend_Name


#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_DKV_PROD_WLPFO
    mode tcp
    # (0) Load Balancing Method.
    balance source
    # (4) Peer Sync: a sticky session is a session maintained by persistence
    stick-table type ip size 1m peers hapeers expire 60m
    stick on src
    # (5) Server List
    # (5.1) Backend
    server Backend_Server1 10.10.10.1 check port 18088
    server Backend_Server2 10.10.10.2 check port 18088 backup


The log directive in above config instructs HAProxy to send logs to the Syslog server listening at 127.0.0.1:514. Messages are sent with facility local2, which is one of the standard, user-defined Syslog facilities. It’s also the facility that our rsyslog configuration is expecting. You can add more than one log statement to send output to multiple Syslog servers.

Once rsyslog and haproxy logging is configured as a minumum you need to restart rsyslog (assuming that haproxy config is already properly loaded):

# systemctl restart rsyslogd.service

To make sure rsyslog reloaded successfully:

systemctl status rsyslogd.service


Restarting HAproxy

If the rsyslogd logging to 127.0.0.1 port 514 was recently added a HAProxy restart should also be run, you can do it with:
 

# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)


Or to restart use systemctl script (if haproxy is not used in a cluster with corosync / heartbeat).

# systemctl restart haproxy.service

You can control how much information is logged by adding a Syslog level by

    log         127.0.0.1 local2 info


The accepted values are the standard syslog security level severity:

Value Severity Keyword Deprecated keywords Description Condition
0 Emergency emerg panic System is unusable A panic condition.
1 Alert alert   Action must be taken immediately A condition that should be corrected immediately, such as a corrupted system database.
2 Critical crit   Critical conditions Hard device errors.
3 Error err error Error conditions  
4 Warning warning warn Warning conditions  
5 Notice notice   Normal but significant conditions Conditions that are not error conditions, but that may require special handling.
6 Informational info   Informational messages  
7 Debug debug   Debug-level messages Messages that contain information normally of use only when debugging a program.

 

Logging only errors / timeouts / retries and errors is done with option:

Note that if the rsyslog is configured to listen on different port for some weird reason you should not forget to set the proper listen port, e.g.:
 

  log         127.0.0.1:514 local2 info

option dontlog-normal

in defaults or frontend section.

You most likely want to enable this only during certain times, such as when performing benchmarking tests.

(or log-format-sd for structured-data syslog) directive in your defaults or frontend
 

Haproxy Logging shortly explained


The type of logging you’ll see is determined by the proxy mode that you set within HAProxy. HAProxy can operate either as a Layer 4 (TCP) proxy or as Layer 7 (HTTP) proxy. TCP mode is the default. In this mode, a full-duplex connection is established between clients and servers, and no layer 7 examination will be performed. When in TCP mode, which is set by adding mode tcp, you should also add option tcplog. With this option, the log format defaults to a structure that provides useful information like Layer 4 connection details, timers, byte count and so on.

Below is example of configured logging with some explanations:

Log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"

haproxy-logged-fields-explained
Example of Log-Format configuration as shown above outputted of haproxy config:

Log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

haproxy_http_log_format-explained1

To understand meaning of this abbreviations you'll have to closely read  haproxy-log-format.txt. More in depth info is to be found in HTTP Log format documentation


haproxy_logging-explained

Logging HTTP request headers

HTTP request header can be logged via:
 

 http-request capture

frontend website
    bind :80
    http-request capture req.hdr(Host) len 10
    http-request capture req.hdr(User-Agent) len 100
    default_backend webservers


The log will show headers between curly braces and separated by pipe symbols. Here you can see the Host and User-Agent headers for a request:

192.168.150.1:57190 [20/Dec/2018:22:20:00.899] website~ webservers/server1 0/0/1/0/1 200 462 – – —- 1/1/0/0/0 0/0 {mywebsite.com|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/71.0.3578.80 } "GET / HTTP/1.1"

 

Haproxy Stats Monitoring Web interface


Haproxy is having a simplistic stats interface which if enabled produces some general useful information like in above screenshot, through which
you can get a very basic in browser statistics and track potential issues with the proxied traffic for all configured backends / frontends incoming outgoing
network packets configured nodes
 experienced downtimes etc.

haproxy-statistics-report-picture

The basic configuration to make the stats interface accessible would be like pointed in above config for example to enable network listener on address
 

https://192.168.0.5:8080/stats


with hproxyuser / password config would be:

# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.5:8080                #Haproxy Monitoring run on port 8080
    mode http
    option httplog
    option http-server-close
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats                            #URL for HAProxy monitoring
    stats realm Haproxy\ Statistics
    stats auth hproxyauser:Password___          #User and Password for login to the monitoring dashboard

 

 

Sessions states and disconnect errors on new application setup

Both TCP and HTTP logs include a termination state code that tells you the way in which the TCP or HTTP session ended. It’s a two-character code. The first character reports the first event that caused the session to terminate, while the second reports the TCP or HTTP session state when it was closed.

Here are some essential termination codes to track in for in the log:
 

Here are some termination code examples most commonly to see on TCP connection establishment errors:

Two-character code    Meaning
—    Normal termination on both sides.
cD    The client did not send nor acknowledge any data and eventually timeout client expired.
SC    The server explicitly refused the TCP connection.
PC    The proxy refused to establish a connection to the server because the process’ socket limit was reached while attempting to connect.


To get all non-properly exited codes the easiest way is to just grep for anything that is different from a termination code –, like that:

tail -f /var/log/haproxy.log | grep -v ' — '


This should output in real time every TCP connection that is exiting improperly.

There’s a wide variety of reasons a connection may have been closed. Detailed information about all possible termination codes can be found in the HAProxy documentation.
To get better understanding a very useful reading to haproxy Debug errors with  is in haproxy-logging.txt in that small file are collected all the cryptic error messages codes you might find in your logs when you're first time configuring the Haproxy frontend / backend and the backend application behind.

Another useful analyze tool which can be used to analyze Layer 7 HTTP traffic is halog for more on it just google around.

Rsync copy files with root privileges between servers with root superuser account disabled

Tuesday, December 3rd, 2019

 

rsync-copy-files-between-two-servers-with-root-privileges-with-root-superuser-account-disabled

Sometimes on servers that follow high security standards in companies following PCI Security (Payment Card Data Security) standards it is necessery to have a very weird configurations on servers,to be able to do trivial things such as syncing files between servers with root privileges in a weird manners.This is the case for example if due to security policies you have disabled root user logins via ssh server and you still need to synchronize files in directories such as lets say /etc , /usr/local/etc/ /var/ with root:root user and group belongings.

Disabling root user logins in sshd is controlled by a variable in /etc/ssh/sshd_config that on most default Linux OS
installations is switched on, e.g. 

grep -i permitrootlogin /etc/ssh/sshd_config
PermitRootLogin yes


Many corporations use Vulnerability Scanners such as Qualys are always having in their list of remote server scan for SSH Port 22 to turn have the PermitRootLogin stopped with:

 

PermitRootLogin no


In this article, I'll explain a scenario where we have synchronization between 2 or more servers Server A / Server B, whatever number of servers that have already turned off this value, but still need to
synchronize traditionally owned and allowed to write directories only by root superuser, here is 4 easy steps to acheive it.

 

1. Add rsyncuser to Source Server (Server A) and Destination (Server B)


a. Execute on Src Host:

 

groupadd rsyncuser
useradd -g 1000 -c 'Rsync user to sync files as root src_host' -d /home/rsyncuser -m rsyncuser

 

b. Execute on Dst Host:

 

groupadd rsyncuser
useradd -g 1000 -c 'Rsync user to sync files dst_host' -d /home/rsyncuser -m rsyncuser

 

2. Generate RSA SSH Key pair to be used for passwordless authentication


a. On Src Host
 

su – rsyncuser

ssh-keygen -t rsa -b 4096

 

b. Check .ssh/ generated key pairs and make sure the directory content look like.

 

[rsyncuser@src-host .ssh]$ cd ~/.ssh/;  ls -1

id_rsa
id_rsa.pub
known_hosts


 

3. Copy id_rsa.pub to Destination host server under authorized_keys

 

scp ~/.ssh/id_rsa.pub  rsyncuser@dst-host:~/.ssh/authorized_keys

 

Next fix permissions of authorized_keys file for rsyncuser as anyone who have access to that file (that exists as a user account) on the system
could steal the key and use it to run rsync commands and overwrite remotely files, like overwrite /etc/passwd /etc/shadow files with his custom crafted credentials
and hence hack you 🙂
 

Hence, On Destionation Host Server B fix permissions with:
 

su – rsyncuser; chmod 0600 ~/.ssh/authorized_keys
[rsyncuser@dst-host ~]$


An alternative way for the lazy sysadmins is to use the ssh-copy-id command

 

$ ssh-copy-id rsyncuser@192.168.0.180
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys
root@192.168.0.180's password: 
 

 

For improved security here to restrict rsyncuser to be able to run only specific command such as very specific script instead of being able to run any command it is good to use little known command= option
once creating the authorized_keys

 

4. Test ssh passwordless authentication works correctly


For that Run as a normal ssh from rsyncuser

On Src Host

 

[rsyncuser@src-host ~]$ ssh rsyncuser@dst-host


Perhaps here is time that for those who, think enabling a passwordless authentication is not enough secure and prefer to authorize rsyncuser via a password red from a secured file take a look in my prior article how to login to remote server with password provided from command line as a script argument / Running same commands on many servers 

5. Enable rsync in sudoers to be able to execute as root superuser (copy files as root)

 


For this step you will need to have sudo package installed on the Linux server.

Then, Execute once logged in as root on Destionation Server (Server B)

 

[root@dst-host ~]# grep 'rsyncuser ALL' /etc/sudoers|wc -l || echo ‘rsyncuser ALL=NOPASSWD:/usr/bin/rsync’ >> /etc/sudoers
 

 

Note that using rsync with a ALL=NOPASSWD in /etc/sudoers could pose a high security risk for the system as anyone authorized to run as rsyncuser is able to overwrite and
respectivle nullify important files on Destionation Host Server B and hence easily mess the system, even shell script bugs could produce a mess, thus perhaps a better solution to the problem
to copy files with root privileges with the root account disabled is to rsync as normal user somewhere on Dst_host and use some kind of additional script running on Dst_host via lets say cron job and
will copy gently files on selective basis.

Perhaps, even a better solution would be if instead of granting ALL=NOPASSWD:/usr/bin/rsync in /etc/sudoers is to do ALL=NOPASSWD:/usr/local/bin/some_copy_script.sh
that will get triggered, once the files are copied with a regular rsyncuser acct.

 

6. Test rsync passwordless authentication copy with superuser works


Do some simple copy, lets say copy files on Encrypted tunnel configurations located under some directory in /etc/stunnel on Server A to /etc/stunnel on Server B

The general command to test is like so:
 

rsync -aPz -e 'ssh' '–rsync-path=sudo rsync' /var/log rsyncuser@$dst_host:/root/tmp/


This will copy /var/log files to /root/tmp, you will get a success messages for the copy and the files will be at destination folder if succesful.

 

On Src_Host run:

 

[rsyncuser@src-host ~]$ dst=FQDN-DST-HOST; user=rsyncuser; src_dir=/etc/stunnel; dst_dir=/root/tmp;  rsync -aP -e 'ssh' '–rsync-path=sudo rsync' $src_dir  $rsyncuser@$dst:$dst_dir;

 

7. Copying files with root credentials via script


The simlest file to use to copy a bunch of predefined files  is best to be handled by some shell script, the most simple version of it, could look something like this.
 

#!/bin/bash
# On server1 use something like this
# On server2 dst server
# add in /etc/sudoers
# rsyncuser ALL=NOPASSWD:/usr/bin/rsync

user='rsyncuser';

dst_dir="/root/tmp";
dst_host='$dst_host';
src[1]="/etc/hosts.deny";
src[2]="/etc/sysctl.conf";
src[3]="/etc/samhainrc";
src[4]="/etc/pki/tls/";
src[5]="/usr/local/bin/";

 

for i in $(echo ${src[@]}); do
rsync -aPvz –delete –dry-run -e 'ssh' '–rsync-path=sudo rsync' "$i" $rsyncuser@$dst_host:$dst_dir"$i";
done


In above script as you can see, we define a bunch of files that will be copied in bash array and then run a loop to take each of them and copy to testination dir.
A very sample version of the script rsync_with_superuser-while-root_account_prohibited.sh 
 

Conclusion


Lets do short overview on what we have done here. First Created rsyncuser on SRC Server A and DST Server B, set up the key pair on both copied the keys to make passwordless login possible,
set-up rsync to be able to write as root on Dst_Host / testing all the setup and pinpointing a small script that can be used as a backbone to develop something more complex
to sync backups or keep system configurations identicatial – for example if you have doubts that some user might by mistake change a config etc.
In short it was pointed the security downsides of using rsync NOPASSWD via /etc/sudoers and few ideas given that could be used to work on if you target even higher
PCI standards.

 

How to activate (enable) telnet client on Windows XP and Windows 7

Thursday, November 22nd, 2012

1. Enable / Disable Telnet Client command on Windows XP

By default telnet is enabled on Windows XP. However on some messed up Windows installations, cause of viruses or some custom users changes the PATH variable set for command prompt to C:\WINDOWS\system32\
is missing and therefore typing in telnet in cmd.exe fails to find the command. Thus it is good idea before trying to enable telnet with dism command to try running in Command Prompt

C:\WINDOWS\system32\telnet.exe,

 If the command is not found  by giving it full path, then you can read down the article how to enable it.
Launch command prompt cmd.exe and type:

dism /online /Enable-Feature /FeatureName:TelnetClient

dism command enable disable telnet client command on windows 7 and windows 2008 servers Launching again:

dism /online /Enable-Feature /FeatureName:TelnetClient

Disables telnet, whether it is enabled on system, on some systems XP admins might want to do this for increasing system security 

2. Enabling Telnet client using XP Graphical interface

Start (button) -> Control Panel -> Add / Remove Programs-> (Click on Turn Windows features on and Off )

3. Turn on Windows Telnet Client on Win 7  using Graphical Environment

Probably out of security, concerns within Microsoft took a wise decision to make Telnet Client disabled by default on Windows 7. Thus just trying to run it via cmd.exe users one gets 'telnet is not recognized as internal or external command, operable program or batch file'

Telnet not recognized as internal or external command message on Windows 7

Start (button) -> Control Panel -> Program and Features -> (Click on Turn Windows features on and Off )

turning on-telnet client control panel turn windows features on and off screenshot

You will get a dialog listing a number of things you can turn on you need to set tick on
Telnet Client

windows 7 windows features on and off enable telnet tickset

 Then a pop-up User Account Control will require you to enter administrator user password (if not already logged in as administrator).

Please wait while Windows makes changes to features, installing telnet client post dialog window

Further on launch cmd.exe and type telnet

Start (menu) -> Start Search -> cmd.exe
or pressing Windows Button + R and type in cmd.exe and type telnet

4. Turn on ( Install ) Win 7 Telnet Client from command line:

Launch command prompt (cmd.exe) and type in:

C:\Windows> pkgmgr /iu:"TelnetClient"

Installing using pkgmgr /ui: "TelnetClient" should be able to install telnet also on Windows 2008, Windows 2008 R2 Servers and Windows Vista

Happy telnet-ing 🙂

Enable festival Linux (text-to-speech-system) to read/speak PDF and DOC files (Speech PDF and DOC in Festival Script)

Sunday, September 20th, 2009

Today I wondered if festival supports reading of PDF files on Linux? The answer due to my short research was NO!.

Well though I couldn’t find official program to speak PDFs for me it’s not such a big deal since it’s not so hard to convert PDF files into plain text files in Linux with pdftotext command. 

pdftotext is part of the poppler-utils which is a nice package which alsocontains pdfimages – enabling you to extract images from pdfs,
pdftohtml – pdf to html converter and
pdffonts – pdf font analyzier. The normal way to read PDF files via festival is: First use pdftotext to convert your PDF to text file

$ pdftotext filename.pdf outputfile.txt

and then to make computer speak it over festival default configured synthesizer:

$ cat outfile.txt | festival –tts
For convenience I’ve created a small shell script I calledfestival-read-pdf.sh which does this directly.

Please download the festival-read-pdf.sh shell script here Furthermore I wondered how to make the Microsoft Office .doc files to be played throughfestival. On that account It was required something to convert again the .doc file extension to plain text. I came across antiword which I’ve blogged about in my previous post. Thus to carry it via festival you need to: antiword filename.doc | festival –tts I’ve fastly scripted it for some convenience. Download the festival-doc-read.sh script here I’ve also created a third bash script which enables you to select either to play DOC or PDF file in Festival.
Here is a link to the festival’s festival-read-doc-en-pdf.sh PDF, DOC speaker script .
Talking about festival it might be interesting to mention fala – A simple text reader.If you’re a Debian user you’ll be glad to know there is already a package containg fala. Well I hope you’ll find the PDF, DOC festival speech scripts useful. Enjoy

END—–

Enable write “write” command between logged in users on Debian GNU / Linux

Tuesday, October 9th, 2012

efault Debian GNU / Linux install does not permit messaging between ssh logged in users. Messages are disabled like this for security reasons as if they are on by default it is quite easy to flood one’s terminal with messages using a little loop like for instance:


while [ 1 ]; do
echo "You're flooded" | mesg username
done

Hence smartly, all users between write is switched off, i.e. mesg n

For those unfamiliar with mesg I suggest you check man mesg – which is one of the shortest UNIX manual written 🙂

Mesg head manual description is:


mesg - control write access to your terminal

Options mesg can accept are either yes or no ( y / n ).
To check on current logged in user if write username messaging is turned on, on any logged in user shell use:


# mesg
mesg is n

While mesg is set to no by default, if you try to message a random logged in system user you will get a message like:


$ write testuser
write: write: you have write permission turned off.

It is actually, quite handy to have messages switched on especially if you have a Linux host with user accounts which are friends of yours and
it is not very likely mesg is used for bad.

To change the default mesg n to mesg y you need to edit /etc/bash.bashscr (in case if all users are configured to use bash) or even better to set mesg y for all existing users add a new line on top or at the end of /etc/profile file:


echo 'mesg y' >> /etc/profile

On next login via ssh or physical tty, messaging will be on. To check re-login and type:


$ mesg
is y

One note to make here, is even though if you set messaging to yes for all users via /etc/profile, still for some reason the root user m
essaging keeps set to NO.


$ mesg
is y

One note to make here, is even though if you set messaging to yes for all users via /etc/profile, still for some reason the root user m
essaging keeps set to NO.


root@debian:~# mesg
is n

I have no clue, like this happens, but if you need to enable mesg to root as well add mesg y to /root/.bashrc

Well that’s all, I hope this helps someone 🙂 Cheers.

How to enable cron log separately in cron.log in Debian GNU / Linux

Thursday, October 4th, 2012

Default logging mechanism in and rest of its forks Ubuntu, Xubuntu whatever is done that there is no separate logging for cron daemon (/usr/sbin/cron). Instead all output messages from cron service gets logged in /var/log/auth.log

I don’t like this and since I ever remember using Linux I always liked it whether cron logs its activities in /var/log/cron.log.

Hence on each and every new Desktop or Server install, one of my first things is to set crond log msg separately in /var/log/cron.log.

In older Debians syslog, was used whether in Debian Squeeze, (Wheezy) and onwards releases rsyslog is used.

On hosts depending on syslogd or rsyslogd to manage logs, to enable cron.log just uncomment in either /etc/syslog.conf or /etc/rsyslog.conf:


# cron.* /var/log/cron.log


cron.* /var/log/cron.log

Create cron.log


linux:~# touch /var/log/cron.log

And restart syslogd / rsyslogd and cron


linux:~# /etc/init.d/rsyslogd restart
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.
linux:~# /etc/init.d/cron restart
Restarting periodic command scheduler: cron.
....

Check cron.log to see it logs fine:


linux:~# tail -n 5 /var/log/cron.log
Oct 4 16:31:27 pcfreak /usr/sbin/cron[32413]: (CRON) STARTUP (fork ok)
Oct 4 16:31:27 pcfreak /usr/sbin/cron[32413]: (CRON) INFO
(Skipping @reboot jobs -- not system startup)
Oct 4 16:32:35 pcfreak /usr/sbin/cron[32468]: (CRON) INFO
(pidfile fd = 3)
Oct 4 16:32:35 pcfreak /usr/sbin/cron[32469]: (CRON) STARTUP
(fork ok)
Oct 4 16:32:35 pcfreak /usr/sbin/cron[32469]: (CRON) INFO
(Skipping @reboot jobs -- not system startup)