Posts Tagged ‘zabbix-agent’

Install specific zabbix-agent version / Downgrade Zabbix Agent client to exact preferred old RPM version on CentOS / Fedora / RHEL Linux from repo

Wednesday, June 7th, 2023

zabbix-update-downgrade-on-centos-rhel-fedora-and-other-rpm-based-linux-zabbix-logo

 

In below article, I'll give you the short Update zabbix procedure to specific version release, if you need to have it running in tandem with rest of zabbix infra, as well as expain shortly how to downgrade zabbix version to a specific release number
to match your central zabbix-serveror central zabbix proxies.

The article is based on personal experience how to install / downgrade the specific zabbuix-agent  release on RPM based distros.
I know this is pretty trivial stuff but still, hope this might be useful to some sysadmin out there thus I decided to quickly blog it.

 

1. Prepare backup of zabbix_agentd.conf
 

# cp -rpf /etc/zabbix/zabbix_agentd.conf /home/your-user/zabbix_agentd.conf.bak.$(date +"%b-%d-%Y")

 

2. Create zabbix repo source file in yum.repos.d directory

# cd /etc/yum.repos.d/
# vim zabbix.repo 

 

[zabbix-5.0]

name=Zabbix 5.0 repo

baseurl=http://zabixx-rpm-mirrors-site.com/centos/external/zabbix-5.0/8/x86_64/

enabled=1

gpgcheck=0

 

3. Update zabbix-agent to a specific defined version

# yum search zabbix-agent –enablerepo zabbix-5.0

To update zabbix-agent for RHEL 7.*

# yum install zabbix-agent-5.0.34-1.el7.x86_64


For RHEL 8.*

# yum install zabbix-agent-5.0.34-1.el8.x86_64


4. Restart zabbix-agentd and check its status to make sure it works correctly
 

# systemctl status zabbix-agentd
# systemctl restart zabbix-agentd
# systemctl status zabbix-agentd


Go to zabbix-server WEB GUI interface and check that data is delivered as normally in Latest Data for the host fom recent time, to make sure host monitoring is continuing flawlessly as before change.

NB !: If yum use something like versionlock is enabled remove the versionlock for package and update then, otherwise it will (weirldly look) look like the package is missing.
I'm saying that because I've hit this issue and was wondering why i cannot install the zabbix-agent even though the version is listed, available and downloadable from the repository.


5. Downgrade agent-client to specific version (Install old version of Zabbix from Repo)
 

Sometimes by mistake you might have raised the Zabbix-agent version to be higher release than the zabbix-server's version and thus breach out the Zabbix documentation official recommendation to keep
up the zabbix-proxy, zabbix-server and zabbix-agent at the exactly same version major and minor version releases. 

If so, then you would want to decrease / downgrade the version, to match your Zabbix overall infrastructure exact version for each of Zabbix server -> Zabbix Proxy server -> Agent clients.

To downgrade the version, I prefer to create some backups, just in case for all /etc/zabbix/ configurations and userparameter scripts (from experience this is useful as sometimes some RPM binary update packages might cause /etc/zabbix/zabbix_agentd.conf file to get overwritten. To prevent from restoring zabbix_agentd.conf from your most recent backup hence, I prefer to just crease the zabbix config backups manually.
 

# cd /root

# mkdir -p /root/backup/zabbix-agent 

# tar -czvf zabbix_agent.tar.gz /etc/zabbix/

# tar -xzvf zabbix_agent.tar.gz 


Then list the available installable zabbix-agent versions
 

[root@sysadminshelp:~]# yum –showduplicates list zabbix-agent
Заредени плъгини: fastestmirror
Determining fastest mirrors
 * base: centos.uni-sofia.bg
 * epel: fedora.ipacct.com
 * extras: centos.uni-sofia.bg
 * remi: mirrors.uni-ruse.bg
 * remi-php74: mirrors.uni-ruse.bg
 * remi-safe: mirrors.uni-ruse.bg
 * updates: centos.uni-sofia.bg
Инсталирани пакети
zabbix-agent.x86_64                                                     5.0.30-1.el7                                                     @zabbix
Налични пакети
zabbix-agent.x86_64                                                     5.0.0-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.1-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.2-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.3-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.4-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.5-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.6-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.7-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.8-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.9-1.el7                                                      zabbix
zabbix-agent.x86_64                                                     5.0.10-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.11-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.12-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.13-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.14-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.15-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.16-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.17-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.18-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.19-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.20-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.21-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.22-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.23-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.24-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.25-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.26-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.27-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.28-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.29-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.30-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.31-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.32-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.33-1.el7                                                     zabbix
zabbix-agent.x86_64                                                     5.0.34-1.el7                                                     zabbix

 

Next lets install the most recent zabbix-versoin from the CentOS repo, which for me as of time of writting this article is 5.0.34.

# yum downgrade -y zabbix-agent-5.0.34-1.el7

# cp -rpf /root/backup/zabbix-agent/etc/zabbix/zabbix_agentd.conf /etc/zabbix/

# systemctl start zabbix-agent.service

# systemctl enable  zabbix-agent.service
 

# zabbix_agentd -V
zabbix_agentd (daemon) (Zabbix) 5.0.30
Revision 2c96c38fb4b 28 November 2022, compilation time: Nov 28 2022 11:27:43

Copyright (C) 2022 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 1.0.1e-fips 11 Feb 2013
Running with OpenSSL 1.0.1e-fips 11 Feb 2013
 

That's all folks you should be at your custom selected preferred version of zabbix-agent.
Enjoy ! 🙂

Tags: , , , , , , , , ,
Posted in Linux, Linux Package Management, Monitoring, Various, Zabbix | 2 Comments »

Enable PSK encryption on Zabbix Agent (client) sent encrypted monitored datas to Zabbix server

Friday, April 7th, 2023

zabbix-client-server-encryption-public-key-exchange

Those concerned of security and in use of their Zabbix monitored data who communicate Zabbix collected agent
data over internet or via some kind of untrusted network might definitely not enjoy the fact that zabbix-agent sents
its collected data to server in a plain text. Clear text data is allowing any network sniffer to possibly collect your
monitored server and hardware devices data and exposes all data sent over the network to same problems like in the past
the old uencrypted SMTP protocol.

To mitigate those great security hole for the paranoid sys admin it is rather easy to enable PSK (Pre Shared Key) based encryption.
To generate Pre Shared key you have to had to important values present

1. PSK Identity
2. PSK Secret

PSK secret should be minimum of 128 bit (16-byte PSK, entered as 32 hexadecimal digits), and supports up to
2048 bit (256-byte PSK, entered as 512 hexadecimal digits)

Usually something like 256 bit PSK secret on the machine should be strong enough and simply generated by running

# openssl rand -hex 32

1. Agent to zabbix server or proxy connection config

In /etc/zabbix/zabbix_agentd.conf for a Server Active (e.g. server to actively request the client to sent its collected data)
On machine running zabbix-agent should have a configuration similar to:

# cat /etc/zabbix/zabbix_agentd.conf

PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0

# IP of the machine
SourceIP=10.10.10.30
# turn it on if you need to execute to remote machine commands
EnableRemoteCommands=0

# IP of the server
servers=10.30.50.80
ListenPort=10080

# IP of the machine
ListenIP=10.30.30.31

# IP of the server
ServerActive=10.30.50.80

HostMetadataItem=system.uname
BufferSize=5400
MaxLinesPerSecond=5
Timeout=10
AllowRoot=0
StartAgents=5
LogRemoteCommands=0


# Machine hostname
Hostname=fqdn-of-zabbix-data-collect-server.com
Include=/etc/zabbix/zabbix_agentd.d/*.conf

# Encryption
TLSConnect=psk
TLSAccept=psk
TLSPSKIdentity=PSK to Zabbix Server5
TLSPSKFile=/etc/zabbix/zabbix_agentd.psk


! Important security note

!!! The TLSPSKIdentity value you decide will not be encrypted on transport, so don't use anything sensitive.

Once you include the TSL config

2 Generate / Create Zabbix Agent Key

Generate the key with pseudo-random bites inside /etc/zabbix/zabbix_agentd_key.psk

# cd /etc/zabbix
# openssl rand -hex 32 > zabbix_agentd_key.psk
# chown zabbix:zabbix zabbix_agentd_key.psk
# chmod 600 zabbix_agentd_key.psk

3. Configure PSK encryption in Zabbix Server Web User interface

Go to Zabbix Server User interface in browser and configure the PSK encryption options for the host.

Select the:

'Connections to host' = PSK

'Connections from host' = PSK

'PSK Identity' = [public-value-configured-in-Zabbix-agent-config]

'PSK' = [paste the long hex string generated from the OpenSSL command above]


In some seconds up to a minute or two the Zabbix Server and Agent will successfully communicate using PSK encryption.
Making the monitored data unreadable in plain text for malignant sniffers hanging in the middle equipment between the zabbix-agent and zabbix-server hosts.

4. PSK encryption behind a Proxy

Many companies, nowadays use zabbix proxy for improvement of network infrastrucutre. For example it is used to offload the zabbix-server when multiple zabbix-agents have to report various datas or to monitor servers and devices that are phyisically in separate networks or data centers (are passing through paranoic built firewalls) or monitor locations are having unreliable communications between each other.
 

To enable PSK for communications between your Zabbix Server and Zabbix Proxy.

1. Create a new secret, and add the PSK Identity and Secret to

Administration ⇾ Proxies ⇾ [Your proxy] ⇾ Encryption

2. Adjust the settings inside the zabbix proxies configuration file at /etc/zabbix/zabbix_proxy.conf


If setting up PSK encryption for agents behind a Zabbix proxy, ensure your have

Zabbix Server ⇽⇾ Proxy PSK enabled
first in Zabbix Server UI.

This is because, when you start the Proxy, or do some testing to send some key value to Zabbix server via the proxy with commands :

# zabbix_get -s 127.0.0.1 -k system.hostname
# zabbix_server -R config_cache_reload


config_cache_reload, the Proxy will download all its host settings from the server, and this also includes the servers copy of the secret.

The proxy needs to know the secret since it is now managing the communications on behalf of the server.

3. To add PSK encryption for any Agents behind a proxy, then you continue to set up the Agents as normal by creating a new secret, editing

Configuration ⇾ Hosts ⇾ [Your Host] ⇾ Encryption page

and also editing /etc/zabbix/zabbix_agentd.conf.

Remember that, since your Agents Host configuration in the Zabbix UI will be set as Monitored by Proxy, the PSK settings will be applicable for communications happening between the Zabbix Proxy and the Agent that it is monitoring, not between the Zabbix Server and the Agent behind the proxy.

You can also add PSK Encryption between your Zabbix Proxy and its own local Agent if you want.
You would set its PSK settings in the Proxy Agents host configuration at

Configuration ⇾ Hosts ⇾ [Your proxy] ⇾ Encryption

and modify the settings in the agents on configuration file at /etc/zabbix/zabbix_agentd.conf.
Keep in mind, this is only applicable to communications between the Zabbix Proxy, and its own Agent process.

When setting up PSK encryption for the Zabbix Server, Proxy and Agents, you may see an error in the Proxy logs,

cannot send proxy data to server at "zabbix.your-domain.tld": connection of type "TLS with PSK" is not allowed for proxy "your-proxy".

If you hit this, check that your

Zabbix Server ⇽⇾ Proxy PSK settings

are correct first.

Don't get confused between the Proxies own optional agent process, and its main Proxy process which is required.

Tags: , , , , , , , , , , , , ,
Posted in Computer Security, Linux, Monitoring, System Administration, Zabbix | 1 Comment »

Enable zabbix agent to work with SeLinux enabled on CentOS 7 Linux

Wednesday, October 19th, 2022

If you have the task to install and use zabbix-agent or zabbix-proxy to report to zabbix-server on CentOS 7 with enabled SeLinux services for security reasons and you have no mean to disable the selinux which is a common step to take under this circumstances, you will have to add the zabbix services to be exluded as permissive in selinux. In below article I'll show you how this is done in few easy steps.

zabbix-agent-service-selinux-linux-real-time-operating-sytems

 

1. Check the system sestatus

[root@linux zabbix]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28


2. Enable zabbix to be permissive in selinux

To be able to set zabbix to be in permissive mode as well as for further troubleshooting if you have to enable other  linux services inside selinux you have to install below RPM packs.

[root@linux zabbix]# yum install setroubleshoot.x86_64 setools.x86_64 setools-console.x86_64 policycoreutils-python.x86_64

Set the zabbix permissive exclude rule in SeLINUX

[root@linux zabbix]# semanage permissive –add zabbix_t

Re-run the zabbix proxy (if you have a local zabbix-proxy running and the zabbix-agent)

[root@linux zabbix]# systemctl start zabbix-proxy.service

[root@linux zabbix]# systemctl start zabbix-agent.service

[root@linux zabbix]# systemctl status zabbix-agent
● zabbix-agent.service – Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-10-18 09:30:16 CEST; 1 day 7h ago
 Main PID: 962952 (zabbix_agentd)
    Tasks: 6 (limit: 100884)
   Memory: 5.1M
   CGroup: /system.slice/zabbix-agent.service
           ├─962952 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─962955 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─962956 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─962957 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─962958 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─962959 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Oct 18 09:30:16 linux systemd[1]: Starting Zabbix Agent…
Oct 18 09:30:16 linux systemd[1]: Started Zabbix Agent.

3. Check inside audit logs all is OK

To make sure zabbix is really enabled to be omitted by selinux rules check audit.log

[root@linux zabbix]# grep zabbix_proxy /var/log/audit/audit.log

That's all folks, Enjoy ! 🙂

Tags: , , , , , , , , , , , , , , ,
Posted in Migration, System Administration, Zabbix | 1 Comment »

Monitor service log is continously growing with Zabbix on Windows with batch userparameter script and trigger Alert if log is unchanged

Thursday, March 17th, 2022

monitor-if-log-file-is-growing-with-zabbix-zabbix-userparameter-script-howto

Recently we had an inteteresting Monitoring work task to achieve. We have an Application that is constantly simulating encrypted connections traffic to a remote side machine and sending specific data on TCP/IP ports.
Communiucation between App Server A -> App Server B should be continous and if all is working as expected App Server A messages output are logged in the Application log file on the machine which by the way Runs
Windows Server 2020.

Sometimes due to Network issues this constant reconnections from the Application S. A to the remote checked machine TCP/IP ports gets interrupted due to LAN issues or a burned Network Switch equipment, misconfiguration on the network due to some Network admin making stoopid stuff etc..

Thus it was important to Monitor somehow whether the log is growing or not and feed the output of whether Application log file is growing or it stuck to a Central Zabbix Server. 
To be able to better understand the task, lets divide the desired outcome in few parts on required:

1. Find The latest file inside a folder C:\Path-to-Service\Monitoring\Log\
2. Open the and check it is current logged records and log the time
3. Re-open file in a short while and check whether in few seconds new records are written
4. Report the status out to Zabbix
5. Make Zabbix Item / Trigger / Action in case if monitored file is not growing

In below article I'll briefly explain how Monitoring a Log on a Machine for growing was implemented using a pure good old WIN .BAT (.batch) script and Zabbix Userparameter key

 

1. Enable userparameter script for Local Zabbix-Agent on the Windows 10 Server Host


Edit Zabbix config file usually on Windows Zabbix installs file is named:

[ zabbix_agentd.win ]


Uncomment the following lines to enable userparameter support for zabbix-agentd:

 

# Include=c:\zabbix\zabbix_agentd.userparams.conf

Include=c:\zabbix\zabbix_agentd.conf.d\

# Include=c:\zabbix\zabbix_agentd.conf.d\*.conf


2. Create folders for userparameter script and for the userparameter.conf

Before creating userparameter you can to create the folder and grant permissions

Folder name under C:\Zabbix -> zabbix_agentd.conf.d

If you don't want to use Windows Explorer) but do it via cmd line:

C:\Users\LOGUser> mkdir \Zabbix\zabbix_agentd.conf\
C:\User\LOGUser> mkdir \Zabbix\zabbix_scripts\


3. Create Userparameter with some name file ( Userparameter-Monitor-Grow.conf )

In the directory C:\Zabbix\zabbix_agentd.conf.d you should create a config file like:
Userparameter-Monitor-Grow.conf and in it you should have a standard userparameter key and script so file content is:

UserParameter=service.check,C:\Zabbix\zabbix_scripts\GROW_LOG_MONITOR-USERPARAMETER.BAT


4. Create the Batch script that will read the latest file in the service log folder and will periodically check and report to zabbix that file is changing

notepad C:\Zabbix\zabbix_scripts\GROW_LOG_MONITOR-USERPARAMETER.BAT

REM "SCRIPT MONITOR IF FILE IS GROWING OR NOT"
@echo off

set work_dir=C:\Path-to-Service\Monitoring\Log\

set client=client Name

set YYYYMMDD=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%

set name=csv%YYYYMMDD%.csv

set mytime=%TIME:~0,8%

for %%I in (..) do set CurrDirName=%%~nxI

 

setlocal EnableDelayedExpansion

set "line1=findstr /R /N "^^" %work_dir%\output.csv | find /C ":""


for /f %%a in ('!line1!') do set number1=%%a

set "line2=findstr /R /N "^^" %work_dir%\%name% | find /C ":""


for /f %%a in ('!line2!') do set number2=%%a

 

IF  %number1% == %number2% (

echo %YYYYMMDD% %mytime% MAJOR the log is not incrementing for %client%

echo %YYYYMMDD% %mytime% MAJOR the log is not incrementing for %client% >> monitor-grow_err.log

) ELSE (

echo %YYYYMMDD% %mytime% NORMAL the log is incrementing for %client%

SETLOCAL DisableDelayedExpansion

del %work_dir%\output.csv

FOR /F "usebackq delims=" %%a in (`"findstr /n ^^ %work_dir%\%name%"`) do (

    set "var=%%a"

    SETLOCAL EnableDelayedExpansion

    set "var=!var:*:=!"

    echo(!var! >> %work_dir%\output.csv

    ENDLOCAL

)

)
 

 

To download GROW_LOG_MONITOR-USERPARAMETER.BAT click here.
The script needs to have configured the path to directory containing multiple logs produced by the Monitored Application.
As prior said it will, list the latest created file based on DATE timestamp in the folder will output a simple messages:

If the log file is being fed with data the script will output to output.csv messages continuously, either:

%%mytime%% NORMAL the log is incrementing for %%client%%

Or if the Monitored application log is not writting anything for a period it will start writting messages

%%mytime%%mytime MAJOR the log is not incrementing for %client%

The messages will also be sent in Zabbix.

Before saving the script make sure you modify the Full Path location to the Monitored file for growing, i.e.:

set work_dir=C:\Path-to-Service\Monitoring\Log\


5. Create The Zabbix Item

Set whatever service.check name you would like and a check interval to fetch the info from the userparameter (if you're dealing with very large log files produced by Monitored log of application, then 10 minutes might be too frequent, in most cases 10 minutes should be fine)
monitor-if-log-grows-windows-zabbix-item-service-check-screenshot
 

6. Create Zabbix Trigger


You will need a Trigger something similar to below:

Now considering that zabbix server receives correctly data from the client and the monitored log is growing you should in Zabbix:

%%mytime%% NORMAL the log is incrementing for %%client%%


7. Lastly create an Action to send Email Alert if log is not growing

Tags: , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Zabbix | No Comments »