Posts Tagged ‘userparameter’

Monitoring chronyd time service is synchronized, get additional time server values with Zabbix userparameter script

Monday, March 21st, 2022

monitoring-chronyc-time-server-synchronization-zabbix-logo

If you''re running a server infrastructure and your main monitoring system is Zabbix. Then a vital check you might want to setup is to monitor the server time synchronization to a central server. In newer Linux OS-es ntpd time server is started to be used lesser and many modern Linux distributions used in the corporate realm are starting to recommend using chrony as a time synchronization client / server.

In this article, I'll show you how you can quickly setup monitoring of chronyd process and monitoring whether the time is successfully synchronizing with remote Chronyd time server. This will be done with a tiny one liner shell script setup as userparameter It is relatively easy then to setup an Action Alert


1. Create userparameter script to send parsed chronyd time synchronization to Zabbix Server

chronyc tracking provides plenty of useful data which can give many details about info such as offset, skew, root delay, stratum, update interval.

[root@server: ~]# chronyc tracking
Reference ID    : 0A32EF0B (fkf-intp01.intcs.meshcore.net)
Stratum         : 3
Ref time (UTC)  : Fri Mar 18 12:42:31 2022
System time     : 0.000032544 seconds fast of NTP time
Last offset     : +0.000031102 seconds
RMS offset      : 0.000039914 seconds
Frequency       : 3.037 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.023 ppm
Root delay      : 0.017352410 seconds
Root dispersion : 0.004285847 seconds
Update interval : 1041.6 seconds
Leap status     : Normal

[root@server zabbix_agentd.d]# cat userparameter_chrony.conf 
UserParameter=chrony.json,chronyc -c tracking | sed -e s/'^'/'{"chrony":[“‘/g -e s/’$’/'”]}'/g -e s/','/'","'/g
[root@server zabbix_agentd.d]#

The -c option passed to chronyc is printing the chronyc tracking command ouput data in comma-separated values ( CSV ) format.

2. Create Necessery Item key to get chronyd processes and catch the userparameter data

 

  • First lets create a an Item key to calculate the chronyd daemon proc.num
    proc.num – simply returns the number of processes in the process list just like a simple
    pgrep servicename command does.


monitroing-chronyc_zabbix_item_report_to-zabbix

Second lets create the Item for the userparameter script, the chrony.json key should be the same as the key given in the userparameter script.

obtain-chronyc-statistic-variables-from-remote-chronyd-to-zabbix-windows

Create Chrony Zabbix Triggers 
 

Expression 

{server-host:proc.num[chronyd].last()}<1


will be triggered if the process of chronyd on the server is less than 1

 

chronyc_monitoring_process-is-not-running-screenshot

Next configure

{server-host:chrony[Leap status].iregexp[Not synchronised) ]=1


to trigger Alert Chronyd is Not synchronized if the Expression check occurs.

chronyd-is-not-synchronized-trigger-iregexp

Reload the zabbix-agent on the server
 

To make zabbix-agent locally installed on the machine read the userparameter into memory  (in my case this is zabbix-agent-4.0.28-1.el8.x86_64) installed on Redhat 8.3 (Ootpa), you have to restart it.

[root@server: ~]# systemctl restart zabbix-agent
[root@server: ~]# systemctl status zabbix-agent

● zabbix-agent.service – Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-12-16 16:41:02 CET; 3 months 0 days ago
 Main PID: 862165 (zabbix_agentd)
    Tasks: 6 (limit: 23662)
   Memory: 20.6M
   CGroup: /system.slice/zabbix-agent.service
           ├─862165 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─862166 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─862167 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─862168 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─862169 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─862170 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


In a short while you should be seeing in the chrony.json key History data fed by the userparameter Script.

In Zabbix Latest data, you will see plenty of interesting time synchronization data get reported such as Skew, Stratum, Root Delay, Update Interval, Frequency etc.

chronyd-zabbix-reported-time-synchronization-offset-leap-residual-freq-root-delay-screenshot

To have an email Alerting further, go and setup a new Zabbix Action based on the Trigger with your likings and you're done. 
The tracked machine will be in zabbix to make sure your OS clock is not afar from the time server. Repeat the same steps if you need to track chronyd is up running and synchronized on few machines, or if you have to make it for dozens setup a Zabbix template.

Monitor service log is continously growing with Zabbix on Windows with batch userparameter script and trigger Alert if log is unchanged

Thursday, March 17th, 2022

monitor-if-log-file-is-growing-with-zabbix-zabbix-userparameter-script-howto

Recently we had an inteteresting Monitoring work task to achieve. We have an Application that is constantly simulating encrypted connections traffic to a remote side machine and sending specific data on TCP/IP ports.
Communiucation between App Server A -> App Server B should be continous and if all is working as expected App Server A messages output are logged in the Application log file on the machine which by the way Runs
Windows Server 2020.

Sometimes due to Network issues this constant reconnections from the Application S. A to the remote checked machine TCP/IP ports gets interrupted due to LAN issues or a burned Network Switch equipment, misconfiguration on the network due to some Network admin making stoopid stuff etc..

Thus it was important to Monitor somehow whether the log is growing or not and feed the output of whether Application log file is growing or it stuck to a Central Zabbix Server. 
To be able to better understand the task, lets divide the desired outcome in few parts on required:

1. Find The latest file inside a folder C:\Path-to-Service\Monitoring\Log\
2. Open the and check it is current logged records and log the time
3. Re-open file in a short while and check whether in few seconds new records are written
4. Report the status out to Zabbix
5. Make Zabbix Item / Trigger / Action in case if monitored file is not growing

In below article I'll briefly explain how Monitoring a Log on a Machine for growing was implemented using a pure good old WIN .BAT (.batch) script and Zabbix Userparameter key

 

1. Enable userparameter script for Local Zabbix-Agent on the Windows 10 Server Host


Edit Zabbix config file usually on Windows Zabbix installs file is named:

zabbix_agentd.win ]


Uncomment the following lines to enable userparameter support for zabbix-agentd:

 

# Include=c:\zabbix\zabbix_agentd.userparams.conf

Include=c:\zabbix\zabbix_agentd.conf.d\

# Include=c:\zabbix\zabbix_agentd.conf.d\*.conf


2. Create folders for userparameter script and for the userparameter.conf

Before creating userparameter you can to create the folder and grant permissions

Folder name under C:\Zabbix -> zabbix_agentd.conf.d

If you don't want to use Windows Explorer) but do it via cmd line:

C:\Users\LOGUser> mkdir \Zabbix\zabbix_agentd.conf\
C:\User\LOGUser> mkdir \Zabbix\zabbix_scripts\


3. Create Userparameter with some name file ( Userparameter-Monitor-Grow.conf )

In the directory C:\Zabbix\zabbix_agentd.conf.d you should create a config file like:
Userparameter-Monitor-Grow.conf and in it you should have a standard userparameter key and script so file content is:

UserParameter=service.check,C:\Zabbix\zabbix_scripts\GROW_LOG_MONITOR-USERPARAMETER.BAT


4. Create the Batch script that will read the latest file in the service log folder and will periodically check and report to zabbix that file is changing

notepad C:\Zabbix\zabbix_scripts\GROW_LOG_MONITOR-USERPARAMETER.BAT

REM "SCRIPT MONITOR IF FILE IS GROWING OR NOT"
@echo off

set work_dir=C:\Path-to-Service\Monitoring\Log\

set client=client Name

set YYYYMMDD=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%

set name=csv%YYYYMMDD%.csv

set mytime=%TIME:~0,8%

for %%I in (..) do set CurrDirName=%%~nxI

 

setlocal EnableDelayedExpansion

set "line1=findstr /R /N "^^" %work_dir%\output.csv | find /C ":""


for /f %%a in ('!line1!') do set number1=%%a

set "line2=findstr /R /N "^^" %work_dir%\%name% | find /C ":""


for /f %%a in ('!line2!') do set number2=%%a

 

IF  %number1% == %number2% (

echo %YYYYMMDD% %mytime% MAJOR the log is not incrementing for %client%

echo %YYYYMMDD% %mytime% MAJOR the log is not incrementing for %client% >> monitor-grow_err.log

) ELSE (

echo %YYYYMMDD% %mytime% NORMAL the log is incrementing for %client%

SETLOCAL DisableDelayedExpansion

del %work_dir%\output.csv

FOR /F "usebackq delims=" %%a in (`"findstr /n ^^ %work_dir%\%name%"`) do (

    set "var=%%a"

    SETLOCAL EnableDelayedExpansion

    set "var=!var:*:=!"

    echo(!var! >> %work_dir%\output.csv

    ENDLOCAL

)

)
 

 

To download GROW_LOG_MONITOR-USERPARAMETER.BAT click here.
The script needs to have configured the path to directory containing multiple logs produced by the Monitored Application.
As prior said it will, list the latest created file based on DATE timestamp in the folder will output a simple messages:

If the log file is being fed with data the script will output to output.csv messages continuously, either:

%%mytime%% NORMAL the log is incrementing for %%client%%

Or if the Monitored application log is not writting anything for a period it will start writting messages

%%mytime%%mytime MAJOR the log is not incrementing for %client%

The messages will also be sent in Zabbix.

Before saving the script make sure you modify the Full Path location to the Monitored file for growing, i.e.:

set work_dir=C:\Path-to-Service\Monitoring\Log\


5. Create The Zabbix Item

Set whatever service.check name you would like and a check interval to fetch the info from the userparameter (if you're dealing with very large log files produced by Monitored log of application, then 10 minutes might be too frequent, in most cases 10 minutes should be fine)
monitor-if-log-grows-windows-zabbix-item-service-check-screenshot
 

6. Create Zabbix Trigger


You will need a Trigger something similar to below:

Now considering that zabbix server receives correctly data from the client and the monitored log is growing you should in Zabbix:

%%mytime%% NORMAL the log is incrementing for %%client%%


7. Lastly create an Action to send Email Alert if log is not growing