Posts Tagged ‘tail’

Zabbix Power Shell PS1 script to write zero or one if string is matched inside log file

Monday, December 2nd, 2024

How to Install and Configure Zabbix Server and Client on Rocky Linux 9 - Cộng Đồng Linux

At work we had setup zabbix log file processing for few servers for a service that is doing a Monitoring Health Checks for a a special application via an encrypted strong encrypted tunnel. The app based on the check reports whether the remote side has processed data or not.
As me and my team are not maintainers of the zabbix-server where the zabbix-agents are sending the data, there is a multiple content of data being sent in simply "" empty strings via a zabbix Item setup. Those empty strings however gets stored in the zabbix-server database and since this check is made frequently about 500 hundred records of empty string lines are being written to the zabbix server, we got complaint by the zabbix adminsitrators, that we have to correct our Monitoring setup to not flood the zabbix-server.

Since zabbix cannot catch up the "" empty string and we cannot supress the string from being written in the Item, we needed a way to change the monitoring so that the configured Application check returns 1 (on error) and 0 (on success).

Zabbix even though advanced has a strange when zabbix log[] function, e.g. 

log[/path/to/log,,,,skip]

log function, used to analyze a log file and cut out last or first lines of a file simmilar to UNIX's  head and tail over log files this is described in the zabbix log file monitoring here . If a string is matched it can return string 1, but if nothing gets matched the result is empty string "" and this empty string cannot be used in a way to analyze the data with Item is used.

There is plenty of discussions online for this weird behavior and many people do offer different approaches to solve the strange situation, but as we have tried with our colleagues sys admins  none of those really worked out.

Thus we decided to use the classical way to work around, e.g. to simply use a powershell script that would check a number of lines inside a provided log file analyze if a string gets found and print out value of "1" if the string is matched or "0" "if not and this PS1 script to be set to run via a standard zabbix userparameter script.

This worked well, as all of us are mainly managing Linux systems, and we don't have enough knowledge on powershell we have used our internal Aartificial Intelligence (AI) clone tool to LibreChat – A free and open source ChatGPT clone.

LibreChat includes OpenAI's models, but also others — both open-source and closed-source — and its website promises "seamless integration" with AI services from OpenAI, Azure, Anthropic, and Google — as well as GPT-4, Gemini Vision, and many others. ("Every AI in one place," explains LibreChat's home page.) Plugins even let you make requests to DALL-E or Stable Diffusion for image generations. (LibreChat also offers a database that tracks "conversation state" — making it possible to switch to a different AI model in mid-conversation…)

$logfile = "C:\path\to\your\logfile.log"
$searchString = "-1"
 
# Get the last 140 lines
$lines = Get-Content $logfile -Tail 140
 
# Filter lines containing the search string
$found = $lines | Where-Object { $_ -match [regex]::Escape($searchString) }
 
# Output found lines or 0 if none were found
if ($found) {
    $found | ForEach-Object { $_ }
} else {
    Write-Host 0
}

You can download and the return_zero_or_one-if-string-matches-in-log-powershell.ps1 script here

Linux: List last 10 (newest) and 10 oldest modified files in a directory with ls

Tuesday, April 8th, 2014

An useful thing on GNU / Linux sometimes is to list last or oldest modified files in directory.

Lets say you want to list last 10 modified files with ls from today / yesterday. Here is how:
 

ls -1t | head -10
my.cnf
wordperss_enabled_plugins.txt
newcerts/
mysql-hipo_pcfreakbiz.dump
NewArchive-Jan-10-15.zip
hipo_pcfreakbiz-mysqldb-any-out-1389361234.tgz
Tisho_Snimki/
wordpress/
wp-cron.php?doing_wp_cron=1.1
wp-cron.php

 

To list 10 oldest modified files on Linux:

 

ls -1t | tail -10
    my.cnf
    pcfreak_sql-15_10_05_2012
    mysql-tuning-primer*
    tuning-primer.sh*
    system-administration-services.html
    blog_backup_15_07_2012.tar.gz
    www-files/
    dump.sql
    courier-imap*
    djbdns-1.05.tar.gz


Cheers 😉

How to count how many files are in a directory with find on Linux

Tuesday, February 21st, 2012

how to count how many directories are on your linux server

Did you ever needed to count, how many files in a directory are there?
Having the concrete number of files in a directory is not a seldom task but still very useful especially for scripts or simply for the sake of learning

The quickest and maybe the easiest way to count all files in a directory in Linux is with a combination of find and wc commands:

Here is how;

linux:~# cd ascii
linux:~/ascii# find . -type f -iname '*' -print |wc -l
407

This will find and list all matched files in any directory and subdirectories, print them out and count them with wc command.
The -type f argument instructs find to look only for files.

Other helpful variance of finding and listing all files in a directory and subdirectories is to list and count all the files with a certain file extension under a directory. For example, lets list all text files (.txt) contained in a directory and all level sub-directories:

linux:~/ascii# find . -type f -iname '*.txt' -print |wc -l
401

If you need to check the number of files in a directory for multiple directories on a server and you're aiming at doing it efficienly, issung above find .. | wc code will definitely be not a good choice. If used it will generate heavy load for the system and along with that will complete the execution in ages if issued on a large number of files containing dirs.

Thanksfully if efficiency is targetted, there is a command written in C called tree which is more efficient than find.
To count the number of files in dir but using tree :

linux:~# cd ascii
linux:/ascii# tree | tail -n 1
32 directories, 407 files

By default tree prints info for both the number of found files and directories.
To print out only the files matched, awk comes handy, e.g.:

linux:/ascii# tree |tail -n 1| awk '{ print $3 }'407

To list only the number of files in a directory without its existing sub-directories ls + wc use is also possible:

linux:~/ascii# ls -l | grep ^- | wc -l68

This result the above command would produce is +1 more than the real number of files, as it counts the directory ".." as one file (in UNIX / LINUX everything is file).

A short one liner script that can calculate all files correctly by substracting 1 is and hence present correct result on number of files is like so:

linux:~/ascii# var=$(ls -l | grep ^- | wc -l); var=$(($var - 1)); echo $var

ls can be used to calculate the number of 1-st level sub-directories under certain directory for instance:

linux:~/ascii# ls -l |grep ^d|wc -l
25

You see the ascii directory has 25 subdirectories in its 1st level.

To check symlinks under a directory with ls the command would be:

linux:~/ascii# ls -l | grep ^l | wc -l
0

Note above 3 ls | grep … examples, will not work properly if the directory contains files with SUID or some special properties set.
Hence to get the same 3 results for active files, directories and symbolic links, a one liner similar to the one below can be used instead:

linux:~/ascii# for t in files links directories; do echo `find . -type ${t:0:1} | wc -l` $t; done 2> /dev/null
407 files
0 links
33 directories

This will show statistics about all files, links and directories for all directory sub-levels.
Just in case if there is need to only count files, links and directories without directory recursion enabled, use:

linux:~/ascii# for t in files links directories; do echo `find . -maxdepth 1 -type ${t:0:1} | wc -l` $t; done 2> /dev/null
68 files
0 links
26 directories

Anyways the above bash loop will be slow, for directories containing thousands of files. For better performance the equivallent of above bash loop rewritten in perl would be:

linux:~/ascii# ls -l |perl -e 'while(<>){$h{substr($_,0,1)}+=1;} END {foreach(keys %h) {print "$_ $h{$_}\n";}}'
- 68
d 25
t 1
linux:~/ascii#
In any case the most preferrable and efficient way to count files en directories is by using tree command.
In my view using always tree command instead of code "hacks" is smart idea.

In Slackware tree command is part of the base install, on Debian and CentOS Linux, tree cmd is not part of the base system and requires install via apt / yum e.g.:

debian:~# apt-get --yes install tree
...

[root@centos:~ ]# yum --yes install tree

Happy counting 😉

How to test if USB Camera is working with Cheese on GNU / Linux

Friday, December 23rd, 2011

I just bought an USB Camera (my notebook doesn't include an embedded camera). The camera is some infamous brand chineese name Eilondo
and on the camera all that is written is SUPER USB2.0 1.3 mega pixel

I bought exactly this camera because I was said by the shop reseller that the camera works without any driver installations on Windows XP and Windows Vista

On my Debian Squeeze GNU / Linux it was detected in dmesg without any troubles, here is how the camera got detected in my kernel log :

debian:~# dmesg |tail -n 10
[25385.734932] usb 2-1: USB disconnect, address 4
[25388.905049] usb 2-1: new high speed USB device using ehci_hcd and address 5
[25389.050753] usb 2-1: New USB device found, idVendor=1e4e, idProduct=0102
[25389.050757] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[25389.050760] usb 2-1: Product: USB2.0 Camera
[25389.050762] usb 2-1: Manufacturer: Etron Technology, Inc.
[25389.050936] usb 2-1: configuration #1 chosen from 1 choice
[25389.056056] uvcvideo: Found UVC 1.00 device USB2.0 Camera (1e4e:0102)
[25389.058242] uvcvideo: UVC non compliance - GET_DEF(PROBE) not supported. Enabling workaround.
[25389.059113] input: USB2.0 Camera as /devices/pci0000:00/0000:00:1d.7/usb2/2-1/2-1:1.0/input/input26

I was troubled by the message uvcvideo: UVC non compliance – GET_DEF(PROBE) not supported. Enabling workaround. , and hence looked for an application to test if the camera can recored properly.

While checking in packages available in Software Center , I found a plenty of programs under the search keyword Camera
I however decided to test it using just one application Cheese A tool to take pictures and videos from your webcam which I've seen to be quite popular among Liunx users.
Cheese is part of GNOME Desktop, so that was another reason I decided to give it a try. I was pleasently surprised about how good these tiny but functional proggie is.

To run Cheese in GNOME I nagivated to the menus:

Applications -> Sound & Video -> Cheese Webcam Booth

Just in case if Cheese is not installed, installing it with apt:

debian:~# apt-get install cheese

Cheese has capabilities to take pictures, a consequential photos take up, as well as create Video movies.

Cheese take camera testing tool Debian GNU Linux

The program has support to apply 12 Effects / (Masks) to add some fun to the pictures or video snapshots.

Test Video Camera on Debian Linux Cheese Effects

Probably the best thing about Cheese is its simplistic interface, which for me personally is a main criterion to evaluate a program quality ;).

How to set up Path to .exe GNUWin32 binary files in Windows XP / Vista / 2003 / 2008 (Setting PATH to executables on Windows)

Tuesday, August 23rd, 2011

I’ve been working on a servers running Windows 2003 and Windows 2008 these days.
As I wanted to be more flexible on what I can do from the command line I decided to install GNUwin (provides port of GNU tools), most of which are common part of any Linux distribution).
Having most of the command line flexibility on a Windows server is a great thing, so I would strongly recommend GNUWin to any Windows server adminsitrator out there.

Actually it’s a wonderful thing that most of the popular Linux tools can easily be installed and used on Windows for more check GnuWin32 on sourceforge

One of the reasons I installed Gnuwin was my intention to use the good old Linux tail command to keep an eye interactive on the IIS server access log files, which by the way for IIS webserver are stored by default in C:WindowsSystem32LogFilesW3SVC1*.log

I’ve managed to install the GNUWin following the install instructions, not with too much difficulties. The install takes a bit of time, cause many packs containing different parts of the GNUWin has to be fetched.

To install I downloaded the GNUWin installer available from GNUWin32’s website and instructed to extracted the files into C:Program FilesGnuwin
Then I followed the install instructions suggestions, e.g.:

C:> cd c:Program FilesGnuWin
C:Program FilesGnuWin> download.bat
...
C:Program FilesGnuWin> install c:gnuwin32
...

After the installation was succesfully completed on the two Windows machines, both of which by the way are running 64 bit Windows, it was necessery to add the newly installed GNU .exe files to my regular cmd.exe PATH variable in order to be able to access the sed, tail and the rest of the gnuwin32 command line tools.

In order to add C:GnuWin32bin directory to the windows defined Command line Path , I had to do the following:

a. Select (Properties) for My Computer

Start (button) -> My Computer (choose properties)

b. Select the My Computer Advanced (tab)

Then, from the My Computer pane press on Advanced tab

c. Next press on Environment Variables

Windows environment variables screenshot

You see in above’s screenshot the Environment Variables config dialog, to add the new path location in System Variables sectiom, between the list I had to add the c:GNUwin32bin path locatiion. To find I pressed on Edit button scrolled down to find the Variable and hence added at the end of the long list defined paths.
After adding in GNUwin, the Windows path looks like this:

C:Program Files (x86)EWANAPI;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;C:Program Files (x86)IntelNGSMSMPFiles;C:Program Files (x86)Microsoft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100DTSBinn;C:Program Files (x86)Microsoft SQL Server100ToolsBinnVSShellCommon7IDE;C:Program Files (x86)Microsoft Visual Studio 9.0Common7IDEPrivateAssemblies;C:Program Files (x86)Microsoft SQL Server100DTSBinn;C:WINDOWSsystem32WindowsPowerShellv1.0;C:gnuwin32bin

Further on, I launched the tail command to intercatively take an eye on who is accessing the IIS webserver.
Sadly this worked not, trying to use tail with the IIS ex10116.log log;

C:WindowsSystem32LogfilesW3SVC1> tail -f ex10116.log

Spit an error tail: ex10116.log: Bad file descriptor

Since I couldn’t use tail -f I looked for alternative and a quick search led me to Tail 4 Win32 . Lest the name suggests it is supposed to work on 32 bit arch Windows the version on tailforwin32’s website is working perfectly fine on 64 bit Windows as well.
What it does is to simulate a normal tail -f command inside a very simplistic window interface. You see it in action with opened IIS log on below’s screenshot:

GUI Tail for Windows screenshot

Finally my goal is achieved and I can take an eye interactively on IIS logs. End of the article, hope it wasn’t too boring 😉