Linux Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Archive for the ‘Linux’ Category

Improve haproxy logging with custom log-format for better readiability

Friday, April 12th, 2024

Haproxy logging is a very big topic, worthy of many articles, but unfortunately not enough is written on the topic, perhaps for the reason haproxy is free software and most people who use it doesn't follow the philosophy of free software sharing but want to keep, the acquired knowledge on the topic for their own and if possible in the capitalist world most of us live to use it for a Load Balancer haproxy consultancy, consultancy fee or in their daily job as system administrators (web and middleware) or cloud specialist etc. 🙂

Having a good haproxy logging is very important as you need to debug issues with backend machines or some other devices throwing traffic to the HA Proxy.
Thus it is important to build a haproxy logging in a way that it provides most important information and the information is as simple as possible, so everyone can understand what is in without much effort and same time it contains enough debug information, to help you if you want to use the output logs with Graylog filters or process data with some monitoring advanced tool as Prometheus etc.

In our effort to optimize the way haproxy logs via a configured handler that sends the haproxy output to logging handler configured to log through rsyslog, we have done some experiments with logging arguments and came up with few variants, that we liked. In that article the idea is I share this set of logging parameters with hope to help some other guy that starts with haproxy to build a good logging readable and easy to process with scripts log output from haproxy.

The criterias for a decent haproxy logging used are:

1. Log should be simple but not dumb
2. Should be concrete (and not too much complicated)
3. Should be easy to read for the novice and advanced sysadmin

Before starting, have to say that building the logging format seems tedious task but to make it fit your preference could take a lot of time, especially as logging parameters naming is hard to remember, thus the haproxy logging documentation log-format description table comes really handy:

Haproxy log-format paremeters ASCII table

Please refer to the table for log-format defined variables :

+---+------+-----------------------------------------------+-------------+
| R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
+---+------+-----------------------------------------------+-------------+
|   | %o   | special variable, apply flags on all next var |             |
+---+------+-----------------------------------------------+-------------+
|   | %B   | bytes_read           (from server to client)  | numeric     |
| H | %CC  | captured_request_cookie                       | string      |
| H | %CS  | captured_response_cookie                      | string      |
|   | %H   | hostname                                      | string      |
| H | %HM  | HTTP method (ex: POST)                        | string      |
| H | %HP  | HTTP request URI without query string (path)  | string      |
| H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string      |
| H | %HU  | HTTP request URI (ex: /foo?bar=baz)           | string      |
| H | %HV  | HTTP version (ex: HTTP/1.0)                   | string      |
|   | %ID  | unique-id                                     | string      |
|   | %ST  | status_code                                   | numeric     |
|   | %T   | gmt_date_time                                 | date        |
|   | %Ta  | Active time of the request (from TR to end)   | numeric     |
|   | %Tc  | Tc                                            | numeric     |
|   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                 | numeric     |
|   | %Tl  | local_date_time                               | date        |
|   | %Th  | connection handshake time (SSL, PROXY proto)  | numeric     |
| H | %Ti  | idle time before the HTTP request             | numeric     |
| H | %Tq  | Th + Ti + TR                                  | numeric     |
| H | %TR  | time to receive the full request from 1st byte| numeric     |
| H | %Tr  | Tr (response time)                            | numeric     |
|   | %Ts  | timestamp                                     | numeric     |
|   | %Tt  | Tt                                            | numeric     |
|   | %Tw  | Tw                                            | numeric     |
|   | %U   | bytes_uploaded       (from client to server)  | numeric     |
|   | %ac  | actconn                                       | numeric     |
|   | %b   | backend_name                                  | string      |
|   | %bc  | beconn      (backend concurrent connections)  | numeric     |
|   | %bi  | backend_source_ip       (connecting address)  | IP          |
|   | %bp  | backend_source_port     (connecting address)  | numeric     |
|   | %bq  | backend_queue                                 | numeric     |
|   | %ci  | client_ip                 (accepted address)  | IP          |
|   | %cp  | client_port               (accepted address)  | numeric     |
|   | %f   | frontend_name                                 | string      |
|   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
|   | %fi  | frontend_ip              (accepting address)  | IP          |
|   | %fp  | frontend_port            (accepting address)  | numeric     |
|   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
|   | %lc  | frontend_log_counter                          | numeric     |
|   | %hr  | captured_request_headers default style        | string      |
|   | %hrl | captured_request_headers CLF style            | string list |
|   | %hs  | captured_response_headers default style       | string      |
|   | %hsl | captured_response_headers CLF style           | string list |
|   | %ms  | accept date milliseconds (left-padded with 0) | numeric     |
|   | %pid | PID                                           | numeric     |
| H | %r   | http_request                                  | string      |
|   | %rc  | retries                                       | numeric     |
|   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
|   | %s   | server_name                                   | string      |
|   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
|   | %si  | server_IP                   (target address)  | IP          |
|   | %sp  | server_port                 (target address)  | numeric     |
|   | %sq  | srv_queue                                     | numeric     |
| S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
| S | %sslv| ssl_version (ex: TLSv1)                       | string      |
|   | %t   | date_time      (with millisecond resolution)  | date        |
| H | %tr  | date_time of HTTP request                     | date        |
| H | %trg | gmt_date_time of start of HTTP request        | date        |
| H | %trl | local_date_time of start of HTTP request      | date        |
|   | %ts  | termination_state                             | string      |
| H | %tsc | termination_state with cookie status          | string      |
+---+------+-----------------------------------------------+-------------+
R = Restrictions : H = mode http only ; S = SSL only

Our custom log-format built in order to fulfill our needs is as this:

log-format %ci:%cp\ %H\ [%t]\ [%f\ %fi:%fp]\ [%b/%s\ %si:%sp]\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%sq/%bq

Once you place the log-format as a default for all haproxy frontend / backends or for a custom defined ones, the output you will get when tailing the log is:

# tail -f /var/log/haproxy.log

Apr 5 21:47:19 10.42.73.83:23262 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:23.879] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME/bk_appserv3 10.75.226.88:61310] 1/0/55250 55 sD 4/2/1/0/0/0
Apr 5 21:48:14 10.42.73.83:57506 haproxy-fqdn-hostname.com [05/Apr/2024:21:47:18.925] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv1 10.35.242.134:61310] 1/0/55236 55 sD 4/2/1/0/0/0
Apr 5 21:49:09 10.42.73.83:46520 haproxy-fqdn-hostname.com [05/Apr/2024:21:48:13.956] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv2 10.75.226.89:61310] 1/0/55209 55 sD 4/2/1/0/0/0

If you don't care about extra space and logs being filled with more naming, another variant of above log-format, that makes it even more readable even for most novice sys admin or programmer would look like this:

log-format [%t]\ %H\ [IN_IP]\ %ci:%cp\ [FT_NAME]\ %f:%fp\ [FT_IP]\ %fi:%fp\ [BK_NAME]\ [%b/%s:%sp]\ [BK_IP]\ %si:%sp\ [TIME_WAIT]\ {%Tw/%Tc/%Tt}\ [CONN_STATE]\ {%B\ %ts}\ [STATUS]\ [%ac/%fc/%bc/%sc/%sq/%bq]

Once you apply the config test the haproxy.cfg to make sure no syntax errors during copy / paste from this page

haproxy-serv:~# haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

Next restart graceously haproxy

haproxy-serv:~# /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

Once you reload haproxy graceously without loosing the established connections in stead of restarting it completely via systemd sysctl restart haproxy:

2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.200.198.195:50714 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.012] [FrotnendProd 10.55.0.20:27800] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/1/0/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.189:54290 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.056] [FrotnendProd 10.55.0.20:27900] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/4/3/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.190:26778 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.134] [FrotnendProd 10.55.0.20:27900] [BackendProd/tsefas02s 10.35.242.134:27900] 1/-1/0 0 CC 4/4/3/0/0/0

Note that in that log localhost haproxy[pid] is written by rsyslog, you can filter it out by modifying rsyslogd configurations

The only problem with this log-format is not everyone wants to have to much repeating information pointer on which field is what, but I personally liked this one as well because using it even though occuping much more space, makes the log much easier to process with perl or python scripting for data visualize and very for programs that does data or even "big data" analysis.

Tags: about, address, admin, AES, ALL, and, another, are, article, better, connections, custom, data, fi, handy, haproxy, HTTP, Improve, logging, string
Posted in Haproxy, Linux, System Administration, Various | No Comments »

How to RIP audio CD and convert to MP3 format on Linux

Thursday, April 11th, 2024

I've been given a very tedious task to Copy music from Audio CD Drive to MP3 / MP4 file format and then copy the content to external Flash drive.
Doing so is pretty trivial, you just need to have a CD / DVD rom on your computer something that becomes rare nowadays and then you need to have installed a bunch of software, if you don't already have it as i've pointed in my previous article Howto craete Music Audio CD from MP3 files, create playable WAV format audio CD Albums from MP3s.

Creating a Audio CD from an MP3 collection is exactly the opposite to what is aim't now (to copy the content of a CD to a computer and prepare it for A Car MP3 player).

1. RIPing audio CDs to WAV and Conver to MP3 from terminal

On Linux there is many ways to do it and many tools that can do it for your both graphical and command line.
But as I prefer command line to do stuff, in this article I'll mention the quickest and most elementary one which is done in 2 steps.

1. Use a tool to dump the CD Audio music to Tracks in WAV format
2. Convert the WAV to MP3 format

We'll need cdparanoia tool installed as well as ffmpeg.

If you don't have them installed do:

# apt-get install –yes cdparanoia dvd+rw-tools cdw cdrdao audiotools cdlabelgen dvd+rw-tools wodim ffmpeg lame normalize-audio libavcodec58

Next create the directory where you want to dump the .wav files.

# mkdir /home/hipo/audiorip/cd1
# cd /home/hipo/audiorip/cd1

Next assumng the Audio CD is plugged in the CD reader, dump its full content into track*.WAV files with cmd:

# paranoia -B

This will produce you the dumped songs into .wav files.

hipo@noah:~/audiorip/cd1$ ls -al *.wav
-rw-r–r– 1 root root 10278284 мар 25 22:49 track01.cdda.wav
-rw-r–r– 1 root root 21666668 мар 25 22:50 track02.cdda.wav
-rw-r–r– 1 root root 88334108 мар 25 22:53 track03.cdda.wav
-rw-r–r– 1 root root 53453948 мар 25 22:55 track04.cdda.wav
-rw-r–r– 1 root root 100846748 мар 25 22:58 track05.cdda.wav
-rw-r–r– 1 root root 41058908 мар 25 22:59 track06.cdda.wav
-rw-r–r– 1 root root 105952940 мар 25 23:02 track07.cdda.wav
-rw-r–r– 1 root root 50074124 мар 25 23:03 track08.cdda.wav
-rw-r–r– 1 root root 92555948 мар 25 23:06 track09.cdda.wav
-rw-r–r– 1 root root 61939964 мар 25 23:07 track10.cdda.wav
-rw-r–r– 1 root root 8521340 мар 25 23:07 track11.cdda.wav

Then you can use a simple for loop with ffmpeg command to conver the .wav files to .mp3s.

hipo@noah:~/audiorip/cd1$ for i in $( ls -1 *); do ffmpeg -i $i $i.wav.mp3; done

ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis libavutil      52. 18.100 / 52. 18.100
libavcodec     54. 92.100 / 54. 92.100
libavformat    54. 63.104 / 54. 63.104
libavdevice    54. 3.103 / 54. 3.103
libavfilter     3. 42.103 / 3. 42.103
libswscale      2. 2.100 / 2. 2.100
libswresample   0. 17.102 / 0. 17.102
libpostproc    52. 2.100 / 52. 2.100
[wav @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'track01.cdda.wav':
Duration: 00:00:23.19, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp3, to 'track01.cdda.wav.wav.mp3':
Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
Stream #0:0 -> #0:0 (pcm_s16le -> libmp3lame)
Press [q] to stop, [?] for help
size=     363kB time=00:00:23.19 bitrate= 128.2kbits/s
video:0kB audio:363kB subtitle:0 global headers:0kB muxing overhead 0.058402%
ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis libavutil      52. 18.100 / 52. 18.100
libavcodec     54. 92.100 / 54. 92.100
libavformat    54. 63.104 / 54. 63.104
libavdevice    54. 3.103 / 54. 3.103
libavfilter     3. 42.103 / 3. 42.103
libswscale      2. 2.100 / 2. 2.100
libswresample   0. 17.102 / 0. 17.102
libpostproc    52. 2.100 / 52. 2.100
[mp3 @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Input #0, mp3, from 'track01.cdda.wav.mp3':
Metadata:
    encoder         : Lavf54.63.104
Duration: 00:00:23.22, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
File 'track01.cdda.wav.mp3.wav.mp3' already exists. Overwrite ? [y/N] y
Output #0, mp3, to 'track01.cdda.wav.mp3.wav.mp3':
Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
Stream #0:0 -> #0:0 (mp3 -> libmp3lame)
Press [q] to stop, [?] for help
Trying to remove 1152 samples, but the queue is emptys
size=     363kB time=00:00:23.24 bitrate= 128.1kbits/s
video:0kB audio:363kB subtitle:0 global headers:0kB muxing overhead 0.058336%
ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis libavutil      52. 18.100 / 52. 18.100
libavcodec     54. 92.100 / 54. 92.100
libavformat    54. 63.104 / 54. 63.104
libavdevice    54. 3.103 / 54. 3.103
libavfilter     3. 42.103 / 3. 42.103
libswscale      2. 2.100 / 2. 2.100
libswresample   0. 17.102 / 0. 17.102
libpostproc    52. 2.100 / 52. 2.100
[mp3 @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Input #0, mp3, from 'track01.cdda.wav.mp3.wav.mp3':
Metadata:
    encoder         : Lavf54.63.104
Duration: 00:00:23.25, start: 0.000000, bitrate: 128 kb/s
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 128 kb/s
Output #0, mp3, to 'track01.cdda.wav.mp3.wav.mp3.wav.mp3':
Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
Stream #0:0 -> #0:0 (mp3 -> libmp3lame)
Press [q] to stop, [?] for help
Trying to remove 1152 samples, but the queue is emptys
size=     364kB time=00:00:23.27 bitrate= 128.1kbits/s
video:0kB audio:364kB subtitle:0 global headers:0kB muxing overhead 0.058271%
ffmpeg version 1.2.12 Copyright (c) 2000-2015 the FFmpeg developers
built on Feb 12 2015 18:03:16 with gcc 4.7 (Debian 4.7.2-5)
configuration: –prefix=/usr –extra-cflags='-g -O2 -fstack-protector –param=ssp-buffer-size=4 -Wformat -Werror=format-security ' –extra-ldflags='-Wl,-z,relro' –cc='ccache cc' –enable-shared –enable-libmp3lame –enable-gpl –enable-nonfree –enable-libvorbis –enable-pthreads –enable-libfaac –enable-libxvid –enable-postproc –enable-x11grab –enable-libgsm –enable-libtheora –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libx264 –enable-libspeex –enable-nonfree –disable-stripping –enable-libvpx –enable-libschroedinger –disable-encoder=libschroedinger –enable-version3 –enable-libopenjpeg –enable-librtmp –enable-avfilter –enable-libfreetype –enable-libvo-aacenc –disable-decoder=amrnb –enable-libvo-amrwbenc –enable-libaacplus –libdir=/usr/lib/x86_64-linux-gnu –disable-vda –enable-libbluray –enable-libcdio –enable-gnutls –enable-frei0r –enable-openssl –enable-libass –enable-libopus –enable-fontconfig –enable-libpulse –disable-mips32r2 –disable-mipsdspr1 –dis libavutil      52. 18.100 / 52. 18.100
libavcodec     54. 92.100 / 54. 92.100
libavformat    54. 63.104 / 54. 63.104
libavdevice    54. 3.103 / 54. 3.103
libavfilter     3. 42.103 / 3. 42.103
libswscale      2. 2.100 / 2. 2.100
libswresample   0. 17.102 / 0. 17.102
libpostproc    52. 2.100 / 52. 2.100
[wav @ 0x66c900] max_analyze_duration 5000000 reached at 5015510 microseconds
Guessed Channel Layout for Input Stream #0.0 : stereo
Input #0, wav, from 'track02.cdda.wav':
Duration: 00:02:21.28, bitrate: 1411 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Output #0, mp3, to 'track02.cdda.wav.wav.mp3':
Metadata:
    TSSE            : Lavf54.63.104
    Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p
Stream mapping:
Stream #0:0 -> #0:0 (pcm_s16le -> libmp3lame)
Press [q] to stop, [?] for help
…

Finally remove the old unneded .wav files and enjoy the mp3s with vlc / mplayer / mpg123 or whatever player you like.

hipo@noah:~/audiorip/cd1$ rm -f *.wav

Now mount the flash drive and copy th files into it.

# mkdir /media/usb-drive
# mount /dev/sdc1 /media/usb-drive/
# mkdir -p /media/usb-drive/cd1
# fdisk -l |grep -i sdc1
/dev/sdc1 on /media/usb-drive type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro

# cp -rpf ~/audiorip/cd1*.mp3 /mnt/usb-drive/cd1
# umount /dev/sdc1

2. RIPping audio CD on Linux with rip-audio-cds-linux.sh script

#!/bin/bash
# A simple shell script to rip audio cd and create mp3 using lame
# and cdparanoia utilities.
# —————————————————————————-
# Written by Vivek Gite <http://www.cyberciti.biz/>
# (c) 2006 nixCraft under GNU GPL v2.0+
# —————————————————————————-
read -p "Starting in 5 seconds ( to abort press CTRL + C ) " -t 5
cdparanoia -B
for i in *.wav
do
lame –vbr-new -b 360 "$i" "${i%%.cdda.wav}.mp3"
rm -f "$i"
done

If you need to automate the task of dumping the audio CDs to WAV and convert them to MP3s you can do it via a small shell script like the one provided by cyberciti.biz that uses paranoia and lame commands in a shell script loop. Script rip-audio-cds-linux.sh is here

3. Dump Audio CD to MP3 with Graphical program ( ripperx )

By default most modern Linux distributions including the Debian GNU / Linux based ones has the ripperx in the default repositories, as well as the tool is downloadable and compilable from source from sourceforge.net

ripping-audio-cds-linux-graphical-program-ripperx-tool

# apt-cache show ripperx|grep -i descript -A3 -B3
Architecture: amd64
Depends: cdparanoia, vorbis-tools (>= 1.0beta3), libatk1.0-0 (>= 1.12.4), libc6 (>= 2.14), libcairo2 (>= 1.2.4), libfontconfig1 (>= 2.12.6), libfreetype6 (>= 2.2.1), libgcc1 (>= 1:3.0), libgdk-pixbuf2.0-0 (>= 2.22.0), libglib2.0-0 (>= 2.16.0), libgtk2.0-0 (>= 2.8.0), libpango-1.0-0 (>= 1.14.0), libpangocairo-1.0-0 (>= 1.14.0), libpangoft2-1.0-0 (>= 1.14.0), libstdc++6 (>= 5.2), libtag1v5 (>= 1.9.1-2.2~)
Suggests: sox, cdtool, mpg321, flac, toolame
Description-en: GTK-based audio CD ripper/encoder
ripperX is a graphical interface for ripping CD audio tracks (using
cdparanoia) and then encoding them into the Ogg, FLAC, or MP2/3
formats using the vorbis tools, FLAC, toolame or other available
MP3 encoders.
.
It includes support for CDDB lookups and ID3v2 tags.
Description-md5: cdeabf4ef72c33d57aecc4b4e2fd5952
Homepage: http://sourceforge.net/projects/ripperx/
Tag: hardware::storage, hardware::storage:cd, interface::graphical,
interface::x11, role::program, scope::application, uitoolkit::gtk,

# apt install –yes ripperx

That's all folks.
Enjoy !

Tags: Audio, audio cd, ccache, cdda, cmd, Convert, ffmpeg, gcc, graphical interface, help, hipo, linux?, need, Press, RIP, root root, shell script
Posted in Educational, File Convert Tools, Linux, Linux and FreeBSD Desktop, Linux Audio & Video | No Comments »

DNS Monitoring: Check and Alert if DNS nameserver resolver of Linux machine is not properly resolving shell script. Monitor if /etc/resolv.conf DNS runs Okay

Thursday, March 14th, 2024

If you happen to have issues occasionally with DNS resolvers and you want to keep up an eye on it and alert if DNS is not properly resolving Domains, because sometimes you seem to have issues due to network disconnects, disturbances (modifications), whatever and you want to have another mean to see whether a DNS was reachable or unreachable for a time, here is a little bash shell script that does the "trick".

Script work mechacnism is pretty straight forward as you can see we check what are the configured nameservers if they properly resolve and if they're properly resolving we write to log everything is okay, otherwise we write to the log DNS is not properly resolvable and send an ALERT email to preconfigured Email address.

Below is the check_dns_resolver.sh script:

#!/bin/bash
# Simple script to Monitor DNS set resolvers hosts for availability and trigger alarm via preset email if any of the nameservers on the host cannot resolve
# Use a configured RESOLVE_HOST to try to resolve it via available configured nameservers in /etc/resolv.conf
# if machines are not reachable send notification email to a preconfigured email
# script returns OK 1 if working correctly or 0 if there is issue with resolving $RESOLVE_HOST on $SELF_HOSTNAME and mail on $ALERT_EMAIL
# output of script is to be kept inside DNS_status.log

ALERT_EMAIL='your.email.address@email-fqdn.com';
log=/var/log/dns_status.log;
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2));

SELF_HOSTNAME=$(hostname –fqdn);
RESOLVE_HOST=$(hostname –fqdn);

for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $RESOLVE_HOST $i);

if [[ “$?” == ‘0’ ]]; then echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i on host $SELF_HOST OK 1" | tee -a $log;
else
echo "$(date "+%y.%m.%d %T")$RESOLVE_HOST $i on host $SELF_HOST NOT_OK 0" | tee -a $log;

echo "$(date "+%y.%m.%d %T") $RESOLVE_HOST $i DNS on host $SELF_HOST resolve ERROR" | mail -s "$RESOLVE_HOST /etc/resolv.conf $i DNS on host $SELF_HOST resolve ERROR";

fi

done

Download check_dns_resolver.sh here set the script to run via a cron job every lets say 5 minutes, for example you can set a cronjob like this:

# crontab -u root -e
*/5 * * * * check_dns_resolver.sh 2>&1 >/dev/null

Then Voila, check the log /var/log/dns_status.log if you happen to run inside a service downtime and check its output with the rest of infrastructure componets, network switch equipment, other connected services etc, that should keep you in-line to proof during eventual RCA (Root Cause Analysis) if complete high availability system gets down to proof your managed Linux servers was not the reason for the occuring service unavailability.

A simplified variant of the check_dns_resolver.sh can be easily integrated to do Monitoring with Zabbix userparameter script and DNS Check Template containing few Triggers, Items and Action if I have time some time in the future perhaps, I'll blog a short article on how to configure such DNS zabbix monitoring, the script zabbix variant of the DNS monitor script is like this:

[root@linux-server bin]# cat check_dns_resolver.sh
#!/bin/bash
TIMEOUT=3; DNS=($(grep -R nameserver /etc/resolv.conf | cut -d ' ' -f2)); for i in ${DNS[@]}; do dns_status=$(timeout $TIMEOUT nslookup $(hostname –fqdn) $i); if [[ “$?” == ‘0’ ]]; then echo "$i OK 1"; else echo "$i NOT OK 0"; fi; done
[root@linux-server bin]#

Hope this article, will help someone to improve his Unix server Infrastucture monitoring.

Enjoy and Cheers !

Tags: /etc/resolv.conf, address, alert, bash, bin, blog, catching dns issues, check dns, crontab, DNS, download, echo, email, Enjoy, linux?, monitor dns, monitoring dns, net, properly, resolv.conf, resolver, script, Timeout, unix, var, www, Zabbix Userparameter
Posted in Bash Scripting, DNS, Linux, Monitoring, System Administration | No Comments »

How to do a port redirect to localhost service with socat or ncat commands to open temporary access to service not seen on the network

Friday, February 23rd, 2024

You know sometimes it is necessery to easily and temporary redirect network TCP ports to be able to be accessible from Internal DMZ-ed Network via some Local Network IP connection or if the computer system is Internet based and has an external "'real" Internet Class A / B address to be reachable directly from the internet via lets say a modern Internet browser such as Mozilla Firefox / Google Chrome Browser etc.

Such things are easy to be done with iptables if you need to do the IP redirect permanent with Firewall rule changes on Linux router with iptables.
One way to create a TCP port redirect using firewall would include few iptable rules like for example:

1. Redirect port traffic from external TCP port source to internal one

# iptables -t nat -I PREROUTING -p tcp –dport 10000 -j REDIRECT –to-ports 80
# iptables -t nat -I OUTPUT -p tcp -o lo –dport 10000 -j REDIRECT –to-ports 80
# iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp –dport 80 -j DNAT –to-destination 192.168.0.50:10000
# iptables -t nat -I OUTPUT –source 0/0 –destination 0/0 -p tcp –dport 80 -j REDIRECT –to-ports 10000

Then you will have 192.168.00.50:10000 listener (assuming that the IP is already configured on some of the host network interface, plugged in to the network).

But as messing up with the firewall is not the best thing to do especially, if you need to just temporary redirect external listener port to a service configured on the server to only run on TCP port on loopback address 127.0.0.1, you can do it instead with another script or command for simplicy.

One simple way to do a port redirect on the fly on GNU / Linux or FreeBSD / OpenBSD is with socat command.

Lets say you have a running statistics of a web server Apache / Nginx / Haproxy frontend / backend statistics or whatever kind of web TCP service on port 80 on your server and this interface is on purpose configured to be reachable only on localhost interface port 80, so you can either access it by creating an ssh tunnel towards the service on 127.0.0.1 or by accessing it by redirecting the traffic towards another external TCP port, lets say 10000.

Here is how you can achieve

2. Redirect Local network accessible IP on all configured Server network interfaces port 10000 to 127.0.0.1 TCP 80 with socat

# socat tcp-l:10000,fork,reuseaddr tcp:127.0.0.1:80

If you need to access later the redirected port in a Browser, pick up the machine first configured IP and open it in a browser (assuming there is no firewall filter prohibiting access to redirected port).

root@pcfreak:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 109.104.212.130 netmask 255.255.255.0 broadcast 109.104.212.255
ether 91:f8:51:03:75:e5 txqueuelen 1000 (Ethernet)
RX packets 652945510 bytes 598369753019 (557.2 GiB)
RX errors 0 dropped 10541 overruns 0 frame 0
TX packets 619726615 bytes 630209829226 (586.9 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Then in a browser open http://102.104.212.130 or https://102.104.212.130 (depending on if remote service has SSL encryption enabled or not) and you're done, the configured listener Server service should pop-up on the screen.

3. Redirect IP Traffic from External IP to Localhost loopback interface with netcat ( ncat ) swiss army knife hackers and sysadmins tool

If you need to redirect lets say TCP / IP port 8000 to Port a server local binded service on TCP 80 with ncat, instead of socat (if lets say socat is not pre-installed on the machine), you can do it by simply running those two commands:

[root@server ~]# mkfifo svr1_to_svr2
[root@server ~]# ncat -vk -l 8000 < svr1_to_svr2 | ncat 127.0.0.1 80 > svr1_to_svr2
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Listening on 0.0.0.0:10000
Ncat: Connection from 10.10.258.39.
Ncat: Connection from 10.10.258.39:51813.
Ncat: Connection from 10.10.258.39.
Ncat: Connection from 10.10.258.39:23179.

I you don't care to log what is going on the background of connection and you simply want to background the process with a one liner command you can achive that with:

[root@server /tmp]# cd tmp; mkfifo svr1_to_svr2; (ncat -vk -l 8000 < svr1_to_svr2 | ncat 127.0.0.1 80 > svr1_to_svr2 &)

Then you can open the Internal Machine Port 80 TCP service on 8000 in a browser as usual.

For those who want a bit of more sophisticated proxy like script I would suggest you take a look at using netcat and a few lines of shell script loop, that can simulate a raw and very primitive proxy with netcat this is exampled in my previous article Create simple proxy server with netcat ( nc ) based utility.

Hope this article is helpful to anyone, there is plenty of other ways to do a port redirect with lets say perl, python and perhaps other micro tools. If you know of one liners or small scripts, that do it please share in comments, so we can learn from each other !

Enjoy ! 🙂

Tags: configured, How to, ifconfig eth0, Internal Machine Port, Mozilla Firefox Google Chrome Browser, Ncat Connection, need, network interface, port, proxy server, say, shell script, swiss army knife, tcp
Posted in Educational, Linux, Linux and FreeBSD Desktop, Networking | No Comments »

Create Haproxy Loadbalancer Access Control Lists and forward incoming frontend traffics based on simple logic

Friday, February 16th, 2024

Haproxy Load Balancers could do pretty much to load balance traffic between application servers. The most straight forward way to use is to balance traffic for incoming Frontends towards a Backend configuration with predefined Application machines and ports to send the traffic, where one can be the leading one and others be set as backup or we can alternatively send the traffic towards a number of machines incoming to a Frontend port bind IP listener and number of backend machine.

Besides this the more interesting capabilities of Haproxy comes with using Access Control Lists (ACLs) to forward Incoming Frontend (FT) traffic towards specific backends and ports based on logic, power ACLs gives to Haproxy to do a sophisticated load balancing are enormous.
In this post I'll give you a very simple example on how you can save some time, if you have already a present Frontend listening to a Range of TCP Ports and it happens you want to redirect some of the traffic towards a spefic predefined Backend.

This is not the best way to it as Access Control Lists will put some extra efforts on the server CPU, but as today machines are quite powerful, it doesn't really matter. By using a simple ACLs as given in below example, one can save much of a time of writting multiple frontends for a complete sequential port range, if lets say only two of the ports in the port range and distinguish and redirect traffic incoming to Haproxy frontend listener in the port range of 61000-61230 towards a certain Ports that are supposed to go to a Common Backends to a separate ones, lets say ports 61115 and 61215.

Here is a short description on the overall screnarios. We have an haproxy with 3 VIP (Virtual Private IPs) with a Single Frontend with 3 binded IPs and 3 Backends, there is a configured ACL rule to redirect traffic for certain ports, the overall Load Balancing config is like so:

Frontend (ft):

ft_PROD:
listen IPs:

192.168.0.77
192.168.0.83
192.168.0.78

On TCP port range: 61000-61299

Backends (bk):

bk_PROD_ROUNDROBIN
bk_APP1
bk_APP2

Config Access Control Liststo seperate incoming haproxy traffic for CUSTOM_APP1 and CUSTOM_APP2

By default send all incoming FT traffic to: bk_PROD_ROUNDROBIN

With exception for frontend configured ports on:
APP1 port 61115
APP2 port 61215

If custom APP1 send to bk:
RULE1
If custom APP2 send to bk:
RULE2

Config on frontends traffic send operation:

bk_PROD_ROUNDROBIN (roundrobin) traffic send to App machines all in parallel
traffic routing mode (roundrobin)
Appl1
Appl2
Appl3
Appl4

bk_APP1 and bk_APP2

traffic routing mode: (balance source)
Appl1 default serving host

If configured check port 61888, 61887 is down, traffic will be resend to configured pre-configured backup hosts:

Appl2
Appl3
Appl4

/etc/haproxy/haproxy.cfg that does what is described with ACL LB capabilities looks like so:

#———————————————————————
# Global settings
#———————————————————————
global
log 127.0.0.1 local2

chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

#———————————————————————
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#———————————————————————
defaults
mode tcp
log global
option tcplog
#option dontlognull
#option http-server-close
#option forwardfor except 127.0.0.0/8
option redispatch
retries 7
#timeout http-request 10s
timeout queue 10m
timeout connect 30s
timeout client 20m
timeout server 10m
#timeout http-keep-alive 10s
timeout check 30s
maxconn 3000

#———————————————————————
# Synchronize server entries in sticky tables
#———————————————————————

peers hapeers
peer haproxy1-fqdn.com 192.168.0.58:8388
peer haproxy2-fqdn.com 192.168.0.79:8388

#———————————————————————
# HAProxy Monitoring Config
#———————————————————————
listen stats 192.168.0.77:8080 #Haproxy Monitoring run on port 8080
mode http
option httplog
option http-server-close
stats enable
stats show-legends
stats refresh 5s
stats uri /stats #URL for HAProxy monitoring
stats realm Haproxy\ Statistics
stats auth hauser:secretpass4321 #User and Password for login to the monitoring dashboard
stats admin if TRUE
#default_backend bk_Prod1 #This is optionally for monitoring backend
#———————————————————————
# HAProxy Monitoring Config
#———————————————————————
#listen stats 192.168.0.83:8080 #Haproxy Monitoring run on port 8080
# mode http
# option httplog
# option http-server-close
# stats enable
# stats show-legends
# stats refresh 5s
# stats uri /stats #URL for HAProxy monitoring
# stats realm Haproxy\ Statistics
# stats auth hauser:secretpass321 #User and Password for login to the monitoring dashboard
# stats admin if TRUE
# #default_backend bk_Prod1 #This is optionally for monitoring backend

#———————————————————————
# HAProxy Monitoring Config
#———————————————————————
# listen stats 192.168.0.78:8080 #Haproxy Monitoring run on port 8080
# mode http
# option httplog
# option http-server-close
# stats enable
# stats show-legends
# stats refresh 5s
# stats uri /stats #URL for HAProxy monitoring
# stats realm Haproxy\ Statistics
# stats auth hauser:secretpass123 #User and Password for login to the monitoring dashboard
# stats admin if TRUE
# #default_backend bk_DKV_PROD_WLPFO #This is optionally for monitoring backend

#———————————————————————
# frontend which proxys to the backends
#———————————————————————
frontend ft_PROD
mode tcp
bind 192.168.0.77:61000-61299
bind 192.168.0.83:51000-51300
bind 192.168.0.78:51000-62300
option tcplog
# (4) Peer Sync: a sticky session is a session maintained by persistence
stick-table type ip size 1m peers hapeers expire 60m
# Commented for change CHG0292890
# stick on src
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
acl RULE1 dst_port 61115
acl RULE2 dst_port 61215
use_backend APP1 if app1
use_backend APP2 if app2
default_backend bk_PROD_ROUNDROBIN

#———————————————————————
# round robin balancing between the various backends
#———————————————————————
backend bk_PROD_ROUNDROBIN
mode tcp
# (0) Load Balancing Method.
balance roundrobin
# (4) Peer Sync: a sticky session is a session maintained by persistence
stick-table type ip size 1m peers hapeers expire 60m
# (5) Server List
# (5.1) Backend
server appl1 10.33.0.50 check port 31232
server appl2 10.33.0.51 check port 31232
server appl2 10.45.0.78 check port 31232
server appl3 10.45.0.79 check port 31232

#———————————————————————
# source balancing for the GUI
#———————————————————————
backend bk_APP2
mode tcp
# (0) Load Balancing Method.
balance source
# (4) Peer Sync: a sticky session is a session maintained by persistence
stick-table type ip size 1m peers hapeers expire 60m
stick on src
# (5) Server List
# (5.1) Backend
server appl1 10.33.0.50 check port 55232
server appl2 10.32.0.51 check port 55232 backup
server appl3 10.45.0.78 check port 55232 backup
server appl4 10.45.0.79 check port 55232 backup

#———————————————————————
# source balancing for the OLW
#———————————————————————
backend bk_APP1
mode tcp
# (0) Load Balancing Method.
balance source
# (4) Peer Sync: a sticky session is a session maintained by persistence
stick-table type ip size 1m peers hapeers expire 60m
stick on src
# (5) Server List
# (5.1) Backend
server appl1 10.33.0.50 check port 53119
server appl2 10.32.0.51 check port 53119 backup
server appl3 10.45.0.78 check port 53119 backup
server appl4 10.45.0.79 check port 53119 backup

You can also check and download the haproxy.cfg here.
Enjjoy !

Tags: backup, based, Create Haproxy Loadbalancer Access Control Lists, frontends, logic, option, peers, ports, range, Server List, session, short description, simple, size, tcp, traffic, User
Posted in Everyday Life, Haproxy, Linux, Web and CMS | No Comments »

How to run SSH server Mac OS X and set it to auto boot on Mac Book system start

Monday, February 5th, 2024

How to run SSH Server on Mac OS X to administrate remotely your MAC OS to access remote MacBook Air or Mac OS

Linux / UNIX users know it is pretty easy to run OpenSSH server on old Linux SystemV releases

it is done with cmd:

# /etc/init.d/sshd start

On newer Linux distros where systemd is the standard it is done wtih:

# systemctl start ssh.service

To enable ssh service on boot on systemd distros

# systemctl enable ssh.service

To enable SSH access on Mac OS X this is done wtih a simple command

To check the status of SSH server being on or OFF, either connect with netcat to TCP port 22, which is usually installed by default on most MAC OS-es or run:

# systemsetup -getremotelogin

To start and enable SSH service on Mac OS X run:

# systemsetup -setremotelogin on

If you later need to turn off the SSH service

# systemsetup -setremotelogin off

Actually systemsetup command can do pretty much on MAC OS X and it is worthy to take a look at it, if you're running a MAC PC or Mac Book laptop.

systemsetup can set the current date, change time server host, set computer name (hostname) and much more.

sh-3.2# systemsetup -help

systemsetup Help Information
————————————-
Usage: systemsetup -getdate
Display current date.

Usage: systemsetup -setdate <mm:dd:yy>
Set current date to <mm:dd:yy>.

Usage: systemsetup -gettime
Display current time.

Usage: systemsetup -settime <hh:mm:ss>
Set current time to <hh:mm:ss>.

Usage: systemsetup -gettimezone
Display current time zone.

Usage: systemsetup -settimezone <timezone>
Set current time zone to <timezone>. Use "-listtimezones" to list time zones.

Usage: systemsetup -listtimezones
List time zones supported by this machine.

Usage: systemsetup -getusingnetworktime
Display whether network time is on or off.

Usage: systemsetup -setusingnetworktime <on off>
Set using network time to either <on> or <off>.

Usage: systemsetup -getnetworktimeserver
Display network time server.

Usage: systemsetup -setnetworktimeserver <timeserver>
Set network time server to <timeserver>.

Usage: systemsetup -getsleep
Display amount of idle time until computer, display and hard disk sleep.

Usage: systemsetup -setsleep <minutes>
Set amount of idle time until computer, display and hard disk sleep to <minutes>.
Specify "Never" or "Off" for never.

Usage: systemsetup -getcomputersleep
Display amount of idle time until computer sleeps.

Usage: systemsetup -setcomputersleep <minutes>
Set amount of idle time until compputer sleeps to <minutes>.
Specify "Never" or "Off" for never.

Usage: systemsetup -getdisplaysleep
Display amount of idle time until display sleeps.

Usage: systemsetup -setdisplaysleep <minutes>
Set amount of idle time until display sleeps to <minutes>.
Specify "Never" or "Off" for never.

Usage: systemsetup -getharddisksleep
Display amount of idle time until hard disk sleeps.

Usage: systemsetup -setharddisksleep <minutes>
Set amount of idle time until hard disk sleeps to <minutes>.
Specify "Never" or "Off" for never.

Usage: systemsetup -getwakeonmodem
Display whether wake on modem is on or off.

Usage: systemsetup -setwakeonmodem <on off>
Set wake on modem to either <on> or <off>.

Usage: systemsetup -getwakeonnetworkaccess
Display whether wake on network access is on or off.

Usage: systemsetup -setwakeonnetworkaccess <on off>
Set wake on network access to either <on> or <off>.

Usage: systemsetup -getrestartpowerfailure
Display whether restart on power failure is on or off.

Usage: systemsetup -setrestartpowerfailure <on off>
Set restart on power failure to either <on> or <off>.

Usage: systemsetup -getrestartfreeze
Display whether restart on freeze is on or off.

Usage: systemsetup -setrestartfreeze <on off>
Set restart on freeze to either <on> or <off>.

Usage: systemsetup -getallowpowerbuttontosleepcomputer
Display whether the power button is able to sleep the computer.

Usage: systemsetup -setallowpowerbuttontosleepcomputer <on off>
Enable or disable whether the power button can sleep the computer.

Usage: systemsetup -getremotelogin
Display whether remote login is on or off.

Usage: systemsetup -setremotelogin <on off>
Set remote login to either <on> or <off>. Use "systemsetup -f -setremotelogin off" to suppress prompting when turning remote login off.

Usage: systemsetup -getremoteappleevents
Display whether remote apple events are on or off.

Usage: systemsetup -setremoteappleevents <on off>
Set remote apple events to either <on> or <off>.

Usage: systemsetup -getcomputername
Display computer name.

Usage: systemsetup -setcomputername <computername>
Set computer name to <computername>.

Usage: systemsetup -getlocalsubnetname
Display local subnet name.

Usage: systemsetup -setlocalsubnetname <name>
Set local subnet name to <name>.

Usage: systemsetup -getstartupdisk
Display current startup disk.

Usage: systemsetup -setstartupdisk <disk>
Set current startup disk to <disk>.

Usage: systemsetup -liststartupdisks
List startup disks on this machine.

Usage: systemsetup -getwaitforstartupafterpowerfailure
Get the number of seconds after which the computer will start up after a power failure.

Usage: systemsetup -setwaitforstartupafterpowerfailure <seconds>
Set the number of seconds after which the computer will start up after a power failure. The <seconds> value must be a multiple of 30 seconds.

Usage: systemsetup -getdisablekeyboardwhenenclosurelockisengaged
Get whether or not the keyboard should be disabled when the X Serve enclosure lock is engaged.

Usage: systemsetup -setdisablekeyboardwhenenclosurelockisengaged <yes no>
Set whether or not the keyboard should be disabled when the X Serve enclosure lock is engaged.

Usage: systemsetup -version
Display version of systemsetup tool.

Usage: systemsetup -help
Display help.

Usage: systemsetup -printCommands
Display commands.

Enabling SSH in Mac OS X computers can be done also from Graphical interface for the lazy ones.

enable-ssh-mac-remote-login-from-mac-OS-X-gui

Tags: Display, either, hard disk, hostname, How to, network, power, Set, sleep, ssh, time server, Usage
Posted in Linux, Mac OS X, Networking, Various | No Comments »

Zabbix script to track arp address cache loss (arp incomplete) from Linux server to gateway IP

Tuesday, January 30th, 2024

Some of the Linux servers recently, I'm responsible had a very annoying issue recently. The problem is ARP address to default configured server gateway is being lost, every now and then and it takes up time, fot the remote CISCO router to realize the problem and resolve it. We have debugged with the Network expert colleague, while he was checking the Cisco router and we were checking the arp table on the Linux server with arp command. And we came to conclusion this behavior is due to some network mess because of too many NAT address configurations on the network or due to a Cisco bug. The colleagues asked Cisco but cisco does not have any solution to the issue and the only close work around for the gateway loosing the mac is to set a network rule on the Cisco router to flush its arp record for the server it was loosing the MAC address for.
This does not really solve completely the problem but at least, once we run into the issue, it gets resolved as quick as 5 minutes time. }

As we run a cluster environment it is useful to Monitor and know immediately once we hit into the MAC gateway disappear issue and if the issue persists, exclude the Linux node from the Cluster so we don't loose connection traffic.
For the purpose of Monitoring MAC state from the Linux haproxy machine towards the Network router GW, I have developed a small userparameter script, that is periodically checking the state of the MAC address of the IP address of remote gateway host and log to a external file for any problems with incomplete MAC address of the Remote configured default router.

In case if you happen to need the same MAC address state monitoring for your servers, I though that might be of a help to anyone out there.
To monitor MAC address incomplete state with Zabbix, do the following:

1. Create userparamater_arp_gw_check.conf Zabbix script

# cat userparameter_arp_gw_check.conf
UserParameter=arp.check,/usr/local/bin/check_gw_arp.sh

2. Create the following shell script /usr/local/bin/check_gw_arp.sh

#!/bin/bash
# simple script to run on cron peridically or via zabbix userparameter
# to track arp loss issues to gateway IP
#gw_ip='192.168.0.55';
gw_ip=$(ip route show|grep -i default|awk '{ print $3 }');
log_f='/var/log/arp_incomplete.log';
grep_word='incomplete';
inactive_status=$(arp -n "$gw_ip" |grep -i $grep_word);
# if GW incomplete record empty all is ok
if [[ $inactive_status == ” ]]; then
echo $gw_ip OK 1;
else
# log inactive MAC to gw_ip
echo "$(date '+%Y-%m-%d %H:%M:%S')" "ARP_ERROR $inactive_status 0" | tee -a $log_f 2>&1 >/dev/null;
# printout to zabbix
echo "1 ARP FAILED: $inactive_status";
fi

You can download the check_gw_arp.sh here.

The script is supposed to automatically grep for the Default Gateway router IP, however before setting it up. Run it and make sure this corresponds correctly to the default Gateway IP MAC you would like to monitor.

3. Create New Zabbix Template for ARP incomplete monitoring

Create Application

*Name
Default Gateway ARP state

4. Create Item and Dependent Item

Create Zabbix Item and Dependent Item like this

5. Create Trigger to trigger WARNING or whatever you like

arp-machine-to-default-gateway-failure-monitoring-trigger-screenshot

arp-machine-to-default-gateway-failure-monitoring-trigger1-screenshot

arp-machine-to-default-gateway-failure-monitoring-trigger2-screenshot

6. Create Zabbix Action to notify via Email etc.

arp-machine-to-default-gateway-failure-monitoring-action1-screenshot

arp-machine-to-default-gateway-failure-monitoring-action2-screenshot

That's all. Once you set up this few little things, you can enjoy having monitoring Alerts for your ARP state incomplete on your Linux / Unix servers.
Enjoy !

Tags: address, arp, bin, checking, cisco, colleagues, echo, Linux Unix, linux?, log, network, run, shell script, state
Posted in Linux, Monitoring, System Administration, Zabbix | No Comments »

KVM Creating LIVE and offline VM snapshot backup of Virtual Machines. Restore KVM VM from backup. Delete old KVM backups

Tuesday, January 16th, 2024

For those who have to manage Kernel-Based Virtual Machines it is a must to create periodic backups of VMs. The backup is usually created as a procedure part of the Update plan (schedule) of the server either after shut down the machine completely or live.

Since KVM is open source the very logical question for starters, whether KVM supports Live backups. The simple answer is Yes it does.

virsh command as most people know is the default command to manage VMs on KVM running Hypervisor servers to manage the guest domains.

KVM is flexible and could restore a VM based on its XML configuration and the VM data (either a static VM single file) or a filesystem laying on LVM filesystem etc.

To create a snapshot out of the KVM HV, list all VMs and create the backup:

# export VM-NAME=fedora;
# export SNAPSHOT-NAME=fedora-backup;
# virsh list –all
…

It is useful to check out the snapshot-create-as sub arguments

# virsh help snapshot-create-as
…

OPTIONS
[–domain] <string> domain name, id or uuid
–name <string> name of snapshot
–description <string> description of snapshot
–print-xml print XML document rather than create
–no-metadata take snapshot but create no metadata
–halt halt domain after snapshot is created
–disk-only capture disk state but not vm state
–reuse-external reuse any existing external files
–quiesce quiesce guest's file systems
–atomic require atomic operation
–live take a live snapshot
–memspec <string> memory attributes: [file=]name[,snapshot=type]
[–diskspec] disk attributes: disk[,snapshot=type][,driver=type][,file=name]

# virsh shutdown $VM_NAME
# virsh snapshot-create-as –domain $VM-NAME –name "$SNAPSHOT-NAME"

1. Creating a KVM VM LIVE (running machine) backup

# virsh snapshot-create-as –domain debian \
–name "debian-snapshot-2024" \
–description "VM Snapshot before upgrading to latest Debian" \
–live

On successful execution of KVM Virtual Machine live backup, should get something like:

Domain snapshot debian-snapshot-2024 created

2. Listing backed-up snapshot content of KVM machine

# virsh snapshot-list –domain debian

a. To get more extended info about a previous snapshot backup

# virsh snapshot-info –domain debian –snapshotname debian-snapshot-2024

b. Listing info for multiple attached storage qcow partition to a VM

# virsh domblklist linux-guest-vm1 –details

Sample Output would be like:

Type Device Target Source
——————————————————————-
file disk vda /kvm/linux-host/linux-guest-vm1_root.qcow2
file disk vdb /kvm/linux-host/linux-guest-vm1_attached_storage.qcow2
file disk vdc /kvm/linux-host/guest01_logging_partition.qcow2
file cdrom sda –
file cdrom sdb

3. Backup KVM only Virtual Machine data files (but not VM state) Live

# virsh snapshot-create-as –name "mint-snapshot-2024" \
–description "Mint Linux snapshot" \
–disk-only \
–live
–domain mint-home-desktop

4. KVM restore snapshot (backup)

To revert backup VM state to older backup snapshot:

# virsh shutdown –domain manjaro
# virsh snapshot-revert –domain manjaro –snapshotname manjaro-linux-back-2024 –running

5. Delete old unnecessery KVM VM backup

# virsh snapshot-delete –domain dragonflybsd –snapshotname dragonfly-freebsd

Tags: backups, cdrom, command, created, Creating, Delete, domain name, filesystem, kvm, Listing, logical question, snapshot, virsh
Posted in Linux, System Administration, Virtual Machines | No Comments »

Use haproxy to dynamically modify haproxy load balancer variables, view stastics, errors and much more via stats UNIX socket with socat via command line

Friday, December 15th, 2023

Haproxy could be configured to use the listen stats interface to provide a tiny web interface with statistics on all configured haproxy frontends / backends state status (UP / DOWN), current connections to proxy, errors and other interesting bandwidth information.

That is mostly useful but not every haproxy has it configured and if you did not configure the HAproxy load balancer machines on your own it might be, the previous person who build the LB infrastructure did not create the haproxy listener.

If that is the case and you still need to get various statistics on how haproxy performs and the status of active connections towards Frotnend i/ Backend interfaces this is still possible via configured stats socket (usually this is in Global or some of the other haproxy.cfg config sections..

It is possible to do many things with haproxy such as disable / enable frotnends / backends / servers

Lets say your Haproxy has a global section that looks like this:

global
stats socket /var/run/haproxy/haproxy.sock mode 0600 level admin #Creates Unix-Like socket to fetch stats
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 99999
nbproc 1
nbthread 2
cpu-map 1 0
cpu-map 2 1

…

1. Listing all available options that can be send via the haproxy.sock UNIX socket interface

root@pcfreak:/home/hipo/info# echo "show help" | socat stdio /var/run/haproxy/haproxy.sock
Unknown command. Please enter one of the following commands only :
help : this message
prompt : toggle interactive mode with prompt
quit : disconnect
show tls-keys [id|*]: show tls keys references or dump tls ticket keys when id specified
set ssl tls-key [id|keyfile] <tlskey>: set the next TLS key for the <id> or <keyfile> listener to <tlskey>
add ssl crt-list <filename> <certfile> [options] : add a line <certfile> to a crt-list <filename>
del ssl crt-list <filename> <certfile[:line]> : delete a line <certfile> in a crt-list <filename>
show ssl crt-list [-n] [] : show the list of crt-lists or the content of a crt-list <filename>
new ssl cert <certfile> : create a new certificate file to be used in a crt-list or a directory
set ssl cert <certfile> <payload> : replace a certificate file
commit ssl cert <certfile> : commit a certificate file
abort ssl cert <certfile> : abort a transaction for a certificate file
del ssl cert <certfile> : delete an unused certificate file
show ssl cert [] : display the SSL certificates used in memory, or the details of a <certfile>
set maxconn global : change the per-process maxconn setting
set rate-limit : change a rate limiting value
set severity-output [none|number|string] : set presence of severity level in feedback information
set timeout : change a timeout setting
show env [var] : dump environment variables known to the process
show cli sockets : dump list of cli sockets
show cli level : display the level of the current CLI session
show fd [num] : dump list of file descriptors in use
show activity : show per-thread activity stats (for support/developers)
operator : lower the level of the current CLI session to operator
user : lower the level of the current CLI session to user
clear counters : clear max statistics counters (add 'all' for all counters)
show info : report information about the running process [desc|json|typed]*
show stat : report counters for each proxy and server [desc|json|typed]*
show schema json : report schema used for stats
show sess [id] : report the list of current sessions or dump this session
shutdown session : kill a specific session
shutdown sessions server : kill sessions on a server
disable agent : disable agent checks (use 'set server' instead)
disable health : disable health checks (use 'set server' instead)
disable server : disable a server for maintenance (use 'set server' instead)
enable agent : enable agent checks (use 'set server' instead)
enable health : enable health checks (use 'set server' instead)
enable server : enable a disabled server (use 'set server' instead)
set maxconn server : change a server's maxconn setting
set server : change a server's state, weight or address
get weight : report a server's current weight
set weight : change a server's weight (deprecated)
show startup-logs : report logs emitted during HAProxy startup
clear table : remove an entry from a table
set table [id] : update or create a table entry's data
show table [id]: report table usage stats or dump this table's contents
add acl : add acl entry
clear acl <id> : clear the content of this acl
del acl : delete acl entry
get acl : report the patterns matching a sample for an ACL
show acl [id] : report available acls or dump an acl's contents
add map : add map entry
clear map <id> : clear the content of this map
del map : delete map entry
get map : report the keys and values matching a sample for a map
set map : modify map entry
show map [id] : report available maps or dump a map's contents
show events [] : show event sink state
show threads : show some threads debugging information
show peers [peers section]: dump some information about all the peers or this peers section
disable frontend : temporarily disable specific frontend
enable frontend : re-enable specific frontend
set maxconn frontend : change a frontend's maxconn setting
show servers conn [id]: dump server connections status (for backend <id>)
show servers state [id]: dump volatile server information (for backend <id>)
show backend : list backends in the current running config
shutdown frontend : stop a specific frontend
set dynamic-cookie-key backend : change a backend secret key for dynamic cookies
enable dynamic-cookie backend : enable dynamic cookies on a specific backend
disable dynamic-cookie backend : disable dynamic cookies on a specific backend
show errors : report last request and response errors for each proxy
show resolvers [id]: dumps counters from all resolvers section and
associated name servers
show pools : report information about the memory pools usage
show profiling : show CPU profiling options
set profiling : enable/disable CPU profiling
show cache : show cache status
trace <module> [cmd [args…]] : manage live tracing
show trace [] : show live tracing state

2. View haproxy running threads

root@pcfreak:/home/hipo/info# echo "show threads" | socat stdio /var/run/haproxy/haproxy.sock
Thread 1 : id=0x7f87b6e2c1c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
stuck=0 prof=0 harmless=1 wantrdv=0
cpu_ns: poll=3061065069437 now=3061065077880 diff=8443
curr_task=0
* Thread 2 : id=0x7f87b6e20700 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
stuck=0 prof=0 harmless=0 wantrdv=0
cpu_ns: poll=2969050092523 now=2969050197848 diff=105325
curr_task=0x7f87b006f740 (task) calls=1 last=0
fct=0x560978846340(task_run_applet) ctx=0x7f87b0190720(<CLI>)
strm=0x56097a763560 src=unix fe=GLOBAL be=GLOBAL dst=<CLI>
rqf=c48200 rqa=0 rpf=80008000 rpa=0 sif=EST,200008 sib=EST,204018
af=(nil),0 csf=0x56097a776ef0,8200
ab=0x7f87b0190720,9 csb=(nil),0
cof=0x56097a77fb00,1300:PASS(0x7f87b019a680)/RAW((nil))/unix_stream(22)
cob=(nil),0:NONE((nil))/NONE((nil))/NONE(0)

3. Show haproxy server connections

root@pcfreak:/home/hipo/info# echo "show servers conn" | socat stdio /var/run/haproxy/haproxy.sock
# bkname/svname bkid/svid addr port – purge_delay used_cur used_max need_est unsafe_nb safe_nb idle_lim idle_cur idle_per_thr[2]
http-websrv/ha1server-1 3/1 192.168.0.209 80 – 5000 0 12 12 0 0 -1 0 0 0
http-websrv/ha1server-2 3/2 192.168.0.200 80 – 5000 1 142 142 0 0 -1 0 0 0
http-websrv/ha1server-3 3/3 192.168.1.30 80 – 5000 0 0 0 0 0 -1 0 0 0
http-websrv/ha1server-4 3/4 192.168.1.14 80 – 5000 0 0 0 0 0 -1 0 0 0
http-websrv/ha1server-5 3/5 192.168.0.1 80 – 5000 0 13 13 0 0 -1 0 0 0
https-websrv/ha1server-1 5/1 192.168.0.209 443 – 5000 0 59 59 0 0 -1 0 0 0
https-websrv/ha1server-2 5/2 192.168.0.200 443 – 5000 11 461 461 0 0 -1 0 0 0
https-websrv/ha1server-3 5/3 192.168.1.30 443 – 5000 0 0 0 0 0 -1 0 0 0
https-websrv/ha1server-4 5/4 192.168.1.14 443 – 5000 0 0 0 0 0 -1 0 0 0
https-websrv/ha1server-5 5/5 192.168.0.1 443 – 5000 1 152 152 0 0 -1 0 0 0
MASTER/cur-1 6/1 – 0 – 0 0 0 0 0 0 0 0

4. Show Load balancer servers state

root@pcfreak:/home/hipo/info# echo "show servers state" | socat stdio /var/run/haproxy/haproxy.sock
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
3 http-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 3929 6 3 4 6 0 0 0 – 80 –
3 http-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 3928 6 3 4 6 0 0 0 – 80 –
3 http-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 3927 6 3 4 6 0 0 0 – 80 –
3 http-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 3929 6 3 4 6 0 0 0 – 80 –
3 http-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1708087 6 3 4 6 0 0 0 – 80 –
5 https-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 3929 6 3 4 6 0 0 0 – 443 –
5 https-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 3928 6 3 4 6 0 0 0 – 443 –
5 https-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 3927 6 3 4 6 0 0 0 – 443 –
5 https-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 3929 6 3 4 6 0 0 0 – 443 –
5 https-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1708087 6 3 4 6 0 0 0 – 443 –
6 MASTER 1 cur-1 – 2 0 0 0 1708087 1 0 0 0 0 0 0 – 0 –

5. Get general haproxy info on variables that can be used for Load Balancer fine tuning

root@pcfreak:/home/hipo/info# echo "show info" | socat stdio /var/run/haproxy/haproxy.sock
Name: HAProxy
Version: 2.2.9-2+deb11u5
Release_date: 2023/04/10
Nbthread: 2
Nbproc: 1
Process_num: 1
Pid: 3103635
Uptime: 19d 18h11m49s
Uptime_sec: 1707109
Memmax_MB: 0
PoolAlloc_MB: 1
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 200059
Maxsock: 200059
Maxconn: 99999
Hard_maxconn: 99999
CurrConns: 8
CumConns: 19677218
CumReq: 2740072
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 1
ConnRateLimit: 0
MaxConnRate: 2161
SessRate: 1
SessRateLimit: 0
MaxSessRate: 2161
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 32
Run_queue: 1
Idle_pct: 100
node: pcfreak
Stopping: 0
Jobs: 13
Unstoppable Jobs: 0
Listeners: 4
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0
FailedResolutions: 0
TotalBytesOut: 744390344175
BytesOutRate: 30080
DebugCommandsIssued: 0
Build info: 2.2.9-2+deb11u5

root@pcfreak:/home/hipo/info# echo "show errors" | socat stdio /var/run/haproxy/haproxy.sock
Total events captured on [14/Dec/2023:17:29:17.930] : 0

6. View all opened sessions and, the session age (time since it has been opened) and session exp (expiry)

root@pcfreak:/home/hipo/info# echo "show sess" | socat stdio /var/run/haproxy/haproxy.sock
0x56097a763560: proto=tcpv4 src=113.120.74.123:54651 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=37s calls=3 rate=0 cpu=0 lat=0 rq[f=848000h,i=0,an=00h,rx=1m58s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=1m58s,ax=] s0=[8,200000h,fd=24,ex=] s1=[8,40018h,fd=25,ex=] exp=1m51s
0x56097a812830: proto=tcpv4 src=190.216.236.134:35526 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=17s calls=3 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m42s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m42s,wx=,ax=] s0=[8,200008h,fd=40,ex=] s1=[8,200018h,fd=41,ex=] exp=12s
0x56097a784ad0: proto=tcpv4 src=103.225.203.131:33835 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=17s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m44s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m44s,wx=,ax=] s0=[8,200008h,fd=20,ex=] s1=[8,200018h,fd=21,ex=] exp=13s
0x7f87b0082cc0: proto=tcpv4 src=190.216.236.134:35528 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=14s calls=3 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m46s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m46s,wx=,ax=] s0=[8,200008h,fd=34,ex=] s1=[8,200018h,fd=35,ex=] exp=15s
0x7f87b0089e10: proto=tcpv4 src=40.130.105.242:50669 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=11s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m49s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m49s,wx=,ax=] s0=[8,200008h,fd=15,ex=] s1=[8,200018h,fd=16,ex=] exp=18s
0x7f87b010b450: proto=tcpv4 src=64.62.202.82:37562 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=7s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m52s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m52s,wx=,ax=] s0=[8,200008h,fd=26,ex=] s1=[8,200018h,fd=27,ex=] exp=22s
0x56097a7b8bc0: proto=tcpv4 src=85.208.96.211:54226 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=0s calls=2 rate=2 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m59s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m59s,wx=,ax=] s0=[8,200008h,fd=22,ex=] s1=[8,200018h,fd=23,ex=] exp=29s
0x7f87b008ec00: proto=tcpv4 src=3.135.192.206:60258 fe=http-in be=http-websrv srv=ha1server-2 ts=00 age=0s calls=2 rate=2 cpu=0 lat=0 rq[f=848000h,i=0,an=00h,rx=1m59s,wx=1m59s,ax=] rp[f=80008000h,i=0,an=00h,rx=1m59s,wx=1m59s,ax=] s0=[8,200008h,fd=28,ex=] s1=[8,200018h,fd=29,ex=] exp=29s
0x56097a7b2490: proto=tcpv4 src=45.147.249.119:62283 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=0s calls=3 rate=3 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m59s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m59s,wx=,ax=] s0=[8,200008h,fd=17,ex=] s1=[8,200018h,fd=18,ex=] exp=29s
0x7f87b0114f90: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c48200h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=

root@pcfreak:/home/hipo/info#

7. Disabling an haproxy frontend via UNIX socket

If you get some frontend that gets broken and this is monitored in Zabbix or other monitoring tool used to monitor you can use the haproxy stats interface to disable frontend

root@pcfreak:/home/hipo/info# echo "disable frontend https-websrv" | socat stdio /var/run/haproxy/haproxy.sock
…

8. Show general haproxy statistics (could tell you much about customer connections health state) and state of connection to backend

Lets check uptime details for frontends / backends, that is done with show stat command.

root@pcfreak:/home/hipo/info# echo "show stat" | socat stdio /var/run/haproxy/haproxy.sock
#

pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp

,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,
pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status

,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx

,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt

,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk

,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,

agent_duration,check_desc,agent_desc,check_rise,

check_fall,check_health,agent_rise,

agent_fall,agent_health,addr,cookie,mode,

algo,conn_rate,conn_rate_max,conn_tot,intercepted

,dcon,dses,wrew,connect,reuse,cache_lookups,

cache_hits,srv_icur,src_ilim,qtime_max,ctime_max,

rtime_max,ttime_max,eint,idle_conn_cur,

safe_conn_cur,used_conn_cur,need_conn_est,

http-in,FRONTEND,,,0,142,99999,371655,166897324,

1462777381,0,0,62,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,

1080,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,1080,

371655,,0,0,0,,,,,,,,,,,0,,,,,

http-websrv,ha1server-1,0,0,0,12,,9635,3893561

,64880833,,0,,0,3,15,0,UP

,254,0,1,41,9,4686,34728,,1,3,1,,4924,,2,0,,56,L4OK

,,0,,,,,,,,,,,900,168,,,,,1292679,,,0,0,0,2843,,,,

Layer4 check passed,,2,3,4,,,,192.168.0.209:80,,tcp,,,,,,,,

0,9635,0,,,0,,0,15024,0,672888,0,0,0,0,12,

http-websrv,ha1server-2,0,0,0,142,,321867,

149300590,1350577153,,0,,

1,4,30,0,UP,255,1,0,37,10,4685,89418,,1,3,2,,111864,,2

,0,,1080,L4OK,,0,,,,,,,,,,,37161,4822,,,,,6,,,0,12,0,

2120,,,,Layer4 check passed,,2,3,4,,,,192.168.0.200:80,,tcp,,,,,,,,0,321867,

0,,,0,,0,30223,0,1783442,0,0,0,0,142,

List continues here
….
…
..
.

9. Using netcat to view UNIX socket instead of socat

If you don't have the socat command on the server but you have netcat installed, you can also send the commands to the running haproxy daemon via nc's capability to send via UNIX socket via nc -U option.

-U Use UNIX-domain sockets. Cannot be used together with -F or -x.

root@pcfreak:/home/hipo/info# echo "set server"|nc -U /var/run/haproxy/haproxy.sock
Require 'backend/server'.

10. Get only statistics about running LB Backends and Frontends

To get only haproxy statistics about running Load Balancer BACKENDs and FRONTENDs

root@pcfreak:/home/hipo/info# echo "show stat" | sudo socat unix-connect:/var/run/haproxy/haproxy.sock stdio | awk -F '.' '/BACKEND/ {print $1, $6}'
http-websrv,BACKEND,0,0,2,142,10000,371880,167022255,1462985601,0,0,,1,7,46,0,UP

,255,1,4,,0,1709835,0,,1,3,0,,118878,,1,0,,1080,,,,,,,,,,,,,,38782,5001,0,0,0,0,5,,,0,8,0,2034

,,,,,,,,,,,,,,tcp,source,,,,,,,0,371864,0,,,,,0,30223,0,1783442,0,,,,,
https-websrv,BACKEND,0,0,5,461,10000,2374328,3083873321,740021649129,0,0,,28,42,626,0,UP
,255,1,4,,0,1709835,0,,1,5,0,,474550,,1,1,,1081,,,,,,,,,,,,,,451783,72307,0,0,0,0,0,,,0,0,0,6651

,,,,,,,,,,,,,,tcp,source,,,,,,,0,2374837,0,,,,,0,32794,0,46414141,0,,,,,

As you can see there are two configured BACKENDs that are in UP state, the other possibility is that they're DOWN if haproxy can't reach the backend.

root@pcfreak:/home/hipo/info# echo "show stat" | sudo socat unix-connect:/var/run/haproxy/haproxy.sock stdio | awk -F '.' '/FRONTEND/ {print $1, $6}'
http-in,FRONTEND,,,2,142,99999,371887,167024040,1462990718,0,0,62,,,,,OPEN

,,,,,,,,,1,2,0,,,,0,1,0,1080,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,1,1080,371887,,0,0,0,,,,,,,,,,,0,,,,,
https-in,FRONTEND,,,4,461,99999,2374337,3083881912,740021909870,0,0,112,,,,,OPEN

,,,,,,,,,1,4,0,,,,0,1,0,1081,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,1,1081,2374337,,0,0,0,,,,,,,,,,,0,,,,,
root@pcfreak:/home/hipo/info#

As you can see from the list of show help you can change maxconns supported, change the proxy rate-limit and even in real time change a haproxy.cfg configured section timeouts or even modify ACLs dynamicly for Backends and Frontends.

If you use those to make a modifications to the haproxy, that modifications should been written also to Haproxy's configured instance haproxy.cfg file.
If you want to check it reload the haproxy instance with the new written haproxy.cfg, through the Unix socket.

11. Shutting down specific opened sessions

Shutting down specific session that has been opened for too long is particularly useful to do, especially if you have some kind of VPN encryption device before the Haproxy server and an Application Backend server that is buggy and fails to properly close sessions at time, to cut off a specific sessions that has been hanging for days after reviewing it with "show sess".

root@pcfreak:/home/hipo/info# echo "shutdown session 0x56097a7707d0" | socat stdio /var/run/haproxy/haproxy.sock

12. Sending shutdown to backend on a certain configured LB service

To bring down a configured backend on a certain server after listing it:

root@pcfreak:/home/hipo/info# echo "disable server bk_mybackend/srv_myserver" | socat /var/run/haproxy.sock stdio

12. Sending multiple commands to haproxy socket

# echo "show info;show stat" | socat /var/run/haproxy/haproxy.sock stdio
…

13. Report table usage information or dump table data content

It is possible to view exact queued connections inside the sticky table. To get a list of available, available configured tables on the haproxy

root@pcfreak:/home/hipo/info# echo "show table" | socat /var/run/haproxy/haproxy.sock stdio
# table: https-websrv, type: ip, size:204800, used:498
# table: http-websrv, type: ip, size:204800, used:74

To get the exact record of queued IPs inside https-websrv.

root@pcfreak:/home/hipo/info# echo "show table https-websrv" | socat /var/run/haproxy/haproxy.sock stdio|head -10
# table: https-websrv, type: ip, size:204800, used:502
0x56097a7444e0: key=2.147.73.42 use=0 exp=1090876 server_id=2 server_name=ha1server-2
0x56097a792ac0: key=3.14.130.119 use=0 exp=1038004 server_id=2 server_name=ha1server-2
0x7f87b006a4e0: key=3.15.203.28 use=0 exp=1536721 server_id=2 server_name=ha1server-2
0x56097a7467f0: key=3.16.54.132 use=0 exp=387191 server_id=2 server_name=ha1server-2
0x7f87b0075f90: key=3.17.180.28 use=0 exp=353211 server_id=2 server_name=ha1server-2
0x56097a821b10: key=3.23.114.130 use=0 exp=1521100 server_id=2 server_name=ha1server-2
0x56097a7475b0: key=3.129.250.144 use=0 exp=121043 server_id=2 server_name=ha1server-2
0x7f87b004d240: key=3.134.112.27 use=0 exp=1182169 server_id=2 server_name=ha1server-2
0x56097a754c90: key=3.135.192.206 use=0 exp=1383882 server_id=2 server_name=ha1server-2

14. Show information about Haproxy startup

Sometimes, where logrotation is integrated on the server and haproxy's logs are log rotated to a central logging server, it might be hard to get information about Haproxy startup messages (warnings, errors etc.).
As digging through old haproxy logs might be tedious, you can simply get it via the stats interface.

root@pcfreak:/home/hipo/info# echo "show startup-logs" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio

[WARNING] 327/231534 (3103633) : parsing [/etc/haproxy/haproxy.cfg:62] : 'fullconn' ignored because frontend 'http-in' has no backend capability. Maybe you want 'maxconn' instead ?
[WARNING] 327/231534 (3103633) : parsing [/etc/haproxy/haproxy.cfg:69] : 'maxconn' ignored because backend 'http-websrv' has no frontend capability. Maybe you want 'fullconn' instead ?
[WARNING] 327/231534 (3103633) : parsing [/etc/haproxy/haproxy.cfg:114] : 'maxconn' ignored because backend 'https-websrv' has no frontend capability. Maybe you want 'fullconn' instead ?
[WARNING] 327/231534 (3103633) : config : missing timeouts for frontend 'http-in'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for frontend 'http-in' as it requires HTTP mode.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for backend 'http-websrv' as it requires HTTP mode.
[WARNING] 327/231534 (3103633) : config : missing timeouts for frontend 'https-in'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for frontend 'https-in' as it requires HTTP mode.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for backend 'https-websrv' as it requires HTTP mode.

15. Disable / Enable health check for haproxy configured backend

Disable health checks is useful, especially on non production server environments, during integration phase of application with Haproxy load balancer.

The general syntax is like this:

> disable health backend/server1

root@pcfreak:/home/hipo/info# echo "show servers state" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio 1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
3 http-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 13709 6 3 4 6 0 0 0 – 80 –
3 http-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 13708 6 3 4 6 0 0 0 – 80 –
3 http-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 13707 6 3 4 6 0 0 0 – 80 –
3 http-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 13709 6 3 4 6 0 0 0 – 80 –
3 http-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1717867 6 3 4 6 0 0 0 – 80 –
5 https-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 13709 6 3 4 6 0 0 0 – 443 –
5 https-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 13708 6 3 4 6 0 0 0 – 443 –
5 https-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 13707 6 3 4 6 0 0 0 – 443 –
5 https-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 13709 6 3 4 6 0 0 0 – 443 –
5 https-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1717867 6 3 4 6 0 0 0 – 443 –
6 MASTER 1 cur-1 – 2 0 0 0 1717867 1 0 0 0 0 0 0 – 0 –

Lets disable health checks for ha1server-1 server and http-websrv backend.

root@pcfreak:/home/hipo/info# echo "disable health http-websrv/ha1server-1" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio

To enable back health checks

root@pcfreak:/home/hipo/info# echo "enable health http-websrv/ha1server-1" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio

16. Change weight for server

if you have a round-robin Load balancing configured and already have a predefined configuration on how many percentage of the server to be sent to which application server (e.g. have a configured weight to dynamically change it via UNIX sock iface).

# Change weight by percentage of its original value

# socat unix-connect:/var/run/haproxy/haproxy.sock stdio

> set server be_app/webserv1 weight 50%

# Change weight in proportion to other servers
> set server be_app/webserv1 weight 100

root@pcfreak:/home/hipo/info# socat unix-connect:/var/run/haproxy/haproxy.sock stdio
set server http-websrv/ha1server-1 weight 50%
Backend is using a static LB algorithm and only accepts weights '0%' and '100%'.

17. Draining traffic from server / backend App in case of Maintenance

You can gradually drain traffic away from a particular server if those backend Application server should be put in maintenance mode for update or whatever. The drain option is very interesting and combined with scripting does open a lot of possibilities for the Load balancer system administrator to put an extra automation.

To drain, set server command with the state argument set to drain:

# Drain traffic
> set server backend_app/server1 state drain

# Allow server to accept traffic again
> set server backend_app/server1 state ready

root@pcfreak:/home/hipo/info# socat unix-connect:/var/run/haproxy/haproxy.sock stdio
set server http-websrv/ha1server-1 state drain

root@pcfreak:/home/hipo/info# socat unix-connect:/var/run/haproxy/haproxy.sock stdio
set server http-websrv/ha1server-1 state ready

18. Run Interactive Mode connection to haproxy UNIX stats socket

For a haproxies that has multiple configured proxied rules backends / frontends, it is nice to use the interactive mode.
Instead of processing a single line of semicolon separate commands, HAProxy takes one command at a time and waits for the user.
In interactive mode, HAProxy sends a “>” character and waits for input command. After command is submitted, HAProxy sends back the result and waits for a new command.
The interactive mode is especially useful during phase of integrating a new haproxy towards an application, where multiple things has to be tuned on the fly without, reloading the haproxy again and again.

On RPM based distros socat is compiled to have the readline interactive capability. Thus to use the haproxy haproxy stats connect interactive mode on RHEL / CentOS / Fedora and other RPM based distros simply use:

# socat /var/run/haproxy.sock readline
> show info
Name: HAProxy
Version: 2.2.9-2+deb11u5
Release_date: 2023/04/10
Nbthread: 2
Nbproc: 1
Process_num: 1
Pid: 3103635
Uptime: 19d 20h48m50s
Uptime_sec: 1716530
Memmax_MB: 0
PoolAlloc_MB: 1
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 200059
Maxsock: 200059
Maxconn: 99999
Hard_maxconn: 99999
CurrConns: 9
CumConns: 19789176
CumReq: 2757976
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 2161
SessRate: 0
SessRateLimit: 0
MaxSessRate: 2161
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 35
Run_queue: 1
Idle_pct: 100
node: pcfreak
Stopping: 0
Jobs: 14
Unstoppable Jobs: 0
Listeners: 4
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0
FailedResolutions: 0
TotalBytesOut: 744964070459
BytesOutRate: 0
DebugCommandsIssued: 0
Build info: 2.2.9-2+deb11u5

On Deb (Debian) based distributions such as Debian, Ubuntu Mint Linux, unfortunately the readline inractive mode is disabled due to licensing issues that makes readline not GPL license compliant.

root@pcfreak:/home/hipo/info# socat -V|awk 'NR < 5 || tolower($0) ~ /readline/'
socat by Gerhard Rieger and contributors – see www.dest-unreach.org
socat version 1.7.4.1 on Feb 3 2021 12:58:17
running on Linux version #1 SMP Debian 5.10.179-3 (2023-07-27), release 5.10.0-23-amd64, machine x86_64
features:
#undef WITH_READLINE

There is a workaround to emulate the Intearactive mode on Debians however like this:

root@pcfreak:/home/hipo/info# while [ 1 ]; do socat – /var/run/haproxy/haproxy.sock ; done

show table
# table: https-websrv, type: ip, size:204800, used:511
# table: http-websrv, type: ip, size:204800, used:67

show sess
0x56097a784ad0: proto=tcpv4 src=45.61.161.66:51416 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=1m13s calls=3 rate=0 cpu=0 lat=0 rq[f=848000h,i=0,an=00h,rx=47s,wx=,ax=] rp[f=80048000h,i=0,an=00h,rx=47s,wx=,ax=] s0=[8,200008h,fd=17,ex=] s1=[8,200018h,fd=23,ex=] exp=47s
0x56097a7707d0: proto=tcpv4 src=47.128.41.242:39372 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=16s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m45s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m45s,wx=,ax=] s0=[8,200008h,fd=35,ex=] s1=[8,200018h,fd=36,ex=] exp=14s
0x56097a781300: proto=tcpv4 src=54.36.148.40:17439 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=13s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m47s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m47s,wx=,ax=] s0=[8,200008h,fd=26,ex=] s1=[8,200018h,fd=28,ex=] exp=17s
0x56097a7fca80: proto=tcpv4 src=18.217.94.243:4940 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=7s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m53s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m53s,wx=,ax=] s0=[8,200008h,fd=21,ex=] s1=[8,200018h,fd=22,ex=] exp=23s
0x7f87b00778c0: proto=tcpv4 src=85.208.96.206:51708 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=4s calls=3 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m56s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m56s,wx=,ax=] s0=[8,200008h,fd=20,ex=] s1=[8,200018h,fd=24,ex=] exp=26s
0x56097a80c1e0: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 age=3s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=10s,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200008h,fd=15,ex=] s1=[8,204018h,fd=-1,ex=] exp=7s

To end the eternal loop press CTRL + z and kill first detached job %1 run:

# kiill %1

Sum it up what learned

What we learned in this article is how to use socat and netcat to connect and manage dynamically haproxy via its haproxy stats interface, without reloading the proxqy itself. We learned how to view various statistics and information on the proxy, its existing tables, caches, session information (such as age, and expiry). Also you've seen how to disable / enable configured backends as well as get available backends and frontends and their state.
You've seen how the drained option could be used to slowly drain connections towards configured backend, in case if you need to a maintenance on a backend node.
Also was pointed how to shutdown a specific long lived sessions that has been hanging and creating troubles towards app backends.

Finally, you've seen how to open an interactive connection towards the haproxy socket and send commands in a raw with socat (on distros where compiled with readline support) as well shown how to emulate the interactive mode of rest of distros whose socat is missing the readline support.

Tags: acl, certificate, client, debugging, Disabling, frontends, many things, Sending, threads, unix
Posted in Haproxy, Linux, Networking | No Comments »

Monitoring network traffic tools to debug network issues in console interactively on Linux

Thursday, December 14th, 2023

transport-layer-fourth-layer-data-transport-diagram

In my last article Debugging and routing network issues on Linux (common approaches), I've given some step by step methology on how to debug a network routing or unreachability issues between network hosts. As the article was mostly targetting a command line tools that can help debugging the network without much interactivity. I've decided to blog of a few other tools that might help the system administrator to debug network issues by using few a bit more interactive tools. Throughout the years of managing multitude of Linux based laptops and servers, as well as being involved in security testing and penetration in the past, these tools has always played an important role and are worthy to be well known and used by any self respecting sys admin or network security expert that has to deal with Linux and *Unix operating systems.

1. Debugging what is going on on a network level interactively with iptraf-ng

Historically iptraf and today's iptraf is also a great tool one can use to further aid the arsenal debug a network issue or Protocol problem, failure of packets or network interaction issues SYN -> ACK etc. proto interactions and check for Flag states and packets flow.

To use iptraf-ng which is a ncurses based tool just install it and launch it and select the interface you would like to debug trafic on.

To install On Debians distros

# apt install iptraf-ng –yes

# iptraf-ng

iptraf-ng-linux-select-interface-screen

Session-Layer-in-OSI-Model-diagram

2. Use hackers old tool sniffit to monitor current ongoing traffic and read plain text messages

Those older who remember the rise of Linux to the masses, should remember sniffit was a great tool to snoop for traffic on the network.

root@pcfreak:~# apt-cache show sniffit|grep -i description -A 10 -B10
Package: sniffit
Version: 0.5-1
Installed-Size: 139
Maintainer: Joao Eriberto Mota Filho <eriberto@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.14), libncurses6 (>= 6), libpcap0.8 (>= 0.9.8), libtinfo6 (>= 6)
Description-en: packet sniffer and monitoring tool
Sniffit is a packet sniffer for TCP/UDP/ICMP packets over IPv4. It is able
to give you a very detailed technical info on these packets, as SEQ, ACK,
TTL, Window, etc. The packet contents also can be viewed, in different
formats (hex or plain text, etc.).
.
Sniffit is based in libpcap and is useful when learning about computer
networks and their security.
Description-md5: 973beeeaadf4c31bef683350f1346ee9
Homepage: https://github.com/resurrecting-open-source-projects/sniffit
Tag: interface::text-mode, mail::notification, role::program, scope::utility,
uitoolkit::ncurses, use::monitor, use::scanning, works-with::mail,
works-with::network-traffic
Section: net
Priority: optional
Filename: pool/main/s/sniffit/sniffit_0.5-1_amd64.deb
Size: 61796
MD5sum: ea4cc0bc73f9e94d5a3c1ceeaa485ee1
SHA256: 7ec76b62ab508ec55c2ef0ecea952b7d1c55120b37b28fb8bc7c86645a43c485

Sniffit is not installed by default on deb distros, so to give it a try install it

# apt install sniffit –yes
# sniffit

sniffit-linux-check-tcp-traffic-screenshot

3. Use bmon to monitor bandwidth and any potential traffic losses and check qdisc pfifo
Linux network stack queues

root@pcfreak:~# apt-cache show bmon |grep -i description
Description-en: portable bandwidth monitor and rate estimator
Description-md5: 3288eb0a673978e478042369c7927d3f
root@pcfreak:~# apt-cache show bmon |grep -i description -A 10 -B10
Package: bmon
Version: 1:4.0-7
Installed-Size: 146
Maintainer: Patrick Matthäi <pmatthaei@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.17), libconfuse2 (>= 3.2.1~), libncursesw6 (>= 6), libnl-3-200 (>= 3.2.7), libnl-route-3-200 (>= 3.2.7), libtinfo6 (>= 6)
Description-en: portable bandwidth monitor and rate estimator
bmon is a commandline bandwidth monitor which supports various output
methods including an interactive curses interface, lightweight HTML output but
also simple ASCII output.
.
Statistics may be distributed over a network using multicast or unicast and
collected at some point to generate a summary of statistics for a set of
nodes.
Description-md5: 3288eb0a673978e478042369c7927d3f
Homepage: http://www.infradead.org/~tgr/bmon/
Tag: implemented-in::c, interface::text-mode, network::scanner,
role::program, scope::utility, uitoolkit::ncurses, use::monitor,
works-with::network-traffic
Section: net
Priority: optional
Filename: pool/main/b/bmon/bmon_4.0-7_amd64.deb
Size: 47348
MD5sum: c210f8317eafa22d9e3a8fb8316e0901
SHA256: 21730fc62241aee827f523dd33c458f4a5a7d4a8cf0a6e9266a3e00122d80645

root@pcfreak:~# apt install bmon –yes

root@pcfreak:~# bmon

4. Use nethogs net diagnosis text interactive tool

NetHogs is a small 'net top' tool.
Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process.

root@pcfreak:~# apt-cache show nethogs|grep -i description -A10 -B10
Package: nethogs
Source: nethogs (0.8.5-2)
Version: 0.8.5-2+b1
Installed-Size: 79
Maintainer: Paulo Roberto Alves de Oliveira (aka kretcheu) <kretcheu@gmail.com>
Architecture: amd64
Depends: libc6 (>= 2.15), libgcc1 (>= 1:3.0), libncurses6 (>= 6), libpcap0.8 (>= 0.9.8), libstdc++6 (>= 5.2), libtinfo6 (>= 6)
Description-en: Net top tool grouping bandwidth per process
NetHogs is a small 'net top' tool. Instead of breaking the traffic down per
protocol or per subnet, like most tools do, it groups bandwidth by process.
NetHogs does not rely on a special kernel module to be loaded.
Description-md5: 04c153c901ad7ca75e53e2ae32565ccd
Homepage: https://github.com/raboof/nethogs
Tag: admin::monitoring, implemented-in::c++, role::program,
uitoolkit::ncurses, use::monitor, works-with::network-traffic
Section: net
Priority: optional
Filename: pool/main/n/nethogs/nethogs_0.8.5-2+b1_amd64.deb
Size: 30936
MD5sum: 500047d154a1fcde5f6eacaee45148e7
SHA256: 8bc69509f6a8c689bf53925ff35a5df78cf8ad76fff176add4f1530e66eba9dc

root@pcfreak:~# apt install nethogs –yes

# nethogs

5;.Use iftop – to display network interface usage

root@pcfreak:~# apt-cache show iftop |grep -i description -A10 -B10
Package: iftop
Version: 1.0~pre4-7
Installed-Size: 97
Maintainer: Markus Koschany <apo@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.29), libncurses6 (>= 6), libpcap0.8 (>= 0.9.8), libtinfo6 (>= 6)
Description-en: displays bandwidth usage information on an network interface
iftop does for network usage what top(1) does for CPU usage. It listens to
network traffic on a named interface and displays a table of current bandwidth
usage by pairs of hosts. Handy for answering the question "Why is my Internet
link so slow?".
Description-md5: f7e93593aba6acc7b5a331b49f97466f
Homepage: http://www.ex-parrot.com/~pdw/iftop/
Tag: admin::monitoring, implemented-in::c, interface::text-mode,
role::program, scope::utility, uitoolkit::ncurses, use::monitor,
works-with::network-traffic
Section: net
Priority: optional
Filename: pool/main/i/iftop/iftop_1.0~pre4-7_amd64.deb
Size: 42044
MD5sum: c9bb9c591b70753880e455f8dc416e0a
SHA256: 0366a4e54f3c65b2bbed6739ae70216b0017e2b7421b416d7c1888e1f1cb98b7

root@pcfreak:~# apt install –yes iftop

iftop-interactive-network-traffic-output-linux-screenshot

6. Ettercap (tool) to active and passive dissect network protocols for in depth network and host analysis

root@pcfreak:/var/www/images# apt-cache show ettercap-common|grep -i description -A10 -B10
Package: ettercap-common
Source: ettercap
Version: 1:0.8.3.1-3
Installed-Size: 2518
Maintainer: Debian Security Tools <team+pkg-security@tracker.debian.org>
Architecture: amd64
Depends: ethtool, geoip-database, libbsd0 (>= 0.0), libc6 (>= 2.14), libcurl4 (>= 7.16.2), libgeoip1 (>= 1.6.12), libluajit-5.1-2 (>= 2.0.4+dfsg), libnet1 (>= 1.1.6), libpcap0.8 (>= 0.9.8), libpcre3, libssl1.1 (>= 1.1.1), zlib1g (>= 1:1.1.4)
Recommends: ettercap-graphical | ettercap-text-only
Description-en: Multipurpose sniffer/interceptor/logger for switched LAN
Ettercap supports active and passive dissection of many protocols
(even encrypted ones) and includes many feature for network and host
analysis.
.
Data injection in an established connection and filtering (substitute
or drop a packet) on the fly is also possible, keeping the connection
synchronized.
.
Many sniffing modes are implemented, for a powerful and complete
sniffing suite. It is possible to sniff in four modes: IP Based, MAC Based,
ARP Based (full-duplex) and PublicARP Based (half-duplex).
.
Ettercap also has the ability to detect a switched LAN, and to use OS
fingerprints (active or passive) to find the geometry of the LAN.
.
This package contains the Common support files, configuration files,
plugins, and documentation. You must also install either
ettercap-graphical or ettercap-text-only for the actual GUI-enabled
or text-only ettercap executable, respectively.
Description-md5: f1d894b138f387661d0f40a8940fb185
Homepage: https://ettercap.github.io/ettercap/
Tag: interface::text-mode, network::scanner, role::app-data, role::program,
uitoolkit::ncurses, use::scanning
Section: net
Priority: optional
Filename: pool/main/e/ettercap/ettercap-common_0.8.3.1-3_amd64.deb
Size: 734972
MD5sum: 403d87841f8cdd278abf20bce83cb95e
SHA256: 500aee2f07e0fae82489321097aee8a97f9f1970f6e4f8978140550db87e4ba9

root@pcfreak:/ # apt install ettercap-text-only –yes

root@pcfreak:/ # ettercap -C

7. iperf and netperf to measure connecitivity speed on Network LAN and between Linux server hosts

iperf and netperf are two very handy tools to measure the speed of a network and various aspects of the bandwidth. It is mostly useful when designing network infrastructure or building networks from scratch.

If you never used netperf in the past here is a description from man netperf

NAME
netperf – a network performance benchmark

SYNOPSIS
netperf [global options] — [test specific options]

DESCRIPTION
Netperf is a benchmark that can be used to measure various aspects of
networking performance. Currently, its focus is on bulk data transfer
and request/response performance using either TCP or UDP, and the
Berkeley Sockets interface. In addition, tests for DLPI, and Unix Do‐
main Sockets, tests for IPv6 may be conditionally compiled-in.

root@freak:~# netperf
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 65536 65536 10.00 17669.96

Testing UDP network throughput using NetPerf

Change the test name from TCP_STREAM to UDP_STREAM. Let’s use 1024 (1MB) as the message size to be sent by the client.
If you receive the following error send_data: data send error: Network is unreachable (errno 101) netperf: send_omni:

send_data failed: Network is unreachable, add option -R 1 to remove the iptable rule that prohibits NetPerf UDP flow.

$ netperf -H 172.31.56.48 -t UDP_STREAM -l 300 — -R 1 -m 1024
MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.31.56.48 () port 0 AF_INET
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

212992 1024 300.00 9193386 0 251.04
212992 300.00 9131380 249.35

UDP Throughput in a WAN

$ netperf -H HOST -t UDP_STREAM -l 300 — -R 1 -m 1024
MIGRATED UDP STREAM TEST from (null) (0.0.0.0) port 0 AF_INET to (null) () port 0 AF_INET : histogram : spin interval
Socket Message Elapsed Messages
Size Size Time Okay Errors Throughput
bytes bytes secs # # 10^6bits/sec

9216 1024 300.01 35627791 0 972.83
212992 300.01 253099 6.91

Testing TCP throughput using iPerf

Here is a short description of iperf

NAME
iperf – perform network throughput tests

SYNOPSIS
iperf -s [options]

iperf -c server [options]

iperf -u -s [options]

iperf -u -c server [options]

DESCRIPTION
iperf 2 is a tool for performing network throughput and latency mea‐
surements. It can test using either TCP or UDP protocols. It supports
both unidirectional and bidirectional traffic. Multiple simultaneous
traffic streams are also supported. Metrics are displayed to help iso‐
late the causes which impact performance. Setting the enhanced (-e) op‐
tion provides all available metrics.

The user must establish both a both a server (to discard traffic) and a
client (to generate traffic) for a test to occur. The client and server
typically are on different hosts or computers but need not be.

Run iPerf3 as server on the server:

$ iperf3 –server –interval 30
———————————————————–
Server listening on 5201
———————————————————–

Test TCP Throughput in Local LAN

$ iperf3 –client 172.31.56.48 –time 300 –interval 30
Connecting to host 172.31.56.48, port 5201
[ 4] local 172.31.100.5 port 44728 connected to 172.31.56.48 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-30.00 sec 1.70 GBytes 488 Mbits/sec 138 533 KBytes
[ 4] 30.00-60.00 sec 260 MBytes 72.6 Mbits/sec 19 489 KBytes
[ 4] 60.00-90.00 sec 227 MBytes 63.5 Mbits/sec 15 542 KBytes
[ 4] 90.00-120.00 sec 227 MBytes 63.3 Mbits/sec 13 559 KBytes
[ 4] 120.00-150.00 sec 228 MBytes 63.7 Mbits/sec 16 463 KBytes
[ 4] 150.00-180.00 sec 227 MBytes 63.4 Mbits/sec 13 524 KBytes
[ 4] 180.00-210.00 sec 227 MBytes 63.5 Mbits/sec 14 559 KBytes
[ 4] 210.00-240.00 sec 227 MBytes 63.5 Mbits/sec 14 437 KBytes
[ 4] 240.00-270.00 sec 228 MBytes 63.7 Mbits/sec 14 516 KBytes
[ 4] 270.00-300.00 sec 227 MBytes 63.5 Mbits/sec 14 524 KBytes
– – – – – – – – – – – – – – – – – – – – – – – – –
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-300.00 sec 3.73 GBytes 107 Mbits/sec 270 sender
[ 4] 0.00-300.00 sec 3.73 GBytes 107 Mbits/sec receiver

Test TCP Throughput in a WAN Network

$ iperf3 –client HOST –time 300 –interval 30
Connecting to host HOST, port 5201
[ 5] local 192.168.1.73 port 56756 connected to HOST port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-30.00 sec 21.2 MBytes 5.93 Mbits/sec
[ 5] 30.00-60.00 sec 27.0 MBytes 7.55 Mbits/sec
[ 5] 60.00-90.00 sec 28.6 MBytes 7.99 Mbits/sec
[ 5] 90.00-120.00 sec 28.7 MBytes 8.02 Mbits/sec
[ 5] 120.00-150.00 sec 28.5 MBytes 7.97 Mbits/sec
[ 5] 150.00-180.00 sec 28.6 MBytes 7.99 Mbits/sec
[ 5] 180.00-210.00 sec 28.4 MBytes 7.94 Mbits/sec
[ 5] 210.00-240.00 sec 28.5 MBytes 7.97 Mbits/sec
[ 5] 240.00-270.00 sec 28.6 MBytes 8.00 Mbits/sec
[ 5] 270.00-300.00 sec 27.9 MBytes 7.81 Mbits/sec
– – – – – – – – – – – – – – – – – – – – – – – – –
[ ID] Interval Transfer Bitrate
[ 5] 0.00-300.00 sec 276 MBytes 7.72 Mbits/sec sender
[ 5] 0.00-300.00 sec 276 MBytes 7.71 Mbits/sec receiver

$ iperf3 –client 172.31.56.48 –interval 30 -u -b 100MB
Accepted connection from 172.31.100.5, port 39444
[ 5] local 172.31.56.48 port 5201 connected to 172.31.100.5 port 36436
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-30.00 sec 354 MBytes 98.9 Mbits/sec 0.052 ms 330/41774 (0.79%)
[ 5] 30.00-60.00 sec 355 MBytes 99.2 Mbits/sec 0.047 ms 355/41903 (0.85%)
[ 5] 60.00-90.00 sec 354 MBytes 98.9 Mbits/sec 0.048 ms 446/41905 (1.1%)
[ 5] 90.00-120.00 sec 355 MBytes 99.4 Mbits/sec 0.045 ms 261/41902 (0.62%)
[ 5] 120.00-150.00 sec 354 MBytes 99.1 Mbits/sec 0.048 ms 401/41908 (0.96%)
[ 5] 150.00-180.00 sec 353 MBytes 98.7 Mbits/sec 0.047 ms 530/41902 (1.3%)
[ 5] 180.00-210.00 sec 353 MBytes 98.8 Mbits/sec 0.059 ms 496/41904 (1.2%)
[ 5] 210.00-240.00 sec 354 MBytes 99.0 Mbits/sec 0.052 ms 407/41904 (0.97%)
[ 5] 240.00-270.00 sec 351 MBytes 98.3 Mbits/sec 0.059 ms 725/41903 (1.7%)
[ 5] 270.00-300.00 sec 354 MBytes 99.1 Mbits/sec 0.043 ms 393/41908 (0.94%)
– – – – – – – – – – – – – – – – – – – – – – – – –
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-300.04 sec 3.45 GBytes 98.94 Mbits/sec 0.043 ms 4344/418913 (1%)

UDP Throughput in a WAN

$ iperf3 –client HOST –time 300 -u -b 7.7MB
Accepted connection from 45.29.190.145, port 60634
[ 5] local 172.31.56.48 port 5201 connected to 45.29.190.145 port 52586
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-30.00 sec 27.4 MBytes 7.67 Mbits/sec 0.438 ms 64/19902 (0.32%)
[ 5] 30.00-60.00 sec 27.5 MBytes 7.69 Mbits/sec 0.446 ms 35/19940 (0.18%)
[ 5] 60.00-90.00 sec 27.5 MBytes 7.68 Mbits/sec 0.384 ms 39/19925 (0.2%)
[ 5] 90.00-120.00 sec 27.5 MBytes 7.68 Mbits/sec 0.528 ms 70/19950 (0.35%)
[ 5] 120.00-150.00 sec 27.4 MBytes 7.67 Mbits/sec 0.460 ms 51/19924 (0.26%)
[ 5] 150.00-180.00 sec 27.5 MBytes 7.69 Mbits/sec 0.485 ms 37/19948 (0.19%)
[ 5] 180.00-210.00 sec 27.5 MBytes 7.68 Mbits/sec 0.572 ms 49/19941 (0.25%)
[ 5] 210.00-240.00 sec 26.8 MBytes 7.50 Mbits/sec 0.800 ms 443/19856 (2.2%)
[ 5] 240.00-270.00 sec 27.4 MBytes 7.66 Mbits/sec 0.570 ms 172/20009 (0.86%)
[ 5] 270.00-300.00 sec 25.3 MBytes 7.07 Mbits/sec 0.423 ms 1562/19867 (7.9%)
– – – – – – – – – – – – – – – – – – – – – – – – –
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-300.00 sec 272 MBytes 7.60 Mbits/sec 0.423 ms 2522/199284 (1.3%)
[SUM] 0.0-300.2 sec 31 datagrams received out-of-order

Sum it up what learned

Debugging network issues and snooping on a Local LAN (DMZ) network on a server or home LAN is useful to debug for various network issues and more importantly track and know abou tsecurity threads such as plain text passowd communication via insecure protocols a failure of proper communication between Linux network nodes at times, or simply to get a better idea on what kind of network is your new purchased dedicated server living in .It can help you also strenghten your security and close up any possible security holes, or even help you start thinking like a security intruder (cracker / hacker) would do. In this article we went through few of my favourite tools I use for many years quite often. These tools are just part of the tons of useful *Unix free tools available to do a network debug. Tools mentioned up are worthy to install on every server you have to administratrate or even your home desktop PCs, these are iptraf, sniffit, iftop, bmon, nethogs, nmon, ettercap, iperf and netperf.
If you have some other useful tools used on Linux sys admin tasks please share, I'll be glad to know it and put them in my arsenal of used tools.

Enjoy ! 🙂

Tags: about, ack, admin, administrator, Aid, amd64, and, ANY, architecture, are, arp, bandwidth, console, information, issues, linux network, Monitoring, network stack, root, use
Posted in Computer Security, Linux, Networking | No Comments »

Archive for the ‘Linux’ Category

Haproxy log-format paremeters ASCII table

1. RIPing audio CDs to WAV and Conver to MP3 from terminal

2. RIPping audio CD on Linux with rip-audio-cds-linux.sh script

3. Dump Audio CD to MP3 with Graphical program ( ripperx )

1. Creating a KVM VM LIVE (running machine) backup

2. Listing backed-up snapshot content of KVM machine

3. Backup KVM only Virtual Machine data files (but not VM state) Live

4. KVM restore snapshot (backup)

5. Delete old unnecessery KVM VM backup

1. Listing all available options that can be send via the haproxy.sock UNIX socket interface

2. View haproxy running threads

3. Show haproxy server connections

4. Show Load balancer servers state

5. Get general haproxy info on variables that can be used for Load Balancer fine tuning

6. View all opened sessions and, the session age (time since it has been opened) and session exp (expiry)

7. Disabling an haproxy frontend via UNIX socket

8. Show general haproxy statistics (could tell you much about customer connections health state) and state of connection to backend

9. Using netcat to view UNIX socket instead of socat

10. Get only statistics about running LB Backends and Frontends

11. Shutting down specific opened sessions

12. Sending shutdown to backend on a certain configured LB service

12. Sending multiple commands to haproxy socket

13. Report table usage information or dump table data content

14. Show information about Haproxy startup

15. Disable / Enable health check for haproxy configured backend

16. Change weight for server

17. Draining traffic from server / backend App in case of Maintenance

18. Run Interactive Mode connection to haproxy UNIX stats socket

Sum it up what learned

1. Debugging what is going on on a network level interactively with iptraf-ng

2. Use hackers old tool sniffit to monitor current ongoing traffic and read plain text messages

3. Use bmon to monitor bandwidth and any potential traffic losses and check qdisc pfifo Linux network stack queues

4. Use nethogs net diagnosis text interactive tool

5;.Use iftop – to display network interface usage

6. Ettercap (tool) to active and passive dissect network protocols for in depth network and host analysis

7. iperf and netperf to measure connecitivity speed on Network LAN and between Linux server hosts

Testing TCP throughput using iPerf

Test TCP Throughput in Local LAN

Test TCP Throughput in a WAN Network

UDP Throughput in a WAN

Sum it up what learned

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Tags

Top Post Views

blogtopsites

3. Use bmon to monitor bandwidth and any potential traffic losses and check qdisc pfifo
Linux network stack queues