data Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘data’

How to boost Linux Server Speed with tmpfs and few smart Optimization tweaks

Monday, October 20th, 2025

speed-up-accelerate-wordpress-joomla-drupal-cms-and-mysql-server-with-tmpfs_ramfs_decrease-pageload-times-with-ram-caching.png

If you’ve ever managed a busy Linux server, you’ve probably noticed how I/O bottlenecks can make your system feel sluggish – even when CPU and memory usage look fine. I recently faced this problem on a Debian box running Nginx, PHP-FPM, and MySQL / MariaDB. The fix turned out to be surprisingly simple: using tmpfs for temporary directories and tweaking a few kernel parameters.

1. Identify the resource issue Bottleneck

Running iostat -x 1 showed my /var/lib/mysql drive constantly pegged at 100% utilization, while CPU load stayed low. Classic disk-bound performance issue.

I checked swap usage:

# swapon –show

# free -h

The system was swapping occasionally — not good for performance-critical workloads.

2. Use tmpfs for Cache and Temporary Files

Linux allows you to use part of your RAM as a fast, volatile filesystem. Perfect for things like cache and sessions.

I edited /etc/fstab and added:

tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=1G 0 0

tmpfs /var/cache/nginx tmpfs defaults,noatime,mode=0755,size=512M 0 0

Then:

# mount -a

Immediately, I noticed fewer disk I/O spikes. Nginx’s cache hits were lightning-fast, and PHP temporary files were written in memory instead of SSD.

3. Tune Kernel Parameters

Adding these lines to /etc/sysctl.conf helped reduce swapping and improve responsiveness:

vm.swappiness=10

vm.vfs_cache_pressure=50

Then apply:

# sysctl -p

4. Use tmpfs for MySQL / MariaDB tmpdir (Optional)

If you have enough RAM, move MySQL’s temporary directory to tmpfs:

# mkdir -p /mnt/mysqltmp

# mount -t tmpfs -o size=512M tmpfs /mnt/mysqltmp

# chown mysql:mysql /mnt/mysqltmp

Then set in /etc/mysql/my.cnf: / /etc/mysql/mariadb.cnf

tmpdir = /mnt/mysqltmp

This dramatically speeds up large sorts and temporary table creation.

5. Monitor the Effects and tune up if necessery

After applying these changes, iostat showed disk utilization dropping from 95% to under 20% under the same workload.
Average response times from Nginx dropped by around 30–40%, and the server felt much more responsive.
However other cases might be different so it is a good idea to play around with tmpfs side according to your CPU / Memory system parameters etc. and find out the best values that would fit your Linux setup best.

Short rephrasal

tmpfs is one of those underused Linux features that can make a real-world difference for sysadmins and self-hosters. Just remember that data in tmpfs disappears after reboot, so only use it for volatile data (cache, temp files, etc.).

If you’re running a VPS or small dedicated box and looking for a quick, low-cost performance boost — give tmpfs a try. You might be surprised how much smoother your system feels.

Tags: cnf, data, filesystem, How to, Linux Server Speed, mnt, response times, Short, tmpfs, use
Posted in Linux, Linux and FreeBSD Desktop, MySQL, Performance Tuning, Various | No Comments »

Monitoring Linux server activity with psacct, accton, lastconn commands. Understanding /var/account/pacct and /var/account/acct Acct File Format

Friday, September 5th, 2025

Intro: Understanding Process Accounting in Linux/Unix Systems

In multi-user Unix and Linux environments, keeping track of what’s happening under the hood is essential—for security, system performance, compliance, and resource management. One powerful but often overlooked feature that provides deep visibility into system activity is process accounting.

Process accounting records every command executed on a system, along with key details such as the user who ran it, the time it was executed, how long it ran, how much CPU and memory it consumed, and more. This information is stored in binary log files such as /var/account/pacct or /var/account/acct, which can then be analyzed using specialized tools.

System administrators, security analysts, and auditors use this data for a wide range of purposes: tracking user activity, analyzing system resource usage, investigating security incidents, or even supporting cost allocation in shared computing environments. In regulated industries, process accounting can also serve as a reliable audit trail, helping organizations meet compliance requirements.

While it doesn’t capture full command-line arguments or real-time process behavior, process accounting provides a concise, system-wide history of what has been run and by whom—making it a valuable component of any robust monitoring or auditing strategy.

In this article, we'll explore what process accounting is, how it works, what kind of data it records, and how it can be used to improve system transparency, security, and efficiency.

Understanding the /var/account/pacct or /var/account/acct files involves diving into process accounting on Unix and Linux systems. These files log detailed information about every process that has run on the system. They are binary files and not human-readable so to parse them you the sa / lastcomm tools.

**What are /var/account/pacct and /var/account/acct dirs and common locations across *nix OS-es?**

These are process accounting files used to log per-process execution details.
Created and maintained by the acct package (acctd or psacct, depending on your distro).
Location may vary slightly by distro:
- /var/account/pacct – common on Debian/Ubuntu as well as on FreeBSD / OpenBSD
- /var/account/acct – common on Red Hat/CentOS.
- /var/adm/pact – is location for Sun Solaris UNIX (if you face some legacy systems somewhere and you wonder what it is)

What Do process account data dirs contain ?

These binary log files record information like:

Field	Description
ac_comm	Command name (truncated to 16 characters)
ac_utime	User CPU time used
ac_stime	System CPU time used
ac_etime	Elapsed time
ac_btime	Process start time (in epoch seconds)
ac_uid	User ID of the process owner
ac_gid	Group ID of the process
ac_mem	Average memory usage (in pages)
ac_io	I/O operations (usually blocks read/written)
ac_tty	Controlling terminal
ac_exitcode	Exit status
ac_flag	Flags (forked, exec'd, etc.)

Note: The exact fields and formats may vary based on your OS and kernel version.

How to View or Parse the acct File

Since the file is binary, use tools to interpret it:

1. lastcomm command – Shows last executed commands

# lastcomm

Output includes user, command, CPU time, and date/time.

Note ! that if lastcomm returns an empty output that means the process accounting (if installed on the system) is not yet enabled. To have it return its purposed data enable it.

2. accton – Enables or disables accounting

To enable proc accounting:

# accton on /var/account/pact

To disable proc accounting:

# accton off /var/account/pact

3. Enable disable process accounting on Linux boot

On Redhat based Linux distributions accton can be switch on / off on boot up with the good old chkconfig command

On Debian based Ubuntu / Mint etc. distros

To enable process accounging service on boot:

# update-rc.d acct defaults
# /etc/init.d/acct start

Alternatively you can do it on Debian Linux via its config /etc/default/acct:

# cat /etc/default/acct

# Defaults for acct

# If you want to keep acct installed, but not started automatically, set this

# variable to 0. Because /etc/cron.daily/acct calls the initscript daily, it is

# not sufficient to stop acct once after booting if your machine remains up.

ACCT_ENABLE="1"

# Amount of days that the logs are kept.

ACCT_LOGGING="30"

On Redhat RPM distros good old chkconfig is used

# chkconfig psacct on

4. sa command (summary accounting)

# sa

    3414   54989.71re      64.59cp         0avio     16834k

      34   54603.57re      61.21cp         0avio     38640k   ***other*

      14       1.33re       1.29cp         0avio     74725k   ls

      39      11.88re       1.16cp         0avio     56746k   find

     149      25.51re       0.83cp         0avio     64912k   spamd

      16     293.95re       0.06cp         0avio         0k   kworker/dying*

      22       5.12re       0.01cp         0avio      3951k   sshd

       3       0.01re       0.01cp         0avio     61611k   awstats.pl

       2      11.93re       0.00cp         0avio     58288k   ssh

       3       0.00re       0.00cp         0avio     76224k   php5.6

       2       0.00re       0.00cp         0avio     75824k   php7.3

       7       0.65re       0.00cp         0avio     56846k   bash

       2       0.00re       0.00cp         0avio     75360k   php7.0

       2       0.00re       0.00cp         0avio     69920k   php8.3

       4       0.00re       0.00cp         0avio     58032k   mkvalidrcptto

       2       0.00re       0.00cp         0avio     75408k   php7.4

       2       0.00re       0.00cp         0avio     69920k   php8.2

      60       2.02re       0.00cp         0avio     56882k   cron*

       3       1.24re       0.00cp         0avio     56144k   less

       2       0.00re       0.00cp         0avio     57272k   dpkg

       5       0.01re       0.00cp         0avio     23664k   dig

       3       0.00re       0.00cp         0avio      1443k   lastcomm

       2       0.31re       0.00cp         0avio       940k   dump-acct

       2       0.00re       0.00cp         0avio     13702k   php8.0

       2       0.02re       0.00cp         0avio      1339k   sa

    1226       1.27re       0.00cp         0avio       631k   watchdog*

     560       0.00re       0.00cp         0avio      5316k   zabbix_agentd*

     153       0.00re       0.00cp         0avio     56612k   bash*

     149       0.12re       0.00cp         0avio     56016k   pwd

     139       2.04re       0.00cp         0avio       644k   sh

     128       0.04re       0.00cp         0avio       644k   sessionclean*

     115       0.00re       0.00cp         0avio       644k   which

      71       0.00re       0.00cp         0avio     56278k   sed

      56       0.00re       0.00cp         0avio     56528k   chown

      39       0.19re       0.00cp         0avio       618k   qmail-queue

      39       0.00re       0.00cp         0avio       705k   vdelivermail

      39       0.00re       0.00cp         0avio       590k   qmail-local

      29       0.53re       0.00cp         0avio     56144k   wc

      28       0.24re       0.00cp         0avio     56032k   who

      28       0.00re       0.00cp         0avio       701k   valias

      28       0.00re       0.00cp         0avio       702k   vuserinfo

      24      17.13re       0.00cp         0avio      2282k   qmail-smtpd

      21       5.04re       0.00cp         0avio      3924k   sshd*

      17       1.16re       0.00cp         0avio     56544k   ping

      12       0.00re       0.00cp         0avio     56032k   date

      10       0.00re       0.00cp         0avio     56512k   whoamiSummarizes command usage statistics.

For example to pick up percentage information based on process accounting regarding which process took over most of server’s CPU time

# sa -c

    3414 100.00%   54989.71re 100.00%      64.59cp 100.00%         0avio     16834k

      34    1.00%   54603.57re   99.30%      61.21cp   94.77%         0avio     38640k   ***other*

      14    0.41%       1.33re    0.00%       1.29cp    1.99%         0avio     74725k   ls

      39    1.14%      11.88re    0.02%       1.16cp    1.80%         0avio     56746k   find

     149    4.36%      25.51re    0.05%       0.83cp    1.28%         0avio     64912k   spamd

      16    0.47%     293.95re    0.53%       0.06cp    0.09%         0avio         0k   kworker/dying*

      22    0.64%       5.12re    0.01%       0.01cp    0.02%         0avio      3951k   sshd

       3    0.09%       0.01re    0.00%       0.01cp    0.02%         0avio     61611k   awstats.pl

       2    0.06%      11.93re    0.02%       0.00cp    0.01%         0avio     58288k   ssh

       3    0.09%       0.00re    0.00%       0.00cp    0.00%         0avio     76224k   php5.6

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     75824k   php7.3

       7    0.21%       0.65re    0.00%       0.00cp    0.00%         0avio     56846k   bash

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     75360k   php7.0

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     69920k   php8.3

       4    0.12%       0.00re    0.00%       0.00cp    0.00%         0avio     58032k   mkvalidrcptto

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     75408k   php7.4

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     69920k   php8.2

      60    1.76%       2.02re    0.00%       0.00cp    0.00%         0avio     56882k   cron*

       3    0.09%       1.24re    0.00%       0.00cp    0.00%         0avio     56144k   less

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     57272k   dpkg

       5    0.15%       0.01re    0.00%       0.00cp    0.00%         0avio     23664k   dig

       3    0.09%       0.00re    0.00%       0.00cp    0.00%         0avio      1443k   lastcomm

       2    0.06%       0.31re    0.00%       0.00cp    0.00%         0avio       940k   dump-acct

       2    0.06%       0.00re    0.00%       0.00cp    0.00%         0avio     13702k   php8.0

       2    0.06%       0.02re    0.00%       0.00cp    0.00%         0avio      1339k   sa

    1226   35.91%       1.27re    0.00%       0.00cp    0.00%         0avio       631k   watchdog*

     560   16.40%       0.00re    0.00%       0.00cp    0.00%         0avio      5316k   zabbix_agentd*

     153    4.48%       0.00re    0.00%       0.00cp    0.00%         0avio     56612k   bash*

5. dump-acct (from GNU acct) – Parses the binary files stored on machine

# dump-acct /var/account/pacct

Outputs detailed info per process in a readable format.

6. Installing the process accounting Tools if not already on the system

On Debian/Ubuntu to install:

# apt install acct

On Red Hat/CentOS to install:

# yum install psacct

File Format Details (Technical details)

The pacct file format is defined in C struct form in kernel headers:

struct acct {

    char ac_flag;

    char ac_version;

    uint16_t ac_tty;

    uint32_t ac_exitcode;

    uint32_t ac_uid;

    uint32_t ac_gid;

    uint32_t ac_pid;

    uint32_t ac_ppid;

    uint32_t ac_btime;

    uint32_t ac_etime;

    uint32_t ac_utime;

    uint32_t ac_stime;

    uint32_t ac_mem;

    uint32_t ac_io;

    char ac_comm[16];

};

(Fields may be compressed using "comp_t" – a 16-bit pseudo-floating point format used to save space.)

7. How to Decode comp_t (Optional)

Fields like ac_utime, ac_stime, ac_etime, ac_mem, and ac_io are often stored in comp_t format:

#define MANTSIZE 13

#define EXPSIZE 3

unsigned int expand(comp_t c) {

return ((c & 0x1FFF) << ((c >> 13) & 0x7));

}

So a comp_t value can be interpreted as:

value = mantissa << (exponent)

8. File Growth and Rotation manage

These files grow over time and can become large.
Best practice is to rotate and archive them regularly using logrotate or cron jobs.
Example rotation script (manual):

# mv /var/account/pacct /var/account/pacct.1

# touch /var/account/pacct

# accton /var/account/pacct

Summary what learned

We have learned on what is Process accounting on Linux. The purpose of process accounting is to keep an Audit of what has happened on the Linux server recently thus it is a must to be present on Linux machines, that needs to keep a good history record for potential Security audits like PCI-DSS etc. Here is a short table to recall once again what learned.

Topic	Summary
What it is	Binary log of all executed processes
Location	/var/account/pacct or /var/account/acct
Tools to read	lastcomm, sa, dump-acct
Format	Binary; contains CPU, memory, I/O, UID info
Use case	Auditing, usage tracking, forensics

Tags: about, Account, Accounting, adm, command, CPU, cron jobs, data, debian linux, distros, enable, kernel version, Output, process, Redhat, sshd, time, Understanding, var, www
Posted in Linux, Various | No Comments »

How to convert .p12 ssl certificate to .pem with openssl command

Friday, March 21st, 2025

In cryptography, PKCS #12 defines an archive file format for storing many cryptography objects as a single file. It is commonly used to bundle a private key with its X.509 certificate or to bundle all the members of a chain of trust.

A PKCS #12 file may be encrypted and signed. The internal storage containers, called "SafeBags", may also be encrypted and signed. A few SafeBags are predefined to store certificates, private keys and CRLs. Another SafeBag is provided to store any other data at individual implementer's choice.

PKCS #12 is one of the family of standards called Public-Key Cryptography Standards (PKCS) published by RSA Laboratories.

Privacy-Enhanced Mail (PEM) is a de facto file format for storing and sending cryptographic keys, certificates, and other data, based on a set of 1993 IETF standards defining "privacy-enhanced mail." While the original standards were never broadly adopted and were supplanted by PGP and S/MIME, the textual encoding they defined became very popular. The PEM format was eventually formalized by the IETF in RFC 7468.

If you already have a .P12 certificate password signed provided by someone and you need to convert it a .PEM, this can be done like so:

To convert .p12 certificate :

# Initialize variable
cert_p12_in=your-domain-name-cert.p12
cert_p12_pass='XXXZZZYYYPPPQQQ'
cert_pem_out=your-domain-name-cert.pem

# Extract the private key
openssl pkcs12 -in $cert_p12_in -nocerts -nodes -passin "pass:$cert_p12_pass" 2>/dev/null | sed -ne '/-BEGIN PRIVATE KEY-/,/-END PRIVATE KEY-/p' > $cert_pem_out

# Extract the certificate
openssl pkcs12 -in $cert_p12_in -clcerts -nokeys -passin "pass:$cert_p12_pass" 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >> $cert_pem_out

# Extract the Chain certificate, potentially nothing
openssl pkcs12 -in $cert_p12_in -cacerts -nokeys -chain -passin "pass:$cert_p12_pass" 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >> $cert_pem_out

# Display the result
cat $cert_pem_out

That's all you should have the .p12 to .pem successfully converted.
Cheers ! 🙂

Tags: bundle, command, data, dev, file format, How to, null, openssl, pem, signed
Posted in Everyday Life, Web and CMS | No Comments »

Zabbix Power Shell PS1 script to write zero or one if string is matched inside log file

Monday, December 2nd, 2024

At work we had setup zabbix log file processing for few servers for a service that is doing a Monitoring Health Checks for a a special application via an encrypted strong encrypted tunnel. The app based on the check reports whether the remote side has processed data or not.
As me and my team are not maintainers of the zabbix-server where the zabbix-agents are sending the data, there is a multiple content of data being sent in simply "" empty strings via a zabbix Item setup. Those empty strings however gets stored in the zabbix-server database and since this check is made frequently about 500 hundred records of empty string lines are being written to the zabbix server, we got complaint by the zabbix adminsitrators, that we have to correct our Monitoring setup to not flood the zabbix-server.

Since zabbix cannot catch up the "" empty string and we cannot supress the string from being written in the Item, we needed a way to change the monitoring so that the configured Application check returns 1 (on error) and 0 (on success).

Zabbix even though advanced has a strange when zabbix log[] function, e.g.

log[/path/to/log,,,,skip]

log function, used to analyze a log file and cut out last or first lines of a file simmilar to UNIX's head and tail over log files this is described in the zabbix log file monitoring here . If a string is matched it can return string 1, but if nothing gets matched the result is empty string "" and this empty string cannot be used in a way to analyze the data with Item is used.

There is plenty of discussions online for this weird behavior and many people do offer different approaches to solve the strange situation, but as we have tried with our colleagues sys admins none of those really worked out.

Thus we decided to use the classical way to work around, e.g. to simply use a powershell script that would check a number of lines inside a provided log file analyze if a string gets found and print out value of "1" if the string is matched or "0" "if not and this PS1 script to be set to run via a standard zabbix userparameter script.

This worked well, as all of us are mainly managing Linux systems, and we don't have enough knowledge on powershell we have used our internal Aartificial Intelligence (AI) clone tool to LibreChat – A free and open source ChatGPT clone.

LibreChat includes OpenAI's models, but also others — both open-source and closed-source — and its website promises "seamless integration" with AI services from OpenAI, Azure, Anthropic, and Google — as well as GPT-4, Gemini Vision, and many others. ("Every AI in one place," explains LibreChat's home page.) Plugins even let you make requests to DALL-E or Stable Diffusion for image generations. (LibreChat also offers a database that tracks "conversation state" — making it possible to switch to a different AI model in mid-conversation…)

$logfile = "C:\path\to\your\logfile.log"
$searchString = "-1"

# Get the last 140 lines
$lines = Get-Content $logfile -Tail 140

# Filter lines containing the search string
$found = $lines | Where-Object { $_ -match [regex]::Escape($searchString) }

# Output found lines or 0 if none were found
if ($found) {
$found | ForEach-Object { $_ }
} else {
Write-Host 0
}

You can download and the return_zero_or_one-if-string-matches-in-log-powershell.ps1 script here

Tags: data, empty, file, found, generations, inside, Item, last, log, log files, make, Monitoring Health Checks, none, page, possible, requests, search string, simply, switch, tail, tool, zabbix
Posted in Educational, System Administration, Various, Windows | No Comments »

Improve haproxy logging with custom log-format for better readiability

Friday, April 12th, 2024

Haproxy logging is a very big topic, worthy of many articles, but unfortunately not enough is written on the topic, perhaps for the reason haproxy is free software and most people who use it doesn't follow the philosophy of free software sharing but want to keep, the acquired knowledge on the topic for their own and if possible in the capitalist world most of us live to use it for a Load Balancer haproxy consultancy, consultancy fee or in their daily job as system administrators (web and middleware) or cloud specialist etc. 🙂

Having a good haproxy logging is very important as you need to debug issues with backend machines or some other devices throwing traffic to the HA Proxy.
Thus it is important to build a haproxy logging in a way that it provides most important information and the information is as simple as possible, so everyone can understand what is in without much effort and same time it contains enough debug information, to help you if you want to use the output logs with Graylog filters or process data with some monitoring advanced tool as Prometheus etc.

In our effort to optimize the way haproxy logs via a configured handler that sends the haproxy output to logging handler configured to log through rsyslog, we have done some experiments with logging arguments and came up with few variants, that we liked. In that article the idea is I share this set of logging parameters with hope to help some other guy that starts with haproxy to build a good logging readable and easy to process with scripts log output from haproxy.

The criterias for a decent haproxy logging used are:

1. Log should be simple but not dumb
2. Should be concrete (and not too much complicated)
3. Should be easy to read for the novice and advanced sysadmin

Before starting, have to say that building the logging format seems tedious task but to make it fit your preference could take a lot of time, especially as logging parameters naming is hard to remember, thus the haproxy logging documentation log-format description table comes really handy:

Haproxy log-format paremeters ASCII table

Please refer to the table for log-format defined variables :

+---+------+-----------------------------------------------+-------------+
| R | var  | field name (8.2.2 and 8.2.3 for description)  | type        |
+---+------+-----------------------------------------------+-------------+
|   | %o   | special variable, apply flags on all next var |             |
+---+------+-----------------------------------------------+-------------+
|   | %B   | bytes_read           (from server to client)  | numeric     |
| H | %CC  | captured_request_cookie                       | string      |
| H | %CS  | captured_response_cookie                      | string      |
|   | %H   | hostname                                      | string      |
| H | %HM  | HTTP method (ex: POST)                        | string      |
| H | %HP  | HTTP request URI without query string (path)  | string      |
| H | %HQ  | HTTP request URI query string (ex: ?bar=baz)  | string      |
| H | %HU  | HTTP request URI (ex: /foo?bar=baz)           | string      |
| H | %HV  | HTTP version (ex: HTTP/1.0)                   | string      |
|   | %ID  | unique-id                                     | string      |
|   | %ST  | status_code                                   | numeric     |
|   | %T   | gmt_date_time                                 | date        |
|   | %Ta  | Active time of the request (from TR to end)   | numeric     |
|   | %Tc  | Tc                                            | numeric     |
|   | %Td  | Td = Tt - (Tq + Tw + Tc + Tr)                 | numeric     |
|   | %Tl  | local_date_time                               | date        |
|   | %Th  | connection handshake time (SSL, PROXY proto)  | numeric     |
| H | %Ti  | idle time before the HTTP request             | numeric     |
| H | %Tq  | Th + Ti + TR                                  | numeric     |
| H | %TR  | time to receive the full request from 1st byte| numeric     |
| H | %Tr  | Tr (response time)                            | numeric     |
|   | %Ts  | timestamp                                     | numeric     |
|   | %Tt  | Tt                                            | numeric     |
|   | %Tw  | Tw                                            | numeric     |
|   | %U   | bytes_uploaded       (from client to server)  | numeric     |
|   | %ac  | actconn                                       | numeric     |
|   | %b   | backend_name                                  | string      |
|   | %bc  | beconn      (backend concurrent connections)  | numeric     |
|   | %bi  | backend_source_ip       (connecting address)  | IP          |
|   | %bp  | backend_source_port     (connecting address)  | numeric     |
|   | %bq  | backend_queue                                 | numeric     |
|   | %ci  | client_ip                 (accepted address)  | IP          |
|   | %cp  | client_port               (accepted address)  | numeric     |
|   | %f   | frontend_name                                 | string      |
|   | %fc  | feconn     (frontend concurrent connections)  | numeric     |
|   | %fi  | frontend_ip              (accepting address)  | IP          |
|   | %fp  | frontend_port            (accepting address)  | numeric     |
|   | %ft  | frontend_name_transport ('~' suffix for SSL)  | string      |
|   | %lc  | frontend_log_counter                          | numeric     |
|   | %hr  | captured_request_headers default style        | string      |
|   | %hrl | captured_request_headers CLF style            | string list |
|   | %hs  | captured_response_headers default style       | string      |
|   | %hsl | captured_response_headers CLF style           | string list |
|   | %ms  | accept date milliseconds (left-padded with 0) | numeric     |
|   | %pid | PID                                           | numeric     |
| H | %r   | http_request                                  | string      |
|   | %rc  | retries                                       | numeric     |
|   | %rt  | request_counter (HTTP req or TCP session)     | numeric     |
|   | %s   | server_name                                   | string      |
|   | %sc  | srv_conn     (server concurrent connections)  | numeric     |
|   | %si  | server_IP                   (target address)  | IP          |
|   | %sp  | server_port                 (target address)  | numeric     |
|   | %sq  | srv_queue                                     | numeric     |
| S | %sslc| ssl_ciphers (ex: AES-SHA)                     | string      |
| S | %sslv| ssl_version (ex: TLSv1)                       | string      |
|   | %t   | date_time      (with millisecond resolution)  | date        |
| H | %tr  | date_time of HTTP request                     | date        |
| H | %trg | gmt_date_time of start of HTTP request        | date        |
| H | %trl | local_date_time of start of HTTP request      | date        |
|   | %ts  | termination_state                             | string      |
| H | %tsc | termination_state with cookie status          | string      |
+---+------+-----------------------------------------------+-------------+
R = Restrictions : H = mode http only ; S = SSL only

Our custom log-format built in order to fulfill our needs is as this:

log-format %ci:%cp\ %H\ [%t]\ [%f\ %fi:%fp]\ [%b/%s\ %si:%sp]\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%sq/%bq

Once you place the log-format as a default for all haproxy frontend / backends or for a custom defined ones, the output you will get when tailing the log is:

# tail -f /var/log/haproxy.log

Apr 5 21:47:19 10.42.73.83:23262 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:23.879] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME/bk_appserv3 10.75.226.88:61310] 1/0/55250 55 sD 4/2/1/0/0/0
Apr 5 21:48:14 10.42.73.83:57506 haproxy-fqdn-hostname.com [05/Apr/2024:21:47:18.925] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv1 10.35.242.134:61310] 1/0/55236 55 sD 4/2/1/0/0/0
Apr 5 21:49:09 10.42.73.83:46520 haproxy-fqdn-hostname.com [05/Apr/2024:21:48:13.956] [ft_FRONTEND_NAME 10.46.108.6:61310] [bk_BACKEND_NAME//bk_appserv2 10.75.226.89:61310] 1/0/55209 55 sD 4/2/1/0/0/0

If you don't care about extra space and logs being filled with more naming, another variant of above log-format, that makes it even more readable even for most novice sys admin or programmer would look like this:

log-format [%t]\ %H\ [IN_IP]\ %ci:%cp\ [FT_NAME]\ %f:%fp\ [FT_IP]\ %fi:%fp\ [BK_NAME]\ [%b/%s:%sp]\ [BK_IP]\ %si:%sp\ [TIME_WAIT]\ {%Tw/%Tc/%Tt}\ [CONN_STATE]\ {%B\ %ts}\ [STATUS]\ [%ac/%fc/%bc/%sc/%sq/%bq]

Once you apply the config test the haproxy.cfg to make sure no syntax errors during copy / paste from this page

haproxy-serv:~# haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

Next restart graceously haproxy

haproxy-serv:~# /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

Once you reload haproxy graceously without loosing the established connections in stead of restarting it completely via systemd sysctl restart haproxy:

2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.200.198.195:50714 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.012] [FrotnendProd 10.55.0.20:27800] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/1/0/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.189:54290 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.056] [FrotnendProd 10.55.0.20:27900] [BackendProd/<NOSRV> -:-] -1/-1/0 0 — 4/4/3/0/0/0
2024-04-05T21:46:03+02:00 localhost haproxy[1897731]: 193.100.193.190:26778 haproxy-fqdn-hostname.com [05/Apr/2024:21:46:03.134] [FrotnendProd 10.55.0.20:27900] [BackendProd/tsefas02s 10.35.242.134:27900] 1/-1/0 0 CC 4/4/3/0/0/0

Note that in that log localhost haproxy[pid] is written by rsyslog, you can filter it out by modifying rsyslogd configurations

The only problem with this log-format is not everyone wants to have to much repeating information pointer on which field is what, but I personally liked this one as well because using it even though occuping much more space, makes the log much easier to process with perl or python scripting for data visualize and very for programs that does data or even "big data" analysis.

Tags: about, address, admin, AES, ALL, and, another, are, article, better, connections, custom, data, fi, handy, haproxy, HTTP, Improve, logging, string
Posted in Haproxy, Linux, System Administration, Various | No Comments »

Must have software on freshly installed windows – Essential Software after fresh Windows install

Friday, March 18th, 2016

If you're into IT industry even if you don't like installing frequently Windows or you're completely Linux / BSD user, you will certainly have a lot of friends which will want help from you to re-install or fix their Windows 7 / 8 / 10 OS. At least this is the case with me every year, I'm kinda of obliged to install fresh windowses on new bought friends or relatives notebooks / desktop PCs.

Of course according to for whom the new Windows OS installed the preferrences of necessery software varies, however more or less there is sort of standard list of Windows Software which is used daily by most of Avarage Computer user, such as:

CD Mounting ISO software ( VirtualClone Drive )
CD Recording ( CDBurnerXP )
Archiving / Unarchiving (Zip Rar) Software ( 7-Zip, Winrar, PeaZip )
Office Software ( Apache Open Office – also known widely as LibreOffice )
Internet Browsers ( Mozilla Firefox, Opera, Google Chrome )
Antivirus ( Avira Antivirus or AVG or Kaspersky, Nod32, McAffee (however rarely install non-freeware AV, usually only for companies or organizations) )
Anti-Malware ( Malware Bytes, SpyBot – Search & Destroy )
Chat Audio / Video Conferencing ( Skype / Pidgin (ICQ Client) / Telegram (A free software multiple platform Chat messanger alternative to Skype) )
Video Player ( VLC (Video Lan Client), KMplayer, GOM Player, Quick time, K-lite Codecs )
Music Player ( Winamp (nowadays discontinued), foobar2000, Audacity )
Graphic Design / Graphic Editor / Web Design tools ( The Gimp, Inkscape, PhotoShop and CorelDraw )
Picture Viewer ( IrfanView, FastStone, Picasa )
Java ( Java 7 / 8 JRE and at some PCs where necessery JDK )
PDF Viewers / DJViewPDF ( Adobe Acrobat Reader, Foxit Reader, Sumatra PDF, WinDJView )
Flash Player ( Adobe Shockware Flash player, Adobe Air (install it only on some PCs) )
SSH Client ( MobaXterm (Home Edition), PuTTY (Free Telnet SSH Client) )
FTP Client ( WinSCP, FileZilla )
Programming Text Editor ( Notepad++, Scintilla (SciTE), GVIM, Eclipse (only on developers and programmers PCs) )
Virtualization Software ( VirtualBox, VMWare (install VMWare only on Admin PCs))
Remote Desktop and VNC connector program ( RealVNC, TeamViewer )
Bittorrent Client ( uTorrent or qBittorrent or Transmission )
E-Mail POP3 / Imap Client ( Mozilla Thunderbird )
Screenshotting Tool ( GreenShot )
Password Safe program ( Password Safe or KeePass )
Remote Backup and Storage (Cloud) synchronization ( Dropbox, GoogleDrive, OneDrive (Comes preinstalled on Win 8.1 and Windows 10 )
Tools
– UnixUtils – Installs following common Linux command package tools for Windows CLI

bc-1.05, bison-1.28, bzip2-1.0.2, diffutils-2.7, fileutils-3.16, findutils-4.1, flex-2.5.4, gawk-3.1.0
grep-2.4.2, gsar110, gzip-1.2.4, indent-2.2.9, jwhois-2.4.1, less-340, m4-1.4, make-3.78.1
patch-2.5, recode-3.6, rman-3.0.7, sed-3.02, shellutils-1.9.4, tar-1.12, textutils-2.1
unrar-3.00, wget-1.8.2, which-2.4

– (WinDirStat or SpaceSniffer – Tools that can show you which is the biggest files and directory inside a directory tree)
– WinGrep (Grep: Grep for Windows)
– Everest Home Edition ( Hardware System Information – Shows you what is the PC hardware )

Not to forget a good candidate from the list to install on new fresh windows Installation candidates are:

Winrar
PeaZIP
WinZip
GreenShot (to be able to easily screenshot stuff and save pictures locally and to the cloud)
AnyDesk (non free but very functional alternative to TeamViewer) to be able to remotely access remote PC
TightVNC
ITunes / Spotify (for people who have also iPhone smart phone)
DropBox or pCloud (to have some extra cloud free space)
FBReader (for those reading a lot of books in different formats)
Rufus – Rufus is an efficient and lightweight tool to create bootable USB drives. It helps you to create BIOS or UEFI bootable devices. It helps you to create Windows TO Go drives. It provides support for various disk, format, and partition.
Recuva is a data recovery software for Windows 10 (non free)
EaseUS (for specific backup / restore data purposes but unfortunately (non free)
For designers
Adobe Photoshop
Adobe Illustrator
f.lux – to control brightness of screen and potentially Save your eyes
ImDisk virtual Disk Driver
KeePass / PasswordSafe – to Securely store your passwords
Putty / MobaXterm / SecureCRT / mPutty (for system administrators and programmers that has to deal with Linux / UNIX)

I tend to install on New Windows installs and thus I have more or less systematized the process.

I try to usually stick to free software where possible for each of the above categories as a Free Software enthusiast and luckily nowadays there is a lot of non-priprietary or at least free as in beer software available out there.

For Windows sysadmins or College and other public institutions networks including multiple of Windows Computers which are not inside a domain and also for people in computer repair shops where daily dozens of windows pre-installs or a set of software Automatic updates are necessery make sure to take a look at Ninite

ninite-automate-windows-program-deploy-and-update-on-new-windows-os-openoffice-screenshot

As official website introduces Ninite:

Ninite – Install and Update All Your Programs at Once

Of course as Ninite is used by organizations as NASA, Harvard Medical School etc. it is likely the tool might reports your installed list of Windows software and various other Win PC statistical data to Ninite developers and most likely NSA, but this probably doesn't much matter as this is probably by the moment you choose to have installed a Windows OS on your PC.

ninite-choises-to-build-an-install-package-with-useful-essential-windows-software-screenshot

For Windows System Administrators managing small and middle sized network PCs that are not inside a Domain Controller, Ninite could definitely save hours and at cases even days of boring install and maintainance work. HP Enterprise or HP Inc. Employees or ex-employees would definitely love Ninite, because what Ninite does is pretty much like the well known HP Internal Tool PC COE.

Ninite could also prepare an installer containing multiple applications based on the choice on Ninite's website, so that's also a great thing especially if you need to deploy a different type of Users PCs (Scientific / Gamers / Working etc.)

Perhaps there are also other useful things to install on a new fresh Windows installations, if you're using something I'm missing let me know in comments.

Tags: Archiving Unarchiving Zip Rar Software, case, Chat, com, common, course, daily, data, directory, dozens, Eclipse, fix, free software, inside, installed, list, look, lot, matter, multiple, necessery, New Windows, notebooks, official, OS, Pc, possible, process, public institutions, remote desktop, software, sort, standard, system administrators, things, various, website, Windows, windows computers, windows software, www
Posted in Everyday Life, Remote System Administration, System Administration, Various, Windows | 1 Comment »

Debugging routing and network issues on Linux common approaches. A step by step guide to find out why routing or network service fails

Thursday, November 30th, 2023

For system administrators having a Network issue is among the Hell-ish stuff that can happen every now and then. That is especially true in Heterogenous / Hybrid and complicated Network topologies (with missing well crafted documentation), that were build without an initial overview "on the fly".
Such a networking connectivity or routing issues are faced by every novice, mid or even expert system administrators as the Company's Network IT environments are becoming more and more complicated day by day.

When the "Disaster" of being unable to connect two servers or at times home laptops / PCs to see each other even though on the Physical layer / Transport Layer (Hardware such as external Switches / Routers / Repeaters / Cabling etc.) is Present machines are connected and everything on the 1 Physical Layer from OSI layears is present happens, then it is time to Debug it with some software tools and methods.

To each operating system the tools and methods to test networking connection and routings is a bit different but generally speaking most concepts are pretty much the same across different types of operating systems (Linux ditros / OpenBSD / FreeBSD / Mac OS / Android / iOS / HP-UX / IBM AIX / DOS / Windows etc.).

Debugging network issues across separate operating systems has its variations but in this specific (ideas) are much close to this article. As the goal at that guide will be to point out how to debug network issues on Linux, in future if I have the time or need to debug other OS-es from Linux, I'll try to put an article on how to debug Network issues on Windows when have some time to do it.

Consider to look for the issue following the basic TCP / IP OSI Level model, every system administrator should have idea about it already, it is part of most basic networking courses such as Cisco's CCNA

1. Check what is the Link status of the Interface with ethtool

root@freak:~# ethtool eno1
Settings for eno1:
   Supported ports: [ TP ]
   Supported link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Supported pause frame use: Symmetric
   Supports auto-negotiation: Yes
   Supported FEC modes: Not reported
   Advertised link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Advertised pause frame use: Symmetric
   Advertised auto-negotiation: Yes
   Advertised FEC modes: Not reported
   Speed: 100Mb/s
   Duplex: Full
   Auto-negotiation: on
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   MDI-X: on (auto)
   Supports Wake-on: pumbg
   Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
   Link detected: yes

root@freak:~# ethtool eno2
Settings for eno2:
   Supported ports: [ TP ]
   Supported link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Supported pause frame use: Symmetric
   Supports auto-negotiation: Yes
   Supported FEC modes: Not reported
   Advertised link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Advertised pause frame use: Symmetric
   Advertised auto-negotiation: Yes
   Advertised FEC modes: Not reported
   Speed: 1000Mb/s
   Duplex: Full
   Auto-negotiation: on
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   MDI-X: on (auto)
   Supports Wake-on: pumbg
   Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
   Link detected: yes

For example lets check only if Cable of Network card is plugged in and detected to have a network connection to remote node or switch and show the connection speed on which the 'autoneg on' (autonegiation option) of the LAN card has detected the network exat maximum speed:

root@pcfreak:~# ethtool eth0|grep -i 'link detected'; ethtool eth0 |grep 'Speed: '
Link detected: yes
Speed: 100Mb/s

1. Check ip command network configuration output

root@freak:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
link/ether 70:e2:84:13:44:15 brd ff:ff:ff:ff:ff:ff
altname enp7s0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 1000
link/ether 70:e2:84:13:44:17 brd ff:ff:ff:ff:ff:ff
altname enp8s0
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 70:e2:84:13:44:13 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.7/24 brd 192.168.1.255 scope global dynamic xenbr0
valid_lft 7361188sec preferred_lft 7361188sec
5: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 70:e2:84:13:44:15 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.5/24 brd 192.168.0.255 scope global dynamic xenbr1
valid_lft 536138sec preferred_lft 536138sec
10: vif2.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: vif2.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
12: vif3.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
13: vif3.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
14: vif4.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
15: vif4.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
16: vif5.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
17: vif5.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
18: vif6.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
19: vif6.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
30: vif17.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
31: vif17.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
34: vif21.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
35: vif21.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
48: vif25.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
49: vif25.1-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr1 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
50: vif25.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
51: vif25.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr1 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
118: vif47.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
119: vif47.1-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr1 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
120: vif47.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
121: vif47.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr1 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
root@freak:~#

ip a s (is a also a shortcut command alias) you can enjoy if you have to deal with ip command frequently.

2. Check the status of the interfaces

Old fashioned way is to just do:

/sbin/ifconfig

root@freak:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:15 txqueuelen 1000 (Ethernet)
RX packets 52366502 bytes 10622469320 (9.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 242622195 bytes 274688121244 (255.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb200000-fb27ffff

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:17 txqueuelen 1000 (Ethernet)
RX packets 220995454 bytes 269698276095 (251.1 GiB)
RX errors 0 dropped 7 overruns 0 frame 0
TX packets 192319925 bytes 166233773782 (154.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb100000-fb17ffff

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 2553 bytes 147410 (143.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2553 bytes 147410 (143.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 14517375 bytes 133226551792 (124.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 139688950 bytes 145111993017 (135.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 86113294 bytes 156944058681 (146.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 181513904 bytes 267892940821 (249.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 1521875 bytes 88282472 (84.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152691174 bytes 278372314505 (259.2 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif2.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 454915 bytes 81069760 (77.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 266953989 bytes 425692364876 (396.4 GiB)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0

vif21.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 20043711 bytes 1283926794 (1.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 141580485 bytes 277396881113 (258.3 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif21.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 73004 bytes 3802174 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267151006 bytes 425621892663 (396.3 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif25.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 2736348 bytes 295661367 (281.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 260385509 bytes 265751226663 (247.5 GiB)
TX errors 0 dropped 200 overruns 0 carrier 0 collisions 0

vif25.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 145387 bytes 36011655 (34.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 370314760 bytes 394725961081 (367.6 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 55382861 bytes 130042280927 (121.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 99040097 bytes 147929196318 (137.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif3.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 5132631 bytes 295493762 (281.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 262314199 bytes 425416945203 (396.2 GiB)
TX errors 0 dropped 16 overruns 0 carrier 0 collisions 0

vif4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4902015 bytes 615387539 (586.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 149342891 bytes 277802504143 (258.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif4.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 276927 bytes 30720101 (29.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267132395 bytes 425745668273 (396.5 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif47.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 208745 bytes 20096596 (19.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 110905731 bytes 110723486135 (103.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 140517 bytes 14596061 (13.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 150831959 bytes 162931572456 (151.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 2030528 bytes 363988589 (347.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152264264 bytes 278131541781 (259.0 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif5.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4169244 bytes 1045889687 (997.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 263561100 bytes 424894400987 (395.7 GiB)
TX errors 0 dropped 7 overruns 0 carrier 0 collisions 0

vif6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 300242 bytes 16210963 (15.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 153909576 bytes 278461295620 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

vif6.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 43 bytes 1932 (1.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 154205631 bytes 278481298141 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.8 netmask 255.255.255.0 broadcast 192.168.1.255
ether 70:e2:84:13:44:11 txqueuelen 1000 (Ethernet)
RX packets 13689902 bytes 923464162 (880.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12072932 bytes 1307055530 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.255.0 broadcast 192.168.0.255
ether 70:e2:84:13:44:12 txqueuelen 1000 (Ethernet)
RX packets 626995 bytes 180026901 (171.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12815 bytes 942092 (920.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@freak:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:11 txqueuelen 1000 (Ethernet)
RX packets 52373358 bytes 10623034427 (9.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 242660000 bytes 274734018669 (255.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb200000-fb27ffff

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:12 txqueuelen 1000 (Ethernet)
RX packets 221197892 bytes 269978137472 (251.4 GiB)
RX errors 0 dropped 7 overruns 0 frame 0
TX packets 192573206 bytes 166491370299 (155.0 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb100000-fb17ffff

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 2553 bytes 147410 (143.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2553 bytes 147410 (143.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 14519247 bytes 133248290251 (124.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 139708738 bytes 145135168676 (135.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 86206104 bytes 157189755115 (146.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 181685983 bytes 268170806613 (249.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 1522072 bytes 88293701 (84.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152712638 bytes 278417240910 (259.2 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif2.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 454933 bytes 81071616 (77.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267218860 bytes 426217224334 (396.9 GiB)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0

vif21.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 20045530 bytes 1284038375 (1.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 141601066 bytes 277441739746 (258.3 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif21.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 73010 bytes 3802474 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267415889 bytes 426146753845 (396.8 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif25.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 2736576 bytes 295678097 (281.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 260429831 bytes 265797660906 (247.5 GiB)
TX errors 0 dropped 200 overruns 0 carrier 0 collisions 0

vif25.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 145425 bytes 36018716 (34.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 370770440 bytes 395263409640 (368.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 55392503 bytes 130064444520 (121.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 99052116 bytes 147951838129 (137.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif3.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 5133054 bytes 295517366 (281.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 262578665 bytes 425941777243 (396.6 GiB)
TX errors 0 dropped 16 overruns 0 carrier 0 collisions 0

vif4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4902949 bytes 615496460 (586.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 149363618 bytes 277847322538 (258.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif4.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 276943 bytes 30721141 (29.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267397268 bytes 426270528575 (396.9 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif47.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 208790 bytes 20100733 (19.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 110950236 bytes 110769932971 (103.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 140551 bytes 14599509 (13.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 151287643 bytes 163469024604 (152.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 2030676 bytes 363997181 (347.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152285777 bytes 278176471509 (259.0 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif5.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4169387 bytes 1045898303 (997.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 263825846 bytes 425419251935 (396.2 GiB)
TX errors 0 dropped 7 overruns 0 carrier 0 collisions 0

vif6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 300266 bytes 16212271 (15.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 153931212 bytes 278506234302 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

vif6.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 43 bytes 1932 (1.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 154227291 bytes 278526238467 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.8 netmask 255.255.255.0 broadcast 192.168.1.255
ether 70:e2:84:13:44:11 txqueuelen 1000 (Ethernet)
RX packets 13690768 bytes 923520126 (880.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12073667 bytes 1307127765 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.255.0 broadcast 192.168.0.255
ether 70:e2:84:13:44:12 txqueuelen 1000 (Ethernet)
RX packets 627010 bytes 180028847 (171.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12815 bytes 942092 (920.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

To see ethernet interfaces that seem up and then do a ifconfig -a to check whether some interfaces are down (e.g. not shown in the simple ifconfig list).
/sbin/ifconfig -a

! Please note that some virtual IP configurations might not appear and noly be visible in an (ip addr show) command.

3. Check iproute2 for special rt_tables (Routing Tables) rules

By default Linux distributions does not have any additional rules in /etc/iproute2/rt_tables however some Linux router machines, needs to have a multiple Gateways. Perhaps the most elegant way to do multiple routings with Linux is to use iproute2's routing tables rt_tables.

Here is example of an OpenXEN system that has 2 Internet providers attached and routes different traffic via

root@freak:~# cat /etc/iproute2/rt_tables
#
# reserved values
#
255   local
254   main
253   default

100   INET1
200 INET2
0   unspec
#
# local
#
#1   inr.ruhep

root@freak:~# ip rule list
0:   from all lookup local
32762:   from all to 192.168.1.8 lookup INET2
32763:   from 192.168.1.8 lookup INET2
32764:   from all to 192.168.0.3 lookup INET1
32765:   from 192.168.0.3 lookup INET1
32766:   from all lookup main
32767:   from all lookup default
root@freak:~#

4. Using ip route get to find out traffic route (path)

root@freak:~# ip route get 192.168.0.1
192.168.0.1 via 192.168.0.1 dev xenbr1 src 192.168.0.3 uid 0
cache

root@freak:~# /sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 xenbr0
192.168.0.0 192.168.0.1 255.255.255.0 UG 0 0 0 xenbr1
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr1
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr0
root@freak:~#

root@freak:~# ip route show
default via 192.168.1.1 dev xenbr0
192.168.0.0/24 via 192.168.0.1 dev xenbr1
192.168.0.0/24 dev xenbr1 proto kernel scope link src 192.168.0.3
192.168.1.0/24 dev xenbr0 proto kernel scope link src 192.168.1.8

If you find that gateway is missing you might want to add it with:

root@freak:~# ip route add default via 192.168.5.1

If you need to add a speicic network IP range via separate gateways, you can use commands like:

To add routing for 192.168.0.1/24 / 192.168.1.1/24 via 192.168.0.1 and 192.168.1.1

# /sbin/route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.1 dev eth1
# /sbin/route add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.0.1 dev eth1

If you need to delete a configured wrong route with ip command

# ip route del 192.168.1.0/24 via 0.0.0.0 dev eth1
# ip route del 192.168.0.0/24 via 0.0.0.0 dev eth1

5. Use ping (ICMP protocol) the Destionation IP

root@freak:~# ping -c 3 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.219 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.295 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.270 ms

— 192.168.0.1 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2048ms
rtt min/avg/max/mdev = 0.219/0.261/0.295/0.031 ms
root@freak:~# ping -c 3 192.168.0.39
PING 192.168.0.39 (192.168.0.39) 56(84) bytes of data.
From 192.168.1.80: icmp_seq=2 Redirect Host(New nexthop: 192.168.0.39)
From 192.168.1.80: icmp_seq=3 Redirect Host(New nexthop: 192.168.0.39)
From 192.168.1.80 icmp_seq=1 Destination Host Unreachable

— 192.168.0.39 ping statistics —
3 packets transmitted, 0 received, +1 errors, 100% packet loss, time 2039ms
pipe 3

Note that sometimes you might get 100% traffic loss but still have connection to the destionation in case if the ICMP protocol is filtered for security.

However if you get something like Network is unreachable that is usually an indicator of some routing problem or wrongly configured network netmask.

root@freak:~# ping 192.168.0.5
ping: connect: Network is unreachable

Test network with different packet size. To send 8972 bytes of payload in a Ethernet frame without fragmentation, the following command can be used:

root@pcfreak:~# ping -s 8972 -M do -c 4 freak
PING xen (192.168.1.8) 8972(9000) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
^C
— xen ping statistics —
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2037ms

root@pcfreak:~#

-M pmtudisc_opt
Select Path MTU Discovery strategy. pmtudisc_option may be either do (prohibit fragmentation, even local one), want (do PMTU discovery, fragment locally when packet size is
large), or dont (do not set DF flag).

root@pcfreak:~# ping -s 8972 -M want -c 4 freak
PING xen (192.168.1.8) 8972(9000) bytes of data.
8980 bytes from xen (192.168.1.5): icmp_seq=1 ttl=64 time=2.18 ms
8980 bytes from xen (192.168.1.5): icmp_seq=2 ttl=64 time=1.90 ms
8980 bytes from xen (192.168.1.5): icmp_seq=3 ttl=64 time=2.10 ms
^C
— xen ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.901/2.059/2.178/0.116 ms
root@pcfreak:~#

-M do: prohibit fragmentation
-s 8972 8972 bytes of data
ICMP header: 8 bytes
IP header: 20 bytes (usually, it can be higher)
8980 bytes of bytes is the IP payload

These commands can be used to capture for MTU (maximum transmition units) related issues between hosts that are preventing for hosts to properly send traffic between themselves.
A common issue for Linux hosts to be unable to see each other on the same network is caused by Jumbo Frames (MTU 9000) packets enabled on one of the sides and MTU of 1500 on the other side.
Thus it is always a good idea to thoroughully look up all configured MTUs for all LAN Devices on each server.

6. Check traceroute path to host

If there is no PING but ip route get shows routing is properly configured and the routes existing in the Linux machine routing tables, next step is to check the output of traceroute / tracepath / mtr

raceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets
1 pcfreak (192.168.0.1) 0.263 ms 0.166 ms 0.119 ms
root@freak:~# tracepath 192.168.1.1
1?: [LOCALHOST] pmtu 1500
1: vivacom-gigabit-router 0.925ms reached
1: vivacom-gigabit-router 0.835ms reached
Resume: pmtu 1500 hops 1 back 1

It might be useful to get a frequent output of the command (especially on Linux hosts) where mtr command is not installed with:

root@freak:~# watch -n 0.1 traceroute 192.168.0.1

root@freak:~# traceroute -4 google.com
traceroute to google.com (172.217.17.110), 30 hops max, 60 byte packets
1 vivacom-gigabit-router (192.168.1.1) 0.657 ms 1.280 ms 1.647 ms
2 213.91.190.130 (213.91.190.130) 7.983 ms 8.168 ms 8.097 ms
3 * * *
4 * * *
5 212-39-66-222.ip.btc-net.bg (212.39.66.222) 16.613 ms 16.336 ms 17.151 ms
6 * * *
7 142.251.92.65 (142.251.92.65) 18.808 ms 13.246 ms 209.85.254.242 (209.85.254.242) 15.541 ms
8 142.251.92.3 (142.251.92.3) 14.223 ms 142.251.227.251 (142.251.227.251) 14.507 ms 142.251.92.3 (142.251.92.3) 15.328 ms
9 ams15s29-in-f14.1e100.net (172.217.17.110) 14.097 ms 14.909 ms 142.251.242.230 (142.251.242.230) 13.481 ms
root@freak:~#

If you have MTR then you can get plenty of useful additional information such as the Network HOP name or the Country location of the HOP.

To get HOP name:

root@freak:~# mtr -z google.com

To get info on where (which Country) exactly network HOP is located physically:

root@freak:~# mtr -y 2 google.com

7. Check iptables INPUT / FORWARD / OUTPUT rules are messing with something

# iptables -L -n
…

# iptables -t nat -L -n
…

Ideally you would not have any firewall

# iptables -L -n

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

In case if something like firewalld is enabled as a default serviceto provide some modern Linux firewall as Ubuntu and Redhat / CentOS / Fedoras has it often turned on as a service stop and disable the service

# systemctl stop firewalld

# systemctl disable firewalld

8. Debug for any possible MAC address duplicates

root@pcfrxen:~# arp -an
? (192.168.1.33) at 00:16:3e:59:96:9e [ether] on eth0
? (192.168.1.1) at 18:45:93:c6:d8:00 [ether] on eth1
? (192.168.0.1) at 8c:89:a5:f2:e8:d9 [ether] on eth1
? (192.168.1.1) at 18:45:93:c6:d8:00 [ether] on eth0
? (192.168.1.11) at 7c:0a:3f:89:b6:fa [ether] on eth1
? (192.168.1.17) at <incomplete> on eth0
? (192.168.1.37) at 00:16:3e:ea:05:ce [ether] on eth0
? (192.168.1.80) at 8c:89:a5:f2:e7:d8 [ether] on eth0
? (192.168.1.11) at 7c:0a:3f:89:a5:fa [ether] on eth0
? (192.168.1.30) at 00:16:3e:bb:46:45 [ether] on eth1
? (192.168.0.210) at 00:16:3e:68:d9:55 [ether] on eth1
? (192.168.1.30) at 00:16:3e:bb:46:45 [ether] on eth0
? (192.168.1.18) at 00:16:3e:0d:40:05 [ether] on eth1
? (192.168.0.211) at 00:16:3e:4d:41:05 [ether] on eth1
? (192.168.1.35) at 00:16:3e:d1:8f:77 [ether] on eth0
? (192.168.1.18) at 00:16:3e:0d:43:05 [ether] on eth0
? (192.168.1.28) at 00:16:3e:04:12:1c [ether] on eth1
? (192.168.0.3) at 70:e2:84:13:43:12 [ether] on eth1
? (192.168.0.208) at 00:16:3e:51:de:9c [ether] on eth1
? (192.168.0.241) at 00:16:3e:0d:48:06 [ether] on eth1
? (192.168.1.28) at 00:16:3e:04:12:1c [ether] on eth0
? (192.168.1.33) at 00:16:3e:59:97:8e [ether] on eth1
? (192.168.0.241) at 00:16:3e:0d:45:06 [ether] on eth0
? (192.168.0.209) at 00:16:3e:5c:df:96 [ether] on eth1

root@pcfrxen:~# ip neigh show
192.168.1.33 dev eth0 lladdr 00:16:3e:59:96:9e REACHABLE
192.168.1.1 dev eth1 lladdr 18:45:93:c6:d8:00 STALE
192.168.0.1 dev eth1 lladdr 8c:89:a5:f2:e8:d9 REACHABLE
192.168.1.1 dev eth0 lladdr 18:45:93:c6:d9:01 REACHABLE
192.168.1.11 dev eth1 lladdr 7c:0a:3f:89:a6:fb STALE
192.168.1.17 dev eth0 FAILED
192.168.1.37 dev eth0 lladdr 00:16:3e:ea:06:ce STALE
192.168.1.80 dev eth0 lladdr 8c:89:a5:f2:e8:d9 REACHABLE
192.168.1.11 dev eth0 lladdr 7c:0a:3f:89:a7:fa STALE
192.168.1.30 dev eth1 lladdr 00:16:3e:bb:45:46 STALE
192.168.0.210 dev eth1 lladdr 00:16:3e:68:d8:56 REACHABLE
192.168.1.30 dev eth0 lladdr 00:16:3e:bb:45:46 STALE
192.168.1.18 dev eth1 lladdr 00:16:3e:0d:48:04 STALE
192.168.0.211 dev eth1 lladdr 00:16:3e:4d:40:04 STALE
192.168.1.35 dev eth0 lladdr 00:16:3e:d2:8f:76 STALE
192.168.1.18 dev eth0 lladdr 00:16:3e:0d:48:06 STALE
192.168.1.28 dev eth1 lladdr 00:16:3e:04:11:2c STALE
192.168.0.3 dev eth1 lladdr 70:e2:84:13:44:13 STALE
192.168.0.208 dev eth1 lladdr 00:16:3e:51:de:9c REACHABLE
192.168.0.241 dev eth1 lladdr 00:16:3e:0d:48:07 STALE
192.168.1.28 dev eth0 lladdr 00:16:3e:04:12:1c REACHABLE
192.168.1.33 dev eth1 lladdr 00:16:3e:59:96:9e STALE
192.168.0.241 dev eth0 lladdr 00:16:3e:0d:49:06 STALE
192.168.0.209 dev eth1 lladdr 00:16:3e:5c:dd:97 STALE
root@pcfrxen:~#

9. Check out with netstat / ss for any irregularities such as high amount of error of faulty ICMP / TCP / UDP network packs

For example check out the netstat network stack output

# netstat -s

root@pcfrxen:~# netstat -s
Ip:
Forwarding: 2
440044929 total packets received
1032 with invalid addresses
0 forwarded
0 incoming packets discarded
439988902 incoming packets delivered
396161852 requests sent out
3 outgoing packets dropped
100 dropped because of missing route
Icmp:
1025 ICMP messages received
540 input ICMP message failed
ICMP input histogram:
destination unreachable: 1014
timeout in transit: 11
519 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 519
IcmpMsg:
InType3: 1014
InType11: 11
OutType3: 519
Tcp:
1077237 active connection openings
1070510 passive connection openings
1398236 failed connection attempts
111345 connection resets received
83 connections established
438293250 segments received
508143650 segments sent out
42567 segments retransmitted
546 bad segments received
329039 resets sent
Udp:
1661295 packets received
278 packets to unknown port received
0 packet receive errors
1545720 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 33046
UdpLite:
TcpExt:
1 invalid SYN cookies received
1398196 resets received for embryonic SYN_RECV sockets
1737473 packets pruned from receive queue because of socket buffer overrun
1118775 TCP sockets finished time wait in fast timer
638 time wait sockets recycled by time stamp
656 packetes rejected in established connections because of timestamp
2218959 delayed acks sent
2330 delayed acks further delayed because of locked socket
Quick ack mode was activated 7172 times
271799723 packet headers predicted
14917420 acknowledgments not containing data payload received
171078735 predicted acknowledgments
52 times recovered from packet loss due to fast retransmit
TCPSackRecovery: 337
Detected reordering 1551 times using SACK
Detected reordering 1501 times using reno fast retransmit
Detected reordering 61 times using time stamp
9 congestion windows fully recovered without slow start
38 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 241
104 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 11550
1 timeouts after reno fast retransmit
TCPSackFailures: 13
3772 fast retransmits
2 retransmits in slow start
TCPTimeouts: 24104
TCPLossProbes: 101748
TCPLossProbeRecovery: 134
TCPSackRecoveryFail: 3
128989224 packets collapsed in receive queue due to low socket buffer
TCPBacklogCoalesce: 715034
TCPDSACKOldSent: 7168
TCPDSACKOfoSent: 341
TCPDSACKRecv: 16612
150689 connections reset due to unexpected data
27063 connections reset due to early user close
17 connections aborted due to timeout
TCPDSACKIgnoredOld: 158
TCPDSACKIgnoredNoUndo: 13514
TCPSpuriousRTOs: 9
TCPSackMerged: 1191
TCPSackShiftFallback: 1011
TCPDeferAcceptDrop: 699473
TCPRcvCoalesce: 3311764
TCPOFOQueue: 14289375
TCPOFOMerge: 356
TCPChallengeACK: 621
TCPSYNChallenge: 621
TCPSpuriousRtxHostQueues: 4
TCPAutoCorking: 1605205
TCPFromZeroWindowAdv: 132380
TCPToZeroWindowAdv: 132441
TCPWantZeroWindowAdv: 1445495
TCPSynRetrans: 23652
TCPOrigDataSent: 388992604
TCPHystartTrainDetect: 69089
TCPHystartTrainCwnd: 3264904
TCPHystartDelayDetect: 4
TCPHystartDelayCwnd: 128
TCPACKSkippedPAWS: 3
TCPACKSkippedSeq: 2001
TCPACKSkippedChallenge: 2
TCPWinProbe: 123043
TCPKeepAlive: 4389
TCPDelivered: 389507445
TCPAckCompressed: 7343781
TcpTimeoutRehash: 23311
TcpDuplicateDataRehash: 8
TCPDSACKRecvSegs: 17335
IpExt:
InMcastPkts: 145100
OutMcastPkts: 9429
InBcastPkts: 18226
InOctets: 722933727848
OutOctets: 759502627470
InMcastOctets: 58227095
OutMcastOctets: 3284379
InBcastOctets: 1756918
InNoECTPkts: 440286946
InECT0Pkts: 936

List all listening established connections to host

# netstat -ltne

List all UDP / TCP connections

# netstat -ltua

or if you prefer to do it with the newer and more comprehensive tool ss:

List all listening TCP connections

# ss -lt

List all listening UDP connections

# ss -ua

Display statistics about recent connections

root@pcfrxen:~# ss -s
Total: 329
TCP: 896 (estab 70, closed 769, orphaned 0, timewait 767)

Transport Total IP IPv6
RAW   0 0 0
UDP   40 36 4
TCP   127 118 9
INET   167 154 13
FRAG   0 0 0

If you need to debug some specific sport or dport filter out the connection you need by port number

# ss -at '( dport = :22 or sport = :22 )'

Debug for any possible issues with ICMP unreachable but ports reachable with NMAP / telnet / Netcat

# nc 192.168.0.1 -vz

root@pcfrxen:/ # nc 192.168.0.1 80 -vz
pcfreak [192.168.0.1] 80 (http) open

root@pcfrxen:/ # nc 192.168.0.1 5555 -vz
pcfreak [192.168.0.1] 5555 (?) : Connection refused

root@pcfrxen:/# telnet 192.168.0.1 3128
Trying 192.168.0.1…
Connected to 192.168.0.1.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

root@pcfrxen:/# nmap -sS -P0 192.168.0.1 -p 443 -O
Starting Nmap 7.80 ( https://nmap.org ) at 2023-11-27 19:51 EET
Nmap scan report for pcfreak (192.168.0.1)
Host is up (0.00036s latency).

PORT STATE SERVICE
443/tcp open https
MAC Address: 8C:89:A5:F2:E8:D8 (Micro-Star INT'L)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Aggressive OS guesses: Linux 3.11 (96%), Linux 3.1 (95%), Linux 3.2 (95%), AXIS 210A or 211 Network Camera (Linux 2.6.17) (94%), Linux 2.6.32 (94%), Linux 3.10 (94%), Linux 2.6.18 (93%), Linux 3.2 – 4.9 (93%), ASUS RT-N56U WAP (Linux 3.4) (93%), Linux 3.16 (93%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop

OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.24 seconds
root@pcfrxen:/#

10. Add static MAC address to Ethernet Interface (if you find a MAC address being wrongly assigned to interface)

Sometimes problems with network unrechability between hosts is caused by wrongly defined MAC addresses on a Switch that did not correspond correctly to the ones assigned on the Linux host.
The easiest resolution here if you don't have access to Switch in work environment is to reassign the default MAC addresses of interfaces to proper MAC addresses, expected by remote router.

root@pcfrxen:/#  /sbin/ifconfig eth2 hw ether 8c:89:a5:f2:e8:d6

root@pcfrxen:/#  /sbin/ifconfig eth1 hw ether 8c:89:a5:f2:e8:d5

root@pcfrxen:/#  ifconfig eth0|grep -i ether
ether 8c:89:a5:f2:e8:d6 txqueuelen 1000 (Ethernet)

11. Check for Network Address Translation (NAT) misconfigurations

If you do use some NAT-ing between Linux host and the remote Network Device you cannot reach, make sure IP Forwarding is enabled (i.e. /etc/sysctl.conf was not mistakenly overwritten by a script or admin for whatever reason).

root@server:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
root@server:~# sysctl net.ipv4.conf.all.forwarding
net.ipv4.conf.all.forwarding = 1

root@server:~# sysctl net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 0

12. Check for Resolving DNS irregularities with /etc/resolv.conf

If network connectivity is okay on TCP / IP , UDP Level but problems with DNS of course, check what you have configured inside /etc/resolv.conf

And if use newer Linux distributions and have resolving managed by systemd check status of resolvectl

root@server:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND — YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.

nameserver 127.0.0.1
search pc-freak.net
domain pc-freak.net
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 109.104.195.2
nameserver 109.104.195.1
nameserver 208.67.222.222
nameserver 208.67.220.220
options timeout:2 rotate
root@pcfreak:~#

root@server:~# resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (ens3)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.5.1
DNS Servers: 192.168.5.1

As seen see, the systemd-resolved service is used to provide domain names resolution and we can modify its configuration file /etc/systemd/resolved.conf to add the DNS server – the following line is set (two DNS servers’ addresses are added):

For example …

DNS=8.8.8.8

13. Fix problems with wrongly configured Network Speed between hosts

It is not uncommon to have a Switch between two Linux hosts that is set to communicate on a certain maximum amount of Speed but a Linux host is set to communicate or lesser or more of Speed, this might create network issues so in such cases make sure either you use the Auto Negitionation network feature
or set both sides to be communicating on the same amount of network speed.

To turn on auto negotiation for ether interface

# ethtool -s eth1 speed 1000 duplex full autoneg on

For example to set a Linux network interface to communicate on 1 Gigabit speed and switch off autonegotiation off.

# ethtool -s eth1 speed 1000 duplex full autoneg off

14. Check arp and icmp traffic with tcpdump

On both sides where the IPs can't see each other we can run a tcpdump to check the ARP and ICMP traffic flowing between the hosts.

# tcpdump -i eth1 arp or icmp

cpdump: verbose output suppressed, use -v[v]… for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:29:07.001841 IP freak-eth1 > pcfr_hware_local_ip: ICMP echo request, id 13348, seq 65, length 64
15:29:07.001887 IP pcfr_hware_local_ip > freak-eth1: ICMP echo reply, id 13348, seq 65, length 64
15:29:07.598413 ARP, Request who-has pcfr_hware_local_ip tell zabbix-server, length 46
15:29:07.598425 ARP, Reply pcfr_hware_local_ip is-at 8c:89:a5:f2:e8:d8 (oui Unknown), length 28
15:29:07.633055 ARP, Request who-has freak_vivacom_auto_assigned_dhcp_ip tell 192.168.1.1, length 46
15:29:08.025824 IP freak-eth1 > pcfr_hware_local_ip: ICMP echo request, id 13348, seq 66, length 64
15:29:08.025864 IP pcfr_hware_local_ip > freak-eth1: ICMP echo reply, id 13348, seq 66, length 64

# tcpdump -i eth1 -vvv
…

If you want to sniff for TCP protocol and specific port and look up for DATA transfered for SMTP you can use something like:

# tcpdump -nNxXi eth0 -s 0 proto TCP and port 25

If you need a bit more thorough explanation on what it would do check out my previous article How to catch / track mail server traffic abusers with tcpdump.

15. Debugging network bridge issues

Having bridge network interface is another brink where things could go totally wrong.
If you have network bridges configured, check out what is the status of the bridge.

root@freak:/etc/network# brctl show
bridge name   bridge id       STP enabled   interfaces
xenbr0       8000.70e284134411   yes       eno1
                           vif1.0
                           vif10.0
                           vif16.0
                           vif16.0-emu
                           vif2.0
                           vif3.0
                           vif4.0
                           vif5.0
                           vif6.0
                           vif9.0
                           vif9.0-emu
xenbr1       8000.70e284134412   yes       eno2
                           vif1.1
                           vif10.1
                           vif16.1
                           vif16.1-emu
                           vif2.1
                           vif3.1
                           vif4.1
                           vif5.1
                           vif6.1
                           vif9.1
                           vif9.1-emu

Check out any configurations such as /etc/sysconfig/network-scripts/ifcfg-* are not misconfigured if on Redhat / CentOS / Fedora.
Or if on Debian / Ubuntu and other deb based Linuxes look up for /etc/network/interfaces config problems that might be causing the bridge to misbehave.

For example one bridge network issue, I've experienced recently is related to bridge_ports variable configured as bridge_ports all.
This was causing the second bridge xenbr1 to be unable to see another local network that was directly connected with a cable to it.

The fix was bridge_ports none. Finding out this trivial issue caused by a restored network config from old backup took me days to debug.
As everything seemed on a network level to be perfect just like in Physical layer, same way and on Software level, routings were okay.
Checked everything multiple times and did not see anything irregular. ping was missing and hosts cannot see each other even though having the right netmask and
network configuration in place.

Below is my /etc/network/interfaces configuration with the correct bridge_ports none changed.

root@freak:/etc/network# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
allow-hotplug eno1
iface eno1 inet manual
dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220
auto eno2
allow-hotplug eno2
iface eno2 inet manual
dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220

auto xenbr0
allow-hotplug xenbr0
# Bridge setup
# fetching dhcp ip from 192.168.1.20 (vivacom fiber optics router) routing traffic via 1Gigabit network
iface xenbr0 inet dhcp
hwaddress ether 70:e2:84:13:44:11
# address 192.168.1.5/22
address 192.168.1.5
netmask 255.255.252.0
# address 192.168.1.8 if dhcp takes from vivacom dhcpd
bridge_ports eno1
gateway 192.168.1.20
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports none
dns-nameservers 8.8.8.8 8.8.4.4

auto xenbr1
# fetching dhcp ip from pc-freak.net (192.168.0.1) bergon.net routing traffic through it
allow-hotplug xenbr1
iface xenbr1 inet dhcp
hwaddress ether 70:e2:84:13:44:11
## address 192.168.0.3/22
address 192.168.0.8
netmask 255.255.252.0
# address 192.168.0.8 if dhcp takes from vivacom dhcpd (currently mac deleted from vivacom router)
# address 192.168.0.9 if dhcp takes from pc-freak.net hware host
# hwaddress ether 70:e2:84:13:44:13
gateway 192.168.0.1
bridge_ports eno2
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports none
dns-nameservers 8.8.8.8 8.8.4.4
root@freak:/etc/network#

root@freak:/etc/network# brctl showstp xenbr0
xenbr0
bridge id       8000.70e284134411
designated root   8000.70e284134411
root port       0           path cost       0
max age       20.00           bridge max age       20.00
hello time       2.00           bridge hello time   2.00
forward delay       15.00           bridge forward delay   15.00
ageing time       0.00
hello timer       1.31           tcn timer       0.00
topology change timer   0.00           gc timer       0.00
flags

eno1 (1)
port id       8001           state       forwarding
designated root   8000.70e284134411   path cost       19
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8001           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif1.0 (2)
port id       8002           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8002           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif10.0 (12)
port id       800c           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800c           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif16.0 (13)
port id       800d           state       disabled
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800d           forward delay timer   0.00
designated cost   0           hold timer       0.00
flags

vif16.0-emu (14)
port id       800e           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800e           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif2.0 (4)
port id       8004           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8004           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif3.0 (5)
port id       8005           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8005           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif4.0 (3)
port id       8003           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8003           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif5.0 (6)
port id       8006           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8006           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif6.0 (7)
port id       8007           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8007           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif9.0 (10)
port id       800a           state       disabled
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800a           forward delay timer   0.00
designated cost   0           hold timer       0.00
flags

vif9.0-emu (11)
port id       800b           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800b           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

root@freak:/etc/network#

Sum it up

We have learned how to debug various routing issues, how to add and remote default gateways, check network reachability with ICMP protocol with ping, traceroute as well check for DNS issues and given some hints how to resolve DNS misconfigurations.
We also learned how to check the configured Network interfaces certain settings and resolve issues caused by Network sides max Speed misconfigurations as well how to track and resolve communication issues caused by wrongly configured MAC addresses.
Further more learned on how to do a basic port and protocol debugging of state of Network packets with netstat and nc and check problems related to iptables Firewall and IP Forwarding misconfigurations.
Finally we learned some basic usage of tcpdump on how to track arp and MAC traffic and look up for a specific TCP / UDP protocol and its contained data.
There is certainly things this article is missing as the topic of debugging network connectivity issues on Linux is a whole ocean, especially as the complexity of Linux has grown dramatically these days.
I gues it is worthy to mention that unable to see remote network could be caused by wrong VLAN configurations on Linux or even buggy switches and router devices, due to hardware or software,
but I hope this article at least covers the very basics of network debugging and Linux.

Enjojy 🙂

Tags: article, broadcast, data, debugging, default, Display, fails, guide, histogram, ifconfig eth0, initial, internal, issues, link, linux?, loopback, mtr, MULTICAST, multiple times, network, network speed, Old, operating systems, ping, pipe, Present, running, RX, Settings, system administrators, tcp, time, Tx
Posted in Debian, Educational, Linux, Networking, System Administration | No Comments »

Enable PSK encryption on Zabbix Agent (client) sent encrypted monitored datas to Zabbix server

Friday, April 7th, 2023

zabbix-client-server-encryption-public-key-exchange

Those concerned of security and in use of their Zabbix monitored data who communicate Zabbix collected agent
data over internet or via some kind of untrusted network might definitely not enjoy the fact that zabbix-agent sents
its collected data to server in a plain text. Clear text data is allowing any network sniffer to possibly collect your
monitored server and hardware devices data and exposes all data sent over the network to same problems like in the past
the old uencrypted SMTP protocol.

To mitigate those great security hole for the paranoid sys admin it is rather easy to enable PSK (Pre Shared Key) based encryption.
To generate Pre Shared key you have to had to important values present

1. PSK Identity
2. PSK Secret

PSK secret should be minimum of 128 bit (16-byte PSK, entered as 32 hexadecimal digits), and supports up to
2048 bit (256-byte PSK, entered as 512 hexadecimal digits)

Usually something like 256 bit PSK secret on the machine should be strong enough and simply generated by running

# openssl rand -hex 32

1. Agent to zabbix server or proxy connection config

In /etc/zabbix/zabbix_agentd.conf for a Server Active (e.g. server to actively request the client to sent its collected data)
On machine running zabbix-agent should have a configuration similar to:

# cat /etc/zabbix/zabbix_agentd.conf

PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0

# IP of the machine
SourceIP=10.10.10.30
# turn it on if you need to execute to remote machine commands
EnableRemoteCommands=0

# IP of the server
servers=10.30.50.80
ListenPort=10080

# IP of the machine
ListenIP=10.30.30.31

# IP of the server
ServerActive=10.30.50.80

HostMetadataItem=system.uname
BufferSize=5400
MaxLinesPerSecond=5
Timeout=10
AllowRoot=0
StartAgents=5
LogRemoteCommands=0

# Machine hostname
Hostname=fqdn-of-zabbix-data-collect-server.com
Include=/etc/zabbix/zabbix_agentd.d/*.conf

# Encryption
TLSConnect=psk
TLSAccept=psk
TLSPSKIdentity=PSK to Zabbix Server5
TLSPSKFile=/etc/zabbix/zabbix_agentd.psk

! Important security note

!!! The TLSPSKIdentity value you decide will not be encrypted on transport, so don't use anything sensitive.

Once you include the TSL config

2 Generate / Create Zabbix Agent Key

Generate the key with pseudo-random bites inside /etc/zabbix/zabbix_agentd_key.psk

# cd /etc/zabbix
# openssl rand -hex 32 > zabbix_agentd_key.psk
# chown zabbix:zabbix zabbix_agentd_key.psk
# chmod 600 zabbix_agentd_key.psk

3. Configure PSK encryption in Zabbix Server Web User interface

Go to Zabbix Server User interface in browser and configure the PSK encryption options for the host.

Select the:

'Connections to host' = PSK

'Connections from host' = PSK

'PSK Identity' = [public-value-configured-in-Zabbix-agent-config]

'PSK' = [paste the long hex string generated from the OpenSSL command above]

In some seconds up to a minute or two the Zabbix Server and Agent will successfully communicate using PSK encryption.
Making the monitored data unreadable in plain text for malignant sniffers hanging in the middle equipment between the zabbix-agent and zabbix-server hosts.

4. PSK encryption behind a Proxy

Many companies, nowadays use zabbix proxy for improvement of network infrastrucutre. For example it is used to offload the zabbix-server when multiple zabbix-agents have to report various datas or to monitor servers and devices that are phyisically in separate networks or data centers (are passing through paranoic built firewalls) or monitor locations are having unreliable communications between each other.

To enable PSK for communications between your Zabbix Server and Zabbix Proxy.

1. Create a new secret, and add the PSK Identity and Secret to

Administration ⇾ Proxies ⇾ [Your proxy] ⇾ Encryption

2. Adjust the settings inside the zabbix proxies configuration file at /etc/zabbix/zabbix_proxy.conf

If setting up PSK encryption for agents behind a Zabbix proxy, ensure your have

Zabbix Server ⇽⇾ Proxy PSK enabled
first in Zabbix Server UI.

This is because, when you start the Proxy, or do some testing to send some key value to Zabbix server via the proxy with commands :

# zabbix_get -s 127.0.0.1 -k system.hostname
# zabbix_server -R config_cache_reload

config_cache_reload, the Proxy will download all its host settings from the server, and this also includes the servers copy of the secret.

The proxy needs to know the secret since it is now managing the communications on behalf of the server.

3. To add PSK encryption for any Agents behind a proxy, then you continue to set up the Agents as normal by creating a new secret, editing

Configuration ⇾ Hosts ⇾ [Your Host] ⇾ Encryption page

and also editing /etc/zabbix/zabbix_agentd.conf.

Remember that, since your Agents Host configuration in the Zabbix UI will be set as Monitored by Proxy, the PSK settings will be applicable for communications happening between the Zabbix Proxy and the Agent that it is monitoring, not between the Zabbix Server and the Agent behind the proxy.

You can also add PSK Encryption between your Zabbix Proxy and its own local Agent if you want.
You would set its PSK settings in the Proxy Agents host configuration at

Configuration ⇾ Hosts ⇾ [Your proxy] ⇾ Encryption

and modify the settings in the agents on configuration file at /etc/zabbix/zabbix_agentd.conf.
Keep in mind, this is only applicable to communications between the Zabbix Proxy, and its own Agent process.

When setting up PSK encryption for the Zabbix Server, Proxy and Agents, you may see an error in the Proxy logs,

cannot send proxy data to server at "zabbix.your-domain.tld": connection of type "TLS with PSK" is not allowed for proxy "your-proxy".

If you hit this, check that your

Zabbix Server ⇽⇾ Proxy PSK settings

are correct first.

Don't get confused between the Proxies own optional agent process, and its main Proxy process which is required.

Tags: configuration file, Configuration Hosts Encryption, data, Generate Create Zabbix Agent Key, hex, key value, machine, network, Proxy Agents, psk, Select, string, zabbix-agent, zabbix-proxy
Posted in Computer Security, Linux, Monitoring, System Administration, Zabbix | 1 Comment »

How to disable Windows pagefile.sys and hiberfil.sys to temporary or permamently save disk space if space is critically low

Monday, March 28th, 2022

Sometimes you have to work with Windows 7 / 8 / 10 PCs etc. that has a very small partition C:\
drive or othertimes due to whatever the disk got filled up with time and has only few megabytes left
and this totally broke up the windows performance as Windows OS becomes terribly sluggish and even
simple things as opening Internet Browser (Chrome / Firefox / Opera ) or Windows Explorer stones the PC performance.

You might of course try to use something like Spacesniffer tool (a great tool to find lost data space on PC s short description on it is found in my previous article how to
delete temporary Internet Files and Folders to to speed up and free disk space ) or use CCleaner to clean up a bit the pc.
Sometimes this is not enough though or it is not possible to do at all the main
partition disk C:\ is anyhow too much low (only 30-50MB are available on HDD) or the Physical or Virtual Machine containing the OS is filled with important data
and you couldn't risk to remove anything including Internet Temporary files, browsing cookies … whatever.

Lets say you are the fate chosen guy as sysadmin to face this uneasy situation and have no easy
way to add disk space from another present free space partition or could not add a new SATA hard drive
SSD drive, what should you do?

The solution wipe off pagefile.sys and hiberfil.sys

Usually every Windows installation has a pagefile.sys and hiberfil.sys.

pagefile.sys – is the default file that is used as a swap file, immediately once the machine runs out of memory. For Unix / Linux users better understanding pagefile.sys is the equivallent of Linux's swap partition. Of course as the pagefile is in a file and not in separate partition the swapping in Windows is perhaps generally worse than in Linux.
hiberfil.sys – is used to store data from the machine on machine Hibernation (for those who use the feature)

Pagefile.sys which depending on the configured RAM memory on the OS could takes up up to 5 – 8 GB, there hanging around doing nothing but just occupying space. Thus a temporary workaround that could free you some space even though it will degrade performance and on servers and production machines this is not a good solution on just user machines, where you temporary need to free space any other important task you can free up space
by seriously reducing the preconfigured default size of pagefile.sys (which usually is 1.5 times the active memory on the OS – hence if you have 4GB you would have a 6Gigabytes of pagefile.sys).

Other possibility especially on laptop and movable devices running Win OS is to disable hiberfil.sys, read below how this is done.

The temporary solution here is to simply free space by either reducing the pagefile.sys or completely disabling it

1. Disable pagefile.sys on Windows XP, Windows 7 / 8 / 10 / 11

The GUI interface to disable pagefile across all NT based Windows OS-es is quite similar, the only difference is newer versions of Windows has slightly more options.

1.1 Disable pagefile on Windows XP

Quickest way is to find pagefile.sys settings from GUI menus

1. Computer (My Computer) – right click mouse
2. Properties (System Preperties will appear)
3. Advanced (tab)
4. Settings
5. Advanced (tab)
6. Change button

windows-xp-pagefile-disable-screenshot

1.2 Disable pagefile on Windows 7

advanced-system-settings-control-panel-system-and-security-screenshot

windows-system-properties-screenshot-properties-advanced-change-Virtual-memory-pagefile-screenshot

system-properties-performance-options

Once applied you'll be required to reboot the PC

How-to-turn-off-Virtual-Memory-Paging_File-in-Windows-7-restart

1.3 Disable Increase / Decrease pagefile.sys on Windows 10 / Win 11

open-system-properties-advanced-win10

configure-virtual-memory-win-10-screenshot

1.4 Make Windows clear pagefile.sys on shutdown

On home PCs it might be useful thing to clear up ( nullify) pagefile.sys on shutdown, that could save you some disk space on every reboot, until file continuously grows to its configured Maximum.

Run

regedit

Modify registry key at location

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

windows-clean-up-pagefile-sys-file-on-shutdown-or-reboot-registry-editor-value-screenshot

You can apply the value also via a registry file you can get the Enable Clearpagefile at shutdown here .reg.

2. Manipulating pagefile.sys size and file delete from command line with wmic tool

For scripting purposes you might want to use the wmic pagefile which can do increase / decrase or delete the file without GUI, that is very helpful if you have to admin a Windows Domain (Active Directory)

[hipo.WINDOWS-PC] ➤ wmic pagefile /?

PAGEFILE – Virtual memory file swapping management.

HINT: BNF for Alias usage.
(<alias> [WMIObject] | [] | [] ) [].

USAGE:

PAGEFILE ASSOC []
PAGEFILE CREATE <assign list>
PAGEFILE DELETE
PAGEFILE GET [] []
PAGEFILE LIST [] []

[hipo.WINDOWS-PC] ➤ wmic pagefile
AllocatedBaseSize Caption CurrentUsage Description InstallDate Name PeakUsage Status TempPageFile
4709 C:\pagefile.sys 499 C:\pagefile.sys 20200912061902.938000+180 C:\pagefile.sys 525 FALSE

[hipo.WINDOWS-PC] ➤ wmic pagefile list /format:list

AllocatedBaseSize=4709
CurrentUsage=499
Description=C:\pagefile.sys
InstallDate=20200912061902.938000+180
Name=C:\pagefile.sys
PeakUsage=525
Status=
TempPageFile=FALSE

To change the Initial Size or Maximum Size of Pagefile use:

➤ wmic pagefileset where name="C:\\pagefile.sys" set InitialSize=2048,MaximumSize=2048

To move the pagefile / change location of pagefile to less occupied disk drive partition (i.e. D:\ drive)

Sometimes you might have multiple drives on the PC and some of them might be having multitudes of gigabytes while main drive C:\ could be fully occupied due to initial install bad drive organization, in that case a good work arount to save you space so you can work normally with the server is just to temporary or permanently move pagefile to another drive.

wmic pagefileset where name="D:\\pagefile.sys" set InitialSize=2048,MaximumSize=2048

!! CONSIDER !!!

That if you have the option to move the pagefile.sys for best performance it is advicable to place the file inside another physical disk, preferrably a Solid State Drive one, SATA disks are too slow and reduced Input / Output disk operations will lead to degraded performance, if there is lack of memory (i.e. pagefile.sys is actively open read and wrote in).

To delete pagefile.sys

➤ wmic pagefileset where name="C:\\pagefile.sys" delete

If for some reason you prefer to not use wmic but simple del command you can delete pagefile.sys also by:

Removing file default "Hidden" and "system" file attributes – set for security reasons as the file is a system file usually not touched by user. This will save you from "permission denied" errors:

➤ attrib -s -h %systemdrive%\pagefile.sys

Delete the file:

➤ del /a /q %systemdrive%\pagefile.sys

3. Disable hibernation on Windows 7 / 8 and Win 10 / 11

Disabling hibernation file hiberfil.sys can also free up some space, especially if the hibernation has been actively used before and the file is written with data. Of course, that is more common on notebooks.
Windows hibernation has significantly improved over time though i didn't have very pleasant experience in the past and I prefer to disable it just in case.

3.1 Disable Windows 7 / 8 / 10 / 11 hibernation from GUI

Disable it through:

Control Panel -> All Control Panel Items -> Power Options -> Edit Plan Settings -> Change advanced power settings

like shown in below screenshot:

3.2 Disable Windows 7 / 8 / 10 / 11 hibernation from command line

Disable hibernation Is done in the same way through the powercfg.exe command, to disable it
if you're cut of disk space and you want to save space from it:

run as Administrator in Command Line Windows (cmd.exe)

powercfg.exe /hibernate off

If you later need to switch on hibernation

powercfg.exe /hibernate on

3.3 Disable Windows hibernation on legacy Windows XP

On XP to disable hibernation open

1. Power Options Properties
2. Select Hibernate.
3. Select Enable Hibernation to clear the checkbox and disable Hibernation mode.
4. Select OK to apply the change.

Close the Power Options Properties box.

To sum it up

We have learned some basics on Windows swapping and hibernation and i've tried to give some insight on how thiese files if misconfigured could lead to degraded Win OS performance. In any case using SSD as of 2022 to store both files is a best practice for machines that has plenty of memory always try to completely disable / remove the files. It was shown how to manage pagefile.sys and hiberfil.sys across Windows Operating Systems different versions both from GUI and via command line as well as how you can configure pagefile.sys to be cleared up on pc reboot.

Tags: 8 and win 10, active directory, admin, administrator, BNF, cmd, data, disk space, drive, exe, free space, HINT, How to, partition, Pc, sys, sysadmin, temporary solution, Windows, workaround
Posted in System Administration, Windows | No Comments »

Monitoring chronyd time service is synchronized, get additional time server values with Zabbix userparameter script

Monday, March 21st, 2022

If you''re running a server infrastructure and your main monitoring system is Zabbix. Then a vital check you might want to setup is to monitor the server time synchronization to a central server. In newer Linux OS-es ntpd time server is started to be used lesser and many modern Linux distributions used in the corporate realm are starting to recommend using chrony as a time synchronization client / server.

In this article, I'll show you how you can quickly setup monitoring of chronyd process and monitoring whether the time is successfully synchronizing with remote Chronyd time server. This will be done with a tiny one liner shell script setup as userparameter It is relatively easy then to setup an Action Alert

1. Create userparameter script to send parsed chronyd time synchronization to Zabbix Server

chronyc tracking provides plenty of useful data which can give many details about info such as offset, skew, root delay, stratum, update interval.

[root@server: ~]# chronyc tracking
Reference ID : 0A32EF0B (fkf-intp01.intcs.meshcore.net)
Stratum : 3
Ref time (UTC) : Fri Mar 18 12:42:31 2022
System time : 0.000032544 seconds fast of NTP time
Last offset : +0.000031102 seconds
RMS offset : 0.000039914 seconds
Frequency : 3.037 ppm slow
Residual freq : +0.000 ppm
Skew : 0.023 ppm
Root delay : 0.017352410 seconds
Root dispersion : 0.004285847 seconds
Update interval : 1041.6 seconds
Leap status : Normal

[root@server zabbix_agentd.d]# cat userparameter_chrony.conf
UserParameter=chrony.json,chronyc -c tracking | sed -e s/'^'/'{"chrony":[“‘/g -e s/’$’/'”]}'/g -e s/','/'","'/g
[root@server zabbix_agentd.d]#

The -c option passed to chronyc is printing the chronyc tracking command ouput data in comma-separated values ( CSV ) format.

2. Create Necessery Item key to get chronyd processes and catch the userparameter data

First lets create a an Item key to calculate the chronyd daemon proc.num
proc.num – simply returns the number of processes in the process list just like a simple
pgrep servicename command does.

monitroing-chronyc_zabbix_item_report_to-zabbix

Second lets create the Item for the userparameter script, the chrony.json key should be the same as the key given in the userparameter script.

obtain-chronyc-statistic-variables-from-remote-chronyd-to-zabbix-windows

Create Chrony Zabbix Triggers

Expression

{server-host:proc.num[chronyd].last()}<1

will be triggered if the process of chronyd on the server is less than 1

chronyc_monitoring_process-is-not-running-screenshot

Next configure

{server-host:chrony[Leap status].iregexp[Not synchronised) ]=1

to trigger Alert Chronyd is Not synchronized if the Expression check occurs.

chronyd-is-not-synchronized-trigger-iregexp

Reload the zabbix-agent on the server

To make zabbix-agent locally installed on the machine read the userparameter into memory (in my case this is zabbix-agent-4.0.28-1.el8.x86_64) installed on Redhat 8.3 (Ootpa), you have to restart it.

[root@server: ~]# systemctl restart zabbix-agent
[root@server: ~]# systemctl status zabbix-agent

● zabbix-agent.service – Zabbix Agent
Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2021-12-16 16:41:02 CET; 3 months 0 days ago
Main PID: 862165 (zabbix_agentd)
Tasks: 6 (limit: 23662)
Memory: 20.6M
CGroup: /system.slice/zabbix-agent.service
├─862165 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
├─862166 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
├─862167 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
├─862168 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
├─862169 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
└─862170 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

In a short while you should be seeing in the chrony.json key History data fed by the userparameter Script.

In Zabbix Latest data, you will see plenty of interesting time synchronization data get reported such as Skew, Stratum, Root Delay, Update Interval, Frequency etc.

chronyd-zabbix-reported-time-synchronization-offset-leap-residual-freq-root-delay-screenshot

To have an email Alerting further, go and setup a new Zabbix Action based on the Trigger with your likings and you're done.
The tracked machine will be in zabbix to make sure your OS clock is not afar from the time server. Repeat the same steps if you need to track chronyd is up running and synchronized on few machines, or if you have to make it for dozens setup a Zabbix template.

Tags: alert, chrony, chronyc, chronyd, create, data, Frequency, Fri Mar, interval, Item, last, leap, Monitoring, plenty, Ref, root, system time, time synchronization, timedate, tracking, trigger, update, userparameter, zabbix
Posted in Linux, Zabbix | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘data’

How to convert .p12 ssl certificate to .pem with openssl command

Zabbix Power Shell PS1 script to write zero or one if string is matched inside log file

How to disable Windows pagefile.sys and hiberfil.sys to temporary or permamently save disk space if space is critically low

The solution wipe off pagefile.sys and hiberfil.sys

1. Disable pagefile.sys on Windows XP, Windows 7 / 8 / 10 / 11

1.1 Disable pagefile on Windows XP

1.2 Disable pagefile on Windows 7

1.3 Disable Increase / Decrease pagefile.sys on Windows 10 / Win 11

1.4 Make Windows clear pagefile.sys on shutdown

regedit

2. Manipulating pagefile.sys size and file delete from command line with wmic tool

3. Disable hibernation on Windows 7 / 8 and Win 10 / 11

3.1 Disable Windows 7 / 8 / 10 / 11 hibernation from GUI

3.2 Disable Windows 7 / 8 / 10 / 11 hibernation from command line

3.3 Disable Windows hibernation on legacy Windows XP

To sum it up

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About

About Myself

Links to Other Places

Links to Other Places

Recent Comments

Top Post Views

blogtopsites