running Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘running’

How to prevent /etc/resolv.conf to overwrite on every Linux boot. Make /etc/resolv.conf DNS records permanent forever

Tuesday, February 4th, 2025

how-to-make-prevent-etc-resolv.conf-to-ovewrite-on-every-linux-boot-make-etc-resolv-conf-permanent-forever

Have you recently been seriously bothered, after one of the updates from older to newer Debian / Ubuntu / CentOS or other Linux distributions by the fact /etc/resolv.conf has become a dynamic file that pretty much in the spirit of cloud technologies is being regenerated and ovewritten on each and every system (server) OS update / reboot and due to that you start getting some wrong inappropriate DNS records /etc/resolv.conf causing you harm to the server infrastructure?

During my set of server infra i have faced that odditty for some years now and i guess every system administrator out there has suffered at a point by having to migrate an older Linux release to a newer one, where something gets messed up with DNS resolving due to that Linux OS new feature of /etc/resolv.conf not being really static any more.

The Dynamic resolv.conf file for glibc resolver is often generated used to be regenerated by resolvconf command and consequentially can be tampered by dhcpd resolved systemd service as well perhaps other mechanism depending on how the different Linux distribution architects make it to behave …

There are more than one ways to stop the annoying /etc/resolv.conf ovewritten behavior

1. Using dhcpd to stop /etc/resolv.conf being overwritten

Using dhcpd either a small null up script can be used or a separate hook script.

The null script would look like this

root@pcfreak:/root# vim /etc/dhcp/dhclient-enter-hooks.d/nodnsupdate

#!/bin/sh
make_resolv_conf() {
:
}

root@pcfreak:/root# chmod +x /etc/dhcp/dhclient-enter-hooks.d/nodnsupdate

This script overrides an internal function called make_resolv_conf() that would normally overwrite resolv.conf and instead does nothing.

On old Ubuntu s and Debian versions this should work.

Alternative method is to use a small hook dhcp script like this:

root@pcfreak:/root# echo 'make_resolv_conf() { :; }' > /etc/dhcp/dhclient-enter-hooks.d/leave_my_resolv_conf_alone
chmod 755 /etc/dhcp/dhclient-enter-hooks.d/leave_my_resolv_conf_alone

Next boot when dhclient runs onreboot or when you manually run sudo ifdown -a ; sudo ifup -a ,
it loads this script nodnsupdate or the hook script and hopefully your manually configured values of /etc/resolv.conf would not mess up your file anymore.

2. Use a chattr and set immutable flag attribute to /etc/resolv.conf to prevent re-boot to ovewrite it

Anyways the universal and simple way "hack" to prevent /etc/resolv.conf many prefer to use instead of dhcp (especially as not everyone is running a dhcp on a server) , to overwrite is to delete the file and make it immutable with chattr (assuming chattr is supported by the filesystem i.e. EXT3 / EXT4 / XFS , you use on the Linux.).

You might need to check the filesystem type, before using chattr.

root@pcfreak:/root# blkid | awk '{print $1 ,$3, $4}'
/dev/xvda1: TYPE="xfs"
/dev/xvda2: TYPE="LVM2_member"
/dev/mapper/centos-root: TYPE="xfs"
/dev/mapper/centos-swap: TYPE="swap"
/dev/loop0:
/dev/loop1:
/dev/loop2:

Normally EXT fs and XFS support it, note that this is not going to be the case with a network filesystem like NFS.

If you have some weird Filesystem type and you try to chattr you will get error like:

chattr: Inappropriate ioctl for device while reading flags on /etc/resolv.conf

To make /etc/resolv.conf file unchangeable on next boot by dhcpd or systemd-resolved

a systemd service that provides network name resolution to local applications via a D-Bus interface, the resolve NSS service (nss-resolve)

root@pcfreak:/root# rm -f /etc/resolv.conf
{ echo "nameserver 1.1.1.1";
echo "nameserver 1.0.0.1;
echo "search mydomain.com"; } > /etc/resolv.conf
chattr +i /etc/resolv.conf
reboot

Also it is a good think if you don't plan after some update to have unexpected results caused by systemd-resolved doing something strange is to rename to /etc/systemd/resolved.conf.dpkg-bak or completely remove file

/etc/systemd/resolved.conf

To prevent dhcpd to overwrite the server /etc/resolv.conf from something automatically taken from preconfigured central DNS inside the network configurations made from /etc/network/interfaces configurations such as:

dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220

You need to change the DHCP configuration file named dhclient.conf and use the supersede option.
To so Edit /etc/dhcp/dhclient.conf.

Look for lines like these:

#supersede domain-name "fugue.com home.vix.com";
#prepend domain-name-servers 127.0.0.1;

Remove the preceding “#” comment and use the domain-name and/or domain-name-servers which you want (your DNS FQDN). Save and hopefully the DNS related ovewrite to /etc/resolv.conf would be stopped, e.g. changes inside /etc/resolv.conf mnually done should stay permanent.

Also it is a good practice to disable ddns-update-style direcive inside /etc/dhcp/dhcpd.conf

root@pcfreak:/root# vim /etc/dhcp/dhcpd.conf
##ddns-update-style none;

However on many newer Debian Linux as of 2025 and its .deb based derivative distros, you have to consider the /etc/resolv.conf is a symlink to another file /etc/resolvconf/run/resolv.conf

If that is the case with you then you'll have to set the immutable chattr attribute flag like so

root@pcfreak:~# chattr -V +i /etc/resolvconf/run/resolv.conf
chattr 1.47.0 (5-Feb-2023)
Flags of /etc/resolvconf/run/resolv.conf set as —-i—————–

root@pcfreak:/root# lsattr /etc/resolvconf/run/resolv.conf
—-i—————– /etc/resolvconf/run/resolv.conf

3. Make /etc/resolv.conf permanent with simple custom a rc.local boot triggered resolv.conf ovewrite from a resolv.conf_actual template file

Consider that due to the increasing complexity of how Linux based OS-es behaves and the fact the Linux is more and more written to fit integration into the Cloud and be as easy as possible to containerize or orchestrate (with lets say docker or some cloud PODs) and other multitude of OS virtualiozation stuff modernities /etc/resolv.conf might still continue to ovewrite ! 🙂

Thus I've come up with my very own unique and KISS (Keep it Simple Stupid) method to make sure /etc/resolv.conf is kept permanent and ovewritten on every boot for that "hack" trick you only need to have the good old /etc/rc.local enabled – i have written a short article how it can be enabled on newer debian / ubuntu / fedora / centos Linux here.

Prepare your permanent and static /etc/resolv.conf file containing your preferred server DNSes under a file /etc/resolv.conf_actual

Here is an example of one of my /etc/resolv.conf template files that gets ovweritten on each boot.

root@pcfreak:/root# cat /etc/resolv.conf_actual
domain pc-freak.net
search pc-freak.net
#nameserver 192.168.0.1

nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 212.39.90.42
nameserver 212.39.90.43
nameserver 208.67.222.222
nameserver 208.67.220.220
options timeout:2 rotate

And in /etc/rc.local place before the exit directive inside the file simple copy over the original /etc/resolv.conf file real location.

Before proceeding to add it to execute /etc/rc.local assure yourself file is being venerated by OS.

root@pcfreak:/etc/dhcp# systemctl status rc-local
● rc-local.service – /etc/rc.local Compatibility
Loaded: loaded (/etc/systemd/system/rc-local.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/rc-local.service.d
└─debian.conf
Active: active (exited) since Sun 2024-12-08 21:59:01 EET; 1 month 27 days ago
Docs: man:systemd-rc-local-generator(8)
Process: 1417 ExecStart=/etc/rc.local start (code=exited, status=0/SUCCESS)
CPU: 302ms

Notice: journal has been rotated since unit was started, output may be incomplete.

root@pcfreak:/root# vim /etc/rc.local

cp -rpf /etc/resolv.conf_actual /etc/resolvconf/run/resolv.conf

NB ! Make sure those line is placed before any exit 0 command in /etc/rc.local otherwise that won''t work

That's it folks 🙂
Using this simple trick you should be no longer bothered by a mysterious /etc/resolv.conf overwritten on next server reboot or system update (via a puppet / ansible or some other centralized update automation stuff) causing you a service or infrastructure outage.

Enjoy !

Tags: /etc/resolv.conf, debian linux, dev, filesystem type, How to, nameserver, pcfreak, root, running, script
Posted in dhcpd, DNS, Linux, Linux and FreeBSD Desktop, OS Update, System Administration | No Comments »

Enable automatic updates on CentOS 8 , CentOS 9 Stream Linux with dnf-automatic and Cockpit Web GUI package management tool

Wednesday, January 15th, 2025

Security for any OS is critical nowadays, thus as a CentOS legacy system admin at work or using CentOS Stream releases 8 and 9 that are to be around for the coming years

CentOS 8 and CentOS 9 Stream Lifecycle

CentOS Stream follows the same lifecycle as Red Hat Enterprise Linux. From version 8 onward this means every version is supported for 10 years, split into 5 years of Full Support and 5 years of maintenance support. Users also have the option to purchase an additional 3 years of Extended Life Cycle Support (ELS) as an add-on.

Version   General Availability   Full Support Ends   Maintenance Support Ends   Extended Life Cycle Support (ELS) Ends
8   May 7, 2019   May 31, 2024   May 31, 2029   May 31, 2032
9   May 18, 2022   May 31, 2027   May 31, 2032   May 31, 2035

In this article, you are going to learn how to enable automatic software updates on CentOS 8 and CentOS 9 ( Stream ) Linux OS-es. I'll show how to set up your system to download and apply security and other updates without user intervention.

It is really useful to use the CentOS automatic updates OS capability, turning on updates and instead typing all the time yum update && yum upgrade (and wasting time to observe the process) as it takes usually some 5 to 10 minutes to make the OS automatically install updates in the background and notify you once all is done so you can periodically check what the dnf-automatic automatic update tool has done that in most cases of success would save you at least few minutes per host. Automatic updates is critical especially if you have to maintain an infrastructure of CentOS virtual servers at version 8 or 9.

Those who use heavily used CentOS might have already enabled and used dnf-automatic, but I guess just like me until recently, most people using CentOS 8 don’t know how to enable and apply CentOS Linux updates automatically and those article might be helpful.

1. Enable Automatic CentOS 8 / 9 Updates Using DNF Automatic RPM Package

Install the DNF-automatic RPM package, it will provide a DNF component that enables start automatically the update process.
To install it on both CentOS 8 / 9.

[root@centos ~]# yum install dnf-automatic
CentOS Stream 9 – BaseOS 78 kB/s | 14 kB 00:00
CentOS Stream 9 – AppStream 28 kB/s | 15 kB 00:00
CentOS Stream 9 – Extras packages 81 kB/s | 18 kB 00:00
Dependencies resolved.
======================================================
Package Architecture Version Repository Size
======================================================
Installing:
dnf-automatic noarch 4.14.0-23.el9 baseos 33 k
Upgrading:
dnf noarch 4.14.0-23.el9 baseos 478 k
dnf-data noarch 4.14.0-23.el9 baseos 37 k
python3-dnf noarch 4.14.0-23.el9 baseos 461 k
yum noarch 4.14.0-23.el9 baseos 88 k

Transaction Summary
=======================================================
Install 1 Package
Upgrade 4 Packages

Total download size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): dnf-data-4.14.0-23.el9.noarch.rpm 556 kB/s | 37 kB 00:00
(2/5): dnf-automatic-4.14.0-23.el9.noarch.rpm 406 kB/s | 33 kB 00:00
(3/5): yum-4.14.0-23.el9.noarch.rpm 1.4 MB/s | 88 kB 00:00
(4/5): python3-dnf-4.14.0-23.el9.noarch.rpm 4.9 MB/s | 461 kB 00:00
(5/5): dnf-4.14.0-23.el9.noarch.rpm 2.6 MB/s | 478 kB 00:00
——————————————————————————————————
Total 1.1 MB/s | 1.1 MB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : dnf-data-4.14.0-23.el9.noarch 1/9
Upgrading : python3-dnf-4.14.0-23.el9.noarch 2/9
Upgrading : dnf-4.14.0-23.el9.noarch 3/9
Running scriptlet: dnf-4.14.0-23.el9.noarch 3/9
Installing : dnf-automatic-4.14.0-23.el9.noarch 4/9
Running scriptlet: dnf-automatic-4.14.0-23.el9.noarch 4/9
Upgrading : yum-4.14.0-23.el9.noarch 5/9
Cleanup : yum-4.14.0-9.el9.noarch 6/9
Running scriptlet: dnf-4.14.0-9.el9.noarch 7/9
Cleanup : dnf-4.14.0-9.el9.noarch 7/9
Running scriptlet: dnf-4.14.0-9.el9.noarch 7/9
Cleanup : python3-dnf-4.14.0-9.el9.noarch 8/9
Cleanup : dnf-data-4.14.0-9.el9.noarch 9/9
Running scriptlet: dnf-data-4.14.0-9.el9.noarch 9/9
Verifying : dnf-automatic-4.14.0-23.el9.noarch 1/9
Verifying : dnf-4.14.0-23.el9.noarch 2/9
Verifying : dnf-4.14.0-9.el9.noarch 3/9
Verifying : dnf-data-4.14.0-23.el9.noarch 4/9
Verifying : dnf-data-4.14.0-9.el9.noarch 5/9
Verifying : python3-dnf-4.14.0-23.el9.noarch 6/9
Verifying : python3-dnf-4.14.0-9.el9.noarch 7/9
Verifying : yum-4.14.0-23.el9.noarch 8/9
Verifying : yum-4.14.0-9.el9.noarch 9/9

Upgraded:
dnf-4.14.0-23.el9.noarch dnf-data-4.14.0-23.el9.noarch python3-dnf-4.14.0-23.el9.noarch yum-4.14.0-23.el9.noarch
Installed:
dnf-automatic-4.14.0-23.el9.noarch

Complete!
[root@centos ~]#

Here is info on what dnf-automatic package will do:

[root@centos ~]# rpm -qi dnf-automatic
Name : dnf-automatic
Version : 4.14.0
Release : 23.el9
Architecture: noarch
Install Date: Wed 15 Jan 2025 08:00:47 AM -03
Group : Unspecified
Size : 57937
License : GPLv2+
Signature : RSA/SHA256, Thu 02 Jan 2025 01:19:43 PM -03, Key ID 05b555b38483c65d
Source RPM : dnf-4.14.0-23.el9.src.rpm
Build Date : Thu 12 Dec 2024 07:30:24 AM -03
Build Host : s390-08.stream.rdu2.redhat.com
Packager : builder@centos.org
Vendor : CentOS
URL : https://github.com/rpm-software-management/dnf
Summary : Package manager – automated upgrades
Description :
Systemd units that can periodically download package upgrades and apply them.

Next up is configuring the dnf-automatic updates. The configuration file is located at /etc/dnf/automatic.conf. Once you have opened the file, you can to set the required values to fit your software requirements.
The values you might want to modify are as so:

[root@centos ~]# grep -v \# /etc/dnf/automatic.conf|sed '/^$/d'
[commands]
upgrade_type = default
random_sleep = 0
network_online_timeout = 60
download_updates = yes
apply_updates = no
reboot = never
reboot_command = "shutdown -r +5 'Rebooting after applying package updates'"
[emitters]
emit_via = stdio
[email]
email_from = root@example.com
email_to = root
email_host = localhost
[command]
[command_email]
email_from = root@example.com
email_to = root
[base]
debuglevel = 1
[root@centos ~]#

The most important things you need to tune in automatic.conf are:

[root@centos ~]# vim /etc/dnf/automatic.conf

apply_updates = no

should be changed to yes

apply_updates = yes

for automatic updates to start by dnf-automatic service

It is nice to set the email server to use configuration values, as well as email from, email to and the way for
email to be set emit_via = stdio is default (check out the other options if to be used inside the commented lines)

Finally, you can now run dnf-automatic, execute the following command to schedule DNF automatic updates for your CentOS 8 machine.

[root@centos ~]# systemctl enable –now dnf-automatic.timer

The command above enables and starts the system timer. To check the status of the dnf-automatic service, run the following.

[root@centos ~]# systemctl list-timers *dnf-*
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2025-01-15 09:31:52 -03 13min left – – dnf-makecache.timer dnf-makecache.service
Thu 2025-01-16 06:21:20 -03 21h left Wed 2025-01-15 08:09:20 -03 1h 8min ago dnf-automatic.timer dnf-automatic.service

2 timers listed.
Pass –all to see loaded but inactive timers, too.
[root@centos ~]#

Enable and Manage Automatic updates with Cockpit GUI web interface

Sooner or later even hard core sysadmins has to enter the 21 century and start using a Web interfaces for server or Desktop Linux management to offload your head for more important stuff.
Cockpit is a great tool to help you automatically manage and update your servers with no need to use the Linux console most of the time.

Cockpit is a very powerful tool you can use to manage remotely updates through a web interface, it is very handy tool for system admins as it gives you overview over updates and supports automatic updates and set RPM package management tasks through web-based console.
Cockpit allows updates over multiple servers and it makes it a kind of server orchestration tool that allows yo to update many same versioned operating system software.

If you haven't it already pre-installed in CentOS 8 / 9 depending on the type ofinstall you have done, you might need to install Cockpit.

To install cockpit

[root@centos ~]# yum install cockpit -y

To make the web service accessible in a browser you'll have to start it with cmds:

[root@centos ~]# systemctl start cockpit
[root@centos ~]# systemctl status cockpit

To access cockpit you'll either have to access it on https://localhost:9090 in case you need to access it locally via https://SERVER_IP:9090/.
Note that of course you will have to have firewalld enabling traffic to SERVER_IP PORT 9090.

By default cockpit will run with self signed certificate, if you need you can set up a certbot certificate or regenerate the self signed one for better managed security risk. For a first time if you haven't changed the certificate simply use the browser exclusion menu and login to Cockpit.

Once logged in you can check the available updates.

centos-steam-cockpit-web-gui-autoupdate-tool-linux-screenshot0

By default you will have to login with non-root account, preferably that should be an account who is authorized to become root via sudo.
To elevate to administrative privileges while in cockpit clock on 'Administrative access' and grant cockpit your superuser privileges.

centos-steam-cockpit-web-gui-autoupdate-tool-linux-screenshot2

Once authorized you can run the upgrade and enojy a coffee or beer in the mean time 🙂

centos-steam-cockpit-web-gui-autoupdate-tool-linux-screenshot-update-ongoing

Among the useful cockpit options, is also the Terminal through which you can run commands like over a normal Web SSH service.

The 'Logs' section is also very useful as it shows you clearly synthesized information on failed services and modules, since last OS system boot.

To add and manage updates for multiple hosts use the 'Add new host' menu that is a expansion of the main machine on which cockpit runs.

centos-steam-cockpit-web-gui-autoupdate-tool-linux-automatic-updates-settings

In the next window, turn automatic updates ON. You can now select the type of updates you want (Apply All Updates or Apply Security Updates), the day and time you want the updates applied, and the server rebooted.

CentOS 9's cockpit even have support for the innovative Kernel live patching, so the machine kernel can be updated even Live and you can save the reboot after complete patching of OS including the kernel.

Note that you cannot set up automatic updates without rebooting the system. Therefore, make sure your server can be rebooted at the time you’ve selected for the updates.

Sum it up

In this post, we learned have learned how to set up automatic updates for your CentOS 8 / 9 Linux. There are two main stream ways you can do it.
1. By using DNF automatic updates tool.
By enabling DNF automatic updates on CentOS 8 Linux the machine updated is faster, seemless and frequent as compared to manual updates.

This protects the OS more about crackers cyber-attacks. Secondly for the more lazy admins or for better structuring of updates (if it has to be executed on multiple hosts), the Cockpit web console is available.

With Cockpit, it’s much easy to enable automatic updates as the GUI is self-explanatory graphical user interface (GUI) as opposed to the DNF automatic updates, which would waste you more time on CLI ( shell ).

Tags: CentOS, check, cmds, command, configuration file, dnf, noarch, package updates, rpm, running, Stream Lifecycle, Stream Linux, Users
Posted in CentOS, Linux, Linux and FreeBSD Desktop | 1 Comment »

Enable Debian Linux automatic updates to keep latest OS Patches / Security Up to Date

Monday, January 13th, 2025

Enable Debian Linux automatic updates to keep latest OS Patches / Security Up to Date

I'm not a big fan of automatism on GNU / Linux as often using automatic updates could totally mess things especially with a complex and a bit chatic OS-es like is Linux nowadays.
Nevertheless as Security is becoming more and more of a problem especially the browser security, having a scheduled way to apply updates like every normal modern Windows and MAC OS as an option is becoming essential to have a fully manageble Operating system.

As I use Debian GNU / Linux for desktop for my own personal computer and I have already a lot of Debian servers, whose OS minor level and package version maintenance takes up too big chunk of my time (a time I could dedicated to more useful activities). Thus I found it worthy at some cases to trigger Debian's way to keep the OS and security at a present level, the so called Debian "unattended upgrades".

In this article, I'll explain how to install and Enable Automatic (" Unattended " ) Updates on Debian, with the hope that other Debian users might start benefiting from it.

Pros of enabling automatic updates, are:

Debian OS Stay secure without constant monitoring.
You Save much time by letting your system handle updates.
Presumably Enjoying more peace of mind, knowing your system is more protected.

Cons of enabling automatic updates:

Some exotic and bad maintained packages (might break after the update)
Customizations made on the OS /etc/sysctl.conf or any other very custom server configs might disappear or not work after the update
At worst scenario (a very rare but possible case) OS might fail to boot after update 🙂

Regular security updates patch vulnerabilities that could otherwise be exploited by attackers, which is especially important for servers and systems exposed to the internet, where threats evolve constantly.

1. Update Debian System to latest

Before applying automatic updates making any changes, run apt to update package lists and upgrade any outdated packages,to have automatic updates for a smooth configuration process.

# apt update && apt upgrade -y

2. Install the Unattended-Upgrades deb Package

# apt install unattended-upgrades -y

Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following additional packages will be installed:
distro-info-data gir1.2-glib-2.0 iso-codes libgirepository-1.0-1 lsb-release python-apt-common python3-apt python3-dbus python3-distro-info python3-gi
Suggested packages:
isoquery python-apt-doc python-dbus-doc needrestart powermgmt-base
The following NEW packages will be installed:
distro-info-data gir1.2-glib-2.0 iso-codes libgirepository-1.0-1 lsb-release python-apt-common python3-apt python3-dbus python3-distro-info python3-gi unattended-upgrades
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,786 kB of archives.
After this operation, 24.4 MB of additional disk space will be used.
Do you want to continue? [Y/n]

# apt install apt-listchanges
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following package was automatically installed and is no longer required:
linux-image-5.10.0-30-amd64
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
python3-debconf
The following NEW packages will be installed:
apt-listchanges python3-debconf
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 137 kB of archives.
After this operation, 452 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian bookworm/main amd64 python3-debconf all 1.5.82 [3,980 B]
Get:2 http://deb.debian.org/debian bookworm/main amd64 apt-listchanges all 3.24 [133 kB]
Fetched 137 kB in 0s (292 kB/s)
Preconfiguring packages …
Deferring configuration of apt-listchanges until /usr/bin/python3
and python's debconf module are available
Selecting previously unselected package python3-debconf.
(Reading database … 84582 files and directories currently installed.)
Preparing to unpack …/python3-debconf_1.5.82_all.deb …
Unpacking python3-debconf (1.5.82) …
Selecting previously unselected package apt-listchanges.
Preparing to unpack …/apt-listchanges_3.24_all.deb …
Unpacking apt-listchanges (3.24) …
Setting up python3-debconf (1.5.82) …
Setting up apt-listchanges (3.24) …

Creating config file /etc/apt/listchanges.conf with new version

Example config for apt-listchanges would be like:

# vim /etc/apt/listchanges.conf
[apt]
frontend=pager
email_address=root
confirm=0
save_seen=/var/lib/apt/listchanges.db
which=both

3. Enable Automatic unattended upgrades

Once installed, enable automatic updates with the following command, which will prompt asking if you want to enable automatic updates. Select Yes and press Enter, which will confirm that the unattended-upgrades service is active and ready to manage updates for you.

# dpkg-reconfigure unattended-upgrades

Or non-interactively by running command:

# echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true | debconf-set-selections
dpkg-reconfigure -f noninteractive unattended-upgrades

4. Set the Schedule for Automatic Updates on Debian

By default, unattended-upgrades runs daily, to verify or modify the schedule, check the systemd timer:

# sudo systemctl status apt-daily.timer
# sudo systemctl status apt-daily-upgrade.timer
# systemctl edit apt-daily-upgrade.timer

Current apt-daily.timer config as of Debian 12 (bookworm) is as follows

root@haproxy2:/etc/apt/apt.conf.d# cat /lib/systemd/system/apt-daily.timer
[Unit]
Description=Daily apt download activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target
root@haproxy2:/etc/apt/apt.conf.d#

# systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
OnCalendar=03:00
RandomizedDelaySec=0

At Line num 2 above is needed to reset (empty) the default value shown below in line num 5.
Line 4 is needed to prevent any random delays coming from the defaults.

Now both timers should be active, if not, activate them with:

# systemctl enable –now apt-daily.timer
# systemctl enable –now apt-daily-upgrade.timer

These timers ensure that updates are checked and applied regularly, without manual intervention.

5.Test one time Automatic Updates on Debian works

To ensure everything is working, simulate an unattended upgrade with a dry run:

# unattended-upgrade –dry-run

You can monitor automatic updates by checking the logs.

# less /var/log/unattended-upgrades/unattended-upgrades.log

Log shows details of installed updates and any issues that occurred. Reviewing logs periodically can help you ensure that updates are being applied correctly and troubleshoot any problems.

6. Advanced Configuration Options

If you’re a power user or managing multiple systems, you might want to explore these additional settings in the configuration file:

# vim /etc/apt/apt.conf.d/50unattended-upgrades

Configure unattended-upgrades to send you an email whenever updates are installed.

Unattended-Upgrade::Mail "your-email-address@email-address.com";

Enable automatic reboots after kernel updates
by adding the line:

Unattended-Upgrade::Automatic-Reboot "true";

To schedule reboots after package upgrade is applied at a specific time:

Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Specify packages you don’t want to be updated by editing the Unattended-Upgrade::Package-Blacklist section in the configuration file.

Here is alternative way to configure the unattended upgrade, by using apt configuration options:

# vim /etc/apt/apt.conf.d/02periodic

// Control parameters for cron jobs by /etc/cron.daily/apt-compat //

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade –download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "21";

// Send report mail to root
// 0: no report (or null string)
// 1: progress report (actually any string)
// 2: + command outputs (remove -qq, remove 2>/dev/null, add -d)
// 3: + trace on
APT::Periodic::Verbose "2";

If you have to simultaneously update multiple machines and you're on a limited connection line, configure download limits if you’re on a metered connection by setting options in /etc/apt/apt.conf.d/20auto-upgrades.

7. Stop Automatic Unattended Upgrade

Under some circumstances if it happens the unattended upgrades are no longer required and you want to revert back to manual package updates, to disable the updates you have to disable the unattended-upgrades service

# systemctl stop unattended-upgrades

8. Stop an ongoing apt deb package set of updates applied on Debian server

Perhaps not often, but it might be you have run an automated upgrade and this has broke a server system or a service and for that reason you would like to stop the upcoming upgrade (some of whose might have started on other servers) immediately, to do so, the easiest way (not always safe thogh) is to kill the unattended-upgrades daemon.

# pkill –signal SIGKILL unattended-upgrades

Note that this a very brutal way to kill it and that might lead to some broken package update, that you might have to later fix manually.

If you have the unattended-upgrade process running on the OS in the process list backgrounded and you want to stop the being on the fly upgrade on the system more safely for the system, you can stop and cancel the ongoing apt upgrade it by running the ncurses prompt interface, through dpkg-reconfigure

# dpkg-reconfigure unattended-upgrades

Then just select No, press Enter. In my case, this has promptly stopped the ongoing unattended upgrade that seemed blocked (at least as promptly as the hardware seemed to allow 🙂 ).

If you want to disable it for future, so it doesn't automatically gets enabled on next manual update, by some update script disable service as well.

# systemctl disable unattended-upgrades
…

Close up

That’s all ! Now, your Debian system will automatically handle security updates, keeping your system secure without you having to do a thing.
The same guide should be good for most Deb based distributions such as Ubuntu / Mint and there rest of other Debian derivative OS-es.
You’ve now set up a reliable way to ensure your system stays protected from vulnerabilities, but anyways it is a good practice to always login and check what the update has done to the system, otherwise expect the unexpected.

Tags: Advanced Configuration Options, background, close, configure, dependency tree, Description Daily, disable, Done, Enable Debian Linux, installed, level, logs, MAC, packages, peace, possible, reading package, root, running, security, servers, setting, state information, sudo, system, systemctl, time, timer, updates, upgrade, without
Posted in Computer Security, Curious Facts, Linux, Linux and FreeBSD Desktop, Linux on Laptops, Various | No Comments »

Flush DNS on Mac OS X ,Windows and Linux from command line

Wednesday, January 8th, 2025

MAC OS X flush DNS cache

Open macOS /Mac OS X terminal application and type the following command under OS X version 10.5 and above:

# dscacheutil -flushcache

Mac OS X Tiger (v10.4) user, enter:

# lookupd -flushcache

macOS 11 Big Sur or 12 Monterey user, try:

# sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

macOS 10.11 El Capitan, 10.12 Sierra, 10.13 High Sierra, 10.14 Mojave, or 10.15 Catalina user try:

# sudo killall -HUP mDNSResponder

On Mac OS X version 10.6 and onwards to dump DNS cached records

$ dscacheutil -cachedump -entries

Display your DNS cached records on Windows

DNS cached records are stored to offload the DNS from querying it all the time in order to resolve the questionable site domain IP.

To view most recent cached DNS records on Windows use cmd:

c:\Windows\> ipconfig /displaydns

Windows IP Configuration

array805.prod.do.dsp.mp.microsoft.com
—————————————-
Record Name . . . . . : array805.prod.do.dsp.mp.microsoft.com
Record Type . . . . . : 1
Time To Live . . . . : 1894
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 52.143.124.236

telemetry-incoming.r53-2.services.mozilla.com
—————————————-
Record Name . . . . . : telemetry-incoming.r53-2.services.mozilla.com
Record Type . . . . . : 1
Time To Live . . . . : 444
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 34.120.208.123

tunnel.googlezip.net
—————————————-
Record Name . . . . . : tunnel.googlezip.net
Record Type . . . . . : 1
Time To Live . . . . : 36756
Data Length . . . . . : 4
…

Use up/down scroll array to see the cached record you need to view in the DNS cache.

Flush DNS Cache on Windows OS

Open the terminal (Start > Run > type cmd and press the [Enter] key) and type the following command to flush DNS on Win XP/2000/7/10 etc:

# ipconfig /flushdns

How to Flush DNS local cache queries on Linux

If you’re using a Linux distro such as Ubuntu, here’s how to flush the DNS cache:

Open the terminal by pressing Ctrl + Alt + T or searching for “Terminal” in the Activities menu.
Run the following command:
If you are running Ubuntu 20.04 LTS Linux or older Linux distro using systemd-resolve, use: sudo systemd-resolve –flush-caches.
If you are running Ubuntu 22.04 LTS or newer, use: sudo resolvectl flush-caches.
Enter your administrator password when prompted, and the DNS cache will be flushed.

To check the change in the DNS record is real, you can check the DNS cache statistics:

# resolvectl statistics

On older Linux distributions where /etc/resolv.conf nameserver directives are used to set the DNS for the server, to flush the cache
only options is to either flush the server used cache or change the nameserver inside the file.

Check the DNS cache records on older Linux distros (requires NSCD daemon)

Assuming the nscd daemon is UP and running (if not you have to install it) to check the cached local DNS records on Linux do

# service nscd status

Reviewing Your DNS Cache on modern Linux distros

The systemd-resolved daemon reacts to USR1 by writing its cache to the system logs. We can then use the journalctl command to filter out the DNS entries.

We'll use the killall command with USR1 To send the signal to the systemd-resolved daemon. Note that although we're using the killall command, the systemd-resolved daemon continues to run. This isn't a termination signal that we're sending.

$ sudo killall -USR1 systemd-resolved

Now we'll use the journalctl command with the -u (filter by systemd unit) option to extract the log entries that have been generated by systemd-resolved . We'll redirect that output into a text files called "dns.txt."

$ sudo journalctl -u systemd-resolved > dns.txt

Then check the DNS records

$ less dns.txt

Then find the cached mappings between domain names and IP addresses in.

How to Flush the dnsmasq Cache on Linux

The dnsmasq application provides a DNS cache and a DHCP server.

dnsmasq is popular among with users who want to run their own DNS server, especially on non-systemd installations.

Flushing the dnsmasq DNS cache is easy. We need to send the SIGHUP signal, which tells the dnsmasq daemon to effectively reinitialize. Doing so clears its DNS cache. To send the signal we use the killall command with the -HUP flag, and the name of the application.

$ sudo killall -HUP dnsmasq

Tags: /etc/resolv.conf, Big Sur, check, cmd, command, Display, enter, ipconfig flushdns, key, line, linux?, Open, running, Start Run, sudo, type, use, Windows
Posted in DNS, Linux, Mac OS X, Windows | No Comments »

How to install and use WSL 2 Windows native Linux emulation Debian and Ubuntu Linux on Windows 10 / Windows 11

Thursday, October 31st, 2024

WSL (Windows Subsystem for Linux) is perhaps relatively rarely known to the old school sys admins rats who usually use stuff like QEMU / KVM for Windows or Virtualbox / VMWare for Host machine.
However most people most lileky heard but never used or heard about the native (container like) virtualization WSL which was introduced in Windows 10 and Windows 11 as an attempt from Microsoft to improve the interoperability between Windows and Linux.
WSL version 1 and ver 2 allows Microsoft Windows for using a Linux environment without the need for a separate virtual machine.

In Windows 10, it is existing in Windows 10 Professional version can be installed either by joining the Windows Insider program or manually via Microsoft Store or Winget.
Hence perhaps you don't know that WSL virtualization can be used by those who want to mix Linux and Windows or for example get an advantages against dual-boot (installing Linux and Windows on the same computer).
Even better most significant WSL pros is you can literally running both systems at the same time without the need to run or stop every software that’s running and reboot to another system.

Procedure to set up a WSL is simple and similar to setting up a real Linux OS, therefore this guide can also be used as a reference to Linux setup.The specifications of WSL setup procedure are mainly in Install WSL and then setup any packages you would like to use for example if you want to be able to access remotely the WSL emulated Debian / Ubuntu or other of the installable distros via OpenSSH server.

1. Requirements to install and use WSL Linux emulation

To have the wsl subsystem used on Windows 10 or Windows 11 requirements:

You must be running Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11 to use the commands below. If you are on earlier versions please see the manual install page.

2. List available installable Linux distributions

WSL subsystem has ported only a certain set of Linux distributions, so if you need a very specific and unique Linux distribution, you would perhaps need to use Hyper-V virtualization or Virtualbox / VMWare.
However for people like me who are mainly using Debian GNU / Linux on daily basis as well as some OracleLinux admins / SUSE it is a perfect solution.

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –list –online
The following is a list of valid distributions that can be installed.
Install using 'wsl.exe –install <Distro>'.

NAME FRIENDLY NAME
Ubuntu Ubuntu
Debian Debian GNU/Linux
kali-linux Kali Linux Rolling
Ubuntu-18.04 Ubuntu 18.04 LTS
Ubuntu-20.04 Ubuntu 20.04 LTS
Ubuntu-22.04 Ubuntu 22.04 LTS
Ubuntu-24.04 Ubuntu 24.04 LTS
OracleLinux_7_9 Oracle Linux 7.9
OracleLinux_8_7 Oracle Linux 8.7
OracleLinux_9_1 Oracle Linux 9.1
openSUSE-Leap-15.6 openSUSE Leap 15.6
SUSE-Linux-Enterprise-15-SP5 SUSE Linux Enterprise 15 SP5
SUSE-Linux-Enterprise-15-SP6 SUSE Linux Enterprise 15 SP6
openSUSE-Tumbleweed openSUSE Tumbleweed

3. Install Linux distribution for a first time

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –install

The default Linux distribution that will get installed inside WLS Virtlualization is Ubuntu.

4. Install Debian GNU / Linux distribution as a second distro

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –install Debian

That second installed distro would make Debian now the default one to boot by WSL.

To run the fresh installed Debian GNU / Linux distribution, run only wsl command with no arguments.

# wsl

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –set-version Debian 2
For information on key differences with WSL 2 please visit https://aka.ms/wsl2
Conversion in progress, this may take a few minutes.
The distribution is already the requested version.
Error code: Wsl/Service/WSL_E_VM_MODE_INVALID_STATE
PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –set-version 2
There is no distribution with the supplied name.
Error code: Wsl/Service/WSL_E_DISTRO_NOT_FOUND
PS C:\Windows\System32\WindowsPowerShell\v1.0>

Simply pressting CTRL + D from the actively running WSL emulated Linux (that is pretty much like a native Windows docker container if we have to compare to Linux) would stop the VM.

5. List runnable / installed VM Linux distributions

To list the available runnable Linux VMs on your Windows status on Windows Subsystem for Linux:

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –list –verbose
NAME STATE VERSION
* Debian Stopped 2
Ubuntu Stopped 2
PS C:\Windows\System32\WindowsPowerShell\v1.0>

6. Run and check recent installed Linux distribution version

wsl2-windows-virtualization-install-virtual-machine-debian4

To run the newly install Debian Virtualized Linux (which as you can see is the default set distribution to run by WSL virtualization) simply type

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl

hipo@PC2LP3:/mnt/c/Windows/System32/WindowsPowerShell/v1.0$hipo@WL-2SLPWL3:/mnt/c/Windows/System32/WindowsPowerShell/v1.0$ cd ~
hipo@PC2LP3:~$

hipo@PC2LP3:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
hipo@WL-2SLPWL3:~$

7. Update the Debian distribuion packages to latest available

hipo@PC2LP3:~$ sudo su – root
hipo@PC2LP3:~# apt update –fix-missing

8. Install openssh server to be able to connect to the WSL hosted Virtual Machine

hipo@PC2LP3:/home/hipo# apt install openssh-server –yes
…

windows-wsl-linux-emulation

root@PC2LP3:/home/hipo# systemctl start openssh-server telnet
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
root@WL-2SLPWL3:/home/hipo# /etc/init.d/ssh start
Starting OpenBSD Secure Shell server: sshd.
root@WL-2SLPWL3:/home/hipo# ps -ef|grep -i ssh
root 30 9 0 18:19 ? 00:00:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
root 32 15 0 18:20 pts/1 00:00:00 grep -i ssh

windows-wsl-linux-emulation

By default a fresh new installed VM would have a process list like below:

root@PC2LP3:/home/hipo# ps axuwef

wsl2-windows-virtualization-install-virtual-machine-debian7

To be able to have ifconfig and a number of other network tools it is useful to install net-tools package

root@PC2LP3:/home/hipo# apt install net-tools –yes
…

root@PC2LP3:/home/hipo# /sbin/ifconfig
…

Once the WSL VM and OpenSSHD is run you can try to telnet or ssh to the VM locally or remotely.

root@PC2LP3:/home/hipo# telnet localhost 22
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u3

9. Run commands directly from Windows command line or Powershell

You can also use the powershell to run commands via the virtualized Linux environment using simple syntax

# wsl [cmd-to-run]

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl ls /
bin dev home lib lost+found mnt proc run srv tmp var
boot etc init lib64 media opt root sbin sys usr
PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 18:07 hvc0 00:00:00 /init
root 5 1 0 18:07 hvc0 00:00:00 plan9 –control-socket 5 –log-level 4 –server-fd 6 –pipe-fd 8 –log-t
root 8 1 0 18:07 ? 00:00:00 /init
root 9 8 0 18:07 ? 00:00:00 /init
hipo 10 9 0 18:07 pts/0 00:00:00 ps -ef
PS C:\Windows\System32\WindowsPowerShell\v1.0>

10. Enable systemd on Linux distribution in WSL 2

Once you boot into the WSL installed distro shell edit /etc/wsl.conf:

$ vim /etc/wsl.conf

[boot]
systemd=true

11. Setting extra useful variables to boot the WSL emulated Linux VM

root@debian-wsl:/home/hipo# cat /etc/wsl.conf
[boot]
systemd=true

# Automatically mount Windows drive when the distribution is launched
[automount]

# Set to true will automount fixed drives (C:/ or D:/) with DrvFs under the root directory set above. Set to false means drives won't be mounted automatically, but need to be mounted manually or with fstab.
enabled = true

# Sets the directory where fixed drives will be automatically mounted. This example changes the mount location, so your C-drive would be /c, rather than the default /mnt/c.
root = /

# DrvFs-specific options can be specified.
options = "metadata,uid=1003,gid=1003,umask=077,fmask=11,case=off"

# Sets the `/etc/fstab` file to be processed when a WSL distribution is launched.
mountFsTab = true

# Network host settings that enable the DNS server used by WSL 2. This example changes the hostname, sets generateHosts to false, preventing WSL from the default behavior of auto-generating /etc/hosts, and sets generateResolvConf to false, preventing WSL from auto-generating /etc/resolv.conf, so that you can create your own (ie. nameserver 1.1.1.1).
[network]
hostname = debian-wsl
generateHosts = true
generateResolvConf = true

# Set whether WSL supports interop processes like launching Windows apps and adding path variables. Setting these to false will block the launch of Windows processes and block adding $PATH environment variables.
[interop]
enabled = false
appendWindowsPath = false

# Set the user when launching a distribution with WSL.
[user]
default = hipo

# Set a command to run when a new WSL instance launches. This example starts the Docker container service.
#[boot]
#command = service docker start
root@debian-wsl:/home/hipo#

To learn about on Advanced settings configuration in WSL check out official Microsoft documentation here

12. Shutting down a running emulated Linux VM

If you have run a WSL VM and you want to shut it down do:

# wsl shutdown

If you at a point want to delete / uninstall the installed distribution you can do

# wsl –terminate Distro_Name
# wsl –uninstall Distro_Name

Or you if you want to do a cleanup of the stored files inside the installed distribution (if you have stored files), do:

# wsl –unregister Distro_Name

For more in depth details check out the manual

PS C:\Windows\System32\WindowsPowerShell\v1.0> wsl –help
Copyright (c) Microsoft Corporation. All rights reserved.
For privacy information about this product please visit https://aka.ms/privacy.

Usage: wsl.exe [Argument] [Options…] [CommandLine]

Arguments for running Linux binaries:

If no command line is provided, wsl.exe launches the default shell.

–exec, -e <CommandLine>
Execute the specified command without using the default Linux shell.

–shell-type <standard|login|none>
Execute the specified command with the provided shell type.

—
Pass the remaining command line as-is.

Options:
–cd <Directory>
Sets the specified directory as the current working directory.
If ~ is used the Linux user's home path will be used. If the path begins
with a / character, it will be interpreted as an absolute Linux path.
Otherwise, the value must be an absolute Windows path.

–distribution, -d <Distro>
Run the specified distribution.

–user, -u <UserName>
Run as the specified user.

–system
Launches a shell for the system distribution.

Arguments for managing Windows Subsystem for Linux:

–help
Display usage information.

–debug-shell
Open a WSL2 debug shell for diagnostics purposes.

–install [Distro] [Options…]
Install a Windows Subsystem for Linux distribution.
For a list of valid distributions, use 'wsl.exe –list –online'.

Options:
–no-launch, -n
Do not launch the distribution after install.

–web-download
Download the distribution from the internet instead of the Microsoft Store.

–no-distribution
Only install the required optional components, does not install a distribution.

–enable-wsl1
Enable WSL1 support.

–manage <Distro> <Options…>
Changes distro specific options.

Options:
–move <Location>
Move the distribution to a new location.

–set-sparse, -s <true|false>
Set the vhdx of distro to be sparse, allowing disk space to be automatically reclaimed.

–mount <Disk>
Attaches and mounts a physical or virtual disk in all WSL 2 distributions.

Options:
–vhd
Specifies that <Disk> refers to a virtual hard disk.

–bare
Attach the disk to WSL2, but don't mount it.

–name <Name>
Mount the disk using a custom name for the mountpoint.

–type <Type>
Filesystem to use when mounting a disk, if not specified defaults to ext4.

–options <Options>
Additional mount options.

–partition <Index>
Index of the partition to mount, if not specified defaults to the whole disk.

–set-default-version <Version>
Changes the default install version for new distributions.

–shutdown
Immediately terminates all running distributions and the WSL 2
lightweight utility virtual machine.

–status
Show the status of Windows Subsystem for Linux.

–unmount [Disk]
Unmounts and detaches a disk from all WSL2 distributions.
Unmounts and detaches all disks if called without argument.

–uninstall
Uninstalls the Windows Subsystem for Linux package from this machine.

–update
Update the Windows Subsystem for Linux package.

Options:
–pre-release
Download a pre-release version if available.

–version, -v
Display version information.

Arguments for managing distributions in Windows Subsystem for Linux:

–export <Distro> <FileName> [Options]
Exports the distribution to a tar file.
The filename can be – for stdout.

Options:
–vhd
Specifies that the distribution should be exported as a .vhdx file.

–import <Distro> <InstallLocation> <FileName> [Options]
Imports the specified tar file as a new distribution.
The filename can be – for stdin.

Options:
–version <Version>
Specifies the version to use for the new distribution.

–vhd
Specifies that the provided file is a .vhdx file, not a tar file.
This operation makes a copy of the .vhdx file at the specified install location.

–import-in-place <Distro> <FileName>
Imports the specified .vhdx file as a new distribution.
This virtual hard disk must be formatted with the ext4 filesystem type.

–list, -l [Options]
Lists distributions.

Options:
–all
List all distributions, including distributions that are
currently being installed or uninstalled.

–running
List only distributions that are currently running.

–quiet, -q
Only show distribution names.

–verbose, -v
Show detailed information about all distributions.

–online, -o
Displays a list of available distributions for install with 'wsl.exe –install'.

–set-default, -s <Distro>
Sets the distribution as the default.

–set-version <Distro> <Version>
Changes the version of the specified distribution.

–terminate, -t <Distro>
Terminates the specified distribution.

–unregister <Distro>
Unregisters the distribution and deletes the root filesystem.
PS C:\Windows\System32\WindowsPowerShell\v1.0>

Once wsl is installed you can run it directly from Windows start menu, by searching for the name of the distribution you would like to run for example to run my Debian WSL running emulator::

Sum it up

What was shown up is how to run in parallel virtualized Linux distribution on Windows 10 and Windows 11 and how to install update to latest and run opensshd server to be able to ssh into the WSL Linux virtual machine remotely.
.Also i've shown you, How to test ssh is reachable and how to stop / start or destroy and cleanup any stored files for VM if necessery, as well as how to apply some extra advanced configurations to boot VM for.

Using WSL is not the best virtualization ever but anyways it is an alternative for people employed in Domain attached Windows PCs part of Big Corporations, where VirtualBox use is blocked / prohibited and you still need to experiment or develop Shell scripts or software on Python / Perl / Ruby on Linux before you do stuff on the PreProd or Production Linux host.

That's all folks, Enjoy ! 🙂

Tags: about, access, Admins, com, Connected, Debian Ubuntu, Debian Virtualized Linux, distributions, emulator, Failed, How to, linux?, Lists, procedure, reference, running, setting, use, Windows, WSL
Posted in Debian, Educational, Linux, Various, Virtual Machines, Windows | 1 Comment »

How to run multiple processes in parallel with xargs

Wednesday, May 29th, 2024

In our company there is a legacy application which developers run in multiple consoles launching its in intedependent components together in different consoles by simply running each component, the question comes then how this can be scripted so no waste of people is done to manually run the different componets from different parallel consoles. To achive the run in parallel of the multiple programs in parallel and background it with xargs and eval (integrated bash command) in Linux from a single script you can use a simple one liner like in the example.

#run in parallel
xargs -P <n> allows you to run <n> commands in parallel.
time xargs -P 3 -I {} sh -c 'eval "$1"' – {} <<'EOF'
program1; sleep 1; echo 1
program2 ; sleep 2; echo 2
program3; sleep 3; echo 3
echo 4
EOF

You can attune the delay up to the exact requirements or completely remove it and the multi run script is ready, enjoy.

Tags: consoles, EOF, How to, linux?, multiple, running, script, simple, sleep, waste
Posted in Bash Scripting, Linux | No Comments »

Debugging routing and network issues on Linux common approaches. A step by step guide to find out why routing or network service fails

Thursday, November 30th, 2023

For system administrators having a Network issue is among the Hell-ish stuff that can happen every now and then. That is especially true in Heterogenous / Hybrid and complicated Network topologies (with missing well crafted documentation), that were build without an initial overview "on the fly".
Such a networking connectivity or routing issues are faced by every novice, mid or even expert system administrators as the Company's Network IT environments are becoming more and more complicated day by day.

When the "Disaster" of being unable to connect two servers or at times home laptops / PCs to see each other even though on the Physical layer / Transport Layer (Hardware such as external Switches / Routers / Repeaters / Cabling etc.) is Present machines are connected and everything on the 1 Physical Layer from OSI layears is present happens, then it is time to Debug it with some software tools and methods.

To each operating system the tools and methods to test networking connection and routings is a bit different but generally speaking most concepts are pretty much the same across different types of operating systems (Linux ditros / OpenBSD / FreeBSD / Mac OS / Android / iOS / HP-UX / IBM AIX / DOS / Windows etc.).

Debugging network issues across separate operating systems has its variations but in this specific (ideas) are much close to this article. As the goal at that guide will be to point out how to debug network issues on Linux, in future if I have the time or need to debug other OS-es from Linux, I'll try to put an article on how to debug Network issues on Windows when have some time to do it.

Consider to look for the issue following the basic TCP / IP OSI Level model, every system administrator should have idea about it already, it is part of most basic networking courses such as Cisco's CCNA

1. Check what is the Link status of the Interface with ethtool

root@freak:~# ethtool eno1
Settings for eno1:
   Supported ports: [ TP ]
   Supported link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Supported pause frame use: Symmetric
   Supports auto-negotiation: Yes
   Supported FEC modes: Not reported
   Advertised link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Advertised pause frame use: Symmetric
   Advertised auto-negotiation: Yes
   Advertised FEC modes: Not reported
   Speed: 100Mb/s
   Duplex: Full
   Auto-negotiation: on
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   MDI-X: on (auto)
   Supports Wake-on: pumbg
   Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
   Link detected: yes

root@freak:~# ethtool eno2
Settings for eno2:
   Supported ports: [ TP ]
   Supported link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Supported pause frame use: Symmetric
   Supports auto-negotiation: Yes
   Supported FEC modes: Not reported
   Advertised link modes: 10baseT/Half 10baseT/Full
   100baseT/Half 100baseT/Full
   1000baseT/Full
   Advertised pause frame use: Symmetric
   Advertised auto-negotiation: Yes
   Advertised FEC modes: Not reported
   Speed: 1000Mb/s
   Duplex: Full
   Auto-negotiation: on
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   MDI-X: on (auto)
   Supports Wake-on: pumbg
   Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
   Link detected: yes

For example lets check only if Cable of Network card is plugged in and detected to have a network connection to remote node or switch and show the connection speed on which the 'autoneg on' (autonegiation option) of the LAN card has detected the network exat maximum speed:

root@pcfreak:~# ethtool eth0|grep -i 'link detected'; ethtool eth0 |grep 'Speed: '
Link detected: yes
Speed: 100Mb/s

1. Check ip command network configuration output

root@freak:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
link/ether 70:e2:84:13:44:15 brd ff:ff:ff:ff:ff:ff
altname enp7s0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 1000
link/ether 70:e2:84:13:44:17 brd ff:ff:ff:ff:ff:ff
altname enp8s0
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 70:e2:84:13:44:13 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.7/24 brd 192.168.1.255 scope global dynamic xenbr0
valid_lft 7361188sec preferred_lft 7361188sec
5: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 70:e2:84:13:44:15 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.5/24 brd 192.168.0.255 scope global dynamic xenbr1
valid_lft 536138sec preferred_lft 536138sec
10: vif2.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: vif2.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
12: vif3.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
13: vif3.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
14: vif4.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
15: vif4.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
16: vif5.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
17: vif5.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
18: vif6.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
19: vif6.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
30: vif17.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
31: vif17.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
34: vif21.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
35: vif21.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
48: vif25.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
49: vif25.1-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr1 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
50: vif25.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
51: vif25.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr1 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
118: vif47.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
119: vif47.1-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr1 state UNKNOWN group default qlen 1000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
120: vif47.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
121: vif47.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr1 state DOWN group default qlen 2000
link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
root@freak:~#

ip a s (is a also a shortcut command alias) you can enjoy if you have to deal with ip command frequently.

2. Check the status of the interfaces

Old fashioned way is to just do:

/sbin/ifconfig

root@freak:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:15 txqueuelen 1000 (Ethernet)
RX packets 52366502 bytes 10622469320 (9.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 242622195 bytes 274688121244 (255.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb200000-fb27ffff

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:17 txqueuelen 1000 (Ethernet)
RX packets 220995454 bytes 269698276095 (251.1 GiB)
RX errors 0 dropped 7 overruns 0 frame 0
TX packets 192319925 bytes 166233773782 (154.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb100000-fb17ffff

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 2553 bytes 147410 (143.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2553 bytes 147410 (143.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 14517375 bytes 133226551792 (124.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 139688950 bytes 145111993017 (135.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 86113294 bytes 156944058681 (146.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 181513904 bytes 267892940821 (249.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 1521875 bytes 88282472 (84.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152691174 bytes 278372314505 (259.2 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif2.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 454915 bytes 81069760 (77.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 266953989 bytes 425692364876 (396.4 GiB)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0

vif21.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 20043711 bytes 1283926794 (1.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 141580485 bytes 277396881113 (258.3 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif21.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 73004 bytes 3802174 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267151006 bytes 425621892663 (396.3 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif25.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 2736348 bytes 295661367 (281.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 260385509 bytes 265751226663 (247.5 GiB)
TX errors 0 dropped 200 overruns 0 carrier 0 collisions 0

vif25.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 145387 bytes 36011655 (34.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 370314760 bytes 394725961081 (367.6 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 55382861 bytes 130042280927 (121.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 99040097 bytes 147929196318 (137.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif3.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 5132631 bytes 295493762 (281.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 262314199 bytes 425416945203 (396.2 GiB)
TX errors 0 dropped 16 overruns 0 carrier 0 collisions 0

vif4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4902015 bytes 615387539 (586.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 149342891 bytes 277802504143 (258.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif4.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 276927 bytes 30720101 (29.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267132395 bytes 425745668273 (396.5 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif47.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 208745 bytes 20096596 (19.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 110905731 bytes 110723486135 (103.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 140517 bytes 14596061 (13.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 150831959 bytes 162931572456 (151.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 2030528 bytes 363988589 (347.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152264264 bytes 278131541781 (259.0 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif5.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4169244 bytes 1045889687 (997.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 263561100 bytes 424894400987 (395.7 GiB)
TX errors 0 dropped 7 overruns 0 carrier 0 collisions 0

vif6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 300242 bytes 16210963 (15.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 153909576 bytes 278461295620 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

vif6.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 43 bytes 1932 (1.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 154205631 bytes 278481298141 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.8 netmask 255.255.255.0 broadcast 192.168.1.255
ether 70:e2:84:13:44:11 txqueuelen 1000 (Ethernet)
RX packets 13689902 bytes 923464162 (880.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12072932 bytes 1307055530 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.255.0 broadcast 192.168.0.255
ether 70:e2:84:13:44:12 txqueuelen 1000 (Ethernet)
RX packets 626995 bytes 180026901 (171.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12815 bytes 942092 (920.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

root@freak:~# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:11 txqueuelen 1000 (Ethernet)
RX packets 52373358 bytes 10623034427 (9.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 242660000 bytes 274734018669 (255.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb200000-fb27ffff

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 70:e2:84:13:44:12 txqueuelen 1000 (Ethernet)
RX packets 221197892 bytes 269978137472 (251.4 GiB)
RX errors 0 dropped 7 overruns 0 frame 0
TX packets 192573206 bytes 166491370299 (155.0 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xfb100000-fb17ffff

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 2553 bytes 147410 (143.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2553 bytes 147410 (143.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 14519247 bytes 133248290251 (124.0 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 139708738 bytes 145135168676 (135.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif17.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 86206104 bytes 157189755115 (146.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 181685983 bytes 268170806613 (249.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 1522072 bytes 88293701 (84.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152712638 bytes 278417240910 (259.2 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif2.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 454933 bytes 81071616 (77.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267218860 bytes 426217224334 (396.9 GiB)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0

vif21.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 20045530 bytes 1284038375 (1.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 141601066 bytes 277441739746 (258.3 GiB)
TX errors 0 dropped 3 overruns 0 carrier 0 collisions 0

vif21.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 73010 bytes 3802474 (3.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267415889 bytes 426146753845 (396.8 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif25.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif25.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 2736576 bytes 295678097 (281.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 260429831 bytes 265797660906 (247.5 GiB)
TX errors 0 dropped 200 overruns 0 carrier 0 collisions 0

vif25.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 145425 bytes 36018716 (34.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 370770440 bytes 395263409640 (368.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 55392503 bytes 130064444520 (121.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 99052116 bytes 147951838129 (137.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif3.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 5133054 bytes 295517366 (281.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 262578665 bytes 425941777243 (396.6 GiB)
TX errors 0 dropped 16 overruns 0 carrier 0 collisions 0

vif4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4902949 bytes 615496460 (586.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 149363618 bytes 277847322538 (258.7 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif4.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 276943 bytes 30721141 (29.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 267397268 bytes 426270528575 (396.9 GiB)
TX errors 0 dropped 14 overruns 0 carrier 0 collisions 0

vif47.0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 208790 bytes 20100733 (19.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 110950236 bytes 110769932971 (103.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif47.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 1000 (Ethernet)
RX packets 140551 bytes 14599509 (13.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 151287643 bytes 163469024604 (152.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vif5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 2030676 bytes 363997181 (347.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 152285777 bytes 278176471509 (259.0 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0

vif5.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 4169387 bytes 1045898303 (997.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 263825846 bytes 425419251935 (396.2 GiB)
TX errors 0 dropped 7 overruns 0 carrier 0 collisions 0

vif6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 300266 bytes 16212271 (15.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 153931212 bytes 278506234302 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

vif6.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 2000 (Ethernet)
RX packets 43 bytes 1932 (1.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 154227291 bytes 278526238467 (259.3 GiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.8 netmask 255.255.255.0 broadcast 192.168.1.255
ether 70:e2:84:13:44:11 txqueuelen 1000 (Ethernet)
RX packets 13690768 bytes 923520126 (880.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12073667 bytes 1307127765 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.255.0 broadcast 192.168.0.255
ether 70:e2:84:13:44:12 txqueuelen 1000 (Ethernet)
RX packets 627010 bytes 180028847 (171.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12815 bytes 942092 (920.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

To see ethernet interfaces that seem up and then do a ifconfig -a to check whether some interfaces are down (e.g. not shown in the simple ifconfig list).
/sbin/ifconfig -a

! Please note that some virtual IP configurations might not appear and noly be visible in an (ip addr show) command.

3. Check iproute2 for special rt_tables (Routing Tables) rules

By default Linux distributions does not have any additional rules in /etc/iproute2/rt_tables however some Linux router machines, needs to have a multiple Gateways. Perhaps the most elegant way to do multiple routings with Linux is to use iproute2's routing tables rt_tables.

Here is example of an OpenXEN system that has 2 Internet providers attached and routes different traffic via

root@freak:~# cat /etc/iproute2/rt_tables
#
# reserved values
#
255   local
254   main
253   default

100   INET1
200 INET2
0   unspec
#
# local
#
#1   inr.ruhep

root@freak:~# ip rule list
0:   from all lookup local
32762:   from all to 192.168.1.8 lookup INET2
32763:   from 192.168.1.8 lookup INET2
32764:   from all to 192.168.0.3 lookup INET1
32765:   from 192.168.0.3 lookup INET1
32766:   from all lookup main
32767:   from all lookup default
root@freak:~#

4. Using ip route get to find out traffic route (path)

root@freak:~# ip route get 192.168.0.1
192.168.0.1 via 192.168.0.1 dev xenbr1 src 192.168.0.3 uid 0
cache

root@freak:~# /sbin/route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 xenbr0
192.168.0.0 192.168.0.1 255.255.255.0 UG 0 0 0 xenbr1
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr1
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 xenbr0
root@freak:~#

root@freak:~# ip route show
default via 192.168.1.1 dev xenbr0
192.168.0.0/24 via 192.168.0.1 dev xenbr1
192.168.0.0/24 dev xenbr1 proto kernel scope link src 192.168.0.3
192.168.1.0/24 dev xenbr0 proto kernel scope link src 192.168.1.8

If you find that gateway is missing you might want to add it with:

root@freak:~# ip route add default via 192.168.5.1

If you need to add a speicic network IP range via separate gateways, you can use commands like:

To add routing for 192.168.0.1/24 / 192.168.1.1/24 via 192.168.0.1 and 192.168.1.1

# /sbin/route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.1 dev eth1
# /sbin/route add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.0.1 dev eth1

If you need to delete a configured wrong route with ip command

# ip route del 192.168.1.0/24 via 0.0.0.0 dev eth1
# ip route del 192.168.0.0/24 via 0.0.0.0 dev eth1

5. Use ping (ICMP protocol) the Destionation IP

root@freak:~# ping -c 3 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.219 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.295 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.270 ms

— 192.168.0.1 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2048ms
rtt min/avg/max/mdev = 0.219/0.261/0.295/0.031 ms
root@freak:~# ping -c 3 192.168.0.39
PING 192.168.0.39 (192.168.0.39) 56(84) bytes of data.
From 192.168.1.80: icmp_seq=2 Redirect Host(New nexthop: 192.168.0.39)
From 192.168.1.80: icmp_seq=3 Redirect Host(New nexthop: 192.168.0.39)
From 192.168.1.80 icmp_seq=1 Destination Host Unreachable

— 192.168.0.39 ping statistics —
3 packets transmitted, 0 received, +1 errors, 100% packet loss, time 2039ms
pipe 3

Note that sometimes you might get 100% traffic loss but still have connection to the destionation in case if the ICMP protocol is filtered for security.

However if you get something like Network is unreachable that is usually an indicator of some routing problem or wrongly configured network netmask.

root@freak:~# ping 192.168.0.5
ping: connect: Network is unreachable

Test network with different packet size. To send 8972 bytes of payload in a Ethernet frame without fragmentation, the following command can be used:

root@pcfreak:~# ping -s 8972 -M do -c 4 freak
PING xen (192.168.1.8) 8972(9000) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
^C
— xen ping statistics —
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2037ms

root@pcfreak:~#

-M pmtudisc_opt
Select Path MTU Discovery strategy. pmtudisc_option may be either do (prohibit fragmentation, even local one), want (do PMTU discovery, fragment locally when packet size is
large), or dont (do not set DF flag).

root@pcfreak:~# ping -s 8972 -M want -c 4 freak
PING xen (192.168.1.8) 8972(9000) bytes of data.
8980 bytes from xen (192.168.1.5): icmp_seq=1 ttl=64 time=2.18 ms
8980 bytes from xen (192.168.1.5): icmp_seq=2 ttl=64 time=1.90 ms
8980 bytes from xen (192.168.1.5): icmp_seq=3 ttl=64 time=2.10 ms
^C
— xen ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.901/2.059/2.178/0.116 ms
root@pcfreak:~#

-M do: prohibit fragmentation
-s 8972 8972 bytes of data
ICMP header: 8 bytes
IP header: 20 bytes (usually, it can be higher)
8980 bytes of bytes is the IP payload

These commands can be used to capture for MTU (maximum transmition units) related issues between hosts that are preventing for hosts to properly send traffic between themselves.
A common issue for Linux hosts to be unable to see each other on the same network is caused by Jumbo Frames (MTU 9000) packets enabled on one of the sides and MTU of 1500 on the other side.
Thus it is always a good idea to thoroughully look up all configured MTUs for all LAN Devices on each server.

6. Check traceroute path to host

If there is no PING but ip route get shows routing is properly configured and the routes existing in the Linux machine routing tables, next step is to check the output of traceroute / tracepath / mtr

raceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets
1 pcfreak (192.168.0.1) 0.263 ms 0.166 ms 0.119 ms
root@freak:~# tracepath 192.168.1.1
1?: [LOCALHOST] pmtu 1500
1: vivacom-gigabit-router 0.925ms reached
1: vivacom-gigabit-router 0.835ms reached
Resume: pmtu 1500 hops 1 back 1

It might be useful to get a frequent output of the command (especially on Linux hosts) where mtr command is not installed with:

root@freak:~# watch -n 0.1 traceroute 192.168.0.1

root@freak:~# traceroute -4 google.com
traceroute to google.com (172.217.17.110), 30 hops max, 60 byte packets
1 vivacom-gigabit-router (192.168.1.1) 0.657 ms 1.280 ms 1.647 ms
2 213.91.190.130 (213.91.190.130) 7.983 ms 8.168 ms 8.097 ms
3 * * *
4 * * *
5 212-39-66-222.ip.btc-net.bg (212.39.66.222) 16.613 ms 16.336 ms 17.151 ms
6 * * *
7 142.251.92.65 (142.251.92.65) 18.808 ms 13.246 ms 209.85.254.242 (209.85.254.242) 15.541 ms
8 142.251.92.3 (142.251.92.3) 14.223 ms 142.251.227.251 (142.251.227.251) 14.507 ms 142.251.92.3 (142.251.92.3) 15.328 ms
9 ams15s29-in-f14.1e100.net (172.217.17.110) 14.097 ms 14.909 ms 142.251.242.230 (142.251.242.230) 13.481 ms
root@freak:~#

If you have MTR then you can get plenty of useful additional information such as the Network HOP name or the Country location of the HOP.

To get HOP name:

root@freak:~# mtr -z google.com

To get info on where (which Country) exactly network HOP is located physically:

root@freak:~# mtr -y 2 google.com

7. Check iptables INPUT / FORWARD / OUTPUT rules are messing with something

# iptables -L -n
…

# iptables -t nat -L -n
…

Ideally you would not have any firewall

# iptables -L -n

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

In case if something like firewalld is enabled as a default serviceto provide some modern Linux firewall as Ubuntu and Redhat / CentOS / Fedoras has it often turned on as a service stop and disable the service

# systemctl stop firewalld

# systemctl disable firewalld

8. Debug for any possible MAC address duplicates

root@pcfrxen:~# arp -an
? (192.168.1.33) at 00:16:3e:59:96:9e [ether] on eth0
? (192.168.1.1) at 18:45:93:c6:d8:00 [ether] on eth1
? (192.168.0.1) at 8c:89:a5:f2:e8:d9 [ether] on eth1
? (192.168.1.1) at 18:45:93:c6:d8:00 [ether] on eth0
? (192.168.1.11) at 7c:0a:3f:89:b6:fa [ether] on eth1
? (192.168.1.17) at <incomplete> on eth0
? (192.168.1.37) at 00:16:3e:ea:05:ce [ether] on eth0
? (192.168.1.80) at 8c:89:a5:f2:e7:d8 [ether] on eth0
? (192.168.1.11) at 7c:0a:3f:89:a5:fa [ether] on eth0
? (192.168.1.30) at 00:16:3e:bb:46:45 [ether] on eth1
? (192.168.0.210) at 00:16:3e:68:d9:55 [ether] on eth1
? (192.168.1.30) at 00:16:3e:bb:46:45 [ether] on eth0
? (192.168.1.18) at 00:16:3e:0d:40:05 [ether] on eth1
? (192.168.0.211) at 00:16:3e:4d:41:05 [ether] on eth1
? (192.168.1.35) at 00:16:3e:d1:8f:77 [ether] on eth0
? (192.168.1.18) at 00:16:3e:0d:43:05 [ether] on eth0
? (192.168.1.28) at 00:16:3e:04:12:1c [ether] on eth1
? (192.168.0.3) at 70:e2:84:13:43:12 [ether] on eth1
? (192.168.0.208) at 00:16:3e:51:de:9c [ether] on eth1
? (192.168.0.241) at 00:16:3e:0d:48:06 [ether] on eth1
? (192.168.1.28) at 00:16:3e:04:12:1c [ether] on eth0
? (192.168.1.33) at 00:16:3e:59:97:8e [ether] on eth1
? (192.168.0.241) at 00:16:3e:0d:45:06 [ether] on eth0
? (192.168.0.209) at 00:16:3e:5c:df:96 [ether] on eth1

root@pcfrxen:~# ip neigh show
192.168.1.33 dev eth0 lladdr 00:16:3e:59:96:9e REACHABLE
192.168.1.1 dev eth1 lladdr 18:45:93:c6:d8:00 STALE
192.168.0.1 dev eth1 lladdr 8c:89:a5:f2:e8:d9 REACHABLE
192.168.1.1 dev eth0 lladdr 18:45:93:c6:d9:01 REACHABLE
192.168.1.11 dev eth1 lladdr 7c:0a:3f:89:a6:fb STALE
192.168.1.17 dev eth0 FAILED
192.168.1.37 dev eth0 lladdr 00:16:3e:ea:06:ce STALE
192.168.1.80 dev eth0 lladdr 8c:89:a5:f2:e8:d9 REACHABLE
192.168.1.11 dev eth0 lladdr 7c:0a:3f:89:a7:fa STALE
192.168.1.30 dev eth1 lladdr 00:16:3e:bb:45:46 STALE
192.168.0.210 dev eth1 lladdr 00:16:3e:68:d8:56 REACHABLE
192.168.1.30 dev eth0 lladdr 00:16:3e:bb:45:46 STALE
192.168.1.18 dev eth1 lladdr 00:16:3e:0d:48:04 STALE
192.168.0.211 dev eth1 lladdr 00:16:3e:4d:40:04 STALE
192.168.1.35 dev eth0 lladdr 00:16:3e:d2:8f:76 STALE
192.168.1.18 dev eth0 lladdr 00:16:3e:0d:48:06 STALE
192.168.1.28 dev eth1 lladdr 00:16:3e:04:11:2c STALE
192.168.0.3 dev eth1 lladdr 70:e2:84:13:44:13 STALE
192.168.0.208 dev eth1 lladdr 00:16:3e:51:de:9c REACHABLE
192.168.0.241 dev eth1 lladdr 00:16:3e:0d:48:07 STALE
192.168.1.28 dev eth0 lladdr 00:16:3e:04:12:1c REACHABLE
192.168.1.33 dev eth1 lladdr 00:16:3e:59:96:9e STALE
192.168.0.241 dev eth0 lladdr 00:16:3e:0d:49:06 STALE
192.168.0.209 dev eth1 lladdr 00:16:3e:5c:dd:97 STALE
root@pcfrxen:~#

9. Check out with netstat / ss for any irregularities such as high amount of error of faulty ICMP / TCP / UDP network packs

For example check out the netstat network stack output

# netstat -s

root@pcfrxen:~# netstat -s
Ip:
Forwarding: 2
440044929 total packets received
1032 with invalid addresses
0 forwarded
0 incoming packets discarded
439988902 incoming packets delivered
396161852 requests sent out
3 outgoing packets dropped
100 dropped because of missing route
Icmp:
1025 ICMP messages received
540 input ICMP message failed
ICMP input histogram:
destination unreachable: 1014
timeout in transit: 11
519 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 519
IcmpMsg:
InType3: 1014
InType11: 11
OutType3: 519
Tcp:
1077237 active connection openings
1070510 passive connection openings
1398236 failed connection attempts
111345 connection resets received
83 connections established
438293250 segments received
508143650 segments sent out
42567 segments retransmitted
546 bad segments received
329039 resets sent
Udp:
1661295 packets received
278 packets to unknown port received
0 packet receive errors
1545720 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 33046
UdpLite:
TcpExt:
1 invalid SYN cookies received
1398196 resets received for embryonic SYN_RECV sockets
1737473 packets pruned from receive queue because of socket buffer overrun
1118775 TCP sockets finished time wait in fast timer
638 time wait sockets recycled by time stamp
656 packetes rejected in established connections because of timestamp
2218959 delayed acks sent
2330 delayed acks further delayed because of locked socket
Quick ack mode was activated 7172 times
271799723 packet headers predicted
14917420 acknowledgments not containing data payload received
171078735 predicted acknowledgments
52 times recovered from packet loss due to fast retransmit
TCPSackRecovery: 337
Detected reordering 1551 times using SACK
Detected reordering 1501 times using reno fast retransmit
Detected reordering 61 times using time stamp
9 congestion windows fully recovered without slow start
38 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 241
104 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 11550
1 timeouts after reno fast retransmit
TCPSackFailures: 13
3772 fast retransmits
2 retransmits in slow start
TCPTimeouts: 24104
TCPLossProbes: 101748
TCPLossProbeRecovery: 134
TCPSackRecoveryFail: 3
128989224 packets collapsed in receive queue due to low socket buffer
TCPBacklogCoalesce: 715034
TCPDSACKOldSent: 7168
TCPDSACKOfoSent: 341
TCPDSACKRecv: 16612
150689 connections reset due to unexpected data
27063 connections reset due to early user close
17 connections aborted due to timeout
TCPDSACKIgnoredOld: 158
TCPDSACKIgnoredNoUndo: 13514
TCPSpuriousRTOs: 9
TCPSackMerged: 1191
TCPSackShiftFallback: 1011
TCPDeferAcceptDrop: 699473
TCPRcvCoalesce: 3311764
TCPOFOQueue: 14289375
TCPOFOMerge: 356
TCPChallengeACK: 621
TCPSYNChallenge: 621
TCPSpuriousRtxHostQueues: 4
TCPAutoCorking: 1605205
TCPFromZeroWindowAdv: 132380
TCPToZeroWindowAdv: 132441
TCPWantZeroWindowAdv: 1445495
TCPSynRetrans: 23652
TCPOrigDataSent: 388992604
TCPHystartTrainDetect: 69089
TCPHystartTrainCwnd: 3264904
TCPHystartDelayDetect: 4
TCPHystartDelayCwnd: 128
TCPACKSkippedPAWS: 3
TCPACKSkippedSeq: 2001
TCPACKSkippedChallenge: 2
TCPWinProbe: 123043
TCPKeepAlive: 4389
TCPDelivered: 389507445
TCPAckCompressed: 7343781
TcpTimeoutRehash: 23311
TcpDuplicateDataRehash: 8
TCPDSACKRecvSegs: 17335
IpExt:
InMcastPkts: 145100
OutMcastPkts: 9429
InBcastPkts: 18226
InOctets: 722933727848
OutOctets: 759502627470
InMcastOctets: 58227095
OutMcastOctets: 3284379
InBcastOctets: 1756918
InNoECTPkts: 440286946
InECT0Pkts: 936

List all listening established connections to host

# netstat -ltne

List all UDP / TCP connections

# netstat -ltua

or if you prefer to do it with the newer and more comprehensive tool ss:

List all listening TCP connections

# ss -lt

List all listening UDP connections

# ss -ua

Display statistics about recent connections

root@pcfrxen:~# ss -s
Total: 329
TCP: 896 (estab 70, closed 769, orphaned 0, timewait 767)

Transport Total IP IPv6
RAW   0 0 0
UDP   40 36 4
TCP   127 118 9
INET   167 154 13
FRAG   0 0 0

If you need to debug some specific sport or dport filter out the connection you need by port number

# ss -at '( dport = :22 or sport = :22 )'

Debug for any possible issues with ICMP unreachable but ports reachable with NMAP / telnet / Netcat

# nc 192.168.0.1 -vz

root@pcfrxen:/ # nc 192.168.0.1 80 -vz
pcfreak [192.168.0.1] 80 (http) open

root@pcfrxen:/ # nc 192.168.0.1 5555 -vz
pcfreak [192.168.0.1] 5555 (?) : Connection refused

root@pcfrxen:/# telnet 192.168.0.1 3128
Trying 192.168.0.1…
Connected to 192.168.0.1.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

root@pcfrxen:/# nmap -sS -P0 192.168.0.1 -p 443 -O
Starting Nmap 7.80 ( https://nmap.org ) at 2023-11-27 19:51 EET
Nmap scan report for pcfreak (192.168.0.1)
Host is up (0.00036s latency).

PORT STATE SERVICE
443/tcp open https
MAC Address: 8C:89:A5:F2:E8:D8 (Micro-Star INT'L)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Aggressive OS guesses: Linux 3.11 (96%), Linux 3.1 (95%), Linux 3.2 (95%), AXIS 210A or 211 Network Camera (Linux 2.6.17) (94%), Linux 2.6.32 (94%), Linux 3.10 (94%), Linux 2.6.18 (93%), Linux 3.2 – 4.9 (93%), ASUS RT-N56U WAP (Linux 3.4) (93%), Linux 3.16 (93%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop

OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.24 seconds
root@pcfrxen:/#

10. Add static MAC address to Ethernet Interface (if you find a MAC address being wrongly assigned to interface)

Sometimes problems with network unrechability between hosts is caused by wrongly defined MAC addresses on a Switch that did not correspond correctly to the ones assigned on the Linux host.
The easiest resolution here if you don't have access to Switch in work environment is to reassign the default MAC addresses of interfaces to proper MAC addresses, expected by remote router.

root@pcfrxen:/#  /sbin/ifconfig eth2 hw ether 8c:89:a5:f2:e8:d6

root@pcfrxen:/#  /sbin/ifconfig eth1 hw ether 8c:89:a5:f2:e8:d5

root@pcfrxen:/#  ifconfig eth0|grep -i ether
ether 8c:89:a5:f2:e8:d6 txqueuelen 1000 (Ethernet)

11. Check for Network Address Translation (NAT) misconfigurations

If you do use some NAT-ing between Linux host and the remote Network Device you cannot reach, make sure IP Forwarding is enabled (i.e. /etc/sysctl.conf was not mistakenly overwritten by a script or admin for whatever reason).

root@server:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
root@server:~# sysctl net.ipv4.conf.all.forwarding
net.ipv4.conf.all.forwarding = 1

root@server:~# sysctl net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 0

12. Check for Resolving DNS irregularities with /etc/resolv.conf

If network connectivity is okay on TCP / IP , UDP Level but problems with DNS of course, check what you have configured inside /etc/resolv.conf

And if use newer Linux distributions and have resolving managed by systemd check status of resolvectl

root@server:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND — YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.

nameserver 127.0.0.1
search pc-freak.net
domain pc-freak.net
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 109.104.195.2
nameserver 109.104.195.1
nameserver 208.67.222.222
nameserver 208.67.220.220
options timeout:2 rotate
root@pcfreak:~#

root@server:~# resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (ens3)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.5.1
DNS Servers: 192.168.5.1

As seen see, the systemd-resolved service is used to provide domain names resolution and we can modify its configuration file /etc/systemd/resolved.conf to add the DNS server – the following line is set (two DNS servers’ addresses are added):

For example …

DNS=8.8.8.8

13. Fix problems with wrongly configured Network Speed between hosts

It is not uncommon to have a Switch between two Linux hosts that is set to communicate on a certain maximum amount of Speed but a Linux host is set to communicate or lesser or more of Speed, this might create network issues so in such cases make sure either you use the Auto Negitionation network feature
or set both sides to be communicating on the same amount of network speed.

To turn on auto negotiation for ether interface

# ethtool -s eth1 speed 1000 duplex full autoneg on

For example to set a Linux network interface to communicate on 1 Gigabit speed and switch off autonegotiation off.

# ethtool -s eth1 speed 1000 duplex full autoneg off

14. Check arp and icmp traffic with tcpdump

On both sides where the IPs can't see each other we can run a tcpdump to check the ARP and ICMP traffic flowing between the hosts.

# tcpdump -i eth1 arp or icmp

cpdump: verbose output suppressed, use -v[v]… for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:29:07.001841 IP freak-eth1 > pcfr_hware_local_ip: ICMP echo request, id 13348, seq 65, length 64
15:29:07.001887 IP pcfr_hware_local_ip > freak-eth1: ICMP echo reply, id 13348, seq 65, length 64
15:29:07.598413 ARP, Request who-has pcfr_hware_local_ip tell zabbix-server, length 46
15:29:07.598425 ARP, Reply pcfr_hware_local_ip is-at 8c:89:a5:f2:e8:d8 (oui Unknown), length 28
15:29:07.633055 ARP, Request who-has freak_vivacom_auto_assigned_dhcp_ip tell 192.168.1.1, length 46
15:29:08.025824 IP freak-eth1 > pcfr_hware_local_ip: ICMP echo request, id 13348, seq 66, length 64
15:29:08.025864 IP pcfr_hware_local_ip > freak-eth1: ICMP echo reply, id 13348, seq 66, length 64

# tcpdump -i eth1 -vvv
…

If you want to sniff for TCP protocol and specific port and look up for DATA transfered for SMTP you can use something like:

# tcpdump -nNxXi eth0 -s 0 proto TCP and port 25

If you need a bit more thorough explanation on what it would do check out my previous article How to catch / track mail server traffic abusers with tcpdump.

15. Debugging network bridge issues

Having bridge network interface is another brink where things could go totally wrong.
If you have network bridges configured, check out what is the status of the bridge.

root@freak:/etc/network# brctl show
bridge name   bridge id       STP enabled   interfaces
xenbr0       8000.70e284134411   yes       eno1
                           vif1.0
                           vif10.0
                           vif16.0
                           vif16.0-emu
                           vif2.0
                           vif3.0
                           vif4.0
                           vif5.0
                           vif6.0
                           vif9.0
                           vif9.0-emu
xenbr1       8000.70e284134412   yes       eno2
                           vif1.1
                           vif10.1
                           vif16.1
                           vif16.1-emu
                           vif2.1
                           vif3.1
                           vif4.1
                           vif5.1
                           vif6.1
                           vif9.1
                           vif9.1-emu

Check out any configurations such as /etc/sysconfig/network-scripts/ifcfg-* are not misconfigured if on Redhat / CentOS / Fedora.
Or if on Debian / Ubuntu and other deb based Linuxes look up for /etc/network/interfaces config problems that might be causing the bridge to misbehave.

For example one bridge network issue, I've experienced recently is related to bridge_ports variable configured as bridge_ports all.
This was causing the second bridge xenbr1 to be unable to see another local network that was directly connected with a cable to it.

The fix was bridge_ports none. Finding out this trivial issue caused by a restored network config from old backup took me days to debug.
As everything seemed on a network level to be perfect just like in Physical layer, same way and on Software level, routings were okay.
Checked everything multiple times and did not see anything irregular. ping was missing and hosts cannot see each other even though having the right netmask and
network configuration in place.

Below is my /etc/network/interfaces configuration with the correct bridge_ports none changed.

root@freak:/etc/network# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eno1
allow-hotplug eno1
iface eno1 inet manual
dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220
auto eno2
allow-hotplug eno2
iface eno2 inet manual
dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220

auto xenbr0
allow-hotplug xenbr0
# Bridge setup
# fetching dhcp ip from 192.168.1.20 (vivacom fiber optics router) routing traffic via 1Gigabit network
iface xenbr0 inet dhcp
hwaddress ether 70:e2:84:13:44:11
# address 192.168.1.5/22
address 192.168.1.5
netmask 255.255.252.0
# address 192.168.1.8 if dhcp takes from vivacom dhcpd
bridge_ports eno1
gateway 192.168.1.20
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports none
dns-nameservers 8.8.8.8 8.8.4.4

auto xenbr1
# fetching dhcp ip from pc-freak.net (192.168.0.1) bergon.net routing traffic through it
allow-hotplug xenbr1
iface xenbr1 inet dhcp
hwaddress ether 70:e2:84:13:44:11
## address 192.168.0.3/22
address 192.168.0.8
netmask 255.255.252.0
# address 192.168.0.8 if dhcp takes from vivacom dhcpd (currently mac deleted from vivacom router)
# address 192.168.0.9 if dhcp takes from pc-freak.net hware host
# hwaddress ether 70:e2:84:13:44:13
gateway 192.168.0.1
bridge_ports eno2
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports none
dns-nameservers 8.8.8.8 8.8.4.4
root@freak:/etc/network#

root@freak:/etc/network# brctl showstp xenbr0
xenbr0
bridge id       8000.70e284134411
designated root   8000.70e284134411
root port       0           path cost       0
max age       20.00           bridge max age       20.00
hello time       2.00           bridge hello time   2.00
forward delay       15.00           bridge forward delay   15.00
ageing time       0.00
hello timer       1.31           tcn timer       0.00
topology change timer   0.00           gc timer       0.00
flags

eno1 (1)
port id       8001           state       forwarding
designated root   8000.70e284134411   path cost       19
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8001           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif1.0 (2)
port id       8002           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8002           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif10.0 (12)
port id       800c           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800c           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif16.0 (13)
port id       800d           state       disabled
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800d           forward delay timer   0.00
designated cost   0           hold timer       0.00
flags

vif16.0-emu (14)
port id       800e           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800e           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif2.0 (4)
port id       8004           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8004           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif3.0 (5)
port id       8005           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8005           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif4.0 (3)
port id       8003           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8003           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif5.0 (6)
port id       8006           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8006           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif6.0 (7)
port id       8007           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   8007           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

vif9.0 (10)
port id       800a           state       disabled
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800a           forward delay timer   0.00
designated cost   0           hold timer       0.00
flags

vif9.0-emu (11)
port id       800b           state       forwarding
designated root   8000.70e284134411   path cost       100
designated bridge   8000.70e284134411   message age timer   0.00
designated port   800b           forward delay timer   0.00
designated cost   0           hold timer       0.31
flags

root@freak:/etc/network#

Sum it up

We have learned how to debug various routing issues, how to add and remote default gateways, check network reachability with ICMP protocol with ping, traceroute as well check for DNS issues and given some hints how to resolve DNS misconfigurations.
We also learned how to check the configured Network interfaces certain settings and resolve issues caused by Network sides max Speed misconfigurations as well how to track and resolve communication issues caused by wrongly configured MAC addresses.
Further more learned on how to do a basic port and protocol debugging of state of Network packets with netstat and nc and check problems related to iptables Firewall and IP Forwarding misconfigurations.
Finally we learned some basic usage of tcpdump on how to track arp and MAC traffic and look up for a specific TCP / UDP protocol and its contained data.
There is certainly things this article is missing as the topic of debugging network connectivity issues on Linux is a whole ocean, especially as the complexity of Linux has grown dramatically these days.
I gues it is worthy to mention that unable to see remote network could be caused by wrong VLAN configurations on Linux or even buggy switches and router devices, due to hardware or software,
but I hope this article at least covers the very basics of network debugging and Linux.

Enjojy 🙂

Tags: article, broadcast, data, debugging, default, Display, fails, guide, histogram, ifconfig eth0, initial, internal, issues, link, linux?, loopback, mtr, MULTICAST, multiple times, network, network speed, Old, operating systems, ping, pipe, Present, running, RX, Settings, system administrators, tcp, time, Tx
Posted in Debian, Educational, Linux, Networking, System Administration | No Comments »

How to turn On or Off Screen Reader ORCA on Linux Desktop enabled by mistype or a kid smash on the keyboard

Wednesday, November 22nd, 2023

For those who type quite fast and use Microsoft Windows, its quite common to start the annoying NARRATOR (Windows Speaking Program) by accidently due to mistyping pressing together Windows key + Control + Enter.
This enables Narrator to read stuff on the screen here and there and to turn it off you just have to either Lock the Windows Computer and press again Windows key + Control + Enter to TURN OFF NARRATOR.

Linux does not have a Narrator but have also embedded Eye impairment Assistive Technology called ORCA.

Orca works with applications and toolkits that support the Assistive Technology Service Provider Interface (AT-SPI), which is the primary assistive technology infrastructure for Linux and Solaris. Applications and toolkits supporting the AT-SPI include the GNOME Gtk+ toolkit, the Java platform's Swing toolkit, LibreOffice, Gecko, and WebKitGtk. AT-SPI support for the KDE Qt toolkit is being pursued.

ORCA is nowadays installed and integrated into many if not most Linux distributions out there. Enabling ORCA is not such a common thing on Linux,so today I got quite puzzled once I came back to the computer, leaving the 3.7 months kid near the Keyboard and finding out that I've enabled aloud screen reader that is reading what is every Window / Menu / Program or object I select with the mouse on my Linux MATE Desktop home GUI environment running on top of Debian Linux.

After a quick look up in Google on what exactly is the Linux program that is reading my screen I came across ORCA, which seem to be visible also as running in my process list:

hipo@jeremiah:~/Downloads$ ps -ef|grep -i orca
hipo 1068376 7960 17 18:48 tty2 00:00:01 orca

After a quick check online I found out that,

To start (Turn On ) Orca Screen Reader using the keyboard:

Windows logo button (Super Key) key + Alt + S

Of course, it is possible to shut off the annoying reader by simply killing it with:

# kill -9 orca

Ubuntu users, could start Orca using a mouse and keyboard:

Open the Activities overview and start typing Accessibility.

Click Accessibility to open the panel.

Select thez to open it.

Switch the Screen Reader switch to on.

Problem solved now Screen Reader on Linux is disabled, maybe it is time to disable Orca key press ability to prevent the kid from enabling it again since I don't need it actively thanksfully. with

xmodmap -e 'keycode <value>='

or simply removing the orca package with apt:

# apt remove orca

Tags: applications, common, hipo, How to, keyboard, linux desktop, microsoft windows, mouse, ORCA, reading, running, simply, Ubuntu
Posted in Curious Facts, Linux, Linux and FreeBSD Desktop, System Administration | No Comments »

Perl Modules via HTTP Proxy installation, update and install perl CPAN modules behind a Firewall DMZ-ed networks

Friday, May 19th, 2023

If you have to maintain perl script written applications on Linux servers that are sitting behind a very paranoid set-up firewalls
and Local DMZ network, but you still need to maintain the servers and applications versions including perl CPAN (Comprehensive Perl Archive networking) module libraries, you could still do so via another Proxy machine Hub on the Local network, where you either have to manually download all the newest perl versions and CPAN module (libraries) or you can set it up that Proxy machine
to access only a specific Secured internet URLs for perl stuff.

Proxying perl downloads can be done via FTP connects, but as FTP is communicating in plain text and the protocol is known
for not behaving very well behind firewalls, it is a better idea to use for CPAN downloads HTTP or HTTPS protocol.

Normlly Perl is using FTP to download files from the internet. To enable Perl using also HTTP, please install the following RPM:

# yum install perl-libwww-perl

After figure out a CPAN-mirror from http://mirrors.cpan.org/search.cgi?country=Germany, we can start with the configuration from CPAN.

or debian package

# apt install libwww-perl

At the first run from /usr/bin/cpan the initial configuration will be started up which will be done automatically:

[username@linux-host ~]$ cpan

/home/linux-username/.cpan/CPAN/MyConfig.pm initialized.

CPAN is the world-wide archive of perl resources. It consists of about
100 sites that all replicate the same contents all around the globe.
Many countries have at least one CPAN site already. The resources
found on CPAN are easily accessible with the CPAN.pm module. If you
want to use CPAN.pm, you have to configure it properly.

If you do not want to enter a dialog now, you can answer 'no' to this
question and I'll try to autoconfigure. (Note: you can revisit this
dialog anytime later by typing 'o conf init' at the cpan prompt.)

Are you ready for manual configuration? [yes] no

— SNIP — SNAP — SNIP — SNAP — SNIP — SNAP —

commit: wrote /home/linux-username/.cpan/CPAN/MyConfig.pm
Terminal does not support AddHistory.

cpan shell — CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan> q
Terminal does not support GetHistory.
Lockfile removed.
[username@linux-host ~]$

After the initial configuration you have to run /usr/bin/cpan again, to configure the HTTP-proxy and an alternative HTTP-URL for the default FTP URL:

[username@linux-host ~]$ cpan
Terminal does not support AddHistory.

cpan shell — CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan> o conf http_proxy http://proxy-host-to-internet.com:8080
http_proxy http://proxy-host-to-internet.com:8080

cpan> o conf urllist push http://mirrors.zerg.biz/cpan/

cpan> o conf commit
commit: wrote /home/linux-username/.cpan/CPAN/MyConfig.pm

cpan> q
Terminal does not support GetHistory.
Lockfile removed.
[username@linux-host ~]$

From now CPAN will load it's files from the internet with the HTTP-proxy:

[username@linux-host ~]$ cpan
Terminal does not support AddHistory.

cpan shell — CPAN exploration and modules installation (v1.7602)
ReadLine support available (try 'install Bundle::CPAN')

cpan> i Example::DB::Oracle
CPAN: Storable loaded ok
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
http://mirrors.zerg.biz/cpan/authors/01mailrc.txt.gz
Going to read /home/linux-host/.cpan/sources/authors/01mailrc.txt.gz
Fetching with LWP:
http://mirrors.zerg.biz/cpan/modules/02packages.details.txt.gz
Going to read /home/linux-host/.cpan/sources/modules/02packages.details.txt.gz
Database was generated on Thu, 07 Jan 2010 10:44:22 GMT

There's a new CPAN.pm version (v1.9402) available!
[Current version is v1.7602]
You might want to try
    install Bundle::CPAN
    reload cpan
without quitting the current session. It should be a seamless upgrade
while we are running…

Fetching with LWP:
http://mirrors.zerg.biz/cpan/modules/03modlist.data.gz
Going to read /home/linux-username/.cpan/sources/modules/03modlist.data.gz
Going to write /home/linux-username/.cpan/Metadata
Strange distribution name [Example::DB::Oracle]
Module id = Example::DB::Oracle
    CPAN_USERID MSERGEANT (MSERGEANT <msergeant@cpan.org>)
    CPAN_VERSION undef
    CPAN_FILE    M/MS/MSERGEANT/DBIx-AnyDBD-2.01.tar.gz
    INST_FILE    (not installed)

cpan> q
Terminal does not support GetHistory.
Lockfile removed.

Now as the new proxy http URL http://proxy-host-to-internet.com:8080 is set on the machine, to upgrade the existing modules non interactively

# perl MCPAN -e upgrade

or do it the old fashioned way via the MCPAN perl shell:

# perl -MCPAN -e shell

Starting with version 2.29 of the cpan shell, a new download mechanism
is the default which exclusively uses cpan.org as the host to download
from. The configuration variable pushy_https can be used to (de)select
the new mechanism. Please read more about it and make your choice
between the old and the new mechanism by running

o conf init pushy_https

Once you have done that and stored the config variable this dialog
will disappear.

cpan shell — CPAN exploration and modules installation (v2.29)
Enter 'h' for help.

cpan[1]> upgrade
…

That's all folks after a while if no errors are spit during the Perl modules update you'll be at the latest versions of CPAN and modules.

Tags: configure, downloads, gz, home, installation, MCPAN, Perl Modules, read, running, terminal
Posted in Linux, System Administration | 2 Comments »

Linux script to periodically log enabled systemctl services, configured network IPs and routings, server established connections and iptables firewall rules

Tuesday, January 25th, 2022

For those who are running some kind of server be it virtual or physical, where multiple people or many systemins have access, sometimes it could be quite a mess as someone due to miscommunication or whatever could change something on the configured Network Ethernet interfaces, or configured routing tables, or simply issue an update which might change the set of automatically set to run systemctl services due to update. Such changes on a Linux server Operating system often can remain unnoticed and could cause quite a harm. Even when the change is noticed the logical question occurs what was the previous network route on the server or what kind of network was configured on Ethernet interface ethX etc.
Problems like the described where, pretty common in many public Private Clouds or VMWare / XEN based Hypervisors that host multiple Virtual machines, for that reason I've developed a small script which is pretty dumb on the first glimpse but mostly useful as it keeps historical records of such important information.

#!/bin/sh
# script to show configured services on system, configured IPs, netstat state and network routes
# Script to be used during CentOS and Redhat Enterprise Linux RPM package updates with yum

output_file=network_ip_routes_services_status;
ddate=$(date '+%Y-%m-%d_%H-%M-%S');
iptables=$(which iptables);
if [ ! -d /root/logs/ ]; then
mkdir /root/logs/;
fi

echo "STARTED: $(date '+%Y-%m-%d_%H-%M-%S'):" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# systemctl list-unit-files\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
systemctl list-unit-files –type=service | grep enabled | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e '# systemctl | grep ".service" | grep "running"\n' | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
systemctl | grep ".service" | grep "running" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# netstat -tulpn\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
netstat -tulpn | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# netstat -r\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
netstat -r | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# ip a s\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
ip a s | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# /sbin/route -n\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
/sbin/route -n | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# $iptables -L -n\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# $iptables -t nat -L" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
$iptables -L -n | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
$iptables -t nat -L | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo "ENDED $(date '+%Y-%m-%d_%H-%M-%S'):" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log

Script produces its logs inside /root/logs/network_ip_routes_services_status*hostname*currentdate*.log, put the script inside /root/ or wherever you like.

To keep an eye how network routing or ip configuration or firewall changed or there was a peak with the established connections towards daemons running on host (lets say requiring a machine upgrade), I've set the script to run as usually via cron job at the end of the predefined cron job tasks, like so:

# crontab -u root -e
# periodic dump and log network routing tables, netstat and systemctl list-unit-files
*/1 01 01,25 * * /root/show_running_services_netstat_ips_route1.sh 2>&1 >/dev/null

You can download a copy of show_running_services_netstat_ips_route1.sh script here.
The script is written without much of efficiency on mind, as you can see the with the multiple tee -a and for critical hosts it might be a good idea to rewrite it to use '>>' OPERAND instead, anyhows as most machines today are pretty powerful it doesn't really matter much.

Of course today such a script is quite archaic, as most big corporations are using much more complex monitoring software such as Zabbix, Prometheus or if some kind of Elastic Search is used Kibana etc. but for a basic needs and even for a double checking and comparing with other more advanced monitoring tools (in case if monitoring tools database gets damaged or temporary down until backupped), still I think such an oldschool simple monitoring script can be of use.

A good addition to that if you use a central logging server is to set another cron to periodically synchronize produced /root/logs/* to somewhere, here is how to do it with simple rsync (considering your host is configured to login with a user without password with ssh key authentication).

# HOSTNAME=$(hostname); rsync -axHv –ignore-existing -e 'ssh -p 22' /bashscripts/ -q -i –out-format="%t %f %b" –log-file=/var/log/rsync_sync_jobs.log –info=progress2 root@BACKUP_SERVER_HOST:/$(HOSTNAME)-logs/

Once something strange occurs with the machine, like the machine needs to be rebuild

I would be glad to hear if some of my readers uses some useful script which I can adopt myself. Cheers 🙂

Tags: configured, cron job, firewall rules, iptables firewall, logs, machines, Network Ethernet, running, script, systemctl
Posted in Linux, Linux Backup tools | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘running’

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites