System Administration Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Archive for the ‘System Administration’ Category

How to Recover deleted /var/lib/dpkg directory on Debian / Ubuntu Linux server

Wednesday, October 6th, 2021

how-to-recover-accidently-deleted-var-lib-dpkg-directory-on-debian-gnu-linux

Sometimes you might do something stupid, in the hurry like running the wrong rm command and ending up deleting /var/lib/dpkg on your Debian / Ubuntu system.

by either wrongly issuing the rm to a directory or mistyping rm -r /var/lib/dpkg.
I know this is pretty dumb but sometimes we're all dumb, if you do so and you try to do the regular

root@debian:/ # apt update && apt upgrade

or try to install some random package onwards you will end up with error message:

E: Could not open lock file /var/lib/dpkg/lock – open (2: No such file or directory)

Ending up with this error, does totally blocks your further system administration activities with both apt / aptitude / apt-get as well as with dpkg package management tool.

1. The /var/backups recovery directory

Thankfully, by Gods mercy some of Debian Linux system architects has foreseen such issues might occur and have integrated into it the automatic periodic creation of some important files into directory /var/backups/

Hence the next step is to check what kind of backups are available, there:

root@debian:/ # ls -al /var/backups/
total 19892
drwxr-xr-x 7 root root 4096 Sep 24 06:25 ./
drwxr-xr-x 22 root root 4096 Dec 21 2020 ../
-rw-r–r– 1 root root 245760 Aug 20 06:25 alternatives.tar.0
-rw-r–r– 1 root root 15910 Aug 14 06:25 alternatives.tar.1.gz
-rw-r–r– 1 root root 15914 May 29 06:25 alternatives.tar.2.gz
-rw-r–r– 1 root root 15783 Jan 29 2021 alternatives.tar.3.gz
-rw-r–r– 1 root root 15825 Nov 20 2020 alternatives.tar.4.gz
-rw-r–r– 1 root root 15778 Jul 16 2020 alternatives.tar.5.gz
-rw-r–r– 1 root root 15799 Jul 4 2020 alternatives.tar.6.gz
-rw-r–r– 1 root root 80417 Aug 19 14:48 apt.extended_states.0
-rw-r–r– 1 root root 8693 Apr 27 22:40 apt.extended_states.1.gz
-rw-r–r– 1 root root 8658 Apr 17 19:45 apt.extended_states.2.gz
-rw-r–r– 1 root root 8601 Apr 15 00:52 apt.extended_states.3.gz
-rw-r–r– 1 root root 8599 Apr 9 00:26 apt.extended_states.4.gz
-rw-r–r– 1 root root 8542 Mar 18 2021 apt.extended_states.5.gz
-rw-r–r– 1 root root 8549 Mar 18 2021 apt.extended_states.6.gz
-rw-r–r– 1 root root 9030483 Jul 4 2020 aptitude.pkgstates.0
-rw-r–r– 1 root root 628958 May 7 2019 aptitude.pkgstates.1.gz
-rw-r–r– 1 root root 534758 Oct 21 2017 aptitude.pkgstates.2.gz
-rw-r–r– 1 root root 503877 Oct 19 2017 aptitude.pkgstates.3.gz
-rw-r–r– 1 root root 423277 Oct 15 2017 aptitude.pkgstates.4.gz
-rw-r–r– 1 root root 420899 Oct 14 2017 aptitude.pkgstates.5.gz
-rw-r–r– 1 root root 229508 May 5 2015 aptitude.pkgstates.6.gz
-rw-r–r– 1 root root 11 Oct 14 2017 dpkg.arch.0
-rw-r–r– 1 root root 43 Oct 14 2017 dpkg.arch.1.gz
-rw-r–r– 1 root root 43 Oct 14 2017 dpkg.arch.2.gz
-rw-r–r– 1 root root 43 Oct 14 2017 dpkg.arch.3.gz
-rw-r–r– 1 root root 43 Oct 14 2017 dpkg.arch.4.gz
-rw-r–r– 1 root root 43 Oct 14 2017 dpkg.arch.5.gz
-rw-r–r– 1 root root 43 Oct 14 2017 dpkg.arch.6.gz
-rw-r–r– 1 root root 1319 Apr 27 22:28 dpkg.diversions.0
-rw-r–r– 1 root root 387 Apr 27 22:28 dpkg.diversions.1.gz
-rw-r–r– 1 root root 387 Apr 27 22:28 dpkg.diversions.2.gz
-rw-r–r– 1 root root 387 Apr 27 22:28 dpkg.diversions.3.gz
-rw-r–r– 1 root root 387 Apr 27 22:28 dpkg.diversions.4.gz
-rw-r–r– 1 root root 387 Apr 27 22:28 dpkg.diversions.5.gz
-rw-r–r– 1 root root 387 Apr 27 22:28 dpkg.diversions.6.gz
-rw-r–r– 1 root root 375 Aug 23 2018 dpkg.statoverride.0
-rw-r–r– 1 root root 247 Aug 23 2018 dpkg.statoverride.1.gz
-rw-r–r– 1 root root 247 Aug 23 2018 dpkg.statoverride.2.gz
-rw-r–r– 1 root root 247 Aug 23 2018 dpkg.statoverride.3.gz
-rw-r–r– 1 root root 247 Aug 23 2018 dpkg.statoverride.4.gz
-rw-r–r– 1 root root 247 Aug 23 2018 dpkg.statoverride.5.gz
-rw-r–r– 1 root root 247 Aug 23 2018 dpkg.statoverride.6.gz
-rw-r–r– 1 root root 3363749 Sep 23 14:32 dpkg.status.0
-rw-r–r– 1 root root 763524 Aug 19 14:48 dpkg.status.1.gz
-rw-r–r– 1 root root 760198 Aug 17 19:41 dpkg.status.2.gz
-rw-r–r– 1 root root 760176 Aug 13 12:48 dpkg.status.3.gz
-rw-r–r– 1 root root 760105 Jul 16 15:25 dpkg.status.4.gz
-rw-r–r– 1 root root 759807 Jun 28 15:18 dpkg.status.5.gz
-rw-r–r– 1 root root 759554 May 28 16:22 dpkg.status.6.gz
drwx—— 2 root root 4096 Oct 15 2017 ejabberd-2017-10-15T00:22:30.p1e5J8/
drwx—— 2 root root 4096 Oct 15 2017 ejabberd-2017-10-15T00:24:02.dAUgDs/
drwx—— 2 root root 4096 Oct 15 2017 ejabberd-2017-10-15T12:29:51.FX27WJ/
drwx—— 2 root root 4096 Oct 15 2017 ejabberd-2017-10-15T21:18:26.bPQWlW/
drwx—— 2 root root 4096 Jul 16 2019 ejabberd-2019-07-16T00:49:52.Gy3sus/
-rw——- 1 root root 2512 Oct 20 2020 group.bak
-rw——- 1 root shadow 1415 Oct 20 2020 gshadow.bak
-rw——- 1 root root 7395 May 11 22:56 passwd.bak
-rw——- 1 root shadow 7476 May 11 22:56 shadow.bak

Considering the situation the important files for us that could, help us restore our previous list of packages, we had installed on the Debian are files under /var/backups/dpkg.status*

Luckily debian based systems keeps backups of its important files that can be used later on for system recovery activities.
Below is a common structure of /var/lib/dpkg on a deb based system.

hipo@debian:/home/hipo$ ls -l /var/lib/dpkg/
total 11504
drwxr-xr-x 2 root root 4096 Aug 19 14:33 alternatives/
-rw-r–r– 1 root root 11 Oct 14 2017 arch
-rw-r–r– 1 root root 2199402 Oct 19 2017 available
-rw-r–r– 1 root root 2197483 Oct 19 2017 available-old
-rw-r–r– 1 root root 8 Sep 6 2012 cmethopt
-rw-r–r– 1 root root 1319 Apr 27 22:28 diversions
-rw-r–r– 1 root root 1266 Nov 18 2020 diversions-old
drwxr-xr-x 2 root root 606208 Sep 23 14:32 info/
-rw-r—– 1 root root 0 Sep 23 14:32 lock
-rw-r—– 1 root root 0 Mar 18 2021 lock-frontend
drwxr-xr-x 2 root root 4096 Sep 17 2012 parts/
-rw-r–r– 1 root root 375 Aug 23 2018 statoverride
-rw-r–r– 1 root root 337 Aug 13 2018 statoverride-old
-rw-r–r– 1 root root 3363749 Sep 23 14:32 status
-rw-r–r– 1 root root 3363788 Sep 23 14:32 status-old
drwxr-xr-x 2 root root 4096 Aug 19 14:48 triggers/
drwxr-xr-x 2 root root 4096 Sep 23 14:32 updates/

2. Recreate basic /var/lib/dpkg directory and files structures

As you can see, there are 5 directories and the status file and some other files.
Hence the first step is to restore the lost directory structure.

hipo@debian: ~$ sudo mkdir -p /var/lib/dpkg/{alternatives,info,parts,triggers,updates}

3. Recover /var/lib/dpkg/status file

Further on recover the dpkg status file from backup

hipo@debian: ~$ sudo cp /var/backups/dpkg.status.0 /var/lib/dpkg/status

4. Check dpkg package installation works again and reinstall base-files

Next check if dpkg – debian package manager is now working, by simply trying to download dpkg*.deb reinstalling it.

root@debian:/root # apt-get download dpkg
# sudo dpkg -i dpkg*.deb

If you get no errors next step is to reinstall base-files which is important package on which dpkg depends.

root@debian:/root # apt-get download base-files
…
root@debian:/root # sudo dpkg -i base-files*.deb
…

5. Update deb system package list and db consistency

Onwards try to update system package list and check dpkg / apt database consistency.

root@debian:/root # dpkg –audit
…
root@debian:/root # sudo apt-get update
…
root@debian:/root # sudo apt-get check
…

The result should be more of the files in /var/lib/dpkg should appear, thus list the directory again and compare to the earlier given list of it, they should be similar.

root@debian:/root # ls -l /var/lib/dpkg
…

6. Reinstall completely from source code dpkg, if nothing else works

If some files are missing they should get created with a normal daily sysadmin package management tasks so no worries.

In case if after attempting to upgrade the system or install a package with apt, you get some nasty error like:

'/usr/local/var/lib/dpkg/status' for reading: No such file or directory

Then the next and final thing to try as a recovery is to download compile from a new and reinstall dpkg from source code!

root@debian:/ # wget https://launchpad.net/ubuntu/+archive/primary/+files/dpkg_1.16.1.2ubuntu7.2.tar.bz2
root@debian:/ # tar -xvf dpkg_1.16*
…
root@debian:/ # cd dpkg-1.16*
…
root@debian:/ # ./configure
…
root@debian:/ # make
…
root@debian:/ # make install
…

Hopefully you'll have gcc and development tools provided by build-essential .deb package , otherwise you have to download and compile this ones as well 🙂
If this doesn't bring you back the installed packages you had priorly (hopefully not), then waste no more time and do a backup of the main things on the server, and reinstall it completely.

The moral out of this incident is always to implement always to your system a good back up system and regularly create backups of /var/lib/dpkg , /etc/ , /usr/local* and other important files on a remote backup server, to be able to easily recover if you do by mistake something whacky.

Hope that helped anyone. Cheers 🙂

Tags: Aug, Debian Ubuntu Linux, dpkg, gz, lib, Oct, package management, reading, recovery, root root, status

Tags: Aug, Debian Ubuntu Linux, dpkg, gz, lib, Oct, package management, reading, recovery, root root, status
Posted in Linux, Linux and FreeBSD Desktop, System Administration | No Comments »

Install and enable Sysstats IO / DIsk / CPU / Network monitoring console suite on Redhat 8.3, Few sar useful command examples

Tuesday, September 28th, 2021

Why to monitoring CPU, Memory, Hard Disk, Network usage etc. with sysstats tools?

Using system monitoring tools such as Zabbix, Nagios Monit is a good approach, however sometimes due to zabbix server interruptions you might not be able to track certain aspects of system performance on time. Thus it is always a good idea to
Gain more insights on system peroformance from command line. Of course there is cmd tools such as iostat and top, free, vnstat that provides plenty of useful info on system performance issues or bottlenecks. However from my experience to have a better historical data that is systimized and all the time accessible from console it is a great thing to have sysstat package at place. Since many years mostly on every server I administer, I've been using sysstats to monitor what is going on servers over a short time frames and I'm quite happy with it. In current company we're using Redhats and CentOS-es and I had to install sysstats on Redhat 8.3. I've earlier done it multiple times on Debian / Ubuntu Linux and while I've faced on some .deb distributions complications of making sysstat collect statistics I've come with an article on Howto fix sysstat Cannot open /var/log/sysstat/sa no such file or directory” on Debian / Ubuntu Linux

Sysstat contains the following tools related to collecting I/O and CPU statistics:
iostat
Displays an overview of CPU utilization, along with I/O statistics for one or more disk drives.
mpstat
Displays more in-depth CPU statistics.
Sysstat also contains tools that collect system resource utilization data and create daily reports based on that data. These tools are:
sadc
Known as the system activity data collector, sadc collects system resource utilization information and writes it to a file.
sar
Producing reports from the files created by sadc, sar reports can be generated interactively or written to a file for more intensive analysis.

My experience with CentOS 7 and Fedora to install sysstat it was pretty straight forward, I just had to install it via yum install sysstat wait for some time and use sar (System Activity Reporter) tool to report collected system activity info stats over time.
Unfortunately it seems on RedHat 8.3 as well as on CentOS 8.XX instaling sysstats does not work out of the box.

To complete a successful installation of it on RHEL 8.3, I had to:

[root@server ~]# yum install -y sysstat

To make sysstat enabled on the system and make it run, I've enabled it in sysstat

[root@server ~]# systemctl enable sysstat

Running immediately sar command, I've faced the shitty error:

“Cannot open /var/log/sysstat/sa18:
No such file or directory. Please check if data collecting is enabled”

Once installed I've waited for about 5 minutes hoping, that somehow automatically sysstat would manage it but it didn't.

To solve it, I've had to create additionally file /etc/cron.d/sysstat (weirdly RPM's post install instructions does not tell it to automatically create it)

[root@server ~]# vim /etc/cron.d/sysstat

# run system activity accounting tool every 10 minutes
0 * * * * root /usr/lib64/sa/sa1 60 59 &
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A &

/usr/local/lib/sa1 is a shell script that we can use for scheduling cron which will create daily binary log file.
/usr/local/lib/sa2 is a shell script will change binary log file to human-readable form.

[root@server ~]# chmod 600 /etc/cron.d/sysstat

[root@server ~]# systemctl restart sysstat

In a while if sysstat is working correctly you should get produced its data history logs inside /var/log/sa

[root@server ~]# ls -al /var/log/sa

Note that the standard sysstat history files on Debian and other modern .deb based distros such as Debian 10 (in y.2021) is stored under /var/log/sysstat

Here is few useful uses of sysstat cmds

1. Check with sysstat machine history SWAP and RAM Memory use

To lets say check last 10 minutes SWAP memory use:

[hipo@server yum.repos.d] $ sar -W |last -n 10

Linux 4.18.0-240.el8.x86_64 (server) 09/28/2021 _x86_64_ (8 CPU)

12:00:00 AM pswpin/s pswpout/s
12:00:01 AM 0.00 0.00
12:01:01 AM 0.00 0.00
12:02:01 AM 0.00 0.00
12:03:01 AM 0.00 0.00
12:04:01 AM 0.00 0.00
12:05:01 AM 0.00 0.00
12:06:01 AM 0.00 0.00

[root@ccnrlb01 ~]# sar -r | tail -n 10
14:00:01 93008 1788832 95.06 0 1357700 725740 9.02 795168 683484 32
14:10:01 78756 1803084 95.81 0 1358780 725740 9.02 827660 652248 16
14:20:01 92844 1788996 95.07 0 1344332 725740 9.02 813912 651620 28
14:30:01 92408 1789432 95.09 0 1344612 725740 9.02 816392 649544 24
14:40:01 91740 1790100 95.12 0 1344876 725740 9.02 816948 649436 36
14:50:01 91688 1790152 95.13 0 1345144 725740 9.02 817136 649448 36
15:00:02 91544 1790296 95.14 0 1345448 725740 9.02 817472 649448 36
15:10:01 91108 1790732 95.16 0 1345724 725740 9.02 817732 649340 36
15:20:01 90844 1790996 95.17 0 1346000 725740 9.02 818016 649332 28
Average: 93473 1788367 95.03 0 1369583 725074 9.02 800965 671266 29

2. Check system load? Are my processes waiting too long to run on the CPU?

[root@server ~ ]# sar -q |head -n 10
Linux 4.18.0-240.el8.x86_64 (server) 09/28/2021 _x86_64_ (8 CPU)

12:00:00 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
12:00:01 AM 0 272 0.00 0.02 0.00 0
12:01:01 AM 1 271 0.00 0.02 0.00 0
12:02:01 AM 0 268 0.00 0.01 0.00 0
12:03:01 AM 0 268 0.00 0.00 0.00 0
12:04:01 AM 1 271 0.00 0.00 0.00 0
12:05:01 AM 1 271 0.00 0.00 0.00 0
12:06:01 AM 1 265 0.00 0.00 0.00 0

3. Show various CPU statistics per CPU use

On a multiprocessor, multi core server sometimes for scripting it is useful to fetch processor per use historic data,
this can be attained with:

[hipo@server ~ ] $ mpstat -P ALL
Linux 4.18.0-240.el8.x86_64 (server) 09/28/2021 _x86_64_ (8 CPU)

06:08:38 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:08:38 PM all 0.17 0.02 0.25 0.00 0.05 0.02 0.00 0.00 0.00 99.49
06:08:38 PM 0 0.22 0.02 0.28 0.00 0.06 0.03 0.00 0.00 0.00 99.39
06:08:38 PM 1 0.28 0.02 0.36 0.00 0.08 0.02 0.00 0.00 0.00 99.23
06:08:38 PM 2 0.27 0.02 0.31 0.00 0.06 0.01 0.00 0.00 0.00 99.33
06:08:38 PM 3 0.15 0.02 0.22 0.00 0.03 0.01 0.00 0.00 0.00 99.57
06:08:38 PM 4 0.13 0.02 0.20 0.01 0.03 0.01 0.00 0.00 0.00 99.60
06:08:38 PM 5 0.14 0.02 0.27 0.00 0.04 0.06 0.01 0.00 0.00 99.47
06:08:38 PM 6 0.10 0.02 0.17 0.00 0.04 0.02 0.00 0.00 0.00 99.65
06:08:38 PM 7 0.09 0.02 0.15 0.00 0.02 0.01 0.00 0.00 0.00 99.70

sar-sysstat-cpu-statistics-screenshot

Monitor processes and threads currently being managed by the Linux kernel.

[hipo@server ~ ] $ pidstat

[hipo@server ~ ] $ pidstat -d 2

pidstat-show-processes-with-most-io-activities-linux-screenshot

This report tells us that there is few processes with heave I/O use Filesystem system journalling daemon jbd2, apache, mysqld and supervise, in 3rd column you see their respective PID IDs.

To show threads used inside a process (like if you press SHIFT + H) inside Linux top command:

[hipo@server ~ ] $ pidstat -t -p 10765 1 3

Linux 4.19.0-14-amd64 (server) 28.09.2021 _x86_64_ (10 CPU)

21:41:22 UID TGID TID %usr %system %guest %wait %CPU CPU Command
21:41:23 108 10765 – 1,98 0,99 0,00 0,00 2,97 1 mysqld
21:41:23 108 – 10765 0,00 0,00 0,00 0,00 0,00 1 |__mysqld
21:41:23 108 – 10768 0,00 0,00 0,00 0,00 0,00 0 |__mysqld
21:41:23 108 – 10771 0,00 0,00 0,00 0,00 0,00 5 |__mysqld
21:41:23 108 – 10784 0,00 0,00 0,00 0,00 0,00 7 |__mysqld
21:41:23 108 – 10785 0,00 0,00 0,00 0,00 0,00 6 |__mysqld
21:41:23 108 – 10786 0,00 0,00 0,00 0,00 0,00 2 |__mysqld
…

10765 – is the Process ID whose threads you would like to list

With pidstat, you can further monitor processes for memory leaks with:

[hipo@server ~ ] $ pidstat -r 2

4. Report paging statistics for some old period

[root@server ~ ]# sar -B -f /var/log/sa/sa27 |head -n 10
Linux 4.18.0-240.el8.x86_64 (server) 09/27/2021 _x86_64_ (8 CPU)

15:42:26 LINUX RESTART (8 CPU)

15:55:30 LINUX RESTART (8 CPU)

04:00:01 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
04:01:01 PM 0.00 14.47 629.17 0.00 502.53 0.00 0.00 0.00 0.00
04:02:01 PM 0.00 13.07 553.75 0.00 419.98 0.00 0.00 0.00 0.00
04:03:01 PM 0.00 11.67 548.13 0.00 411.80 0.00 0.00 0.00 0.00

5. Monitor Received RX and Transmitted TX network traffic perl Network interface real time

To print out Received and Send traffic per network interface 4 times in a raw

sar-sysstats-network-traffic-statistics-screenshot

[hipo@server ~ ] $ sar -n DEV 1 4

To continusly monitor all network interfaces I/O traffic

[hipo@server ~ ] $ sar -n DEV 1

To only monitor a certain network interface lets say loopback interface (127.0.0.1) received / transmitted bytes

[hipo@server yum.repos.d] $ sar -n DEV 1 2|grep -i lo
06:29:53 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
06:29:54 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

6. Monitor block devices use

To check block devices use 3 times in a raw

[hipo@server yum.repos.d] $ sar -d 1 3

sar-sysstats-blockdevice-statistics-screenshot

7. Output server monitoring data in CSV database structured format

For preparing a nice graphs with Excel from CSV strucuted file format, you can dump the collected data as so:

[root@server yum.repos.d]# sadf -d /var/log/sa/sa27 — -n DEV | grep -v lo|head -n 10
server-name-fqdn;-1;2021-09-27 13:42:26 UTC;LINUX-RESTART (8 CPU)
# hostname;interval;timestamp;IFACE;rxpck/s;txpck/s;rxkB/s;txkB/s;rxcmp/s;txcmp/s;rxmcst/s;%ifutil
server-name-fqdn;-1;2021-09-27 13:55:30 UTC;LINUX-RESTART (8 CPU)
# hostname;interval;timestamp;IFACE;rxpck/s;txpck/s;rxkB/s;txkB/s;rxcmp/s;txcmp/s;rxmcst/s;%ifutil
server-name-fqdn;60;2021-09-27 14:01:01 UTC;eth1;19.42;16.12;1.94;1.68;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:01:01 UTC;eth0;7.18;9.65;0.55;0.78;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:01:01 UTC;eth2;5.65;5.13;0.42;0.39;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:02:01 UTC;eth1;18.90;15.55;1.89;1.60;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:02:01 UTC;eth0;7.15;9.63;0.55;0.74;0.00;0.00;0.00;0.00
server-name-fqdn;60;2021-09-27 14:02:01 UTC;eth2;5.67;5.15;0.42;0.39;0.00;0.00;0.00;0.00
…

To graph the output data you can use Excel / LibreOffice's Excel equivalent Calc or if you need to dump a CSV sar output and generate it on the fly from a script use gnuplot

What we've learned?

How to install and enable on cron sysstats on Redhat and CentOS 8 Linux ?
How to continuously monitor CPU / Disk and Network, block devices, paging use and processes and threads used by the kernel per process ?
As well as how to export previously collected data to CSV to import to database or for later use inrder to generate graphic presentation of data.
Cheers ! 🙂

Tags: access, cmds, com, command, console, cron, data, Debian Ubuntu Linux, eth0, eth1, eth2, How to, Install, installation, iowait, Output, period, Redhat, root, servers, sysstat, systemctl, timestamp, use, usr

How to redirect TCP port traffic from Internet Public IP host to remote local LAN server, Redirect traffic for Apache Webserver, MySQL, or other TCP service to remote host

Thursday, September 23rd, 2021

1. Use the good old times rinetd – internet “redirection server” service

Perhaps, many people who are younger wouldn't remember rinetd's use was pretty common on old Linuxes in the age where iptables was not on the scene and its predecessor ipchains was so common.
In the raise of mass internet rinetd started loosing its popularity because the service was exposed to the outer world and due to security holes and many exploits circulating the script kiddie communities
many servers get hacked "pwned" in the jargon of the script kiddies.

rinetd is still available even in modern Linuxes and over the last years I did not heard any severe security concerns regarding it, but the old paranoia perhaps and the set to oblivion makes it still unpopular soluttion for port redirect today in year 2021.
However for a local secured DMZ lans I can tell you that its use is mostly useful and I chooes to use it myself, everynow and then due to its simplicity to configure and use.
rinetd is pretty standard among unixes and is also available in old Sun OS / Solaris and BSD-es and pretty much everything on the Unix scene.

Below is excerpt from 'man rinetd':

DESCRIPTION
rinetd redirects TCP connections from one IP address and port to another. rinetd is a single-process server which handles any number of connections to the address/port pairs
specified in the file /etc/rinetd.conf. Since rinetd runs as a single process using nonblocking I/O, it is able to redirect a large number of connections without a severe im‐
pact on the machine. This makes it practical to run TCP services on machines inside an IP masquerading firewall. rinetd does not redirect FTP, because FTP requires more than
one socket. rinetd is typically launched at boot time, using the following syntax: /usr/sbin/rinetd The configuration file is found in the file /etc/rinetd.conf, unless another file is specified using the -c command line option.

To use rinetd on any LInux distro you have to install and enable it with apt or yum as usual. For example on my Debian GNU / Linux home machine to use it I had to install .deb package, enable and start it it via systemd :

server:~# apt install –yes rinetd
…

server:~# systemctl enable rinetd

server:~# systemctl start rinetd

server:~# systemctl status rinetd
● rinetd.service
Loaded: loaded (/etc/init.d/rinetd; generated)
Active: active (running) since Tue 2021-09-21 10:48:20 EEST; 2 days ago
Docs: man:systemd-sysv-generator(8)
Tasks: 1 (limit: 4915)
Memory: 892.0K
CGroup: /system.slice/rinetd.service
└─1364 /usr/sbin/rinetd

rinetd is doing the traffic redirect via a separate process daemon, in order for it to function once you have service up check daemon is up as well.

root@server:/home/hipo# ps -ef|grep -i rinet
root 359 1 0 16:10 ? 00:00:00 /usr/sbin/rinetd
root 824 26430 0 16:10 pts/0 00:00:00 grep -i rinet

+ Configuring a new port redirect with rinetd

Is pretty straight forward everything is handled via one single configuration – /etc/rinetd.conf

The format (syntax) of a forwarding rule is as follows:

[bindaddress] [bindport] [connectaddress] [connectport]

Besides that rinetd , could be used as a primitive firewall substitute to iptables, general syntax of allow deny an IP address is done with (allow, deny) keywords:

allow 192.168.2.*
deny 192.168.2.1?

To enable logging to external file ,you'll have to include in the configuration:

# logging information
logfile /var/log/rinetd.log

Here is an example rinetd.conf configuration, redirecting tcp mysql 3306, nginx on port 80 and a second web service frontend for ILO to server reachable via port 8888 and a redirect from External IP to local IP SMTP server.

#
# this is the configuration file for rinetd, the internet redirection server
#
# you may specify global allow and deny rules here
# only ip addresses are matched, hostnames cannot be specified here
# the wildcards you may use are * and ?
#
# allow 192.168.2.*
# deny 192.168.2.1?

#
# forwarding rules come here
#
# you may specify allow and deny rules after a specific forwarding rule
# to apply to only that forwarding rule
#
# bindadress bindport connectaddress connectport

# logging information
logfile /var/log/rinetd.log
83.228.93.76 80 192.168.0.20 80
192.168.0.2 3306           192.168.0.19 3306
83.228.93.76 443 192.168.0.20 443
# enable for access to ILO
83.228.93.76       8888           192.168.1.25 443
127.0.0.1   25   192.168.0.19 25

83.228.93.76 is my external ( Public ) IP internet address where 192.168.0.20, 192.168.0.19, 192.168.0.20 (are the DMZ-ed Lan internal IPs) with various services.

To identify the services for which rinetd is properly configured to redirect / forward traffic you can see it with netstat or the newer ss command

root@server:/home/hipo# netstat -tap|grep -i rinet
tcp 0 0 www.pc-freak.net:8888 0.0.0.0:* LISTEN 13511/rinetd
tcp 0 0 www.pc-freak.n:http-alt 0.0.0.0:* LISTEN 21176/rinetd
tcp 0 0 www.pc-freak.net:443 0.0.0.0:* LISTEN 21176/rinetd
…

+ Using rinetd to redirect External interface IP to loopback's port (127.0.0.1)

If you have the need to redirect an External connectable living service be it apache mysql / privoxy / squid or whatever rinetd is perhaps the tool of choice (especially since there is no way to do it with iptables.

If you want to redirect all traffic which is accessed via Linux's loopback interface (localhost) to be reaching a remote host 11.5.8.1 on TCP port 1083 and 1888, use below config

# bindadress bindport connectaddress connectport
11.5.8.1 1083 127.0.0.1 1083
11.5.8.1 1888 127.0.0.1 1888

For a quick and dirty solution to redirect traffic rinetd is very useful, however you'll have to keep in mind that if you want to redirect traffic for tens of thousands of connections constantly originating from the internet you might end up with some disconnects as well as notice a increased use of rinetd CPU use with the incrased number of forwarded connections.

2. Redirect TCP / IP port using DNAT iptables firewall rules

Lets say you have some proxy, webservice or whatever service running on port 5900 to be redirected with iptables.
The easeiest legacy way is to simply add the redirection rules to /etc/rc.local. In newer Linuxes rc.local so if you decide to use,
you'll have to enable rc.local , I've written earlier a short article on how to enable rc.local on newer Debian, Fedora, CentOS

# redirect 5900 TCP service
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -I PREROUTING -p tcp –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -I OUTPUT -p tcp -o lo –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp –dport 5900 -j DNAT –to-destination 192.168.1.8:5900
iptables -t nat -I OUTPUT –source 0/0 –destination 0/0 -p tcp –dport 5900 -j REDIRECT –to-ports 5900

Here is another two example which redirects port 2208 (which has configured a bind listener for SSH on Internal host 192.168.0.209:2208) from External Internet IP address (XXX.YYY.ZZZ.XYZ)

# Port redirect for SSH to VM on openxen internal Local lan server 192.168.0.209
-A PREROUTING -p tcp –dport 2208 -j DNAT –to-destination 192.168.0.209:2208
-A POSTROUTING -p tcp –dst 192.168.0.209 –dport 2208 -j SNAT –to-source 83.228.93.76

3. Redirect TCP traffic connections with redir tool

If you look for an easy straight forward way to redirect TCP traffic, installing and using redir (ready compiled program) might be a good idea.

root@server:~# apt-cache show redir|grep -i desc -A5 -B5
Version: 3.2-1
Installed-Size: 60
Maintainer: Lucas Kanashiro <kanashiro@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.15)
Description-en: Redirect TCP connections
It can run under inetd or stand alone (in which case it handles multiple
connections). It is 8 bit clean, not limited to line mode, is small and
light. Supports transparency, FTP redirects, http proxying, NAT and bandwidth
limiting.
.
redir is all you need to redirect traffic across firewalls that authenticate
based on an IP address etc. No need for the firewall toolkit. The
functionality of inetd/tcpd and "redir" will allow you to do everything you
need without screwy telnet/ftp etc gateways. (I assume you are running IP
Masquerading of course.)
Description-md5: 2089a3403d126a5a0bcf29b22b68406d
Homepage: https://github.com/troglobit/redir
Tag: interface::daemon, network::server, network::service, role::program,
use::proxying
Section: net
Priority: optional

server:~# apt-get install –yes redir
…

Here is a short description taken from its man page 'man redir'

DESCRIPTION
redir redirects TCP connections coming in on a local port, [SRC]:PORT, to a specified address/port combination, [DST]:PORT. Both the SRC and DST arguments can be left out,
redir will then use 0.0.0.0.

redir can be run either from inetd or as a standalone daemon. In –inetd mode the listening SRC:PORT combo is handled by another process, usually inetd, and a connected
socket is handed over to redir via stdin. Hence only [DST]:PORT is required in –inetd mode. In standalone mode redir can run either in the foreground, -n, or in the back‐
ground, detached like a proper UNIX daemon. This is the default. When running in the foreground log messages are also printed to stderr, unless the -s flag is given.

Depending on how redir was compiled, not all options may be available.

+ Use redir to redirect TCP traffic one time

Lets say you have a MySQL running on remote machine on some internal or external IP address, lets say 192.168.0.200 and you want to redirect all traffic from remote host to the machine (192.168.0.50), where you run your Apache Webserver, which you want to configure to use
as MySQL localhost TCP port 3306.
Assuming there are no irewall restrictions between Host A (192.168.0.50) and Host B (192.168.0.200) is already permitting connectivity on TCP/IP port 3306 between the two machines.

To open redirection from localhost on 192.168.0.50 -> 192.168.0.200:

server:~# redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306

If you need other third party hosts to be additionally reaching 192.168.0.200 via 192.168.0.50 TCP 3306.

root@server:~# redir –laddr=192.168.0.50 –lport=3306 –caddr=192.168.0.200 –cport=3306

Of course once you close, the /dev/tty or /dev/vty console the connection redirect will be cancelled.

+ Making TCP port forwarding from Host A to Host B permanent

One solution to make the redir setup rules permanent is to use –rinetd option or simply background the process, nevertheless I prefer to use instead GNU Screen.
If you don't know screen is a vVrtual Console Emulation manager with VT100/ANSI terminal emulation to so, if you don't have screen present on the host install it with whatever Linux OS package manager is present and run:

root@server:~#screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306'

That would run it into screen session and detach so you can later connect, if you want you can make redir to also log connections via syslog with ( -s) option.

I found also useful to be able to track real time what's going on currently with the opened redirect socket by changing redir log level.

Accepted log level is:

-l, –loglevel=LEVEL
Set log level: none, err, notice, info, debug. Default is notice.

root@server:/ # screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3308 –caddr=192.168.0.200 –cport=3306 -l debug'

To test connectivity works as expected use telnet:

root@server:/ # telnet localhost 3308
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
g
5.5.5-10.3.29-MariaDB-0+deb10u1-log�+c2nWG>B��o+#ly=bT^]79mysql_native_password

6#HY000Proxy header is not accepted from 192.168.0.19 Connection closed by foreign host.

once you attach to screen session with

root@server:/home # screen -r

You will get connectivity attempt from localhost logged : .

redir[10640]: listening on 127.0.0.1:3306
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10793]: peer IP is 127.0.0.1
redir[10793]: peer socket is 25592
redir[10793]: target IP address is 192.168.0.200
redir[10793]: target port is 3306
redir[10793]: Connecting 127.0.0.1:25592 to 127.0.0.1:3306
redir[10793]: Entering copyloop() – timeout is 0
redir[10793]: Disconnect after 1 sec, 165 bytes in, 4 bytes out

The downsides of using redir is redirection is handled by the separate process which is all time hanging in the process list, as well as the connection redirection speed of incoming connections might be about at least 30% slower to if you simply use a software (firewall ) redirect such as iptables. If you use something like kernel IP set ( ipsets ). If you hear of ipset for a first time and you wander whta it is below is short package description.

root@server:/root# apt-cache show ipset|grep -i description -A13 -B5
Maintainer: Debian Netfilter Packaging Team <pkg-netfilter-team@lists.alioth.debian.org>
Architecture: amd64
Provides: ipset-6.38
Depends: iptables, libc6 (>= 2.4), libipset11 (>= 6.38-1~)
Breaks: xtables-addons-common (<< 1.41~)
Description-en: administration tool for kernel IP sets
IP sets are a framework inside the Linux 2.4.x and 2.6.x kernel which can be
administered by the ipset(8) utility. Depending on the type, currently an
IP set may store IP addresses, (TCP/UDP) port numbers or IP addresses with
MAC addresses in a way which ensures lightning speed when matching an
entry against a set.
.
If you want to
.
* store multiple IP addresses or port numbers and match against the
entire collection using a single iptables rule.
* dynamically update iptables rules against IP addresses or ports without
performance penalty.
* express complex IP address and ports based rulesets with a single
iptables rule and benefit from the speed of IP sets.
.
then IP sets may be the proper tool for you.
Description-md5: d87e199641d9d6fbb0e52a65cf412bde
Homepage: http://ipset.netfilter.org/
Tag: implemented-in::c, role::program
Section: net
Priority: optional
Filename: pool/main/i/ipset/ipset_6.38-1.2_amd64.deb
Size: 50684
MD5sum: 095760c5db23552a9ae180bd58bc8efb
SHA256: 2e2d1c3d494fe32755324bf040ffcb614cf180327736c22168b4ddf51d462522

Tags: amd64, Below, CentOS, Configuring, Connected, External, How to, incoming connections, information, ip addresses, lan, network server, package description, redirection, remote server, server port, server root, standard, Supports, systemd, tcp, use

Change Windows 10 default lock screen image via win registry LockScreenImage key change

Tuesday, September 21st, 2021

If you do work for a corporation on a Windows machine that is part of Windows Active Directory domain or a Microsoft 365 environment and your Domain admimistrator after some of the scheduled updates. Has enforced a Windows lock screen image change.
You might be surprised to have some annoying corporation logo picture shown as a default Lock Screen image on your computer on next reoboot. Perhaps for some people it doesn't matter but for as a person who seriously like customizations, and a valuer of
freedom having an enforced picture logo each time I press CTRL + L (To lock my computer) is really annoying.

The logical question hence was how to reverse my desired image as a default lock screen to enkoy. Some would enjoy some relaxing picture of a Woods, Cave or whatever Natural place landscape. I personally prefer simplicity so I simply use a simple purely black
background.

To do it you'll have anyways to have some kind of superuser access to the computer. At the company I'm epmloyeed, it is possible to temporary request Administrator access this is done via a software installed on the machine. So once I request it I become
Administratof of machine for 20 minutes. In that time I do used a 'Run as Administartor' command prompt cmd.exe and inside Windows registry do the following Registry change.

The first logical thing to do is to try to manually set the picture via:

Settings -> Lock Screen

But unfortunately as you can see in below screenshot, there was no way to change the LockScreen background image.

Windows-settings-lockscreen-screenshot

In Windows Registry Editor

I had to go to registry path

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\.]

And from there in create a new "String Value" key

"LockScreenImage"

so full registry key path should be equal to:

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Personalization\LockScreenImage]"

The value to set is:

C:\Users\a768839\Desktop\var-stuff\background\Desired-background-picture.jpg

windows-registry-change-lock-screen-background-picture-from-registry-screenshot

If you want to set a black background picture for LockScreen like me you can download my black background picture from here.

That's all press CTRL + L key combination and the black screen background lock screen picture will appear !

Hopefully the Domain admin, would not soon enforce some policty to update the registry keys or return your old registry database from backup if something crashs out with something strange to break just set configuration.

To test whether the setting will stay permanent after the next scheduled Windows PC update of policies enforced by the Active Directory (AD) sysadmin, run manually from CMD.EXE

C:\> gpupdate /force

The command will download latest policies from Windows Domain, try to lock the screen once again with Control + L, if the background picture is still there most likely the Picture change would stay for a long.
If you get again the corporation preset domain background instead, you're out of luck and will have to follow the same steps every, now and then after a domani policy update.

Enjoy your new smooth LockScreen Image 🙂

Tags: access, active directory, AD, admin, administrator, after, again, ALL, and, annoying, Anyways, background, Below, black, break, cave, change, change windows, default, screen

Tags: access, active directory, AD, admin, administrator, after, again, ALL, and, annoying, Anyways, background, Below, black, break, cave, change, change windows, default, screen
Posted in System Administration | No Comments »

Defining multiple short Server Hostname aliases via SSH config files and defining multiple ssh options for it, Use passwordless authentication via public keys

Thursday, September 16th, 2021

In case you have to access multiple servers from your terminal client such as gnome-terminal, kterminal (if on Linux) or something such as mobaxterm + cygwin (if on Windows) with an opens ssh client (ssh command). There is a nifty trick to save time and keyboard typing through creating shortcuts aliases by adding few definitions inside your $HOME/.ssh/config ( ~/.ssh/config ) for your local non root user or even make the configuration system wide (for all existing local /etc/passwd users) via /etc/ssh/ssh_config.
By adding a pseudonym alias for each server it makes sysadmin life much easier as you don't have to type in each time the FQDN (Fully Qualified Domain Name) hostname of remote accessed Linux / Unix / BSD / Mac OS or even Windows sshd ready hosts accessible via remote TCP/IP port 22.

1. Adding local user remote server pointer aliases via ~/.ssh/config

The file ~/.ssh/config is read by the ssh client part of the openssh-client (Linux OS package) on each invokement of the client, and besides defining a pseudonym for the hosts you like to save you time when accessing remote host and hence increase your productivity. Moreover you can also define various other nice options through it to define specifics of remote ssh session for each desired host such as remote host default SSH port (for example if your OpenSSHD is configured to run on non-standard SSH port as lets say 2022 instead of default port TCP 22 for some reason, e.g. security through obscurity etc.).

The general syntax of .ssh/config file si simplistic, it goes like this:

Host MACHNE_HOSTNAME

SSH_OPTION1 value1
SSH_OPTION1 value1 value2
SSH_OPTION2 value1 value2

Host MACHNE_HOSTNAME

SSH_OPTION value
SSH_OPTION1 value1 value2

Another understood syntax if you prefer to not have empty whitespaces is to use ( = )
between the parameter name and values.

Host MACHINE_HOSTNAME
SSH_config=value
SSH_config1=value1 value2

All empty lines and lines starting with the hash shebang sign ( # ) would be ignored.
All values are case-sensitive, but parameter names are not.

If you have never so far used the $HOME/.ssh/config you would have to create the file and set the proper permissions to it like so:

mkdir -p $HOME/.ssh
chmod 0700 $HOME/.ssh

Below are examples taken from my .ssh/config configuration for all subdomains for my pcfreak.org domain

# Ask for password for every subdomain under pc-freak.net for security
Host *.pcfreak.org
user hipopo
passwordauthentication yes
StrictHostKeyChecking no

# ssh public Key authentication automatic login
Host www1.pc-freak.net
user hipopo
Port 22
passwordauthentication no
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Host haproxy2
Hostname 213.91.190.233
User root
Port 2218
  PubkeyAuthentication yes
IdentityFile ~/.ssh/haproxy2.pub
  StrictHostKeyChecking no
LogLevel INFO

Host pcfrxenweb
Hostname 83.228.93.76
User root
Port 2218
  PubkeyAuthentication yes
IdentityFile ~/.ssh/pcfrxenweb.key
  StrictHostKeyChecking no

Host pcfreak-sf
Hostname 91.92.15.51
User root
Port 2209
  PreferredAuthentications password
  StrictHostKeyChecking no
Compression yes

As you can see from above configuration the Hostname could be referring either to IP address or to Hostname.

Now to connect to defined IP 91.92.15.51 you can simply refer to its alias

$ ssh pcfreak-sf -v

and you end up into the machine ssh on port 2209 and you will be prompted for a password.

$ ssh pcfrxenweb -v

would lead to IP 83.228.93.76 SSH on Port 2218 and will use the defined public key for a passwordless login and will save you the password typing each time.

Above ssh command is a short alias you can further use instead of every time typing:

$ ssh -i ~/.ssh/pcfrxenweb.key -p 2218 root@83.228.93.76

There is another nifty trick worthy to mention, if you have a defined hostname such as the above config haproxy2 to use a certain variables, but you would like to override some option for example you don't want to connet by default with User root, but some other local account, lets say ssh as devuser@haproxy2 you can type:

$ ssh -o "User=dev" devuser

StrictHostKeyChecking no

– variable will instruct the ssh to not check if the finger print of remote host has changed. Usually this finger print check sum changes in case if for example for some reason the opensshd gets updated or the default /etc/ssh/ssh_host_dsa_key /etc/ssh/sshd_host_dsa_* files have changed due to some reason.
Of course you should use this option only if you tend to access your remote host via a secured VPN or local network, otherwise the Host Key change could be an indicator someone is trying to intercept your ssh session.

Compression yes

– variable enables compression of connection saves few bits was useful in the old modem telephone lines but still could save you few bits
It is also possible to define a full range of IP addresses to be accessed with one single public rsa / dsa key

Below .ssh/config

Host 192.168.5.?
Hostname 192.168.2.18
User admin
IdentityFile ~/.ssh/id_ed25519.pub

Would instruct each host attemted to be reached in the IP range of 192.168.2.1-254 to be automatically reachable by default with ssh client with admin user and the respective ed25519.pub key.

$ ssh 192.168.1.[1-254] -v

2. Adding ssh client options system wide for all existing local or remote LDAP login users

The way to add any Host block is absolutely the same as with a default user except you need to add the configuration to /etc/ssh/ssh_config. Here is a confiugaration from mine Latest Debian Linux

$ cat /etc/ssh/ssh_config

# This is the ssh client system-wide configuration file. See
# ssh_config(5) for more information. This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
# 1. command line options
# 2. user-specific file
# 3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options. For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
# ForwardAgent no
# ForwardX11 no
# ForwardX11Trusted yes
# PasswordAuthentication yes
# HostbasedAuthentication no
# GSSAPIAuthentication no
# GSSAPIDelegateCredentials no
# GSSAPIKeyExchange no
# GSSAPITrustDNS no
# BatchMode no
# CheckHostIP yes
# AddressFamily any
# ConnectTimeout 0
# StrictHostKeyChecking ask
# IdentityFile ~/.ssh/id_rsa
# IdentityFile ~/.ssh/id_dsa
# IdentityFile ~/.ssh/id_ecdsa
# IdentityFile ~/.ssh/id_ed25519
# Port 22
# Protocol 2
# Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
# MACs hmac-md5,hmac-sha1,umac-64@openssh.com
# EscapeChar ~
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes

As you can see pretty much can be enabled by default such as the forwarding of the Authentication agent option ( -A ) option, necessery for some Company server environments to be anbled. So if you have to connect to remote host with enabled Agent Forwarding instead of typing

ssh -A user@remotehostname

To enable Agent Forwarding instead of

ssh -X user@remotehostname

Simply uncomment and set to yes

ForwardX11 yes
ForwardX11Trusted yes

Just simply uncomment above's config ForwardAgent no

As you can see ssh could do pretty much, you can configure enable SSH Tunneling or run via a Proxy with the ProxyCommand (If it is the first time you hear about ProxyCommand I warmly recommend you check my previous article – How to pass SSH traffic through a secured Corporate Proxy Server with corkscrew).

Sometimes for a defines hostname, due to changes on remote server ssh configuration, SSH encryption type or a host key removal you might end up with issues connecting, therefore to override all the previously defined options inside .ssh/config by ignoring the configuration with -F /dev/null

$ ssh -F /dev/null user@freak -v

What we learned ?

To sum it up In this article, we have learned how to easify the stressed sysadmin life, by adding Aliases with certain port numbering and configurations for different remote SSH administrated Linux / Unix, hosts via local ~/.ssh/config or global wide /etc/ssh/ssh_config configuration options, as well as how already applied configuration from ~/.ssh/config affecting each user ssh command execution, could be overriden.

Tags: authentication, Below, Ciphers, configuration file, default user, machines, options, server hostname, ssh client, ssh command, sshd, time

Tags: authentication, Below, Ciphers, configuration file, default user, machines, options, server hostname, ssh client, ssh command, sshd, time
Posted in Linux, Linux and FreeBSD Desktop, Remote System Administration, System Administration | No Comments »

Fix Out of inodes on Postfix Linux Mail Cluster. How to clean up filesystem running out of Inodes, Filesystem inodes on partition is 100% full

Wednesday, August 25th, 2021

Inode_Entry_inode-table-content

Recently we have faced a strange issue with with one of our Clustered Postfix Mail servers (the cluster is with 2 nodes that each has configured Postfix daemon mail servers (running on an OpenVZ virtualized environment).
A heartbeat that checks liveability of clusters and switches nodes in case of one of the two gets broken due to some reason), pretty much a standard SMTP cluster.

So far so good but since the cluster is a kind of abondoned and is pretty much legacy nowadays and used just for some Monitoring emails from different scripts and systems on servers, it was not really checked thoroughfully for years and logically out of sudden the alarming email content sent via the cluster stopped working.

The normal sysadmin job here was to analyze what is going on with the cluster and fix it ASAP. After some very basic analyzing we catched the problem is caused by a "inodes full" (100% of available inodes were occupied) problem, e.g. file system run out of inodes on both machines perhaps due to a pengine heartbeat process bug leading to producing a high number of .bz2 pengine recovery archive files stored in /var/lib/pengine>

Below are the few steps taken to analyze and fix the problem.

1. Finding out about the the system run out of inodes problem

After logging on to system and not finding something immediately is wrong with inodes, all I can see from crm_mon is cluster was broken.
A plenty of emails were left inside the postfix mail queue visible with a standard command

[root@smtp1: ~ ]# postqueue -p

It took me a while to find ot the problem is with inodes because a simple df -h was showing systems have enough space but still cluster quorum was not complete.
A bit of further investigation led me to a simple df -i reporting the number of inodes on the local filesystems on both our SMTP1 and SMTP2 got all occupied.

[root@smtp1: ~ ]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 500000 0 100% /
none 65536 61 65475 1% /dev

As you can see the number of inodes on the Virual Machine are unfortunately depleted

Next step was to check directories occupying most inodes, as this is the place from where files could be temporary moved to a remote server filesystem or moved to another partition with space on a server locally attached drives.
Below command gives an ordered list with directories locally under the mail root filesystem / and its respective occupied number files / inodes,
the more files under a directory the more inodes are being occupied by the files on the filesystem.

1.1 Getting which directory consumes most of the inodes on the systems

[root@smtp1: ~ ]# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
….
…..

…….
586 /usr/lib64/python2.4
664 /usr/lib64
671 /usr/share/man/man8
860 /usr/bin
1006 /usr/share/man/man1
1124 /usr/share/man/man3p
1246 /var/lib/Pegasus/prev_repository_2009-03-10-1236698426.308128000.rpmsave/root#cimv2/classes
1246 /var/lib/Pegasus/prev_repository_2009-05-18-1242636104.524113000.rpmsave/root#cimv2/classes
1246 /var/lib/Pegasus/prev_repository_2009-11-06-1257494054.380244000.rpmsave/root#cimv2/classes
1246 /var/lib/Pegasus/prev_repository_2010-08-04-1280907760.750543000.rpmsave/root#cimv2/classes
1381 /var/lib/Pegasus/prev_repository_2010-11-15-1289811714.398469000.rpmsave/root#cimv2/classes
1381 /var/lib/Pegasus/prev_repository_2012-03-19-1332151633.572875000.rpmsave/root#cimv2/classes
1398 /var/lib/Pegasus/repository/root#cimv2/classes
1696 /usr/share/man/man3
400816 /var/lib/pengine

Note, the above command orders the files from bottom to top order and obviosuly the bottleneck directory that is over-eating Filesystem inodes with an exceeding amount of files is
/var/lib/pengine

2. Backup old multitude of files just in case of something goes wrong with the cluster after some files are wiped out

The next logical step of course is to check what is going on inside /var/lib/pengine just to find a very ,very large amount of pe-input-*NUMBER*.bz2 files were suddenly produced.

[root@smtp1: ~ ]# ls -1 pe-input*.bz2 | wc -l
400816

The files are produced by the pengine process which is one of the processes that is controlling the heartbeat cluster state, presumably it is done by running process:

[root@smtp1: ~ ]# ps -ef|grep -i pengine
24 5649 5521 0 Aug10 ? 00:00:26 /usr/lib64/heartbeat/pengine

Hence in order to fix the issue, to prevent some inconsistencies in the cluster due to the file deletion, copied the whole directory to another mounted parition (you can mount it remotely with sshfs for example) or use a local one if you have one:

[root@smtp1: ~ ]# cp -rpf /var/lib/pengine /mnt/attached_storage

and proceeded to clean up some old multitde of files that are older than 2 years of times (720 days):

3. Clean up /var/lib/pengine files that are older than two years with short loop and find command

First I made a list with all the files to be removed in external text file and quickly reviewed it by lessing it like so

[root@smtp1: ~ ]# cd /var/lib/pengine
[root@smtp1: ~ ]# find . -type f -mtime +720|grep -v pe-error.last | grep -v pe-input.last |grep -v pe-warn.last -fprint /home/myuser/pengine_older_than_720days.txt
[root@smtp1: ~ ]# less /home/myuser/pengine_older_than_720days.txt

Once reviewing commands I've used below command to delete the files you can run below command do delete all older than 2 years that are different from pe-error.last / pe-input.last / pre-warn.last which might be needed for proper cluster operation.

[root@smtp1: ~ ]# for i in $(find . -type f -mtime +720 -exec echo '{}' \;|grep -v pe-error.last | grep -v pe-input.last |grep -v pe-warn.last); do echo $i; done

Another approach to the situation is to simply review all the files inside /var/lib/pengine and delete files based on year of creation, for example to delete all files in /var/lib/pengine from 2010, you can run something like:

[root@smtp1: ~ ]# for i in $(ls -al|grep -i ' 2010 ' | awk '{ print $9 }' |grep -v 'pe-warn.last'); do rm -f $i; done

4. Monitor real time inodes freeing

While doing the clerance of old unnecessery pengine heartbeat archives you can open another ssh console to the server and view how the inodes gets freed up with a command like:

# check if inodes is not being rapidly decreased

[root@csmtp1: ~ ]# watch 'df -i'

5. Restart basic Linux services producing pid files and logs etc. to make then workable (some services might not be notified the inodes on the Hard drive are freed up)

Because the hard drive on the system was full some services started to misbehaving and /var/log logging was impacted so I had to also restart them in our case this is the heartbeat itself
that checks clusters nodes availability as well as the logging daemon service rsyslog

# restart rsyslog and heartbeat services
[root@csmtp1: ~ ]# /etc/init.d/heartbeat restart
[root@csmtp1: ~ ]# /etc/init.d/rsyslog restart

The systems had been a data integrity legacy service samhain so I had to restart this service as well to reforce the /var/log/samhain log file to again continusly start writting data to HDD.

# Restart samhain service init script
[root@csmtp1: ~ ]# /etc/init.d/samhain restart

6. Check up enough inodes are freed up with df

[root@smtp1 log]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 410531 19469 91% /
none 65536 61 65475 1% /dev

I had to repeat the same process on the second Postfix cluster node smtp2, and after all the steps like below check the status of smtp2 node and the postfix queue, following same procedure made the second smtp2 cluster member as expected 🙂

7. Check the cluster node quorum is complete, e.g. postfix cluster is operating normally

# Test if email cluster is ok with pacemaker resource cluster manager – lt-crm_mon

[root@csmtp1: ~ ]# crm_mon -1
============
Last updated: Tue Aug 10 18:10:48 2021
Stack: Heartbeat
Current DC: smtp2.fqdn.com (bfb3d029-89a8-41f6-a9f0-52d377cacd83) – partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============

Online: [ smtp2.fqdn.com smtp1.fqdn.com ]

failover-ip (ocf::heartbeat:IPaddr2): Started csmtp1.ikossvan.de
Clone Set: postfix_clone
Started: [ smtp2.fqdn.com smtp1fqdn.com ]
Clone Set: pingd_clone
Started: [ smtp2.fqdn.com smtp1.fqdn.com ]
Clone Set: mailto_clone
Started: [ smtp2.fqdn.com smtp1.fqdn.com ]

8. Force resend a few hundred thousands of emails left in the email queue

After some inodes gets freed up due to the file deletion, i've reforced a couple of times the queued mail servers to be immediately resent to remote mail destinations with cmd:

# force emails in queue to be resend with postfix

[root@smtp1: ~ ]# sendmail -q

– It was useful to watch in real time how the queued emails are quickly decreased (queued mails are successfully sent to destination addresses) with:

# Monitor the decereasing size of the email queue
[root@smtp1: ~ ]# watch 'postqueue -p|grep -i '@'|wc -l'

Tags: Below, check, checks, Clone Set, cmd, command, dev, email, Free, How to, init, inodes, last, lib, Linux Fix, linux server, manager, none, Postfix Linux Mail Cluster, server, var

Tags: Below, check, checks, Clone Set, cmd, command, dev, email, Free, How to, init, inodes, last, lib, Linux Fix, linux server, manager, none, Postfix Linux Mail Cluster, server, var
Posted in Linux, System Administration, Various | No Comments »

Linux: Howto Fix “N: Repository ‘http://deb.debian.org/debian buster InRelease’ changed its ‘Version’ value from ‘10.9’ to ‘10.10’” error to resolve apt-get release update issue

Friday, August 13th, 2021

Linux's surprises and disorganization is continuously growing day by day and I start to realize it is becoming mostly impossible to support easily this piece of hackware bundled together.
Usually so far during the last 5 – 7 years, I rarely had any general issues with using:

apt-get update && apt-get upgrade && apt-get dist-upgrade

to raise a server's working stable Debian Linux version packages e.g. version X.Y to verzion X.Z (for example up the release from Debian Jessie from 8.1 to 8.2).

Today I just tried to follow this well known and established procedure that, of course nowdays is better to be done with the newer "apt" command instead with the legacy "apt-get"
And the set of

# apt-get update && apt-get upgrade && apt-get dist-upgrade

has triggered below shitty error:

root@zabbix:~# apt-get update && apt-get upgrade
Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://security.debian.org buster/updates/non-free Sources [688 B]
Get:4 http://repo.zabbix.com/zabbix/5.0/debian buster InRelease [7096 B]
Get:5 http://security.debian.org buster/updates/main Sources [198 kB]
Get:6 http://security.debian.org buster/updates/main amd64 Packages [300 kB]
Get:7 http://security.debian.org buster/updates/main Translation-en [157 kB]
Get:8 http://security.debian.org buster/updates/non-free amd64 Packages [556 B]
Get:9 http://deb.debian.org/debian buster/main Sources [7836 kB]
Get:10 http://repo.zabbix.com/zabbix/5.0/debian buster/main Sources [1192 B]
Get:11 http://repo.zabbix.com/zabbix/5.0/debian buster/main amd64 Packages [4785 B]
Get:12 http://deb.debian.org/debian buster/non-free Sources [85.7 kB]
Get:13 http://deb.debian.org/debian buster/contrib Sources [42.5 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:15 http://deb.debian.org/debian buster/main Translation-en [5968 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 Contents (deb) [37.3 MB]
Get:17 http://deb.debian.org/debian buster/contrib amd64 Packages [50.1 kB]
Get:18 http://deb.debian.org/debian buster/non-free amd64 Packages [87.7 kB]
Get:19 http://deb.debian.org/debian buster/non-free Translation-en [88.9 kB]
Get:20 http://deb.debian.org/debian buster/non-free amd64 Contents (deb) [861 kB]
Fetched 61.1 MB in 22s (2774 kB/s)
Reading package lists… Done
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'
…

As I used to realize nowdays, as Linux started originally as 'Hackers' operating system, its legacy is just one big hack and everything from simple maintenance up to the higher and more sophisticated things requires a workaround 'hack''.

This time the hack to resolve error:

N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'

is up to running cmd:

debian-server:~# apt-get update –allow-releaseinfo-change
Поп:1 http://ftp.de.debian.org/debian buster-backports InRelease
Поп:2 http://ftp.debian.org/debian stable InRelease
Поп:3 http://security.debian.org stable/updates InRelease
Изт:5 https://packages.sury.org/php buster InRelease [6837 B]
Изт:6 https://download.docker.com/linux/debian stretch InRelease [44,8 kB]
Изт:7 https://packages.sury.org/php buster/main amd64 Packages [317 kB]
Игн:4 https://attic.owncloud.org/download/repositories/production/Debian_10 InRelease
Изт:8 https://download.owncloud.org/download/repositories/production/Debian_10 Release [964 B]
Изт:9 https://packages.sury.org/php buster/main i386 Packages [314 kB]
Изт:10 https://download.owncloud.org/download/repositories/production/Debian_10 Release.gpg [481 B]
Грш:10 https://download.owncloud.org/download/repositories/production/Debian_10 Release.gpg
Следните подписи са невалидни: DDA2C105C4B73A6649AD2BBD47AE7F72479BC94B
Грш:11 https://ookla.bintray.com/debian generic InRelease
403 Forbidden [IP: 52.39.193.126 443]
Четене на списъците с пакети… Готово
N: Repository 'https://packages.sury.org/php buster InRelease' changed its 'Suite' value from '' to 'buster'
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://download.owncloud.org/download/repositories/production/Debian_10 Release:
…

apt-get-update-allow-releaseinfo-change-debian-linux-screenshot

Onwards to upgrade the system up to the latest .deb packages, as usual run:

# apt-get -y update && apt-get upgrade -y
…

and updates should be applied as usual with some prompts on whether you prefer to keep or replace existing service configuration and some information on some general changes that might affect your installed services. In a few minutes and few prompts hopefully your Debian OS should be up to the latest stable.

Tags: apt-get, cmd, command, complete, deb, download, issues, Linux Fix Repository, org, owncloud, packages, repositories, security, updates, zabbix

Tags: apt-get, cmd, command, complete, deb, download, issues, Linux Fix Repository, org, owncloud, packages, repositories, security, updates, zabbix
Posted in Linux, System Administration, Various | No Comments »

How to yum Install Gnome GUI, Latest Guest Addition Tools, Google Chrome latest version and rdesktop / xfreerdp / remmina remote RDP VNC clients On CentOS 7 / 8

Thursday, July 29th, 2021

I've just reinstalled my CentOS 7 Virtual Machine since after I tried to migrate a .vdi Virtual Box image to the new company laptop using a copy of Virtualbox VM via Microsoft OneDrive was a failure.
Thus I have rebuild all my CentOS Linux programs preinstalled on the old Virtual Machines from scratch, I use this virtual machine for a very simple tasks, so basicly most imporant tools I use is a plain SSH and VNC and Remote Desktop clients just to be able to remotely connect to remote Home based server.

1.Install GNOME Graphical Environment from command line on CentOS 7 with yum and configure it to start GUI on next OS boot

I've used a minimal CentOS installation – ISO CentOS-7-x86_64-DVD-1908.iso and this brings up the OS with a text mode only as usually CentOS is used to roll on Servers and rarely and many times admins did not want to have GUI at all, however my case is different since I do like to use Graphical Environment as I use my CentOS for all kind of testing that can be later applied to a Production machines that doesn't have a GUI, hence to install GNOME on CentOS run below cmds:

[root@centos ~ ]# yum group list
Loaded plugins: fastestmirror
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Loading mirror speeds from cached hostfile
Available Environment Groups:
Minimal Install
Compute Node
Infrastructure Server
File and Print Server
Basic Web Server
Virtualization Host
Server with GUI
GNOME Desktop
KDE Plasma Workspaces
Development and Creative Workstation
Available Groups:
Compatibility Libraries
Console Internet Tools
Development Tools
Graphical Administration Tools
Legacy UNIX Compatibility
Scientific Support
Security Tools
Smart Card Support
System Administration Tools
System Management
Done

[root@centos ~ ]# yum groupinstall "GNOME Desktop" "Graphical Administration Tools" -y

Enable GUI to be automatically start on CentOS VM boot in systemd this is configured with the "targets" instead of the well known classical runlevels (the well known /etc/inittab) is now obsolete in newer Linux distros.

[root@centos ~ ]# ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target

2. Install Guest Additions Tools on CentOS

The most basic thing to do once I've had the CentOS Linux release 7.7.1908 (Core) rolled out on the VirtualBox is of course to enable Guest Additions Tools

First I had to install of course Guest Additions Tools to allow myself to have a copy paste in clip board via the Host Machine (Windows 10) and the Guest Machine.
To do I had to:

[root@centos ~ ]# yum install kernel-headers.x86_64 -y

[root@centos ~ ]# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

[root@centos ~ ]# yum install perl gcc dkms kernel-devel kernel-headers make bzip2

To check the required VBoxLinuxAdditions.run script kernel headers are at place:

[root@centos ~ ]# ls -l /usr/src/kernels/$(uname -r)

…
You should get a list of kernel header files

Then once I've done the Insert Guest Additions CD Image from the VirtualBox VM upper menu. I've had to mount and load the guest additions via the script:

[root@centos ~ ]# mkdir /mnt/cdrom
[root@centos ~ ]# mount /dev/cdrom /mnt/cdrom
[root@centos ~ ]# sh VBoxLinuxAdditions.run

After rebooting the Virtual Machine, I've used the full screen functionality to test and configured immediately Shared Clipboard and Drag and Drop to be both set to (Bidirectional) as well as configured a Shared folder to provide my Windows Desktop under /mnt/shared_folder (as read write) as I usually do to be able to easily copy files from the VM and to the Windows.

3. Install Google Chrome on the CentOS Virtual Machine with yum

Next I've installed the chrome browser that was pretty trivial it is up to fetching the required 32 or 64 bit latest chrome binary this is usually on URL:

[root@centos ~ ]# wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

and installing Google Chrome with superuser with command:

[root@centos ~ ]# yum install ./google-chrome-stable_current_*.rpm -y

Loaded plugins: fastestmirror, langpacks
Examining ./google-chrome-stable_current_x86_64.rpm: google-chrome-stable-92.0.4515.107-1.x86_64
Marking ./google-chrome-stable_current_x86_64.rpm to be installed
Resolving Dependencies
–> Running transaction check
—> Package google-chrome-stable.x86_64 0:92.0.4515.107-1 will be installed
–> Processing Dependency: liberation-fonts for package: google-chrome-stable-92.0.4515.107-1.x86_64
Loading mirror speeds from cached hostfile
* base: mirror.digitalnova.at
* epel: fedora.ipacct.com
* extras: mirror.digitalnova.at
* updates: mirror.digitalnova.at
–> Processing Dependency: libvulkan.so.1()(64bit) for package: google-chrome-stable-92.0.4515.107-1.x86_64
–> Running transaction check
—> Package liberation-fonts.noarch 1:1.07.2-16.el7 will be installed
–> Processing Dependency: liberation-narrow-fonts = 1:1.07.2-16.el7 for package: 1:liberation-fonts-1.07.2-16.el7.noarch
—> Package vulkan.x86_64 0:1.1.97.0-1.el7 will be installed
–> Processing Dependency: vulkan-filesystem = 1.1.97.0-1.el7 for package: vulkan-1.1.97.0-1.el7.x86_64
–> Running transaction check
—> Package liberation-narrow-fonts.noarch 1:1.07.2-16.el7 will be installed
—> Package vulkan-filesystem.noarch 0:1.1.97.0-1.el7 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
google-chrome-stable x86_64 92.0.4515.107-1 /google-chrome-stable_current_x86_64
259 M
Installing for dependencies:
liberation-fonts noarch 1:1.07.2-16.el7 base 13 k
liberation-narrow-fonts noarch 1:1.07.2-16.el7 base 202 k
vulkan x86_64 1.1.97.0-1.el7 base 3.6 M
vulkan-filesystem noarch 1.1.97.0-1.el7 base 6.3 k

Transaction Summary
================================================================================
Install 1 Package (+4 Dependent packages)

Total size: 263 M
Total download size: 3.8 M
Installed size: 281 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): liberation-fonts-1.07.2-16.el7.noarch.rpm | 13 kB 00:00
(2/4): liberation-narrow-fonts-1.07.2-16.el7.noarch.rpm | 202 kB 00:00
(3/4): vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm | 6.3 kB 00:00
(4/4): vulkan-1.1.97.0-1.el7.x86_64.rpm | 3.6 MB 00:00
——————————————————————————–
Total 3.0 MB/s | 3.8 MB 00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : vulkan-filesystem-1.1.97.0-1.el7.noarch 1/5
Installing : vulkan-1.1.97.0-1.el7.x86_64 2/5
Installing : 1:liberation-narrow-fonts-1.07.2-16.el7.noarch 3/5
Installing : 1:liberation-fonts-1.07.2-16.el7.noarch 4/5
Installing : google-chrome-stable-92.0.4515.107-1.x86_64 5/5
Verifying : vulkan-1.1.97.0-1.el7.x86_64 1/5
Verifying : 1:liberation-narrow-fonts-1.07.2-16.el7.noarch 2/5
Verifying : 1:liberation-fonts-1.07.2-16.el7.noarch 3/5
Verifying : google-chrome-stable-92.0.4515.107-1.x86_64 4/5
Verifying : vulkan-filesystem-1.1.97.0-1.el7.noarch 5/5

Installed:
google-chrome-stable.x86_64 0:92.0.4515.107-1

Dependency Installed:
liberation-fonts.noarch 1:1.07.2-16.el7
liberation-narrow-fonts.noarch 1:1.07.2-16.el7
vulkan.x86_64 0:1.1.97.0-1.el7
vulkan-filesystem.noarch 0:1.1.97.0-1.el7
…

4. Install usable Windows VNC and remote desktop (RDP Client) for CentOS Linux

There is a plenty of clients to choice from if you need to have an RDP client for Linux, but perhaps the most useful ones I usually use are remmina / rdesktop and freerdp. Usually I use remmina on Debian Linux, but under the VM somehow I was not able to make remmina work in Full Screen mode while connected to remote Windows 7 VPS server, thus I've first tried xfreerdp (that comes from default CentOS repositories) and is open source alternative to rdesktop (which is non free distributed binary).

[root@centos ~ ]$ sudo yum -y install freerdp

The basic use is:

[hipo@centos ~ ]$ xfreerdp –toggle-fullscreen <remote-server-address>

Unfortunately I did not succeeded to make xfreerdp be able to show me remote desktop in FullScreen mode so had to use additional repository package called nux-dextop to have rdesktop at my disposal.

To install it had to run:

[root@centos ~ ]# rpm –import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
[root@centos ~ ]# rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
[root@centos ~ ]# yum install rdesktop

To connect to the remote RDP host in Fullscreen with rdesktop :

rdesktop -f <remote-server-address>

windows-7-remote-desktop-screenshot-connected-with-rdesktop

As telnet is not installed by default and it is so useful to check ports

5. Install GNU Image Manipulation Program for better screnshotting and Graphic edits

I usually do install GIMP (GNU Image Manipulation Program) since this is my favourite tool to make screenshot on Linux as well as do some minor graphic edits whenever necessery. I warmly recommend gimp to anyone. If you don't have basic GIMP tool and you plan to be daily working a lot with Linux sooner or later some skills with the program will be of a major use even for the most advanced sysadmin :)_

root@centos ~ ]# yum install -y gimp

6. Install useful administration tools for daily sysadmin work – telnet, nmap, iftop, htop, iotop, iptraf-ng, tcpdump

Having basic analys tools and remote communication port testing, DNS, resolving and connection, cpu, mem statistics I find mostly useful.

[root@centos .ssh]# yum install telnet nmap iftop htop vnstat sysstat iptraf-ng bind-utils -y
…

7. Set Open Explorer and SHOW Desktop key binding shortcuts for GNOME (to make daily work easier)

Another useful I do use in my newly installed Virtual Machines is the key combination of Windows (button key) + E – to easily open the GNOME equivalent of Windows Explorer (Nautilus) and Windows (key) + D to hide the active selected Window and Show Desktop. This is configured pretty easy in GNOME through:

$ gnome-control-center

Keyboard (Section)

Perhaps there is other stuff I need to add on the freshly installed Operating System if I remember something else interesting

configure-home-folder-and-hide-all-normal-windows-gnome-key-binding-howto-screenshot

8. Install gnome-tweaks to tweak a bit the desktop icon positionsing and additional gnome-shell extras

[root@centos hipo]# yum install -y gnome-shell-extension-workspace-indicator.noarch gnome-shell-extension-workspace-indicator.noarch gnome-shell-extension-suspend-button.noarch gnome-shell-extension-refresh-wifi.noarch gnome-shell-extension-updates-dialog.noarch gnome-shell-extension-windowoverlay-icons.noarch gnome-shell-extension-places-menu.noarch gnome-shell-extension-drive-menu.noarch gnome-shell-extension-apps-menu.noarch gnome-shell-extension-auto-move-windows.noarch gnome-tweaks gnome-shell-extension-systemMonitor.noarch gnome-shell-extension-openweather.noarch gnome-shell-extension-user-theme.noarch gnome-shell-extension-topicons-plus.noarch

Next step is to use gnome-tweaks to set multiple custom preference stuff you like on the gnome 3.28 GUI

gnome-tweak-tool1

gnome-tweak-tool2

gnome-tweak-tool3

9. Change ( Fix) timezone to correct time on the Virtual Machine

[root@localhost ~]# timedatectl
Local time: Fri 2021-07-30 12:20:51 CEST
Universal time: Fri 2021-07-30 10:20:51 UTC
RTC time: Fri 2021-07-30 10:20:48
Time zone: Europe/Berlin (CEST, +0200)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: yes
Last DST change: DST began at
Sun 2021-03-28 01:59:59 CET
Sun 2021-03-28 03:00:00 CEST
Next DST change: DST ends (the clock jumps one hour backwards) at
Sun 2021-10-31 02:59:59 CEST
Sun 2021-10-31 02:00:00 CET

[root@localhost ~]# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 35 Jul 29 14:03 /etc/localtime -> ../usr/share/zoneinfo/Europe/Berlin

To change to correct timezone, you need to find out the long name for the timezone you want to use. The timezone naming convention usually uses “Region/City” format.

To list all available time zones, you can either list the files in the /usr/share/zoneinfo directory or use the timedatectl command.

[root@centos ~]# timedatectl list-timezones|tail -n 10
Pacific/Pohnpei
Pacific/Port_Moresby
Pacific/Rarotonga
Pacific/Saipan
Pacific/Tahiti
Pacific/Tarawa
Pacific/Tongatapu
Pacific/Wake
Pacific/Wallis
UTC

As I'm situated in Sofia Bulgaria to set the correct timezone to UTC (Universal Time Clock) + 2 Hrs, i've checked the correct Continent/Country like so:

[root@centos ~]# timedatectl list-timezones|grep -i Sofia
Europe/Sofia

Once I've my Capital / Country time location identified to set to it:

[root@centos ~]# timedatectl set-timezone your_time_zone

10. Configure remote connection hostname SSH aliases via ssh config ( ~/.ssh/config)

I'm having separate Virtual Machines running on my OpenXen virtualization Hypervisor server at different ports which I remember by heart under different hostnames, this saves me time to always type on command line long commands such as:

# ssh long-hostname -p Port_number

to make accessibility to remote machines via a simple Hostname Aliases, that forwards to remote port (that gets forwarded via a Local Network configure Netwrork Address Translation), I use the .ssh/config nice Host / Hostname / User / Port directives like below samples:

[hipo@centos .ssh]$ cat config
Host pcfreak
User root
Port 2248
HostName 83.228.93.76

Host freak
User root
Port 2249
HostName 213.91.190.233

Host pcfrxenweb
User root
Port 2251
Hostname 83.228.93.76

Host pcfrxen
User root
Port 2250
Hostname 213.91.190.233
…

Now to connect to pcfrxen for example I simply type:

ssh pcfrxen

type in the password to remote VM and I'm in 🙂

The same could be achieved also with Adding Custom Hostname IP Aliases via ~/.bashrc or iteration script as I've explained earlier that fakes like custom /etc/hosts, but I usuaully prefer to use .ssh/config instead like explained above.

Note that above steps should work also on RHEL / Fedora Linux with a minor modifications, as usually this two distros share the RPM package manager. If someone tries to follow the guide and have success on any of this distros please drop a comment with feedback.

Tags: Available Environment Groups, Available Groups, cmds, Dependency Installed, fedora linux, guide, How to, Package Arch Version Repository Size, rhel, superuser, url

Tags: Available Environment Groups, Available Groups, cmds, Dependency Installed, fedora linux, guide, How to, Package Arch Version Repository Size, rhel, superuser, url
Posted in Linux, System Administration, Virtual Machines | No Comments »

CentOS disable SELinux permanently or one time on grub Linux kernel boot time

Saturday, July 24th, 2021

1. Office 365 cloud connected computer and a VirtualBox hosted machine with SELINUX preventing it to boot

At my job we're in process of migrating my old Lenovo Laptop Thinkpad model L560 Laptop to Dell Latitude 5510 wiith Intel Core i5 vPro CPU and 256 Gb SSD Hard Drive. The new laptops are generally fiine though they're not even a middle class computers and generally I prefer thinkpads. The sad thing out of this is our employee decided to migrate to Office 365 (again perhaps another stupid managerial decision out of an excel sheet wtih a balance to save some money …

As you can imagine Office 365 is not really PCI Standards compliant and not secure since our data is stored in Microsoft cloud and theoretically Microsoft has and owns our data or could wipe loose the data if they want to. The other obvious security downside I've noticed with the new "Secure PCI complaint laptop" is the initial PC login screen which by default offers fingerprint authentication or the even worse and even less secure face recognition, but obviosly everyhing becomes more and more crazy and people become less and less cautious for security if that would save money or centralize the data … In the name of security we completely waste security that is very dubious paradox I don't really understand but anyways, enough rant back to the main topic of this article is how to and I had to disable selinux?

As part of Migration I've used Microsoft OneDrive to copy old files from the Thinkpad to the Latitude (as on the old machine USB's are forbidden and I cannot copy over wiith a siimple USB driive, as well as II have no right to open the laptop and copy data from the Hard driive, and even if we had this right without breaking up some crazy company policy that will not be possible as the hard drive data on old laptop is encrypted, the funny thing is that the new laptop data comes encrypted and there is no something out of the box as BitDefender or McAffee incryption (once again, obviously our data security is a victim of some managarial decisions) …

2. OneDrive copy problems unable to sync some of the copied files to Onedrive

Anyways as the Old Laptop's security is quite paranoid and we're like Fort Nox, only port 80 and port 443 connections to the internet can be initiated to get around this harsh restrictions it was as simple to use a Virtualbox Virtual Machine. So on old laptop I've installed a CentOS 7 image which I used so far and I used one drive to copy my vbox .vdi image on the new laptop work machine.

The first head buml was the .vdi which seems to be prohibited to be copied to OneDrive, so to work around this I had to rename the origianl CentOS7.vdi to CentOS7.vdi-renamed on old laptop and once the data is in one drive copy my Vitualbox VM/ directory from one drive to the Dell Latitude machine and rename the .vdi-named towards .vdi as well as import it from the latest installed VirtualBox on the new machine.

3. Disable SELINUX from initial grub boot

So far so good but as usual happens with miigrations I've struck towards another blocker, the VM image once initiated to boot from Virtualbox badly crashed with some complains that selinux cannot be loaded.
Realizing CentOS 7 has the more or less meaningless Selinux, I've took the opportunity to disable SeLinux.

To do so I've booted the Kernel with Selinux disabled from GRUB2 loader prompt before Kernel and OS Userland boots.

I thought I need to type the information on the source in grub. What I did is very simple, on the Linux GRUB boot screen I've pressed

'e' keyboard letter

that brought the grub boot loader into edit mode.

Then I had to add selinux=0 on the edited selected kernel version, as shown in below screenshot:

Next to boot the Linux VM without Selinux enabled one time, just had to press together

Ctrl+X then add selinux=0 on the edited selected kernel version, that should be added as shown in the screenshot somewhere after the line of
root=/dev/mapper/…..

4. Permanently Disable Selinux on CentOS 7

Once I managed to boot Virtual Machine properly with Oracle Virtualbox, to permanently disabled selinux I had to:

Once booted into CentOS, to check the status of selinux run:

# sestatus
Copy
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

5. Disable SELinux one time with setenforce command

You can temporarily change the SELinux mode from targeted to permissive with the following command:

# setenforce 0

Next o permanently disable SELinux on your CentOS 7 next time the system boots, Open the /etc/selinux/config file and set the SELINUX mod parameter to disabled.

On CentOS 7 you can edit the kernel parameters in /etc/default/grub (in the GRUB_CMDLINE_LINUX= key) and set selinux=0 so on next VM / PC boot we boot with a SELINUX disabled for example add RUB_CMDLINE_LINUX=selinux=0 to the file then you have to regenerate your Grub config like this:

# grub2-mkconfig -o /etc/grub2.cfg
# grub2-mkconfig -o /etc/grub2-efi.cfg

Further on to disable SeLinux on OS level edit /etc/selinux

Default /etc/selinux/config with selinux enabled should look like so:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing – SELinux security policy is enforced.
# permissive – SELinux prints warnings instead of enforcing.
# disabled – No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
# targeted – Targeted processes are protected,
# mls – Multi Level Security protection.
SELINUXTYPE=targeted

To disable SeLinux modify the file to be something like:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing – SELinux security policy is enforced.
# permissive – SELinux prints warnings instead of enforcing.
# disabled – No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted – Targeted processes are protected,
# mls – Multi Level Security protection.
SELINUXTYPE=targeted

6. Check SELINUX status is disabled

# sestatus

SELinux status: disabled

So in this article shottly was explained shortly the fake security adopted by using Microsoft Cloud environment Offiice 365, my faced OneDrive copy issues (which prevented even my old laptop Virtual Machine to boot properly and the handy trick to rename the file that is unwilling to get copied from old PC towards m$ OneDrive as well as the grub trick to disable Selinux permanently from grub2.

Tags: boot time, CentOS, config, copy, file, Mode, Office, SELINUXTYPE, system, Virtualbox Virtual Machine

Tags: boot time, CentOS, config, copy, file, Mode, Office, SELINUXTYPE, system, Virtualbox Virtual Machine
Posted in Linux, Security, System Administration | No Comments »

Linux: How to set KVM Virtual Machine to autostart on system boot

Thursday, July 15th, 2021

linux-kvm-set-vm-to-automatically-boot-on-linux-system-boot

Recently I've written a short article on how to make auto boot OpenVZ container Virtual Machine if system gets rebooted as I had the task to do so in daily job as sysadmin.
Few days ago after some power outage in one of Frankfurt based Data Centers, some Rack tech equipment has been temporary powered of and once the electricity was up again, some Linux servers running Kernel Based Virtualization ( KVM ) some of the virtual machines did not managed to automatically start up and we had to manually start them one by one.
To manually start each of the machines that did not start up had to do the trivial:

[root@hypervisor ~]# virsh list
Id Name State
—————————————————-
3 VM500 running

[root@hypervisor ~]# virsh dominfo VM500
Id: 3
Name: VM500
UUID: 82075903-a4ce-5ec0-a3d8-9ee83d85fc75
OS Type: hvm
State: running
CPU(s): 2
CPU time: 247407.9s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: disable
Managed save: no
Security model: selinux
Security DOI: 0
Security label: system_u:system_r:svirt_t:s0:c447,c723 (permissive)

[root@hypervisor ~]# virsh start VM_domain_Name

Of course logcally to prevent future issues in case if Linux server gets suddenly rebooted due to whatever reason, we had to configure the machines in questions to automatically boot on OS system boot.

In some rare cases if above start command does not help you might want to check what is the status of libvirtd and investigate the logs in /var/log/libvirt/ i.e. /var/log/libvirt/libvirt.log

An alternative but more complicated way to set virtual machine to automatically start on boot is by using default location for automatic VM start just like OpenXEn has its /etc/xen/auto/ dirtory from where each soft symlinked VM configuration from /etc/xen/VM_name.cfg is started in KVM Hypervisor hosts to auto boot a certain vm you have to link /etc/libvirt/qemu/VM-to-autoboot-name.xml to /etc/libvirt/qemu/autostart/VM-to-autoboot-name.xml.

[root@hypervisor ~]# systemctl status libvirtd
● libvirtd.service – Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-06-29 10:21:30 CEST; 2 weeks 2 days ago
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 1809 (libvirtd)
Tasks: 22 (limit: 32768)
CGroup: /system.slice/libvirtd.service
├─1809 /usr/sbin/libvirtd
├─2335 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
├─2336 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
├─2386 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…
└─2387 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…

If for some reason libvirtd is not running or disabled state you will have to enable it with:

[root@hypervisor ~]# systemctl enable libvirtd

If the virtualization is running on some RPM based distribtuion OpenSuse whatever and libvirtd is controlled via chkconfig (redhat runlevel alternative system), you will have to :

[root@hypervisor ~]# chkconfig libvirtd on

Disabling KVM Virtual Machine from auto start on server boot

[root@hypervisor ~]# virsh autostart Debian10 –disable
Domain Debian10 unmarked as autostarted

Tags: default location, libvirtd.service, logs, running, sbin, status, system boot, usr, virsh, vm

Tags: default location, libvirtd.service, logs, running, sbin, status, system boot, usr, virsh, vm
Posted in Linux, System Administration, Virtual Machines | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Archive for the ‘System Administration’ Category

Change Windows 10 default lock screen image via win registry LockScreenImage key change

Settings -> Lock Screen

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites

Archive for the ‘System Administration’ Category

1. The /var/backups recovery directory

2. Recreate basic /var/lib/dpkg directory and files structures

3. Recover /var/lib/dpkg/status file

4. Check dpkg package installation works again and reinstall base-files

5. Update deb system package list and db consistency

6. Reinstall completely from source code dpkg, if nothing else works

'/usr/local/var/lib/dpkg/status' for reading: No such file or directory

“Cannot open /var/log/sysstat/sa18: No such file or directory. Please check if data collecting is enabled”

1. Check with sysstat machine history SWAP and RAM Memory use

2. Check system load? Are my processes waiting too long to run on the CPU?

3. Show various CPU statistics per CPU use

4. Report paging statistics for some old period

5. Monitor Received RX and Transmitted TX network traffic perl Network interface real time

6. Monitor block devices use

7. Output server monitoring data in CSV database structured format

What we've learned?

1. Use the good old times rinetd – internet “redirection server” service

+ Configuring a new port redirect with rinetd

[bindaddress] [bindport] [connectaddress] [connectport]

+ Using rinetd to redirect External interface IP to loopback's port (127.0.0.1)

2. Redirect TCP / IP port using DNAT iptables firewall rules

3. Redirect TCP traffic connections with redir tool

+ Use redir to redirect TCP traffic one time

+ Making TCP port forwarding from Host A to Host B permanent

Settings -> Lock Screen

1. Adding local user remote server pointer aliases via ~/.ssh/config

2. Adding ssh client options system wide for all existing local or remote LDAP login users

What we learned ?

1. Finding out about the the system run out of inodes problem

1.1 Getting which directory consumes most of the inodes on the systems

2. Backup old multitude of files just in case of something goes wrong with the cluster after some files are wiped out

3. Clean up /var/lib/pengine files that are older than two years with short loop and find command

4. Monitor real time inodes freeing

5. Restart basic Linux services producing pid files and logs etc. to make then workable (some services might not be notified the inodes on the Hard drive are freed up)

6. Check up enough inodes are freed up with df

7. Check the cluster node quorum is complete, e.g. postfix cluster is operating normally

8. Force resend a few hundred thousands of emails left in the email queue

N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'

1.Install GNOME Graphical Environment from command line on CentOS 7 with yum and configure it to start GUI on next OS boot

2. Install Guest Additions Tools on CentOS

3. Install Google Chrome on the CentOS Virtual Machine with yum

4. Install usable Windows VNC and remote desktop (RDP Client) for CentOS Linux

5. Install GNU Image Manipulation Program for better screnshotting and Graphic edits

6. Install useful administration tools for daily sysadmin work – telnet, nmap, iftop, htop, iotop, iptraf-ng, tcpdump

7. Set Open Explorer and SHOW Desktop key binding shortcuts for GNOME (to make daily work easier)

8. Install gnome-tweaks to tweak a bit the desktop icon positionsing and additional gnome-shell extras

9. Change ( Fix) timezone to correct time on the Virtual Machine

10. Configure remote connection hostname SSH aliases via ssh config ( ~/.ssh/config)

2. OneDrive copy problems unable to sync some of the copied files to Onedrive

3. Disable SELINUX from initial grub boot

4. Permanently Disable Selinux on CentOS 7

5. Disable SELinux one time with setenforce command

6. Check SELINUX status is disabled

Disabling KVM Virtual Machine from auto start on server boot

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Tags

Top Post Views

blogtopsites

“Cannot open /var/log/sysstat/sa18:
No such file or directory. Please check if data collecting is enabled”