Posts Tagged ‘CentOS’

CentOS 8 / Redhat 8 insert additional guests additions to VM to enable Fullscreen, Copy / Paste and Shared Folder from host OS

Monday, January 10th, 2022

virtualbox-guest-additions-install-on-centos-8.3-linux-oracle-logo

My experience with enabling virtualbox additions guest tools on many of the separate Linux distributions throughout time is pretty bad as it always is a pain in the ass to enable fully functional full screen and copy paste for Virtualbox…
 
For those who installed it for a first time vbox guest addition tools for Virtualbox are additional software components added so the Emulated Operating system
could allow better screen resolution and better mouse integration support.

So far I've installed virtualbox additions tools to CentOS 7 and Debian Linux various releases and faced complications there as well.
Few days ago my colleague Georgi Stoyanov have installed CentOS 8.3 with current version of VirtualBox 6.1 (vesrsion from beginning of 2022) and he has also shared had issues with enabling the CentOS 8.3 Linux to work with guestadditions but eventually found a resolution.

Thus he has shared with me the solution and I share it with you, so hopefully someone else could enable Guesttools on his CentOS 8.3 with less digging online.
The error received is:

# ./VBoxLinuxAdditions.run

Trying to install Guest Additions in RHEL 8.3.

VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules. This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup
VirtualBox Guest Additions: or
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Building the modules for kernel
4.18.0-193.el8.x86_64.

VirtualBox Guest Additions: Look at /var/log/vboxadd-setup.log to find out what
went wrong
ValueError: File context for /opt/VBoxGuestAdditions-6.0.20/other/mount.vboxsf already defined
VirtualBox Guest Additions: Running kernel modules will not be replaced until
the system is restarted
Press Return to close this window…

No idea what to do next. Been trying for sometime.


To enable guestaddtions in CentOS 8.3, e.g. get arount the error you have to:


1. Install all necessery dependncies RPMs required by GuestAddition tools

 

# dnf install tar bzip2 kernel-devel-$(uname -r) kernel-headers perl gcc make elfutils-libelf-devel

# dnf -y install gcc automake make kernel-headers dkms bzip2 libxcrypt-compat kernel-devel perl

2.  Run below semanage and restorecon commands

 

# semanage fcontext -d /opt/VBoxGuestAdditions-/other/mount.vboxsf
# restorecon /opt/VBoxGuestAdditions-/other/mount.vboxsf

 

3.  Insert Virtualbox guest additions ISO and Run it

 

centos-insert-guest-additions-linux-virtualbox-screenshot
 

Devices -> Insert Guest Additions CD Image

 

Click Run button to exec Vbox_GAs_6.0.18 script or run it manually

Run-Guest-Additions-screenshot-virtualbox-centos-8

or mount it manually with mount command and execute the VBoxLinuxAdditions.run to do so:

 

$ cd /run/media/`whoami`/VB*
$ su
# ./VBoxLinuxAdditions.run
Installing additional modules …
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
VirtualBox Guest Additions: Running kernel modules will not be replaced until the system is restarted
VirtualBox Guest Additions: Starting.

 

4. Reboot the VM
 

# reboot

5. Check and Confirm Virtualbox guest additions are properly installed and running
 

# lsmod | grep vbox

 

6. Enable Copy / Paste from to Virttual Machine e.g. Shared Clipboard / Shared Folder etc.

 

Share-Clipboard-in-Virtualbox-screenshot-centos-8

 

The three options most useful besides the support for FullScreen OS emulation by Virtualbox to enable right after
guesttools is on are:


1. Devices -> Shared Clipboard -> Bidirectional
2. Devices -> Drag and Drop -> Bidirectional
3. Devices -> Shared Folders -> Shared Folder Settings

 

How to test RAM Memory for errors in Linux / UNIX OS servers. Find broken memory RAM banks

Friday, December 3rd, 2021

test-ram-memory-for-errors-linux-unix-find-broken-memory-logo

 

1. Testing the memory with motherboard integrated tools
 

Memory testing has been integral part of Computers for the last 50 years. In the dawn of computers those older perhaps remember memory testing was part of the computer initialization boot. And this memory testing was delaying the boot with some seconds and the user could see the memory numbers being counted up to the amount of memory. With the increased memory modern computers started to have and the annoyance to wait for a memory check program to check the computer hardware memory on modern computers this check has been mitigated or completely removed on some hardware.
Thus under some circumstances sysadmins or advanced computer users might need to check the memory, especially if there is some suspicion for memory damages or if for example a home PC starts crashing with Blue screens of Death on Windows without reason or simply the PC or some old arcane Linux / UNIX servers gets restarted every now and then for now apparent reason. When such circumstances occur it is an idea to start debugging the hardware issue with a simple memory check.

There are multiple ways to test installed memory banks on a server laptop or local home PC both integrated and using external programs.
On servers that is usually easily done from ILO or IPMI or IDRAC access (usually web) interface of the vendor, on laptops and home usage from BIOS or UEFI (Unified Extensible Firmware Interface) acces interface on system boot that is possible as well.

memtest-hp
HP BIOS Setup

An old but gold TIP, more younger people might not know is the

 

Prolonged SHIFT key press which once held with the user instructs the machine to initiate a memory test before the computer starts reading what is written in the boot loader.

So before anything else from below article it might be a good idea to just try HOLD SHIFT for 15-20 seconds after a complete Shut and ON from the POWER button.

If this test does not triggered or it is triggered and you end up with some corrupted memory but you're not sure which exact Memory bank is really crashing and want to know more on what memory Bank and segments are breaking up you might want to do a more thorough testing. In below article I'll try to explain shortly how this can be done.


2. Test the memory using a boot USB Flash Drive / DVD / CD 
 

Say hello to memtest86+. It is a Linux GRUB boot loader bootable utility that tests physical memory by writing various patterns to it and reading them back. Since memtest86+ runs directly off the hardware it does not require any operating system support for execution. Perhaps it is important to mention that memtest86 (is PassMark memtest86)and memtest86+ (An Advanced Memory diagnostic tool) are different tools, the first is freeware and second one is FOSS software.

To use it all you'll need is some version of Linux. If you don't already have some burned in somewhere at your closet, you might want to burn one.
For Linux / Mac users this is as downloading a Linux distribution ISO file and burning it with

# dd if=/path/to/iso of=/dev/sdbX bs=80M status=progress


Windows users can burn a Live USB with whatever Linux distro or download and burn the latest versionof memtest86+ from https://www.memtest.org/  on Windows Desktop with some proggie like lets say UnetBootIn.
 

2.1. Run memtest86+ on Ubuntu

Many Linux distributions such as Ubuntu 20.0 comes together with memtest86+, which can be easily invoked from GRUB / GRUB2 Kernel boot loader.
Ubuntu has a separate menu pointer for a Memtest.

ubuntu-grub-2-04-boot-loader-memtest86-menu-screenshot

Other distributions RPM based distributions such as CentOS, Fedora Linux, Redhat things differ.

2.2. memtest86+ on Fedora


Fedora used to have the memtest86+ menu at the GRUB boot selection prompt, but for some reason removed it and in newest Fedora releases as of time such as Fedora 35 memtest86+ is preinstalled and available but not visible, to start on  already and to start a memtest memory test tool:

  •   Boot a Fedora installation or Rescue CD / USB. At the prompt, type "memtest86".

boot: memtest86

2.3 memtest86+ on RHEL Linux

The memtest86+tool is available as an RPM package from Red Hat Network (RHN) as well as a boot option from the Red Hat Enterprise Linux rescue disk.
And nowadays Red Hat Enterprise Linux ships by default with the tool.

Prior redhat (now legacy) releases such as on RHEL 5.0 it has to be installed and configure it with below 3 commands.

[root@rhel ~]# yum install memtest86+
[root@rhel ~]# memtest-setup
[root@rhel ~]# grub2-mkconfig -o /boot/grub2/grub.cfg


    Again as with CentOS to boot memtest86+ from the rescue disk, you will need to boot your system from CD 1 of the Red Hat Enterprise Linux installation media, and type the following at the boot prompt (before the Linux kernel is started):

boot: memtest86

memtestx86-8gigabytes-of-memory-boot-screenshot
memtest86+ testing 5 memory slots

As you see all on above screenshot the Memory banks are listed as Slots. There are a number of Tests to be completed until
it can be said for sure memory does not have any faulty cells. 
The

Pass: 0
Errors: 0 

Indicates no errors, so in the end if memtest86 does not find anything this values should stay at zero.
memtest86+ is also usable to detecting issues with temperature of CPU. Just recently I've tested a PC thinking that some memory has defects but it turned out the issue on the Computer was at the CPU's temperature which was topping up at 80 – 82 Celsius.

If you're unfortunate and happen to get some corrupted memory segments you will get some red fields with the memory addresses found to have corrupted on Read / Write test operations:

memtest86-returning-memory-address-errors-screenshot


2.4. Install and use memtest and memtest86+ on Debian / Mint Linux

You can install either memtest86+ or just for the fun put both of them and play around with both of them as they have a .deb package provided out of debian non-free /etc/apt/sources.list repositories.


root@jeremiah:/home/hipo# apt-cache show memtest86 memtest86+
Package: memtest86
Version: 4.3.7-3
Installed-Size: 302
Maintainer: Yann Dirson <dirson@debian.org>
Architecture: amd64
Depends: debconf (>= 0.5) | debconf-2.0
Recommends: memtest86+
Suggests: hwtools, memtester, kernel-patch-badram, grub2 (>= 1.96+20090523-1) | grub (>= 0.95+cvs20040624), mtools
Description-en: thorough real-mode memory tester
 Memtest86 scans your RAM for errors.
 .
 This tester runs independently of any OS – it is run at computer
 boot-up, so that it can test *all* of your memory.  You may want to
 look at `memtester', which allows testing your memory within Linux,
 but this one won't be able to test your whole RAM.
 .
 It can output a list of bad RAM regions usable by the BadRAM kernel
 patch, so that you can still use you old RAM with one or two bad bits.
 .
 This is the last DFSG-compliant version of this software, upstream
 has opted for a proprietary development model starting with 5.0.  You
 may want to consider using memtest86+, which has been forked from an
 earlier version of memtest86, and provides a different set of
 features.  It is available in the memtest86+ package.
 .
 A convenience script is also provided to make a grub-legacy-based
 floppy or image.

Description-md5: 0ad381a54d59a7d7f012972f613d7759
Homepage: http://www.memtest86.com/
Section: misc
Priority: optional
Filename: pool/main/m/memtest86/memtest86_4.3.7-3_amd64.deb
Size: 45470
MD5sum: 8dd2a4c52910498d711fbf6b5753bca9
SHA256: 09178eca21f8fd562806ccaa759d0261a2d3bb23190aaebc8cd99071d431aeb6

Package: memtest86+
Version: 5.01-3
Installed-Size: 2391
Maintainer: Yann Dirson <dirson@debian.org>
Architecture: amd64
Depends: debconf (>= 0.5) | debconf-2.0
Suggests: hwtools, memtester, kernel-patch-badram, memtest86, grub-pc | grub-legacy, mtools
Description-en: thorough real-mode memory tester
 Memtest86+ scans your RAM for errors.
 .
 This tester runs independently of any OS – it is run at computer
 boot-up, so that it can test *all* of your memory.  You may want to
 look at `memtester', which allows to test your memory within Linux,
 but this one won't be able to test your whole RAM.
 .
 It can output a list of bad RAM regions usable by the BadRAM kernel
 patch, so that you can still use your old RAM with one or two bad bits.
 .
 Memtest86+ is based on memtest86 3.0, and adds support for recent
 hardware, as well as a number of general-purpose improvements,
 including many patches to memtest86 available from various sources.
 .
 Both memtest86 and memtest86+ are being worked on in parallel.
Description-md5: aa685f84801773ef97fdaba8eb26436a
Homepage: http://www.memtest.org/

Tag: admin::benchmarking, admin::boot, hardware::storage:floppy,
 interface::text-mode, role::program, scope::utility, use::checking
Section: misc
Priority: optional
Filename: pool/main/m/memtest86+/memtest86+_5.01-3_amd64.deb
Size: 75142
MD5sum: 4f06523532ddfca0222ba6c55a80c433
SHA256: ad42816e0b17e882713cc6f699b988e73e580e38876cebe975891f5904828005
 

 

root@jeremiah:/home/hipo# apt-get install –yes memtest86+

root@jeremiah:/home/hipo# apt-get install –yes memtest86

Reading package lists… Done
Building dependency tree       
Reading state information… Done
Suggested packages:
  hwtools kernel-patch-badram grub2 | grub
The following NEW packages will be installed:
  memtest86
0 upgraded, 1 newly installed, 0 to remove and 21 not upgraded.
Need to get 45.5 kB of archives.
After this operation, 309 kB of additional disk space will be used.
Get:1 http://ftp.de.debian.org/debian buster/main amd64 memtest86 amd64 4.3.7-3 [45.5 kB]
Fetched 45.5 kB in 0s (181 kB/s)     
Preconfiguring packages …
Selecting previously unselected package memtest86.
(Reading database … 519985 files and directories currently installed.)
Preparing to unpack …/memtest86_4.3.7-3_amd64.deb …
Unpacking memtest86 (4.3.7-3) …
Setting up memtest86 (4.3.7-3) …
Generating grub configuration file …
Found background image: saint-John-of-Rila-grub.jpg
Found linux image: /boot/vmlinuz-4.19.0-18-amd64
Found initrd image: /boot/initrd.img-4.19.0-18-amd64
Found linux image: /boot/vmlinuz-4.19.0-17-amd64
Found initrd image: /boot/initrd.img-4.19.0-17-amd64
Found linux image: /boot/vmlinuz-4.19.0-8-amd64
Found initrd image: /boot/initrd.img-4.19.0-8-amd64
Found linux image: /boot/vmlinuz-4.19.0-6-amd64
Found initrd image: /boot/initrd.img-4.19.0-6-amd64
Found linux image: /boot/vmlinuz-4.19.0-5-amd64
Found initrd image: /boot/initrd.img-4.19.0-5-amd64
Found linux image: /boot/vmlinuz-4.9.0-8-amd64
Found initrd image: /boot/initrd.img-4.9.0-8-amd64
Found memtest86 image: /boot/memtest86.bin
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
File descriptor 3 (pipe:[66049]) leaked on lvs invocation. Parent PID 22581: /bin/sh
done
Processing triggers for man-db (2.8.5-2) …

 

After this both memory testers memtest86+ and memtest86 will appear next to the option of booting a different version kernels and the Advanced recovery kernels, that you usually get in the GRUB boot prompt.

2.5. Use memtest embedded tool on any Linux by adding a kernel variable

Edit-Grub-Parameters-add-memtest-4-to-kernel-boot

2.4.1. Reboot your computer

# reboot

2.4.2. At the GRUB boot screen (with UEFI, press Esc).

2.4.3 For 4 passes add temporarily the memtest=4 kernel parameter.
 

memtest=        [KNL,X86,ARM,PPC,RISCV] Enable memtest
                Format: <integer>
                default : 0 <disable>
                Specifies the number of memtest passes to be
                performed. Each pass selects another test
                pattern from a given set of patterns. Memtest
                fills the memory with this pattern, validates
                memory contents and reserves bad memory
                regions that are detected.


3. Install and use memtester Linux tool
 

At some condition, memory is the one of the suspcious part, or you just want have a quick test. memtester  is an effective userspace tester for stress-testing the memory subsystem.  It is very effective at finding intermittent and non-deterministic faults.

The advantage of memtester "live system check tool is", you can check your system for errors while it's still running. No need for a restart, just run that application, the downside is that some segments of memory cannot be thoroughfully tested as you already have much preloaded data in it to have the Operating Sytstem running, thus always when possible try to stick to rule to test the memory using memtest86+  from OS Boot Loader, after a clean Machine restart in order to clean up whole memory heap.

Anyhow for a general memory test on a Critical Legacy Server  (if you lets say don't have access to Remote Console Board, or don't trust the ILO / IPMI Hardware reported integrity statistics), running memtester from already booted is still a good idea.


3.1. Install memtester on any Linux distribution from source

wget http://pyropus.ca/software/memtester/old-versions/memtester-4.2.2.tar.gz
# tar zxvf memtester-4.2.2.tar.gz
# cd memtester-4.2.2
# make && make install

3.2 Install on RPM based distros

 

On Fedora memtester is available from repositories however on many other RPM based distros it is not so you have to install it from source.

[root@fedora ]# yum install -y memtester

 

3.3. Install memtester on Deb based Linux distributions from source
 

To install it on Debian / Ubuntu / Mint etc. , open a terminal and type:
 

root@linux:/ #  apt install –yes memtester

The general run syntax is:

memtester [-p PHYSADDR] [ITERATIONS]


You can hence use it like so:

hipo@linux:/ $ sudo memtester 1024 5

This should allocate 1024MB of memory, and repeat the test 5 times. The more repeats you run the better, but as a memtester run places a great overall load on the system you either don't increment the runs too much or at least run it with  lowered process importance e.g. by nicing the PID:

hipo@linux:/ $ nice -n 15 sudo memtester 1024 5

 

  • If you have more RAM like 4GB or 8GB, it is upto you how much memory you want to allocate for testing.
  • As your operating system, current running process might take some amount of RAM, Please check available Free RAM and assign that too memtester.
  • If you are using a 32 Bit System, you cant test more than 4 GB even though you have more RAM( 32 bit systems doesnt support more than 3.5 GB RAM as you all know).
  • If your system is very busy and you still assigned higher than available amount of RAM, then the test might get your system into a deadlock, leads to system to halt, be aware of this.
  • Run the memtester as root user, so that memtester process can malloc the memory, once its gets hold on that memory it will try to apply lock. if specified memory is not available, it will try to reduce required RAM automatically and try to lock it with mlock.
  • if you run it as a regular user, it cant auto reduce the required amount of RAM, so it cant lock it, so it tries to get hold on that specified memory and starts exhausting all system resources.


If you have 8 Gigas of RAM plugged into the PC motherboard you have to multiple 1024*8 this is easily done with bc (An arbitrary precision calculator language) tool:

root@linux:/ # bc -l
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
8*1024
8192


 for example you should run:

root@linux:/ # memtester 8192 5

memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 8192MB (2083520512 bytes)
got  8192MB (2083520512 bytes), trying mlock …Loop 1/1:
  Stuck Address       : ok        
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : ok        
  Block Sequential    : ok        
  Checkerboard        : ok        
  Bit Spread          : ok        
  Bit Flip            : ok        
  Walking Ones        : ok        
  Walking Zeroes      : ok        
  8-bit Writes        : ok
  16-bit Writes       : ok

Done.

 

4. Shell Script to test server memory for corruptions
 

If for some reason the machine you want to run a memory test doesn't have connection to the external network such as the internet and therefore you cannot configure a package repository server and install memtester, the other approach is to use a simple memory test script such as memtestlinux.sh
 

#!/bin/bash
# Downloaded from https://www.srv24x7.com/memtest-linux/
echo "ByteOnSite Memory Test"
cpus=`cat /proc/cpuinfo | grep processor | wc -l`
if [ $cpus -lt 6 ]; then
threads=2
else
threads=$(($cpus / 2))
fi
echo "Detected $cpus CPUs, using $threads threads.."
memory=`free | grep 'Mem:' | awk {'print $2'}`
memoryper=$(($memory / $threads))
echo "Detected ${memory}K of RAM ($memoryper per thread).."
freespace=`df -B1024 . | tail -n1 | awk {'print $4'}`
if [ $freespace -le $memory ]; then
echo You do not have enough free space on the current partition. Minimum: $memory bytes
exit 1
fi
echo "Clearing RAM Cache.."
sync; echo 3 > /proc/sys/vm/drop_cachesfile
echo > dump.memtest.img
echo "Writing to dump file (dump.memtest.img).."
for i in `seq 1 $threads`;
do
# 1044 is used in place of 1024 to ensure full RAM usage (2% over allocation)
dd if=/dev/urandom bs=$memoryper count=1044 >> dump.memtest.img 2>/dev/null &
pids[$i]=$!
echo $i
done
for pid in "${pids[@]}"
do
wait $pid
done

echo "Reading and analyzing dump file…"
echo "Pass 1.."
md51=`md5sum dump.memtest.img | awk {'print $1'}`
echo "Pass 2.."
md52=`md5sum dump.memtest.img | awk {'print $1'}`
echo "Pass 3.."
md53=`md5sum dump.memtest.img | awk {'print $1'}`
if [ “$md51” != “$md52” ]; then
fail=1
elif [ “$md51” != “$md53” ]; then
fail=1
elif [ “$md52” != “$md53” ]; then
fail=1
else
fail=0
fi
if [ $fail -eq 0 ]; then
echo "Memory test PASSED."
else
echo "Memory test FAILED. Bad memory detected."
fi
rm -f dump.memtest.img
exit $fail

Nota Bene !: Again consider the restults might not always be 100% trustable if possible restart the server and test with memtest86+

Consider also its important to make sure prior to script run,  you''ll have enough disk space to produce the dump.memtest.img file – file is created as a test bed for the memory tests and if not scaled properly you might end up with a full ( / ) root directory!

 

4.1 Other memory test script with dd and md5sum checksum

I found this solution on the well known sysadmin site nixCraft cyberciti.biz, I think it makes sense and quicker.

First find out memory site using free command.
 

# free
             total       used       free     shared    buffers     cached
Mem:      32867436   32574160     293276          0      16652   31194340
-/+ buffers/cache:    1363168   31504268
Swap:            0          0          0


It shows that this server has 32GB memory,
 

# dd if=/dev/urandom bs=32867436 count=1050 of=/home/memtest


free reports by k and use 1050 is to make sure file memtest is bigger than physical memory.  To get better performance, use proper bs size, for example 2048 or 4096, depends on your local disk i/o,  the rule is to make bs * count > 32 GB.
run

# md5sum /home/memtest; md5sum /home/memtest; md5sum /home/memtest


If you see md5sum mismatch in different run, you have faulty memory guaranteed.
The theory is simple, the file /home/memtest will cache data in memory by filling up all available memory during read operation. Using md5sum command you are reading same data from memory.


5. Other ways to test memory / do a machine stress test

Other good tools you might want to check for memory testing is mprime – ftp://mersenne.org/gimps/ 
(https://www.mersenne.org/ftp_root/gimps/)

  •  (mprime can also be used to stress test your CPU)

Alternatively, use the package stress-ng to run all kind of stress tests (including memory test) on your machine.
Perhaps there are other interesting tools for a diagnosis of memory if you know other ones I miss, let me know in the comment section.

How to redirect TCP port traffic from Internet Public IP host to remote local LAN server, Redirect traffic for Apache Webserver, MySQL, or other TCP service to remote host

Thursday, September 23rd, 2021

 

 

Linux-redirect-forward-tcp-ip-port-traffic-from-internet-to-remote-internet-LAN-IP-server-rinetd-iptables-redir

 

 

1. Use the good old times rinetd – internet “redirection server” service


Perhaps, many people who are younger wouldn't remember rinetd's use was pretty common on old Linuxes in the age where iptables was not on the scene and its predecessor ipchains was so common.
In the raise of mass internet rinetd started loosing its popularity because the service was exposed to the outer world and due to security holes and many exploits circulating the script kiddie communities
many servers get hacked "pwned" in the jargon of the script kiddies.

rinetd is still available even in modern Linuxes and over the last years I did not heard any severe security concerns regarding it, but the old paranoia perhaps and the set to oblivion makes it still unpopular soluttion for port redirect today in year 2021.
However for a local secured DMZ lans I can tell you that its use is mostly useful and I chooes to use it myself, everynow and then due to its simplicity to configure and use.
rinetd is pretty standard among unixes and is also available in old Sun OS / Solaris and BSD-es and pretty much everything on the Unix scene.

Below is excerpt from 'man rinetd':

 

DESCRIPTION
     rinetd redirects TCP connections from one IP address and port to another. rinetd is a single-process server which handles any number of connections to the address/port pairs
     specified in the file /etc/rinetd.conf.  Since rinetd runs as a single process using nonblocking I/O, it is able to redirect a large number of connections without a severe im‐
     pact on the machine. This makes it practical to run TCP services on machines inside an IP masquerading firewall. rinetd does not redirect FTP, because FTP requires more than
     one socket.
     rinetd is typically launched at boot time, using the following syntax:      /usr/sbin/rinetd      The configuration file is found in the file /etc/rinetd.conf, unless another file is specified using the -c command line option.

To use rinetd on any LInux distro you have to install and enable it with apt or yum as usual. For example on my Debian GNU / Linux home machine to use it I had to install .deb package, enable and start it it via systemd :

 

server:~# apt install –yes rinetd

server:~#  systemctl enable rinetd


server:~#  systemctl start rinetd


server:~#  systemctl status rinetd
● rinetd.service
   Loaded: loaded (/etc/init.d/rinetd; generated)
   Active: active (running) since Tue 2021-09-21 10:48:20 EEST; 2 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 1 (limit: 4915)
   Memory: 892.0K
   CGroup: /system.slice/rinetd.service
           └─1364 /usr/sbin/rinetd


rinetd is doing the traffic redirect via a separate process daemon, in order for it to function once you have service up check daemon is up as well.

root@server:/home/hipo# ps -ef|grep -i rinet
root       359     1  0 16:10 ?        00:00:00 /usr/sbin/rinetd
root       824 26430  0 16:10 pts/0    00:00:00 grep -i rinet

+ Configuring a new port redirect with rinetd

 

Is pretty straight forward everything is handled via one single configuration – /etc/rinetd.conf

The format (syntax) of a forwarding rule is as follows:

     [bindaddress] [bindport] [connectaddress] [connectport]


Besides that rinetd , could be used as a primitive firewall substitute to iptables, general syntax of allow deny an IP address is done with (allow, deny) keywords:
 

allow 192.168.2.*
deny 192.168.2.1?


To enable logging to external file ,you'll have to include in the configuration:

# logging information
logfile /var/log/rinetd.log

Here is an example rinetd.conf configuration, redirecting tcp mysql 3306, nginx on port 80 and a second web service frontend for ILO to server reachable via port 8888 and a redirect from External IP to local IP SMTP server.

 

#
# this is the configuration file for rinetd, the internet redirection server
#
# you may specify global allow and deny rules here
# only ip addresses are matched, hostnames cannot be specified here
# the wildcards you may use are * and ?
#
# allow 192.168.2.*
# deny 192.168.2.1?


#
# forwarding rules come here
#
# you may specify allow and deny rules after a specific forwarding rule
# to apply to only that forwarding rule
#
# bindadress    bindport  connectaddress  connectport


# logging information
logfile /var/log/rinetd.log
83.228.93.76        80            192.168.0.20       80
192.168.0.2        3306            192.168.0.19        3306
83.228.93.76        443            192.168.0.20       443
# enable for access to ILO
83.228.93.76        8888            192.168.1.25 443

127.0.0.1    25    192.168.0.19    25


83.228.93.76 is my external ( Public )  IP internet address where 192.168.0.20, 192.168.0.19, 192.168.0.20 (are the DMZ-ed Lan internal IPs) with various services.

To identify the services for which rinetd is properly configured to redirect / forward traffic you can see it with netstat or the newer ss command
 

root@server:/home/hipo# netstat -tap|grep -i rinet
tcp        0      0 www.pc-freak.net:8888   0.0.0.0:*               LISTEN      13511/rinetd      
tcp        0      0 www.pc-freak.n:http-alt 0.0.0.0:*               LISTEN      21176/rinetd        
tcp        0      0 www.pc-freak.net:443   0.0.0.0:*               LISTEN      21176/rinetd      

 

+ Using rinetd to redirect External interface IP to loopback's port (127.0.0.1)

 

If you have the need to redirect an External connectable living service be it apache mysql / privoxy / squid or whatever rinetd is perhaps the tool of choice (especially since there is no way to do it with iptables.

If you want to redirect all traffic which is accessed via Linux's loopback interface (localhost) to be reaching a remote host 11.5.8.1 on TCP port 1083 and 1888, use below config

# bindadress    bindport  connectaddress  connectport
11.5.8.1        1083            127.0.0.1       1083
11.5.8.1        1888            127.0.0.1       1888

 

For a quick and dirty solution to redirect traffic rinetd is very useful, however you'll have to keep in mind that if you want to redirect traffic for tens of thousands of connections constantly originating from the internet you might end up with some disconnects as well as notice a increased use of rinetd CPU use with the incrased number of forwarded connections.

 

2. Redirect TCP / IP port using DNAT iptables firewall rules

 

Lets say you have some proxy, webservice or whatever service running on port 5900 to be redirected with iptables.
The easeiest legacy way is to simply add the redirection rules to /etc/rc.local​. In newer Linuxes rc.local so if you decide to use,
you'll have to enable rc.local , I've written earlier a short article on how to enable rc.local on newer Debian, Fedora, CentOS

 

# redirect 5900 TCP service 
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -I PREROUTING -p tcp –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -I OUTPUT -p tcp -o lo –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp –dport 5900 -j DNAT  –to-destination 192.168.1.8:5900
iptables -t nat -I OUTPUT –source 0/0 –destination 0/0 -p tcp –dport 5900 -j REDIRECT –to-ports 5900

 

Here is another two example which redirects port 2208 (which has configured a bind listener for SSH on Internal host 192.168.0.209:2208) from External Internet IP address (XXX.YYY.ZZZ.XYZ) 
 

# Port redirect for SSH to VM on openxen internal Local lan server 192.168.0.209 
-A PREROUTING  -p tcp –dport 2208 -j DNAT –to-destination 192.168.0.209:2208
-A POSTROUTING -p tcp –dst 192.168.0.209 –dport 2208 -j SNAT –to-source 83.228.93.76

 

3. Redirect TCP traffic connections with redir tool

 

If you look for an easy straight forward way to redirect TCP traffic, installing and using redir (ready compiled program) might be a good idea.


root@server:~# apt-cache show redir|grep -i desc -A5 -B5
Version: 3.2-1
Installed-Size: 60
Maintainer: Lucas Kanashiro <kanashiro@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.15)
Description-en: Redirect TCP connections
 It can run under inetd or stand alone (in which case it handles multiple
 connections).  It is 8 bit clean, not limited to line mode, is small and
 light. Supports transparency, FTP redirects, http proxying, NAT and bandwidth
 limiting.
 .
 redir is all you need to redirect traffic across firewalls that authenticate
 based on an IP address etc. No need for the firewall toolkit. The
 functionality of inetd/tcpd and "redir" will allow you to do everything you
 need without screwy telnet/ftp etc gateways. (I assume you are running IP
 Masquerading of course.)

Description-md5: 2089a3403d126a5a0bcf29b22b68406d
Homepage: https://github.com/troglobit/redir
Tag: interface::daemon, network::server, network::service, role::program,
 use::proxying
Section: net
Priority: optional

 

 

server:~# apt-get install –yes redir

Here is a short description taken from its man page 'man redir'

 

DESCRIPTION
     redir redirects TCP connections coming in on a local port, [SRC]:PORT, to a specified address/port combination, [DST]:PORT.  Both the SRC and DST arguments can be left out,
     redir will then use 0.0.0.0.

     redir can be run either from inetd or as a standalone daemon.  In –inetd mode the listening SRC:PORT combo is handled by another process, usually inetd, and a connected
     socket is handed over to redir via stdin.  Hence only [DST]:PORT is required in –inetd mode.  In standalone mode redir can run either in the foreground, -n, or in the back‐
     ground, detached like a proper UNIX daemon.  This is the default.  When running in the foreground log messages are also printed to stderr, unless the -s flag is given.

     Depending on how redir was compiled, not all options may be available.

 

+ Use redir to redirect TCP traffic one time

 

Lets say you have a MySQL running on remote machine on some internal or external IP address, lets say 192.168.0.200 and you want to redirect all traffic from remote host to the machine (192.168.0.50), where you run your Apache Webserver, which you want to configure to use
as MySQL localhost TCP port 3306.

Assuming there are no irewall restrictions between Host A (192.168.0.50) and Host B (192.168.0.200) is already permitting connectivity on TCP/IP port 3306 between the two machines.

To open redirection from localhost on 192.168.0.50 -> 192.168.0.200:

 

server:~# redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306

 

If you need other third party hosts to be additionally reaching 192.168.0.200 via 192.168.0.50 TCP 3306.

root@server:~# redir –laddr=192.168.0.50 –lport=3306 –caddr=192.168.0.200 –cport=3306


Of course once you close, the /dev/tty or /dev/vty console the connection redirect will be cancelled.

 

+ Making TCP port forwarding from Host A to Host B permanent


One solution to make the redir setup rules permanent is to use –rinetd option or simply background the process, nevertheless I prefer to use instead GNU Screen.
If you don't know screen is a vVrtual Console Emulation manager with VT100/ANSI terminal emulation to so, if you don't have screen present on the host install it with whatever Linux OS package manager is present and run:

 

root@server:~#screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306'

 

That would run it into screen session and detach so you can later connect, if you want you can make redir to also log connections via syslog with ( -s) option.

I found also useful to be able to track real time what's going on currently with the opened redirect socket by changing redir log level.

Accepted log level is:

 

  -l, –loglevel=LEVEL
             Set log level: none, err, notice, info, debug.  Default is notice.

 

root@server:/ # screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3308 –caddr=192.168.0.200 –cport=3306 -l debug'

 

To test connectivity works as expected use telnet:
 

root@server:/ # telnet localhost 3308
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
g
5.5.5-10.3.29-MariaDB-0+deb10u1-log�+c2nWG>B���o+#ly=bT^]79mysql_native_password

6#HY000Proxy header is not accepted from 192.168.0.19 Connection closed by foreign host.

once you attach to screen session with

 

root@server:/home #  screen -r

 

You will get connectivity attempt from localhost logged : .
 

redir[10640]: listening on 127.0.0.1:3306
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10793]: peer IP is 127.0.0.1
redir[10793]: peer socket is 25592
redir[10793]: target IP address is 192.168.0.200
redir[10793]: target port is 3306
redir[10793]: Connecting 127.0.0.1:25592 to 127.0.0.1:3306
redir[10793]: Entering copyloop() – timeout is 0
redir[10793]: Disconnect after 1 sec, 165 bytes in, 4 bytes out

The downsides of using redir is redirection is handled by the separate process which is all time hanging in the process list, as well as the connection redirection speed of incoming connections might be about at least 30% slower to if you simply use a software (firewall ) redirect such as iptables. If you use something like kernel IP set ( ipsets ). If you hear of ipset for a first time and you wander whta it is below is short package description.

 

root@server:/root# apt-cache show ipset|grep -i description -A13 -B5
Maintainer: Debian Netfilter Packaging Team <pkg-netfilter-team@lists.alioth.debian.org>
Architecture: amd64
Provides: ipset-6.38
Depends: iptables, libc6 (>= 2.4), libipset11 (>= 6.38-1~)
Breaks: xtables-addons-common (<< 1.41~)
Description-en: administration tool for kernel IP sets
 IP sets are a framework inside the Linux 2.4.x and 2.6.x kernel which can be
 administered by the ipset(8) utility. Depending on the type, currently an
 IP set may store IP addresses, (TCP/UDP) port numbers or IP addresses with
 MAC addresses in a  way which ensures lightning speed when matching an
 entry against a set.
 .
 If you want to
 .
  * store multiple IP addresses or port numbers and match against the
    entire collection using a single iptables rule.
  * dynamically update iptables rules against IP addresses or ports without
    performance penalty.
  * express complex IP address and ports based rulesets with a single
    iptables rule and benefit from the speed of IP sets.

 .
 then IP sets may be the proper tool for you.
Description-md5: d87e199641d9d6fbb0e52a65cf412bde
Homepage: http://ipset.netfilter.org/
Tag: implemented-in::c, role::program
Section: net
Priority: optional
Filename: pool/main/i/ipset/ipset_6.38-1.2_amd64.deb
Size: 50684
MD5sum: 095760c5db23552a9ae180bd58bc8efb
SHA256: 2e2d1c3d494fe32755324bf040ffcb614cf180327736c22168b4ddf51d462522

CentOS disable SELinux permanently or one time on grub Linux kernel boot time

Saturday, July 24th, 2021

selinux-artistic-penguin-logo-protect-data

 

1. Office 365 cloud connected computer and a VirtualBox hosted machine with SELINUX preventing it to boot

At my job we're in process of migrating my old Lenovo Laptop Thinkpad model L560 Laptop to Dell Latitude 5510 wiith Intel Core i5 vPro CPU and 256 Gb SSD Hard Drive.  The new laptops are generally fiine though they're not even a middle class computers and generally I prefer thinkpads. The sad thing out of this is our employee decided to migrate to Office 365 (again perhaps another stupid managerial decision out of an excel sheet wtih a balance to save some money … 

As you can imagine Office 365 is not really PCI Standards compliant and not secure since our data is stored in Microsoft cloud and theoretically Microsoft has and owns our data or could wipe loose the data if they want to. The other obvious security downside I've noticed with the new "Secure PCI complaint laptop" is the initial PC login screen which by default offers fingerprint authentication or the even worse  and even less secure face recognition, but obviosly everyhing becomes more and more crazy and people become less and less cautious for security if that would save money or centralize the data … In the name of security we completely waste security that is very dubious paradox I don't really understand but anyways, enough rant back to the main topic of this article is how to and I had to disable selinux?

As part of Migration I've used Microsoft OneDrive to copy old files from the Thinkpad to the Latitude (as on the old machine USB's are forbidden and I cannot copy over wiith a siimple USB driive, as well as II have no right to open the laptop and copy data from the Hard driive, and even if we had this right without breaking up some crazy company policy that will not be possible as the hard drive data on old laptop is encrypted, the funny thing is that the new laptop data comes encrypted and there is no something out of the box as BitDefender or McAffee incryption (once again, obviously our data security is a victim of some managarial decisions) …
 

2. OneDrive copy problems unable to sync some of the copied files to Onedrive


Anyways as the Old Laptop's security is quite paranoid and we're like Fort Nox, only port 80 and port 443 connections to the internet can be initiated to get around this harsh restrictions it was as simple to use a Virtualbox Virtual Machine. So on old laptop I've installed a CentOS 7 image which I used so far and I used one drive to copy my vbox .vdi image on the new laptop work machine.

The first head buml was the .vdi which seems to be prohibited to be copied to OneDrive, so to work around this I had to rename the origianl CentOS7.vdi to CentOS7.vdi-renamed on old laptop and once the data is in one drive copy my Vitualbox VM/ directory from one drive to the Dell Latitude machine and rename the .vdi-named towards .vdi as well as import it from the latest installed VirtualBox on the new machine.
 

3. Disable SELINUX from initial grub boot


So far so good but as usual happens with miigrations I've struck towards another blocker, the VM image once initiated to boot from Virtualbox badly crashed with some complains that selinux cannot be loaded.
Realizing CentOS 7 has the more or less meaningless Selinux, I've took the opportunity to disable SeLinux.

To do so I've booted the Kernel with Selinux disabled from GRUB2 loader prompt before Kernel and OS Userland boots.

 

 

I thought I need to type the information on the source in grub. What I did is very simple, on the Linux GRUB boot screen I've pressed

'e' keyboard letter

that brought the grub boot loader into edit mode.

Then I had to add selinux=0 on the edited selected kernel version, as shown in below screenshot:

selinux-disable-from-grub.png

Next to boot the Linux VM without Selinux enabled one time,  just had to press together

Ctrl+X then add selinux=0 on the edited selected kernel version, that should be added as shown in the screenshot somewhere after the line of
root=/dev/mapper/….

4. Permanently Disable Selinux on CentOS 7


Once I managed to boot Virtual Machine properly with Oracle Virtualbox, to permanently disabled selinux I had to:

 

Once booted into CentOS, to check the status of selinux run:

 

# sestatus
Copy
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

 

5. Disable SELinux one time with setenforce command


You can temporarily change the SELinux mode from targeted to permissive with the following command:

 

# setenforce 0


Next o permanently disable SELinux on your CentOS 7 next time the system boots, Open the /etc/selinux/config file and set the SELINUX mod parameter to disabled.

On CentOS 7 you can  edit the kernel parameters in /etc/default/grub (in the GRUB_CMDLINE_LINUX= key) and set selinux=0 so on next VM / PC boot we boot with a SELINUX disabled for example add   RUB_CMDLINE_LINUX=selinux=0 to the file then you have to regenerate your Grub config like this:
 

# grub2-mkconfig -o /etc/grub2.cfg
# grub2-mkconfig -o /etc/grub2-efi.cfg


Further on to disable SeLinux on OS level edit /etc/selinux
 

Default /etc/selinux/config with selinux enabled should look like so:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.
SELINUXTYPE=targeted


To disable SeLinux modify the file to be something like:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.
SELINUXTYPE=targeted

6. Check SELINUX status is disabled

# sestatus

SELinux status:                 disabled

So in this article shottly was explained shortly the fake security adopted by using Microsoft Cloud environment Offiice 365, my faced OneDrive copy issues (which prevented even my old laptop Virtual Machine to boot properly and the handy trick to rename the file that is unwilling to get copied from old PC towards m$ OneDrive as well as the grub trick to disable Selinux permanently from grub2.

How to configure static DNS and Search domain for Redhat / CentOS and Redhat Linux

Wednesday, February 3rd, 2021

Fedora-Red-Hat-and-CentOS-fix-DNS-resolv-conf-automatically-deleted-records
In latest Redhat based OS-es Fedora / CentOS / Redhat etc. just like on many other Linux distributions, we have /etc/resolv.conf being overwritten by NetworkManager and / or systemd configurations setup which since some time has been introduced to be a "more sophisticated" (default)  so the file is being written by Network Manager / dhcp or systemd config. Though the idea is good, having other programs modify /etc/resolv.conf is a real pain in the ass especially as you end up with an empty file because some service has overwritten what you have placed in the file and the DNS records and Search Domain is deleted forever. If you're not aware of this "new cool" linux feature you might first think that it was a bug that has ovewritten /etc/resolv.conf but ok guys as Bill Gates loved to say "this is not a bug it is a feature", so any attemps you make to manually change /etc/resolv.conf will be soon gone 🙂

This is pretty annoying for old school sysadmins which like to just set the necessery Domain name server resolving

search Subdomain.SearchDomain.Com
nameserver xxx.xxx.xxx.xxx
nameserver yyy.yyy.yyy.yyy


However as said Nowdays if you just place the desired config with in /etc/resolv.conf on next Server reboot or Network restart (or next fetch of DHCP if the ethernet interface IPs are being obtained via DHCP protocol) you will end up in a situation with an empty /etc/resolv.conf  with one commented line reading:

[root@redhat ~]# cat /etc/resolv.conf
# Generated by NetworkManager

To make the DNS and Search Domain be always presented on any network restart or reboot on the server hence you will need to define  DNS1 DNS2 DNS3 etc. and SEARCH variable inside the network configuration files for Bridge or Network interfaces located in /etc/sysconfig/network-scripts/ifcfg-br0 /etc/sysconfig/network-scripts/ifcfg-eno1np0 etc.  that will automatically append above search / nameserver fields in /etc/resolv.conf on any NetworkManager or system restart.
Below is example with the variables added to a Network bridge configuration on Redhat 8.3 (Ootpa):

[root@redhat ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
UUID=f87e54a8-0fc4-4197-8ccc-0d8a671f30d0
DEVICE=br0
ONBOOT=yes
IPADDR=10.10.51.16
PREFIX=26
GATEWAY=10.10.51.1

DNS1="172.80.11.2"
DNS2="172.80.11.3"
DNS3="172.80.11.4"
SEARCH="sub.search-domain.com"


To test the configuration does append proper records into /etc/resolv.conf on Network restart  or /sbin/reboot reload the network.

[root@redhat ~]# systemctl restart NetworkManager


The result is you should have a good looking resolv.conf as so

[root@redhat ~]#  cat /etc/resolv.conf
# Generated by NetworkManager
search sub.search-domain.com
nameserver 172.20.88.2
nameserver 172.20.88.3
nameserver 172.20.88.4

 

Debug and fix Virtuozzo / KVM broken Hypervisor error: ‘PrlSDKError(‘SDK error: 0x80000249: Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down.’ on CentOS Linux howto

Thursday, January 28th, 2021

fix-sdkerror-virtuozzo-kvm-how-to-debug-problems-with-hypervisor-host-linux

I've recently yum upgraded a CentOS Linux server runinng Virtuozzo kernel and Virtuozzo virtualization Virtual Machines to the latest available CentOS Linux release 7.9.2009 (Core) just to find out after the upgrade there was issues with both virtuozzo (VZ) way to list installed VZ enabled VMs reporting Unable to connect to Virtuozzo error like below:
 

[root@CENTOS etc]# prlctl list -a
Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down. Contact your Virtuozzo administrator for assistance.


Even the native QEMU KVM VMs installed on the Hypervisor system failed to work to list and bring up the VMs producing another unexplainable error with virsh unable to connect to the hypervisor socket

[root@CENTOS etc]# virsh list –all
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory


In dmesg cmd kernel log messages the error found looked as so:

[root@CENTOS etc]# dmesg|grep -i sdk


[    5.314601] PrlSDKError('SDK error: 0x80000249: Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down. Contact your Virtuozzo administrator for assistance.',)

To fix it I had to experiment a bit based on some suggestions from Google results as usual and what turned to be the cause is a now obsolete setting for disk probing that is breaking libvirtd

Disable allow_disk_format_probing in /etc/libvirt/qemu.conf

The fix to PrlSDKError('SDK error: 0x80000249: Unable to connect to Virtuozzo comes to commenting a parameter inside 

/etc/libvirt/qemu.conf

which for historical reasons seems to be turned on by default it is like this

allow_disk_format_probing = 1


Resolution is to either change the value to 0 or completely comment the line:

[root@CENTOS etc]# grep allow_disk_format_probing /etc/libvirt/qemu.conf
# If allow_disk_format_probing is enabled, libvirt will probe disk
#allow_disk_format_probing = 1
#allow_disk_format_probing = 1


Debug problems with Virtuozzo services and validate virtualization setup

What really helped to debug the issue was to check the extended status info of libvirtd.service vzevent vz.service libvirtguestd.service prl-disp systemd services

[root@CENTOS etc]# systemctl -l status libvirtd.service vzevent vz.service libvirtguestd.service prl-disp

Here I had to analyze the errors and googled a little bit about it


Once this is changed I had to of course restart libvirtd.service and rest of virtuozzo / kvm services

[root@CENTOS etc]# systemctl restart libvirtd.service ibvirtd.service vzevent vz.service libvirtguest.service prl-disp


Another useful tool part of a standard VZ install that I've used to make sure each of the Host OS Hypervisor components is running smoothly is virt-host-validate (tool is part of libvirt-client rpm package)

[root@CENTOS etc]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller mount-point                  : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpu' controller mount-point                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'cpuset' controller mount-point                  : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'devices' controller mount-point                 : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking for cgroup 'blkio' controller mount-point                   : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS


One thing to note here that virt-host-validate helped me to realize the  fuse (File system in userspace) module kernel support enabled on the HV was missing so I've enabled temporary for this boot with modprobe and permanently via a configuration like so:

# to load it one time
[root@CENTOS etc]#  modprobe fuse
# to load fuse permnanently on next boot

[root@CENTOS etc]#  echo fuse >> /etc/modules-load.d/fuse.conf

Disable selinux on CentOS HV

Another thing was selinux was enabled on the HV. Selinux is really annoying thing and to be honest I never used it on any server and though its idea is quite good the consequences it creates for daily sysadmin work are terrible so I usually disable it. It could be that a Hypervisor Host OS might work just normal with the selinux enabled but just in case I decided to remove it. This is how

[root@CENTOS etc]#  sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

To temporarily change the SELinux mode from targeted to permissive with the following command:

[root@CENTOS etc]#  setenforce 0

Edit /etc/selinux/config file and set the SELINUX mod to disabled

[root@CENTOS etc]# vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.
SELINUXTYPE=targeted

Finally rebooted graceously the machine just in case with the good recommended way to reboot servers with shutdown command instead of /sbin/reboot

[root@CENTOS etc]# shutdown -r now

The advantage of shutdown is that it tries to shutdown each service by sending stop requests but usually this takes some time and even a shutdown request could take longer to proccess as each service such as a WebServer application is being waited to close all its network connections etc. |
However if you want to have a quick reboot and you don't care about any established network connections to third party IPs you can go for the brutal old fashioned /sbin/reboot 🙂

Listing installed RPMs by vendor installed on CentOS / RedHat Linux

Friday, January 8th, 2021

Listing installed RPMs by vendor installed on CentOS / RedHat Linux

Listing installed RPMs by vendor is useful sysadmin stuff if you have third party software installed that is not part of official CentOS / RedHat Linux and you want to only list this packages, here is how this is done

 

[root@redhat ~]# rpm -qa –qf '%{NAME} %{VENDOR} %{PACKAGER} \n' | grep -v 'CentOS' | sort

criu Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
gskcrypt64 IBM IBM
gskssl64 IBM IBM
ipxe-roms-qemu Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libevent (none) (none)
libguestfs-appliance Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libguestfs-tools-c Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libguestfs Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlcommon Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlsdk-python Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlsdk Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlxmlmodel Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libtcmu Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvcmmd Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-client Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-config-nwfilter Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-interface Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-network Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-nodedev Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-nwfilter Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-qemu Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-storage-core Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-storage Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-kvm Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-libs Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-python Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvzctl Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvzevent Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
openvz-logos Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
p7zip-plugins Fedora Project Fedora Project
ploop-lib Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
ploop Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
prlctl Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
prl-disk-tool Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
prl-disp-service Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
python2-lockfile Fedora Project Fedora Project
python2-psutil Fedora Project Fedora Project
python-daemon Fedora Project Fedora Project
python-subprocess32 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qemu-img-vz Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qemu-kvm-common-vz Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qemu-kvm-vz Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qt Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
rkhunter Fedora Project Fedora Project
seabios-bin Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
seavgabios-bin Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
spfs Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
TIVsm-API64 IBM (none)
TIVsm-APIcit IBM (none)
TIVsm-BAcit IBM (none)
TIVsm-BA IBM (none)
vcmmd Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vmauth Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vzctl Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vzkernel Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vzkernel Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt_checker Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt_checker Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt-lib Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
zabbix-agent (none) (none)

 


That instructs rpm to output each package's name and vendor, then we exclude those from "Red Hat, Inc." (which is the exact string Red Hat conveniently uses in the "vendor" field of all RPMs they pacakge).

By default, rpm -qa uses the format '%{NAME}-%{VERSION}-%{RELEASE}', and it's nice to see version and release, and on 64-bit systems, it's also nice to see the architecture since both 32- and 64-bit packages are often installed. Here's how I did that:

[root@redhat ~]# rpm -qa –qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR} %{PACKAGER} \n' | grep -v 'CentOS' | sort

criu-3.10.0.23-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
gskcrypt64-8.0-55.17.x86_64 IBM IBM
gskssl64-8.0-55.17.x86_64 IBM IBM
ipxe-roms-qemu-20170123-1.git4e85b27.1.vz7.5.noarch Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libevent-2.0.22-1.rhel7.x86_64 (none) (none)
libguestfs-1.36.10-6.2.vz7.12.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libguestfs-appliance-1.36.10-6.2.vz7.12.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libguestfs-tools-c-1.36.10-6.2.vz7.12.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlcommon-7.0.162-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlsdk-7.0.226-2.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlsdk-python-7.0.226-2.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libprlxmlmodel-7.0.80-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libtcmu-1.2.0-16.2.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvcmmd-7.0.22-3.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-client-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-config-nwfilter-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-interface-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-network-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-nodedev-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-nwfilter-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-qemu-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-storage-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-driver-storage-core-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-daemon-kvm-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-libs-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvirt-python-3.9.0-1.vz7.1.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvzctl-7.0.506-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
libvzevent-7.0.7-5.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
openvz-logos-70.0.13-1.vz7.noarch Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
p7zip-plugins-16.02-10.el7.x86_64 Fedora Project Fedora Project
ploop-7.0.137-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
ploop-lib-7.0.137-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
prlctl-7.0.164-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
prl-disk-tool-7.0.43-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
prl-disp-service-7.0.925-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
python2-lockfile-0.11.0-17.el7.noarch Fedora Project Fedora Project
python2-psutil-5.6.7-1.el7.x86_64 Fedora Project Fedora Project
python-daemon-1.6-4.el7.noarch Fedora Project Fedora Project
python-subprocess32-3.2.7-1.vz7.5.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qemu-img-vz-2.10.0-21.7.vz7.67.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qemu-kvm-common-vz-2.10.0-21.7.vz7.67.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qemu-kvm-vz-2.10.0-21.7.vz7.67.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
qt-4.8.7-2.vz7.2.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
rkhunter-1.4.6-2.el7.noarch Fedora Project Fedora Project
seabios-bin-1.10.2-3.1.vz7.3.noarch Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
seavgabios-bin-1.10.2-3.1.vz7.3.noarch Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
spfs-0.09.0010-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
TIVsm-API64-8.1.11-0.x86_64 IBM (none)
TIVsm-APIcit-8.1.11-0.x86_64 IBM (none)
TIVsm-BA-8.1.11-0.x86_64 IBM (none)
TIVsm-BAcit-8.1.11-0.x86_64 IBM (none)
vcmmd-7.0.160-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vmauth-7.0.10-2.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vzctl-7.0.194-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vzkernel-3.10.0-862.11.6.vz7.64.7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vzkernel-3.10.0-862.20.2.vz7.73.29.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt-7.0.63-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt_checker-7.0.2-1.vz7.i686 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt_checker-7.0.2-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
vztt-lib-7.0.63-1.vz7.x86_64 Virtuozzo Virtuozzo (http://www.virtuozzo.com/support/)
zabbix-agent-3.2.11-1.el7.x86_64 (none) (none)

Howto Upgrade IBM Spectrum Protect Backup Client TSM 7.X to 8.1.8, Update Tivoli 8.1.8 to 8.1.11 on CentOS and Redhat Linux

Thursday, December 3rd, 2020

 

IBM-spectrum-protect-backup-logo-tivoli-tsm-logo

Having another day of a system administrator boredom, we had a task to upgrade some Tivoli TSM Backup clients running on a 20+ machines powered by CentOS and RHEL Linux to prepare the systems to be on the latest patched IBM Spectrum Backup client version available from IBM. For the task of patching I've used a central server where, I've initially downloaded the provided TSM client binaries archives. From this machine, we have copied TivSM*.tar to each and every system that needs to be patched and then patched. The task is not too complex as the running TSM in the machines are all at the same version and all running a recent patched version of Linux. Hence to make sure all works as expected we have tested TSM is upgraded from 7.X.X to 8.X.X on one machine and then test 8.1.8 to 8.1.11 upgrade on another one. Once having confirmed that Backups works as expected after upgrade. We have proceeded to do it massively on each of the rest 20+ hosts.
Below article's goal is to help some lazy sysadmin with the task to prepare an TSM Backup upgrade procedure to standartize TSM Upgrade, which as many of the IBM's softwares is very specific and its upgrade requires, a bit of manual work and extra cautious as there seems to be no easy way (or at least I don't know it), to do the upgrade by simply adding an RPM repository and doing, something like yum install tivsm*.


0. Check if there is at least 2G free of space

According to documentation the minimum space you need to a functional install without having it half installed or filling up your filesystem is 2 Gigabytes of Free Memory on a filesystem where the .tar and rpms will be living.

Thus check what is the situation with your filesystem where you wills store the .tar archice and extract .RPM files / install the RPM files.

# df -h

1. Download the correct tarball with 8.1 Client

On one central machine you would need to download the Tivoli you can do that via wget / curl / lynx whatever is at hand on the Linux server.

As of time of writting this article TSM's 8.1.11 location is at
URL:

http://public.dhe.ibm.com/storage/tivoli-storage-management/maintenance/client/v8r1/Linux/LinuxX86/BA/v8111/

I've made a local download mirror of Tivoli TSM 8.1.11 here.
In case you need to install IBM Spectrum Backup Client to a PCI secured environment to a DMZ-ed LAN network from a work PC you can Download it first from your local PC and via Citrix client upload program or WinSCP upload it to a central replication host from where you will later copy to each of the other server nodes that needs to be upgraded.

Lets Copy archive to all Server hosts where you want it later installed, using a small hack

Assuming you already have an Excel document or a Plain text document with all the IPs of the affected hosts where you will need to get TSM upgraded. Extract this data and from it create a plain text file /home/user/hosts.txt containing all the machine IPs lined up separated with carriage return separations (\n), so you can loop over each one and use scp to send the files.

– Replicate Tivoli tar to all machine hosts where you want to get IBM Spectrum installed or upgraded.
Do it with a loop like this:

# for i in $(cat hosts.txt); do scp 8.1.11.0-TIV-TSMBAC-LinuxX86.tar user@$i:/home/user/; done

 Copy to a Copy buffer temporary your server password assuming all your passwords to each machine are identical and paste your login user pass for each host to initiate transfer
 

2. SSH to each of the Machine hosts IPs

Once you login to the host you want to upgrade
Go to your user $HOME /home/user and create files where we'll temporary store Tivoli archive files and extract RPMs

[root@linux-server user]# mkdir -p ~/tsm/TSM_BCK/
[root@linux-server user]# mv 8.1.11.0-TIV-TSMBAC-LinuxX86.tar ~/tsm
[root@linux-server user]# cd tsm
[root@linux-server user]# tar -xvvf 8.1.11.0-TIV-TSMBAC-LinuxX86.tar
gskcrypt64-8.0.55.17.linux.x86_64.rpm
GSKit.pub.pgp
gskssl64-8.0.55.17.linux.x86_64.rpm
README_api.htm
README.htm
RPM-GPG-KEY-ibmpkg
TIVsm-API64.x86_64.rpm
TIVsm-APIcit.x86_64.rpm
TIVsm-BAcit.x86_64.rpm
TIVsm-BAhdw.x86_64.rpm
TIVsm-BA.x86_64.rpm
TIVsm-filepath-source.tar.gz
TIVsm-JBB.x86_64.rpm
TIVsm-WEBGUI.x86_64.rpm
update.txt

3. Create backup of old backup files

It is always a good idea to keep old backup files

[root@linux-server tsm]# cp -av /opt/tivoli/tsm/client/ba/bin/dsm.opt ~/tsm/TSM_BCK/dsm.opt_bak_$(date +'%Y_%M_%H')
[root@linux-server tsm]# cp -av /opt/tivoli/tsm/client/ba/bin/dsm.sys ~/tsm/TSM_BCK/dsm.sys_bak_$(date +'%Y_%M_%H')

[root@linux-server tsm]# [[ -f /etc/adsm/TSM.PWD ]] && cp -av /etc/adsm/TSM.PWD ~/TSM_BCK/ || echo 'file doesnt exist'

/etc/adsm/TSM.PWD this file is only there as legacy for TSM it contained encrypted passwords inver 7 for updates. In TSM v.8 encryption file is not there as new mechanism for sensitive data was introduced.
Be aware that from Tivoli 8.X it will return error
exist'

!! Note – if dsm.opt , dsm.sys files are on different locations – please use correct full path locations !!

4. Stop  dsmcad – TSM Service daemon

[root@linux-server tsm]# systemctl stop dsmcad

5. Locate and deinstall all old Clients

Depending on the version to upgrade if you're upgrading from TSM version 7 to 8, you will get output like.

[root@linux-server tsm]# rpm -qa | grep 'TIVsm-'
TIVsm-BA-7.1.6-2.x86_64
TIVsm-API64-7.1.6-2.x86_64

If you're one of this paranoid admins you can remove TIVsm packs  one by one.

[root@linux-server tsm]# rpm -e TIVsm-BA-7.1.6-2.x86_64
[root@linux-server tsm]# rpm -e TIVsm-API64-7.1.6-2.x86_64

Instead if upgrading from version 8.1.8 to 8.1.11 due to the Security CVE advisory recently published by IBM e.g. (IBM Runtime Vulnerability affects IBM Spectrum Backup archive Client) and  vulnerability in Apache Commons Log4J affecting IBM Spectrum Protect Backup Archive Client.

[root@linux-server tsm]# rpm -qa | grep 'TIVsm-'
TIVsm-API64-8.1.8-0.x86_64
TIVsm-BA-8.1.8-0.x86_64

Assuming you're not scared of a bit automation you can straight do it with below one liner too 🙂

# rpm -e $(rpm -qa | grep TIVsm)

[root@linux-server tsm]# rpm -qa | grep gsk
[root@linux-server tsm]# rpm -e gskcrypt64 gskssl64

6. Check uninstallation success:

[root@linux-server tsm]# rpm -qa | grep TIVsm
[root@linux-server tsm]# rpm -qa | grep gsk

Here you should an Empty output, if packages are not on the system, e.g. Empty output is good output ! 🙂

7. Install new client IBM Spectrum Client (Tivoli Storage Manager) and lib dependencies

[root@linux-server tsm]# rpm -ivh gskcrypt64-8.0.55.4.linux.x86_64.rpm
[root@linux-server tsm]# rpm -ivh gskssl64-8.0.55.4.linux.x86_64.rpm

 If you're lazy to type you can do as well

[root@linux-server tsm]# rpm -Uvh gsk*

Next step is to install main Tivoli SM components the the API files and BA (The Backup Archive Client)

[root@linux-server tsm]# rpm -ivh TIVsm-API64.x86_64.rpm
[root@linux-server tsm]# rpm -ivh TIVsm-BA.x86_64.rpm

If you have to do it on multiple servers and you do it manually following a guide like this, you might instead want to install them with one liner.

[root@linux-server tsm]# rpm -ivh TIVsm-API64.x86_64.rpm TIVsm-BA.x86_64.rpm

There are some Not mandatory "Common Inventory Technology" components (at some cases if you're using the API install it we did not need that), just for the sake if you need them on your servers due to backup architecture, install also below commented rpm files.

## rpm -ivh TIVsm-APIcit.x86_64.rpm

## rpm -ivh TIVsm-BAcit.x86_64.rpm

These packages not needed only for operation WebGUI TSM GUI management, (JBB) Journal Based Backup, BAhdw (the ONTAP library)


— TIVsm-WEBGUI.x86_64.rpm
— TIVsm-JBB.x86_64.rpm
— TIVsm-BAhdw.x86_64.rpm

8. Start and enable dsmcad service

[root@linux-server tsm]# systemctl stop dsmcad

You will get

##Warning: dsmcad.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[root@linux-server tsm]# systemctl daemon-reload

[root@linux-server tsm]# systemctl start dsmcad


## enable dsmcad – it is disabled by default after install

[root@linux-server ~]# systemctl enable dsmcad

[root@linux-server tsm]# systemctl status dsmcad

9. Check dmscad service is really running

Once enabled IBM TSM will spawn a process in the bacground dmscad if it started properly you should have the process backgrounded.

[root@linux-server tsm]# ps -ef|grep -i dsm|grep -v grep
root      2881     1  0 18:05 ?        00:00:01 /usr/bin/dsmcad

If process is not there there might be some library or something not at place preventing the process to start …

10. Check DSMCAD /var/tsm logs for errors

After having dsmcad process enabled and running in background

[root@linux-server tsm]# grep -i Version /var/tsm/sched.log|tail -1
12/03/2020 18:06:29   Server Version 8, Release 1, Level 10.000

 

[root@linux-server tsm]# cat /var/tsm/dsmerror.log

To see the current TSM configuration files we can  grep out comments *

[root@linux-server tsm]# grep -v '*' /opt/tivoli/tsm/client/ba/bin/dsm.sys

Example Configuration of the agent:
—————————————————-
   *TSM SERVER NODE Location
   Servername           tsm_server
   COMMmethod           TCPip
   TCPPort              1400
   TCPServeraddress     tsmserver2.backuphost.com
   NodeName             NODE.SERVER-TO-BACKUP-HOSTNAME.COM
   Passwordaccess       generate
   SCHEDLOGNAME         /var/tsm/sched.log
   SCHEDLOGRETENTION    21 D
   SCHEDMODE            POLLING
   MANAGEDServices      schedule
   ERRORLOGNAME         /var/tsm/dsmerror.log
   ERRORLOGRETENTION    30 D
   INCLEXCL             /opt/tivoli/tsm/client/ba/bin/inclexcl.tsm

11. Remove tsm install directory tar ball and rpms to save space on system

The current version of Tivoli service manager is 586 Megabytes.

[root@linux-server tsm]# du -hsc 8.1.11.0-TIV-TSMBAC-LinuxX86.tar
586M    8.1.11.0-TIV-TSMBAC-LinuxX86.tar

Some systems are on purpose configured to have less space under their /home directory,
hence it is a good idea to clear up unnecessery files after completion.

Lets get rid of all the IBM Spectrum archive source files and the rest of RPMs used for installation.

[root@linux-server tsm]# rm -rf ~/tsm/{*.tar,*.rpm,*.gpg,*.htm,*.txt}

12. Check backups are really created on the configured remote Central backup server

To make sure after the upgrade the backups are continuously created and properly stored on the IBM Tivoly remote central backup server, either manually initiate a backup or wait for lets say a day and run dsmc client to show all created backups from previous day. To make sure you'll not get empty output you can on purpose modify some file by simply opening it and writting over without chaning anything e.g. modify your ~/.bashrc or ~/.bash_profile

## List all backups for '/' root directory from -fromdate='DD/MM/YY'

[root@linux-server tsm]# dsmc
Protect>
IBM Spectrum Protect
Command Line Backup-Archive Client Interface
  Client Version 8, Release 1, Level 11.0
  Client date/time: 12/03/2020 18:14:03
(c) Copyright by IBM Corporation and other(s) 1990, 2020. All Rights Reserved.

Node Name: NODE.SERVER-TO-BACKUP-HOSTNAME.COM
Session established with server TSM2_SERVER: AIX
  Server Version 8, Release 1, Level 10.000
  Server date/time: 12/03/2020 18:14:04  Last access: 12/03/2020 18:06:29
 
Protect> query backup -subdir=yes "/" -fromdate=12/3/2020
           Size        Backup Date                Mgmt Class           A/I File
           —-        ———–                ———-           — —-
         6,776  B  12/03/2020 01:26:53             DEFAULT              A  /etc/freshclam.conf
         6,685  B  12/03/2020 01:26:53             DEFAULT              A  /etc/freshclam.conf-2020-12-02
         5,602  B  12/03/2020 01:26:53             DEFAULT              A  /etc/hosts
         5,506  B  12/03/2020 01:26:53             DEFAULT              A  /etc/hosts-2020-12-02
           398  B  12/03/2020 01:26:53             DEFAULT              A  /opt/tivoli/tsm/client/ba/bin/tsmstats.ini
       114,328  B  12/03/2020 01:26:53             DEFAULT              A  /root/.bash_history
           403  B  12/03/2020 01:26:53             DEFAULT              A  /root/.lesshst