Virtual Machines Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Archive for the ‘Virtual Machines’ Category

Howto convert KVM QCOW2 format Virtual Machine to Vmdk to migrate to VMware ESXi

Thursday, November 17th, 2022

Why you would want to convert qcow2 to vmdk?

When managing the heterogeneous virtual environment or changing the virtualization solutions that become so common nowadays, you might need to migrate qcow2 from a Linux based KVM virtualization solution to VMWare's proprietary vmdk – the file format in which a VMWare does keep stored it's VMs, especially if you have a small business or work in a small start-up company where you cannot afford to buy something professional as VMware vCenter Converter Standalone or Microsoft virtual machine converter (MVMC)- usually used to to migrate VMware hosts to Hyper-V hosts, but also capable to migrate .qcow2 to .vmdk. The reason is that your old datacenter based on Linux OS custom KVM virtual machines might be moved to VMWare ESX to guarantee better and more systemized management (which though is very questionable, since most of my experiences with VMWare was that though the software was a great one, the people who manage it was not very much specialists in managing it).

Another common reason is that running a separate Linux virtual machine, costs you more than a well organized VMWare farm because you need more qualified Linux specialists to manage the KVMs thus KVM to VMWare management as in most big corporations nowadays’s main target is to cut the costs.
Even with successful migrations like that, though you might often expect a drop in the quality of the service when your VM ends in the VMWare farm.

Nomatter what’s the reason to migrate qcow2 to VMDK So lets proceed with how the .QCOW2 to .VMDK can be easily done.

1. Get information about the VM you would like to migrate to VMDK

In QEMU-KVM environment, the popular image format is qcow2, which outperforms the first generation of qcow format and raw format. You can find the files of virtual disks by checking the information of virtual machine by virsh command:

[root@hypervisor-machine ~]# virsh dominfo virtual-machine-name

INFO
ID: {e59ae416-9314-4e4b-af07-21c31d91b3fb}
EnvID: 1704649750
Name: CentOS7minimal
Description:
Type: VM
State: stopped
OS: centos7
Template: no
Uptime: 00:00:00 (since 2019-04-25 13:04:11)
Home: /vz/vmprivate/e39ae416-9314-4e4b-af05-21c31d91b3fb/
Owner: root@.
GuestTools: state=not_installed
GuestTools autoupdate: on
Autostart: off
Autostop: shutdown
Autocompact: off
Boot order: hdd0 cdrom0
EFI boot: off
Allow select boot device: off
External boot device:
On guest crash: restart
Remote display: mode=manual port=6903 address=0.0.0.0
Remote display state: stopped
Hardware:
cpu sockets=1 cpus=2 cores=2 VT-x accl=high mode=64 ioprio=4 iolimit='0'
memory 2048Mb
video 32Mb 3d acceleration=off vertical sync=yes
memory_guarantee auto
hdd0 (+) scsi:0 image='/vz/vmprivate/e59ae415-9314-4e4b-af05-21c31d91b3fb/harddisk.hdd' type='expanded' 5120Mb subtype=virtio-scsi
cdrom0 (+) scsi:1 image='/home/CentOS-7-x86_64-Minimal-1611.iso' state=disconnected subtype=virtio-scsi
usb (+)
net0 (+) dev='vme42bef5f3' network='Bridged' mac=001C42BEF5F3 card=virtio ips='10.50.50.27/255.255.255.192 ' gw='10.50.50.1'
SmartMount: (-)
Disabled Windows logo: on
Nested virtualization: off
Offline management: (-)
Hostname: kvmhost.fqdn.com

2. Convert the harddrive to VMDK

[root@hypervisor-machine e59ae415-9314-4e4b-af05-21c31d91b3fb]# ls -lsah

1.3G -rw-r—– 1 root root 1.3G Apr 25 14:43 harddisk.hdd

a. Converstion with qemu:

You can use qemu-img tool that is installable via cmds:

yum install quemu-img / apt install qemu-img / zipper install qemu-img (depending on the distribution RedHat / Debian / SuSE Linux)

-f: format of the source image

-O: format of the target image

[root@hypervisor-machine ~]# qemu-img convert -f qcow2 -O vmdk \-o adapter_type=lsilogic,subformat=streamOptimized,compat6 harddisk.hdd harddisklsilogic.vmdk

[root@ hypervisor-machine e59ae415-9314-4e4b-af05-21c31d91b3fb]# ls -lsah

1.3G -rw-r—– 1 root root 1.3G Apr 25 14:43 harddisk.hdd

536M -rw-r–r– 1 root root 536M Apr 26 14:52 harddisklsilogic.vmdk

3. Upload the new harddrive to the ESXi Hypervisor and adapt it to ESX

This vmdk might not be able to used on ESXi, but you can use it on VMware Workstation. To let it work on ESXi, you need to use vmkfstools to convert it again.

a. Adapt the filesystem to ESXi

[root@hypervisor-machine ~]# vmkfstools -i harddisklsilogic.vmdk -d thin harddisk.vmdk

4. Create a VM and add the converted harddrive to the machine.

Futher

Recreate the initramfs

But of course this won’t work directly as it often happens with Linux 🙂 !!.
We need to make adjustments to the virtual machine as well with few manual interventions:

1. Start the machine from the VMWare interface

2. Grub CentOS Linux rescue will appear from the prompt

3. Run command

dracut –regenerate-all –force

to Recreate the initramfs.

Note that You might also have to edit your network configuration since your network device usually get’s a different name.

Finally reboot the host:

[root@hypervisor-machine ~]# reboot

And voila you’re ready to play the VM inside the ESX after some testing, you might switch off the KVM Hypervisor hosted VM and reroute the network to point to the ESX Cluster.

Tags: command, Convert, External, MVMC, Offline, qemu, Remote, shutdown, Start, video
Posted in Linux, Migration, System Administration, Virtual Machines | No Comments »

How to RPM update Hypervisors and Virtual Machines running Haproxy High Availability cluster on KVM, Virtuozzo without a downtime on RHEL / CentOS Linux

Friday, May 20th, 2022

Here is the scenario, lets say you have on your daily task list two Hypervisor (HV) hosts running CentOS or RHEL Linux with KVM or Virutozzo technology and inside the HV hosts you have configured at least 2 pairs of virtual machines one residing on HV Host 1 and one residing on HV Host 2 and you need to constantly keep the hosts to the latest distribution major release security patchset.

The Virtual Machines has been running another set of Redhat Linux or CentOS configured to work in a High Availability Cluster running Haproxy / Apache / Postfix or any other kind of HA solution on top of corosync / keepalived or whatever application cluster scripts Free or Open Source technology that supports a switch between clustered Application nodes.

The logical question comes how to keep up the CentOS / RHEL Machines uptodate without interfering with the operations of the Applications running on the cluster?

Assuming that the 2 or more machines are configured to run in Active / Passive App member mode, e.g. one machine is Active at any time and the other is always Passive, a switch is possible between the Active and Passive node.

HAProxy--Load-Balancer-cluster-2-nodes-your-Servers

In this article I'll give a simple step by step tested example on how you I succeeded to update (for security reasons) up to the latest available Distribution major release patchset on one by one first the Clustered App on Virtual Machines 1 and VM2 on Linux Hypervisor Host 1. Then the App cluster VM 1 / VM 2 on Hypervisor Host 2.
And finally update the Hypervisor1 (after moving the Active resources from it to Hypervisor2) and updating the Hypervisor2 after moving the App running resources back on HV1.
I know the procedure is a bit monotonic but it tries to go through everything step by step to try to mitigate any possible problems. In case of failure of some rpm dependencies during yum / dnf tool updates you can always revert to backups so in anyways don't forget to have a fully functional backup of each of the HV hosts and the VMs somewhere on a separate machine before proceeding further, any possible failures due to following my aritcle literally is your responsibility 🙂

0. Check situation before the update on HVs / get VM IDs etc.

Check the virsion of each of the machines to be updated both Hypervisor and Hosted VMs, on each machine run:

# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

The machine setup I'll be dealing with is as follows:

hypervisor-host1 -> hypervisor-host1.fqdn.com
• virt-mach-centos1
• virt-machine-zabbix-proxy-centos (zabbix proxy)

hypervisor-host2 -> hypervisor-host2.fqdn.com
• virt-mach-centos2
• virt-machine-zabbix2-proxy-centos (zabbix proxy)

To check what is yours check out with virsh cmd –if on KVM or with prlctl if using Virutozzo, you should get something like:

[root@hypervisor-host2 ~]# virsh list
Id Name State
—————————————————-
1 vm-host1 running
2 virt-mach-centos2 running

# virsh list –all

[root@hypervisor-host1 ~]# virsh list
Id Name State
—————————————————-
1 vm-host2 running
3 virt-mach-centos1 running

[root@hypervisor-host1 ~]# prlctl list
UUID STATUS IP_ADDR T NAME
{dc37c201-08c9-589d-aa20-9386d63ce3f3} running – VM virt-mach-centos1
{76e8a5f8-caa8-5442-830e-aa4bfe8d42d9} running – VM vm-host2
[root@hypervisor-host1 ~]#

If you have stopped VMs with Virtuozzo to list the stopped ones as well.

# prlctl list -a

[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# vzlist -a
CTID NPROC STATUS IP_ADDR HOSTNAME
[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# prlctl list
UUID STATUS IP_ADDR T NAME
{92075803-a4ce-5ec0-a3d8-9ee83d85fc76} running – VM virt-mach-centos2
{74a7bbe8-9245-5385-ac0d-d10299100789} running – VM vm-host1

# prlctl list -a

If due to Virtuozzo version above command does not return you can manually check in the VM located folder, VM ID etc.

[root@hypervisor-host2 vmprivate]# ls
74a7bbe8-9245-4385-ac0d-d10299100789 92075803-a4ce-4ec0-a3d8-9ee83d85fc76
[root@hypervisor-host2 vmprivate]# pwd
/vz/vmprivate
[root@hypervisor-host2 vmprivate]#

[root@hypervisor-host1 ~]# ls -al /vz/vmprivate/
total 20
drwxr-x—. 5 root root 4096 Feb 14 2019 .
drwxr-xr-x. 7 root root 4096 Feb 13 2019 ..
drwxr-x–x. 4 root root 4096 Feb 18 2019 1c863dfc-1deb-493c-820f-3005a0457627
drwxr-x–x. 4 root root 4096 Feb 14 2019 76e8a5f8-caa8-4442-830e-aa4bfe8d42d9
drwxr-x–x. 4 root root 4096 Feb 14 2019 dc37c201-08c9-489d-aa20-9386d63ce3f3
[root@hypervisor-host1 ~]#

Before doing anything with the VMs, also don't forget to check the Hypervisor hosts has enough space, otherwise you'll get in big troubles !

[root@hypervisor-host2 vmprivate]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos_hypervisor-host2-root 20G 1.8G 17G 10% /
devtmpfs 20G 0 20G 0% /dev
tmpfs 20G 0 20G 0% /dev/shm
tmpfs 20G 2.0G 18G 11% /run
tmpfs 20G 0 20G 0% /sys/fs/cgroup
/dev/sda1 992M 159M 766M 18% /boot
/dev/mapper/centos_hypervisor-host2-home 9.8G 37M 9.2G 1% /home
/dev/mapper/centos_hypervisor-host2-var 9.8G 355M 8.9G 4% /var
/dev/mapper/centos_hypervisor-host2-vz 755G 25G 692G 4% /vz

[root@hypervisor-host1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 1.8G 45G 4% /
devtmpfs 20G 0 20G 0% /dev
tmpfs 20G 0 20G 0% /dev/shm
tmpfs 20G 2.1G 18G 11% /run
tmpfs 20G 0 20G 0% /sys/fs/cgroup
/dev/sda2 992M 153M 772M 17% /boot
/dev/mapper/centos-home 9.8G 37M 9.2G 1% /home
/dev/mapper/centos-var 9.8G 406M 8.9G 5% /var
/dev/mapper/centos-vz 689G 12G 643G 2% /vz

Another thing to do before proceeding with update is to check and tune if needed the amount of CentOS repositories used, before doing anything with yum.

[root@hypervisor-host2 yum.repos.d]# ls -al
total 68
drwxr-xr-x. 2 root root 4096 Oct 6 13:13 .
drwxr-xr-x. 110 root root 12288 Oct 7 11:13 ..
-rw-r–r–. 1 root root 4382 Mar 14 2019 CentOS7.repo
-rw-r–r–. 1 root root 1664 Sep 5 2019 CentOS-Base.repo
-rw-r–r–. 1 root root 1309 Sep 5 2019 CentOS-CR.repo
-rw-r–r–. 1 root root 649 Sep 5 2019 CentOS-Debuginfo.repo
-rw-r–r–. 1 root root 314 Sep 5 2019 CentOS-fasttrack.repo
-rw-r–r–. 1 root root 630 Sep 5 2019 CentOS-Media.repo
-rw-r–r–. 1 root root 1331 Sep 5 2019 CentOS-Sources.repo
-rw-r–r–. 1 root root 6639 Sep 5 2019 CentOS-Vault.repo
-rw-r–r–. 1 root root 1303 Mar 14 2019 factory.repo
-rw-r–r–. 1 root root 666 Sep 8 10:13 openvz.repo
[root@hypervisor-host2 yum.repos.d]#

[root@hypervisor-host1 yum.repos.d]# ls -al
total 68
drwxr-xr-x. 2 root root 4096 Oct 6 13:13 .
drwxr-xr-x. 112 root root 12288 Oct 7 11:09 ..
-rw-r–r–. 1 root root 1664 Sep 5 2019 CentOS-Base.repo
-rw-r–r–. 1 root root 1309 Sep 5 2019 CentOS-CR.repo
-rw-r–r–. 1 root root 649 Sep 5 2019 CentOS-Debuginfo.repo
-rw-r–r–. 1 root root 314 Sep 5 2019 CentOS-fasttrack.repo
-rw-r–r–. 1 root root 630 Sep 5 2019 CentOS-Media.repo
-rw-r–r–. 1 root root 1331 Sep 5 2019 CentOS-Sources.repo
-rw-r–r–. 1 root root 6639 Sep 5 2019 CentOS-Vault.repo
-rw-r–r–. 1 root root 1303 Mar 14 2019 factory.repo
-rw-r–r–. 1 root root 300 Mar 14 2019 obsoleted_tmpls.repo
-rw-r–r–. 1 root root 666 Sep 8 10:13 openvz.repo

1. Dump VM definition XMs (to have it in case if it gets wiped during update)

There is always a possibility that something will fail during the update and you might be unable to restore back to the old version of the Virtual Machine due to some config misconfigurations or whatever thus a very good idea, before proceeding to modify the working VMs is to use KVM's virsh and dump the exact set of XML configuration that makes the VM roll properly.

To do so:
Check a little bit up in the article how we have listed the IDs that are part of the directory containing the VM.

[root@hypervisor-host1 ]# virsh dumpxml (Id of VM virt-mach-centos1 ) > /root/virt-mach-centos1_config_bak.xml
[root@hypervisor-host2 ]# virsh dumpxml (Id of VM virt-mach-centos2) > /root/virt-mach-centos2_config_bak.xml

2. Set on standby virt-mach-centos1 (virt-mach-centos1)

As I'm upgrading two machines that are configured to run an haproxy corosync cluster, before proceeding to update the active host, we have to switch off
the proxied traffic from node1 to node2, – e.g. standby the active node, so the cluster can move up the traffic to other available node.

[root@virt-mach-centos1 ~]# pcs cluster standby virt-mach-centos1

3. Stop VM virt-mach-centos1 & backup on Hypervisor host (hypervisor-host1) for VM1

Another prevention step to make sure you don't get into damaged VM or broken haproxy cluster after the upgrade is to of course backup

[root@hypervisor-host1 ]# prlctl backup virt-mach-centos1

[root@hypervisor-host1 ]# prlctl stop virt-mach-centos1
[root@hypervisor-host1 ]# cp -rpf /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3 /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3-bak
[root@hypervisor-host1 ]# tar -czvf virt-mach-centos1_vm_virt-mach-centos1.tar.gz /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3

[root@hypervisor-host1 ]# prlctl start virt-mach-centos1

4. Remove package version locks on all hosts

If you're using package locking to prevent some other colleague to not accidently upgrade the machine (if multiple sysadmins are managing the host), you might use the RPM package locking meachanism, if that is used check RPM packs that are locked and release the locking.

+ List actual list of locked packages

[root@hypervisor-host1 ]# yum versionlock list
…
…..
0:libtalloc-2.1.16-1.el7.*
0:libedit-3.0-12.20121213cvs.el7.*
0:p11-kit-trust-0.23.5-3.el7.*
1:quota-nls-4.01-19.el7.*
0:perl-Exporter-5.68-3.el7.*
0:sudo-1.8.23-9.el7.*
0:libxslt-1.1.28-5.el7.*
versionlock list done

+ Clear the locking

# yum versionlock clear

+ List actual list / == clear all entries

[root@virt-mach-centos2 ]# yum versionlock list; yum versionlock clear
[root@virt-mach-centos1 ]# yum versionlock list; yum versionlock clear
[root@hypervisor-host1 ~]# yum versionlock list; yum versionlock clear
[root@hypervisor-host2 ~]# yum versionlock list; yum versionlock clear

5. Do yum update virt-mach-centos1

For some clarity if something goes wrong, it is really a good idea to make a dump of the basic packages installed before the RPM package update is initiated,
The exact versoin of RHEL or CentOS as well as the list of locked packages, if locking is used.

Enter virt-mach-centos1 (ssh virt-mach-centos1) and run following cmds:

# cat /etc/redhat-release > /root/logs/redhat-release-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

+ Only if needed!!

# yum versionlock clear
# yum versionlock list

Clear any previous RPM packages – careful with that as you might want to keep the old RPMs, if unsure comment out below line

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Proceed with the update and monitor closely the output of commands and log out everything inside files using a small script that you should place under /root/status the script is given at the end of the aritcle.:

yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
yum check-update | wc -l
yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

6. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

7. Stop VM virt-mach-centos2 & backup on Hypervisor host (hypervisor-host2)

Same backup step as prior

# prlctl backup virt-mach-centos2

# prlctl stop virt-mach-centos2
# cp -rpf /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76 /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76-bak
## tar -czvf virt-mach-centos2_vm_virt-mach-centos2.tar.gz /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76

# prctl start virt-mach-centos2

8. Do yum update on virt-mach-centos2

Log system state, before the update

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list

+ Clean old install update / packages if required

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Initiate the update

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

9. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

10. Stop VM vm-host2 & backup

# prlctl backup vm-host2

# prlctl stop vm-host2

Or copy the actual directory containig the Virtozzo VM (use the correct ID)

# cp -rpf /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9 /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9-bak
## tar -czvf vm-host2.tar.gz /vz/vmprivate/76e8a5f8-caa8-4442-830e-aa5bfe8d42d9

# prctl start vm-host2

11. Do yum update vm-host2

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear only if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Do the rpm upgrade

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

12. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

13. Do yum update hypervisor-host2

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Update rpms

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

14. Stop VM vm-host1 & backup

Some as ealier

# prlctl backup vm-host1

# prlctl stop vm-host1

# cp -rpf /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789 /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789-bak
# tar -czvf vm-host1.tar.gz /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789

# prctl start vm-host1

15. Do yum update vm-host2

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

16. Check if everything is running fine after upgrade

+ Reboot VM

# shutdown -r now

17. Do yum update hypervisor-host1

Same procedure for HV host 1

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

18. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

Check hypervisor-host1 all VMs run as expected

19. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

Check hypervisor-host2 all VMs run as expected afterwards

20. Check once more VMs and haproxy or any other contained services in VMs run as expected

21. Haproxy Unstandby virt-mach-centos1

Assuming that the virt-mach-centos1 and virt-mach-centos2 are running a Haproxy / corosync cluster you can try to standby node1 and check the result
hopefully all should be fine and traffic should come to host node2.

[root@virt-mach-centos1 ~]# pcs cluster unstandby virt-mach-centos1

Monitor logs and make sure HAproxy works fine on virt-mach-centos1

22. If necessery to redefine VMs (in case they disappear from virsh) or virtuosso is not working

[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos1_config_bak.xml
[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos2_config_bak.xml

23. Set versionlock to RPMs to prevent accident updates and check OS version release

[root@virt-mach-centos2 ]# yum versionlock \*
[root@virt-mach-centos1 ]# yum versionlock \*
[root@hypervisor-host1 ~]# yum versionlock \*
[root@hypervisor-host2 ~]# yum versionlock \*

[root@hypervisor-host2 ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)

Other useful hints

[root@hypervisor-host1 ~]# virsh console dc37c201-08c9-489d-aa20-9386d63ce3f3
Connected to domain virt-mach-centos1
..

! Compare packages count before the upgrade on each of the supposable identical VMs and HVs – if there is difference in package count review what kind of packages are different and try to make the machines to look as identical as possible !

Packages to update on hypervisor-host1 Count: XXX
Packages to update on hypervisor-host2 Count: XXX
Packages to update virt-mach-centos1 Count: – 254
Packages to update virt-mach-centos2 Count: – 249

The /root/status script

+++

#!/bin/sh
echo '======================================================= '
echo '= Systemctl list-unit-files –type=service | grep enabled '
echo '======================================================= '
systemctl list-unit-files –type=service | grep enabled

echo '======================================================= '
echo '= systemctl | grep ".service" | grep "running" '
echo '======================================================= '
systemctl | grep ".service" | grep "running"

echo '======================================================= '
echo '= chkconfig –list '
echo '======================================================= '
chkconfig –list

echo '======================================================= '
echo '= netstat -tulpn '
echo '======================================================= '
netstat -tulpn

echo '======================================================= '
echo '= netstat -r '
echo '======================================================= '
netstat -r

+++

That's all folks, once going through the article, after some 2 hours of efforts or so you should have an up2date machines.
Any problems faced or feedback is mostly welcome as this might help others who have the same setup.

Thanks for reading me 🙂

Tags: backups, check, cmds, Connected, downtime, echo, Linux Hypervisor Host, list, logs, repo, root root, rpm, Set, setup, updates, virtual machines, yum versionlock
Posted in CentOS, Linux, Linux Package Management, System Administration, Virtual Machines | 1 Comment »

Disable VNC on KVM Virtual Machine without VM restart / How to Change VNC listen address

Monday, February 28th, 2022

disable-vnc-port-listener-on-a-KVM-ran-virtual-machine-virsh-libvirt-libvirt-architecture-design

Say you have recently run a new KVM Virtual machine, have connected via VNC on lets say the default tcp port 5900
installed a brand new Linux OS using a VNC client to connect, such as:
TightVNC / RealVNC if connecting from Windows Client machine or Vncviewer / Remmina if connecting from Linux / BSD and now
you want to turn off the VNC VM listener server either for security reasons to make sure some script kiddie random scanner did not manage to connect and take control over your VM or just because, you will be only further using the new configured VM only via SSH console sessions as they call it in modern times to make a buziness buzz out of it a headless UNIX server (server machines connected a network without a Physical monitor attached to it).

The question comes then how can be the KVM VNC listener on TCP port 5900 be completely disabled?

One way of course is to filter out with a firewall 5900 completely either on a Switch Level (lets say on a Cisco equipment catalist in front of the machine) or the worst solution to locally filter directly on the server with firewalld or iptables chain rules.

1. Disable KVM VNC Port listener via VIRSH VM XML edit

The better way of course is to completely disable the VNC using KVM, that is possible through the virsh command interface.
By editing the XML Virtual Machine configuration and finding the line about vnc confiuguration with:

root@server:/kvm/disk# virsh edit pcfreakweb
Domain pcfreakweb XML configuration not changed.

like:

<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>

and set value to undefined:

port='-1'

virsh-KVM-disable-VNC-port-listener-virsh-xml-edit-screenshot

Modifying the XML however will require you to reboot the Virtual Machine for which XML was editted. This might be not possible
if you have a running production server already configured with Apache / Proxy / PostgreSQL / Mail or any other Internet public service.

2. Disable VNC KVM TCP port 5900 to a dynamic running VM without a machine reboot

Thus if you want to remove the KVM VNC Port Listener on 5900 without a VM shutdown / reboot you can do it via KVM's virsh client interface.

root@server:/kvm/disk# virsh
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
'quit' to quit

virsh # qemu-monitor-command pcfreakweb –hmp change vnc none

The virsh management user interface client, can do pretty much more of real time VM changes, it is really useful to use it if you have KVM Hypervisor hosts with 10+ Virtual machines and it if you have to deal with KVM machines on daily, do specific changes to the VMs on how VM networks are configured, information on HV hardware, configure / reconfigure storage volumes to VMs etc, take some time to play with it 🙂

Tags: address, apache2, domain, help, kvm, machines, none, port, root, root server, tcp, terminal, user interface, virsh, Vncviewer Remmina, welcome, Windows Client, xml
Posted in Curious Facts, Linux, Virtual Machines | No Comments »

How to filter dhcp traffic between two networks running separate DHCP servers to prevent IP assignment issues and MAC duplicate addresses

Tuesday, February 8th, 2022

Tracking the Problem of MAC duplicates on Linux routers

If you have two networks that see each other and they're not separated in VLANs but see each other sharing a common netmask lets say 255.255.254.0 or 255.255.252.0, it might happend that there are 2 dhcp servers for example (isc-dhcp-server running on 192.168.1.1 and dhcpd running on 192.168.0.1 can broadcast their services to both LANs 192.168.1.0.1/24 (netmask 255.255.255.0) and Local Net LAN 192.168.1.1/24. The result out of this is that some devices might pick up their IP address via DHCP from the wrong dhcp server.

Normally if you have a fully controlled little or middle class home or office network (10 – 15 electronic devices nodes) connecting to the LAN in a mixed moth some are connected via one of the Networks via connected Wifi to 192.168.1.0/22 others are LANned and using static IP adddresses and traffic is routed among two ISPs and each network can see the other network, there is always a possibility of things to go wrong. This is what happened to me so this is how this post was born.

The best practice from my experience so far is to define each and every computer / phone / laptop host joining the network and hence later easily monitor what is going on the network with something like iptraf-ng / nethogs / iperf – described in prior how to check internet spepeed from console and in check server internet connectivity speed with speedtest-cli) iftop / nload or for more complex stuff wireshark or even a simple tcpdump. No matter the tools network monitoring is only part on solving network issues. A very must have thing in a controlled network infrastructure is defining every machine part of it to easily monitor later with the monitoring tools. Defining each and every host on the Hybrid computer networks makes administering the network much easier task and tracking irregularities on time is much more likely.

Since I have such a hybrid network here hosting a couple of XEN virtual machines with Linux, Windows 7 and Windows 10, together with Mac OS X laptops as well as MacBook Air notebooks, I have followed this route and tried to define each and every host based on its MAC address to pick it up from the correct DHCP1 server 192.168.1.1 (that is distributing IPs for Internet Provider 1 (ISP 1), that is mostly few computers attached UTP LAN cables via LiteWave LS105G Gigabit Switch as well from DHCP2 – used only to assigns IPs to servers and a a single Wi-Fi Access point configured to route incoming clients via 192.168.0.1 Linux NAT gateway server.

To filter out the unwanted IPs from the DHCPD not to propagate I've so far used a little trick to Deny DHCP MAC Address for unwanted clients and not send IP offer for them.

To give you more understanding, I have to clear it up I don't want to have automatic IP assignments from DHCP2 / LAN2 to DHCP1 / LAN1 because (i don't want machines on DHCP1 to end up with IP like 192.168.0.50 or DHCP2 (to have 192.168.1.80), as such a wrong IP delegation could potentially lead to MAC duplicates IP conflicts. MAC Duplicate IP wrong assignments for those older or who have been part of administrating large ISP network infrastructures makes the network communication unstable for no apparent reason and nodes partially unreachable at times or full time …

However it seems in the 21-st century which is the century of strangeness / computer madness in the 2022, technology advanced so much that it has massively started to break up some good old well known sysadmin standards well documented in the RFCs I know of my youth, such as that every electronic equipment manufactured Vendor should have a Vendor Assigned Hardware MAC Address binded to it that will never change (after all that was the idea of MAC addresses wasn't it !).
Many mobile devices nowadays however, in the developers attempts to make more sophisticated software and Increase Anonimity on the Net and Security, use a technique called MAC Address randomization (mostly used by hackers / script kiddies of the early days of computers) for their Wi-Fi Net Adapter OS / driver controlled interfaces for the sake of increased security (the so called Private WiFi Addresses). If a sysadmin 10-15 years ago has seen that he might probably resign his profession and turn to farming or agriculture plant growing, but in the age of digitalization and "cloud computing", this break up of common developed network standards starts to become the 'new normal' standard.

I did not suspected there might be a MAC address oddities, since I spare very little time on administering the the network. This was so till recently when I accidently checked the arp table with:

Hypervisor:~# arp -an
192.168.1.99 5c:89:b5:f2:e8:d8 (Unknown)
192.168.1.99 00:15:3e:d3:8f:76 (Unknown)

..

and consequently did a network MAC Address ARP Scan with arp-scan (if you never used this little nifty hacker tool I warmly recommend it !!!)
If you don't have it installed it is available in debian based linuces from default repos to install

Hypervisor:~# apt-get install –yes arp-scan
…

It is also available on CentOS / Fedora / Redhat and other RPM distros via:

Hypervisor:~# yum install -y arp-scan
…

Hypervisor:~# arp-scan –interface=eth1 192.168.1.0/24
…

192.168.1.19 00:16:3e:0f:48:05 Xensource, Inc.
192.168.1.22 00:16:3e:04:11:1c Xensource, Inc.
192.168.1.31 00:15:3e:bb:45:45 Xensource, Inc.
192.168.1.38 00:15:3e:59:96:8e Xensource, Inc.
192.168.1.34 00:15:3e:d3:8f:77 Xensource, Inc.
192.168.1.60 8c:89:b5:f2:e8:d8 Micro-Star INT'L CO., LTD
192.168.1.99 5c:89:b5:f2:e8:d8 (Unknown)
192.168.1.99   00:15:3e:d3:8f:76 (Unknown)
192.168.x.91     02:a0:xx:xx:d6:64        (Unknown)
192.168.x.91     02:a0:xx:xx:d6:64        (Unknown) (DUP: 2)
…

N.B. !. I found it helpful to check all available interfaces on my Linux NAT router host.

As you see the scan revealed, a whole bunch of MAC address mess duplicated MAC hanging around, destroying my network topology every now and then
So far so good, the MAC duplicates and strangely hanging around MAC addresses issue, was solved relatively easily with enabling below set of systctl kernel variables.

1. Fixing Linux ARP common well known Problems through disabling arp_announce / arp_ignore / send_redirects kernel variables disablement

Linux answers ARP requests on wrong and unassociated interfaces per default. This leads to the following two problems:

ARP requests for the loopback alias address are answered on the HW interfaces (even if NOARP on lo0:1 is set). Since loopback aliases are required for DSR (Direct Server Return) setups this problem is very common (but easy to fix fortunately).

If the machine is connected twice to the same switch (e.g. with eth0 and eth1) eth2 may answer ARP requests for the address on eth1 and vice versa in a race condition manner (confusing almost everything).

This can be prevented by specific arp kernel settings. Take a look here for additional information about the nature of the problem (and other solutions): ARP flux.

To fix that generally (and reboot safe) we include the following lines into

Hypervisor:~# cp -rpf /etc/sysctl.conf /etc/sysctl.conf_bak_07-feb-2022
Hypervisor:~# cat >> /etc/sysctl.conf

# LVS tuning
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.eth1.send_redirects=0
net.ipv4.conf.default.send_redirects=0

Press CTRL + D simultaneusly to Write out up-pasted vars.

2. Digging further the IP conflict / dulicate MAC Problems

Even after this arp tunings (because I do have my Hypervisor 2 LAN interfaces connected to 1 switch) did not resolved the issues and still my Wireless Connected devices via network 192.168.1.1/24 (ISP2) were randomly assigned the wrong range IPs 192.168.0.XXX/24 as well as the wrong gateway 192.168.0.1 (ISP1).
After thinking thoroughfully for hours and checking the network status with various tools and thanks to the fact that my wife has a MacBook Air that was always complaining that the IP it tried to assign from the DHCP was already taken, i"ve realized, something is wrong with DHCP assignment.
Since she owns a IPhone 10 with iOS and this two devices are from the same vendor e.g. Apple Inc. And Apple's products have been having strange DHCP assignment issues from my experience for quite some time, I've thought initially problems are caused by software on Apple's devices.
I turned to be partially right after expecting the logs of DHCP server on the Linux host (ISP1) finding that the phone of my wife takes IP in 192.168.0.XXX, insetad of IP from 192.168.1.1 (which has is a combined Nokia Router with 2.4Ghz and 5Ghz Wi-Fi and LAN router provided by ISP2 in that case Vivacom). That was really puzzling since for me it was completely logical thta the iDevices must check for DHCP address directly on the Network of the router to whom, they're connecting. Guess my suprise when I realized that instead of that the iDevices does listen to the network on a wide network range scan for any DHCPs reachable baesd on the advertised (i assume via broadcast) address traffic and try to connect and take the IP to the IP of the DHCP which responds faster !!!! Of course the Vivacom Chineese produced Nokia router responded DHCP requests and advertised much slower, than my Linux NAT gateway on ISP1 and because of that the Iphone and iOS and even freshest versions of Android devices do take the IP from the DHCP that responds faster, even if that router is not on a C class network (that's invasive isn't it??). What was even more puzzling was the automatic MAC Randomization of Wifi devices trying to connect to my ISP1 configured DHCPD and this of course trespassed any static MAC addresses filtering, I already had established there.

Anyways there was also a good think out of tthat intermixed exercise 🙂 While playing around with the Gigabit network router of vivacom I found a cozy feature SCHEDULEDING TURNING OFF and ON the WIFI ACCESS POINT – a very useful feature to adopt, to stop wasting extra energy and lower a bit of radiation is to set a swtich off WIFI AP from 12:30 – 06:30 which are the common sleeping hours or something like that.

3. What is MAC Randomization and where and how it is configured across different main operating systems as of year 2022?

Depending on the operating system of your device, MAC randomization will be available either by default on most modern mobile OSes or with possibility to have it switched on:

Android Q: Enabled by default
Android P: Available as a developer option, disabled by default
iOS 14: Available as a user option, disabled by default
Windows 10: Available as an option in two ways – random for all networks or random for a specific network

Lately I don't have much time to play around with mobile devices, and I do not my own a luxury mobile phone so, the fact this ne Androids have this MAC randomization was unknown to me just until I ended a small mess, based on my poor configured networks due to my tight time constrains nowadays.

Finding out about the new security feature of MAC Randomization, on all Android based phones (my mother's Nokia smartphone and my dad's phone, disabled the feature ASAP:

4. Disable MAC Wi-Fi Ethernet device Randomization on Android

MAC Randomization creates a random MAC address when joining a Wi-Fi network for the first time or after “forgetting” and rejoining a Wi-Fi network. It Generates a new random MAC address after 24 hours of last connection.

Disabling MAC Randomization on your devices. It is done on a per SSID basis so you can turn off the randomization, but allow it to function for hotspots outside of your home.

Open the Settings app
Select Network and Internet
Select WiFi
Connect to your home wireless network
Tap the gear icon next to the current WiFi connection
Select Advanced
Select Privacy
Select "Use device MAC"

5. Disabling MAC Randomization on MAC iOS, iPhone, iPad, iPod

To Disable MAC Randomization on iOS Devices:

Open the Settings on your iPhone, iPad, or iPod, then tap Wi-Fi or WLAN

Tap the information button next to your network
Turn off Private Address
Re-join the network

Of course next I've collected their phone Wi-Fi adapters and made sure the included dhcp MAC deny rules in /etc/dhcp/dhcpd.conf are at place.

The effect of the MAC Randomization for my Network was terrible constant and strange issues with my routings and networks, which I always thought are caused by the openxen hypervisor Virtualization VM bugs etc.

That continued for some months now, and the weird thing was the issues always started when I tried to update my Operating system to the latest packetset, do a reboot to load up the new piece of software / libraries etc. and plus it happened very occasionally and their was no obvious reason for it.

6. How to completely filter dhcp traffic between two network router hosts
IP 192.168.0.1 / 192.168.1.1 to stop 2 or more configured DHCP servers
on separate networks see each other

To prevent IP mess at DHCP2 server side (which btw is ISC DHCP server, taking care for IP assignment only for the Servers on the network running on Debian 11 Linux), further on I had to filter out any DHCP UDP traffic with iptables completely.
To prevent incorrect route assignments assuming that you have 2 networks and 2 routers that are configurred to do Network Address Translation (NAT)-ing Router 1: 192.168.0.1, Router 2: 192.168.1.1.

You have to filter out UDP Protocol data on Port 67 and 68 from the respective source and destination addresses.

In firewall rules configuration files on your Linux you need to have some rules as:

# filter outgoing dhcp traffic from 192.168.1.1 to 192.168.0.1
-A INPUT -p udp -m udp –dport 67:68 -s 192.168.1.1 -d 192.168.0.1 -j DROP
-A OUTPUT -p udp -m udp –dport 67:68 -s 192.168.1.1 -d 192.168.0.1 -j DROP
-A FORWARD -p udp -m udp –dport 67:68 -s 192.168.1.1 -d 192.168.0.1 -j DROP

-A INPUT -p udp -m udp –dport 67:68 -s 192.168.0.1 -d 192.168.1.1 -j DROP
-A OUTPUT -p udp -m udp –dport 67:68 -s 192.168.0.1 -d 192.168.1.1 -j DROP
-A FORWARD -p udp -m udp –dport 67:68 -s 192.168.0.1 -d 192.168.1.1 -j DROP

-A INPUT -p udp -m udp –sport 67:68 -s 192.168.1.1 -d 192.168.0.1 -j DROP
-A OUTPUT -p udp -m udp –sport 67:68 -s 192.168.1.1 -d 192.168.0.1 -j DROP
-A FORWARD -p udp -m udp –sport 67:68 -s 192.168.1.1 -d 192.168.0.1 -j DROP

You can download also filter_dhcp_traffic.sh with above rules from here

Applying this rules, any traffic of DHCP between 2 routers is prohibited and devices from Net: 192.168.1.1-255 will no longer wrongly get assinged IP addresses from Network range: 192.168.0.1-255 as it happened to me.

7. Filter out DHCP traffic based on MAC completely on Linux with arptables

If even after disabling MAC randomization on all devices on the network, and you know physically all the connecting devices on the Network, if you still see some weird MAC addresses, originating from a wrongly configured ISP traffic router host or whatever, then it is time to just filter them out with arptables.

## drop traffic prevent mac duplicates due to vivacom and bergon placed in same network – 255.255.255.252
dchp1-server:~# arptables -A INPUT –source-mac 70:e2:83:12:44:11 -j DROP

To list arptables configured on Linux host

dchp1-server:~# arptables –list -n

If you want to be paranoid sysadmin you can implement a MAC address protection with arptables by only allowing a single set of MAC Addr / IPs and dropping the rest.

dchp1-server:~# arptables -A INPUT –source-mac 70:e2:84:13:45:11 -j ACCEPT
dchp1-server:~# arptables -A INPUT –source-mac 70:e2:84:13:45:12 -j ACCEPT

dchp1-server:~# arptables -L –line-numbers
Chain INPUT (policy ACCEPT)
1 -j DROP –src-mac 70:e2:84:13:45:11
2 -j DROP –src-mac 70:e2:84:13:45:12
3

Once MACs you like are accepted you can set the INPUT chain policy to DROP as so:

dchp1-server:~# arptables -P INPUT DROP

If you later need to temporary, clean up the rules inside arptables on any filtered hosts flush all rules inside INPUT chain, like that

dchp1-server:~# arptables -t INPUT -F

Tags: android, Anyways, apple, apt-get, arp, arp-scan, asap, based, connection, default, DHCP, eth0, eth1, filter, How to, hybrid network, iftop, ios, ip arptables, ipv4, key, linux?, lvs, MAC, mac duplicate, mac os, machines, mess, Monitoring, network, Networking, Open, router, something, sysctl, traffic, two, UDP, variables, Vivacom Chineese, WiFi, Windows, WLAN, wrong, yum
Posted in Debian, Linux, Networking, System Administration, Virtual Machines | No Comments »

CentOS 8 / Redhat 8 insert additional guests additions to VM to enable Fullscreen, Copy / Paste and Shared Folder from host OS

Monday, January 10th, 2022

My experience with enabling virtualbox additions guest tools on many of the separate Linux distributions throughout time is pretty bad as it always is a pain in the ass to enable fully functional full screen and copy paste for Virtualbox…

For those who installed it for a first time vbox guest addition tools for Virtualbox are additional software components added so the Emulated Operating system
could allow better screen resolution and better mouse integration support.

So far I've installed virtualbox additions tools to CentOS 7 and Debian Linux various releases and faced complications there as well.
Few days ago my colleague Georgi Stoyanov have installed CentOS 8.3 with current version of VirtualBox 6.1 (vesrsion from beginning of 2022) and he has also shared had issues with enabling the CentOS 8.3 Linux to work with guestadditions but eventually found a resolution.

Thus he has shared with me the solution and I share it with you, so hopefully someone else could enable Guesttools on his CentOS 8.3 with less digging online.
The error received is:

# ./VBoxLinuxAdditions.run

Trying to install Guest Additions in RHEL 8.3.

VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules. This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup
VirtualBox Guest Additions: or
VirtualBox Guest Additions: /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Building the modules for kernel
4.18.0-193.el8.x86_64.

VirtualBox Guest Additions: Look at /var/log/vboxadd-setup.log to find out what
went wrong
ValueError: File context for /opt/VBoxGuestAdditions-6.0.20/other/mount.vboxsf already defined
VirtualBox Guest Additions: Running kernel modules will not be replaced until
the system is restarted
Press Return to close this window…

No idea what to do next. Been trying for sometime.

To enable guestaddtions in CentOS 8.3, e.g. get arount the error you have to:

1. Install all necessery dependncies RPMs required by GuestAddition tools

# dnf install tar bzip2 kernel-devel-$(uname -r) kernel-headers perl gcc make elfutils-libelf-devel

# dnf -y install gcc automake make kernel-headers dkms bzip2 libxcrypt-compat kernel-devel perl

2. Run below semanage and restorecon commands

# semanage fcontext -d /opt/VBoxGuestAdditions-/other/mount.vboxsf
# restorecon /opt/VBoxGuestAdditions-/other/mount.vboxsf

3. Insert Virtualbox guest additions ISO and Run it

Devices -> Insert Guest Additions CD Image

Click Run button to exec Vbox_GAs_6.0.18 script or run it manually

or mount it manually with mount command and execute the VBoxLinuxAdditions.run to do so:

$ cd /run/media/`whoami`/VB*
$ su
# ./VBoxLinuxAdditions.run
Installing additional modules …
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules. This may take a while.
VirtualBox Guest Additions: Running kernel modules will not be replaced until the system is restarted
VirtualBox Guest Additions: Starting.

4. Reboot the VM

# reboot

5. Check and Confirm Virtualbox guest additions are properly installed and running

# lsmod | grep vbox
…

6. Enable Copy / Paste from to Virttual Machine e.g. Shared Clipboard / Shared Folder etc.

The three options most useful besides the support for FullScreen OS emulation by Virtualbox to enable right after
guesttools is on are:

1. Devices -> Shared Clipboard -> Bidirectional
2. Devices -> Drag and Drop -> Bidirectional
3. Devices -> Shared Folders -> Shared Folder Settings

Tags: additions, CentOS, copy paste, enable, fullscreen, installed, Installing, log, mount, org, Reboot, Redhat, resolution, sudo, support, vm
Posted in Linux, Various, Virtual Machines | No Comments »

How to yum Install Gnome GUI, Latest Guest Addition Tools, Google Chrome latest version and rdesktop / xfreerdp / remmina remote RDP VNC clients On CentOS 7 / 8

Thursday, July 29th, 2021

I've just reinstalled my CentOS 7 Virtual Machine since after I tried to migrate a .vdi Virtual Box image to the new company laptop using a copy of Virtualbox VM via Microsoft OneDrive was a failure.
Thus I have rebuild all my CentOS Linux programs preinstalled on the old Virtual Machines from scratch, I use this virtual machine for a very simple tasks, so basicly most imporant tools I use is a plain SSH and VNC and Remote Desktop clients just to be able to remotely connect to remote Home based server.

1.Install GNOME Graphical Environment from command line on CentOS 7 with yum and configure it to start GUI on next OS boot

I've used a minimal CentOS installation – ISO CentOS-7-x86_64-DVD-1908.iso and this brings up the OS with a text mode only as usually CentOS is used to roll on Servers and rarely and many times admins did not want to have GUI at all, however my case is different since I do like to use Graphical Environment as I use my CentOS for all kind of testing that can be later applied to a Production machines that doesn't have a GUI, hence to install GNOME on CentOS run below cmds:

[root@centos ~ ]# yum group list
Loaded plugins: fastestmirror
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Loading mirror speeds from cached hostfile
Available Environment Groups:
Minimal Install
Compute Node
Infrastructure Server
File and Print Server
Basic Web Server
Virtualization Host
Server with GUI
GNOME Desktop
KDE Plasma Workspaces
Development and Creative Workstation
Available Groups:
Compatibility Libraries
Console Internet Tools
Development Tools
Graphical Administration Tools
Legacy UNIX Compatibility
Scientific Support
Security Tools
Smart Card Support
System Administration Tools
System Management
Done

[root@centos ~ ]# yum groupinstall "GNOME Desktop" "Graphical Administration Tools" -y

Enable GUI to be automatically start on CentOS VM boot in systemd this is configured with the "targets" instead of the well known classical runlevels (the well known /etc/inittab) is now obsolete in newer Linux distros.

[root@centos ~ ]# ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target

2. Install Guest Additions Tools on CentOS

The most basic thing to do once I've had the CentOS Linux release 7.7.1908 (Core) rolled out on the VirtualBox is of course to enable Guest Additions Tools

First I had to install of course Guest Additions Tools to allow myself to have a copy paste in clip board via the Host Machine (Windows 10) and the Guest Machine.
To do I had to:

[root@centos ~ ]# yum install kernel-headers.x86_64 -y

[root@centos ~ ]# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

[root@centos ~ ]# yum install perl gcc dkms kernel-devel kernel-headers make bzip2

To check the required VBoxLinuxAdditions.run script kernel headers are at place:

[root@centos ~ ]# ls -l /usr/src/kernels/$(uname -r)

…
You should get a list of kernel header files

Then once I've done the Insert Guest Additions CD Image from the VirtualBox VM upper menu. I've had to mount and load the guest additions via the script:

[root@centos ~ ]# mkdir /mnt/cdrom
[root@centos ~ ]# mount /dev/cdrom /mnt/cdrom
[root@centos ~ ]# sh VBoxLinuxAdditions.run

After rebooting the Virtual Machine, I've used the full screen functionality to test and configured immediately Shared Clipboard and Drag and Drop to be both set to (Bidirectional) as well as configured a Shared folder to provide my Windows Desktop under /mnt/shared_folder (as read write) as I usually do to be able to easily copy files from the VM and to the Windows.

3. Install Google Chrome on the CentOS Virtual Machine with yum

Next I've installed the chrome browser that was pretty trivial it is up to fetching the required 32 or 64 bit latest chrome binary this is usually on URL:

[root@centos ~ ]# wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

and installing Google Chrome with superuser with command:

[root@centos ~ ]# yum install ./google-chrome-stable_current_*.rpm -y

Loaded plugins: fastestmirror, langpacks
Examining ./google-chrome-stable_current_x86_64.rpm: google-chrome-stable-92.0.4515.107-1.x86_64
Marking ./google-chrome-stable_current_x86_64.rpm to be installed
Resolving Dependencies
–> Running transaction check
—> Package google-chrome-stable.x86_64 0:92.0.4515.107-1 will be installed
–> Processing Dependency: liberation-fonts for package: google-chrome-stable-92.0.4515.107-1.x86_64
Loading mirror speeds from cached hostfile
* base: mirror.digitalnova.at
* epel: fedora.ipacct.com
* extras: mirror.digitalnova.at
* updates: mirror.digitalnova.at
–> Processing Dependency: libvulkan.so.1()(64bit) for package: google-chrome-stable-92.0.4515.107-1.x86_64
–> Running transaction check
—> Package liberation-fonts.noarch 1:1.07.2-16.el7 will be installed
–> Processing Dependency: liberation-narrow-fonts = 1:1.07.2-16.el7 for package: 1:liberation-fonts-1.07.2-16.el7.noarch
—> Package vulkan.x86_64 0:1.1.97.0-1.el7 will be installed
–> Processing Dependency: vulkan-filesystem = 1.1.97.0-1.el7 for package: vulkan-1.1.97.0-1.el7.x86_64
–> Running transaction check
—> Package liberation-narrow-fonts.noarch 1:1.07.2-16.el7 will be installed
—> Package vulkan-filesystem.noarch 0:1.1.97.0-1.el7 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
google-chrome-stable x86_64 92.0.4515.107-1 /google-chrome-stable_current_x86_64
259 M
Installing for dependencies:
liberation-fonts noarch 1:1.07.2-16.el7 base 13 k
liberation-narrow-fonts noarch 1:1.07.2-16.el7 base 202 k
vulkan x86_64 1.1.97.0-1.el7 base 3.6 M
vulkan-filesystem noarch 1.1.97.0-1.el7 base 6.3 k

Transaction Summary
================================================================================
Install 1 Package (+4 Dependent packages)

Total size: 263 M
Total download size: 3.8 M
Installed size: 281 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): liberation-fonts-1.07.2-16.el7.noarch.rpm | 13 kB 00:00
(2/4): liberation-narrow-fonts-1.07.2-16.el7.noarch.rpm | 202 kB 00:00
(3/4): vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm | 6.3 kB 00:00
(4/4): vulkan-1.1.97.0-1.el7.x86_64.rpm | 3.6 MB 00:00
——————————————————————————–
Total 3.0 MB/s | 3.8 MB 00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : vulkan-filesystem-1.1.97.0-1.el7.noarch 1/5
Installing : vulkan-1.1.97.0-1.el7.x86_64 2/5
Installing : 1:liberation-narrow-fonts-1.07.2-16.el7.noarch 3/5
Installing : 1:liberation-fonts-1.07.2-16.el7.noarch 4/5
Installing : google-chrome-stable-92.0.4515.107-1.x86_64 5/5
Verifying : vulkan-1.1.97.0-1.el7.x86_64 1/5
Verifying : 1:liberation-narrow-fonts-1.07.2-16.el7.noarch 2/5
Verifying : 1:liberation-fonts-1.07.2-16.el7.noarch 3/5
Verifying : google-chrome-stable-92.0.4515.107-1.x86_64 4/5
Verifying : vulkan-filesystem-1.1.97.0-1.el7.noarch 5/5

Installed:
google-chrome-stable.x86_64 0:92.0.4515.107-1

Dependency Installed:
liberation-fonts.noarch 1:1.07.2-16.el7
liberation-narrow-fonts.noarch 1:1.07.2-16.el7
vulkan.x86_64 0:1.1.97.0-1.el7
vulkan-filesystem.noarch 0:1.1.97.0-1.el7
…

4. Install usable Windows VNC and remote desktop (RDP Client) for CentOS Linux

There is a plenty of clients to choice from if you need to have an RDP client for Linux, but perhaps the most useful ones I usually use are remmina / rdesktop and freerdp. Usually I use remmina on Debian Linux, but under the VM somehow I was not able to make remmina work in Full Screen mode while connected to remote Windows 7 VPS server, thus I've first tried xfreerdp (that comes from default CentOS repositories) and is open source alternative to rdesktop (which is non free distributed binary).

[root@centos ~ ]$ sudo yum -y install freerdp

The basic use is:

[hipo@centos ~ ]$ xfreerdp –toggle-fullscreen <remote-server-address>

Unfortunately I did not succeeded to make xfreerdp be able to show me remote desktop in FullScreen mode so had to use additional repository package called nux-dextop to have rdesktop at my disposal.

To install it had to run:

[root@centos ~ ]# rpm –import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
[root@centos ~ ]# rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
[root@centos ~ ]# yum install rdesktop

To connect to the remote RDP host in Fullscreen with rdesktop :

rdesktop -f <remote-server-address>

windows-7-remote-desktop-screenshot-connected-with-rdesktop

As telnet is not installed by default and it is so useful to check ports

5. Install GNU Image Manipulation Program for better screnshotting and Graphic edits

I usually do install GIMP (GNU Image Manipulation Program) since this is my favourite tool to make screenshot on Linux as well as do some minor graphic edits whenever necessery. I warmly recommend gimp to anyone. If you don't have basic GIMP tool and you plan to be daily working a lot with Linux sooner or later some skills with the program will be of a major use even for the most advanced sysadmin :)_

root@centos ~ ]# yum install -y gimp

6. Install useful administration tools for daily sysadmin work – telnet, nmap, iftop, htop, iotop, iptraf-ng, tcpdump

Having basic analys tools and remote communication port testing, DNS, resolving and connection, cpu, mem statistics I find mostly useful.

[root@centos .ssh]# yum install telnet nmap iftop htop vnstat sysstat iptraf-ng bind-utils -y
…

7. Set Open Explorer and SHOW Desktop key binding shortcuts for GNOME (to make daily work easier)

Another useful I do use in my newly installed Virtual Machines is the key combination of Windows (button key) + E – to easily open the GNOME equivalent of Windows Explorer (Nautilus) and Windows (key) + D to hide the active selected Window and Show Desktop. This is configured pretty easy in GNOME through:

$ gnome-control-center

Keyboard (Section)

Perhaps there is other stuff I need to add on the freshly installed Operating System if I remember something else interesting

configure-home-folder-and-hide-all-normal-windows-gnome-key-binding-howto-screenshot

8. Install gnome-tweaks to tweak a bit the desktop icon positionsing and additional gnome-shell extras

[root@centos hipo]# yum install -y gnome-shell-extension-workspace-indicator.noarch gnome-shell-extension-workspace-indicator.noarch gnome-shell-extension-suspend-button.noarch gnome-shell-extension-refresh-wifi.noarch gnome-shell-extension-updates-dialog.noarch gnome-shell-extension-windowoverlay-icons.noarch gnome-shell-extension-places-menu.noarch gnome-shell-extension-drive-menu.noarch gnome-shell-extension-apps-menu.noarch gnome-shell-extension-auto-move-windows.noarch gnome-tweaks gnome-shell-extension-systemMonitor.noarch gnome-shell-extension-openweather.noarch gnome-shell-extension-user-theme.noarch gnome-shell-extension-topicons-plus.noarch

Next step is to use gnome-tweaks to set multiple custom preference stuff you like on the gnome 3.28 GUI

gnome-tweak-tool1

gnome-tweak-tool2

gnome-tweak-tool3

9. Change ( Fix) timezone to correct time on the Virtual Machine

[root@localhost ~]# timedatectl
Local time: Fri 2021-07-30 12:20:51 CEST
Universal time: Fri 2021-07-30 10:20:51 UTC
RTC time: Fri 2021-07-30 10:20:48
Time zone: Europe/Berlin (CEST, +0200)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: yes
Last DST change: DST began at
Sun 2021-03-28 01:59:59 CET
Sun 2021-03-28 03:00:00 CEST
Next DST change: DST ends (the clock jumps one hour backwards) at
Sun 2021-10-31 02:59:59 CEST
Sun 2021-10-31 02:00:00 CET

[root@localhost ~]# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 35 Jul 29 14:03 /etc/localtime -> ../usr/share/zoneinfo/Europe/Berlin

To change to correct timezone, you need to find out the long name for the timezone you want to use. The timezone naming convention usually uses “Region/City” format.

To list all available time zones, you can either list the files in the /usr/share/zoneinfo directory or use the timedatectl command.

[root@centos ~]# timedatectl list-timezones|tail -n 10
Pacific/Pohnpei
Pacific/Port_Moresby
Pacific/Rarotonga
Pacific/Saipan
Pacific/Tahiti
Pacific/Tarawa
Pacific/Tongatapu
Pacific/Wake
Pacific/Wallis
UTC

As I'm situated in Sofia Bulgaria to set the correct timezone to UTC (Universal Time Clock) + 2 Hrs, i've checked the correct Continent/Country like so:

[root@centos ~]# timedatectl list-timezones|grep -i Sofia
Europe/Sofia

Once I've my Capital / Country time location identified to set to it:

[root@centos ~]# timedatectl set-timezone your_time_zone

10. Configure remote connection hostname SSH aliases via ssh config ( ~/.ssh/config)

I'm having separate Virtual Machines running on my OpenXen virtualization Hypervisor server at different ports which I remember by heart under different hostnames, this saves me time to always type on command line long commands such as:

# ssh long-hostname -p Port_number

to make accessibility to remote machines via a simple Hostname Aliases, that forwards to remote port (that gets forwarded via a Local Network configure Netwrork Address Translation), I use the .ssh/config nice Host / Hostname / User / Port directives like below samples:

[hipo@centos .ssh]$ cat config
Host pcfreak
User root
Port 2248
HostName 83.228.93.76

Host freak
User root
Port 2249
HostName 213.91.190.233

Host pcfrxenweb
User root
Port 2251
Hostname 83.228.93.76

Host pcfrxen
User root
Port 2250
Hostname 213.91.190.233
…

Now to connect to pcfrxen for example I simply type:

ssh pcfrxen

type in the password to remote VM and I'm in 🙂

The same could be achieved also with Adding Custom Hostname IP Aliases via ~/.bashrc or iteration script as I've explained earlier that fakes like custom /etc/hosts, but I usuaully prefer to use .ssh/config instead like explained above.

Note that above steps should work also on RHEL / Fedora Linux with a minor modifications, as usually this two distros share the RPM package manager. If someone tries to follow the guide and have success on any of this distros please drop a comment with feedback.

Tags: Available Environment Groups, Available Groups, cmds, Dependency Installed, fedora linux, guide, How to, Package Arch Version Repository Size, rhel, superuser, url
Posted in Linux, System Administration, Virtual Machines | No Comments »

Linux: How to set KVM Virtual Machine to autostart on system boot

Thursday, July 15th, 2021

linux-kvm-set-vm-to-automatically-boot-on-linux-system-boot

Recently I've written a short article on how to make auto boot OpenVZ container Virtual Machine if system gets rebooted as I had the task to do so in daily job as sysadmin.
Few days ago after some power outage in one of Frankfurt based Data Centers, some Rack tech equipment has been temporary powered of and once the electricity was up again, some Linux servers running Kernel Based Virtualization ( KVM ) some of the virtual machines did not managed to automatically start up and we had to manually start them one by one.
To manually start each of the machines that did not start up had to do the trivial:

[root@hypervisor ~]# virsh list
Id Name State
—————————————————-
3 VM500 running

[root@hypervisor ~]# virsh dominfo VM500
Id: 3
Name: VM500
UUID: 82075903-a4ce-5ec0-a3d8-9ee83d85fc75
OS Type: hvm
State: running
CPU(s): 2
CPU time: 247407.9s
Max memory: 2097152 KiB
Used memory: 2097152 KiB
Persistent: yes
Autostart: disable
Managed save: no
Security model: selinux
Security DOI: 0
Security label: system_u:system_r:svirt_t:s0:c447,c723 (permissive)

[root@hypervisor ~]# virsh start VM_domain_Name

Of course logcally to prevent future issues in case if Linux server gets suddenly rebooted due to whatever reason, we had to configure the machines in questions to automatically boot on OS system boot.

In some rare cases if above start command does not help you might want to check what is the status of libvirtd and investigate the logs in /var/log/libvirt/ i.e. /var/log/libvirt/libvirt.log

An alternative but more complicated way to set virtual machine to automatically start on boot is by using default location for automatic VM start just like OpenXEn has its /etc/xen/auto/ dirtory from where each soft symlinked VM configuration from /etc/xen/VM_name.cfg is started in KVM Hypervisor hosts to auto boot a certain vm you have to link /etc/libvirt/qemu/VM-to-autoboot-name.xml to /etc/libvirt/qemu/autostart/VM-to-autoboot-name.xml.

[root@hypervisor ~]# systemctl status libvirtd
● libvirtd.service – Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-06-29 10:21:30 CEST; 2 weeks 2 days ago
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 1809 (libvirtd)
Tasks: 22 (limit: 32768)
CGroup: /system.slice/libvirtd.service
├─1809 /usr/sbin/libvirtd
├─2335 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
├─2336 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
├─2386 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…
└─2387 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…

If for some reason libvirtd is not running or disabled state you will have to enable it with:

[root@hypervisor ~]# systemctl enable libvirtd

If the virtualization is running on some RPM based distribtuion OpenSuse whatever and libvirtd is controlled via chkconfig (redhat runlevel alternative system), you will have to :

[root@hypervisor ~]# chkconfig libvirtd on

Disabling KVM Virtual Machine from auto start on server boot

[root@hypervisor ~]# virsh autostart Debian10 –disable
Domain Debian10 unmarked as autostarted

Tags: default location, libvirtd.service, logs, running, sbin, status, system boot, usr, virsh, vm
Posted in Linux, System Administration, Virtual Machines | No Comments »

OpenVZ enable or disable auto start on Linux Hypervisor host boot for Virtual Machine containers

Wednesday, July 7th, 2021

To make OpenVZ / Virtuozzo Hypervisor servers and you are not sure whether your configured container virtual machines are configured to automatically boot on Linux Physical OS host boot in case of restart after patch update set or after unexpected shutdown due to Kernel / OS bug a hang or due to some electricity Power outage.

To check what is your current configuration for Virtual Environment on CentOS Linux you need to check inside /etc/sysconfig/vz-scripts/VEID.conf
You need to check the value for inside the file

ONBOOT=""

To get the exact ID of "VEID.conf of the current openvz guest VM containers exec:

[root@openvz vz-scripts]# vzlist -a
CTID NPROC STATUS IP_ADDR HOSTNAME
300 23 running 10.10.10.1 VirtualMachine1
301 25 running 10.10.10.2 VirtualMachine2

[root@openvz ~]# cd /etc/sysconfig/vz-scripts
[root@gbapp2 vz-scripts]# pwd
/etc/sysconfig/vz-scripts

[root@openvz vz-scripts]# grep -i ONBOOT 300.conf 301.conf
300.conf:ONBOOT="yes"
301.conf:ONBOOT="yes"

If you happen to have configured ONBOOT="no" you will need to the change to respective VEID.conf:

vi /etc/sysconfig/vz-scripts/VEID.conf

search for

ONBOOT=”no”

and change to

ONBOOT=”yes”

OpenVZ_virtuozzo-standard-process-tree-landscape

OpenVZ server process tree. The colors of the virtual severs are indicated by colors.

OpenVZ Quick cheat sheet commands

This change will auto-start the VPS container next time the host Hypervisor node is rebooted.
If you happen to have daily work with OpenVZ legacy systems like I do you might find also useful the following OpenVZ Cheatsheet pdf document.

A miniature quick cheatsheet for OpenVZ Virtualion, in case if you are like me and you have to use various virtualization technologies and tend to forget is as below:

vzlist # List running instances
vzlist -a # List all instances

vzctl stop <instance>
vzctl start <instance>
vzctl status <instance>

vzctl exec <instance> <command> # Run a command

vzctl enter <instance> # Get console

vzyum <instance> install <package> # Install a package

# Change properties
vzctl set <instance> –hostname <hostname> –save
vzctl set <instance> –ipadd <IP> –save
vzctl set <instance> –userpasswd root:<password> –save

If need to get more insight on how OpenVZ Virtualization does work on a low level and stretch out its possibilities, an old but useful document you might want to check is OpenVZ-Users-Guide PDF.

If you need it to hava e copy of it openvz_cheat_sheet.txt.

Tags: auto start, check, configured, disable, Linux Hypervisor, need, ONBOOT, root password, sysconfig, virtual machines
Posted in Linux, System Administration, Virtual Machines | No Comments »

How to automate open xen Hypervisor Virtual Machines backups shell script

Tuesday, June 22nd, 2021

As a sysadmin that have my own Open Xen Debian Hypervisor running on a Lenovo ThinkServer few months ago due to a human error I managed to mess up one of my virtual machines and rebuild the Operating System from scratch and restore files service and MySQl data from backup that really pissed me of and this brought the need for having a decent Virtual Machine OpenXen backup solution I can implement on the Debian ( Buster) 10.10 running the free community Open Xen version 4.11.4+107-gef32c7afa2-1. The Hypervisor is a relative small one holding just 7 VM s:

HypervisorHost:~# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 11102 24 r—– 214176.4
pcfrxenweb 11 12288 4 -b—- 247425.5
pcfrxen 12 16384 10 -b—- 1371621.4
windows7 20 4096 2 -b—- 97887.2
haproxy2 21 4096 2 -b—- 11806.9
jitsi-meet 22 2048 2 -b—- 12843.9
zabbix 23 2048 2 -b—- 20275.1
centos7 24 2040 2 -b—- 10898.2

HypervisorHost:~# xl list|grep -v 'Name ' |grep -v 'Domain-0' |wc -l
7

The backup strategy of the script is very simple to shutdown the running VM machine, make a copy with rsync to a backup location the image of each of the Virtual Machines in a bash shell loop for each virtual machine shown in output of xl command and backup to a preset local directory in my case this is /backups/ the backup of each virtual machine is produced within a separate backup directory with a respective timestamp. Backup VM .img files are produced in my case to mounted 2x external attached hard drives each of which is a 4 Terabyte Seagate Plus Backup (Storage). The original version of the script was made to be a slightly different by Zhiqiang Ma whose script I used for a template to come up with my xen VM backup solution. To prevent the Hypervisor's load the script is made to do it with a nice of (nice -n 10) this might be not required or you might want to modify it to better suit your needs. Below is the script itself you can fetch a copy of it /usr/sbin/xen_vm_backups.sh :

#!/bin/bash

# Author: Zhiqiang Ma (http://www.ericzma.com/)
# Modified to work with xl and OpenXen by Georgi Georgiev – https://pc-freak.net
# Original creation dateDec. 27, 2010
# Script takes all defined vms under xen_name_list and prepares backup of each
# after shutting down the machine prepares archive and copies archive in externally attached mounted /backup/disk1 HDD
# Latest update: 08.06.2021 G. Georgiev – hipo@pc-freak.net

mark_file=/backups/disk1/tmp/xen-bak-marker
log_file=/var/log/xen/backups/bak-$(date +%Y_%m_%d).log
err_log_file=/var/log/xen/backups/bak_err-$(date +%H_%M_%Y_%m_%d).log
xen_dir=/xen/domains
xen_vmconfig_dir=/etc/xen/
local_bak_dir=/backups/disk1/tmp
#bak_dir=xenbak@backup_host1:/lhome/xenbak
bak_dir=/backups/disk1/xen-backups/xen_images/$(date +%Y_%m_%d)/xen/domains
#xen_name_list="haproxy2 pcfrxenweb jitsi-meet zabbix windows7 centos7 pcfrxenweb pcfrxen"
xen_name_list="windows7 haproxy2 jitsi-meet zabbix centos7"

if [ ! -d /var/log/xen/backups ]; then
echo mkdir -p /var/log/xen/backups
mkdir -p /var/log/xen/backups
fi

if [ ! -d $bak_dir ]; then
echo mkdir -p $bak_dir
mkdir -p $bak_dir

fi

# check whether bak runned last week
if [ -e $mark_file ] ; then
echo rm -f $mark_file
rm -f $mark_file
else
echo touch $mark_file
touch $mark_file
# exit 0
fi

# set std and stderr to log file
echo mv $log_file $log_file.old
mv $log_file $log_file.old
echo mv $err_log_file $err_log_file.old
mv $err_log_file $err_log_file.old
echo "exec 2> $err_log_file"
exec 2> $err_log_file
echo "exec > $log_file"
exec > $log_file

# check whether the VM is running
# We only backup running VMs

echo "*** Check alive VMs"

xen_name_list_tmp=""

for i in $xen_name_list
do
echo "/usr/sbin/xl list > /tmp/tmp-xen-list"
/usr/sbin/xl list > /tmp/tmp-xen-list
grepinlist=`grep $i" " /tmp/tmp-xen-list`;
if [[ “$grepinlist” == “” ]]
then
echo $i is not alive.
else
echo $i is alive.
xen_name_list_tmp=$xen_name_list_tmp" "$i
fi
done

xen_name_list=$xen_name_list_tmp

echo "Alive VM list:"

for i in $xen_name_list
do
echo $i
done

echo "End alive VM list."

###############################
date
echo "*** Backup starts"

###############################
date
echo "*** Copy VMs to local disk"

for i in $xen_name_list
do
date
echo "Shutdown $i"
echo /usr/sbin/xl shutdown $i
/usr/sbin/xl shutdown $i
if [ $? != ‘0’ ]; then
echo 'Not Xen Disk image destroying …';
/usr/sbin/xl destroy $i
fi
sleep 30

echo "Copy $i"
echo "Copy to local_bak_dir: $local_bak_dir"
echo /usr/bin/rsync -avhW –no-compress –progress $xen_dir/$i/{disk.img,swap.img} $local_bak_dir/$i/
time /usr/bin/rsync -avhW –no-compress –progress $xen_dir/$i/{disk.img,swap.img} $local_bak_dir/$i/
echo /usr/bin/rsync -avhW –no-compress –progress $xen_vmconfig_dir/$i.cfg $local_bak_dir/$i.cfg
time /usr/bin/rsync -avhW –no-compress –progress $xen_vmconfig_dir/$i.cfg $local_bak_dir/$i.cfg
date
echo "Create $i"
# with vmmem=1024"
# /usr/sbin/xm create $xen_dir/vm.run vmid=$i vmmem=1024
echo /usr/sbin/xl create $xen_vmconfig_dir/$i.cfg
/usr/sbin/xl create $xen_vmconfig_dir/$i.cfg
## Uncomment if you need to copy with scp somewhere
### echo scp $log_file $bak_dir/xen-bak-111.log
### echo /usr/bin/rsync -avhW –no-compress –progress $log_file $local_bak_dir/xen-bak-111.log
done

####################
date
echo "*** Compress local bak vmdisks"

for i in $xen_name_list
do
date
echo "Compress $i"
echo tar -z -cfv $bak_dir/$i-$(date +%Y_%m_%d).tar.gz $local_bak_dir/$i-$(date +%Y_%m_%d) $local_bak_dir/$i.cfg
time nice -n 10 tar -z -cvf $bak_dir/$i-$(date +%Y_%m_%d).tar.gz $local_bak_dir/$i/ $local_bak_dir/$i.cfg
echo rm -vf $local_bak_dir/$i/ $local_bak_dir/$i.cfg
rm -vrf $local_bak_dir/$i/{disk.img,swap.img} $local_bak_dir/$i.cfg
done

####################
date
echo "*** Copy local bak vmdisks to remote machines"

copy_remote () {
for i in $xen_name_list
do
date
echo "Copy to remote: vm$i"
echo scp $local_bak_dir/vmdisk0-$i.tar.gz $bak_dir/vmdisk0-$i.tar.gz
done

#####################
date
echo "Backup finishes"
echo scp $log_file $bak_dir/bak-111.log

}

date
echo "Backup finished"

Things to configure before start using using the script to prepare backups for you is the xen_name_list variable

# directory skele where to store already prepared backups
bak_dir=/backups/disk1/xen-backups/xen_images/$(date +%Y_%m_%d)/xen/domains
# The configurations of the running Xen Virtual Machines
xen_vmconfig_dir=/etc/xen/
# a local directory that will be used for backup creation ( I prefer this directory to be on the backup storage location )
local_bak_dir=/backups/disk1/tmp
#bak_dir=xenbak@backup_host1:/lhome/xenbak
# the structure on the backup location where daily .img backups with be produced with rsync and tar archived with bzip2
bak_dir=/backups/disk1/xen-backups/xen_images/$(date +%Y_%m_%d)/xen/domains
# list here all the Virtual Machines you want the script to create backups of
xen_name_list="windows7 haproxy2 jitsi-meet zabbix centos7"

If you need the script to copy the backup of Virtual Machine images to external Backup server via Local Lan or to a remote Internet located encrypted connection with a passwordless ssh authentication (once you have prepared the Machines to automatically login without pass over ssh with specific user), you can uncomment the script commented section to adapt it to copy to remote host.

Once you have placed at place /usr/sbin/xen_vm_backups.sh use a cronjob to prepare backups on a regular basis, for example I use the following cron to produce a working copy of the Virtual Machine backups everyday.

# crontab -u root -l

# create windows7 haproxy2 jitsi-meet centos7 zabbix VMs backup once a month
00 06 1,2,3,4,5,6,7,8,9,10,11,12 * * /usr/sbin/xen_vm_backups.sh 2>&1 >/dev/null

I do clean up virtual machines Images that are older than 95 days with another cron job

# crontab -u root -l

# Delete xen image files older than 95 days to clear up space from backup HDD
45 06 17 * * find /backups/disk1/xen-backups/xen_images* -type f -mtime +95 -exec rm {} \; 2>&1 >/dev/null

#### Delete xen config backups older than 1 year+3 days (368 days)
45 06 17 * * find /backups/disk1/xen-backups/xen_config* -type f -mtime +368 -exec rm {} \; 2>&1 >/dev/null

# Delete xen image files older than 95 days to clear up space from backup HDD
45 06 17 * * find /backups/disk1/xen-backups/xen_images* -type f -mtime +95 -exec rm {} \; 2>&1 >/dev/null

#### Delete xen config backups older than 1 year+3 days (368 days)
45 06 17 * * find /backups/disk1/xen-backups/xen_config* -type f -mtime +368 -exec rm {} \; 2>&1 >/dev/null

Tags: after, Alive, ALL, amp, and, another, archive, are, authentication, automate, backups, bak, cfg, configure, echo, How to, hypervisor, Hypervisor Virtual Machines, machines, sbin, script, Shell, shell script, shutdown, Virtual, vm
Posted in Backups, Linux, Linux and FreeBSD Desktop, Linux Package Management, System Administration, Various, Virtual Machines | No Comments »

How to configure bond0 bonding and network bridging for KVM Virtual machines on Redhat / CentOS / Fedora Linux

Tuesday, February 16th, 2021

configure-bond0-bonding-channel-with-bridges-on-hypervisor-host-for-guest-KVM-virtual-machines-howto-sample-Hypervisor-Virtual-machines-pic
1. Intro to Redhat RPM based distro /etc/sysconfig/network-scripts/* config vars shortly explained

On RPM based Linux distributions configuring network has a very specific structure. As a sysadmin just recently I had a task to configure Networking on 2 Machines to be used as Hypervisors so the servers could communicate normally to other Networks via some different intelligent switches that are connected to each of the interfaces of the server. The idea is the 2 redhat 8.3 machines to be used as Hypervisor (HV) and each of the 2 HVs to each be hosting 2 Virtual guest Machines with preinstalled another set of Redhat 8.3 Ootpa. I've recently blogged on how to automate a bit installing the KVM Virtual machines with using predefined kickstart.cfg file.

The next step after install was setting up the network. Redhat has a very specific network configuration well known under /etc/sysconfig/network-scripts/ifcfg-eno*# or if you have configured the Redhats to fix the changing LAN card naming ens, eno, em1 to legacy eth0, eth1, eth2 on CentOS Linux – e.g. to be named as /etc/sysconfig/network-scripts/{ifcfg-eth0,1,2,3}.

The first step to configure the network from that point is to come up with some network infrastrcture that will be ready on the HV nodes server-node1 server-node2 for the Virtual Machines to be used by server-vm1, server-vm2.

Thus for the sake of myself and some others I decide to give here the most important recognized variables that can be placed inside each of the ifcfg-eth0,ifcfg-eth1,ifcfg-eth2 …

A standard ifcfg-eth0 confing would look something this:

[root@redhat1 :~ ]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME=eth0
UUID=…
ONBOOT=yes
HWADDR=0e:a4:1a:b6:fc:86
IPADDR0=10.31.24.10
PREFIX0=23
GATEWAY0=10.31.24.1
DNS1=192.168.50.3
DNS2=10.215.105.3
DOMAIN=example.com
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

Lets say few words to each of the variables to make it more clear to people who never configured Newtork on redhat without the help of some of the console ncurses graphical like tools such as nmtui or want to completely stop the Network-Manager to manage the network and thus cannot take the advantage of using nmcli (a command-line tool for controlling NetworkManager).

Here is a short description of each of above configuration parameters:

TYPE=device_type: The type of network interface device
BOOTPROTO=protocol: Where protocol is one of the following:

none: No boot-time protocol is used.
bootp: Use BOOTP (bootstrap protocol).
dhcp: Use DHCP (Dynamic Host Configuration Protocol).
static: if configuring static IP

EFROUTE|IPV6_DEFROUTE=answer:

yes: This interface is set as the default route for IPv4|IPv6 traffic.
no: This interface is not set as the default route.

Usually most people still don't use IPV6 so better to disable that

IPV6INIT=answer: Where answer is one of the following:

yes: Enable IPv6 on this interface. If IPV6INIT=yes, the following parameters could also be set in this file:

IPV6ADDR=IPv6 address

IPV6_DEFAULTGW=The default route through the specified gateway

no: Disable IPv6 on this interface.

IPV4_FAILURE_FATAL|IPV6_FAILURE_FATAL=answer: Where answer is one of the following:

yes: This interface is disabled if IPv4 or IPv6 configuration fails.
no: This interface is not disabled if configuration fails.

ONBOOT=answer: Where answer is one of the following:

yes: This interface is activated at boot time.
no: This interface is not activated at boot time.

HWADDR=MAC-address: The hardware address of the Ethernet device
IPADDRN=address: The IPv4 address assigned to the interface
PREFIXN=N: Length of the IPv4 netmask value
GATEWAYN=address: The IPv4 gateway address assigned to the interface. Because an interface can be associated with several combinations of IP address, network mask prefix length, and gateway address, these are numbered starting from 0.
DNSN=address: The address of the Domain Name Servers (DNS)
DOMAIN=DNS_search_domain: The DNS search domain (this is the search Domain-name.com you usually find in /etc/resolv.conf)

Other interesting file that affects how routing is handled on a Redhat Linux is

/etc/sysconfig/network

[root@redhat1 :~ ]# cat /etc/sysconfig/network
# Created by anaconda
GATEWAY=10.215.105.
Having this gateway defined does add a default gateway

This file specifies global network settings. For example, you can specify the default gateway, if you want to apply some network settings such as routings, Alias IPs etc, that will be valid for all configured and active configuration red by systemctl start network scripts or the (the network-manager if such is used), just place it in that file.

Other files of intesresting to control how resolving is being handled on the server worthy to check are

/etc/nsswitch.conf

and

/etc/hosts

If you want to set a preference of /etc/hosts being red before /etc/resolv.conf and DNS resolving for example you need to have inside it, below is default behavior of it.

root@redhat1 :~ ]# grep -i hosts /etc/nsswitch.conf
# hosts: files dns
# hosts: files dns # from user file
# Valid databases are: aliases, ethers, group, gshadow, hosts,
hosts: files dns myhostname

As you can see the default order is to read first files (meaning /etc/hosts) and then the dns (/etc/resolv.conf)
hosts: files dns
…

Now with this short intro description on basic values accepted by Redhat's /etc/sysconfig/network-scripts/ifcfg* prepared configurations.

I will give a practical example of configuring a bond0 interface with 2 members which were prepared based on Redhat's Official documentation found in above URLs:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-network-bonding_configuring-and-managing-networking

# Bonding on RHEL 7 documentation
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-network_bonding_using_the_command_line_interface

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-verifying_network_configuration_bonding_for_redundancy

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-networkscripts-interfaces_network-bridge

# Network Bridge with Bond documentation
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-Configuring_a_VLAN_over_a_Bond

https://docs.fedoraproject.org/en-US/Fedora/24/html/Networking_Guide/sec-Network_Bridge_with_Bond.html

2. Configuring a single bond connection on eth0 / eth2 and setting 3 bridge interfaces bond -> br0, br1 -> eth1, br2 -> eth2

The task on my machines was to set up from 4 lan cards one bonded interface as active-backup type of bond with bonded lines on eth0, eth2 and 3 other 2 eth1, eth2 which will be used for private communication network that is connected via a special dedicated Switches and Separate VLAN 50, 51 over a tagged dedicated gigabit ports.

As said the 2 Servers had each 4 Broadcom Network CARD interfaces each 2 of which are paired (into a single card) and 2 of which are a solid Broadcom NetXtreme Dual Port 10GbE SFP+ and Dell Broadcom 5720 Dual Port 1Gigabit Network.

2-ports-broadcom-netxtreme-dual-port-10GBe-spf-plus

On each of server-node1 and server-node2 we had 4 Ethernet Adapters properly detected on the Redhat

root@redhat1 :~ ]# lspci |grep -i net
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
19:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
19:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)

I've already configured as prerogative net.ifnames=0 to /etc/grub2/boot.cfg and Network-Manager service disabled on the host (hence to not use Network Manager you'll see in below configuration NM_CONTROLLED="no" is telling the Redhat servers is not to be trying NetworkManager for more on that check my previous article Disable NetworkManager automatic Ethernet Interface Management on Redhat Linux , CentOS 6 / 7 / 8.

3. Types of Network Bonding

mode=0 (balance-rr)

This mode is based on Round-robin policy and it is the default mode. This mode offers fault tolerance and load balancing features. It transmits the packets in Round robin fashion that is from the first available slave through the last.

mode-1 (active-backup)

This mode is based on Active-backup policy. Only one slave is active in this band, and another one will act only when the other fails. The MAC address of this bond is available only on the network adapter part to avoid confusing the switch. This mode also provides fault tolerance.

mode=2 (balance-xor)

This mode sets an XOR (exclusive or) mode that is the source MAC address is XOR’d with destination MAC address for providing load balancing and fault tolerance. Each destination MAC address the same slave is selected.

mode=3 (broadcast)

This method is based on broadcast policy that is it transmitted everything on all slave interfaces. It provides fault tolerance. This can be used only for specific purposes.

mode=4 (802.3ad)

This mode is known as a Dynamic Link Aggregation mode that has it created aggregation groups having same speed. It requires a switch that supports IEEE 802.3ad dynamic link. The slave selection for outgoing traffic is done based on a transmit hashing method. This may be changed from the XOR method via the xmit_hash_policy option.

mode=5 (balance-tlb)

This mode is called Adaptive transmit load balancing. The outgoing traffic is distributed based on the current load on each slave and the incoming traffic is received by the current slave. If the incoming traffic fails, the failed receiving slave is replaced by the MAC address of another slave. This mode does not require any special switch support.

mode=6 (balance-alb)

This mode is called adaptive load balancing. This mode does not require any special switch support.

Lets create the necessery configuration for the bond and bridges

[root@redhat1 :~ ]# cat ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
#IPADDR=10.50.21.16
#PREFIX=26
#GATEWAY=10.50.0.1
#DNS1=172.20.88.2
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=1 miimon=100 primary=eth0"
NM_CONTROLLED="no"
BRIDGE=br0

[root@redhat1 :~ ]# cat ifcfg-bond0.10
DEVICE=bond0.10
BOOTPROTO=none
ONPARENT=yes
#IPADDR=10.50.21.17
#NETMASK=255.255.255.0
VLAN=yes

[root@redhat1 :~ ]# cat ifcfg-br0
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
UUID=4451286d-e40c-4d8c-915f-7fc12a16d595
DEVICE=br0
ONBOOT=yes
IPADDR=10.50.50.16
PREFIX=26
GATEWAY=10.50.0.1
DNS1=172.20.0.2
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-br1
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br1
UUID=40360c3c-47f5-44ac-bbeb-77f203390d29
DEVICE=br1
ONBOOT=yes
##IPADDR=10.50.51.241
PREFIX=28
##GATEWAY=10.50.0.1
##DNS1=172.20.0.2
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-br2
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br2
UUID=fbd5c257-2f66-4f2b-9372-881b783276e0
DEVICE=br2
ONBOOT=yes
##IPADDR=10.50.51.243
PREFIX=28
##GATEWAY=10.50.0.1
##DNS1=172.20.10.1
NM_CONTROLLED=no
NM_CONTROLLED=no
BRIDGE=br0

[root@redhat1 :~ ]# cat ifcfg-eth0
TYPE=Ethernet
NAME=bond0-slaveeth0
BOOTPROTO=none
#UUID=61065574-2a9d-4f16-b16e-00f495e2ee2b
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-eth1
TYPE=Ethernet
NAME=eth1
UUID=b4c359ae-7a13-436b-a904-beafb4edee94
DEVICE=eth1
ONBOOT=yes
BRIDGE=br1
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-eth2
TYPE=Ethernet
NAME=bond0-slaveeth2
BOOTPROTO=none
#UUID=821d711d-47b9-490a-afe7-190811578ef7
DEVICE=eth2
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-eth3
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
#BOOTPROTO=dhcp
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
BRIDGE=br2
NAME=eth3
UUID=61065574-2a9d-4f16-b16e-00f495e2ee2b
DEVICE=eth3
ONBOOT=yes
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
#IPADDR=10.50.21.16
#PREFIX=26
#GATEWAY=10.50.21.1
#DNS1=172.20.88.2
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=1 miimon=100 primary=eth0"
NM_CONTROLLED="no"
BRIDGE=br0

# cat ifcfg-bond0.10
DEVICE=bond0.10
BOOTPROTO=none
ONPARENT=yes
#IPADDR=10.50.21.17
#NETMASK=255.255.255.0
VLAN=yes
NM_CONTROLLED=no
BRIDGE=br0

[root@redhat2 :~ ]# cat ifcfg-br0
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
#UUID=f87e55a8-0fb4-4197-8ccc-0d8a671f30d0
UUID=4451286d-e40c-4d8c-915f-7fc12a16d595
DEVICE=br0
ONBOOT=yes
IPADDR=10.50.21.17
PREFIX=26
GATEWAY=10.50.21.1
DNS1=172.20.88.2
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-br1
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=no
#IPV6_AUTOCONF=no
#IPV6_DEFROUTE=no
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br1
UUID=40360c3c-47f5-44ac-bbeb-77f203390d29
DEVICE=br1
ONBOOT=yes
##IPADDR=10.50.21.242
PREFIX=28
##GATEWAY=10.50.21.1
##DNS1=172.20.88.2
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-br2
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=no
#IPV6_AUTOCONF=no
#IPV6_DEFROUTE=no
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br2
UUID=fbd5c257-2f66-4f2b-9372-881b783276e0
DEVICE=br2
ONBOOT=yes
##IPADDR=10.50.21.244
PREFIX=28
##GATEWAY=10.50.21.1
##DNS1=172.20.88.2
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-eth0
TYPE=Ethernet
NAME=bond0-slaveeth0
BOOTPROTO=none
#UUID=ee950c07-7eb2-463b-be6e-f97e7ad9d476
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-eth1
TYPE=Ethernet
NAME=eth1
UUID=ffec8039-58f0-494a-b335-7a423207c7e6
DEVICE=eth1
ONBOOT=yes
BRIDGE=br1
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-eth2
TYPE=Ethernet
NAME=bond0-slaveeth2
BOOTPROTO=none
#UUID=2c097475-4bef-47c3-b241-f5e7f02b3395
DEVICE=eth2
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

Notice that the bond0 configuration does not have an IP assigned this is done on purpose as we're using the interface channel bonding together with attached bridge for the VM. Usual bonding on a normal physical hardware hosts where no virtualization use is planned is perhaps a better choice. If you however try to set up an IP address in that specific configuration shown here and you try to reboot the machine, you will end up with inacessible machine over the network like I did and you will need to resolve configuration via some kind of ILO / IDRAC interface.

4. Generating UUID for ethernet devices bridges and bonds

One thing to note is the command uuidgen you might need that to generate UID identificators to fit in the new network config files.

Example:

[root@redhat2 :~ ]#uuidgen br2
e7995e15-7f23-4ea2-80d6-411add78d703
[root@redhat2 :~ ]# uuidgen br1
05e0c339-5998-414b-b720-7adf91a90103
[root@redhat2 :~ ]# uuidgen br0
e6d7ff74-4c15-4d93-a150-ff01b7ced5fb

5. How to make KVM Virtual Machines see configured Network bridges (modify VM XML)

To make the Virtual machines installed see the bridges I had to

[root@redhat1 :~ ]#virsh edit VM_name1
[root@redhat1 :~ ]#virsh edit VM_name2

[root@redhat2 :~ ]#virsh edit VM_name1
[root@redhat2 :~ ]#virsh edit VM_name2

Find the interface network configuration and change it to something like:

<interface type='bridge'>
<mac address='22:53:00:56:5d:ac'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='22:53:00:2a:5f:01'/>
<source bridge='br1'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='22:34:00:4a:1b:6c'/>
<source bridge='br2'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</interface>

6. Testing the bond is up and works fine

# ip addr show bond0
The result is the following:

4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:cb:25:82 brd ff:ff:ff:ff:ff:ff

The bond should be visible in the normal network interfaces with ip address show or /sbin/ifconfig

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:ab:2a:fa
Slave queue ID: 0

According to the output eth0 is the active slave.

The active slaves device files (eth0 in this case) is found in virtual file system /sys/

# find /sys -name *eth0
/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/net/eth0
/sys/devices/virtual/net/bond0/lower_eth0
/sys/class/net/eth0

You can remove a bond member say eth0 by

cd to the pci* directory
Example: /sys/devices/pci000:00/000:00:15.0

# echo 1 > remove

At this point the eth0 device directory structure that was previously located under /sys/devices/pci000:00/000:00:15.0 is no longer there. It was removed and the device no longer exists as seen by the OS.

You can verify this is the case with a simple ifconfig which will no longer list the eth0 device.
You can also repeat the cat /proc/net/bonding/bond0 command from Step 1 to see that eth0 is no longer listed as active or available.
You can also see the change in the messages file. It might look something like this:

2021-02-12T14:13:23.363414-06:00 redhat1 device eth0: device has been deleted
2021-02-12T14:13:23.368745-06:00 redhat1 kernel: [81594.846099] bonding: bond0: releasing active interface eth0
2021-02-12T14:13:23.368763-06:00 redhat1 kernel: [81594.846105] bonding: bond0: Warning: the permanent HWaddr of eth0 – 00:0c:29:ab:2a:f0 – is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
2021-02-12T14:13:23.368765-06:00 redhat1 kernel: [81594.846132] bonding: bond0: making interface eth1 the new active one.

Another way to test the bonding is correctly switching between LAN cards on case of ethernet hardware failure is to bring down one of the 2 or more bonded interfaces, lets say you want to switch from active-backup from eth1 to eth2, do:

# ip link set dev eth0 down

That concludes the test for fail over on active slave failure.

7. Bringing bond updown (rescan) bond with no need for server reboot

You know bonding is a tedious stuff that sometimes breaks up badly so only way to fix the broken bond seems to be a init 6 (reboot) cmd but no actually that is not so.

You can also get the deleted device back with a simple pci rescan command:

# echo 1 > /sys/bus/pci/rescan

The eth0 interface should now be back
You can see that it is back with an ifconfig command, and you can verify that the bond sees it with this command:

# cat /proc/net/bonding/bond0

That concludes the test of the bond code seeing the device when it comes back again.

The same steps can be repeated only this time using the eth1 device and file structure to fail the active slave in the bond back over to eth0.

8. Testing the bond with ifenslave command (ifenslave command examples)

Below is a set of useful information to test the bonding works as expected with ifenslave command comes from "iputils-20071127" package

– To show information of all the inerfaces

# ifenslave -a
# ifenslave –all-interfaces

– To change the active slave

# ifenslave -c bond0 eth1
# ifenslave –change-active bond0 eth1

– To remove the slave interface from the bonding device

# ifenslave -d eth1
# ifenslave –detach bond0 eth1

– To show master interface info

# ifenslave bond0

– To set the bond device down and automatically release all the slaves

# ifenslave bond1 down

– To get the usage info

# ifenslave -u
# ifenslave –usage

– To set to verbose mode

# ifenslave -v
# ifenslave –verbose

9. Testing the bridge works fine

Historically over the years all kind of bridges are being handled with the brctl part of bridge-utils .deb / .rpm installable package.

The classical way to check a bridge is working is to do

# brctl show
# brctl show br0; brctl show br1; brctl show br2
# brctl showmacs br0

etc.

Unfortunately with redhat 8 this command is no longer available so to get information about configured bridges you need to use instead:

# bridge link show
3:eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
4:eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state listening priority 32 cost 100
…

10. Troubleshooting network connectivity issues on bond bridges and LAN cards

Testing the bond connection and bridges can route proper traffic sometimes is a real hassle so here comes at help the good old tcpdump

If you end up with issues with some of the ethernet interfaces between HV1 and HV2 to be unable to talk to each other and you have some suspiciousness that some colleague from the network team has messed up a copper (UTP) cable or there is a connectivity fiber optics issues. To check the VLAN tagged traffic headers on the switch you can listen to each and every bond0 and br0, br1, br2 eth0, eth1, eth2, eth3 configured on the server like so:

# tcpdump -i bond0 -nn -e vlan

Some further investigation on where does a normal ICMP traffic flows once everything is setup is a normal thing to do, hence just try to route a normal ping via the different server interfaces:

# ping -I bond0 DSTADDR
…

# ping -i eth0 DSTADDR
…

# ping -i eth1 DSTADDR
…

# ping -i eth2 DSTADDR

After conducting the ping do the normal for network testing big ICMP packages (64k) ping to make sure there are no packet losses etc., e.g:

# ping -I eth3 -s 64536 DSTADDR
…

If for 10 – 20 seconds the ping does not return package losses then you should be good.

Tags: boot time, BOOTPROTO, cat, configure, default gateway, DEFROUTE, Device, hosts, How to, IPADDRN, network interface, network scripts, none, ONBOOT, prefix, Redhat Linux, server, sysconfig, systemctl, UUID
Posted in Linux, Networking, System Administration, Virtual Machines | 2 Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Archive for the ‘Virtual Machines’ Category

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites

Archive for the ‘Virtual Machines’ Category

1. Get information about the VM you would like to migrate to VMDK

2. Convert the harddrive to VMDK

3. Upload the new harddrive to the ESXi Hypervisor and adapt it to ESX

4. Create a VM and add the converted harddrive to the machine.

0. Check situation before the update on HVs / get VM IDs etc.

1. Dump VM definition XMs (to have it in case if it gets wiped during update)

2. Set on standby virt-mach-centos1 (virt-mach-centos1)

3. Stop VM virt-mach-centos1 & backup on Hypervisor host (hypervisor-host1) for VM1

4. Remove package version locks on all hosts

5. Do yum update virt-mach-centos1

6. Check if everything is running fine after upgrade

7. Stop VM virt-mach-centos2 & backup on Hypervisor host (hypervisor-host2)

8. Do yum update on virt-mach-centos2

9. Check if everything is running fine after upgrade

10. Stop VM vm-host2 & backup

11. Do yum update vm-host2

12. Check if everything is running fine after upgrade

13. Do yum update hypervisor-host2

14. Stop VM vm-host1 & backup

15. Do yum update vm-host2

16. Check if everything is running fine after upgrade

17. Do yum update hypervisor-host1

18. Check if everything is running fine after upgrade

19. Check if everything is running fine after upgrade

20. Check once more VMs and haproxy or any other contained services in VMs run as expected

21. Haproxy Unstandby virt-mach-centos1

22. If necessery to redefine VMs (in case they disappear from virsh) or virtuosso is not working

23. Set versionlock to RPMs to prevent accident updates and check OS version release

Other useful hints

The question comes then how can be the KVM VNC listener on TCP port 5900 be completely disabled?

1. Disable KVM VNC Port listener via VIRSH VM XML edit

2. Disable VNC KVM TCP port 5900 to a dynamic running VM without a machine reboot

Tracking the Problem of MAC duplicates on Linux routers

1. Fixing Linux ARP common well known Problems through disabling arp_announce / arp_ignore / send_redirects kernel variables disablement

2. Digging further the IP conflict / dulicate MAC Problems

3. What is MAC Randomization and where and how it is configured across different main operating systems as of year 2022?

4. Disable MAC Wi-Fi Ethernet device Randomization on Android

5. Disabling MAC Randomization on MAC iOS, iPhone, iPad, iPod

6. How to completely filter dhcp traffic between two network router hosts IP 192.168.0.1 / 192.168.1.1 to stop 2 or more configured DHCP servers on separate networks see each other

7. Filter out DHCP traffic based on MAC completely on Linux with arptables

1. Install all necessery dependncies RPMs required by GuestAddition tools

2. Run below semanage and restorecon commands

3. Insert Virtualbox guest additions ISO and Run it

Devices -> Insert Guest Additions CD Image

4. Reboot the VM

5. Check and Confirm Virtualbox guest additions are properly installed and running

6. Enable Copy / Paste from to Virttual Machine e.g. Shared Clipboard / Shared Folder etc.

1.Install GNOME Graphical Environment from command line on CentOS 7 with yum and configure it to start GUI on next OS boot

2. Install Guest Additions Tools on CentOS

3. Install Google Chrome on the CentOS Virtual Machine with yum

4. Install usable Windows VNC and remote desktop (RDP Client) for CentOS Linux

5. Install GNU Image Manipulation Program for better screnshotting and Graphic edits

6. Install useful administration tools for daily sysadmin work – telnet, nmap, iftop, htop, iotop, iptraf-ng, tcpdump

7. Set Open Explorer and SHOW Desktop key binding shortcuts for GNOME (to make daily work easier)

8. Install gnome-tweaks to tweak a bit the desktop icon positionsing and additional gnome-shell extras

9. Change ( Fix) timezone to correct time on the Virtual Machine

10. Configure remote connection hostname SSH aliases via ssh config ( ~/.ssh/config)

Disabling KVM Virtual Machine from auto start on server boot

OpenVZ Quick cheat sheet commands

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Tags

Top Post Views

blogtopsites

6. How to completely filter dhcp traffic between two network router hosts
IP 192.168.0.1 / 192.168.1.1 to stop 2 or more configured DHCP servers
on separate networks see each other