Archive for the ‘Virtual Machines’ Category

Linux: How to set KVM Virtual Machine to autostart on system boot

Thursday, July 15th, 2021

linux-kvm-set-vm-to-automatically-boot-on-linux-system-boot

Recently I've written a short article on how to make auto boot OpenVZ container Virtual Machine if system gets rebooted as I had the task to do so in daily job as sysadmin.
Few days ago after some power outage in one of Frankfurt based Data Centers, some Rack tech equipment has been temporary powered of and once the electricity was up again, some Linux servers  running Kernel Based Virtualization ( KVM ) some of the virtual machines did not managed to automatically start up and we had to manually start them one by one.
To manually start each of the machines that did not start up had to do the trivial:

[root@hypervisor ~]# virsh list
 Id    Name                           State
—————————————————-
 3     VM500                   running

[root@hypervisor ~]# virsh dominfo VM500
Id:             3
Name:           VM500
UUID:           82075903-a4ce-5ec0-a3d8-9ee83d85fc75
OS Type:        hvm
State:          running
CPU(s):         2
CPU time:       247407.9s
Max memory:     2097152 KiB
Used memory:    2097152 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c447,c723 (permissive)

[root@hypervisor ~]# virsh start VM_domain_Name

Of course logcally to prevent future issues in case if Linux server gets suddenly rebooted due to whatever reason, we had to configure the machines in questions to automatically boot on OS system boot.

In some rare cases if above start command does not help you might want to check what is the status of libvirtd and investigate the logs in /var/log/libvirt/ i.e. /var/log/libvirt/libvirt.log

An alternative but more complicated way to set virtual machine to automatically start on boot is by using default location for automatic VM start just like OpenXEn has its /etc/xen/auto/ dirtory from where each soft symlinked VM configuration from /etc/xen/VM_name.cfg is started in KVM Hypervisor hosts to auto boot a certain vm you have to link /etc/libvirt/qemu/VM-to-autoboot-name.xml to /etc/libvirt/qemu/autostart/VM-to-autoboot-name.xml

[root@hypervisor ~]# systemctl status libvirtd
● libvirtd.service – Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-06-29 10:21:30 CEST; 2 weeks 2 days ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 1809 (libvirtd)
    Tasks: 22 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─1809 /usr/sbin/libvirtd
           ├─2335 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
           ├─2336 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
           ├─2386 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…
           └─2387 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…

If for some reason libvirtd is not running or disabled state you will have to enable it with:

[root@hypervisor ~]# systemctl enable libvirtd

If the virtualization is running on some RPM based distribtuion OpenSuse whatever and libvirtd is controlled via chkconfig (redhat runlevel alternative system), you will have to :

[root@hypervisor ~]# chkconfig libvirtd on

Disabling KVM Virtual Machine from auto start on server boot

[root@hypervisor ~]# virsh autostart Debian10 –disable
Domain Debian10 unmarked as autostarted

OpenVZ enable or disable auto start on Linux Hypervisor host boot for Virtual Machine containers

Wednesday, July 7th, 2021

howto-add-virtual-machine-to-auto-start-with-vz-openvz-linux-containers-4-logo-slogan-vertical-big

To make OpenVZ / Virtuozzo Hypervisor servers and you are not sure whether your configured container virtual machines are configured to automatically boot on Linux Physical OS host boot in case of restart after patch update set or after unexpected shutdown due to Kernel / OS bug a hang or due to some electricity Power outage.

To check what is your current configuration for Virtual Environment on CentOS Linux you need to check inside /etc/sysconfig/vz-scripts/VEID.conf
You need to check the value for inside the file

ONBOOT="" 

To get the exact ID of "VEID.conf of the current openvz guest VM containers exec:

[root@openvz vz-scripts]# vzlist -a
      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
       300         23 running   10.10.10.1     VirtualMachine1
       301         25 running   10.10.10.2     VirtualMachine2

[root@openvz ~]# cd /etc/sysconfig/vz-scripts
[root@gbapp2 vz-scripts]# pwd
/etc/sysconfig/vz-scripts

[root@openvz vz-scripts]# grep -i ONBOOT 300.conf 301.conf
300.conf:ONBOOT="yes"
301.conf:ONBOOT="yes"

If you happen to have configured ONBOOT="no" you will need to the change to respective VEID.conf:

vi /etc/sysconfig/vz-scripts/VEID.conf

search for

ONBOOT=”no”

and change to

ONBOOT=”yes”

OpenVZ_virtuozzo-standard-process-tree-landscape

OpenVZ server process tree. The colors of the virtual severs are indicated by colors.

OpenVZ Quick cheat sheet commands

This change will auto-start the VPS container next time the host Hypervisor node is rebooted.
If you happen to have daily work with OpenVZ legacy systems like I do you might find also useful the following OpenVZ Cheatsheet pdf document.

A miniature quick cheatsheet for OpenVZ Virtualion, in case if you are like me and you have to use various virtualization technologies and tend to forget is as below:

vzlist                               # List running instances
vzlist -a                            # List all instances

 

vzctl stop <instance>
vzctl start <instance>
vzctl status <instance>

vzctl exec <instance> <command>      # Run a command

vzctl enter <instance>               # Get console

vzyum <instance> install <package>   # Install a package


# Change properties
vzctl set <instance> –hostname <hostname> –save
vzctl set <instance> –ipadd <IP> –save
vzctl set <instance> –userpasswd root:<password> –save

If need to get more insight on how OpenVZ Virtualization does work on a low level and stretch out its possibilities, an old but useful document you might want to check is OpenVZ-Users-Guide PDF.


If you need it to hava e copy of it openvz_cheat_sheet.txt.

How to automate open xen Hypervisor Virtual Machines backups shell script

Tuesday, June 22nd, 2021

openxen-backup-logo As a sysadmin that have my own Open Xen Debian Hypervisor running on a Lenovo ThinkServer few months ago due to a human error I managed to mess up one of my virtual machines and rebuild the Operating System from scratch and restore files service and MySQl data from backup that really pissed me of and this brought the need for having a decent Virtual Machine OpenXen backup solution I can implement on the Debian ( Buster) 10.10 running the free community Open Xen version 4.11.4+107-gef32c7afa2-1. The Hypervisor is a relative small one holding just 7 VM s:

HypervisorHost:~#  xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 11102    24     r—–  214176.4
pcfrxenweb                                  11 12288     4     -b—-  247425.5
pcfrxen                                     12 16384    10     -b—-  1371621.4
windows7                                    20  4096     2     -b—-   97887.2
haproxy2                                    21  4096     2     -b—-   11806.9
jitsi-meet                                  22  2048     2     -b—-   12843.9
zabbix                                      23  2048     2     -b—-   20275.1
centos7                                     24  2040     2     -b—-   10898.2

HypervisorHost:~# xl list|grep -v 'Name ' |grep  -v 'Domain-0'  |wc -l
7


The backup strategy of the script is very simple to shutdown the running VM machine, make a copy with rsync to a backup location the image of each of the Virtual Machines in a bash shell loop for each virtual machine shown in output of xl command and backup to a preset local directory in my case this is /backups/ the backup of each virtual machine is produced within a separate backup directory with a respective timestamp. Backup VM .img files are produced in my case to mounted 2x external attached hard drives each of which is a 4 Terabyte Seagate Plus Backup (Storage). The original version of the script was made to be a slightly different by Zhiqiang Ma whose script I used for a template to come up with my xen VM backup solution. To prevent the Hypervisor's load the script is made to do it with a nice of (nice -n 10) this might be not required or you might want to modify it to better suit your needs. Below is the script itself you can fetch a copy of it /usr/sbin/xen_vm_backups.sh :

#!/bin/bash

# Author: Zhiqiang Ma (http://www.ericzma.com/)
# Modified to work with xl and OpenXen by Georgi Georgiev – https://pc-freak.net
# Original creation dateDec. 27, 2010
# Script takes all defined vms under xen_name_list and prepares backup of each
# after shutting down the machine prepares archive and copies archive in externally attached mounted /backup/disk1 HDD
# Latest update: 08.06.2021 G. Georgiev – hipo@pc-freak.net

mark_file=/backups/disk1/tmp/xen-bak-marker
log_file=/var/log/xen/backups/bak-$(date +%Y_%m_%d).log
err_log_file=/var/log/xen/backups/bak_err-$(date +%H_%M_%Y_%m_%d).log
xen_dir=/xen/domains
xen_vmconfig_dir=/etc/xen/
local_bak_dir=/backups/disk1/tmp
#bak_dir=xenbak@backup_host1:/lhome/xenbak
bak_dir=/backups/disk1/xen-backups/xen_images/$(date +%Y_%m_%d)/xen/domains
#xen_name_list="haproxy2 pcfrxenweb jitsi-meet zabbix windows7 centos7 pcfrxenweb pcfrxen"
xen_name_list="windows7 haproxy2 jitsi-meet zabbix centos7"

if [ ! -d /var/log/xen/backups ]; then
echo mkdir -p /var/log/xen/backups
 mkdir -p /var/log/xen/backups
fi

if [ ! -d $bak_dir ]; then
echo mkdir -p $bak_dir
 mkdir -p $bak_dir

fi


# check whether bak runned last week
if [ -e $mark_file ] ; then
        echo  rm -f $mark_file
 rm -f $mark_file
else
        echo  touch $mark_file
 touch $mark_file
  # exit 0
fi

# set std and stderr to log file
        echo mv $log_file $log_file.old
       mv $log_file $log_file.old
        echo mv $err_log_file $err_log_file.old
       mv $err_log_file $err_log_file.old
        echo "exec 2> $err_log_file"
       exec 2> $err_log_file
        echo "exec > $log_file"
       exec > $log_file


# check whether the VM is running
# We only backup running VMs

echo "*** Check alive VMs"

xen_name_list_tmp=""

for i in $xen_name_list
do
        echo "/usr/sbin/xl list > /tmp/tmp-xen-list"
        /usr/sbin/xl list > /tmp/tmp-xen-list
  grepinlist=`grep $i" " /tmp/tmp-xen-list`;
  if [[ “$grepinlist” == “” ]]
  then
    echo $i is not alive.
  else
    echo $i is alive.
    xen_name_list_tmp=$xen_name_list_tmp" "$i
  fi
done

xen_name_list=$xen_name_list_tmp

echo "Alive VM list:"

for i in $xen_name_list
do
   echo $i
done

echo "End alive VM list."

###############################
date
echo "*** Backup starts"

###############################
date
echo "*** Copy VMs to local disk"

for i in $xen_name_list
do
  date
  echo "Shutdown $i"
        echo  /usr/sbin/xl shutdown $i
        /usr/sbin/xl shutdown $i
        if [ $? != ‘0’ ]; then
                echo 'Not Xen Disk image destroying …';
                /usr/sbin/xl destroy $i
        fi
  sleep 30

  echo "Copy $i"
  echo "Copy to local_bak_dir: $local_bak_dir"
      echo /usr/bin/rsync -avhW –no-compress –progress $xen_dir/$i/{disk.img,swap.img} $local_bak_dir/$i/
     time /usr/bin/rsync -avhW –no-compress –progress $xen_dir/$i/{disk.img,swap.img} $local_bak_dir/$i/
      echo /usr/bin/rsync -avhW –no-compress –progress $xen_vmconfig_dir/$i.cfg $local_bak_dir/$i.cfg
     time /usr/bin/rsync -avhW –no-compress –progress $xen_vmconfig_dir/$i.cfg $local_bak_dir/$i.cfg
  date
  echo "Create $i"
  # with vmmem=1024"
  # /usr/sbin/xm create $xen_dir/vm.run vmid=$i vmmem=1024
          echo /usr/sbin/xl create $xen_vmconfig_dir/$i.cfg
          /usr/sbin/xl create $xen_vmconfig_dir/$i.cfg
## Uncomment if you need to copy with scp somewhere
###       echo scp $log_file $bak_dir/xen-bak-111.log
###      echo  /usr/bin/rsync -avhW –no-compress –progress $log_file $local_bak_dir/xen-bak-111.log
done

####################
date
echo "*** Compress local bak vmdisks"

for i in $xen_name_list
do
  date
  echo "Compress $i"
      echo tar -z -cfv $bak_dir/$i-$(date +%Y_%m_%d).tar.gz $local_bak_dir/$i-$(date +%Y_%m_%d) $local_bak_dir/$i.cfg
     time nice -n 10 tar -z -cvf $bak_dir/$i-$(date +%Y_%m_%d).tar.gz $local_bak_dir/$i/ $local_bak_dir/$i.cfg
    echo rm -vf $local_bak_dir/$i/ $local_bak_dir/$i.cfg
    rm -vrf $local_bak_dir/$i/{disk.img,swap.img}  $local_bak_dir/$i.cfg
done

####################
date
echo "*** Copy local bak vmdisks to remote machines"

copy_remote () {
for i in $xen_name_list
do
  date
  echo "Copy to remote: vm$i"
        echo  scp $local_bak_dir/vmdisk0-$i.tar.gz $bak_dir/vmdisk0-$i.tar.gz
done

#####################
date
echo "Backup finishes"
        echo scp $log_file $bak_dir/bak-111.log

}

date
echo "Backup finished"

 

Things to configure before start using using the script to prepare backups for you is the xen_name_list variable

#  directory skele where to store already prepared backups
bak_dir=/backups/disk1/xen-backups/xen_images/$(date +%Y_%m_%d)/xen/domains

# The configurations of the running Xen Virtual Machines
xen_vmconfig_dir=/etc/xen/
# a local directory that will be used for backup creation ( I prefer this directory to be on the backup storage location )
local_bak_dir=/backups/disk1/tmp
#bak_dir=xenbak@backup_host1:/lhome/xenbak
# the structure on the backup location where daily .img backups with be produced with rsync and tar archived with bzip2
bak_dir=/backups/disk1/xen-backups/xen_images/$(date +%Y_%m_%d)/xen/domains

# list here all the Virtual Machines you want the script to create backups of
xen_name_list="windows7 haproxy2 jitsi-meet zabbix centos7"

If you need the script to copy the backup of Virtual Machine images to external Backup server via Local Lan or to a remote Internet located encrypted connection with a passwordless ssh authentication (once you have prepared the Machines to automatically login without pass over ssh with specific user), you can uncomment the script commented section to adapt it to copy to remote host.

Once you have placed at place /usr/sbin/xen_vm_backups.sh use a cronjob to prepare backups on a regular basis, for example I use the following cron to produce a working copy of the Virtual Machine backups everyday.
 

# crontab -u root -l 

# create windows7 haproxy2 jitsi-meet centos7 zabbix VMs backup once a month
00 06 1,2,3,4,5,6,7,8,9,10,11,12 * * /usr/sbin/xen_vm_backups.sh 2>&1 >/dev/null


I do clean up virtual machines Images that are older than 95 days with another cron job

# crontab -u root -l

# Delete xen image files older than 95 days to clear up space from backup HDD
45 06 17 * * find /backups/disk1/xen-backups/xen_images* -type f -mtime +95 -exec rm {} \; 2>&1 >/dev/null

#### Delete xen config backups older than 1 year+3 days (368 days)
45 06 17 * * find /backups/disk1/xen-backups/xen_config* -type f -mtime +368 -exec rm {} \; 2>&1 >/dev/null

 

# Delete xen image files older than 95 days to clear up space from backup HDD
45 06 17 * * find /backups/disk1/xen-backups/xen_images* -type f -mtime +95 -exec rm {} \; 2>&1 >/dev/null

#### Delete xen config backups older than 1 year+3 days (368 days)
45 06 17 * * find /backups/disk1/xen-backups/xen_config* -type f -mtime +368 -exec rm {} \; 2>&1 >/dev/null

How to configure bond0 bonding and network bridging for KVM Virtual machines on Redhat / CentOS / Fedora Linux

Tuesday, February 16th, 2021

configure-bond0-bonding-channel-with-bridges-on-hypervisor-host-for-guest-KVM-virtual-machines-howto-sample-Hypervisor-Virtual-machines-pic
 1. Intro to Redhat RPM based distro /etc/sysconfig/network-scripts/* config vars shortly explained

On RPM based Linux distributions configuring network has a very specific structure. As a sysadmin just recently I had a task to configure Networking on 2 Machines to be used as Hypervisors so the servers could communicate normally to other Networks via some different intelligent switches that are connected to each of the interfaces of the server. The idea is the 2 redhat 8.3 machines to be used as  Hypervisor (HV) and each of the 2 HVs to each be hosting 2 Virtual guest Machines with preinstalled another set of Redhat 8.3 Ootpa. I've recently blogged on how to automate a bit installing the KVM Virtual machines with using predefined kickstart.cfg file.

The next step after install was setting up the network. Redhat has a very specific network configuration well known under /etc/sysconfig/network-scripts/ifcfg-eno*# or if you have configured the Redhats to fix the changing LAN card naming ens, eno, em1 to legacy eth0, eth1, eth2 on CentOS Linux – e.g. to be named as /etc/sysconfig/network-scripts/{ifcfg-eth0,1,2,3}.

The first step to configure the network from that point is to come up with some network infrastrcture that will be ready on the HV nodes server-node1 server-node2 for the Virtual Machines to be used by server-vm1, server-vm2.

Thus for the sake of myself and some others I decide to give here the most important recognized variables that can be placed inside each of the ifcfg-eth0,ifcfg-eth1,ifcfg-eth2 …

A standard ifcfg-eth0 confing would look something this:
 

[root@redhat1 :~ ]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME=eth0
UUID=…
ONBOOT=yes
HWADDR=0e:a4:1a:b6:fc:86
IPADDR0=10.31.24.10
PREFIX0=23
GATEWAY0=10.31.24.1
DNS1=192.168.50.3
DNS2=10.215.105.3
DOMAIN=example.com
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes


Lets say few words to each of the variables to make it more clear to people who never configured Newtork on redhat without the help of some of the console ncurses graphical like tools such as nmtui or want to completely stop the Network-Manager to manage the network and thus cannot take the advantage of using nmcli (a command-line tool for controlling NetworkManager).

Here is a short description of each of above configuration parameters:

TYPE=device_type: The type of network interface device
BOOTPROTO=protocol: Where protocol is one of the following:

  • none: No boot-time protocol is used.
  • bootp: Use BOOTP (bootstrap protocol).
  • dhcp: Use DHCP (Dynamic Host Configuration Protocol).
  • static: if configuring static IP

EFROUTE|IPV6_DEFROUTE=answer

  • yes: This interface is set as the default route for IPv4|IPv6 traffic.
  • no: This interface is not set as the default route.

Usually most people still don't use IPV6 so better to disable that

IPV6INIT=answer: Where answer is one of the following:

  • yes: Enable IPv6 on this interface. If IPV6INIT=yes, the following parameters could also be set in this file:

IPV6ADDR=IPv6 address

IPV6_DEFAULTGW=The default route through the specified gateway

  • no: Disable IPv6 on this interface.

IPV4_FAILURE_FATAL|IPV6_FAILURE_FATAL=answer: Where answer is one of the following:

  • yes: This interface is disabled if IPv4 or IPv6 configuration fails.
  • no: This interface is not disabled if configuration fails.

ONBOOT=answer: Where answer is one of the following:

  • yes: This interface is activated at boot time.
  • no: This interface is not activated at boot time.

HWADDR=MAC-address: The hardware address of the Ethernet device
IPADDRN=address: The IPv4 address assigned to the interface
PREFIXN=N: Length of the IPv4 netmask value
GATEWAYN=address: The IPv4 gateway address assigned to the interface. Because an interface can be associated with several combinations of IP address, network mask prefix length, and gateway address, these are numbered starting from 0.
DNSN=address: The address of the Domain Name Servers (DNS)
DOMAIN=DNS_search_domain: The DNS search domain (this is the search Domain-name.com you usually find in /etc/resolv.conf)

Other interesting file that affects how routing is handled on a Redhat Linux is

/etc/sysconfig/network

[root@redhat1 :~ ]# cat /etc/sysconfig/network
# Created by anaconda
GATEWAY=10.215.105.

Having this gateway defined does add a default gateway

This file specifies global network settings. For example, you can specify the default gateway, if you want to apply some network settings such as routings, Alias IPs etc, that will be valid for all configured and active configuration red by systemctl start network scripts or the (the network-manager if such is used), just place it in that file.

Other files of intesresting to control how resolving is being handled on the server worthy to check are 

/etc/nsswitch.conf

and

/etc/hosts

If you want to set a preference of /etc/hosts being red before /etc/resolv.conf and DNS resolving for example you need to have inside it, below is default behavior of it.
 

root@redhat1 :~ ]#   grep -i hosts /etc/nsswitch.conf
#     hosts: files dns
#     hosts: files dns  # from user file
# Valid databases are: aliases, ethers, group, gshadow, hosts,
hosts:      files dns myhostname

As you can see the default order is to read first files (meaning /etc/hosts) and then the dns (/etc/resolv.conf)
hosts: files dns

Now with this short intro description on basic values accepted by Redhat's /etc/sysconfig/network-scripts/ifcfg* prepared configurations.


I will give a practical example of configuring a bond0 interface with 2 members which were prepared based on Redhat's Official documentation found in above URLs:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-network-bonding_configuring-and-managing-networking
 

# Bonding on RHEL 7 documentation
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-network_bonding_using_the_command_line_interface

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-verifying_network_configuration_bonding_for_redundancy

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-networkscripts-interfaces_network-bridge

# Network Bridge with Bond documentation
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-Configuring_a_VLAN_over_a_Bond

https://docs.fedoraproject.org/en-US/Fedora/24/html/Networking_Guide/sec-Network_Bridge_with_Bond.html


2. Configuring a single bond connection on eth0 / eth2 and setting 3 bridge interfaces bond -> br0, br1 -> eth1, br2 -> eth2

The task on my machines was to set up from 4 lan cards one bonded interface as active-backup type of bond with bonded lines on eth0, eth2 and 3 other 2 eth1, eth2 which will be used for private communication network that is connected via a special dedicated Switches and Separate VLAN 50, 51 over a tagged dedicated gigabit ports.

As said the 2 Servers had each 4 Broadcom Network CARD interfaces each 2 of which are paired (into a single card) and 2 of which are a solid Broadcom NetXtreme Dual Port 10GbE SFP+ and Dell Broadcom 5720 Dual Port 1Gigabit Network​.

2-ports-broadcom-netxtreme-dual-port-10GBe-spf-plus

On each of server-node1 and server-node2 we had 4 Ethernet Adapters properly detected on the Redhat

root@redhat1 :~ ]# lspci |grep -i net
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
19:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
19:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)


I've already configured as prerogative net.ifnames=0 to /etc/grub2/boot.cfg and Network-Manager service disabled on the host (hence to not use Network Manager you'll see in below configuration NM_CONTROLLED="no" is telling the Redhat servers is not to be trying NetworkManager for more on that check my previous article Disable NetworkManager automatic Ethernet Interface Management on Redhat Linux , CentOS 6 / 7 / 8.

3. Types of Network Bonding

mode=0 (balance-rr)

This mode is based on Round-robin policy and it is the default mode. This mode offers fault tolerance and load balancing features. It transmits the packets in Round robin fashion that is from the first available slave through the last.

mode-1 (active-backup)

This mode is based on Active-backup policy. Only one slave is active in this band, and another one will act only when the other fails. The MAC address of this bond is available only on the network adapter part to avoid confusing the switch. This mode also provides fault tolerance.

mode=2 (balance-xor)

This mode sets an XOR (exclusive or) mode that is the source MAC address is XOR’d with destination MAC address for providing load balancing and fault tolerance. Each destination MAC address the same slave is selected.

mode=3 (broadcast)

This method is based on broadcast policy that is it transmitted everything on all slave interfaces. It provides fault tolerance. This can be used only for specific purposes.

mode=4 (802.3ad)

This mode is known as a Dynamic Link Aggregation mode that has it created aggregation groups having same speed. It requires a switch that supports IEEE 802.3ad dynamic link. The slave selection for outgoing traffic is done based on a transmit hashing method. This may be changed from the XOR method via the xmit_hash_policy option.

mode=5 (balance-tlb)

This mode is called Adaptive transmit load balancing. The outgoing traffic is distributed based on the current load on each slave and the incoming traffic is received by the current slave. If the incoming traffic fails, the failed receiving slave is replaced by the MAC address of another slave. This mode does not require any special switch support.

mode=6 (balance-alb)

This mode is called adaptive load balancing. This mode does not require any special switch support.

Lets create the necessery configuration for the bond and bridges

[root@redhat1 :~ ]# cat ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
#IPADDR=10.50.21.16
#PREFIX=26
#GATEWAY=10.50.0.1
#DNS1=172.20.88.2
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=1 miimon=100 primary=eth0"
NM_CONTROLLED="no"
BRIDGE=br0


[root@redhat1 :~ ]# cat ifcfg-bond0.10
DEVICE=bond0.10
BOOTPROTO=none
ONPARENT=yes
#IPADDR=10.50.21.17
#NETMASK=255.255.255.0
VLAN=yes

[root@redhat1 :~ ]# cat ifcfg-br0
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
UUID=4451286d-e40c-4d8c-915f-7fc12a16d595
DEVICE=br0
ONBOOT=yes
IPADDR=10.50.50.16
PREFIX=26
GATEWAY=10.50.0.1
DNS1=172.20.0.2
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-br1
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br1
UUID=40360c3c-47f5-44ac-bbeb-77f203390d29
DEVICE=br1
ONBOOT=yes
##IPADDR=10.50.51.241
PREFIX=28
##GATEWAY=10.50.0.1
##DNS1=172.20.0.2
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-br2
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br2
UUID=fbd5c257-2f66-4f2b-9372-881b783276e0
DEVICE=br2
ONBOOT=yes
##IPADDR=10.50.51.243
PREFIX=28
##GATEWAY=10.50.0.1
##DNS1=172.20.10.1
NM_CONTROLLED=no
NM_CONTROLLED=no
BRIDGE=br0

[root@redhat1 :~ ]# cat ifcfg-eth0
TYPE=Ethernet
NAME=bond0-slaveeth0
BOOTPROTO=none
#UUID=61065574-2a9d-4f16-b16e-00f495e2ee2b
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

[root@redhat1 :~ ]# cat ifcfg-eth1
TYPE=Ethernet
NAME=eth1
UUID=b4c359ae-7a13-436b-a904-beafb4edee94
DEVICE=eth1
ONBOOT=yes
BRIDGE=br1
NM_CONTROLLED=no

[root@redhat1 :~ ]#  cat ifcfg-eth2
TYPE=Ethernet
NAME=bond0-slaveeth2
BOOTPROTO=none
#UUID=821d711d-47b9-490a-afe7-190811578ef7
DEVICE=eth2
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

[root@redhat1 :~ ]#  cat ifcfg-eth3
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
#BOOTPROTO=dhcp
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
BRIDGE=br2
NAME=eth3
UUID=61065574-2a9d-4f16-b16e-00f495e2ee2b
DEVICE=eth3
ONBOOT=yes
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
#IPADDR=10.50.21.16
#PREFIX=26
#GATEWAY=10.50.21.1
#DNS1=172.20.88.2
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=1 miimon=100 primary=eth0"
NM_CONTROLLED="no"
BRIDGE=br0

# cat ifcfg-bond0.10
DEVICE=bond0.10
BOOTPROTO=none
ONPARENT=yes
#IPADDR=10.50.21.17
#NETMASK=255.255.255.0
VLAN=yes
NM_CONTROLLED=no
BRIDGE=br0

[root@redhat2 :~ ]# cat ifcfg-br0
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
#UUID=f87e55a8-0fb4-4197-8ccc-0d8a671f30d0
UUID=4451286d-e40c-4d8c-915f-7fc12a16d595
DEVICE=br0
ONBOOT=yes
IPADDR=10.50.21.17
PREFIX=26
GATEWAY=10.50.21.1
DNS1=172.20.88.2
NM_CONTROLLED=no

[root@redhat2 :~ ]#  cat ifcfg-br1
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=no
#IPV6_AUTOCONF=no
#IPV6_DEFROUTE=no
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br1
UUID=40360c3c-47f5-44ac-bbeb-77f203390d29
DEVICE=br1
ONBOOT=yes
##IPADDR=10.50.21.242
PREFIX=28
##GATEWAY=10.50.21.1
##DNS1=172.20.88.2
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-br2
STP=yes
BRIDGING_OPTS=priority=32768
TYPE=Bridge
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=no
IPV4_FAILURE_FATAL=no
#IPV6INIT=no
#IPV6_AUTOCONF=no
#IPV6_DEFROUTE=no
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br2
UUID=fbd5c257-2f66-4f2b-9372-881b783276e0
DEVICE=br2
ONBOOT=yes
##IPADDR=10.50.21.244
PREFIX=28
##GATEWAY=10.50.21.1
##DNS1=172.20.88.2
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-eth0
TYPE=Ethernet
NAME=bond0-slaveeth0
BOOTPROTO=none
#UUID=ee950c07-7eb2-463b-be6e-f97e7ad9d476
DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-eth1
TYPE=Ethernet
NAME=eth1
UUID=ffec8039-58f0-494a-b335-7a423207c7e6
DEVICE=eth1
ONBOOT=yes
BRIDGE=br1
NM_CONTROLLED=no

[root@redhat2 :~ ]# cat ifcfg-eth2
TYPE=Ethernet
NAME=bond0-slaveeth2
BOOTPROTO=none
#UUID=2c097475-4bef-47c3-b241-f5e7f02b3395
DEVICE=eth2
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED=no


Notice that the bond0 configuration does not have an IP assigned this is done on purpose as we're using the interface channel bonding together with attached bridge for the VM. Usual bonding on a normal physical hardware hosts where no virtualization use is planned is perhaps a better choice. If you however try to set up an IP address in that specific configuration shown here and you try to reboot the machine, you will end up with inacessible machine over the network like I did and you will need to resolve configuration via some kind of ILO / IDRAC interface.

4. Generating UUID for ethernet devices bridges and bonds

One thing to note is the command uuidgen you might need that to generate UID identificators to fit in the new network config files.

Example:
 

[root@redhat2 :~ ]#uuidgen br2
e7995e15-7f23-4ea2-80d6-411add78d703
[root@redhat2 :~ ]# uuidgen br1
05e0c339-5998-414b-b720-7adf91a90103
[root@redhat2 :~ ]# uuidgen br0
e6d7ff74-4c15-4d93-a150-ff01b7ced5fb


5. How to make KVM Virtual Machines see configured Network bridges (modify VM XML)

To make the Virtual machines installed see the bridges I had to

[root@redhat1 :~ ]#virsh edit VM_name1
[root@redhat1 :~ ]#virsh edit VM_name2

[root@redhat2 :~ ]#virsh edit VM_name1
[root@redhat2 :~ ]#virsh edit VM_name2

Find the interface network configuration and change it to something like:

    <interface type='bridge'>
      <mac address='22:53:00:56:5d:ac'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='22:53:00:2a:5f:01'/>
      <source bridge='br1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='22:34:00:4a:1b:6c'/>
      <source bridge='br2'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </interface>


6. Testing the bond  is up and works fine

# ip addr show bond0
The result is the following:

 

4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:cb:25:82 brd ff:ff:ff:ff:ff:ff


The bond should be visible in the normal network interfaces with ip address show or /sbin/ifconfig

 

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:ab:2a:fa
Slave queue ID: 0

 

According to the output eth0 is the active slave.

The active slaves device files (eth0 in this case) is found in virtual file system /sys/

# find /sys -name *eth0
/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/net/eth0
/sys/devices/virtual/net/bond0/lower_eth0
/sys/class/net/eth0


You can remove a bond member say eth0 by 

 

 cd to the pci* directory
Example: /sys/devices/pci000:00/000:00:15.0

 

# echo 1 > remove


At this point the eth0 device directory structure that was previously located under /sys/devices/pci000:00/000:00:15.0 is no longer there.  It was removed and the device no longer exists as seen by the OS.

You can verify this is the case with a simple ifconfig which will no longer list the eth0 device.
You can also repeat the cat /proc/net/bonding/bond0 command from Step 1 to see that eth0 is no longer listed as active or available.
You can also see the change in the messages file.  It might look something like this:

2021-02-12T14:13:23.363414-06:00 redhat1  device eth0: device has been deleted
2021-02-12T14:13:23.368745-06:00 redhat1 kernel: [81594.846099] bonding: bond0: releasing active interface eth0
2021-02-12T14:13:23.368763-06:00 redhat1 kernel: [81594.846105] bonding: bond0: Warning: the permanent HWaddr of eth0 – 00:0c:29:ab:2a:f0 – is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts.
2021-02-12T14:13:23.368765-06:00 redhat1 kernel: [81594.846132] bonding: bond0: making interface eth1 the new active one.

 

Another way to test the bonding is correctly switching between LAN cards on case of ethernet hardware failure is to bring down one of the 2 or more bonded interfaces, lets say you want to switch from active-backup from eth1 to eth2, do:
 

# ip link set dev eth0 down


That concludes the test for fail over on active slave failure.

7. Bringing bond updown (rescan) bond with no need for server reboot

You know bonding is a tedious stuff that sometimes breaks up badly so only way to fix the broken bond seems to be a init 6 (reboot) cmd but no actually that is not so.

You can also get the deleted device back with a simple pci rescan command:

# echo 1 > /sys/bus/pci/rescan


The eth0 interface should now be back
You can see that it is back with an ifconfig command, and you can verify that the bond sees it with this command:

# cat /proc/net/bonding/bond0


That concludes the test of the bond code seeing the device when it comes back again.

The same steps can be repeated only this time using the eth1 device and file structure to fail the active slave in the bond back over to eth0.

8. Testing the bond with ifenslave command (ifenslave command examples)

Below is a set of useful information to test the bonding works as expected with ifenslave command  comes from "iputils-20071127" package

– To show information of all the inerfaces

                  # ifenslave -a
                  # ifenslave –all-interfaces 

 

– To change the active slave

                  # ifenslave -c bond0 eth1
                  # ifenslave –change-active bond0 eth1 

 

– To remove the slave interface from the bonding device

                  # ifenslave -d eth1
                  # ifenslave –detach bond0 eth1 

 

– To show master interface info

                  # ifenslave bond0 

 

– To set the bond device down and automatically release all the slaves

                  # ifenslave bond1 down 

– To get the usage info

                  # ifenslave -u
                  # ifenslave –usage 

– To set to verbose mode

                  # ifenslave -v
                  # ifenslave –verbose 

9. Testing the bridge works fine

Historically over the years all kind of bridges are being handled with the brctl part of bridge-utils .deb / .rpm installable package.

The classical way to check a bridge is working is to do

# brctl show
# brctl show br0; brctl show br1; brctl show br2

# brctl showmacs br0
 

etc.

Unfortunately with redhat 8 this command is no longer available so to get information about configured bridges you need to use instead:

 

# bridge link show
3:eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
4:eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state listening priority 32 cost 100


10. Troubleshooting network connectivity issues on bond bridges and LAN cards

Testing the bond connection and bridges can route proper traffic sometimes is a real hassle so here comes at help the good old tcpdump

If you end up with issues with some of the ethernet interfaces between HV1 and HV2 to be unable to talk to each other and you have some suspiciousness that some colleague from the network team has messed up a copper (UTP) cable or there is a connectivity fiber optics issues. To check the VLAN tagged traffic headers on the switch you can listen to each and every bond0 and br0, br1, br2 eth0, eth1, eth2, eth3 configured on the server like so:

# tcpdump -i bond0 -nn -e vlan


Some further investigation on where does a normal ICMP traffic flows once everything is setup is a normal thing to do, hence just try to route a normal ping via the different server interfaces:

# ping -I bond0 DSTADDR

# ping -i eth0 DSTADDR

# ping -i eth1 DSTADDR

# ping -i eth2 DSTADDR


After conducting the ping do the normal for network testing big ICMP packages (64k) ping to make sure there are no packet losses etc., e.g:

# ping -I eth3 -s 64536  DSTADDR


If for 10 – 20 seconds the ping does not return package losses then you should be good.

Debug and fix Virtuozzo / KVM broken Hypervisor error: ‘PrlSDKError(‘SDK error: 0x80000249: Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down.’ on CentOS Linux howto

Thursday, January 28th, 2021

fix-sdkerror-virtuozzo-kvm-how-to-debug-problems-with-hypervisor-host-linux

I've recently yum upgraded a CentOS Linux server runinng Virtuozzo kernel and Virtuozzo virtualization Virtual Machines to the latest available CentOS Linux release 7.9.2009 (Core) just to find out after the upgrade there was issues with both virtuozzo (VZ) way to list installed VZ enabled VMs reporting Unable to connect to Virtuozzo error like below:
 

[root@CENTOS etc]# prlctl list -a
Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down. Contact your Virtuozzo administrator for assistance.


Even the native QEMU KVM VMs installed on the Hypervisor system failed to work to list and bring up the VMs producing another unexplainable error with virsh unable to connect to the hypervisor socket

[root@CENTOS etc]# virsh list –all
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory


In dmesg cmd kernel log messages the error found looked as so:

[root@CENTOS etc]# dmesg|grep -i sdk


[    5.314601] PrlSDKError('SDK error: 0x80000249: Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down. Contact your Virtuozzo administrator for assistance.',)

To fix it I had to experiment a bit based on some suggestions from Google results as usual and what turned to be the cause is a now obsolete setting for disk probing that is breaking libvirtd

Disable allow_disk_format_probing in /etc/libvirt/qemu.conf

The fix to PrlSDKError('SDK error: 0x80000249: Unable to connect to Virtuozzo comes to commenting a parameter inside 

/etc/libvirt/qemu.conf

which for historical reasons seems to be turned on by default it is like this

allow_disk_format_probing = 1


Resolution is to either change the value to 0 or completely comment the line:

[root@CENTOS etc]# grep allow_disk_format_probing /etc/libvirt/qemu.conf
# If allow_disk_format_probing is enabled, libvirt will probe disk
#allow_disk_format_probing = 1
#allow_disk_format_probing = 1


Debug problems with Virtuozzo services and validate virtualization setup

What really helped to debug the issue was to check the extended status info of libvirtd.service vzevent vz.service libvirtguestd.service prl-disp systemd services

[root@CENTOS etc]# systemctl -l status libvirtd.service vzevent vz.service libvirtguestd.service prl-disp

Here I had to analyze the errors and googled a little bit about it


Once this is changed I had to of course restart libvirtd.service and rest of virtuozzo / kvm services

[root@CENTOS etc]# systemctl restart libvirtd.service ibvirtd.service vzevent vz.service libvirtguest.service prl-disp


Another useful tool part of a standard VZ install that I've used to make sure each of the Host OS Hypervisor components is running smoothly is virt-host-validate (tool is part of libvirt-client rpm package)

[root@CENTOS etc]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller mount-point                  : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpu' controller mount-point                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'cpuset' controller mount-point                  : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'devices' controller mount-point                 : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking for cgroup 'blkio' controller mount-point                   : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS


One thing to note here that virt-host-validate helped me to realize the  fuse (File system in userspace) module kernel support enabled on the HV was missing so I've enabled temporary for this boot with modprobe and permanently via a configuration like so:

# to load it one time
[root@CENTOS etc]#  modprobe fuse
# to load fuse permnanently on next boot

[root@CENTOS etc]#  echo fuse >> /etc/modules-load.d/fuse.conf

Disable selinux on CentOS HV

Another thing was selinux was enabled on the HV. Selinux is really annoying thing and to be honest I never used it on any server and though its idea is quite good the consequences it creates for daily sysadmin work are terrible so I usually disable it. It could be that a Hypervisor Host OS might work just normal with the selinux enabled but just in case I decided to remove it. This is how

[root@CENTOS etc]#  sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

To temporarily change the SELinux mode from targeted to permissive with the following command:

[root@CENTOS etc]#  setenforce 0

Edit /etc/selinux/config file and set the SELINUX mod to disabled

[root@CENTOS etc]# vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.
SELINUXTYPE=targeted

Finally rebooted graceously the machine just in case with the good recommended way to reboot servers with shutdown command instead of /sbin/reboot

[root@CENTOS etc]# shutdown -r now

The advantage of shutdown is that it tries to shutdown each service by sending stop requests but usually this takes some time and even a shutdown request could take longer to proccess as each service such as a WebServer application is being waited to close all its network connections etc. |
However if you want to have a quick reboot and you don't care about any established network connections to third party IPs you can go for the brutal old fashioned /sbin/reboot 🙂

Fix Oracle virtualbox Virtual Machine inacessible error in Linux / Windows Host OS

Monday, November 2nd, 2020

Fix Oracle virtualbox Virtual Machine inacessible error in Linux / Windows Host OS

Say you have installed a Virtual Machine on a Host OS be it Windows 7 / 10 or some GNU / Linux / FreeBSD OS and suddenly after a PC shutdown / restart the Virtual machine shows as it couldn't be run by Ora Virtualbox anymore with a message of VM Machine "inaccessible", what is causing this and how to fix it?

Virtualbox-virtual-machine-in-inaccessible-state-screenshot
This error is usually caused by the Guest Operating System's forceful behavior or an OS system crash such as sudden hang due to kernel error or due to sudden electricity drop. Whatever the reason if improper sending of termination process signals to Oracle Vbox are not properly handled this is causing the VBox to not recreate its own files and kill (close) the application of Oracle Virtualbox program in gracious way.

On Windows Virtualization Host OS the common files that needs to be tampered by Virtualbox are found under Virtualbox VMs\Name-of-VM:

virtualbox-vm-inacessible-_error-screenshot.png

For example if you have a Ubuntu installed

Virtualbox-virtual-machine-in-inaccessible-state-screenshot.png
C:\Users\\VirtualBox VMs\Ubuntu

If you have CentOS7 installed instead it will be under

C:\Users\\VirtualBox VMs\CentOS7

Usually under this dir you will find files which are with .vbox extension.

The virtual box files with extension .vbox contain metadata the virtualbox hypervisor requires to resolve the guest virtual OS' configuration.
If the main .vbox file is corrupted (i.e. reporting that it is empty) then use the backup .vbox-prev file to recover the contents of the original file.

How to solve the Virtualbox inacessible weird error:

Usually under the Virtualbox VMs\VM_name you'll find a directory / filestructure like:
 

Logs/
Snapshot/
VM_Name.vbox
VM_Name.vbox-prev
VM_Name.vbox-tmp

 

 

1. Considering .vbox is empty Rename the empty .vbox files a temporary name (e.g. rename VM_Name.vbox to VM_Name-empty.vbox). If you're unsure whether it is empty you can check by opening the file in a text editor and make sure it is empty.

2. Then make a copy of the backup file VM_Name.vbox-prev, where the copy will have the same name as the original but with the word "copy" appended to it (i.e. VM_Name.vbox-prev is renamed to VM_Name_copy.vbox-prev) as well as copy VM_Name.vbox-tmp to VM_Name_copy.vbox-tmp.

! Note that it is important to retain the original backup .vbox-prev file it should not be altered or itself renamed.

3. Now go rename the copy of the newly created .vbox-prev file VM_Name.vbox-prev to the name of the empty .vbox file (VM_Name.vbox).

Now that this is done you may add the .vbox file (guest os) back into the VBOX hypervisor.

If for some reason this did not work and you have a proper working copy of the Virtual Machine under VM_Name.vbox-tmp another approach to try is:
 

1.  Go to your Virtualbox folder i.e. C:\Users\hipo\VirtualBox VMs\Ubuntu

2. Check for file extensions files Ubuntu.vbox-tmp or Ubuntu.vbox-prev needed are there.

3. Overwrite Ubuntu.vbox file with the -tmp one, Just rename file from Ubuntu.vbox-tmp to Ubuntu.vbox

4.  Exit from Virtual Machine and Power it on again.

Hopefully this should recovered the state and snapshot of the "inaccessible" guest VM and
you should now see error gone away and VM will work as before.

 

How to install KVM Kernel-based Virtual Machine Virtualization on Linux

Sunday, October 14th, 2018

install-KVM-Kernel-based-Virtual-Machine-virtualization-on-Linux

If you want to run multiple virtual machines on GNU / Linux server or your Linux powered Desktop you have the possibility to use a couple of Virtual Machines just to name a few VirtualBox and VMWare are the option the native way to do it is using the Linux kernel via a loadable kernel module called KVM (Kernel-based Virtual Machine).
Though Oracle's Virtualbox generally works and you could add new test beds virtual machines (install multiple Linux / *BSD OS) it is not fully Free Software and not even fully open source licensed, VMWare even though superior as a Virtualization product is proprietary and its application costs a lot of money which not each develpoper or small / mid-sized company could afford.

Once the kvm.ko module is loaded your Linux kernel turns into a full-featured Virtual Machine Hypervisor.
Starting with Linux kernel 2.6.X the KVM Hypervisor is available and easy to install virtually all modern Linux distributions Redhat / CentOS Debian / Ubuntu etc. support it and its up to running few commands to install and start using the Power of Kernel embedded Virtualization.

KVM could be used to run in parallel multiple Operating Systems such as Windows / Linux / FreeBSD and others of BSDs family,  each running under a separate virtual machine with its private dedicated (isolated), disc, graphic card, network card etc.

To start up I assume you have already installed some kind of Linux distribution either locally or on a remote dedicated server.
 

1. Installing KVM on Debian GNU / Linux / Ubuntu / Mint and other deb based distros

 

Using APT tool install below packages:

 

root@jeremiah:~# apt install –yes qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils libguestfs-tools genisoimage virtinst libosinfo-bin

 

2. Installing virt-manager GUI to manage Virtual servers

 

root@jeremiah:~# apt-cache show virt-manager|grep -i desc -A 1
Description-en: desktop application for managing virtual machines
 It presents a summary view of running domains and their live performance &

Description-md5: 9f7f584744b77cdacc2291f2a8ac220e
Homepage: http://virt-manager.et.redhat.com/

 

root@jeremiah:~# apt install –yes virt-manager

 


virtual-manager-kvm-gnu-linux-virtual-machines-cpu-hdd-load-statistics-screenshot

 

 

virtual-manager-fedora-28-linux-virtual-machine-settings-screenshot


3. Configure bridged networking to allow access to newly configured VMs

Bridging has to be added via /etc/network/interfaces therefore it is a good idea to create a backup of it before modifying:

 

# cp -rpf /etc/network/interfaces /etc/network/interfaces.bakup-$(echo $(date '+%Y-%m-%d-%H'))

 

# vim /etc/network/interfaces

auto br0
 iface br0 inet static
         address 10.15.44.26
         netmask 255.255.255.192
         broadcast 10.15.44.63
         dns-nameservers 10.0.80.11 10.0.80.12
         # set static route for LAN
      post-up route add -net 10.0.0.0 netmask 255.0.0.0 gw 10.18.44.1
      post-up route add -net 161.26.0.0 netmask 255.255.0.0 gw 10.18.44.1
         bridge_ports eth0
         bridge_stp off
         bridge_fd 0
         bridge_maxwait 0
 
 # br1 setup with static wan IPv4 with ISP router as a default gateway
 auto br1
 iface br1 inet static
         address 192.168.222.51
         netmask 255.255.255.248
         broadcast 192.168.222.55
         gateway 192.168.222.49
         bridge_ports eth1
         bridge_stp off
         bridge_fd 0
         bridge_maxwait 0

 

Once file is saved in vim editor restart the networking.

 

# systemctl restart network.manager

 

To verify whether the bridge has been succesfully upped.

 

root@jeremiah:/home/hipo/kvm# brctl show
bridge name    bridge id        STP enabled    interfaces
virbr0        8000.525400cb1cd1    yes        virbr0-nic

 

4. List all installable Virtual OS images
 

root@jeremiah:/home/hipo/kvm# virt-builder -list
centos-6                 x86_64     CentOS 6.6
centos-7.0               x86_64     CentOS 7.0
centos-7.1               x86_64     CentOS 7.1
centos-7.2               aarch64    CentOS 7.2 (aarch64)
centos-7.2               x86_64     CentOS 7.2
centos-7.3               x86_64     CentOS 7.3
centos-7.4               x86_64     CentOS 7.4
centos-7.5               x86_64     CentOS 7.5
cirros-0.3.1             x86_64     CirrOS 0.3.1
cirros-0.3.5             x86_64     CirrOS 0.3.5
debian-6                 x86_64     Debian 6 (Squeeze)
debian-7                 sparc64    Debian 7 (Wheezy) (sparc64)
debian-7                 x86_64     Debian 7 (wheezy)
debian-8                 x86_64     Debian 8 (jessie)
debian-9                 x86_64     Debian 9 (stretch)
fedora-18                x86_64     Fedora® 18
fedora-19                x86_64     Fedora® 19
fedora-20                x86_64     Fedora® 20
fedora-21                aarch64    Fedora® 21 Server (aarch64)
fedora-21                armv7l     Fedora® 21 Server (armv7l)
fedora-21                ppc64      Fedora® 21 Server (ppc64)
fedora-21                ppc64le    Fedora® 21 Server (ppc64le)
fedora-21                x86_64     Fedora® 21 Server
fedora-22                aarch64    Fedora® 22 Server (aarch64)
fedora-22                armv7l     Fedora® 22 Server (armv7l)
fedora-22                i686       Fedora® 22 Server (i686)
fedora-22                x86_64     Fedora® 22 Server
fedora-23                aarch64    Fedora® 23 Server (aarch64)
fedora-23                armv7l     Fedora® 23 Server (armv7l)
fedora-23                i686       Fedora® 23 Server (i686)
fedora-23                ppc64      Fedora® 23 Server (ppc64)
fedora-23                ppc64le    Fedora® 23 Server (ppc64le)
fedora-23                x86_64     Fedora® 23 Server
fedora-24                aarch64    Fedora® 24 Server (aarch64)
fedora-24                armv7l     Fedora® 24 Server (armv7l)
fedora-24                i686       Fedora® 24 Server (i686)
fedora-24                x86_64     Fedora® 24 Server
fedora-25                aarch64    Fedora® 25 Server (aarch64)
fedora-25                armv7l     Fedora® 25 Server (armv7l)
fedora-25                i686       Fedora® 25 Server (i686)
fedora-25                ppc64      Fedora® 25 Server (ppc64)
fedora-25                ppc64le    Fedora® 25 Server (ppc64le)
fedora-25                x86_64     Fedora® 25 Server
fedora-26                aarch64    Fedora® 26 Server (aarch64)
fedora-26                armv7l     Fedora® 26 Server (armv7l)
fedora-26                i686       Fedora® 26 Server (i686)
fedora-26                ppc64      Fedora® 26 Server (ppc64)
fedora-26                ppc64le    Fedora® 26 Server (ppc64le)
fedora-26                x86_64     Fedora® 26 Server
fedora-27                aarch64    Fedora® 27 Server (aarch64)
fedora-27                armv7l     Fedora® 27 Server (armv7l)
fedora-27                i686       Fedora® 27 Server (i686)
fedora-27                ppc64      Fedora® 27 Server (ppc64)
fedora-27                ppc64le    Fedora® 27 Server (ppc64le)
fedora-27                x86_64     Fedora® 27 Server
fedora-28                i686       Fedora® 28 Server (i686)
fedora-28                x86_64     Fedora® 28 Server
freebsd-11.1             x86_64     FreeBSD 11.1
scientificlinux-6        x86_64     Scientific Linux 6.5
ubuntu-10.04             x86_64     Ubuntu 10.04 (Lucid)
ubuntu-12.04             x86_64     Ubuntu 12.04 (Precise)
ubuntu-14.04             x86_64     Ubuntu 14.04 (Trusty)
ubuntu-16.04             x86_64     Ubuntu 16.04 (Xenial)
ubuntu-18.04             x86_64     Ubuntu 18.04 (bionic)
opensuse-13.1            x86_64     openSUSE 13.1
opensuse-13.2            x86_64     openSUSE 13.2
opensuse-42.1            x86_64     openSUSE Leap 42.1
opensuse-tumbleweed      x86_64     openSUSE Tumbleweed


5. Create Virtual Machine OS-es from scratch with virt-builder

Below we'll create two images one for Fedora 28 and 1 for Debian 9 using the virt-builder (a tool to build virtual images quickly), the images that could be used are shown through below virt-builder –list command.
 

# iso='fedora-28';
# iso1='debian-9';

 

# sudo virt-builder $iso \
     –size=10G \
     –format qcow2 -o /var/lib/libvirt/images/$iso-vm1.qcow2 \
     –hostname $iso-vm1 \
     –network \
     –timezone Europe/Sofia

 

[   3.3] Downloading: http://libguestfs.org/download/builder/fedora-28.xz
[   5.2] Planning how to build this image
[   5.2] Uncompressing
[  20.8] Resizing (using virt-resize) to expand the disk to 10.0G
[  50.8] Opening the new disk
[  53.7] Setting a random seed
[  53.7] Setting the hostname: fedora-28-vm1
[  53.7] Setting the timezone: Europe/Sofia
[  53.7] Setting passwords
virt-builder: Setting random password of root to YMTkxaJIkEU24Ytf

[  54.7] Finishing off
                   Output file: /var/lib/libvirt/images/fedora-28-vm1.qcow2
                   Output size: 10.0G
                 Output format: qcow2
            Total usable space: 9.3G
                    Free space: 8.2G (87%)

 

# sudo virt-builder $iso1 \
     –size=10G \
     –format qcow2 -o /var/lib/libvirt/images/$iso-vm1.qcow2 \
     –hostname $iso1-vm1 \
     –network \
     –timezone Europe/Sofia

 

[   3.2] Downloading: http://libguestfs.org/download/builder/debian-9.xz
[   4.1] Planning how to build this image
[   4.1] Uncompressing
[  16.9] Resizing (using virt-resize) to expand the disk to 10.0G
[  40.1] Opening the new disk
[  42.9] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
[  42.9] Setting the hostname: debian-9-vm1
[  43.6] Setting the timezone: Europe/Sofia
[  43.6] Setting passwords
virt-builder: Setting random password of root to JtzEYGff9KxL5jCR
[  44.3] Finishing off
                   Output file: /var/lib/libvirt/images/debian-9-vm1.qcow2
                   Output size: 10.0G
                 Output format: qcow2
            Total usable space: 9.8G
                    Free space: 9.0G (91%)


vim bridged.xml

<network>
  <name>br0</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
</network>

 

# sudo virsh net-define –file bridged.xml
# sudo virsh net-autostart br0
# sudo virsh net-start br0

 

Above two commands will download pre-packaged KVM isos and store them inside /var/lib/libvirt/images/ you see also the root (administrator) password for both ISOs printed out.

 

root@jeremiah:/home/hipo/kvm# ls -ld /var/lib/libvirt/images/*
-rw-r–r– 1 root         root         10739318784 Oct 12 23:45 /var/lib/libvirt/images/debian-9-vm1.qcow2
-rw-r–r– 1 root         root         10739318784 Oct 12 23:46 /var/lib/libvirt/images/fedora-28-vm1.qcow2

 

To access directly the new created VMs as we have specified the –vnc option it is possible to directly vnc to the new host with VNC client (in linux I use vncviewer), on Windows you can use something like TightVNC.
 

6. Use official Linux distributions ISO boot files to install into KVM VM


Those who would like to run inside KVM VM Linux could do it directly using installable ISO files and install the set of Linux with the required packages, just like installing a fresh new Linux on a bare-metal machine.
To do so download your ISO image from the net (either from official distro website or a mirror website, in case if you need to spin an older version) and use virt-install to run the installer inside KVM.

 

root@jeremiah:~# cd /var/lib/libvirt/boot/;
root@jeremiah:~# wget http://mirrors.netix.net/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-DVD-1804.iso

 

# sudo virt-install \
–virt-type=kvm \
–name centos7 \
–ram 2048 \
–vcpus=2 \
–os-variant=centos7.0 \
–virt-type=kvm \
–hvm \
–cdrom=/var/lib/libvirt/boot/CentOS-7-x86_64-DVD-1804.iso \
–network=bridge=br0,model=virtio \
–network=bridge=br1,model=virtio \
–graphics vnc \
–disk path=/var/lib/libvirt/images/centos7.qcow2,size=40,bus=virtio,format=qcow2


7. List newly created VMs with Virsh command

 

root@jeremiah:/home/hipo/kvm# virsh list –all
 Id    Name                           State
—————————————————-
 3     fedora-28                      running
 –     debian9                        shut off

 

The –all parameter lists all available VMs ready to spin, if you want to check what are the VMs that are only running use instead:

 

root@jeremiah:/home/hipo/kvm# virsh list
 Id    Name                           State
—————————————————-
 3     fedora-28                      running

 

8. Install Virtual Machine OS-es

Below lines will install 2 Virtual machines one Fedora 28 and Debian 9

 

 os='fedora-28';
virt-install –import –name $os \
    –ram 2048 \
    –vcpu 2 \
    –disk path=/var/lib/libvirt/images/$os-vm1.qcow2,format=qcow2 \
    –os-variant fedora-unknown \
    –network=bridge=br0,model=virtio \
    –noautoconsole \
  –hvm \
  –graphics vnc

os='debian9';
virt-install –import –name $os     \
–ram 2048     \
–vcpu 2     \
–disk path=/var/lib/libvirt/images/$os-vm1.qcow2,format=qcow2     \
–os-variant debian9     –network=bridge=br0,model=virtio     \
–noautoconsole \
–hvm \
–graphics vnc


To deploy more just change the virtual machine type in os variable and modify the –os-variant variable to match the distribution name, to get the correct –os-variant variables that can be passed use osinfo-query below is output of the cmd:

 

root@jeremiah:/home/hipo/kvm# osinfo-query os
 Short ID             | Name                                               | Version  | ID                                      
———————-+—————————————————-+———-+—————————————–
 altlinux1.0          | Mandrake RE Spring 2001                            | 1.0      | http://altlinux.org/altlinux/1.0        
 altlinux2.0          | ALT Linux 2.0                                      | 2.0      | http://altlinux.org/altlinux/2.0        
 altlinux2.2          | ALT Linux 2.2                                      | 2.2      | http://altlinux.org/altlinux/2.2        
 altlinux2.4          | ALT Linux 2.4                                      | 2.4      | http://altlinux.org/altlinux/2.4        
 altlinux3.0          | ALT Linux 3.0                                      | 3.0      | http://altlinux.org/altlinux/3.0        
 altlinux4.0          | ALT Linux 4.0                                      | 4.0      | http://altlinux.org/altlinux/4.0        
 altlinux4.1          | ALT Linux 4.1                                      | 4.1      | http://altlinux.org/altlinux/4.1        
 altlinux5.0          | ALT Linux 5.0                                      | 5.0      | http://altlinux.org/altlinux/5.0        
 altlinux6.0          | ALT Linux 6.0                                      | 6.0      | http://altlinux.org/altlinux/6.0        
 altlinux7.0          | ALT Linux 7.0                                      | 7.0      | http://altlinux.org/altlinux/7.0        
 centos6.0            | CentOS 6.0                                         | 6.0      | http://centos.org/centos/6.0            
 centos6.1            | CentOS 6.1                                         | 6.1      | http://centos.org/centos/6.1            
 centos6.2            | CentOS 6.2                                         | 6.2      | http://centos.org/centos/6.2            
 centos6.3            | CentOS 6.3                                         | 6.3      | http://centos.org/centos/6.3            
 centos6.4            | CentOS 6.4                                         | 6.4      | http://centos.org/centos/6.4            
 centos6.5            | CentOS 6.5                                         | 6.5      | http://centos.org/centos/6.5            
 centos6.6            | CentOS 6.6                                         | 6.6      | http://centos.org/centos/6.6            
 centos6.7            | CentOS 6.7                                         | 6.7      | http://centos.org/centos/6.7            
 centos6.8            | CentOS 6.8                                         | 6.8      | http://centos.org/centos/6.8            
 centos6.9            | CentOS 6.9                                         | 6.9      | http://centos.org/centos/6.9            
 centos7.0            | CentOS 7.0                                         | 7.0      | http://centos.org/centos/7.0            
 debian1.1            | Debian Buzz                                        | 1.1      | http://debian.org/debian/1.1            
 debian1.2            | Debian Rex                                         | 1.2      | http://debian.org/debian/1.2            
 debian1.3            | Debian Bo                                          | 1.3      | http://debian.org/debian/1.3            
 debian2.0            | Debian Hamm                                        | 2.0      | http://debian.org/debian/2.0            
 debian2.1            | Debian Slink                                       | 2.1      | http://debian.org/debian/2.1            
 debian2.2            | Debian Potato                                      | 2.2      | http://debian.org/debian/2.2            
 debian3              | Debian Woody                                       | 3        | http://debian.org/debian/3              
 debian3.1            | Debian Sarge                                       | 3.1      | http://debian.org/debian/3.1            
 debian4              | Debian Etch                                        | 4        | http://debian.org/debian/4              
 debian5              | Debian Lenny                                       | 5        | http://debian.org/debian/5              
 debian6              | Debian Squeeze                                     | 6        | http://debian.org/debian/6              
 debian7              | Debian Wheezy                                      | 7        | http://debian.org/debian/7              
 debian8              | Debian Jessie                                      | 8        | http://debian.org/debian/8              
 debian9              | Debian Stretch                                     | 9        | http://debian.org/debian/9              
 debiantesting        | Debian Testing                                     | testing  | http://debian.org/debian/testing        
 fedora-unknown       | Fedora                                             | unknown  | http://fedoraproject.org/fedora/unknown
 fedora1              | Fedora Core 1                                      | 1        | http://fedoraproject.org/fedora/1       
 fedora10             | Fedora 10                                          | 10       | http://fedoraproject.org/fedora/10      
 fedora11             | Fedora 11                                          | 11       | http://fedoraproject.org/fedora/11      
 fedora12             | Fedora 12                                          | 12       | http://fedoraproject.org/fedora/12      
 fedora13             | Fedora 13                                          | 13       | http://fedoraproject.org/fedora/13      
 fedora14             | Fedora 14                                          | 14       | http://fedoraproject.org/fedora/14      
 fedora15             | Fedora 15                                          | 15       | http://fedoraproject.org/fedora/15      
 fedora16             | Fedora 16                                          | 16       | http://fedoraproject.org/fedora/16      
 fedora17             | Fedora 17                                          | 17       | http://fedoraproject.org/fedora/17      
 fedora18             | Fedora 18                                          | 18       | http://fedoraproject.org/fedora/18      
 fedora19             | Fedora 19                                          | 19       | http://fedoraproject.org/fedora/19      
 fedora2              | Fedora Core 2                                      | 2        | http://fedoraproject.org/fedora/2       
 fedora20             | Fedora 20                                          | 20       | http://fedoraproject.org/fedora/20      
 fedora21             | Fedora 21                                          | 21       | http://fedoraproject.org/fedora/21      
 fedora22             | Fedora 22                                          | 22       | http://fedoraproject.org/fedora/22      
 fedora23             | Fedora 23                                          | 23       | http://fedoraproject.org/fedora/23      
 fedora24             | Fedora 24                                          | 24       | http://fedoraproject.org/fedora/24      
 fedora25             | Fedora 25                                          | 25       | http://fedoraproject.org/fedora/25      
 fedora26             | Fedora 26                                          | 26       | http://fedoraproject.org/fedora/26      
 fedora3              | Fedora Core 3                                      | 3        | http://fedoraproject.org/fedora/3       
 fedora4              | Fedora Core 4                                      | 4        | http://fedoraproject.org/fedora/4       
 fedora5              | Fedora Core 5                                      | 5        | http://fedoraproject.org/fedora/5       
 fedora6              | Fedora Core 6                                      | 6        | http://fedoraproject.org/fedora/6       
 fedora7              | Fedora 7                                           | 7        | http://fedoraproject.org/fedora/7       
 fedora8              | Fedora 8                                           | 8        | http://fedoraproject.org/fedora/8       
 fedora9              | Fedora 9                                           | 9        | http://fedoraproject.org/fedora/9       
 freebsd1.0           | FreeBSD 1.0                                        | 1.0      | http://freebsd.org/freebsd/1.0          
 freebsd10.0          | FreeBSD 10.0                                       | 10.0     | http://freebsd.org/freebsd/10.0         
 freebsd10.1          | FreeBSD 10.1                                       | 10.1     | http://freebsd.org/freebsd/10.1         
 freebsd10.2          | FreeBSD 10.2                                       | 10.2     | http://freebsd.org/freebsd/10.2         
 freebsd10.3          | FreeBSD 10.3                                       | 10.3     | http://freebsd.org/freebsd/10.3         
 freebsd10.4          | FreeBSD 10.4                                       | 10.4     | http://freebsd.org/freebsd/10.4         
 freebsd11.0          | FreeBSD 11.0                                       | 11.0     | http://freebsd.org/freebsd/11.0         
 freebsd11.1          | FreeBSD 11.1                                       | 11.1     | http://freebsd.org/freebsd/11.1         
 freebsd2.0           | FreeBSD 2.0                                        | 2.0      | http://freebsd.org/freebsd/2.0          
 freebsd2.0.5         | FreeBSD 2.0.5                                      | 2.0.5    | http://freebsd.org/freebsd/2.0.5        
 freebsd2.2.8         | FreeBSD 2.2.8                                      | 2.2.8    | http://freebsd.org/freebsd/2.2.8        
 freebsd2.2.9         | FreeBSD 2.2.9                                      | 2.2.9    | http://freebsd.org/freebsd/2.2.9        
 freebsd3.0           | FreeBSD 3.0                                        | 3.0      | http://freebsd.org/freebsd/3.0          
 freebsd3.2           | FreeBSD 3.2                                        | 3.2      | http://freebsd.org/freebsd/3.2          
 freebsd4.0           | FreeBSD 4.0                                        | 4.0      | http://freebsd.org/freebsd/4.0          
 freebsd4.1           | FreeBSD 4.1                                        | 4.1      | http://freebsd.org/freebsd/4.1          
 freebsd4.10          | FreeBSD 4.10                                       | 4.10     | http://freebsd.org/freebsd/4.10         
 freebsd4.11          | FreeBSD 4.11                                       | 4.11     | http://freebsd.org/freebsd/4.11         
 freebsd4.2           | FreeBSD 4.2                                        | 4.2      | http://freebsd.org/freebsd/4.2          
 freebsd4.3           | FreeBSD 4.3                                        | 4.3      | http://freebsd.org/freebsd/4.3          
 freebsd4.4           | FreeBSD 4.4                                        | 4.4      | http://freebsd.org/freebsd/4.4          
 freebsd4.5           | FreeBSD 4.5                                        | 4.5      | http://freebsd.org/freebsd/4.5          
 freebsd4.6           | FreeBSD 4.6                                        | 4.6      | http://freebsd.org/freebsd/4.6          
 freebsd4.7           | FreeBSD 4.7                                        | 4.7      | http://freebsd.org/freebsd/4.7          
 freebsd4.8           | FreeBSD 4.8                                        | 4.8      | http://freebsd.org/freebsd/4.8          
 freebsd4.9           | FreeBSD 4.9                                        | 4.9      | http://freebsd.org/freebsd/4.9          
 freebsd5.0           | FreeBSD 5.0                                        | 5.0      | http://freebsd.org/freebsd/5.0          
 freebsd5.1           | FreeBSD 5.1                                        | 5.1      | http://freebsd.org/freebsd/5.1          
 freebsd5.2           | FreeBSD 5.2                                        | 5.2      | http://freebsd.org/freebsd/5.2          
 freebsd5.2.1         | FreeBSD 5.2.1                                      | 5.2.1    | http://freebsd.org/freebsd/5.2.1        
 freebsd5.3           | FreeBSD 5.3                                        | 5.3      | http://freebsd.org/freebsd/5.3          
 freebsd5.4           | FreeBSD 5.4                                        | 5.4      | http://freebsd.org/freebsd/5.4          
 freebsd5.5           | FreeBSD 5.5                                        | 5.5      | http://freebsd.org/freebsd/5.5          
 freebsd6.0           | FreeBSD 6.0                                        | 6.0      | http://freebsd.org/freebsd/6.0          
 freebsd6.1           | FreeBSD 6.1                                        | 6.1      | http://freebsd.org/freebsd/6.1          
 freebsd6.2           | FreeBSD 6.2                                        | 6.2      | http://freebsd.org/freebsd/6.2          
 freebsd6.3           | FreeBSD 6.3                                        | 6.3      | http://freebsd.org/freebsd/6.3          
 freebsd6.4           | FreeBSD 6.4                                        | 6.4      | http://freebsd.org/freebsd/6.4          
 freebsd7.0           | FreeBSD 7.0                                        | 7.0      | http://freebsd.org/freebsd/7.0          
 freebsd7.1           | FreeBSD 7.1                                        | 7.1      | http://freebsd.org/freebsd/7.1          
 freebsd7.2           | FreeBSD 7.2                                        | 7.2      | http://freebsd.org/freebsd/7.2          
 freebsd7.3           | FreeBSD 7.3                                        | 7.3      | http://freebsd.org/freebsd/7.3          
 freebsd7.4           | FreeBSD 7.4                                        | 7.4      | http://freebsd.org/freebsd/7.4          
 freebsd8.0           | FreeBSD 8.0                                        | 8.0      | http://freebsd.org/freebsd/8.0          
 freebsd8.1           | FreeBSD 8.1                                        | 8.1      | http://freebsd.org/freebsd/8.1          
 freebsd8.2           | FreeBSD 8.2                                        | 8.2      | http://freebsd.org/freebsd/8.2          
 freebsd8.3           | FreeBSD 8.3                                        | 8.3      | http://freebsd.org/freebsd/8.3          
 freebsd8.4           | FreeBSD 8.4                                        | 8.4      | http://freebsd.org/freebsd/8.4          
 freebsd9.0           | FreeBSD 9.0                                        | 9.0      | http://freebsd.org/freebsd/9.0          
 freebsd9.1           | FreeBSD 9.1                                        | 9.1      | http://freebsd.org/freebsd/9.1          
 freebsd9.2           | FreeBSD 9.2                                        | 9.2      | http://freebsd.org/freebsd/9.2          
 freebsd9.3           | FreeBSD 9.3                                        | 9.3      | http://freebsd.org/freebsd/9.3          
 freedos1.2           | FreeDOS 1.2                                        | 1.2      | http://freedos.org/freedos/1.2          
 gnome-continuous-3.10 | GNOME 3.10                                         | 3.10     | http://gnome.org/gnome-continuous/3.10  
 gnome-continuous-3.12 | GNOME 3.12                                         | 3.12     | http://gnome.org/gnome-continuous/3.12  
 gnome-continuous-3.14 | GNOME 3.14                                         | 3.14     | http://gnome.org/gnome-continuous/3.14  
 gnome3.6             | GNOME 3.6                                          | 3.6      | http://gnome.org/gnome/3.6              
 gnome3.8             | GNOME 3.8                                          | 3.8      | http://gnome.org/gnome/3.8              
 macosx10.0           | MacOS X Cheetah                                    | 10.0     | http://apple.com/macosx/10.0            
 macosx10.1           | MacOS X Puma                                       | 10.1     | http://apple.com/macosx/10.1            
 macosx10.2           | MacOS X Jaguar                                     | 10.2     | http://apple.com/macosx/10.2            
 macosx10.3           | MacOS X Panther                                    | 10.3     | http://apple.com/macosx/10.3            
 macosx10.4           | MacOS X Tiger                                      | 10.4     | http://apple.com/macosx/10.4            
 macosx10.5           | MacOS X Leopard                                    | 10.5     | http://apple.com/macosx/10.5            
 macosx10.6           | MacOS X Snow Leopard                               | 10.6     | http://apple.com/macosx/10.6            
 macosx10.7           | MacOS X Lion                                       | 10.7     | http://apple.com/macosx/10.7            
 mageia1              | Mageia 1                                           | 1        | http://mageia.org/mageia/1              
 mageia2              | Mageia 2                                           | 2        | http://mageia.org/mageia/2              
 mageia3              | Mageia 3                                           | 3        | http://mageia.org/mageia/3              
 mageia4              | Mageia 4                                           | 4        | http://mageia.org/mageia/4              
 mageia5              | Mageia 5                                           | 5        | http://mageia.org/mageia/5              
 mageia6              | Mageia 6                                           | 6        | http://mageia.org/mageia/6              
 mandrake10.0         | Mandrake Linux 10.0                                | 10.0     | http://mandriva.com/mandrake/10.0       
 mandrake10.1         | Mandrake Linux 10.1                                | 10.1     | http://mandriva.com/mandrake/10.1       
 mandrake10.2         | Mandrake Linux 10.2                                | 10.2     | http://mandriva.com/mandrake/10.2       
 mandrake5.1          | Mandrake Linux 5.1                                 | 5.1      | http://mandriva.com/mandrake/5.1        
 mandrake5.2          | Mandrake Linux 5.2                                 | 5.2      | http://mandriva.com/mandrake/5.2        
 mandrake5.3          | Mandrake Linux 5.3                                 | 5.3      | http://mandriva.com/mandrake/5.3        
 mandrake6.0          | Mandrake Linux 6.0                                 | 6.0      | http://mandriva.com/mandrake/6.0        
 mandrake6.1          | Mandrake Linux 6.1                                 | 6.1      | http://mandriva.com/mandrake/6.1        
 mandrake7.0          | Mandrake Linux 7.0                                 | 7.0      | http://mandriva.com/mandrake/7.0        
 mandrake7.1          | Mandrake Linux 7.1                                 | 7.1      | http://mandriva.com/mandrake/7.1        
 mandrake7.2          | Mandrake Linux 7.2                                 | 7.2      | http://mandriva.com/mandrake/7.2        
 mandrake8.0          | Mandrake Linux 8.0                                 | 8.0      | http://mandriva.com/mandrake/8.0        
 mandrake8.1          | Mandrake Linux 8.1                                 | 8.1      | http://mandriva.com/mandrake/8.1        
 mandrake8.2          | Mandrake Linux 8.2                                 | 8.2      | http://mandriva.com/mandrake/8.2        
 mandrake9.0          | Mandrake Linux 9.0                                 | 9.0      | http://mandriva.com/mandrake/9.0        
 mandrake9.1          | Mandrake Linux 9.1                                 | 9.1      | http://mandriva.com/mandrake/9.1        
 mandrake9.2          | Mandrake Linux 9.2                                 | 9.2      | http://mandriva.com/mandrake/9.2        
 mandriva2006.0       | Mandriva Linux 2006.0                              | 2006.0   | http://mandriva.com/mandriva/2006.0     
 mandriva2007         | Mandriva Linux 2007                                | 2007     | http://mandriva.com/mandriva/2007       
 mandriva2007.1       | Mandriva Linux 2007 Spring                         | 2007.1   | http://mandriva.com/mandriva/2007.1     
 mandriva2008.0       | Mandriva Linux 2008                                | 2008.0   | http://mandriva.com/mandriva/2008.0     
 mandriva2008.1       | Mandriva Linux 2008 Spring                         | 2008.1   | http://mandriva.com/mandriva/2008.1     
 mandriva2009.0       | Mandriva Linux 2009                                | 2009.0   | http://mandriva.com/mandriva/2009.0     
 mandriva2009.1       | Mandriva Linux 2009 Spring                         | 2009.1   | http://mandriva.com/mandriva/2009.1     
 mandriva2010.0       | Mandriva Linux 2010                                | 2010.0   | http://mandriva.com/mandriva/2010.0     
 mandriva2010.1       | Mandriva Linux 2010 Spring                         | 2010.1   | http://mandriva.com/mandriva/2010.1     
 mandriva2010.2       | Mandriva Linux 2010.2                              | 2010.2   | http://mandriva.com/mandriva/2010.2     
 mandriva2011         | Mandriva Linux 2011                                | 2011     | http://mandriva.com/mandriva/2011       
 mbs1.0               | Mandriva Business Server 1.0                       | 1.0      | http://mandriva.com/mbs/1.0             
 mes5                 | Mandriva Enterprise Server 5.0                     | 5.0      | http://mandriva.com/mes/5.0             
 mes5.1               | Mandriva Enterprise Server 5.1                     | 5.1      | http://mandriva.com/mes/5.1             
 msdos6.22            | Microsoft MS-DOS 6.22                              | 6.22     | http://microsoft.com/msdos/6.22         
 netbsd0.8            | NetBSD 0.8                                         | 0.8      | http://netbsd.org/netbsd/0.8            
 netbsd0.9            | NetBSD 0.9                                         | 0.9      | http://netbsd.org/netbsd/0.9            
 netbsd1.0            | NetBSD 1.0                                         | 1.0      | http://netbsd.org/netbsd/1.0            
 netbsd1.1            | NetBSD 1.1                                         | 1.1      | http://netbsd.org/netbsd/1.1            
 netbsd1.2            | NetBSD 1.2                                         | 1.2      | http://netbsd.org/netbsd/1.2            
 netbsd1.3            | NetBSD 1.3                                         | 1.3      | http://netbsd.org/netbsd/1.3            
 netbsd1.4            | NetBSD 1.4                                         | 1.4      | http://netbsd.org/netbsd/1.4            
 netbsd1.5            | NetBSD 1.5                                         | 1.5      | http://netbsd.org/netbsd/1.5            
 netbsd1.6            | NetBSD 1.6                                         | 1.6      | http://netbsd.org/netbsd/1.6            
 netbsd2.0            | NetBSD 2.0                                         | 2.0      | http://netbsd.org/netbsd/2.0            
 netbsd3.0            | NetBSD 3.0                                         | 3.0      | http://netbsd.org/netbsd/3.0            
 netbsd4.0            | NetBSD 4.0                                         | 4.0      | http://netbsd.org/netbsd/4.0            
 netbsd5.0            | NetBSD 5.0                                         | 5.0      | http://netbsd.org/netbsd/5.0            
 netbsd5.1            | NetBSD 5.1                                         | 5.1      | http://netbsd.org/netbsd/5.1            
 netbsd6.0            | NetBSD 6.0                                         | 6.0      | http://netbsd.org/netbsd/6.0            
 netbsd6.1            | NetBSD 6.1                                         | 6.1      | http://netbsd.org/netbsd/6.1            
 netbsd7.0            | NetBSD 7.0                                         | 7.0      | http://netbsd.org/netbsd/7.0            
 netbsd7.1            | NetBSD 7.1                                         | 7.1      | http://netbsd.org/netbsd/7.1            
 netbsd7.1.1          | NetBSD 7.1.1                                       | 7.1.1    | http://netbsd.org/netbsd/7.1.1          
 netware4             | Novell Netware 4                                   | 4        | http://novell.com/netware/4             
 netware5             | Novell Netware 5                                   | 5        | http://novell.com/netware/5             
 netware6             | Novell Netware 6                                   | 6        | http://novell.com/netware/6             
 openbsd4.2           | OpenBSD 4.2                                        | 4.2      | http://openbsd.org/openbsd/4.2          
 openbsd4.3           | OpenBSD 4.3                                        | 4.3      | http://openbsd.org/openbsd/4.3          
 openbsd4.4           | OpenBSD 4.4                                        | 4.4      | http://openbsd.org/openbsd/4.4          
 openbsd4.5           | OpenBSD 4.5                                        | 4.5      | http://openbsd.org/openbsd/4.5          
 openbsd4.8           | OpenBSD 4.8                                        | 4.8      | http://openbsd.org/openbsd/4.8          
 openbsd4.9           | OpenBSD 4.9                                        | 4.9      | http://openbsd.org/openbsd/4.9          
 openbsd5.0           | OpenBSD 5.0                                        | 5.0      | http://openbsd.org/openbsd/5.0          
 openbsd5.1           | OpenBSD 5.1                                        | 5.1      | http://openbsd.org/openbsd/5.1          
 openbsd5.2           | OpenBSD 5.2                                        | 5.2      | http://openbsd.org/openbsd/5.2          
 openbsd5.3           | OpenBSD 5.3                                        | 5.3      | http://openbsd.org/openbsd/5.3          
 openbsd5.4           | OpenBSD 5.4                                        | 5.4      | http://openbsd.org/openbsd/5.4          
 openbsd5.5           | OpenBSD 5.5                                        | 5.5      | http://openbsd.org/openbsd/5.5          
 openbsd5.6           | OpenBSD 5.6                                        | 5.6      | http://openbsd.org/openbsd/5.6          
 openbsd5.7           | OpenBSD 5.7                                        | 5.7      | http://openbsd.org/openbsd/5.7          
 openbsd5.8           | OpenBSD 5.8                                        | 5.8      | http://openbsd.org/openbsd/5.8          
 openbsd5.9           | OpenBSD 5.9                                        | 5.9      | http://openbsd.org/openbsd/5.9          
 openbsd6.0           | OpenBSD 6.0                                        | 6.0      | http://openbsd.org/openbsd/6.0          
 openbsd6.1           | OpenBSD 6.1                                        | 6.1      | http://openbsd.org/openbsd/6.1          
 openbsd6.2           | OpenBSD 6.2                                        | 6.2      | http://openbsd.org/openbsd/6.2          
 opensolaris2009.06   | OpenSolaris 2009.06                                | 2009.06  | http://sun.com/opensolaris/2009.06      
 opensuse-factory     | openSUSE                                           | factory  | http://opensuse.org/opensuse/factory    
 opensuse-unknown     | openSUSE                                           | unknown  | http://opensuse.org/opensuse/unknown    
 opensuse10.2         | openSUSE 10.2                                      | 10.2     | http://opensuse.org/opensuse/10.2       
 opensuse10.3         | openSUSE 10.3                                      | 10.3     | http://opensuse.org/opensuse/10.3       
 opensuse11.0         | openSUSE 11.0                                      | 11.0     | http://opensuse.org/opensuse/11.0       
 opensuse11.1         | openSUSE 11.1                                      | 11.1     | http://opensuse.org/opensuse/11.1       
 opensuse11.2         | openSUSE 11.2                                      | 11.2     | http://opensuse.org/opensuse/11.2       
 opensuse11.3         | openSUSE 11.3                                      | 11.3     | http://opensuse.org/opensuse/11.3       
 opensuse11.4         | openSUSE 11.4                                      | 11.4     | http://opensuse.org/opensuse/11.4       
 opensuse12.1         | openSUSE 12.1                                      | 12.1     | http://opensuse.org/opensuse/12.1       
 opensuse12.2         | openSUSE 12.2                                      | 12.2     | http://opensuse.org/opensuse/12.2       
 opensuse12.3         | openSUSE 12.3                                      | 12.3     | http://opensuse.org/opensuse/12.3       
 opensuse13.1         | openSUSE 13.1                                      | 13.1     | http://opensuse.org/opensuse/13.1       
 opensuse13.2         | openSUSE 13.2                                      | 13.2     | http://opensuse.org/opensuse/13.2       
 opensuse42.1         | openSUSE Leap 42.1                                 | 42.1     | http://opensuse.org/opensuse/42.1       
 opensuse42.2         | openSUSE Leap 42.2                                 | 42.2     | http://opensuse.org/opensuse/42.2       
 opensuse42.3         | openSUSE Leap 42.3                                 | 42.3     | http://opensuse.org/opensuse/42.3       
 opensusetumbleweed   | openSUSE Tumbleweed                                | tumbleweed | http://opensuse.org/opensuse/tumbleweed
 rhel-atomic-7.0      | Red Hat Enterprise Linux Atomic Host 7.0           | 7.0      | http://redhat.com/rhel-atomic/7.0       
 rhel-atomic-7.1      | Red Hat Enterprise Linux Atomic Host 7.1           | 7.1      | http://redhat.com/rhel-atomic/7.1       
 rhel-atomic-7.2      | Red Hat Enterprise Linux Atomic Host 7.2           | 7.2      | http://redhat.com/rhel-atomic/7.2       
 rhel2.1              | Red Hat Enterprise Linux 2.1                       | 2.1      | http://redhat.com/rhel/2.1              
 rhel2.1.1            | Red Hat Enterprise Linux 2.1 Update 1  
/etc/bind/masters/elinvent.com            | 2.1.1    | http://redhat.com/rhel/2.1.1            
 rhel2.1.2            | Red Hat Enterprise Linux 2.1 Update 2              | 2.1.2    | http://redhat.com/rhel/2.1.2            
 rhel2.1.3            | Red Hat Enterprise Linux 2.1 Update 3              | 2.1.3    | http://redhat.com/rhel/2.1.3            
 rhel2.1.4            | Red Hat Enterprise Linux 2.1 Update 4              | 2.1.4    | http://redhat.com/rhel/2.1.4            
 rhel2.1.5            | Red Hat Enterprise Linux 2.1 Update 5              | 2.1.5    | http://redhat.com/rhel/2.1.5            
 rhel2.1.6            | Red Hat Enterprise Linux 2.1 Update 6              | 2.1.6    | http://redhat.com/rhel/2.1.6            
 rhel2.1.7            | Red Hat Enterprise Linux 2.1 Update 7              | 2.1.7    | http://redhat.com/rhel/2.1.7            
 rhel3                | Red Hat Enterprise Linux 3                         | 3        | http://redhat.com/rhel/3                
 rhel3.1              | Red Hat Enterprise Linux 3 Update 1                | 3.1      | http://redhat.com/rhel/3.1              
 rhel3.2              | Red Hat Enterprise Linux 3 Update 2                | 3.2      | http://redhat.com/rhel/3.2              
 rhel3.3              | Red Hat Enterprise Linux 3 Update 3                | 3.3      | http://redhat.com/rhel/3.3              
 rhel3.4              | Red Hat Enterprise Linux 3 Update 4                | 3.4      | http://redhat.com/rhel/3.4              
 rhel3.5              | Red Hat Enterprise Linux 3 Update 5                | 3.5      | http://redhat.com/rhel/3.5              
 rhel3.6              | Red Hat Enterprise Linux 3 Update 6                | 3.6      | http://redhat.com/rhel/3.6              
 rhel3.7              | Red Hat Enterprise Linux 3 Update 7                | 3.7      | http://redhat.com/rhel/3.7              
 rhel3.8              | Red Hat Enterprise Linux 3 Update 8                | 3.8      | http://redhat.com/rhel/3.8              
 rhel3.9              | Red Hat Enterprise Linux 3 Update 9                | 3.9      | http://redhat.com/rhel/3.9              
 rhel4.0              | Red Hat Enterprise Linux 4.0                       | 4.0      | http://redhat.com/rhel/4.0              
 rhel4.1              | Red Hat Enterprise Linux 4.1                       | 4.1      | http://redhat.com/rhel/4.1              
 rhel4.2              | Red Hat Enterprise Linux 4.2                       | 4.2      | http://redhat.com/rhel/4.2              
 rhel4.3              | Red Hat Enterprise Linux 4.3                       | 4.3      | http://redhat.com/rhel/4.3              
 rhel4.4              | Red Hat Enterprise Linux 4.4                       | 4.4      | http://redhat.com/rhel/4.4              
 rhel4.5              | Red Hat Enterprise Linux 4.5                       | 4.5      | http://redhat.com/rhel/4.5              
 rhel4.6              | Red Hat Enterprise Linux 4.6                       | 4.6      | http://redhat.com/rhel/4.6              
 rhel4.7              | Red Hat Enterprise Linux 4.7                       | 4.7      | http://redhat.com/rhel/4.7              
 rhel4.8              | Red Hat Enterprise Linux 4.8                       | 4.8      | http://redhat.com/rhel/4.8              
 rhel4.9              | Red Hat Enterprise Linux 4.9                       | 4.9      | http://redhat.com/rhel/4.9              
 rhel5.0              | Red Hat Enterprise Linux 5.0                       | 5.0      | http://redhat.com/rhel/5.0              
 rhel5.1              | Red Hat Enterprise Linux 5.1                       | 5.1      | http://redhat.com/rhel/5.1              
 rhel5.10             | Red Hat Enterprise Linux 5.10                      | 5.10     | http://redhat.com/rhel/5.10             
 rhel5.11             | Red Hat Enterprise Linux 5.11                      | 5.11     | http://redhat.com/rhel/5.11             
 rhel5.2              | Red Hat Enterprise Linux 5.2                       | 5.2      | http://redhat.com/rhel/5.2              
 rhel5.3              | Red Hat Enterprise Linux 5.3                       | 5.3      | http://redhat.com/rhel/5.3              
 rhel5.4              | Red Hat Enterprise Linux 5.4                       | 5.4      | http://redhat.com/rhel/5.4              
 rhel5.5              | Red Hat Enterprise Linux 5.5                       | 5.5      | http://redhat.com/rhel/5.5              
 rhel5.6              | Red Hat Enterprise Linux 5.6                       | 5.6      | http://redhat.com/rhel/5.6              
 rhel5.7              | Red Hat Enterprise Linux 5.7                       | 5.7      | http://redhat.com/rhel/5.7              
 rhel5.8              | Red Hat Enterprise Linux 5.8                       | 5.8      | http://redhat.com/rhel/5.8              
 rhel5.9              | Red Hat Enterprise Linux 5.9                       | 5.9      | http://redhat.com/rhel/5.9              
 rhel6.0              | Red Hat Enterprise Linux 6.0                       | 6.0      | http://redhat.com/rhel/6.0              
 rhel6.1              | Red Hat Enterprise Linux 6.1                       | 6.1      | http://redhat.com/rhel/6.1              
 rhel6.2              | Red Hat Enterprise Linux 6.2                       | 6.2      | http://redhat.com/rhel/6.2              
 rhel6.3              | Red Hat Enterprise Linux 6.3                       | 6.3      | http://redhat.com/rhel/6.3              
 rhel6.4              | Red Hat Enterprise Linux 6.4                       | 6.4      | http://redhat.com/rhel/6.4              
 rhel6.5              | Red Hat Enterprise Linux 6.5                       | 6.5      | http://redhat.com/rhel/6.5              
 rhel6.6              | Red Hat Enterprise Linux 6.6                       | 6.6      | http://redhat.com/rhel/6.6              
 rhel6.7              | Red Hat Enterprise Linux 6.7                       | 6.7      | http://redhat.com/rhel/6.7              
 rhel6.8              | Red Hat Enterprise Linux 6.8                       | 6.8      | http://redhat.com/rhel/6.8              
 rhel6.9              | Red Hat Enterprise Linux 6.9                       | 6.9      | http://redhat.com/rhel/6.9              
 rhel7.0              | Red Hat Enterprise Linux 7.0                       | 7.0      | http://redhat.com/rhel/7.0              
 rhel7.1              | Red Hat Enterprise Linux 7.1                       | 7.1      | http://redhat.com/rhel/7.1              
 rhel7.2              | Red Hat Enterprise Linux 7.2                       | 7.2      | http://redhat.com/rhel/7.2              
 rhel7.3              | Red Hat Enterprise Linux 7.3                       | 7.3      | http://redhat.com/rhel/7.3              
 rhel7.4              | Red Hat Enterprise Linux 7.4                       | 7.4      | http://redhat.com/rhel/7.4              
 rhl1.0               | Red Hat Linux 1.0                                  | 1.0      | http://redhat.com/rhl/1.0               
 rhl1.1               | Red Hat Linux 1.1                                  | 1.1      | http://redhat.com/rhl/1.1               
 rhl2.0               | Red Hat Linux 2.0                                  | 2.0      | http://redhat.com/rhl/2.0               
 rhl2.1               | Red Hat Linux 2.1                                  | 2.1      | http://redhat.com/rhl/2.1               
 rhl3.0.3             | Red Hat Linux 3.0.3                                | 3.0.3    | http://redhat.com/rhl/3.0.3             
 rhl4.0               | Red Hat Linux 4.0                                  | 4.0      | http://redhat.com/rhl/4.0               
 rhl4.1               | Red Hat Linux 4.1                                  | 4.1      | http://redhat.com/rhl/4.1               
 rhl4.2               | Red Hat Linux 4.2                                  | 4.2      | http://redhat.com/rhl/4.2               
 rhl5.0               | Red Hat Linux 5.0                                  | 5.0      | http://redhat.com/rhl/5.0               
 rhl5.1               | Red Hat Linux 5.1                                  | 5.1      | http://redhat.com/rhl/5.1               
 rhl5.2               | Red Hat Linux 5.2                                  | 5.2      | http://redhat.com/rhl/5.2               
 rhl6.0               | Red Hat Linux 6.0                                  | 6.0      | http://redhat.com/rhl/6.0               
 rhl6.1               | Red Hat Linux 6.1                                  | 6.1      | http://redhat.com/rhl/6.1               
 rhl6.2               | Red Hat Linux 6.2                                  | 6.2      | http://redhat.com/rhl/6.2               
 rhl7                 | Red Hat Linux 7                                    | 7        | http://redhat.com/rhl/7                 
 rhl7.1               | Red Hat Linux 7.1                                  | 7.1      | http://redhat.com/rhl/7.1               
 rhl7.2               | Red Hat Linux 7.2                                  | 7.2      | http://redhat.com/rhl/7.2               
 rhl7.3               | Red Hat Linux 7.3                                  | 7.3      | http://redhat.com/rhl/7.3               
 rhl8.0               | Red Hat Linux 8.0                                  | 8.0      | http://redhat.com/rhl/8.0               
 rhl9                 | Red Hat Linux 9                                    | 9        | http://redhat.com/rhl/9                 
 sled10               | SUSE Linux Enterprise Desktop 10                   | 10       | http://suse.com/sled/10                 
 sled10sp1            | SUSE Linux Enterprise Desktop 10 SP1               | 10.1     | http://suse.com/sled/10.1               
 sled10sp2            | SUSE Linux Enterprise Desktop 10 SP2               | 10.2     | http://suse.com/sled/10.2               
 sled10sp3            | SUSE Linux Enterprise Desktop 10 SP3               | 10.3     | http://suse.com/sled/10.3               
 sled10sp4            | SUSE Linux Enterprise Desktop 10 SP4               | 10.4     | http://suse.com/sled/10.4               
 sled11               | SUSE Linux Enterprise Desktop 11                   | 11       | http://suse.com/sled/11                 
 sled11sp1            | SUSE Linux Enterprise Desktop 11 SP1               | 11.1     | http://suse.com/sled/11.1               
 sled11sp2            | SUSE Linux Enterprise Desktop 11 SP2               | 11.2     | http://suse.com/sled/11.2               
 sled11sp3            | SUSE Linux Enterprise Desktop 11 SP3               | 11.3     | http://suse.com/sled/11.3               
 sled11sp4            | SUSE Linux Enterprise Desktop 11 SP4               | 11.4     | http://suse.com/sled/11.4               
 sled12               | SUSE Linux Enterprise Desktop 12                   | 12       | http://suse.com/sled/12                 
 sled12sp1            | SUSE Linux Enterprise Desktop 12 SP1               | 12.1     | http://suse.com/sled/12.1               
 sled12sp2            | SUSE Linux Enterprise Desktop 12 SP2               | 12.2     | http://suse.com/sled/12.2               
 sled9                | SUSE Linux Enterprise Desktop 9                    | 9        | http://suse.com/sled/9                  
 sles10               | SUSE Linux Enterprise Server 10            
/etc/bind/masters/elinvent.com        | 10       | http://suse.com/sles/10                 
 sles10sp1            | SUSE Linux Enterprise Server 10 SP1                | 10.1     | http://suse.com/sles/10.1               
 sles10sp2            | SUSE Linux Enterprise Server 10 SP2                | 10.2     | http://suse.com/sles/10.2               
 sles10sp3            | SUSE Linux Enterprise Server 10 SP3                | 10.3     | http://suse.com/sles/10.3               
 sles10sp4            | SUSE Linux Enterprise Server 10 SP4                | 10.4     | http://suse.com/sles/10.4               
 sles11               | SUSE Linux Enterprise Server 11                    | 11       | http://suse.com/sles/11                 
 sles11sp1            | SUSE Linux Enterprise Server 11 SP1                | 11.1     | http://suse.com/sles/11.1               
 sles11sp2            | SUSE Linux Enterprise Server 11 SP2                | 11.2     | http://suse.com/sles/11.2               
 sles11sp3            | SUSE Linux Enterprise Server 11 SP3                | 11.3     | http://suse.com/sles/11.3               
 sles11sp4            | SUSE Linux Enterprise Server 11 SP4                | 11.4     | http://suse.com/sles/11.4               
 sles12               | SUSE Linux Enterprise Server 12                    | 12       | http://suse.com/sles/12                 
 sles12sp1            | SUSE Linux Enterprise Server 12 SP1                | 12.1     | http://suse.com/sles/12.1               
 sles12sp2            | SUSE Linux Enterprise Server 12 SP2                | 12.2     | http://suse.com/sles/12.2               
 sles9                | SUSE Linux Enterprise Server 9                     | 9        | http://suse.com/sles/9                  
 solaris10            | Solaris 10                                         | 10       | http://sun.com/solaris/10               
 solaris11            | Oracle Solaris 11                                  | 11       | http://oracle.com/solaris/11            
 solaris9             | Solaris 9                                          | 9        | http://sun.com/solaris/9                
 ubuntu10.04          | Ubuntu 10.04 LTS                                   | 10.04    | http://ubuntu.com/ubuntu/10.04          
 ubuntu10.10          | Ubuntu 10.10                                       | 10.10    | http://ubuntu.com/ubuntu/10.10          
 ubuntu11.04          | Ubuntu 11.04                                       | 11.04    | http://ubuntu.com/ubuntu/11.04          
 ubuntu11.10          | Ubuntu 11.10                                       | 11.10    | http://ubuntu.com/ubuntu/11.10          
 ubuntu12.04          | Ubuntu 12.04 LTS                                   | 12.04    | http://ubuntu.com/ubuntu/12.04          
 ubuntu12.10          | Ubuntu 12.10                                       | 12.10    | http://ubuntu.com/ubuntu/12.10          
 ubuntu13.04          | Ubuntu 13.04                                       | 13.04    | http://ubuntu.com/ubuntu/13.04          
 ubuntu13.10          | Ubuntu 13.10                                       | 13.10    | http://ubuntu.com/ubuntu/13.10          
 ubuntu14.04          | Ubuntu 14.04 LTS                                   | 14.04    | http://ubuntu.com/ubuntu/14.04          
 ubuntu14.10          | Ubuntu 14.10                                       | 14.10    | http://ubuntu.com/ubuntu/14.10          
 ubuntu15.04          | Ubuntu 15.04                                       | 15.04    | http://ubuntu.com/ubuntu/15.04          
 ubuntu15.10          | Ubuntu 15.10                                       | 15.10    | http://ubuntu.com/ubuntu/15.10          
 ubuntu16.04          | Ubuntu 16.04                                       | 16.04    | http://ubuntu.com/ubuntu/16.04          
 ubuntu16.10          | Ubuntu 16.10                                       | 16.10    | http://ubuntu.com/ubuntu/16.10          
 ubuntu17.04          | Ubuntu 17.04                                       | 17.04    | http://ubuntu.com/ubuntu/17.04          
 ubuntu17.10          | Ubuntu 17.10                                       | 17.10    | http://ubuntu.com/ubuntu/17.10          
 ubuntu4.10           | Ubuntu 4.10                                        | 4.10     | http://ubuntu.com/ubuntu/4.10           
 ubuntu5.04           | Ubuntu 5.04                                        | 5.04     | http://ubuntu.com/ubuntu/5.04           
 ubuntu5.10           | Ubuntu 5.10                                        | 5.10     | http://ubuntu.com/ubuntu/5.10           
 ubuntu6.06           | Ubuntu 6.06 LTS                                    | 6.06     | http://ubuntu.com/ubuntu/6.06           
 ubuntu6.10           | Ubuntu 6.10                                        | 6.10     | http://ubuntu.com/ubuntu/6.10           
 ubuntu7.04           | Ubuntu 7.04                                        | 7.04     | http://ubuntu.com/ubuntu/7.04           
 ubuntu7.10           | Ubuntu 7.10                                        | 7.10     | http://ubuntu.com/ubuntu/7.10           
 ubuntu8.04           | Ubuntu 8.04 LTS                                    | 8.04     | http://ubuntu.com/ubuntu/8.04           
 ubuntu8.10           | Ubuntu 8.10                                        | 8.10     | http://ubuntu.com/ubuntu/8.10           
 ubuntu9.04           | Ubuntu 9.04                                        | 9.04     | http://ubuntu.com/ubuntu/9.04           
 ubuntu9.10           | Ubuntu 9.10                                        | 9.10     | http://ubuntu.com/ubuntu/9.10           
 win1.0               | Microsoft Windows 1.0                              | 1.0      | http://microsoft.com/win/1.0            
 win10                | Microsoft Windows 10                               | 10.0     | http://microsoft.com/win/10             
 win2.0               | Microsoft Windows 2.0                              | 2.0      | http://microsoft.com/win/2.0            
 win2.1               | Microsoft Windows 2.1                              | 2.1      | http://microsoft.com/win/2.1            
 win2k                | Microsoft Windows 2000                             | 5.0      | http://microsoft.com/win/2k             
 win2k12              | Microsoft Windows Server 2012                      | 6.3      | http://microsoft.com/win/2k12           
 win2k12r2            | Microsoft Windows Server 2012 R2                   | 6.3      | http://microsoft.com/win/2k12r2         
 win2k3               | Microsoft Windows Server 2003                      | 5.2      | http://microsoft.com/win/2k3            
 win2k3r2             | Microsoft Windows Server 2003 R2                   | 5.2      | http://microsoft.com/win/2k3r2          
 win2k8               | Microsoft Windows Server 2008                      | 6.0      | http://microsoft.com/win/2k8            
 win2k8r2             | Microsoft Windows Server 2008 R2                   | 6.1      | http://microsoft.com/win/2k8r2          
 win3.1               | Microsoft Windows 3.1                              | 3.1      | http://microsoft.com/win/3.1            
 win7                 | Microsoft Windows 7                                | 6.1      | http://microsoft.com/win/7              
 win8                 | Microsoft Windows 8                                | 6.2      | http://microsoft.com/win/8              
 win8.1               | Microsoft Windows 8.1                              | 6.3      | http://microsoft.com/win/8.1            
 win95                | Microsoft Windows 95                               | 4.0      | http://microsoft.com/win/95             
 win98                | Microsoft Windows 98                               | 4.1      | http://microsoft.com/win/98             
 winme                | Microsoft Windows Millennium Edition               | 4.9      | http://microsoft.com/win/me             
 winnt3.1             | Microsoft Windows NT Server 3.1                    | 3.1      | http://microsoft.com/winnt/3.1          
 winnt3.5             | Microsoft Windows NT Server 3.5                    | 3.5      | http://microsoft.com/winnt/3.5          
 winnt3.51            | Microsoft Windows NT Server 3.51                   | 3.51     | http://microsoft.com/winnt/3.51         
 winnt4.0             | Microsoft Windows NT Server 4.0                    | 4.0      | http://microsoft.com/winnt/4.0          
 winvista             | Microsoft Windows Vista                            | 6.0      | http://microsoft.com/win/vista          
 winxp                | Microsoft Windows XP                               | 5.1      | http://microsoft.com/win/xp  

 

9. Start / Stop listed KVM Virtual Machine

 

root@jeremiah:~# virsh list –all
 Id    Name                           State
—————————————————-
 3     fedora-28                      running
 –     debian9                        shut off

 

To start debian9 linux virtual machine that is currently off

 

root@jeremiah:~# virsh start fedora-28
Domain fedora-28 started

 

root@jeremiah:/home/hipo# virsh start debian9
error: Failed to start domain debian9
error: Requested operation is not valid: network 'default' is not active

root@jeremiah:/home/hipo# virsh net-list –all
Name                 State      Autostart     Persistent
———————————————————-
br0                  active     yes           yes
default              inactive   no            yes

 

root@jeremiah:/home/hipo# virsh net-start default
Network default started

root@jeremiah:/home/hipo# virsh start debian9
Domain debian9 started

 

10. Attach to running VM with virsh or virt-manager

 

root@jeremiah:~# virsh list
 Id    Name                           State
—————————————————-
 1     fedora-28                      running
 3     debian9                        running

root@jeremiah:~# virsh connect debian9

 


Note that to make the login prompt appear you have to press enter once after the ^] connection string appears


kvm-connect-to-virtual-machine-with-virsh-command-screenshot-howto

An alternative way is to use virt-manager GUI KVM desktop management interface and click over the Virtual Machine Guest name, in same fashion like in VirtualBox.

virtual-manager-virt-manager-screenshot-with-Virtual-Machines-inside-on-Debian-Linux

virt-manager-gui-interface-connect-to-fedora-28-virtual-machine

If you have KVM running on your Linux desktop PC / notebook you can also connect via VNC with virsh command.

 

root@jericho:~# virsh vncdisplay centos7


Another handy thing is to expose the Virtualized Guest OS with VNC in order to be able to connect and manage installation or further Linux configuration via VNC using an SSH Tunnel with port forwarding:

 

$ ssh hipo@www.pc-freak.net -L 5901:127.0.0.1:5901

 

11.  Start / Shutdown / Suspend / Reboot (safe reboot) a VM guest machine domain

 

 

root@jericho:~# virsh shutdown debian9
root@jericho:~# virsh start fedora-28
root@jericho:~# virsh suspend debian9
root@jericho:~# virsh reboot fedora-28

 

12. Remove / Delete KVM Virtual Machines domain

 

root@jeremiah:~# virsh undefine fedora-28
root@jeremiah:~# virsh destroy fedora-28


Closing words


Using KVM to experiment with different OS distributions is really fun just like you can easily run a number of the major most popular Linux Distributions and a set of different versions. It takes few minutes to have a fully functional Linux to play with and it saves a lot of hassles when dealing with GNU / Linux and FreeBSD, doing so in Virtualbox for me prooved to be much more complicated (not to mention that often Virtualbox had an ugly bugs so even Importing an Appliance as a Guest VM with an official distro OS-es failed with weird errors.
One other very practical use of Kerkel-based Virtualization is if you want to run your servers using own Micro-Services architecture (e.g. run multiple Linux OS-es each running a separate Apache / Nginx / MySQL / PostGreSQL / Backup / Storage) etc. all of it running on a single dedicated server or a self-hosted bare-metal
There are plenty of Web Interfaces for Management KVM (proprietary and free software) that could even futher simplify the use and deploy / destory of KVM VMs.
All that makes possible running your own Linux or Web hosting provider a relatively easy task and seriously could cut business expenses and operational (maintenance) costs.

If you plan to run youw own hosting company, I can help you establish your infrastructure and advise you on the right technologies to use.

 

Virtualbox Shared folder set up on Linux between Host and Guest OS – Set up Virtualbox shared folder to Copy files from PC Host to Guest

Wednesday, September 12th, 2018

mount-shares-between-host-OS-and-guest-virtual-machine-howto-virtualbox-vbox-logo

How to set-up Virtualbox shared folder to Copy files from PC Host  and Guest Virtualized OS?

Running VirtualBox Host is an easy thing to set-up across all Operating Systems.  Once you have it sooner or later you will need to copy files from the VM Host OS (that in my case is GNU / Linux) to the virtualized Guest operating system (again in my case that's again another Linux ISO running indide the Virtual Machine).

Below are steps to follow To use Virtualbox Shared Folder functionality to copy files between VBox and your Desktop / server Linux install.

1. Install Virtualbox Guest Additions CD Image ISO

I've explained how to add the Guest Additions CD image thoroughfully in my previous article Howto enable Copy / Paste Virtualbox betwen Linux guest and Host OS
Anyways I'll repeat myself below for sake of clarity:

To do so use Oracle VBox menus (on the booted virtualized OS VBox window):

 

Devices -> Insert Guest additions CD Image

 

Mount the ISO inside the Linux Virtual Machine:

root@debian:~# mount /media/cdrom1/
 

If the mount fails and there are no files inside the mount point it might be because the virtualbox-dkms and virtualbox-guest-dkms packages might be missing on the Host OS.

To install them (on Debian GNU / Linux) assuming that you're using virtualbox default distro packages /etc/apt/sources.list :
 

apt-get install –yes -qq virtualbox-dkms virtualbox-guest-dkms


and run:

 

root@debian:/media/cdrom1# cd /media/cdrom1; sh VBoxLinuxAdditions.run


2. Create directory for Shared Folder that will be used to access Host / OS files from the Guest Virtualized OS
 

root@debian:~# mkdir /mnt/shared_folder

 

3. Map from VBox program interface Shared folder settings and Mount /mnt/shared_folder location

virtualbox-virtual-machine-devices-shared-folders-shared-folder-settings-linux-screenshot

 

Devices -> Shared Folder -> Shared Folder settings -> Transient Folders (click blue folder add small button right)

 

From Transient Folders add whatever directory you want to be shared from your local notebook / PC to the VM.

virtualbox-devices-Shared-Folder-Add-Shared-Folder-add-share-linux-screenshotDepending on whether you would like to mount the shared folder only for reading files (choose Read Only) to make it a permanent shared folder (and not just for the one session of current running Virtual Machine until its killed use Make Permanent) or check Auto-Mount tick if you want the shared_folder mapping to be mounted on every VM boot.

Once the shared_folder directory location is set-up from GUI menu click OK and in order for the settings to take effect, you'll need to restart the VM Guest with Linux (use halt command from terminal) or Power Off the Machine via the VBox menus.

To mount use command like:

mount -t vboxsf name_of_folder_linked_from_vbox  /mnt/name_folder_guest_os/


mount-vboxsf-shared-folder-mnt-shared-linux-guest-screenshot

In my case I wanted to share home folder /home so the command I used is:

root@debian:~# mount -t vboxsf  shared_folder /mnt/shared_folder


If everything is fine your Host OS file content from /home will be visible (for read and write if you Mapped it so) 
under /mnt/shared_folder …

And as Turtles Ninja used to heavily say Cowabunga !!! 🙂
You have it mounted and ready for file share between Desktop -> Virtualized OS.

 

Bear in mind that above mount command has to run as root (superuser) to succeed.

You now could copy files from your Host OS (running the Virtual Machine) and the Guest OS (Virtualized OS) using /mnt/shared_folder mount point without problems.

The example is if you want to share files between VirtualBox installed Linux and the Guest (Desktop / server) OS, however at many cases mounting your Host OS directory for root users might be not very practical but, instead you might prefer to do the mount for specific non admin user, for example I prefer to do the shared folder mount with my pointed non-root username hipo.

Here is how to do above VM shared_folder mount for non-root user:

First you need to know the exact UID / GID (User ID / Group ID) of user, you can get that with id command:

 

hipo@linux:~$  id
uid=1000(hipo) gid=1000(hipo) groups=1000(hipo),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),108(netdev),114(bluetooth),115(lpadmin),119(scanner)

 

As you see UID / GID in my case are 1000 / 1000

hipo@linux:~$ sudo mount -t vboxsf -o rw,uid=1000,gid=1000 shared_folder /mnt/shared_folder

 

mount-virtual-box-shared_folder-with-non-administration-permissions-non-root-permissions-id-and-mount-command-screenshot-linux


4. Mounting configured shared_folder to automatically mount into the Guest OS Linux on every boot

a) Configuring shared_folder auto-mount using /etc/rc.local

If you need the shared_folder to automatically mount next-time you boot the virtual machine quickest way is to add the mount command to /etc/rc.local (on Debian 8 and Debian 9 and newer Ubuntu Linuxes rc.local is missing by default to enable it to work like it worked before read follow my previous article ).

b) Configuring auto-mount for shared_folder through /etc/fstab

The more professional way to auto-mount on emulated OS VM boot time,  you could add the vboxsf mount definitions to /etc/fstab with your favourite text-editor mcedit, nano, joe etc. … (for me that's vim).

Syntax of /etc/fstab is as follows:
 

<Device> <Mount Point> <Type> <Options> <Dump> <Pass>

root@linux:~# vim /etc/fstab

 

shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0 0

Note that you will want to change 1000 / 1000, id / gid with the ones of the non-admin user you would like to add to mount it for.

A quick way to add it to /etc/fstab with a shell one-liner is with command
 

root@linux:~# echo 'shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0  0' >> /etc/fstab

An alternative way to add a user to have permissions for vboxsf file system (without specifying the long -o uid=1000,gid=1000 options is to simply add the username in question to group vboxsf like so:

c) Adding non super user username to vboxsf group

root@linux:~# usermod -G vboxsf hipo
root@linux:~# grep -i vboxsf /etc/group
vboxsf:x:999:hipo

 

hipo@linux:~$ sudo mount -t vboxsf  shared_folder /mnt/shared_folder

 

without the extra arguments and the options to pass to /etc/fstab (for eventual requirement to auto mount the shared_folder) would be more simple e.g.:

 

echo 'shared_folder /mnt/shared_folder                                vboxsf ' >> /etc/fstab

 

One note to make here is if the uesr is added to vboxsf the line for /etc/fstab to auto mount to mount for root user and non-root will be identical.

Then you can get the /etc/fstab auto-mount configured tested by running:

c) Checking auto-mount is working

hipo@linux:~# mount -a
hipo@linux:~# mount |grep -i vboxsf
shared_folder on /mnt/shared_folder type vboxsf (rw,nodev,relatime)


5. What if you end up with mounting failed errors ? – What might be causing the mounting failed Protocol error (a few things to check to solve)


In case of troubles with the mount you might get an error like:

hipo@linux:~# mount -t vboxsf  share_folder /mnt/shared_folder

/sbin/mount.vboxsf: mounting failed with the error: Protocol error


This error might be caused because of Insert Guest Additions CD Image might be not properly enabled and installed using the ISO provided VBoxLinuxAdditions.sh shell script.
Other common reason you might get this error if you have mistyped the Folder name: given in Shared Folders -> Folder Path -> Add Share for example I have given shared_folder as a Map name but as you can see in above mount -t vboxsf, I've mistyped share_folder instead of the correct one shared_folder inserted.
In some VBox releases this error was caused by bugs in the Virtual Machine.
 

virtualbox-virtual-machine-shared-folder-transient-folder-add-folder-linux-VM-guest-linux

One useful tip is to be able to check whether a Virtualbox Virtual Machine has a configured shared_folder (if you're logging to manage the machine on remote server – nomatter whether you have logged in with VNC / Teamviewer / Citrix etc. or via SSH session.

To do so use VBoxControl as of time of writting usually located on most distributions under (/usr/bin/VBoxControl)
 

 

hipo@linux:~# VBoxControl sharedfolder list -automount
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Auto-mounted Shared Folder mappings (0):

No Shared Folders available.

You can use VBoxControl command to get set and list a number of settings on the VBox VM, here is an useful example with it where you get information about numerous VBox info values:

 

root@linux:~# VBoxControl guestproperty enumerate
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Name: /VirtualBox/GuestInfo/OS/Product, value: Linux, timestamp: 1536681633430852000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/V4/IP, value: 10.0.2.15, timestamp: 1536681633438717000, flags: <NULL>
Name: /VirtualBox/HostInfo/GUI/LanguageID, value: en_US, timestamp: 1536697521395621000, flags: RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/MAC, value: 08002762FA1C, timestamp: 1536681633442120000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/ServicePack, value: <NULL>, timestamp: 1536681633431259000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVerExt, value: 5.2.18, timestamp: 1536681619002646000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Netmask, value: 255.255.255.0, timestamp: 1536681633440157000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Version, value: #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13), timestamp: 1536681633431125000, flags: <NULL>
Name: /VirtualBox/GuestAdd/VersionExt, value: 5.2.18, timestamp: 1536681633431582000, flags: <NULL>
Name: /VirtualBox/GuestAdd/Revision, value: 124319, timestamp: 1536681633432515000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepExec, value: <NULL>, timestamp: 1536681619002355000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsers, value: 1, timestamp: 1536681673447293000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/0/Status, value: Up, timestamp: 1536681633443911000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/Name, value: enp0s3, timestamp: 1536681633445302000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepArgs, value: <NULL>, timestamp: 1536681619002387000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestAdd/Version, value: 5.2.18, timestamp: 1536681633431419000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxRev, value: 124319, timestamp: 1536681619002668000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Broadcast, value: 10.0.2.255, timestamp: 1536681633439531000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVer, value: 5.2.18, timestamp: 1536681619002613000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsersList, value: hipo, timestamp: 1536681673446498000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/Count, value: 1, timestamp: 1536698949773993000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Release, value: 4.9.0-7-amd64, timestamp: 1536681633431001000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/NoLoggedInUsers, value: false, timestamp: 1536681673447965000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestAdd/HostVerLastChecked, value: 5.2.18, timestamp: 1536681702832389000, flags: <NULL>

Hope you enjoyed ! Have phun! 🙂

Copy Paste Virtualbox enable between Linux Host and Guest Virtual Machine

Thursday, September 6th, 2018

enable-copy-paste-between-linux-host-and-guest-OS-virtual-machine-virtualbox-vbox-logo

Copy / Pasting has become a vital part of our digital lifes, not having it in our days is insane, most of our System Administration jobs / Programmer or Office (Secretary) stuff daily without Copy / Paste is a unthinkable.

Lately I have been playing with Virtual Machines as I need to test some Dev Ops related stuff I install Jenkins write and test scripts and tools inside a VM, for that to happen I needed a working VM and because I am a Free Software supporter my VM of choice is usually Virtualbox. Even though Virtualbox is not the best and most performance efficient Virtual Machine out there for a small tasks and home usage dev / testing it does a great job, I have faced a problem with Copy / Pasting not working in VBox between the VM -> Guest OS, that's why I decided to write this step by step guide to help people who face the same issue.

For people who are starting with Oracle VirtualBox and you just have Created a New Virtual Machine and installed GNU / Linux or FreeBSD one common set back problem you will experience is the the Copy / Paste functionality between the VM Host Machine and the Guest won't be working out of the box. That's pretty much annoying thing and since usually one installs different Virtual Machines with different Operating Systems as a test bed for Software installs on top of various versions of OSes the lack of Copy, Paste functionality which is not there by default makes copy / pasting programming code / commands etc. between your local Desktop notebook or remote  server Machine and the VBox Virtualized Guest mostly unusuable …

Besides that integration (sharing resources / improved VM performance) between the Host (which is for example your Desktop notebook or PC) and the Virtualized Operating system be it some kind of UNIX (FreeBSD) etc. is not optimal until you enable an extra integration between VirtualBox and your (Host OS – that in my case is Debian 9 GNU / Linux but it can be Microsoft Windows XX as well or any other *nix like OS).In order for this Integration between your Host OS and Guest OS and Copy / Paste features between the two to work you need to Enable Guest Additions CD image (Software) that is part of VirtualBox installation on the Host OS and once enabled to mount (Attach the Vbox provided Virtual CD) from within the Guest OS (that in my case is another Debian 9 Stretch Linux that is installed inside the VM) and run a script that will install the necessery VirtualBox software inside the Guest OS Linux.
 

This is done in 3 easy steps:

STEP 1: Enable Vbox (Guest Additions CD Image)


virtualbox-VM-enable-insert-guest-additions-CD-image-menu-screenshot-debian-linux

 

Devices -> Insert Guest Additions CD image

 


2. Enable Shared Clipboard and Drag and Drop from the Virtual Machine

virtualbox-VM-enable-devices-shared-slipboard-bidirectional-menu-screenshot-debian-linux
 

Devices -> Shared Clipboard -> Bidirectional

 

 

virtualbox-VM-enable-devices-drag-and-drop-bidirectional-menu-screenshot-debian-linux
 

Devices -> Drag and Drop -> Bidirectional


3. Install development packages that are required by VBoxLinuxAdditions.run script

a) First thing is to use Virtual Box program interface menu and enable Insert Guest Additions CD Image option (assuming that you have the Linux Guest OS running).

virtualbox-VM-enable-insert-guest-additions-CD-image-menu-screenshot-debian-linux
 

Devices -> Insert Guest Additions CD image


https://www.pc-freak.net/images/virtualbox-virtual-machine-guest-os-settings

You can do it also from main VirtualBox interface (without having a Virtual machine running) by clicking over the desired Guest OS or pressting CTRL + S key combination

b) Next step is to mount the Virtual CD Guest Additions inside the Guest OS VM

To mount the Virtual content provided by the just ticked VBOX "Insert Guest Additions CD" menu, inside the Guest OS Linux, you have to run simple mount command:

 

 

root@debian:~# mount /media/cdrom1
mount: /dev/sr1 is write-protected, mounting read-only

 

c) Install necessery packages that VBoxLinuxAdditions.run will use

Further on VBoxLinuxAdditions.run (additions script) that adds the integration between the Host operating system and the Guest OS depends on some essential tools necessery for building (compiling) / developing software on Linux, this are:

1.) build-essential – a meta package that will install the following set of packages:

 

dpkg-dev fakeroot g++ g++-6 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libfakeroot libfile-fcntllock-perl libstdc++-6-dev


which are necessery for creation .deb packages, compiling code and other various activitions. a) build-essential is a very important Debian based distro package and if you happen to be new to Linux or just starting learning how to develop programs on Linux, you will soon realize you will need it installed across almost all Linux OS hosts you maintain or develop on …

VirtualBox-VBoxLinuxAdditions-run-script-missing-headers-gcc-perl-how-to-fix

 

root@debian:~# apt-get install –yes build-essential linux-headers-$(uname -r) gcc perl make

 


2.) The linux-headers package in my case that it is is linux-headers-4.9.0-7-amd64 is important as it provides kernel header files that are necessery when compiling and integrating kernel modules VBoxLinuxAdditions.run script depends on it as it builds a kernel module that adds the integration (that enables Copy / Paste and performance inprovements) between the Host VM machine and Guest OS

3.) gcc perl packages this two provides the C compiler and perl scripting programming language interpreter which are also used by VBoxLinuxAdditions.run script.


Now run VBoxLinuxAdditions.run:

As you see in above screenshot the

root@debian:~#
root@debian:/media/cdrom1# sh VBoxLinuxAdditions.run
Verifying archive integrity… All good.
Uncompressing VirtualBox 5.2.18 Guest Additions for Linux……..
VirtualBox Guest Additions installer
Removing installed version 5.2.18 of VirtualBox Guest Additions…
Copying additional installer modules …
Installing additional modules …
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.  This may take a while.
VirtualBox Guest Additions: Running kernel modules will not be replaced until the system is restarted
VirtualBox Guest Additions: Starting.

 


If everything is fine you should get command output like above.

Above are the steps given to follow on current latest Debian Linux 9.5 (Stretch) as a Guest OS (this tutorial should be working also on Ubuntu / Mint and other deb package based distributions for RPM package based Linuxes Fedora / CentOS check out my previous toturial how to enable Linux Guest Virtualbox Addition on Redhat based distos.)

P. S.

What about Mac OS X since it is based on FreeBSD and it includes a lot of GNU licensing programs?

If you wonder whether on Mac OS X you don't need to run guest additions script such as (VBoxLinuxAdditions.run) on Mac Host -> Guest OS integration is done only by enabling the feature from VBox GUI interface via:
 

Devices -> Insert Guest Additions CD Image

What about enabling Copy / Paste on Linux Host VirtualBox hosting a Windows Guest OS?

 

 

Enabling Copy / Paste on a Windows Virtual Machine (I wrote about it earlier here), follows the same steps as in this tutorial.
The only difference is instead of running the VBoxLinuxAdditions.sh you have to navigate to the CD from Windows Explorer and run the appropriate .exe file depending on your architecture (X86 or 64 bit), the files to run on the virtual CDROM are one of the 3:

1. VBoxWindowsAdditions-amd64.exe
2. VBoxWindowsAdditions.exe
3. VBoxWindowsAdditions-x86.exe

virtualbox-vm-guest-additions-autoplay-web-on-windows-os

Follow the click through interface and you will have the Copy / Paste between your Linux -> Windows OS working.

Happy Copy / Pasting ! 🙂

Resize a VirtualBox guest Linux VDI Disk under Linux Host

Tuesday, September 4th, 2018


https://www.pc-freak.net/images/virtualbox-virtual-machine-guest-Windows-10-Linux-Debian-9-screenshot

I am using Oracle VirtualBox virtual machine to experiment with things before I deploy them on my servers. Recently I installed Debian 9 in a hurry (without much taking attention on the configuration options during VM machine creation) and the negative results of my lack of attention came quickly as my Linux Virtualbox guest OS just ran of space a little time after I have installed a Debian Stretch 9 OS into the virtual machine ..

I didn't realized immediately the "No free disk space" error was there because, after installing a few sets of Debian packages (Exim MTA – exim4-damon-heavy .. related etc.) immediately, shutdown the Virtual Machine running (even of this very moment) on my Thinkpad T420 notebook and went to bed.

This morning I switched on the PC booted my notebook and launched the VM to do another set of experiments inside it, however it seemed it failed to load all services printed in text on Linux boot and the GDM3 Greater never appearead on screen- for those who don't know GDM3 is Linux Graphical environment Authentication / Login program.

gdm3-debian-9-login-errors-low-disk-space-no-free-space-on-disk

I have tried a VirtualBox VM Restart,  hoping that the cause of my problems might be VMachine bug (as I'm running the latest stable version of VirtualBox (Virtualbox 5.2) taken from VBox Linux download page . On Next boot weirdly the OS booted until the GDM greater but inputting the password did not load into GNOME but kept repeatedly asking for password.

After another VM restart (shutdown + re-boot), I could eventually login but I got a Low disk space error like in below screenshot

www.pc-freak.net/images/debian-linux-9-running-inside-virtual-machine-low-disk-space-stroke-selection

So how to fix the Virtual Debian Install Low Disk Space?

As of today September 4 2018 the way to extend the hard disk, that was prepared originally for the Virtual Machine install you need to give a check inside your Linux home (~/) directory under Virtualbox VMs find the folder that is equivallent to the Virtual Machine (whose Hard Disk Drive) you would like to extend, do a ls command check to identify the exact name of the VDI – [Virtual Desktop Infrastructure] file (in my case, that's is Debian 9.vdi) and use VboxManage modifyhd options command like shown below to resize VM HDD to more gigabytes

1. Resize the Virtualbox VM .VDI file using command line

root@jericho:/home/hipo/VirtualBox VMs/Debian 9# VBoxManage modifyhd 'Debian 9.vdi' –resize 40000
0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%

As you see I am resizing my Hard Drive for this Virtual Machine to 40000 = (40 Gigabytes), if you have need / more disk space you might want to set something like 100000 (100GB).

 

2. Resizing .VDI file (Virtual Machine) using VirtualBox GUI

Click on Global Tools button (located leftside upper side on the screen

virtualbox-resize-debian-linux-harddrive-global-tools-screenshot

Drag the bar Size: to the Hard Drive partition size you want it to be …
 

So far so good, now the image file is resized but the GNU / Linux partition is still being small, so we need to also do repartitioning, once booted into the VM in order to stretch the HDD size to 40 Gigas.

Next install and use gparted Linux tool to add the free space to your root partition, use the gparted

Inside the VM instal the tool with:
 

Inside virtualmachine's in gnome-terminal / xterm etc. depending on the graphical environment used do install with apt-get:

 

debian:# apt-get install –yes gparted

 

debian~:# gparted


Notice that gparted has to be ran as a root superuser.

 

Run GParted and create new EXT3 filesystem that is 10GB (the size of the new created partition).

If you have installed Debian to place all partitions under / (root directory /dev/sda1) then the fresh new partition to create should be
/dev/sda3, anyways just look closely in EXT3 and in your case if the partiition is named differently create according to proper partition /dev/ naming.

I'll not run into details on how to create the partition with GParted as the program interface is very self-explanatory, the only thing is to apply the update to create partition and the ext3 filesystem, that's being done
with a green tick:

 

Next step is to check with fdisk whether, we have ext3 properly created  filesystem as we've done already with GPARTED:

Once we have the partition created with EXT3 filesystem, we're ready to move /usr temporary to other folder, I use usually /root for the move but you can create anywhere a new folder for that and move to there.

To move to /root directory run again in terminal:

 

debian:~# mv /usr /root
debian:~# mkdir /usr

 

 

 

Note that during the move operations, your Desktop icons will become without (with broken) pictures and the default Debian background picture is to disappear, that's because the GUI environment will soon realize /usr/ libraries that're periodically reloaded in memory are missing and will be unable to reload them as it does in a cycle.

That should take a few minutes, so grab a coffee or if you're a smoker (hope not as smoking kills 🙂 ), in 5 / 10 minutes time depending on your computer / server configuration, it will be over, so we're ready to create new /usr dir and mount the  new partition:

 

debian:~# mount /dev/sda3 /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux1

 

Now we check with mount command whether mount is fine:

 

check-if-filesystem-is-properly-mounted-linux-debian-screenshot


Now  /dev/sda3 is mounted under /usr  and we have to move back /root/usr directory content back to the newly mounted /usr so we run command:
 

debian:~#  mv /root/usr/* /usr/*


Finally we need to create proper records for the new partition inside /etc/fstab (fstab –FileSystem Tab file – the file which describes instructs the Linux OS what partition to boot where, what)