Archive for the ‘Linux’ Category

How to redirect TCP port traffic from Internet Public IP host to remote local LAN server, Redirect traffic for Apache Webserver, MySQL, or other TCP service to remote host

Thursday, September 23rd, 2021

 

 

Linux-redirect-forward-tcp-ip-port-traffic-from-internet-to-remote-internet-LAN-IP-server-rinetd-iptables-redir

 

 

1. Use the good old times rinetd – internet “redirection server” service


Perhaps, many people who are younger wouldn't remember rinetd's use was pretty common on old Linuxes in the age where iptables was not on the scene and its predecessor ipchains was so common.
In the raise of mass internet rinetd started loosing its popularity because the service was exposed to the outer world and due to security holes and many exploits circulating the script kiddie communities
many servers get hacked "pwned" in the jargon of the script kiddies.

rinetd is still available even in modern Linuxes and over the last years I did not heard any severe security concerns regarding it, but the old paranoia perhaps and the set to oblivion makes it still unpopular soluttion for port redirect today in year 2021.
However for a local secured DMZ lans I can tell you that its use is mostly useful and I chooes to use it myself, everynow and then due to its simplicity to configure and use.
rinetd is pretty standard among unixes and is also available in old Sun OS / Solaris and BSD-es and pretty much everything on the Unix scene.

Below is excerpt from 'man rinetd':

 

DESCRIPTION
     rinetd redirects TCP connections from one IP address and port to another. rinetd is a single-process server which handles any number of connections to the address/port pairs
     specified in the file /etc/rinetd.conf.  Since rinetd runs as a single process using nonblocking I/O, it is able to redirect a large number of connections without a severe im‐
     pact on the machine. This makes it practical to run TCP services on machines inside an IP masquerading firewall. rinetd does not redirect FTP, because FTP requires more than
     one socket.
     rinetd is typically launched at boot time, using the following syntax:      /usr/sbin/rinetd      The configuration file is found in the file /etc/rinetd.conf, unless another file is specified using the -c command line option.

To use rinetd on any LInux distro you have to install and enable it with apt or yum as usual. For example on my Debian GNU / Linux home machine to use it I had to install .deb package, enable and start it it via systemd :

 

server:~# apt install –yes rinetd

server:~#  systemctl enable rinetd


server:~#  systemctl start rinetd


server:~#  systemctl status rinetd
● rinetd.service
   Loaded: loaded (/etc/init.d/rinetd; generated)
   Active: active (running) since Tue 2021-09-21 10:48:20 EEST; 2 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 1 (limit: 4915)
   Memory: 892.0K
   CGroup: /system.slice/rinetd.service
           └─1364 /usr/sbin/rinetd


rinetd is doing the traffic redirect via a separate process daemon, in order for it to function once you have service up check daemon is up as well.

root@server:/home/hipo# ps -ef|grep -i rinet
root       359     1  0 16:10 ?        00:00:00 /usr/sbin/rinetd
root       824 26430  0 16:10 pts/0    00:00:00 grep -i rinet

+ Configuring a new port redirect with rinetd

 

Is pretty straight forward everything is handled via one single configuration – /etc/rinetd.conf

The format (syntax) of a forwarding rule is as follows:

     [bindaddress] [bindport] [connectaddress] [connectport]


Besides that rinetd , could be used as a primitive firewall substitute to iptables, general syntax of allow deny an IP address is done with (allow, deny) keywords:
 

allow 192.168.2.*
deny 192.168.2.1?


To enable logging to external file ,you'll have to include in the configuration:

# logging information
logfile /var/log/rinetd.log

Here is an example rinetd.conf configuration, redirecting tcp mysql 3306, nginx on port 80 and a second web service frontend for ILO to server reachable via port 8888 and a redirect from External IP to local IP SMTP server.

 

#
# this is the configuration file for rinetd, the internet redirection server
#
# you may specify global allow and deny rules here
# only ip addresses are matched, hostnames cannot be specified here
# the wildcards you may use are * and ?
#
# allow 192.168.2.*
# deny 192.168.2.1?


#
# forwarding rules come here
#
# you may specify allow and deny rules after a specific forwarding rule
# to apply to only that forwarding rule
#
# bindadress    bindport  connectaddress  connectport


# logging information
logfile /var/log/rinetd.log
83.228.93.76        80            192.168.0.20       80
192.168.0.2        3306            192.168.0.19        3306
83.228.93.76        443            192.168.0.20       443
# enable for access to ILO
83.228.93.76        8888            192.168.1.25 443

127.0.0.1    25    192.168.0.19    25


83.228.93.76 is my external ( Public )  IP internet address where 192.168.0.20, 192.168.0.19, 192.168.0.20 (are the DMZ-ed Lan internal IPs) with various services.

To identify the services for which rinetd is properly configured to redirect / forward traffic you can see it with netstat or the newer ss command
 

root@server:/home/hipo# netstat -tap|grep -i rinet
tcp        0      0 www.pc-freak.net:8888   0.0.0.0:*               LISTEN      13511/rinetd      
tcp        0      0 www.pc-freak.n:http-alt 0.0.0.0:*               LISTEN      21176/rinetd        
tcp        0      0 www.pc-freak.net:443   0.0.0.0:*               LISTEN      21176/rinetd      

 

+ Using rinetd to redirect External interface IP to loopback's port (127.0.0.1)

 

If you have the need to redirect an External connectable living service be it apache mysql / privoxy / squid or whatever rinetd is perhaps the tool of choice (especially since there is no way to do it with iptables.

If you want to redirect all traffic which is accessed via Linux's loopback interface (localhost) to be reaching a remote host 11.5.8.1 on TCP port 1083 and 1888, use below config

# bindadress    bindport  connectaddress  connectport
11.5.8.1        1083            127.0.0.1       1083
11.5.8.1        1888            127.0.0.1       1888

 

For a quick and dirty solution to redirect traffic rinetd is very useful, however you'll have to keep in mind that if you want to redirect traffic for tens of thousands of connections constantly originating from the internet you might end up with some disconnects as well as notice a increased use of rinetd CPU use with the incrased number of forwarded connections.

 

2. Redirect TCP / IP port using DNAT iptables firewall rules

 

Lets say you have some proxy, webservice or whatever service running on port 5900 to be redirected with iptables.
The easeiest legacy way is to simply add the redirection rules to /etc/rc.local​. In newer Linuxes rc.local so if you decide to use,
you'll have to enable rc.local , I've written earlier a short article on how to enable rc.local on newer Debian, Fedora, CentOS

 

# redirect 5900 TCP service 
sysctl -w net.ipv4.conf.all.route_localnet=1
iptables -t nat -I PREROUTING -p tcp –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -I OUTPUT -p tcp -o lo –dport 5900 -j REDIRECT –to-ports 5900
iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp –dport 5900 -j DNAT  –to-destination 192.168.1.8:5900
iptables -t nat -I OUTPUT –source 0/0 –destination 0/0 -p tcp –dport 5900 -j REDIRECT –to-ports 5900

 

Here is another two example which redirects port 2208 (which has configured a bind listener for SSH on Internal host 192.168.0.209:2208) from External Internet IP address (XXX.YYY.ZZZ.XYZ) 
 

# Port redirect for SSH to VM on openxen internal Local lan server 192.168.0.209 
-A PREROUTING  -p tcp –dport 2208 -j DNAT –to-destination 192.168.0.209:2208
-A POSTROUTING -p tcp –dst 192.168.0.209 –dport 2208 -j SNAT –to-source 83.228.93.76

 

3. Redirect TCP traffic connections with redir tool

 

If you look for an easy straight forward way to redirect TCP traffic, installing and using redir (ready compiled program) might be a good idea.


root@server:~# apt-cache show redir|grep -i desc -A5 -B5
Version: 3.2-1
Installed-Size: 60
Maintainer: Lucas Kanashiro <kanashiro@debian.org>
Architecture: amd64
Depends: libc6 (>= 2.15)
Description-en: Redirect TCP connections
 It can run under inetd or stand alone (in which case it handles multiple
 connections).  It is 8 bit clean, not limited to line mode, is small and
 light. Supports transparency, FTP redirects, http proxying, NAT and bandwidth
 limiting.
 .
 redir is all you need to redirect traffic across firewalls that authenticate
 based on an IP address etc. No need for the firewall toolkit. The
 functionality of inetd/tcpd and "redir" will allow you to do everything you
 need without screwy telnet/ftp etc gateways. (I assume you are running IP
 Masquerading of course.)

Description-md5: 2089a3403d126a5a0bcf29b22b68406d
Homepage: https://github.com/troglobit/redir
Tag: interface::daemon, network::server, network::service, role::program,
 use::proxying
Section: net
Priority: optional

 

 

server:~# apt-get install –yes redir

Here is a short description taken from its man page 'man redir'

 

DESCRIPTION
     redir redirects TCP connections coming in on a local port, [SRC]:PORT, to a specified address/port combination, [DST]:PORT.  Both the SRC and DST arguments can be left out,
     redir will then use 0.0.0.0.

     redir can be run either from inetd or as a standalone daemon.  In –inetd mode the listening SRC:PORT combo is handled by another process, usually inetd, and a connected
     socket is handed over to redir via stdin.  Hence only [DST]:PORT is required in –inetd mode.  In standalone mode redir can run either in the foreground, -n, or in the back‐
     ground, detached like a proper UNIX daemon.  This is the default.  When running in the foreground log messages are also printed to stderr, unless the -s flag is given.

     Depending on how redir was compiled, not all options may be available.

 

+ Use redir to redirect TCP traffic one time

 

Lets say you have a MySQL running on remote machine on some internal or external IP address, lets say 192.168.0.200 and you want to redirect all traffic from remote host to the machine (192.168.0.50), where you run your Apache Webserver, which you want to configure to use
as MySQL localhost TCP port 3306.

Assuming there are no irewall restrictions between Host A (192.168.0.50) and Host B (192.168.0.200) is already permitting connectivity on TCP/IP port 3306 between the two machines.

To open redirection from localhost on 192.168.0.50 -> 192.168.0.200:

 

server:~# redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306

 

If you need other third party hosts to be additionally reaching 192.168.0.200 via 192.168.0.50 TCP 3306.

root@server:~# redir –laddr=192.168.0.50 –lport=3306 –caddr=192.168.0.200 –cport=3306


Of course once you close, the /dev/tty or /dev/vty console the connection redirect will be cancelled.

 

+ Making TCP port forwarding from Host A to Host B permanent


One solution to make the redir setup rules permanent is to use –rinetd option or simply background the process, nevertheless I prefer to use instead GNU Screen.
If you don't know screen is a vVrtual Console Emulation manager with VT100/ANSI terminal emulation to so, if you don't have screen present on the host install it with whatever Linux OS package manager is present and run:

 

root@server:~#screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3306 –caddr=192.168.0.200 –cport=3306'

 

That would run it into screen session and detach so you can later connect, if you want you can make redir to also log connections via syslog with ( -s) option.

I found also useful to be able to track real time what's going on currently with the opened redirect socket by changing redir log level.

Accepted log level is:

 

  -l, –loglevel=LEVEL
             Set log level: none, err, notice, info, debug.  Default is notice.

 

root@server:/ # screen -dm bash -c 'redir –laddr=127.0.0.1 –lport=3308 –caddr=192.168.0.200 –cport=3306 -l debug'

 

To test connectivity works as expected use telnet:
 

root@server:/ # telnet localhost 3308
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
g
5.5.5-10.3.29-MariaDB-0+deb10u1-log�+c2nWG>B���o+#ly=bT^]79mysql_native_password

6#HY000Proxy header is not accepted from 192.168.0.19 Connection closed by foreign host.

once you attach to screen session with

 

root@server:/home #  screen -r

 

You will get connectivity attempt from localhost logged : .
 

redir[10640]: listening on 127.0.0.1:3306
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10640]: target is 192.168.0.200:3306
redir[10640]: Waiting for client to connect on server socket …
redir[10793]: peer IP is 127.0.0.1
redir[10793]: peer socket is 25592
redir[10793]: target IP address is 192.168.0.200
redir[10793]: target port is 3306
redir[10793]: Connecting 127.0.0.1:25592 to 127.0.0.1:3306
redir[10793]: Entering copyloop() – timeout is 0
redir[10793]: Disconnect after 1 sec, 165 bytes in, 4 bytes out

The downsides of using redir is redirection is handled by the separate process which is all time hanging in the process list, as well as the connection redirection speed of incoming connections might be about at least 30% slower to if you simply use a software (firewall ) redirect such as iptables. If you use something like kernel IP set ( ipsets ). If you hear of ipset for a first time and you wander whta it is below is short package description.

 

root@server:/root# apt-cache show ipset|grep -i description -A13 -B5
Maintainer: Debian Netfilter Packaging Team <pkg-netfilter-team@lists.alioth.debian.org>
Architecture: amd64
Provides: ipset-6.38
Depends: iptables, libc6 (>= 2.4), libipset11 (>= 6.38-1~)
Breaks: xtables-addons-common (<< 1.41~)
Description-en: administration tool for kernel IP sets
 IP sets are a framework inside the Linux 2.4.x and 2.6.x kernel which can be
 administered by the ipset(8) utility. Depending on the type, currently an
 IP set may store IP addresses, (TCP/UDP) port numbers or IP addresses with
 MAC addresses in a  way which ensures lightning speed when matching an
 entry against a set.
 .
 If you want to
 .
  * store multiple IP addresses or port numbers and match against the
    entire collection using a single iptables rule.
  * dynamically update iptables rules against IP addresses or ports without
    performance penalty.
  * express complex IP address and ports based rulesets with a single
    iptables rule and benefit from the speed of IP sets.

 .
 then IP sets may be the proper tool for you.
Description-md5: d87e199641d9d6fbb0e52a65cf412bde
Homepage: http://ipset.netfilter.org/
Tag: implemented-in::c, role::program
Section: net
Priority: optional
Filename: pool/main/i/ipset/ipset_6.38-1.2_amd64.deb
Size: 50684
MD5sum: 095760c5db23552a9ae180bd58bc8efb
SHA256: 2e2d1c3d494fe32755324bf040ffcb614cf180327736c22168b4ddf51d462522

Defining multiple short Server Hostname aliases via SSH config files and defining multiple ssh options for it, Use passwordless authentication via public keys

Thursday, September 16th, 2021

using-ssh-host-acronym-aliases-ssh-client-explained-openssh-logo

In case you have to access multiple servers from your terminal client such as gnome-terminal, kterminal (if on Linux) or something such as mobaxterm + cygwin (if on Windows) with an opens ssh client (ssh command). There is a nifty trick to save time and keyboard typing through creating shortcuts aliases by adding few definitions inside your $HOME/.ssh/config ( ~/.ssh/config ) for your local non root user or even make the configuration system wide (for all existing local /etc/passwd users) via /etc/ssh/ssh_config.
By adding a pseudonym alias for each server it makes sysadmin life much easier as you don't have to type in each time the FQDN (Fully Qualified Domain Name) hostname of remote accessed Linux / Unix / BSD / Mac OS or even Windows sshd ready hosts accessible via remote TCP/IP port 22.


1. Adding local user remote server pointer aliases via ~/.ssh/config


The file ~/.ssh/config is read by the ssh client part of the openssh-client (Linux OS package) on each invokement of the client, and besides defining a pseudonym for the hosts you like to save you time when accessing remote host and hence increase your productivity. Moreover you can also define various other nice options through it to define specifics of remote ssh session for each desired host such as remote host default SSH port (for example if your OpenSSHD is configured to run on non-standard SSH port as lets say 2022 instead of default port TCP 22 for some reason, e.g. security through obscurity etc.).

 

The general syntax of .ssh/config file si simplistic, it goes like this:
 

Host MACHNE_HOSTNAME

SSH_OPTION1 value1
SSH_OPTION1 value1 value2
SSH_OPTION2 value1 value2

 

Host MACHNE_HOSTNAME

SSH_OPTION value
SSH_OPTION1 value1 value2

  • Another understood syntax if you prefer to not have empty whitespaces is to use ( = )
    between the parameter name and values.

Host MACHINE_HOSTNAME
SSH_config=value
SSH_config1=value1 value2

  • All empty lines and lines starting with the hash shebang sign ( # ) would be ignored.
  • All values are case-sensitive, but parameter names are not.

If you have never so far used the $HOME/.ssh/config you would have to create the file and set the proper permissions to it like so:

mkdir -p $HOME/.ssh
chmod 0700 $HOME/.ssh


Below are examples taken from my .ssh/config configuration for all subdomains for my pcfreak.org domain

 

# Ask for password for every subdomain under pc-freak.net for security
Host *.pcfreak.org
user hipopo
passwordauthentication yes
StrictHostKeyChecking no

# ssh public Key authentication automatic login
Host www1.pc-freak.net
user hipopo
Port 22
passwordauthentication no
StrictHostKeyChecking no

UserKnownHostsFile /dev/null

Host haproxy2
    Hostname 213.91.190.233
    User root
    Port 2218
    PubkeyAuthentication yes
    IdentityFile ~/.ssh/haproxy2.pub    
    StrictHostKeyChecking no
    LogLevel INFO     

Host pcfrxenweb
    Hostname 83.228.93.76
    User root
    Port 2218

    PubkeyAuthentication yes
    IdentityFile ~/.ssh/pcfrxenweb.key    
    StrictHostKeyChecking no

Host pcfreak-sf
    Hostname 91.92.15.51
    User root
    Port 2209
    PreferredAuthentications password
    StrictHostKeyChecking no

    Compression yes


As you can see from above configuration the Hostname could be referring either to IP address or to Hostname.

Now to connect to defined IP 91.92.15.51 you can simply refer to its alias

$ ssh pcfreak-sf -v

and you end up into the machine ssh on port 2209 and you will be prompted for a password.

$ ssh pcfrxenweb -v


would lead to IP 83.228.93.76 SSH on Port 2218 and will use the defined public key for a passwordless login and will save you the password typing each time.

Above ssh command is a short alias you can further use instead of every time typing:

$ ssh -i ~/.ssh/pcfrxenweb.key -p 2218 root@83.228.93.76

There is another nifty trick worthy to mention, if you have a defined hostname such as the above config haproxy2 to use a certain variables, but you would like to override some option for example you don't want to connet by default with User root, but some other local account, lets say ssh as devuser@haproxy2 you can type:

$ ssh -o "User=dev" devuser

StrictHostKeyChecking no

– variable will instruct the ssh to not check if the finger print of remote host has changed. Usually this finger print check sum changes in case if for example for some reason the opensshd gets updated or the default /etc/ssh/ssh_host_dsa_key /etc/ssh/sshd_host_dsa_* files have changed due to some reason.
Of course you should use this option only if you tend to access your remote host via a secured VPN or local network, otherwise the Host Key change could be an indicator someone is trying to intercept your ssh session.

 

Compression yes


– variable  enables compression of connection saves few bits was useful in the old modem telephone lines but still could save you few bits
It is also possible to define a full range of IP addresses to be accessed with one single public rsa / dsa key

Below .ssh/config
 

Host 192.168.5.?
     Hostname 192.168.2.18
     User admin
     IdentityFile ~/.ssh/id_ed25519.pub


Would instruct each host attemted to be reached in the IP range of 192.168.2.1-254 to be automatically reachable by default with ssh client with admin user and the respective ed25519.pub key.
 

$ ssh 192.168.1.[1-254] -v

 

2. Adding ssh client options system wide for all existing local or remote LDAP login users


The way to add any Host block is absolutely the same as with a default user except you need to add the configuration to /etc/ssh/ssh_config. Here is a confiugaration from mine Latest Debian Linux

$ cat /etc/ssh/ssh_config

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Protocol 2
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

As you can see pretty much can be enabled by default such as the forwarding of the Authentication agent option ( -A ) option, necessery for some Company server environments to be anbled. So if you have to connect to remote host with enabled Agent Forwarding instead of typing

ssh -A user@remotehostname


To enable Agent Forwarding instead of

ssh -X user@remotehostname


Simply uncomment and set to yes
 

ForwardX11 yes
ForwardX11Trusted yes


Just simply uncomment above's config ForwardAgent no

As you can see ssh could do pretty much, you can configure enable SSH Tunneling or run via a Proxy with the ProxyCommand (If it is the first time you hear about ProxyCommand I warmly recommend you check my previous article – How to pass SSH traffic through a secured Corporate Proxy Server with corkscrew).

Sometimes for a defines hostname, due to changes on remote server ssh configuration, SSH encryption type or a host key removal you might end up with issues connecting, therefore to override all the previously defined options inside .ssh/config by ignoring the configuration with -F /dev/null

$ ssh -F /dev/null user@freak -v


What we learned ?

To sum it up In this article, we have learned how to easify the stressed sysadmin life, by adding Aliases with certain port numbering and configurations for different remote SSH administrated Linux / Unix, hosts via local ~/.ssh/config or global wide /etc/ssh/ssh_config configuration options, as well as how already applied configuration from ~/.ssh/config affecting each user ssh command execution, could be overriden.

Adding proxy to yum repository on Redhat / Fedora / CentOS and other RPM based Linux distributions, Listing and enabling new RPM repositories

Tuesday, September 7th, 2021

yum-add-proxy-host-for-redhat-linux-centos-list-rpm-repositories-enable-disable-repositories

Sometimes if you work in a company that is following PCI standards with very tight security you might need to use a custom company prepared RPM repositories that are accessible only via a specific custom maintained repositories or alternatively you might need the proxy node  to access an external internet repository from the DMZ-ed firewalled zone where the servers lays .
Hence to still be able to maintain the RPM based servers up2date to the latest security patches and install software with yumone very useful feature of yum package manager is to use a proxy host through which you will reach your Redhat Package Manager files  files.

1. The http_proxy and https_proxy shell variables 

To set  a proxy host you need to define there the IP / Hostname or the Fully Qualified Domain Name (FQDN).

By default "http_proxy and https_proxy are empty. As you can guess https_proxy is used if you have a Secure Socket Layer (SSL) certificate for encrypting the communication channel (e.g. you have https:// URL).

[root@rhel: ~]# echo $http_proxy
[root@rhel: ~]#

2. Setting passwordless or password protected proxy host via http_proxy, https_proxy variables

There is a one time very straight forward to configure proxying of traffic via a specific remote configured server with server bourne again  shell (BASH)'s understood variables:
 

a.) Set password free open proxy to shell environment.

[root@centos: ~]# export https_proxy="https://remote-proxy-server:8080"


Now use yum as usual to update the available installabe package list or simply upgrade to the latest packages with lets say:

[root@rhel: ~]# yum check-update && yum update

b.) Configuring password protected proxy for yum

If your proxy is password protected for even tigher security you can provide the password on the command line as well.

[root@centos: ~]# export http_proxy="http://username:pAssW0rd@server:port/"

Note that if you have some special characters you will have to pass the string inside single quotes or escape them to make sure the password will properly handled to server, before trying out the proxy with yum, echo the variable.

[root@centos: ~]# export http_proxy='http://username:p@s#w:E@192.168.0.1:3128/'
  [root@centos: ~]# echo $http_proxy
http://username:p@s#w:E@server:port/

Then do whatever with yum:

[root@centos: ~]# yum check-update && yum search sharutils


If something is wrong and proxy is not properly connected try to reach for the repository manually with curl or wget

[root@centos: ~]# curl -ilk http://download.fedoraproject.org/pub/epel/7/SRPMS/ /epel/7/SRPMS/
HTTP/1.1 302 Found
Date: Tue, 07 Sep 2021 16:49:59 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Location: http://mirror.telepoint.bg/epel/7/SRPMS/
Content-Type: text/plain
Content-Length: 0
AppTime: D=2264
X-Fedora-ProxyServer: proxy01.iad2.fedoraproject.org
X-Fedora-RequestID: YTeYOE3mQPHH_rxD0sdlGAAAA80
X-Cache: MISS from pcfreak
X-Cache-Lookup: MISS from pcfreak:3128
Via: 1.1 pcfreak (squid/4.6)
Connection: keep-alive


Or if you need, you can test the user, password protected proxy with wget as so:

[root@centos: ~]# wget –proxy-user=USERNAME –proxy-password=PASSWORD http://your-proxy-domain.com/optional-rpms/


If you have lynx installed on the machine you can do the remote proxy successful authentication check with it with less typing:

[root@centos: ~]# lynx -pauth=USER:PASSWORD http://proxy-domain.com/optional-rpm/

 

3. Making yum proxy connection permanent via /etc/yum.conf

 

Perhaps the easiest and quickest way to add the http_proxy / https_proxy configured is to store it to automatically load on each server ssh login in your admin user (root) in /root/.bashrc or /root/.bash_profile or in the global /etc/profile or /etc/profile.d/custom.sh etc.

However if you don't want to have hacks and have more cleanness on the systems, the recommended "Redhat way" so to say is to store the configuration inside /etc/yum.conf

To do it via /etc/yum.conf you have to have some records there like:

# The proxy server – proxy server:port number 
proxy=http://mycache.mydomain.com:3128 
# The account details for yum connections 
proxy_username=yum-user 
proxy_password=qwerty-secret-pass

4. Listing RPM repositories and their state

As I had to install sharutils RPM package to the server which contains the file /bin/uuencode (that is provided on CentOS 7.9 Linux from Repo: base/7/x86_64 I had to check whether the repository was installed on the server.

To get a list of all yum repositories avaiable 

[root@centos:/etc/yum.repos.d]# yum repolist all
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.telecoms.bg
 * epel: mirrors.netix.net
 * extras: centos.telecoms.bg
 * remi: mirrors.netix.net
 * remi-php74: mirrors.netix.net
 * remi-safe: mirrors.netix.net
 * updates: centos.telecoms.bg
repo id                                repo name                                                                         status
base/7/x86_64                          CentOS-7 – Base                                                                   enabled: 10,072
base-debuginfo/x86_64                  CentOS-7 – Debuginfo                                                              disabled
base-source/7                          CentOS-7 – Base Sources                                                           disabled
c7-media                               CentOS-7 – Media                                                                  disabled
centos-kernel/7/x86_64                 CentOS LTS Kernels for x86_64                                                     disabled
centos-kernel-experimental/7/x86_64    CentOS Experimental Kernels for x86_64                                            disabled
centosplus/7/x86_64                    CentOS-7 – Plus                                                                   disabled
centosplus-source/7                    CentOS-7 – Plus Sources                                                           disabled
cr/7/x86_64                            CentOS-7 – cr                                                                     disabled
epel/x86_64                            Extra Packages for Enterprise Linux 7 – x86_64                                    enabled: 13,667
epel-debuginfo/x86_64                  Extra Packages for Enterprise Linux 7 – x86_64 – Debug                            disabled
epel-source/x86_64                     Extra Packages for Enterprise Linux 7 – x86_64 – Source                           disabled
epel-testing/x86_64                    Extra Packages for Enterprise Linux 7 – Testing – x86_64                          disabled
epel-testing-debuginfo/x86_64          Extra Packages for Enterprise Linux 7 – Testing – x86_64 – Debug                  disabled
epel-testing-source/x86_64             Extra Packages for Enterprise Linux 7 – Testing – x86_64 – Source                 disabled
extras/7/x86_64                        CentOS-7 – Extras                                                                 enabled:    500
extras-source/7                        CentOS-7 – Extras Sources                                                         disabled
fasttrack/7/x86_64                     CentOS-7 – fasttrack                                                              disabled
remi                                   Remi's RPM repository for Enterprise Linux 7 – x86_64                             enabled:  7,229
remi-debuginfo/x86_64                  Remi's RPM repository for Enterprise Linux 7 – x86_64 – debuginfo                 disabled
remi-glpi91                            Remi's GLPI 9.1 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-glpi92                            Remi's GLPI 9.2 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-glpi93                            Remi's GLPI 9.3 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-glpi94                            Remi's GLPI 9.4 RPM repository for Enterprise Linux 7 – x86_64                    disabled
remi-modular                           Remi's Modular repository for Enterprise Linux 7 – x86_64                         disabled
remi-modular-test                      Remi's Modular testing repository for Enterprise Linux 7 – x86_64                 disabled
remi-php54                             Remi's PHP 5.4 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php55                             Remi's PHP 5.5 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php55-debuginfo/x86_64            Remi's PHP 5.5 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
!remi-php56                            Remi's PHP 5.6 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php56-debuginfo/x86_64            Remi's PHP 5.6 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php70                             Remi's PHP 7.0 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php70-debuginfo/x86_64            Remi's PHP 7.0 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php70-test                        Remi's PHP 7.0 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php70-test-debuginfo/x86_64       Remi's PHP 7.0 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php71                             Remi's PHP 7.1 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php71-debuginfo/x86_64            Remi's PHP 7.1 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php71-test                        Remi's PHP 7.1 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php71-test-debuginfo/x86_64       Remi's PHP 7.1 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
!remi-php72                            Remi's PHP 7.2 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php72-debuginfo/x86_64            Remi's PHP 7.2 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php72-test                        Remi's PHP 7.2 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php72-test-debuginfo/x86_64       Remi's PHP 7.2 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php73                             Remi's PHP 7.3 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php73-debuginfo/x86_64            Remi's PHP 7.3 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php73-test                        Remi's PHP 7.3 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php73-test-debuginfo/x86_64       Remi's PHP 7.3 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php74                             Remi's PHP 7.4 RPM repository for Enterprise Linux 7 – x86_64                     enabled:    423
remi-php74-debuginfo/x86_64            Remi's PHP 7.4 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php74-test                        Remi's PHP 7.4 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php74-test-debuginfo/x86_64       Remi's PHP 7.4 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-php80                             Remi's PHP 8.0 RPM repository for Enterprise Linux 7 – x86_64                     disabled
remi-php80-debuginfo/x86_64            Remi's PHP 8.0 RPM repository for Enterprise Linux 7 – x86_64 – debuginfo         disabled
remi-php80-test                        Remi's PHP 8.0 test RPM repository for Enterprise Linux 7 – x86_64                disabled
remi-php80-test-debuginfo/x86_64       Remi's PHP 8.0 test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo    disabled
remi-safe                              Safe Remi's RPM repository for Enterprise Linux 7 – x86_64                        enabled:  4,549
remi-safe-debuginfo/x86_64             Remi's RPM repository for Enterprise Linux 7 – x86_64 – debuginfo                 disabled
remi-test                              Remi's test RPM repository for Enterprise Linux 7 – x86_64                        disabled
remi-test-debuginfo/x86_64             Remi's test RPM repository for Enterprise Linux 7 – x86_64 – debuginfo            disabled
updates/7/x86_64                       CentOS-7 – Updates                                                                enabled:  2,741
updates-source/7                       CentOS-7 – Updates Sources                                                        disabled
zabbix/x86_64                          Zabbix Official Repository – x86_64                                               enabled:    178
zabbix-debuginfo/x86_64                Zabbix Official Repository debuginfo – x86_64                                     disabled
zabbix-frontend/x86_64                 Zabbix Official Repository frontend – x86_64                                      disabled
zabbix-non-supported/x86_64            Zabbix Official Repository non-supported – x86_64                                 enabled:      5
repolist: 39,364

[root@centos:/etc/yum.repos.d]# yum repolist all|grep -i 'base/7/x86_64'
base/7/x86_64                       CentOS-7 – Base              enabled: 10,072

 

As you can see in CentOS 7 sharutils is enabled from default repositories, however this is not the case on Redhat 7.9, hence to install sharutils there you can one time enable RPM repository to install sharutils 

[root@centos:/etc/yum.repos.d]# yum –enablerepo=rhel-7-server-optional-rpms install sharutils

To install zabbix-agent on the same Redhat server, without caring that I need precisely  know the RPM repository that is providing zabbix agent that in that was (Repo: 3party/7Server/x86_64)  I had to:

[root@centos:/etc/yum.repos.d]# yum –enablerepo \* install zabbix-agent zabbix-sender


Permanently enabling repositories of course is possible via editting or creating fresh new file configuration manually on CentOS / Fedora under directory /etc/yum.repos.d/
On Redhat Enterprise Linux  servers it is easier to use the subscription-manager command instead, like this:
 

[root@rhel:/root]# subscription-manager repos –disable=epel/7Server/x86_64

[root@rhel:/root]# subscription-manager repos –enable=rhel-6-server-optional-rpms

Fix Out of inodes on Postfix Linux Mail Cluster. How to clean up filesystem running out of Inodes, Filesystem inodes on partition is 100% full

Wednesday, August 25th, 2021

Inode_Entry_inode-table-content

Recently we have faced a strange issue with with one of our Clustered Postfix Mail servers (the cluster is with 2 nodes that each has configured Postfix daemon mail servers (running on an OpenVZ virtualized environment).
A heartbeat that checks liveability of clusters and switches nodes in case of one of the two gets broken due to some reason), pretty much a standard SMTP cluster.

So far so good but since the cluster is a kind of abondoned and is pretty much legacy nowadays and used just for some Monitoring emails from different scripts and systems on servers, it was not really checked thoroughfully for years and logically out of sudden the alarming email content sent via the cluster stopped working.

The normal sysadmin job here  was to analyze what is going on with the cluster and fix it ASAP. After some very basic analyzing we catched the problem is caused by a  "inodes full" (100% of available inodes were occupied) problem, e.g. file system run out of inodes on both machines perhaps due to a pengine heartbeat process  bug  leading to producing a high number of .bz2 pengine recovery archive files stored in /var/lib/pengine>

Below are the few steps taken to analyze and fix the problem.
 

1. Finding out about the the system run out of inodes problem


After logging on to system and not finding something immediately is wrong with inodes, all I can see from crm_mon is cluster was broken.
A plenty of emails were left inside the postfix mail queue visible with a standard command

[root@smtp1: ~ ]# postqueue -p

It took me a while to find ot the problem is with inodes because a simple df -h  was showing systems have enough space but still cluster quorum was not complete.
A bit of further investigation led me to a  simple df -i reporting the number of inodes on the local filesystems on both our SMTP1 and SMTP2 got all occupied.

[root@smtp1: ~ ]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/simfs            500000   500000  0   100% /
none                   65536      61   65475    1% /dev

As you can see the number of inodes on the Virual Machine are unfortunately depleted

Next step was to check directories occupying most inodes, as this is the place from where files could be temporary moved to a remote server filesystem or moved to another partition with space on a server locally attached drives.
Below command gives an ordered list with directories locally under the mail root filesystem / and its respective occupied number files / inodes,
the more files under a directory the more inodes are being occupied by the files on the filesystem.

 

run-out-if-inodes-what-is-inode-find-out-which-filesystem-or-directory-eating-up-all-your-system-inodes-linux_inode_diagram.gif
1.1 Getting which directory consumes most of the inodes on the systems

 

[root@smtp1: ~ ]# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
….
…..

…….
    586 /usr/lib64/python2.4
    664 /usr/lib64
    671 /usr/share/man/man8
    860 /usr/bin
   1006 /usr/share/man/man1
   1124 /usr/share/man/man3p
   1246 /var/lib/Pegasus/prev_repository_2009-03-10-1236698426.308128000.rpmsave/root#cimv2/classes
   1246 /var/lib/Pegasus/prev_repository_2009-05-18-1242636104.524113000.rpmsave/root#cimv2/classes
   1246 /var/lib/Pegasus/prev_repository_2009-11-06-1257494054.380244000.rpmsave/root#cimv2/classes
   1246 /var/lib/Pegasus/prev_repository_2010-08-04-1280907760.750543000.rpmsave/root#cimv2/classes
   1381 /var/lib/Pegasus/prev_repository_2010-11-15-1289811714.398469000.rpmsave/root#cimv2/classes
   1381 /var/lib/Pegasus/prev_repository_2012-03-19-1332151633.572875000.rpmsave/root#cimv2/classes
   1398 /var/lib/Pegasus/repository/root#cimv2/classes
   1696 /usr/share/man/man3
   400816 /var/lib/pengine

Note, the above command orders the files from bottom to top order and obviosuly the bottleneck directory that is over-eating Filesystem inodes with an exceeding amount of files is
/var/lib/pengine
 

2. Backup old multitude of files just in case of something goes wrong with the cluster after some files are wiped out


The next logical step of course is to check what is going on inside /var/lib/pengine just to find a very ,very large amount of pe-input-*NUMBER*.bz2 files were suddenly produced.

 

[root@smtp1: ~ ]# ls -1 pe-input*.bz2 | wc -l
 400816


The files are produced by the pengine process which is one of the processes that is controlling the heartbeat cluster state, presumably it is done by running process:

[root@smtp1: ~ ]# ps -ef|grep -i pengine
24        5649  5521  0 Aug10 ?        00:00:26 /usr/lib64/heartbeat/pengine


Hence in order to fix the issue, to prevent some inconsistencies in the cluster due to the file deletion,  copied the whole directory to another mounted parition (you can mount it remotely with sshfs for example) or use a local one if you have one:

[root@smtp1: ~ ]# cp -rpf /var/lib/pengine /mnt/attached_storage


and proceeded to clean up some old multitde of files that are older than 2 years of times (720 days):


3. Clean  up /var/lib/pengine files that are older than two years with short loop and find command

 


First I made a list with all the files to be removed in external text file and quickly reviewed it by lessing it like so

[root@smtp1: ~ ]#  cd /var/lib/pengine
[root@smtp1: ~ ]# find . -type f -mtime +720|grep -v pe-error.last | grep -v pe-input.last |grep -v pe-warn.last -fprint /home/myuser/pengine_older_than_720days.txt
[root@smtp1: ~ ]# less /home/myuser/pengine_older_than_720days.txt


Once reviewing commands I've used below command to delete the files you can run below command do delete all older than 2 years that are different from pe-error.last / pe-input.last / pre-warn.last which might be needed for proper cluster operation.

[root@smtp1: ~ ]#  for i in $(find . -type f -mtime +720 -exec echo '{}' \;|grep -v pe-error.last | grep -v pe-input.last |grep -v pe-warn.last); do echo $i; done


Another approach to the situation is to simply review all the files inside /var/lib/pengine and delete files based on year of creation, for example to delete all files in /var/lib/pengine from 2010, you can run something like:
 

[root@smtp1: ~ ]# for i in $(ls -al|grep -i ' 2010 ' | awk '{ print $9 }' |grep -v 'pe-warn.last'); do rm -f $i; done


4. Monitor real time inodes freeing

While doing the clerance of old unnecessery pengine heartbeat archives you can open another ssh console to the server and view how the inodes gets freed up with a command like:

 

# check if inodes is not being rapidly decreased

[root@csmtp1: ~ ]# watch 'df -i'


5. Restart basic Linux services producing pid files and logs etc. to make then workable (some services might not be notified the inodes on the Hard drive are freed up)

Because the hard drive on the system was full some services started to misbehaving and /var/log logging was impacted so I had to also restart them in our case this is the heartbeat itself
that  checks clusters nodes availability as well as the logging daemon service rsyslog

 

# restart rsyslog and heartbeat services
[root@csmtp1: ~ ]# /etc/init.d/heartbeat restart
[root@csmtp1: ~ ]# /etc/init.d/rsyslog restart

The systems had been a data integrity legacy service samhain so I had to restart this service as well to reforce the /var/log/samhain log file to again continusly start writting data to HDD.

# Restart samhain service init script 
[root@csmtp1: ~ ]# /etc/init.d/samhain restart


6. Check up enough inodes are freed up with df

[root@smtp1 log]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 500000 410531 19469 91% /
none 65536 61 65475 1% /dev


I had to repeat the same process on the second Postfix cluster node smtp2, and after all the steps like below check the status of smtp2 node and the postfix queue, following same procedure made the second smtp2 cluster member as expected 🙂

 

7. Check the cluster node quorum is complete, e.g. postfix cluster is operating normally

 

# Test if email cluster is ok with pacemaker resource cluster manager – lt-crm_mon
 

[root@csmtp1: ~ ]# crm_mon -1
============
Last updated: Tue Aug 10 18:10:48 2021
Stack: Heartbeat
Current DC: smtp2.fqdn.com (bfb3d029-89a8-41f6-a9f0-52d377cacd83) – partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
4 Resources configured.
============

Online: [ smtp2.fqdn.com smtp1.fqdn.com ]

failover-ip (ocf::heartbeat:IPaddr2): Started csmtp1.ikossvan.de
Clone Set: postfix_clone
Started: [ smtp2.fqdn.com smtp1fqdn.com ]
Clone Set: pingd_clone
Started: [ smtp2.fqdn.com smtp1.fqdn.com ]
Clone Set: mailto_clone
Started: [ smtp2.fqdn.com smtp1.fqdn.com ]

 

8.  Force resend a few hundred thousands of emails left in the email queue


After some inodes gets freed up due to the file deletion, i've reforced a couple of times the queued mail servers to be immediately resent to remote mail destinations with cmd:

 

# force emails in queue to be resend with postfix

[root@smtp1: ~ ]# sendmail -q


– It was useful to watch in real time how the queued emails are quickly decreased (queued mails are successfully sent to destination addresses) with:

 

# Monitor  the decereasing size of the email queue
[root@smtp1: ~ ]# watch 'postqueue -p|grep -i '@'|wc -l'

Linux: Howto Fix “N: Repository ‘http://deb.debian.org/debian buster InRelease’ changed its ‘Version’ value from ‘10.9’ to ‘10.10’” error to resolve apt-get release update issue

Friday, August 13th, 2021

Linux's surprises and disorganization is continuously growing day by day and I start to realize it is becoming mostly impossible to support easily this piece of hackware bundled together.
Usually so far during the last 5 – 7 years, I rarely had any general issues with using:

 apt-get update && apt-get upgrade && apt-get dist-upgrade 

to raise a server's working stable Debian Linux version packages e.g. version X.Y to verzion X.Z (for example up the release from Debian Jessie from 8.1 to 8.2). 

Today I just tried to follow this well known and established procedure that, of course nowdays is better to be done with the newer "apt" command instead with the legacy "apt-get"
And the set of 

 

# apt-get update && apt-get upgrade && apt-get dist-upgrade

 

has triggered below shitty error:
 

root@zabbix:~# apt-get update && apt-get upgrade
Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://security.debian.org buster/updates/non-free Sources [688 B]
Get:4 http://repo.zabbix.com/zabbix/5.0/debian buster InRelease [7096 B]
Get:5 http://security.debian.org buster/updates/main Sources [198 kB]
Get:6 http://security.debian.org buster/updates/main amd64 Packages [300 kB]
Get:7 http://security.debian.org buster/updates/main Translation-en [157 kB]
Get:8 http://security.debian.org buster/updates/non-free amd64 Packages [556 B]
Get:9 http://deb.debian.org/debian buster/main Sources [7836 kB]
Get:10 http://repo.zabbix.com/zabbix/5.0/debian buster/main Sources [1192 B]
Get:11 http://repo.zabbix.com/zabbix/5.0/debian buster/main amd64 Packages [4785 B]
Get:12 http://deb.debian.org/debian buster/non-free Sources [85.7 kB]
Get:13 http://deb.debian.org/debian buster/contrib Sources [42.5 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:15 http://deb.debian.org/debian buster/main Translation-en [5968 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 Contents (deb) [37.3 MB]
Get:17 http://deb.debian.org/debian buster/contrib amd64 Packages [50.1 kB]
Get:18 http://deb.debian.org/debian buster/non-free amd64 Packages [87.7 kB]
Get:19 http://deb.debian.org/debian buster/non-free Translation-en [88.9 kB]
Get:20 http://deb.debian.org/debian buster/non-free amd64 Contents (deb) [861 kB]
Fetched 61.1 MB in 22s (2774 kB/s)
Reading package lists… Done
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'


As I used to realize nowdays, as Linux started originally as 'Hackers' operating system, its legacy is just one big hack and everything from simple maintenance up to the higher and more sophisticated things requires a workaround 'hack''.

 

This time the hack to resolve error:
 

N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'


is up to running cmd:
 

debian-server:~# apt-get update –allow-releaseinfo-change
Поп:1 http://ftp.de.debian.org/debian buster-backports InRelease
Поп:2 http://ftp.debian.org/debian stable InRelease
Поп:3 http://security.debian.org stable/updates InRelease
Изт:5 https://packages.sury.org/php buster InRelease [6837 B]
Изт:6 https://download.docker.com/linux/debian stretch InRelease [44,8 kB]
Изт:7 https://packages.sury.org/php buster/main amd64 Packages [317 kB]
Игн:4 https://attic.owncloud.org/download/repositories/production/Debian_10  InRelease
Изт:8 https://download.owncloud.org/download/repositories/production/Debian_10  Release [964 B]
Изт:9 https://packages.sury.org/php buster/main i386 Packages [314 kB]
Изт:10 https://download.owncloud.org/download/repositories/production/Debian_10  Release.gpg [481 B]
Грш:10 https://download.owncloud.org/download/repositories/production/Debian_10  Release.gpg
  Следните подписи са невалидни: DDA2C105C4B73A6649AD2BBD47AE7F72479BC94B
Грш:11 https://ookla.bintray.com/debian generic InRelease
  403  Forbidden [IP: 52.39.193.126 443]
Четене на списъците с пакети… Готово
N: Repository 'https://packages.sury.org/php buster InRelease' changed its 'Suite' value from '' to 'buster'
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://download.owncloud.org/download/repositories/production/Debian_10  Release: 

apt-get-update-allow-releaseinfo-change-debian-linux-screenshot

Onwards to upgrade the system up to the latest .deb packages, as usual run:

# apt-get -y update && apt-get upgrade -y

 

and updates should be applied as usual with some prompts on whether you prefer to keep or replace existing service configuration and some information on some general changes that might affect your installed services. In a few minutes and few prompts hopefully your Debian OS should be up to the latest stable.

How to yum Install Gnome GUI, Latest Guest Addition Tools, Google Chrome latest version and rdesktop / xfreerdp / remmina remote RDP VNC clients On CentOS 7 / 8

Thursday, July 29th, 2021

centos7-logo

I've just reinstalled my CentOS 7 Virtual Machine since after I tried to migrate a .vdi Virtual Box image to the new company laptop using a copy of Virtualbox VM via Microsoft OneDrive was a failure.
Thus I have rebuild all my CentOS Linux programs preinstalled on the old Virtual Machines from scratch, I use this virtual machine for a very simple tasks, so basicly most imporant tools I use is a plain SSH and VNC and Remote Desktop clients just to be able to remotely connect to remote Home based server.


1.Install GNOME Graphical Environment from command line on CentOS 7 with yum and configure it to start GUI on next OS boot


I've used a minimal CentOS installation – ISO CentOS-7-x86_64-DVD-1908.iso and this brings up the OS with a text mode only as usually CentOS is used to roll on Servers and rarely and many times admins did not want to have GUI at all, however my case is different since I do like to use Graphical Environment as I use my CentOS for all kind of testing that can be later applied to a Production machines that doesn't have a GUI, hence to install GNOME on CentOS run below cmds:
 

[root@centos ~ ]# yum group list
Loaded plugins: fastestmirror
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
Loading mirror speeds from cached hostfile
Available Environment Groups:
 Minimal Install
 Compute Node
 Infrastructure Server
 File and Print Server
 Basic Web Server
 Virtualization Host
 Server with GUI
 GNOME Desktop
 KDE Plasma Workspaces
 Development and Creative Workstation
Available Groups:
 Compatibility Libraries
 Console Internet Tools
 Development Tools
 Graphical Administration Tools
 Legacy UNIX Compatibility
 Scientific Support
 Security Tools
 Smart Card Support
 System Administration Tools
 System Management
Done


[root@centos ~ ]# yum groupinstall "GNOME Desktop" "Graphical Administration Tools" -y


Enable GUI to be automatically start on CentOS VM boot in systemd this is configured with the "targets" instead of the well known classical runlevels (the well known /etc/inittab) is now obsolete in newer Linux distros.

[root@centos ~ ]# ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target


2. Install Guest Additions Tools on CentOS


The most basic thing to do once I've had the CentOS Linux release 7.7.1908 (Core) rolled out on the VirtualBox is of course to enable Guest Additions Tools

First I had to install of course Guest Additions Tools to allow myself to have a copy paste in clip board via the Host Machine (Windows 10) and the Guest Machine.
To do I had to:

[root@centos ~ ]# yum install kernel-headers.x86_64 -y

[root@centos ~ ]# rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

[root@centos ~ ]#  yum install perl gcc dkms kernel-devel kernel-headers make bzip2

To check the required VBoxLinuxAdditions.run script kernel headers are at place:

[root@centos ~ ]# ls -l /usr/src/kernels/$(uname -r)


You should get a list of kernel header files

Then once I've done the Insert Guest Additions CD Image from the VirtualBox VM upper menu. I've had to mount and load the guest additions via the script:
 

[root@centos ~ ]# mkdir /mnt/cdrom
[root@centos ~ ]# mount /dev/cdrom /mnt/cdrom
[root@centos ~ ]# sh VBoxLinuxAdditions.run

After rebooting the Virtual Machine, I've used the full screen functionality to test and configured immediately Shared Clipboard and Drag and Drop to be both set to (Bidirectional) as well as configured a Shared folder to provide my Windows Desktop under /mnt/shared_folder (as read write) as I usually do to be able to easily copy files from the VM and to the Windows.

3. Install Google Chrome on the CentOS Virtual Machine with yum
 

Next I've installed the chrome browser that was pretty trivial it is up to fetching the required 32 or 64 bit latest chrome binary this is usually on URL:

[root@centos ~ ]# wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

and installing Google Chrome with superuser with command:

[root@centos ~ ]# yum install ./google-chrome-stable_current_*.rpm -y

 

Loaded plugins: fastestmirror, langpacks
Examining ./google-chrome-stable_current_x86_64.rpm: google-chrome-stable-92.0.4515.107-1.x86_64
Marking ./google-chrome-stable_current_x86_64.rpm to be installed
Resolving Dependencies
–> Running transaction check
—> Package google-chrome-stable.x86_64 0:92.0.4515.107-1 will be installed
–> Processing Dependency: liberation-fonts for package: google-chrome-stable-92.0.4515.107-1.x86_64
Loading mirror speeds from cached hostfile
 * base: mirror.digitalnova.at
 * epel: fedora.ipacct.com
 * extras: mirror.digitalnova.at
 * updates: mirror.digitalnova.at
–> Processing Dependency: libvulkan.so.1()(64bit) for package: google-chrome-stable-92.0.4515.107-1.x86_64
–> Running transaction check
—> Package liberation-fonts.noarch 1:1.07.2-16.el7 will be installed
–> Processing Dependency: liberation-narrow-fonts = 1:1.07.2-16.el7 for package: 1:liberation-fonts-1.07.2-16.el7.noarch
—> Package vulkan.x86_64 0:1.1.97.0-1.el7 will be installed
–> Processing Dependency: vulkan-filesystem = 1.1.97.0-1.el7 for package: vulkan-1.1.97.0-1.el7.x86_64
–> Running transaction check
—> Package liberation-narrow-fonts.noarch 1:1.07.2-16.el7 will be installed
—> Package vulkan-filesystem.noarch 0:1.1.97.0-1.el7 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch   Version         Repository                 Size
================================================================================
Installing:
 google-chrome-stable    x86_64 92.0.4515.107-1 /google-chrome-stable_current_x86_64
                                                                          259 M
Installing for dependencies:
 liberation-fonts        noarch 1:1.07.2-16.el7 base                       13 k
 liberation-narrow-fonts noarch 1:1.07.2-16.el7 base                      202 k
 vulkan                  x86_64 1.1.97.0-1.el7  base                      3.6 M
 vulkan-filesystem       noarch 1.1.97.0-1.el7  base                      6.3 k

Transaction Summary
================================================================================
Install  1 Package (+4 Dependent packages)

Total size: 263 M
Total download size: 3.8 M
Installed size: 281 M
Is this ok [y/d/N]: y
Downloading packages:
(1/4): liberation-fonts-1.07.2-16.el7.noarch.rpm           |  13 kB   00:00     
(2/4): liberation-narrow-fonts-1.07.2-16.el7.noarch.rpm    | 202 kB   00:00     
(3/4): vulkan-filesystem-1.1.97.0-1.el7.noarch.rpm         | 6.3 kB   00:00     
(4/4): vulkan-1.1.97.0-1.el7.x86_64.rpm                    | 3.6 MB   00:00     
——————————————————————————–
Total                                              3.0 MB/s | 3.8 MB  00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : vulkan-filesystem-1.1.97.0-1.el7.noarch                      1/5 
  Installing : vulkan-1.1.97.0-1.el7.x86_64                                 2/5 
  Installing : 1:liberation-narrow-fonts-1.07.2-16.el7.noarch               3/5 
  Installing : 1:liberation-fonts-1.07.2-16.el7.noarch                      4/5 
  Installing : google-chrome-stable-92.0.4515.107-1.x86_64                  5/5 
  Verifying  : vulkan-1.1.97.0-1.el7.x86_64                                 1/5 
  Verifying  : 1:liberation-narrow-fonts-1.07.2-16.el7.noarch               2/5 
  Verifying  : 1:liberation-fonts-1.07.2-16.el7.noarch                      3/5 
  Verifying  : google-chrome-stable-92.0.4515.107-1.x86_64                  4/5 
  Verifying  : vulkan-filesystem-1.1.97.0-1.el7.noarch                      5/5 

Installed:
  google-chrome-stable.x86_64 0:92.0.4515.107-1                                 

Dependency Installed:
  liberation-fonts.noarch 1:1.07.2-16.el7                                       
  liberation-narrow-fonts.noarch 1:1.07.2-16.el7                                
  vulkan.x86_64 0:1.1.97.0-1.el7                                                
  vulkan-filesystem.noarch 0:1.1.97.0-1.el7             


4. Install usable Windows VNC and remote desktop (RDP Client) for CentOS Linux


There is a plenty of clients to choice from if you need to have an RDP client for Linux, but perhaps the most useful ones I usually use are remmina / rdesktop and freerdp. Usually I use remmina on Debian Linux, but under the VM somehow I was not able to make remmina work in Full Screen mode while connected to remote Windows 7 VPS server, thus I've first tried xfreerdp (that comes from default CentOS repositories) and is open source alternative to rdesktop (which is non free distributed binary).
 

[root@centos ~ ]$ sudo yum -y install freerdp


The basic use is:

[hipo@centos ~ ]$ xfreerdp –toggle-fullscreen <remote-server-address>


Unfortunately I did not succeeded to make xfreerdp be able to show me remote desktop in FullScreen mode so had to use additional repository package called nux-dextop to have rdesktop at my disposal.

To install it had to run:

[root@centos ~ ]# rpm –import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro 
[root@centos ~ ]# rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm    
[root@centos ~ ]# yum install rdesktop

To connect to the remote RDP host in Fullscreen with rdesktop :
 

rdesktop -f <remote-server-address>

windows-7-remote-desktop-screenshot-connected-with-rdesktop

As telnet is not installed by default and it is so useful to check ports

5. Install GNU Image Manipulation Program for better screnshotting and Graphic edits


I usually do install GIMP (GNU Image Manipulation Program) since this is my favourite tool to make screenshot on Linux as well as do some minor graphic edits whenever necessery. I warmly recommend gimp to anyone. If you don't have basic GIMP tool and you plan to be daily working a lot with Linux sooner or later some skills with the program will be of a major use even for the most advanced sysadmin :)_

root@centos ~ ]# yum install -y gimp

 

6. Install useful administration tools for daily sysadmin work – telnet, nmap, iftop, htop, iotop, iptraf-ng, tcpdump

 

Having basic analys tools and remote communication port testing, DNS, resolving and connection, cpu, mem statistics I find mostly useful. 

[root@centos .ssh]# yum install telnet nmap iftop htop vnstat sysstat iptraf-ng bind-utils -y

 

 

7. Set Open Explorer and SHOW Desktop key binding shortcuts for GNOME (to make daily work easier)

 


Another useful I do use in my newly installed Virtual Machines is the key combination of Windows (button key) + E – to easily open the GNOME equivalent of Windows Explorer (Nautilus) and Windows (key) + D to hide the active selected Window and Show Desktop. This is configured pretty easy in GNOME through:
 

gnome-control-center

Keyboard (Section)

Perhaps there is other stuff I need to add on the freshly installed Operating System if I remember something else interesting

configure-home-folder-and-hide-all-normal-windows-gnome-key-binding-howto-screenshot

 

8. Install gnome-tweaks to tweak a bit the desktop icon positionsing and additional gnome-shell extras

[root@centos hipo]# yum install -y gnome-shell-extension-workspace-indicator.noarch gnome-shell-extension-workspace-indicator.noarch gnome-shell-extension-suspend-button.noarch gnome-shell-extension-refresh-wifi.noarch gnome-shell-extension-updates-dialog.noarch gnome-shell-extension-windowoverlay-icons.noarch gnome-shell-extension-places-menu.noarch gnome-shell-extension-drive-menu.noarch gnome-shell-extension-apps-menu.noarch gnome-shell-extension-auto-move-windows.noarch gnome-tweaks gnome-shell-extension-systemMonitor.noarch gnome-shell-extension-openweather.noarch gnome-shell-extension-user-theme.noarch gnome-shell-extension-topicons-plus.noarch


Next step is to use gnome-tweaks to set multiple custom preference stuff you like on the gnome 3.28 GUI 

 

gnome-tweak-tool1

gnome-tweak-tool2

gnome-tweak-tool3

9. Change ( Fix) timezone to correct time on the Virtual Machine

[root@localhost ~]# timedatectl 
      Local time: Fri 2021-07-30 12:20:51 CEST
  Universal time: Fri 2021-07-30 10:20:51 UTC
        RTC time: Fri 2021-07-30 10:20:48
       Time zone: Europe/Berlin (CEST, +0200)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2021-03-28 01:59:59 CET
                  Sun 2021-03-28 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2021-10-31 02:59:59 CEST
                  Sun 2021-10-31 02:00:00 CET

[root@localhost ~]# ls -l /etc/localtime
lrwxrwxrwx. 1 root root 35 Jul 29 14:03 /etc/localtime -> ../usr/share/zoneinfo/Europe/Berlin


To change to correct timezone, you need to find out the long name for the timezone you want to use. The timezone naming convention usually uses “Region/City” format.

To list all available time zones, you can either list the files in the /usr/share/zoneinfo directory or use the timedatectl command.

[root@centos ~]# timedatectl list-timezones|tail -n 10
Pacific/Pohnpei
Pacific/Port_Moresby
Pacific/Rarotonga
Pacific/Saipan
Pacific/Tahiti
Pacific/Tarawa
Pacific/Tongatapu
Pacific/Wake
Pacific/Wallis
UTC


As I'm situated in Sofia Bulgaria to set the correct timezone to UTC (Universal Time Clock)  + 2 Hrs, i've checked the correct Continent/Country like so:

[root@centos ~]# timedatectl list-timezones|grep -i Sofia
Europe/Sofia

Once I've my Capital / Country time location  identified to set to it:

[root@centos ~]# timedatectl set-timezone your_time_zone

 

10. Configure remote connection hostname SSH aliases via ssh config ( ~/.ssh/config)

 


I'm having separate Virtual Machines running on my OpenXen virtualization Hypervisor server at different ports which I remember by heart under different hostnames, this saves me time to always type on command line long commands such as:
 

 

 

#  ssh long-hostname -p Port_number

 to make accessibility to remote machines via a simple Hostname Aliases, that forwards to remote port (that gets forwarded via a Local Network configure Netwrork Address Translation), I use the .ssh/config nice Host / Hostname / User / Port directives like below samples:

[hipo@centos .ssh]$ cat config 
Host pcfreak
User root
Port 2248
HostName 83.228.93.76

Host freak
User root
Port 2249
HostName 213.91.190.233


Host pcfrxenweb
User root
Port 2251
Hostname 83.228.93.76

Host pcfrxen
User root
Port 2250
Hostname 213.91.190.233

Now to connect to pcfrxen for example I simply type:

ssh pcfrxen

type in the password to remote VM and I'm in 🙂

The same could be achieved also with Adding Custom Hostname IP Aliases via ~/.bashrc or iteration script as I've explained earlier that fakes like custom /etc/hosts, but I usuaully prefer to use .ssh/config instead like explained above.

Note that above steps should work also on RHEL / Fedora Linux with a minor modifications, as usually this two distros share the RPM package manager. If someone tries to follow the guide and have success on any of this distros please drop a comment with feedback.

CentOS disable SELinux permanently or one time on grub Linux kernel boot time

Saturday, July 24th, 2021

selinux-artistic-penguin-logo-protect-data

 

1. Office 365 cloud connected computer and a VirtualBox hosted machine with SELINUX preventing it to boot

At my job we're in process of migrating my old Lenovo Laptop Thinkpad model L560 Laptop to Dell Latitude 5510 wiith Intel Core i5 vPro CPU and 256 Gb SSD Hard Drive.  The new laptops are generally fiine though they're not even a middle class computers and generally I prefer thinkpads. The sad thing out of this is our employee decided to migrate to Office 365 (again perhaps another stupid managerial decision out of an excel sheet wtih a balance to save some money … 

As you can imagine Office 365 is not really PCI Standards compliant and not secure since our data is stored in Microsoft cloud and theoretically Microsoft has and owns our data or could wipe loose the data if they want to. The other obvious security downside I've noticed with the new "Secure PCI complaint laptop" is the initial PC login screen which by default offers fingerprint authentication or the even worse  and even less secure face recognition, but obviosly everyhing becomes more and more crazy and people become less and less cautious for security if that would save money or centralize the data … In the name of security we completely waste security that is very dubious paradox I don't really understand but anyways, enough rant back to the main topic of this article is how to and I had to disable selinux?

As part of Migration I've used Microsoft OneDrive to copy old files from the Thinkpad to the Latitude (as on the old machine USB's are forbidden and I cannot copy over wiith a siimple USB driive, as well as II have no right to open the laptop and copy data from the Hard driive, and even if we had this right without breaking up some crazy company policy that will not be possible as the hard drive data on old laptop is encrypted, the funny thing is that the new laptop data comes encrypted and there is no something out of the box as BitDefender or McAffee incryption (once again, obviously our data security is a victim of some managarial decisions) …
 

2. OneDrive copy problems unable to sync some of the copied files to Onedrive


Anyways as the Old Laptop's security is quite paranoid and we're like Fort Nox, only port 80 and port 443 connections to the internet can be initiated to get around this harsh restrictions it was as simple to use a Virtualbox Virtual Machine. So on old laptop I've installed a CentOS 7 image which I used so far and I used one drive to copy my vbox .vdi image on the new laptop work machine.

The first head buml was the .vdi which seems to be prohibited to be copied to OneDrive, so to work around this I had to rename the origianl CentOS7.vdi to CentOS7.vdi-renamed on old laptop and once the data is in one drive copy my Vitualbox VM/ directory from one drive to the Dell Latitude machine and rename the .vdi-named towards .vdi as well as import it from the latest installed VirtualBox on the new machine.
 

3. Disable SELINUX from initial grub boot


So far so good but as usual happens with miigrations I've struck towards another blocker, the VM image once initiated to boot from Virtualbox badly crashed with some complains that selinux cannot be loaded.
Realizing CentOS 7 has the more or less meaningless Selinux, I've took the opportunity to disable SeLinux.

To do so I've booted the Kernel with Selinux disabled from GRUB2 loader prompt before Kernel and OS Userland boots.

 

 

I thought I need to type the information on the source in grub. What I did is very simple, on the Linux GRUB boot screen I've pressed

'e' keyboard letter

that brought the grub boot loader into edit mode.

Then I had to add selinux=0 on the edited selected kernel version, as shown in below screenshot:

selinux-disable-from-grub.png

Next to boot the Linux VM without Selinux enabled one time,  just had to press together

Ctrl+X then add selinux=0 on the edited selected kernel version, that should be added as shown in the screenshot somewhere after the line of
root=/dev/mapper/….

4. Permanently Disable Selinux on CentOS 7


Once I managed to boot Virtual Machine properly with Oracle Virtualbox, to permanently disabled selinux I had to:

 

Once booted into CentOS, to check the status of selinux run:

 

# sestatus
Copy
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

 

5. Disable SELinux one time with setenforce command


You can temporarily change the SELinux mode from targeted to permissive with the following command:

 

# setenforce 0


Next o permanently disable SELinux on your CentOS 7 next time the system boots, Open the /etc/selinux/config file and set the SELINUX mod parameter to disabled.

On CentOS 7 you can  edit the kernel parameters in /etc/default/grub (in the GRUB_CMDLINE_LINUX= key) and set selinux=0 so on next VM / PC boot we boot with a SELINUX disabled for example add   RUB_CMDLINE_LINUX=selinux=0 to the file then you have to regenerate your Grub config like this:
 

# grub2-mkconfig -o /etc/grub2.cfg
# grub2-mkconfig -o /etc/grub2-efi.cfg


Further on to disable SeLinux on OS level edit /etc/selinux
 

Default /etc/selinux/config with selinux enabled should look like so:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.
SELINUXTYPE=targeted


To disable SeLinux modify the file to be something like:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.
SELINUXTYPE=targeted

6. Check SELINUX status is disabled

# sestatus

SELinux status:                 disabled

So in this article shottly was explained shortly the fake security adopted by using Microsoft Cloud environment Offiice 365, my faced OneDrive copy issues (which prevented even my old laptop Virtual Machine to boot properly and the handy trick to rename the file that is unwilling to get copied from old PC towards m$ OneDrive as well as the grub trick to disable Selinux permanently from grub2.

Linux: How to set KVM Virtual Machine to autostart on system boot

Thursday, July 15th, 2021

linux-kvm-set-vm-to-automatically-boot-on-linux-system-boot

Recently I've written a short article on how to make auto boot OpenVZ container Virtual Machine if system gets rebooted as I had the task to do so in daily job as sysadmin.
Few days ago after some power outage in one of Frankfurt based Data Centers, some Rack tech equipment has been temporary powered of and once the electricity was up again, some Linux servers  running Kernel Based Virtualization ( KVM ) some of the virtual machines did not managed to automatically start up and we had to manually start them one by one.
To manually start each of the machines that did not start up had to do the trivial:

[root@hypervisor ~]# virsh list
 Id    Name                           State
—————————————————-
 3     VM500                   running

[root@hypervisor ~]# virsh dominfo VM500
Id:             3
Name:           VM500
UUID:           82075903-a4ce-5ec0-a3d8-9ee83d85fc75
OS Type:        hvm
State:          running
CPU(s):         2
CPU time:       247407.9s
Max memory:     2097152 KiB
Used memory:    2097152 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c447,c723 (permissive)

[root@hypervisor ~]# virsh start VM_domain_Name

Of course logcally to prevent future issues in case if Linux server gets suddenly rebooted due to whatever reason, we had to configure the machines in questions to automatically boot on OS system boot.

In some rare cases if above start command does not help you might want to check what is the status of libvirtd and investigate the logs in /var/log/libvirt/ i.e. /var/log/libvirt/libvirt.log

An alternative but more complicated way to set virtual machine to automatically start on boot is by using default location for automatic VM start just like OpenXEn has its /etc/xen/auto/ dirtory from where each soft symlinked VM configuration from /etc/xen/VM_name.cfg is started in KVM Hypervisor hosts to auto boot a certain vm you have to link /etc/libvirt/qemu/VM-to-autoboot-name.xml to /etc/libvirt/qemu/autostart/VM-to-autoboot-name.xml

[root@hypervisor ~]# systemctl status libvirtd
● libvirtd.service – Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-06-29 10:21:30 CEST; 2 weeks 2 days ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 1809 (libvirtd)
    Tasks: 22 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─1809 /usr/sbin/libvirtd
           ├─2335 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
           ├─2336 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
           ├─2386 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…
           └─2387 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…

If for some reason libvirtd is not running or disabled state you will have to enable it with:

[root@hypervisor ~]# systemctl enable libvirtd

If the virtualization is running on some RPM based distribtuion OpenSuse whatever and libvirtd is controlled via chkconfig (redhat runlevel alternative system), you will have to :

[root@hypervisor ~]# chkconfig libvirtd on

Disabling KVM Virtual Machine from auto start on server boot

[root@hypervisor ~]# virsh autostart Debian10 –disable
Domain Debian10 unmarked as autostarted

OpenVZ enable or disable auto start on Linux Hypervisor host boot for Virtual Machine containers

Wednesday, July 7th, 2021

howto-add-virtual-machine-to-auto-start-with-vz-openvz-linux-containers-4-logo-slogan-vertical-big

To make OpenVZ / Virtuozzo Hypervisor servers and you are not sure whether your configured container virtual machines are configured to automatically boot on Linux Physical OS host boot in case of restart after patch update set or after unexpected shutdown due to Kernel / OS bug a hang or due to some electricity Power outage.

To check what is your current configuration for Virtual Environment on CentOS Linux you need to check inside /etc/sysconfig/vz-scripts/VEID.conf
You need to check the value for inside the file

ONBOOT="" 

To get the exact ID of "VEID.conf of the current openvz guest VM containers exec:

[root@openvz vz-scripts]# vzlist -a
      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
       300         23 running   10.10.10.1     VirtualMachine1
       301         25 running   10.10.10.2     VirtualMachine2

[root@openvz ~]# cd /etc/sysconfig/vz-scripts
[root@gbapp2 vz-scripts]# pwd
/etc/sysconfig/vz-scripts

[root@openvz vz-scripts]# grep -i ONBOOT 300.conf 301.conf
300.conf:ONBOOT="yes"
301.conf:ONBOOT="yes"

If you happen to have configured ONBOOT="no" you will need to the change to respective VEID.conf:

vi /etc/sysconfig/vz-scripts/VEID.conf

search for

ONBOOT=”no”

and change to

ONBOOT=”yes”

OpenVZ_virtuozzo-standard-process-tree-landscape

OpenVZ server process tree. The colors of the virtual severs are indicated by colors.

OpenVZ Quick cheat sheet commands

This change will auto-start the VPS container next time the host Hypervisor node is rebooted.
If you happen to have daily work with OpenVZ legacy systems like I do you might find also useful the following OpenVZ Cheatsheet pdf document.

A miniature quick cheatsheet for OpenVZ Virtualion, in case if you are like me and you have to use various virtualization technologies and tend to forget is as below:

vzlist                               # List running instances
vzlist -a                            # List all instances

 

vzctl stop <instance>
vzctl start <instance>
vzctl status <instance>

vzctl exec <instance> <command>      # Run a command

vzctl enter <instance>               # Get console

vzyum <instance> install <package>   # Install a package


# Change properties
vzctl set <instance> –hostname <hostname> –save
vzctl set <instance> –ipadd <IP> –save
vzctl set <instance> –userpasswd root:<password> –save

If need to get more insight on how OpenVZ Virtualization does work on a low level and stretch out its possibilities, an old but useful document you might want to check is OpenVZ-Users-Guide PDF.


If you need it to hava e copy of it openvz_cheat_sheet.txt.