Posts Tagged ‘logs’

« Older Entries

How to Build a Linux Port Scan Honeypot with iptables and test with nmap

Wednesday, October 8th, 2025

buiild-own-honeypot-with-iptables-and-test-with-nmap-linux-howto

Let’s build a simple, practical Linux port‑scan honeypot that uses iptables to capture/redirect incoming connection attempts and simple listeners that log interaction, and use nmap to test it. I’ll give runnable commands, a small Python honeypot listener, iptables rules (with rate‑limited logging), how to test with nmap, and notes on analysis, hardening, and safety.

Important safety notes (read first)

  • Only deploy this on machines/networks you own or are explicitly authorized to test. Scanning other people’s systems or letting an exposed honeypot be used to attack others is illegal and unethical.
  • Run the honeypot inside an isolated VM or protected network segment (use separate host or VLAN), and don't forward traffic from other networks unless you understand the risks.
  • Keep logs rotated and watch disk usage — honeypots can attract high traffic.

Overview

  1. Prepare an isolated Linux VM.
  2. Add iptables rules to log and optionally redirect connection attempts to local listeners.
  3. Run simple service emulators (Python) to accept connections and log data.
  4. Use nmap to test and validate behavior.
  5. Collect, parse, and analyze logs; optionally integrate with syslog/ELK.

Setup assumptions

  • Ubuntu/Debian or similar Linux.
  • Root (sudo) access.
  • iptables, python3, nmap, and socat/netcat available (I’ll show alternatives).

1) Create a safe environment

  • Use a VM (VirtualBox/QEMU/KVM) or container with no sensitive data.
  • Give it a fixed private IP on an isolated network.
  • Optionally place the VM behind another firewall (so you can control exposure).

2) Simple iptables logging rules (filter table)

We want to log connection attempts but avoid log flooding by using limit. Put these commands in a root shell:

# create a honeypot chain

# iptables -N HONEY

# rate-limited LOG then return so traffic continues to normal behavior

# iptables -A HONEY -m limit –limit 5/min –limit-burst 10 -j LOG \
–log-prefix "HPOT_CONN: " –log-level 4

# record raw packets to kernel log (optional NFLOG/ulogd is better for high volume)

# allow the packet to continue (or drop if you want it to appear closed)

# iptables -A HONEY -j ACCEPT

 

# Apply HONEY chain to incoming TCP attempts to all ports (or a subset)

# iptables -A INPUT -p tcp -j HONEY

Notes:

  • This logs packets hitting INPUT. If your VM is behind NAT or you want to catch forwarded packets, apply to FORWARD too.
  • Using -j LOG writes to kernel log, often /var/log/kern.log or via rsyslog to /var/log/messages.
  • Use NFLOG (-j NFLOG) with ulogd if you want structured logging and higher throughput.

If you prefer to log only SYNs (new connection attempts) to reduce noise:

# iptables -A HONEY -p tcp –syn -m limit –limit 5/min –limit-burst 10 \
-j LOG –log-prefix "HPOT_SYN: " –log-level 4

If you want to redirect certain destination ports into local port listeners (so local fake services can accept connections), use the nat table PREROUTING:

# Example redirect incoming port 80 to local port 8080

# iptables -t nat -A PREROUTING -p tcp –dport 80 -j REDIRECT –to-ports 8080

# For ports other than 80/443, add other rules

# iptables -t nat -A PREROUTING -p tcp –dport 22 -j REDIRECT –to-ports 2222

Notes:

  • -t nat -A PREROUTING catches traffic before routing. Use OUTPUT if traffic originates on host itself.
  • REDIRECT only works for local listeners.

3) Simple Python multi‑port honeypot listener

This script will bind multiple ports and log incoming connections and first bytes (banner or payload). Save as simple_honeypot.py:

#!/usr/bin/env python3

# simple_honeypot.py

import socket, threading, logging, time

 

LISTEN_PORTS = [22, 80, 443, 8080, 2222]   # customize

BANNER = b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nOK"

 

logging.basicConfig(

    filename="/var/log/honeypot.log",

    level=logging.INFO,

    format="%(asctime)s %(message)s"

)

 

def handle_client(conn, addr, port):

    try:

        conn.settimeout(5.0)

        data = conn.recv(4096)

        logged = data[:100].decode('utf-8', errors='replace')

        logging.info("CONNECT %s:%s -> port=%d first=%s", addr[0], addr[1], port, logged)

        # Send a small believable banner depending on port

        if port in (80, 8080):

            conn.sendall(BANNER)

        elif port in (22, 2222):

            conn.sendall(b"SSH-2.0-OpenSSH_7.4p1\r\n")

        else:

            conn.sendall(b"220 smtp.example.com ESMTP\r\n")

    except Exception as e:

        logging.info("ERROR %s", e)

    finally:

        try:

            conn.close()

        except:

            pass

 

def start_listener(port):

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    s.bind(('0.0.0.0', port))

    s.listen(50)

    logging.info("Listening on %d", port)

    while True:

        conn, addr = s.accept()

        t = threading.Thread(target=handle_client, args=(conn, addr, port), daemon=True)

        t.start()

 

if __name__ == "__main__":

    for p in LISTEN_PORTS:

        t = threading.Thread(target=start_listener, args=(p,), daemon=True)

        t.start()

    # keep alive

    while True:

        time.sleep(3600)

Run it as root (or with capability binding) so it can listen on low ports:
 

# python3 simple_honeypot.py


# check logs

# tail -F /var/log/honeypot.log

Alternatives:

  • Use socat to spawn a logger for individual ports.
  • Use cowrie, glastopf, honeyd, sshpot etc. if you want full featured honeypots.

4) Using nmap to test the honeypot

Only scan your honeypot IP. Example commands:

Basic TCP port scan:

# nmap -sS -p- -T4 -oN scan_ports.txt <HONEYPOT_IP>

Service/version detection:

# nmap -sV -sC -p 22,80,443,2222 -oN scan_svcs.txt <HONEYPOT_IP>

Scan while avoiding ping (if your VM blocks ICMP):

# nmap -Pn -sS -p1-2000 <HONEYPOT_IP>

Aggressive scan + scripts (use on your own host only):

# nmap -A -T4 <HONEYPOT_IP>

Watch your honeypot logs while scanning to confirm entries.

5) Log collection & analysis

  • Kernel log: sudo tail -F /var/log/kern.log or sudo journalctl -f
  • Python listener log: /var/log/honeypot.log
  • For structured logging and higher throughput: configure ulogd (NFLOG) or forward logs to syslog/rsyslog and ELK/Graylog.
  • Example simple grep: grep HPOT_CONN /var/log/kern.log | tail -n 200

6) Hardening & operational tips

  • Isolate honeypot inside a VM and snapshot it so you can revert.
  • Use rate limits in iptables to avoid log flooding: -m limit –limit 5/min.
  • Rotate logs: configure logrotate for /var/log/honeypot.log.
  • Do not allow outbound traffic from the honeypot unless needed – attackers may use it to pivot. Use egress firewall rules to restrict outbound connections.
  • Consider running the honeypot under an unprivileged user wherever possible.
  • If expecting a lot of traffic, use NFLOG + ulogd or suricata/zeek for more scalable capture.

7) Optional: Richer visibility with NFLOG / ulogd

If you anticipate higher volume or want structured logs, use:

# example: mark packets from all TCP and send to NFLOG group 1

# iptables -I INPUT -p tcp -m limit –limit 10/min -j NFLOG –nflog-group 1

# run ulogd to write NFLOG to a file or DB (configure /etc/ulogd.conf)

8) Example scenario — bring it all together

  1. Start VM and ensure IP 192.168.56.101.
  2. Add iptables logging and redirect HTTP->8080.
  3. Run simple_honeypot.py with ports [22,80,2222,8080].
  4. From your scanner machine: nmap -sS -sV -p22,80,2222,8080 192.168.56.101
  5. Watch /var/log/honeypot.log and kernel logs for HPOT_* prefixes to see connections and payloads.

Quick reference for commands used:

  • Create honeypot chain and log SYNs:

# iptables -N HONEY

# iptables -A HONEY -p tcp –syn -m limit –limit 5/min –limit-burst 10 \
-j LOG –log-prefix "HPOT_SYN: "

# iptables -A INPUT -p tcp -j HONEY

  • Redirect port 80 -> 8080

# iptables -t nat -A PREROUTING -p tcp –dport 80 -j REDIRECT –to-ports 8080

  • Run listener:

# python3 simple_honeypot.py

  • Test with nmap:

# nmap -sS -sV -p22,80,2222,8080 -Pn <IP>

Conclusion

Building a simple Linux port‑scan honeypot with iptables and lightweight listeners gives you practical, immediate visibility into who’s probing your network and how they probe. With a few well‑scoped iptables rules you can capture and rate‑limit connection attempts, redirect selected ports to local emulators, and keep tidy, analysable logs. The small Python listener shown is enough to collect banners and initial payloads; for higher volume or more fidelity you can step up to NFLOG / ulogd, Zeek / Suricata, or production honeypots like Cowrie.

Remember the two rules that keep a honeypot useful and safe: isolate it (VMs, VLANs, strict egress rules) and log thoughtfully (rate limits, rotation, structured formats). That protects your environment and makes the data you collect far more valuable. Over time, turn raw logs into indicators (IP reputations, patterns of ports/probes, common payloads) and feed them into alerts or dashboards to turn passive observation into active defense.

Tags: , , , , , , , , , , ,
Posted in Computer Security, Linux, Security, System Administration, Various | No Comments »

Windows 10 install local Proxy server to Save bandwidth on a slow and limited Mobile Phone HotSpot network Shared connections

Wednesday, August 20th, 2025

https://pc-freak.net/images/how-to-use-local-proxy-to-speed-up-internet-speed-connectivity-on-windows-os-with-squid-and-privoxy

If you're running on Internet ISP that is providing via a Internet / Wifi Router device with a 3G / 4G / 5G etc. but your receiving point location is situated somewhere very far in a places like High mountains lets say Rila Mountain or  Alps on a very distant places where Internet coverate of Inetner Service Provider is low or very low but you need still to Work / Play / Entertain on the Net frequently.
Hence you will cenrtainly be looking for a ways to Speed Up / Optimize the Internet connectivity somehow.
You cannot do miracles but certainly the daily operations and a pack up of repeating traffic can be achieved by using installing and using simple local proxy server.

The advantages of using a proxy are even more besides the speed up of Internet connection lines, here is the Pros you get by using the proxy:
 

  • Using Caches frequently accessed content (e.g., images, scripts, web pages).
  • Blocks ads and trackers (reduces bandwidth).
  • Compresses data (if needed)
  • Can serve multiple local devices if needed.
     

To save bandwidth on a slow and limited connectivity Internet router or mobile phone hotspot using Windows 10, you can install a local proxy server that:

Here’s a step-by-step guide to set this up:
 

Install a local caching proxy server on Windows 10 to reduce bandwidth usage over a mobile hotspot.


1. Install Squid (Caching Proxy Server)

Squid is a powerful and widely used open-source caching proxy.

Download Squid for Windows

Download Squid for Windows from:

https://squid.acmeconsulting.it/download (Unofficial, stable build)

or compile it manually (if you're having an own Linux or BSD router that is passing on the traffic)

2. Install Squid Proxy sever on Windows


2.1. Extract or install the downloaded Squid package.

…

 

2.2. Install it as a Windows Service

Open Command Prompt (Admin) and run:

C:\\Users\\hipo\\Downloads> squid -i

Initialize cache directories:
 

C:\\Users\\hipo\\Downloads> squid -z

 

3. Configure Squid Proxy via squid.conf


3.1. Open squid.conf

usually in

C:\\Squid\\etc\\squid\\squid.conf
 

3.2. Edit key lines:  

http_port 3128
cache_dir ufs c:/squid/var/cache 100 16 256
access_log c:/squid/var/logs/access.log
cache_log c:/squid/var/logs/cache.log
maximum_object_size 4096 KB
cache_mem 64 MB
 

 

3.3. Allow local access:

 

acl localnet src 192.168.0.0/16
http_access allow localnet

(Adjust IP ranges according to your network.)

 

Here's a ready-to-use Squid configuration file optimized for Running on Windows 10:

  • Caching web content to save bandwidth
  • Blocking ads and trackers
  • Allowing local device connections

 

Location for the squid Config File
 

The Windows squid installer should have setup the Squid proxy by default inside C:\Squid so the full path to squid.conf should be:
Place this as

squid.conf

in:

C:\\Squid\\etc\\squid\\squid.conf

 

# BASIC CONFIGURATION
http_port 3128
visible_hostname localhost

# CACHE SETTINGS
cache_mem 128 MB
maximum_object_size 4096 KB
maximum_object_size_in_memory 512 KB
cache_dir ufs c:/squid/var/cache 100 16 256
cache_log c:/squid/var/logs/cache.log
access_log c:/squid/var/logs/access.log

# DNS
dns_nameservers 8.8.8.8 1.1.1.1

# ACLs (Access Control Lists)
acl localhost src 127.0.0.1/32
acl localnet src 192.168.0.0/16
acl Safe_ports port 80      # HTTP
acl Safe_ports port 443     # HTTPS
acl Safe_ports port 21      # FTP
acl CONNECT method CONNECT

# BLOCKED DOMAINS (Ad/Tracking)
acl ads dstdomain .doubleclick.net .googlesyndication.com .googleadservices.com
acl ads dstdomain .ads.yahoo.com .adnxs.com .track.adform.net
http_access deny ads

# SECURITY & ACCESS CONTROL
http_access allow localhost
http_access allow localnet
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all

# REFRESH PATTERNS (Cache aggressively)
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i \.jpg$       10080   90%     43200
refresh_pattern -i \.png$       10080   90%     43200
refresh_pattern -i \.gif$       10080   90%     43200
refresh_pattern -i \.css$       10080   90%     43200
refresh_pattern -i \.js$        10080   90%     43200
refresh_pattern -i \.html$      1440    90%     10080
refresh_pattern .               0       20%     4320

# LOGGING
logfile_rotate 10
 

 

4. Start the Squid Win Service from Admin command prompt

C:\Users\hipo> net start squid


5. Test the Proxy

 

Set the proxy server in your Windows proxy settings:
 

  • Go to Settings > Network & Internet > Proxy
     
  • Enable Manual proxy setup:

Address: 127.0.0.1

Port: 3128

Browse the web — Squid will now cache content locally.

Make sure

C:\Squid\var\cache

and

C:\Squid\var\logs

exist.

You can expand the ad block list by importing public blocklists. Let me know if you want help with that.

To share this proxy with other local devices, ensure they’re on the same network and allowed via ACL.
 

6. Block Ads and Save More Bandwidth with the Proxy

You can modify Squid to:

Block ad domains (using

acl

rules or a blacklist)

Limit download sizes

Restrict background updates or telemetry

Example rule to block a domain:

acl ads dstdomain .doubleclick.net .ads.google.com http_access deny ads


7. Use Aternative lightweight Proxy Privoxy (Lightweight filtering proxy) 

What is Privoxy?

Privoxy is a lightweight, highly customizable proxy server focused on privacy protection, content filtering, and web page optimization.

Unlike caching proxies (like Squid), Privoxy doesn’t store data locally—but it filters and blocks unnecessary traffic before it even reaches your browser.

7.1. Why Use Privoxy to Speed Up Internet?

Here's how Privoxy helps:

Feature Benefit
 Blocks Ads & Banners Reduces page load size and clutter
 Stops Trackers Prevents background data requests
Filters Pop-ups Improves usability and safety
Speeds Up Web Browsing By stripping unwanted content
Low Resource Usage Works on older or low-spec systems

 

Privoxy is easier to set up than Squid and usually much more simple and fits well if you want something simpler and more light weight and is also great for ad/tracker blocking.
To install and use it it comes to 4 simple steps

  1. Download from: https://www.privoxy.org/

  2. Install and run it.

  3. Configure browser/system to use proxy lets say on:

    127.0.0.1:8118

  4. Customize

    config.txt

    to add block rules.

7.2. Configure Your Web Browser or System Proxy

Set your browser/system to use the local Privoxy proxy:

Proxy address:

127.0.0.1

Port:

8118

On Windows:

Go to Settings > Network & Internet > Proxy

Enable Manual Proxy Setup

Enter:

Address:

127.0.0.1

Port:

8118

Save

7.3: Enable Privoxy Filtering and Blocking Rules

Privoxy comes with built-in rules for:

  • Ad blocking
  • Tracker blocking
  • Cookie management
  • Script filtering

You can customize filters in the configuration files via following configs:

Main config:

C:\\Program Files (x86)\\Privoxy\\config

 

Action files:

C:\\Program Files (x86)\\Privoxy\\default.action

 

Filter files:

C:\\Program Files (x86)\\Privoxy\\default.filter

 

7.4. Example to Block All Ads with Privoxy

Look in

default.action

and ensure these are uncommented:

 

{ +block }


Or add specific ad server domains:

{ +block{Ad Servers} }
.com.doubleclick.net
.ads.google.com
.adnxs.com

 

You can further use community-maintained blocklists for stronger Ads filtering.

 

Privoxy does not compress traffic, so to speed up even further with privoxy you might Compress traffic to do so use ziproxy (the http traffic compressor).

Now all your HTTP traffic is routed through Privoxy and you will notice search engines and repeatingly accessed websites pictures and Internet resources such as css / javscript / htmls etc. will give a boost !

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Computer Security, Curious Facts, Windows | No Comments »

Fix Update KB5060999 Not Installing On Windows 11 Version 23H2/22H2 / Fix windows Update failed

Thursday, July 24th, 2025

fix-Update-KB5060999-Not_Installing_On_Windows_11_Version_23H2_22H2-howto-update-screenshot

 

Recently I've stumbled across a very annoying issue on my work Laptop. Suddenly windows stopped being upadted it took me really long time of hours of researching to find out how to resolve the error:

Fix Update KB5060999 Not Installing On Windows 11 Version 23H2/22H2 / Fix windows Update failed

After a lot of catch / tries I can manage to Find a fix Thanks God! 

Here are the few steps I took to resolve it, first I've taken all the steps pointed at the Complete Guide Windows Problems it took really long time but even thouigh I did not manage to resolve the issue it give a lot of understanding on different ways of how windows components stick together and how to debug and solve errors if such ones occurs. Thus I warmly recommend it to anyone working as HelpDesk support admin within corporation or if you are a Windows specialist who makes a living from resolving weird unexpected Windows errors for some little money.

Once I went through a lot of underwater stones and nothing word finally  I managed to solve it by following these concrete steps:

1. Check logs and find error message / problems in CBS.log

C:\> notepad c:\windows\Logs\CBS\CBS.log

The root of the issue I found there as a repeating error messages is:

"InternalOpenPackage failed for Package_for_KB3025096~31bf3856ad364e35~amd64~~6.4.1.0 [HRESULT = 0x800f0805 – CBS_E_INVALID_PACKAGE]"

 

2. Download from Windows catalog the Update


windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7.msu to C:\User\myuser\Downloads

3. Use Winrar or 7Zip to extract the msu in local Directory

E.g.open Administrator command line cmd.exe extract  in lets say in \Users\Username\Downloads\
windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7

C:\Users/a768839/Downloads/windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7 > dir
WSUSSCAN.cab             

4. Remove old package existing on the Windows 11 OS with the same name using DISM tool with /online /remove-package options

C:\Users\myuser\Downloads\windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7>DISM /online /remove-package /packagepath:C:\Users\a768839\Downloads\windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7\Windows11.0-KB5049624-x64-NDP481.cab                                                                                                                                                                    Deployment Image Servicing and Management tool                                                                                                                                                                                                                                            Version: 10.0.22621.2792                                                                                                                                                                                                                                                                                                                                           Image Version: 10.0.22631.5335                                                                                                                                                                                                                                                                                                                 Processing 1 of 1 – Removing package Package_for_DotNetRollup_481~31bf3856ad364e35~amd64~~10.0.9294.1                                                                                                                                                                                     [==========================100.0%================]                                                                                                                                                                                                                            
 The operation completed successfully.      

                                                                                                                                                                                                                                              5. Use DISM /online /add-package to manually insatall  windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7\Windows11.0-KB5049624-x64-NDP481.cab   
                                             

  Assuming you have already downloaded previously and exctracted the .msu file and you have the .cab file at hand run again as cmd.exe Admin                                                                                                                                                                                       

C:\Users\myuser\Downloads\windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7>DISM /online /add-package /packagepath:C:\Users\a768839\Downloads\windows11.0-kb5049624-x64-ndp481_6990e824379adc100fd7895adb30e692697381d7\Windows11.0-KB5049624-x64-NDP481.cab                                                                                                                                                                                           Deployment Image Servicing and Management tool                                                                                                                                                                                                                                            Version: 10.0.22621.2792                                                                                                                                                                                                                                                                                                                                                Image Version: 10.0.22631.5335                                                                                                                                                                                                                                                                                                                  Processing 1 of 1 – Adding package Package_for_DotNetRollup_481~31bf3856ad364e35~amd64~~10.0.9294.1                                                                                                                                                                                       [==========================100.0%==========================]                                                                                                                                                                                                                              The operation completed successfully.    


The idea to remove and install the failing package found in the CBS.log was initially found on                                                                   
 http://datadump.ru/windows-update-error-800f0831/
 

6. Force a Windows chkdsk (Check disk on next restart) to make sure no physical hard drive or some other inode Windows Filesystem errors are there

Within same Admin cmd.exe

C:\Users\myuser\> chkdsk /f C:\


Once command executes reboot the PC and wait for the chkdsk scan to complete and PC to Boot as usual

7. Stop / start wuauserv and do sfc /scannow to refresh some Windows update components

Run following set of commands within Admin cmd

C:\> net stop wuauserv
C:\> command prompt in admin mode
C:\>  dism /online /cleanup-image /startcomponentcleanup
C:\>  sfc /scannow
C:\> restart
C:\> sfc /scannow
C:\> net start wuauserv

8. Download Manually failing update from Windows catalog download site:

As of time of writting this article the URL to download is https://catalog.update.microsoft.com/Search.aspx?q=KB5049624%20%20

C:\Users\myuser\Downloads> dir *.msu
windows11.0-kb5060999-x64_99e39c1cf8a8976d9b3313efb38069876c417f70.msu

9. Run Manually again failing Update from command line

Run cmd as Administrator and exec the file:

C:\Users\myuser\Downloads\windows11.0-kb5060999-x64_99e39c1cf8a8976d9b3313efb38069876c417f70.msu

Wait for the install to complete The computer will need to restart

10. Resync the device to the Computer Domain (mandatory step) only if your PC is part of large organizational Domain

Open Microsoft Store / Company Portal and Sync the device (if necessery)

Settings -> Sync

11. Force the PC to sync itself with remote Global Domain policies (mandatory only if PC is part of Domain)

C:\Users\myuser> gpupdate /force
C:\Users\myuser> repadmin /showrepl

Wait for Synchronization and wait for some time for computer compliancy to get back to normal (Computer compliancy might not be an issue if this is a Personal Windows installation) but for computers part of Larger Windows Domains, where a Domain policy requires a compliancy to set of rules)

To come up with this guide and better understand what is going on to resolve it I have to thank my colleague Eduard for assisting me to read the CBS.log and analyze it and also the following forum thread reading explaining what causes the mysterious windows update Update KB5060999 Not Installing On Windows 11 Version 23H2/22H2 to fail.
 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Everyday Life, Hacks, System Administration, Windows | No Comments »

Enable Debian Linux automatic updates to keep latest OS Patches / Security Up to Date

Monday, January 13th, 2025

Enable Debian Linux automatic updates to keep latest OS Patches / Security Up to Date

Debian: Entenda a Importância Para o Mundo GNU/LINUX

I'm not a big fan of automatism on GNU / Linux as often using automatic updates could totally mess things especially with a complex and a bit chatic OS-es like is Linux nowadays. 
Nevertheless as Security is becoming more and more of a problem especially the browser security, having a scheduled way to apply updates like every normal modern Windows and MAC OS as an option is becoming essential to have a fully manageble Operating system.

As I use Debian GNU / Linux for desktop for my own personal computer and I have already a lot of Debian servers, whose OS minor level and package version maintenance takes up too big chunk of my time (a time I could dedicated to more useful activities). Thus I found it worthy at some cases to trigger Debian's way to keep the OS and security at a present level, the so called Debian "unattended upgrades".

In this article, I'll explain how to install and Enable Automatic (" Unattended " ) Updates on Debian, with the hope that other Debian users might start benefiting from it.
 

Pros of  enabling automatic updates, are:

  • Debian OS Stay secure without constant monitoring.
  • You Save much time by letting your system handle updates.
  • Presumably Enjoying more peace of mind, knowing your system is more protected.

Cons of enabling automatic updates:

  • Some exotic and bad maintained packages (might break after the update)
  • Customizations made on the OS /etc/sysctl.conf or any other very custom server configs might disappear or not work after the update
  • At worst scenario (a very rare but possible case) OS might fail to boot after update 🙂

Regular security updates patch vulnerabilities that could otherwise be exploited by attackers, which is especially important for servers and systems exposed to the internet, where threats evolve constantly.

1. Update Debian System to latest

Before applying automatic updates making any changes, run apt to update package lists and upgrade any outdated packages,to have automatic updates for a smooth configuration process.

# apt update && apt upgrade -y

2. Install the Unattended-Upgrades deb Package 

# apt install unattended-upgrades -y

Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following additional packages will be installed:
  distro-info-data gir1.2-glib-2.0 iso-codes libgirepository-1.0-1 lsb-release python-apt-common python3-apt python3-dbus python3-distro-info python3-gi
Suggested packages:
  isoquery python-apt-doc python-dbus-doc needrestart powermgmt-base
The following NEW packages will be installed:
  distro-info-data gir1.2-glib-2.0 iso-codes libgirepository-1.0-1 lsb-release python-apt-common python3-apt python3-dbus python3-distro-info python3-gi unattended-upgrades
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,786 kB of archives.
After this operation, 24.4 MB of additional disk space will be used.
Do you want to continue? [Y/n]
 

 

# apt install apt-listchanges
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following package was automatically installed and is no longer required:
  linux-image-5.10.0-30-amd64
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  python3-debconf
The following NEW packages will be installed:
  apt-listchanges python3-debconf
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 137 kB of archives.
After this operation, 452 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian bookworm/main amd64 python3-debconf all 1.5.82 [3,980 B]
Get:2 http://deb.debian.org/debian bookworm/main amd64 apt-listchanges all 3.24 [133 kB]
Fetched 137 kB in 0s (292 kB/s)
Preconfiguring packages …
Deferring configuration of apt-listchanges until /usr/bin/python3
and python's debconf module are available
Selecting previously unselected package python3-debconf.
(Reading database … 84582 files and directories currently installed.)
Preparing to unpack …/python3-debconf_1.5.82_all.deb …
Unpacking python3-debconf (1.5.82) …
Selecting previously unselected package apt-listchanges.
Preparing to unpack …/apt-listchanges_3.24_all.deb …
Unpacking apt-listchanges (3.24) …
Setting up python3-debconf (1.5.82) …
Setting up apt-listchanges (3.24) …

Creating config file /etc/apt/listchanges.conf with new version

 

Example config for apt-listchanges would be like:

# vim /etc/apt/listchanges.conf
[apt]
frontend=pager
email_address=root
confirm=0
save_seen=/var/lib/apt/listchanges.db
which=both

3. Enable Automatic unattended upgrades

Once installed, enable automatic updates with the following command, which will prompt asking if you want to enable automatic updates. Select Yes and press Enter, which will confirm that the unattended-upgrades service is active and ready to manage updates for you.

# dpkg-reconfigure unattended-upgrades

Configure-Unattended-Upgrades-on-Debian_Linux-dpkg-reconfigure-screenshot

Or non-interactively by running command:

# echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true | debconf-set-selections
dpkg-reconfigure -f noninteractive unattended-upgrades


4. Set the Schedule for Automatic Updates on Debian

By default, unattended-upgrades runs daily, to verify or modify the schedule, check the systemd timer:

# sudo systemctl status apt-daily.timer
# sudo systemctl status apt-daily-upgrade.timer
# systemctl edit apt-daily-upgrade.timer

Current apt-daily.timer config as of Debian 12 (bookworm) is as follows

root@haproxy2:/etc/apt/apt.conf.d# cat  /lib/systemd/system/apt-daily.timer
[Unit]
Description=Daily apt download activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target
root@haproxy2:/etc/apt/apt.conf.d#


 

# systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
OnCalendar=03:00
RandomizedDelaySec=0

 

At Line  num 2 above is needed to reset (empty) the default value shown below in line  num 5.
Line 4 is needed to prevent any random delays coming from the defaults.


Now both timers should be active, if not, activate them with:

# systemctl enable –now apt-daily.timer
# systemctl enable –now apt-daily-upgrade.timer


These timers ensure that updates are checked and applied regularly, without manual intervention.

5.Test one time Automatic Updates on Debian works

To ensure everything is working, simulate an unattended upgrade with a dry run:

# unattended-upgrade –dry-run

 

You can monitor automatic updates by checking the logs.

# less /var/log/unattended-upgrades/unattended-upgrades.log

Log shows details of installed updates and any issues that occurred. Reviewing logs periodically can help you ensure that updates are being applied correctly and troubleshoot any problems.

6. Advanced Configuration Options

If you’re a power user or managing multiple systems, you might want to explore these additional settings in the configuration file:

# vim /etc/apt/apt.conf.d/50unattended-upgrades


Configure unattended-upgrades to send you an email whenever updates are installed.

Unattended-Upgrade::Mail "your-email-address@email-address.com";


Enable automatic reboots after kernel updates
by adding the line:

Unattended-Upgrade::Automatic-Reboot "true";

To schedule reboots after package upgrade is applied  at a specific time:

Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Specify packages you don’t want to be updated by editing the Unattended-Upgrade::Package-Blacklist section in the configuration file.

 

Here is alternative way to configure the unattended upgrade, by using apt configuration options:

# vim /etc/apt/apt.conf.d/02periodic

// Control parameters for cron jobs by /etc/cron.daily/apt-compat //


// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";


// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";


// Do "apt-get upgrade –download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";


// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";


// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "21";


// Send report mail to root
//     0:  no report             (or null string)
//     1:  progress report       (actually any string)
//     2:  + command outputs     (remove -qq, remove 2>/dev/null, add -d)
//     3:  + trace on
APT::Periodic::Verbose "2";

If you have to simultaneously update multiple machines and you're on a limited connection line, configure download limits if you’re on a metered connection by setting options in /etc/apt/apt.conf.d/20auto-upgrades.

7. Stop Automatic Unattended Upgrade

Under some circumstances if it happens the unattended upgrades are no longer required and you want to revert back to manual package updates, to disable the updates you have to disable the unattended-upgrades service

# systemctl stop unattended-upgrades


8.  Stop an ongoing apt deb package set of updates applied on Debian server

Perhaps not often, but it might be you have run an automated upgrade and this has broke a server system or a service and for that reason you would like to stop the upcoming upgrade (some of whose might have started on other servers) immediately, to do so, the easiest way (not always safe thogh) is to kill the unattended-upgrades daemon.
 

# pkill –signal SIGKILL unattended-upgrades


Note that this a very brutal way to kill it and that might lead to some broken package update, that you might have to later fix manually.

If you have the unattended-upgrade process running on the OS in the process list backgrounded and you want to stop the being on the fly upgrade on the system more safely for the system, you can stop and cancel the ongoing apt upgrade  it by running the ncurses prompt interface, through dpkg-reconfigure

# dpkg-reconfigure unattended-upgrades


Then just select No, press Enter. In my case, this has promptly stopped the ongoing unattended upgrade that seemed blocked (at least as promptly as the hardware seemed to allow 🙂 ).

If you want to disable it for future, so it doesn't automatically gets enabled on next manual update, by some update script disable service as well.
 

# systemctl disable unattended-upgrades
…

 

Close up

That’s all ! Now, your Debian system will automatically handle security updates, keeping your system secure without you having to do a thing.
The same guide should be good for most Deb based distributions such as Ubuntu / Mint and there rest of other Debian derivative OS-es.
You’ve now set up a reliable way to ensure your system stays protected from vulnerabilities, but anyways it is a good practice to always login and check what the update has done to the system, otherwise expect the unexpected. 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Computer Security, Curious Facts, Linux, Linux and FreeBSD Desktop, Linux on Laptops, Various | No Comments »

How to log multiple haproxy / apache / mysql instance via haproxy log-tagging / Segregating log management for multiple HAProxy instances using rsyslog

Tuesday, May 23rd, 2023

rsyslog-logo-picture-use-programname-and-haproxy-log-tag-directives-together-to-log-as-many-process-streams-as-you-like

 

Introduction

This article provides a guide on refining haproxy  logging mechanism by leveraging the `programname` property in rsyslog, coupled with the `log-tag` directive in haproxy.
This approach will create a granular logging setup, separating logs according to their originating services and specific custom tags, enhancing overall log readability.

Though the article is written concretely for logging multiple log streams from haproxy this can be successfully applied
for any other Linux service to log as many concrete log-tagged data streams as you prefer.

Scope

The guide focuses on tailoring the logging mechanisms for two haproxy  instances named `haproxy` and `haproxyssl`, utilizing the `programname` property in rsyslog and the `log-tag` directive in haproxy for precise log management.

The haproxy and haproxyssl instances are two separate systemd config file prepared instances.
haproxy instance is simple haproxy proxying tcp traffic in non-encrypted form, whether haproxyssl is a special instance
prepared to tunnel the incoming http traffic in ssl form. Both instances of haproxy runs as a separate processes on the server.

Here is the systemd configuration of haproxy systemd service file:

# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target


As well as the systemd service configuration for haproxyssl:
 

# cat /usr/lib/systemd/system/haproxyssl.service
[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Environment="CONFIG=/etc/haproxy/haproxy_ssl_prod.cfg" "PIDFILE=/run/haproxy_ssl_prod.pid"
EnvironmentFile=/etc/sysconfig/haproxy
ExecStartPre=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecStart=/usr/sbin/haproxyssl -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxyssl -f $CONFIG -c -q $OPTIONS
ExecReload=/bin/kill -USR2 $MAINPID
SuccessExitStatus=143
KillMode=mixed
Type=notify

[Install]
WantedBy=multi-user.target

 

Step 1: Configuring HAProxy instances with `log-tag`
 

To distinguish between logs from two HAProxy instances, `log-tag` directive is used to add tags to logs. This tag is used to filter these logs in rsyslog.
Modify the HAProxy configuration file in `/etc/haproxy/haproxy.*.cfg`

HAProxy Instance 1 (haproxy)
 

#———————————————————————
# Global settings
#———————————————————————
global
      log          127.0.0.1 local6 debug
      log-tag      haproxy

HAProxy Instance 2 (haproxyssl)


#———————————————————————
# Global settings
#———————————————————————
global
    log          127.0.0.1 local5 debug
    log-tag      haproxyssl

 

Step 2: Implementing rsyslog configuration for haproxy logs
 

Next, create a new rsyslog configuration file, stored in /etc/rsyslog.d/. Ensure the new configuration file ends in `.conf`

HAProxy Instance 1 (haproxy)

Now add rsyslog rules to filters logs based on the `programname` and the custom log tag:
 

# vi /etc/rsyslog.d/55_haproxy.conf
if $programname == 'haproxy' then /var/log/haproxy.log
&stop

HAProxy Instance 2 (haproxyssl)
# vi /etc/rsyslog.d/51_haproxy_ssl.conf
if $programname == 'haproxy_ssl' then /var/log/haproxy_ssl.log
&stop


These rules filter logs that originate from haproxy  and contain the respective string haproxy   or haproxy_ssl , directing them to their respective log files. The `& stop` directive ensures that rsyslog stops processing the log once a match is found, preventing dublication.

Finally, restart both the haproxy and rsyslog services for the changes to take effect:

# systemctl restart haproxy
# systemctl restart haproxyssl
# systemctl restart rsyslog


Reading References

haproxy:   log-tag directive

rsyslog:    rsyslogd documentation

This is a guest article originally written by: Dimitar Paskalev, guest blogging with good interesting articles is always mostly welcome 

Tags: , , , , , , , , , , ,
Posted in Educational, Hacks, Linux | 1 Comment »

Configure aide file integrity check server monitoring in Zabbix to track for file changes on servers

Tuesday, March 28th, 2023

zabbix-aide-log-monitoring-logo

Earlier I've written a small article on how to setup AIDE monitoring for Server File integrity check on Linux, which put the basics on how this handy software to improve your server overall Security software can be installed and setup without much hassle.

Once AIDE is setup and a preset custom configuration is prepared for AIDE it is pretty useful to configure AIDE to monitor its critical file changes for better server security by monitoring the AIDE log output for new record occurs with Zabbix. Usually if no files monitored by AIDE are modified on the machine, the log size will not grow, but if some file is modified once Advanced Linux Intrusion Detecting (aide) binary runs via the scheduled Cron job, the /var/log/app_aide.log file will grow zabbix-agentd will continuously check the file for size increases and will react.

Before setting up the Zabbix required Template, you will have to set few small scripts that will be reading a preconfigured list of binaries or application files etc. that aide will monitor lets say via /etc/aide-custom.conf
 

1. Configure aide to monitor files for changes


Before running aide, it is a good idea to prepare a file with custom defined directories and files that you plan to monitor for integrity checking e.g. future changes with aide, for example to capture bad intruders who breaks into server which runs aide and modifies critical files such as /etc/passwd /etc/shadow /etc/group or / /usr/local/etc/* or /var/* / /usr/* critical files that shouldn't be allowed to change without the admin to soon find out.

# cat /etc/aide-custom.conf

# Example configuration file for AIDE.
@@define DBDIR /var/lib/aide
@@define LOGDIR /var/log/aide
# The location of the database to be read.
database=file:@@{DBDIR}/app_custom.db.gz
database_out=file:@@{DBDIR}/app_aide.db.new.gz
gzip_dbout=yes
verbose=5

report_url=file:@@{LOGDIR}/app_custom.log
#report_url=syslog:LOG_LOCAL2
#report_url=stderr
#NOT IMPLEMENTED report_url=mailto:root@foo.com
#NOT IMPLEMENTED report_url=syslog:LOG_AUTH

# These are the default rules.
#
#p:      permissions
#i:      inode:
#n:      number of links
#u:      user
#g:      group
#s:      size
#b:      block count
#m:      mtime
#a:      atime
#c:      ctime
#S:      check for growing size
#acl:           Access Control Lists
#selinux        SELinux security context
#xattrs:        Extended file attributes
#md5:    md5 checksum
#sha1:   sha1 checksum
#sha256:        sha256 checksum
#sha512:        sha512 checksum
#rmd160: rmd160 checksum
#tiger:  tiger checksum

#haval:  haval checksum (MHASH only)
#gost:   gost checksum (MHASH only)
#crc32:  crc32 checksum (MHASH only)
#whirlpool:     whirlpool checksum (MHASH only)

FIPSR = p+i+n+u+g+s+m+c+acl+selinux+xattrs+sha256

#R:             p+i+n+u+g+s+m+c+acl+selinux+xattrs+md5
#L:             p+i+n+u+g+acl+selinux+xattrs
#E:             Empty group
#>:             Growing logfile p+u+g+i+n+S+acl+selinux+xattrs

# You can create custom rules like this.
# With MHASH…
# ALLXTRAHASHES = sha1+rmd160+sha256+sha512+whirlpool+tiger+haval+gost+crc32
ALLXTRAHASHES = sha1+rmd160+sha256+sha512+tiger
# Everything but access time (Ie. all changes)
EVERYTHING = R+ALLXTRAHASHES

# Sane, with multiple hashes
# NORMAL = R+rmd160+sha256+whirlpool
NORMAL = FIPSR+sha512

# For directories, don't bother doing hashes
DIR = p+i+n+u+g+acl+selinux+xattrs

# Access control only
PERMS = p+i+u+g+acl+selinux

# Logfile are special, in that they often change
LOG = >

# Just do sha256 and sha512 hashes
LSPP = FIPSR+sha512

# Some files get updated automatically, so the inode/ctime/mtime change
# but we want to know when the data inside them changes
DATAONLY =  p+n+u+g+s+acl+selinux+xattrs+sha256

##############TOUPDATE
#To delegate to app team create a file like /app/aide.conf
#and uncomment the following line
#@@include /app/aide.conf
#Then remove all the following lines
/etc/zabbix/scripts/check.sh FIPSR
/etc/zabbix/zabbix_agentd.conf FIPSR
/etc/sudoers FIPSR
/etc/hosts FIPSR
/etc/keepalived/keepalived.conf FIPSR
# monitor haproxy.cfg
/etc/haproxy/haproxy.cfg FIPSR
# monitor keepalived
/home/keepalived/.ssh/id_rsa FIPSR
/home/keepalived/.ssh/id_rsa.pub FIPSR
/home/keepalived/.ssh/authorized_keys FIPSR
/usr/local/bin/script_to_run.sh FIPSR
/usr/local/bin/another_script_to_monitor_for_changes FIPSR

#  cat /usr/local/bin/aide-config-check.sh
#!/bin/bash
/sbin/aide -c /etc/aide-custom.conf -D

# cat /usr/local/bin/aide-init.sh
#!/bin/bash
/sbin/aide -c /etc/custom-aide.conf -B database_out=file:/var/lib/aide/custom-aide.db.gz -i

 

# cat /usr/local/bin/aide-check.sh

#!/bin/bash
/sbin/aide -c /etc/custom-aide.conf -Breport_url=stdout -B database=file:/var/lib/aide/custom-aide.db.gz -C|/bin/tee -a /var/log/aide/custom-aide-check.log|/bin/logger -t custom-aide-check-report
/usr/local/bin/aide-init.sh

 

# cat /usr/local/bin/aide_app_cron_daily.txt

#!/bin/bash
#If first time, we need to init the DB
if [ ! -f /var/lib/aide/app_aide.db.gz ]
   then
    logger -p local2.info -t app-aide-check-report  "Generating NEW AIDE DATABASE for APPLICATION"
    nice -n 18 /sbin/aide –init -c /etc/aide_custom.conf
    mv /var/lib/aide/app_aide.db.new.gz /var/lib/aide/app_aide.db.gz
fi

nice -n 18 /sbin/aide –update -c /etc/aide_app.conf
#since the option for syslog seems not fully implemented we need to push logs via logger
/bin/logger -f /var/log/aide/app_aide.log -p local2.info -t app-aide-check-report
#Acknoledge the new database as the primary (every results are sended to syslog anyway)
mv /var/lib/aide/app_aide.db.new.gz /var/lib/aide/app_aide.db.gz

What above cron job does is pretty simple, as you can read it yourself. If the configuration predefined aide database store file /var/lib/aide/app_aide.db.gz, does not
exists aide will create its fresh empty database and generate a report for all predefined files with respective checksums to be stored as a comparison baseline for file changes. 

Next there is a line to write aide file changes via rsyslog through the logger and local2.info handler


2. Setup Zabbix Template with Items, Triggers and set Action

2.1 Create new Template and name it YourAppName APP-LB File integrity Check

aide-itengrity-check-zabbix_ Configuration of templates

Then setup the required Items, that will be using zabbix's Skip embedded function to scan file in a predefined period of file, this is done by the zabbix-agent that is
supposed to run on the server.

2.2 Configure Item like

aide-zabbix-triggers-screenshot
 

*Name: check aide log file

Type: zabbix (active)

log[/var/log/aide/app_aide.log,^File.*,,,skip]

Type of information: Log

Update Interval: 30s

Applications: File Integrity Check

Configure Trigger like

Enabled: Tick On

images/aide-zabbix-screenshots/check-aide-log-item


2.3 Create Triggers with the respective regular expressions, that would check the aide generated log file for file modifications


aide-zabbix-screenshot-minor-config

Configure Trigger like
 

Enabled: Tick On


*Name: Someone modified {{ITEM.VALUE}.regsub("(.*)", \1)}

*Expression: {PROD APP-LB File Integrity Check:log[/var/log/aide/app_aide.log,^File.*,,,skip].strlen()}>=1

Allow manual close: yes tick

*Description: Someone modified {{ITEM.VALUE}.regsub("(.*)", \1)} on {HOST.NAME}

 

2.4 Configure Action

 

aide-zabbix-file-monitoring-action-screensho

Now assuming the Zabbix server has  a properly set media for communication and you set Alerting rules zabbix-server can be easily set tosend mails to a Support email to get Notifications Alerts, everytime a monitored file by aide gets changed.

That's all folks ! Enjoy being notified on every file change on your servers  !
 

Tags: , , , , , , , , , , , , , , , , , , , , ,
Posted in Computer Security, Linux, System Administration, Zabbix | 4 Comments »

Zabbix: Monitor Linux rsyslog configured central log server is rechable with check_log_server_status.sh userparameter script

Wednesday, June 8th, 2022

zabbix-monitor-central-log-server-is-reachable-from-host-with-a-userparamater-script-zabbix-logo

On modern Linux OS servers on Redhat / CentOS / Fedora and Debian based distros log server service is usually running on the system  such as rsyslog (rsyslogd) to make sure the logging from services is properly logged in separate logs under /var/log.

A very common practice on critical server machines in terms of data security, where logs produced by rsyslog daermon needs to be copied over network via TCP or UDP protocol immediately is to copy over the /var/log produced logs to another configured central logging server. Then later every piece of bit generated by rsyslogd could be  overseen by a third party auditor person and useful for any investigation in case of logs integrity is required or at worse case if there is a suspicion that system in question is hacked by a malicious hax0r and logs have been "cleaned" up from any traces leading to the intruder (things usually done locally by hackers) or by any automated script exploit tools since yesr.

This doubled logging of system events to external log server  ipmentioned is very common practice by companies to protect their log data and quite useful for logs to be recovered easily later on from the central logging server machine that could be also setup for example to use rsyslogd to receive logs from other Linux machines in circumstances where some log disappears just like that (things i've seen happen) for any strange reason or gets destroyed by the admins mistake locally on machine / or by any other mean such as filesystem gets damaged. a very common practice by companies to protect their log data.  

Monitor remote logging server is reachable with userparameter script

Assuming that you already have setup a logging from the server hostname A towards the Central logging server log storepool and everything works as expected the next logical step is to have at least some basic way to monitor remote logging server configured is still reachable all the time and respectively rsyslog /var/log/*.* logs gets properly produced on remote side for example with something like a simple TCP remote server port check and reported in case of troubles in zabbix.

To solve that simple task for company where I'm employed, I've developed below check_log_server_status.sh:
 

#!/bin/bash
# @@ for TCP @ for UDP
# check_log_server_status.sh Script to check if configured TCP / UDP logging server in /etc/rsyslog.conf is rechable
# report to zabbix
DELIMITER='@@';
GREP_PORT='5145';
CONNECT_TIMEOUT=5;

PORT=$(grep -Ei "*.* $DELIMITER.*:$GREP_PORT" /etc/rsyslog.conf|awk -F : '{ print $2 }'|sort -rn |uniq);

#for i in $(grep -Ei "*.* $DELIMITER.*:$GREP_PORT" /etc/rsyslog.conf |grep -v '\#'|awk -F"$DELIMITER" '{ print $2 }' | awk -F ':' '{ print $1 }'|sort -rn); do
HOST=$(grep -Ei "*.* $DELIMITER.*:$GREP_PORT" /etc/rsyslog.conf |grep -v '\#'|awk -F"$DELIMITER" '{ print $2 }' | awk -F ':' '{ print $1 }'|sort -rn)

# echo $PORT

if [[ ! -z $PORT ]] && [[ ! -z $HOST ]]; then
SSH_RETURN=$(/bin/ssh $HOST -p $PORT -o ConnectTimeout=$CONNECT_TIMEOUT 2>&1);
else
echo "PROBLEM Port $GREP_PORT not defined in /etc/rsyslog.conf";
fi

##echo SSH_RETURN $SSH_RETURN;
#exit 1;
if [[ $(echo $SSH_RETURN |grep -i ‘Connection timed out during banner exchange’ | wc -l) -eq ‘1’ ]]; then
echo "rsyslogd $HOST:$PORT OK";
fi

if [[ $(echo $SSH_RETURN |grep -i ‘Connection refused’ | wc -l) -eq ‘1’ ]]; then
echo "rsyslogd $HOST:$PORT PROBLEM";
fi

#sleep 2;
#done


You can download a copy of the script check_log_server_status.sh here

Depending on the port the remote rsyslogd central logging server is using configure it in the script with respective port through the DELIMITER='@@', GREP_PORT='5145', CONNECT_TIMEOUT=5 values.

The delimiter is setup as usually in /etc/rsyslog.conf this the remote logging server for TCP IP is configured with @@ prefix to indicated TCP mode should be used.

Below is example from /etc/rsyslog.conf of how the rsyslogd server is configured:

[root@Server-hostA /root]# grep -i @@ /etc/rsyslogd.conf
# central remote Log server IP / port
*.* @@10.10.10.1:5145

To use the script on a machine, where you have a properly configured zabbix-agentd service host connected and reporting data to a zabbix-server monitoring server.

1. Set up the script under /usr/local/bin/check_log_server_status.sh

[root@Server-hostA /root ]# vim /usr/local/bin/check_log_server_status.sh
…

[root@Server-hostA /root ]# chmod +x /usr/local/bin/check_log_server_status.sh

2. Prepare userparameter_check_log_server.conf with log_server.check Item key

[root@Server-hostA zabbix_agentd.d]# cat userparameter_check_log_server.conf 
UserParameter=log_server.check, /usr/local/bin/check_log_server_status.sh

3. Set in Zabbix some Item such as on below screenshot

 

check-log-server-status-screenshot-linux-item-zabbix.png4. Create a Zabbix trigger 

check-log-server-status-trigger-logserver-is-unreachable-zabbix


The redded hided field in Expression field should be substituted with your actual hostname on which the monitor script will run.

Tags: , , , , , , , , , ,
Posted in Monitoring, System Administration, Various, Zabbix | No Comments »

How to RPM update Hypervisors and Virtual Machines running Haproxy High Availability cluster on KVM, Virtuozzo without a downtime on RHEL / CentOS Linux

Friday, May 20th, 2022

virtuozzo-kvm-virtual-machines-and-hypervisor-update-manual-haproxy-logo


Here is the scenario, lets say you have on your daily task list two Hypervisor (HV) hosts running CentOS or RHEL Linux with KVM or Virutozzo technology and inside the HV hosts you have configured at least 2 pairs of virtual machines one residing on HV Host 1 and one residing on HV Host 2 and you need to constantly keep the hosts to the latest distribution major release security patchset.

The Virtual Machines has been running another set of Redhat Linux or CentOS configured to work in a High Availability Cluster running Haproxy / Apache / Postfix or any other kind of HA solution on top of corosync / keepalived or whatever application cluster scripts Free or Open Source technology that supports a switch between clustered Application nodes.

The logical question comes how to keep up the CentOS / RHEL Machines uptodate without interfering with the operations of the Applications running on the cluster?

Assuming that the 2 or more machines are configured to run in Active / Passive App member mode, e.g. one machine is Active at any time and the other is always Passive, a switch is possible between the Active and Passive node.

HAProxy--Load-Balancer-cluster-2-nodes-your-Servers

In this article I'll give a simple step by step tested example on how you I succeeded to update (for security reasons) up to the latest available Distribution major release patchset on one by one first the Clustered App on Virtual Machines 1 and VM2 on Linux Hypervisor Host 1. Then the App cluster VM 1 / VM 2 on Hypervisor Host 2.
And finally update the Hypervisor1 (after moving the Active resources from it to Hypervisor2) and updating the Hypervisor2 after moving the App running resources back on HV1.
I know the procedure is a bit monotonic but it tries to go through everything step by step to try to mitigate any possible problems. In case of failure of some rpm dependencies during yum / dnf tool updates you can always revert to backups so in anyways don't forget to have a fully functional backup of each of the HV hosts and the VMs somewhere on a separate machine before proceeding further, any possible failures due to following my aritcle literally is your responsibility 🙂

 

0. Check situation before the update on HVs / get VM IDs etc.

Check the virsion of each of the machines to be updated both Hypervisor and Hosted VMs, on each machine run:
 

# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)


The machine setup I'll be dealing with is as follows:
 

hypervisor-host1 -> hypervisor-host1.fqdn.com 
•    virt-mach-centos1
•    virt-machine-zabbix-proxy-centos (zabbix proxy)

hypervisor-host2 -> hypervisor-host2.fqdn.com
•    virt-mach-centos2
•    virt-machine-zabbix2-proxy-centos (zabbix proxy)

To check what is yours check out with virsh cmd –if on KVM or with prlctl if using Virutozzo, you should get something like:

[root@hypervisor-host2 ~]# virsh list
 Id Name State
—————————————————-
 1 vm-host1 running
 2 virt-mach-centos2 running

 # virsh list –all

[root@hypervisor-host1 ~]# virsh list
 Id Name State
—————————————————-
 1 vm-host2 running
 3 virt-mach-centos1 running

[root@hypervisor-host1 ~]# prlctl list
UUID                                    STATUS       IP_ADDR         T  NAME
{dc37c201-08c9-589d-aa20-9386d63ce3f3}  running      –               VM virt-mach-centos1
{76e8a5f8-caa8-5442-830e-aa4bfe8d42d9}  running      –               VM vm-host2
[root@hypervisor-host1 ~]#

If you have stopped VMs with Virtuozzo to list the stopped ones as well.
 

# prlctl list -a

[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# vzlist -a
                                CTID      NPROC STATUS    IP_ADDR         HOSTNAME
[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# prlctl list
UUID                                    STATUS       IP_ADDR         T  NAME
{92075803-a4ce-5ec0-a3d8-9ee83d85fc76}  running      –               VM virt-mach-centos2
{74a7bbe8-9245-5385-ac0d-d10299100789}  running      –               VM vm-host1

# prlctl list -a


If due to Virtuozzo version above command does not return you can manually check in the VM located folder, VM ID etc.
 

[root@hypervisor-host2 vmprivate]# ls
74a7bbe8-9245-4385-ac0d-d10299100789  92075803-a4ce-4ec0-a3d8-9ee83d85fc76
[root@hypervisor-host2 vmprivate]# pwd
/vz/vmprivate
[root@hypervisor-host2 vmprivate]#


[root@hypervisor-host1 ~]# ls -al /vz/vmprivate/
total 20
drwxr-x—. 5 root root 4096 Feb 14  2019 .
drwxr-xr-x. 7 root root 4096 Feb 13  2019 ..
drwxr-x–x. 4 root root 4096 Feb 18  2019 1c863dfc-1deb-493c-820f-3005a0457627
drwxr-x–x. 4 root root 4096 Feb 14  2019 76e8a5f8-caa8-4442-830e-aa4bfe8d42d9
drwxr-x–x. 4 root root 4096 Feb 14  2019 dc37c201-08c9-489d-aa20-9386d63ce3f3
[root@hypervisor-host1 ~]#


Before doing anything with the VMs, also don't forget to check the Hypervisor hosts has enough space, otherwise you'll get in big troubles !
 

[root@hypervisor-host2 vmprivate]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centos_hypervisor-host2-root   20G  1.8G   17G  10% /
devtmpfs                          20G     0   20G   0% /dev
tmpfs                             20G     0   20G   0% /dev/shm
tmpfs                             20G  2.0G   18G  11% /run
tmpfs                             20G     0   20G   0% /sys/fs/cgroup
/dev/sda1                        992M  159M  766M  18% /boot
/dev/mapper/centos_hypervisor-host2-home  9.8G   37M  9.2G   1% /home
/dev/mapper/centos_hypervisor-host2-var   9.8G  355M  8.9G   4% /var
/dev/mapper/centos_hypervisor-host2-vz    755G   25G  692G   4% /vz

 

[root@hypervisor-host1 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  1.8G   45G   4% /
devtmpfs                  20G     0   20G   0% /dev
tmpfs                     20G     0   20G   0% /dev/shm
tmpfs                     20G  2.1G   18G  11% /run
tmpfs                     20G     0   20G   0% /sys/fs/cgroup
/dev/sda2                992M  153M  772M  17% /boot
/dev/mapper/centos-home  9.8G   37M  9.2G   1% /home
/dev/mapper/centos-var   9.8G  406M  8.9G   5% /var
/dev/mapper/centos-vz    689G   12G  643G   2% /vz

Another thing to do before proceeding with update is to check and tune if needed the amount of CentOS repositories used, before doing anything with yum.
 

[root@hypervisor-host2 yum.repos.d]# ls -al
total 68
drwxr-xr-x.   2 root root  4096 Oct  6 13:13 .
drwxr-xr-x. 110 root root 12288 Oct  7 11:13 ..
-rw-r–r–.   1 root root  4382 Mar 14  2019 CentOS7.repo
-rw-r–r–.   1 root root  1664 Sep  5  2019 CentOS-Base.repo
-rw-r–r–.   1 root root  1309 Sep  5  2019 CentOS-CR.repo
-rw-r–r–.   1 root root   649 Sep  5  2019 CentOS-Debuginfo.repo
-rw-r–r–.   1 root root   314 Sep  5  2019 CentOS-fasttrack.repo
-rw-r–r–.   1 root root   630 Sep  5  2019 CentOS-Media.repo
-rw-r–r–.   1 root root  1331 Sep  5  2019 CentOS-Sources.repo
-rw-r–r–.   1 root root  6639 Sep  5  2019 CentOS-Vault.repo
-rw-r–r–.   1 root root  1303 Mar 14  2019 factory.repo
-rw-r–r–.   1 root root   666 Sep  8 10:13 openvz.repo
[root@hypervisor-host2 yum.repos.d]#

 

[root@hypervisor-host1 yum.repos.d]# ls -al
total 68
drwxr-xr-x.   2 root root  4096 Oct  6 13:13 .
drwxr-xr-x. 112 root root 12288 Oct  7 11:09 ..
-rw-r–r–.   1 root root  1664 Sep  5  2019 CentOS-Base.repo
-rw-r–r–.   1 root root  1309 Sep  5  2019 CentOS-CR.repo
-rw-r–r–.   1 root root   649 Sep  5  2019 CentOS-Debuginfo.repo
-rw-r–r–.   1 root root   314 Sep  5  2019 CentOS-fasttrack.repo
-rw-r–r–.   1 root root   630 Sep  5  2019 CentOS-Media.repo
-rw-r–r–.   1 root root  1331 Sep  5  2019 CentOS-Sources.repo
-rw-r–r–.   1 root root  6639 Sep  5  2019 CentOS-Vault.repo
-rw-r–r–.   1 root root  1303 Mar 14  2019 factory.repo
-rw-r–r–.   1 root root   300 Mar 14  2019 obsoleted_tmpls.repo
-rw-r–r–.   1 root root   666 Sep  8 10:13 openvz.repo


1. Dump VM definition XMs (to have it in case if it gets wiped during update)

There is always a possibility that something will fail during the update and you might be unable to restore back to the old version of the Virtual Machine due to some config misconfigurations or whatever thus a very good idea, before proceeding to modify the working VMs is to use KVM's virsh and dump the exact set of XML configuration that makes the VM roll properly.

To do so:
Check a little bit up in the article how we have listed the IDs that are part of the directory containing the VM.
 

[root@hypervisor-host1 ]# virsh dumpxml (Id of VM virt-mach-centos1 ) > /root/virt-mach-centos1_config_bak.xml
[root@hypervisor-host2 ]# virsh dumpxml (Id of VM virt-mach-centos2) > /root/virt-mach-centos2_config_bak.xml

 


2. Set on standby virt-mach-centos1 (virt-mach-centos1)

As I'm upgrading two machines that are configured to run an haproxy corosync cluster, before proceeding to update the active host, we have to switch off
the proxied traffic from node1 to node2, – e.g. standby the active node, so the cluster can move up the traffic to other available node.
 

[root@virt-mach-centos1 ~]# pcs cluster standby virt-mach-centos1


3. Stop VM virt-mach-centos1 & backup on Hypervisor host (hypervisor-host1) for VM1

Another prevention step to make sure you don't get into damaged VM or broken haproxy cluster after the upgrade is to of course backup 

 

[root@hypervisor-host1 ]# prlctl backup virt-mach-centos1

or
 

[root@hypervisor-host1 ]# prlctl stop virt-mach-centos1
[root@hypervisor-host1 ]# cp -rpf /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3 /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3-bak
[root@hypervisor-host1 ]# tar -czvf virt-mach-centos1_vm_virt-mach-centos1.tar.gz /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3

[root@hypervisor-host1 ]# prlctl start virt-mach-centos1


4. Remove package version locks on all hosts

If you're using package locking to prevent some other colleague to not accidently upgrade the machine (if multiple sysadmins are managing the host), you might use the RPM package locking meachanism, if that is used check RPM packs that are locked and release the locking.

+ List actual list of locked packages

[root@hypervisor-host1 ]# yum versionlock list  
…
…..
0:libtalloc-2.1.16-1.el7.*
0:libedit-3.0-12.20121213cvs.el7.*
0:p11-kit-trust-0.23.5-3.el7.*
1:quota-nls-4.01-19.el7.*
0:perl-Exporter-5.68-3.el7.*
0:sudo-1.8.23-9.el7.*
0:libxslt-1.1.28-5.el7.*
versionlock list done                          

+ Clear the locking            

# yum versionlock clear                               


+ List actual list / == clear all entries
 

[root@virt-mach-centos2 ]# yum versionlock list; yum versionlock clear
[root@virt-mach-centos1 ]# yum versionlock list; yum versionlock clear
[root@hypervisor-host1 ~]# yum versionlock list; yum versionlock clear
[root@hypervisor-host2 ~]# yum versionlock list; yum versionlock clear


5. Do yum update virt-mach-centos1


For some clarity if something goes wrong, it is really a good idea to make a dump of the basic packages installed before the RPM package update is initiated,
The exact versoin of RHEL or CentOS as well as the list of locked packages, if locking is used.

Enter virt-mach-centos1 (ssh virt-mach-centos1) and run following cmds:
 

# cat /etc/redhat-release  > /root/logs/redhat-release-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


+ Only if needed!!
 

# yum versionlock clear
# yum versionlock list


Clear any previous RPM packages – careful with that as you might want to keep the old RPMs, if unsure comment out below line
 

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

Proceed with the update and monitor closely the output of commands and log out everything inside files using a small script that you should place under /root/status the script is given at the end of the aritcle.:
 

yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
yum check-update | wc -l
yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

 

6. Check if everything is running fine after upgrade

Reboot VM
 

# shutdown -r now


7. Stop VM virt-mach-centos2 & backup  on Hypervisor host (hypervisor-host2)

Same backup step as prior 

# prlctl backup virt-mach-centos2


or
 

# prlctl stop virt-mach-centos2
# cp -rpf /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76 /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76-bak
## tar -czvf virt-mach-centos2_vm_virt-mach-centos2.tar.gz /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76

# prctl start virt-mach-centos2


8. Do yum update on virt-mach-centos2

Log system state, before the update
 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list

 

+ Clean old install update / packages if required
 

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Initiate the update

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l 
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


9. Check if everything is running fine after upgrade
 

Reboot VM
 

# shutdown -r now

 

10. Stop VM vm-host2 & backup
 

# prlctl backup vm-host2


or

# prlctl stop vm-host2

Or copy the actual directory containig the Virtozzo VM (use the correct ID)
 

# cp -rpf /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9 /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9-bak
## tar -czvf vm-host2.tar.gz /vz/vmprivate/76e8a5f8-caa8-4442-830e-aa5bfe8d42d9

# prctl start vm-host2


11. Do yum update vm-host2
 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Clear only if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Do the rpm upgrade

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


12. Check if everything is running fine after upgrade
 

Reboot VM
 

# shutdown -r now


13. Do yum update hypervisor-host2

 

 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock   if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


Update rpms
 

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


14. Stop VM vm-host1 & backup


Some as ealier
 

# prlctl backup vm-host1

or
 

# prlctl stop vm-host1

# cp -rpf /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789 /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789-bak
# tar -czvf vm-host1.tar.gz /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789

# prctl start vm-host1


15. Do yum update vm-host2
 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


16. Check if everything is running fine after upgrade

+ Reboot VM

# shutdown -r now


17. Do yum update hypervisor-host1

Same procedure for HV host 1 

# cat /etc/redhat-release  > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock
 

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out


18. Check if everything is running fine after upgrade

Reboot VM
 

# shutdown -r now


Check hypervisor-host1 all VMs run as expected 


19. Check if everything is running fine after upgrade

Reboot VM
 

# shutdown -r now


Check hypervisor-host2 all VMs run as expected afterwards


20. Check once more VMs and haproxy or any other contained services in VMs run as expected

Login to hosts and check processes and logs for errors etc.
 

21. Haproxy Unstandby virt-mach-centos1

Assuming that the virt-mach-centos1 and virt-mach-centos2 are running a Haproxy / corosync cluster you can try to standby node1 and check the result
hopefully all should be fine and traffic should come to host node2.

[root@virt-mach-centos1 ~]# pcs cluster unstandby virt-mach-centos1


Monitor logs and make sure HAproxy works fine on virt-mach-centos1


22. If necessery to redefine VMs (in case they disappear from virsh) or virtuosso is not working

[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos1_config_bak.xml
[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos2_config_bak.xml


23. Set versionlock to RPMs to prevent accident updates and check OS version release

[root@virt-mach-centos2 ]# yum versionlock \*
[root@virt-mach-centos1 ]# yum versionlock \*
[root@hypervisor-host1 ~]# yum versionlock \*
[root@hypervisor-host2 ~]# yum versionlock \*

[root@hypervisor-host2 ~]# cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)

Other useful hints

[root@hypervisor-host1 ~]# virsh console dc37c201-08c9-489d-aa20-9386d63ce3f3
Connected to domain virt-mach-centos1
..

! Compare packages count before the upgrade on each of the supposable identical VMs and HVs – if there is difference in package count review what kind of packages are different and try to make the machines to look as identical as possible  !

Packages to update on hypervisor-host1 Count: XXX
Packages to update on hypervisor-host2 Count: XXX
Packages to update virt-mach-centos1 Count: – 254
Packages to update virt-mach-centos2 Count: – 249

The /root/status script

+++

#!/bin/sh
echo  '=======================================================   '
echo  '= Systemctl list-unit-files –type=service | grep enabled '
echo  '=======================================================   '
systemctl list-unit-files –type=service | grep enabled

echo  '=======================================================   '
echo  '= systemctl | grep ".service" | grep "running"            '
echo  '=======================================================   '
systemctl | grep ".service" | grep "running"

echo  '=======================================================   '
echo  '= chkconfig –list                                        '
echo  '=======================================================   '
chkconfig –list

echo  '=======================================================   '
echo  '= netstat -tulpn                                          '
echo  '=======================================================   '
netstat -tulpn

echo  '=======================================================   '
echo  '= netstat -r                                              '
echo  '=======================================================   '
netstat -r


+++

That's all folks, once going through the article, after some 2 hours of efforts or so you should have an up2date machines.
Any problems faced or feedback is mostly welcome as this might help others who have the same setup.

Thanks for reading me 🙂

Tags: , , , , , , , , , , , , , , , ,
Posted in CentOS, Linux, Linux Package Management, System Administration, Virtual Machines | 1 Comment »

Linux script to periodically log enabled systemctl services, configured network IPs and routings, server established connections and iptables firewall rules

Tuesday, January 25th, 2022

bash-script-command-line-script-logo

For those who are running some kind of server be it virtual or physical, where multiple people or many systemins have access, sometimes it could be quite a mess as someone due to miscommunication or whatever could change something on the configured Network Ethernet interfaces, or configured routing tables, or simply issue an update which might change the set of automatically set to run systemctl services due to update. Such changes on a Linux server Operating system often can remain unnoticed and could cause quite a harm. Even when the change is noticed the logical question occurs what was the previous network route on the server or what kind of network was configured on Ethernet interface ethX etc. 
Problems like the described where, pretty common in many public Private Clouds or VMWare / XEN based Hypervisors that host multiple  Virtual machines, for that reason I've developed a small script which is pretty dumb on the first glimpse but mostly useful as it keeps historical records of such important information.
 

#!/bin/sh
# script to show configured services on system, configured IPs, netstat state and network routes
# Script to be used during CentOS and Redhat Enterprise Linux RPM package updates with yum

output_file=network_ip_routes_services_status;
ddate=$(date '+%Y-%m-%d_%H-%M-%S');
iptables=$(which iptables);
if [ ! -d /root/logs/ ]; then
mkdir /root/logs/;
fi

echo "STARTED: $(date '+%Y-%m-%d_%H-%M-%S'):" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# systemctl list-unit-files\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
systemctl list-unit-files –type=service | grep enabled | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e '# systemctl | grep ".service" | grep "running"\n' | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
systemctl | grep ".service" | grep "running" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# netstat -tulpn\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
netstat -tulpn | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# netstat -r\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
netstat -r | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# ip a s\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
ip a s | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# /sbin/route -n\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
/sbin/route -n | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# $iptables -L -n\n" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo -e "# $iptables -t nat -L" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
$iptables -L -n | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
$iptables -t nat -L | tee -a /root/logs/$output_file-$(hostname)-$ddate.log
echo "ENDED $(date '+%Y-%m-%d_%H-%M-%S'):" | tee -a /root/logs/$output_file-$(hostname)-$ddate.log

 

Script produces its logs inside  /root/logs/network_ip_routes_services_status*hostname*currentdate*.log, put the script inside /root/ or wherever you like.

To keep an eye how network routing or ip configuration or firewall changed or there was a peak with the established connections towards daemons running on host (lets say requiring a machine upgrade), I've set the script to run as usually via cron job at the end of the predefined cron job tasks, like so:

# crontab -u root -e
# periodic dump and log network routing tables, netstat and systemctl list-unit-files
*/1 01 01,25 * * /root/show_running_services_netstat_ips_route1.sh 2>&1 >/dev/null

You can download a copy of show_running_services_netstat_ips_route1.sh script here.
The script is written without much of efficiency on mind, as you can see the with the multiple tee -a and for critical hosts it might be a good idea to rewrite it to use '>>' OPERAND instead, anyhows as most machines today are pretty powerful it doesn't really matter much.

Of course today such a script is quite archaic, as most big corporations are using much more complex monitoring software such as Zabbix, Prometheus or if some kind of Elastic Search is used Kibana etc. but for a basic needs and even for a double checking and comparing with other more advanced monitoring tools  (in case if monitoring tools  database gets damaged or temporary down until backupped), still I think such an oldschool simple monitoring script can be of use.

A good addition to that if you use a central logging server is to set another cron to periodically synchronize produced /root/logs/* to somewhere, here is how to do it with simple rsync (considering your host is configured to login with a user without password with ssh key authentication).

# HOSTNAME=$(hostname); rsync -axHv –ignore-existing -e 'ssh -p 22' /bashscripts/  -q -i –out-format="%t %f %b" –log-file=/var/log/rsync_sync_jobs.log –info=progress2 root@BACKUP_SERVER_HOST:/$(HOSTNAME)-logs/

Once something strange occurs with the machine, like the machine needs to be rebuild

I would be glad to hear if some of my readers uses some useful script which I can adopt myself. Cheers  🙂

Tags: , , , , , , , , ,
Posted in Linux, Linux Backup tools | No Comments »

Linux: How to set KVM Virtual Machine to autostart on system boot

Thursday, July 15th, 2021

linux-kvm-set-vm-to-automatically-boot-on-linux-system-boot

Recently I've written a short article on how to make auto boot OpenVZ container Virtual Machine if system gets rebooted as I had the task to do so in daily job as sysadmin.
Few days ago after some power outage in one of Frankfurt based Data Centers, some Rack tech equipment has been temporary powered of and once the electricity was up again, some Linux servers  running Kernel Based Virtualization ( KVM ) some of the virtual machines did not managed to automatically start up and we had to manually start them one by one.
To manually start each of the machines that did not start up had to do the trivial:

[root@hypervisor ~]# virsh list
 Id    Name                           State
—————————————————-
 3     VM500                   running

[root@hypervisor ~]# virsh dominfo VM500
Id:             3
Name:           VM500
UUID:           82075903-a4ce-5ec0-a3d8-9ee83d85fc75
OS Type:        hvm
State:          running
CPU(s):         2
CPU time:       247407.9s
Max memory:     2097152 KiB
Used memory:    2097152 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c447,c723 (permissive)

[root@hypervisor ~]# virsh start VM_domain_Name

Of course logcally to prevent future issues in case if Linux server gets suddenly rebooted due to whatever reason, we had to configure the machines in questions to automatically boot on OS system boot.

In some rare cases if above start command does not help you might want to check what is the status of libvirtd and investigate the logs in /var/log/libvirt/ i.e. /var/log/libvirt/libvirt.log

An alternative but more complicated way to set virtual machine to automatically start on boot is by using default location for automatic VM start just like OpenXEn has its /etc/xen/auto/ dirtory from where each soft symlinked VM configuration from /etc/xen/VM_name.cfg is started in KVM Hypervisor hosts to auto boot a certain vm you have to link /etc/libvirt/qemu/VM-to-autoboot-name.xml to /etc/libvirt/qemu/autostart/VM-to-autoboot-name.xml. 

[root@hypervisor ~]# systemctl status libvirtd
● libvirtd.service – Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-06-29 10:21:30 CEST; 2 weeks 2 days ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 1809 (libvirtd)
    Tasks: 22 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─1809 /usr/sbin/libvirtd
           ├─2335 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
           ├─2336 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/default.conf –leasefile-ro –dhcp-script=/us…
           ├─2386 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…
           └─2387 /usr/sbin/dnsmasq –conf-file=/var/lib/libvirt/dnsmasq/Host-Only.conf –leasefile-ro –dhcp-script=/…

If for some reason libvirtd is not running or disabled state you will have to enable it with:

[root@hypervisor ~]# systemctl enable libvirtd

If the virtualization is running on some RPM based distribtuion OpenSuse whatever and libvirtd is controlled via chkconfig (redhat runlevel alternative system), you will have to :

[root@hypervisor ~]# chkconfig libvirtd on

Disabling KVM Virtual Machine from auto start on server boot

[root@hypervisor ~]# virsh autostart Debian10 –disable
Domain Debian10 unmarked as autostarted

Tags: , , , , , , , , ,
Posted in Linux, System Administration, Virtual Machines | No Comments »

« Older Entries