updates Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘updates’

How to Fix Windows Update When It Says “Up to Date” But Updates Are Missing

Monday, March 2nd, 2026

Knowing your system isn’t fully updated as OS BUILD Release does not match the latest one it has to but still Windows Update insists everything is “green” good and “Up to date.” is really weird and frustrating stuff Windows user can experience. It makes it even worser if you are like me and your computer is in a large corporate domain that is using Azure (Office 365) services for Auth.

If some updates fail silently or don’t install properly, your notebook / PC may be missing important security patches, Video / Sound Driver / Chipset driver fixes, or feature improvements etc, and with time it can lead due to Windows domain applied policies to left over your computer be considered Unsafe or Broken even dis-joined from the Domain.

Why Windows Update Says Up to Date but Update Are missing Happens ?

There might be mutiple scenarios but Common causes include:

Corrupted update cache
Interrupted installations (PC got hard shut down electricity power outage or
laptop battery has discharged during update)
Broken Windows services (due to)
System file corruption (cause of viruses / malware or during mess left over of multiple windows updates over years)
Registry conflicts (Windows registry conflicts due to installed PC apps etc.)
Failed cumulative updates

Windows may mark updates as “processed” even if installation didn’t complete correctly.
Identifying Missing or Broken Windows updates is really hard sometimes.

Usually to capture it you will have to:
Check the Windows OS Build Release

from: Settings -> System -> About

In this guide, will walk through proven methods to fix Windows Update when it’s stuck or falsely reporting success.

1. Try PC Restart First

Before diving into advanced fixes:

Restart PC.
Go Check Settings → Windows Update → Check for updates again.

Sometimes updates are downloaded but waiting for a reboot to complete installation and thus this oddity is observed.

2. Run the Built-In Windows Update Troubleshooter

Both Windows 10 and Windows 11 include a built-in repair tool (that is starting to get Legacy nowadays but still sometimes can help)

Steps:

Open Settings
Go to System → Troubleshoot → Other troubleshooters
Find Windows Update
Click Run

Let it complete the scan and apply any recommended fixes. Most time this won’t solve it but as it is easy to try out give it a try.

3. Manually Reset Windows Update Components

If Windows still erroneously thinks everything is installed but something is broken internally, resetting the update components often solves the problem.

CleanUp SoftwareDistribution update cache folder is perhaps Most Effective FIX

Cleaning the C:\Windows\SoftwareDistribution folder is actually one of the most effective fixes when Windows refuses to install updates but claims everything is up to date.

C:\Windows\SoftwareDistribution

This is where Windows temporarily stores:

Downloaded update files
Update installation logs
Temporary metadata
Cached update database

If this cache becomes corrupted, Windows Update may:

Fail silently
Not detect new updates
Show “Up to date” incorrectly
Get stuck at 0% or 100%

This method works in both Windows 10 and Windows 11.

What Happens When You Delete SoftwareDistribution?

Deleting (or renaming) the folder:

Does NOT delete installed updates
Does NOT break Windows
Forces Windows to rebuild the update cache
Forces a fresh update scan

It’s completely safe if you do it correct.

Recommended Method (Play Safe)

N!B! Do NOT delete the folder while update services are running.

Step 1: Stop Windows Update Services

Open Command Prompt as Administrator and run:

net stop wuauserv
net stop bits
net stop cryptSvc
net stop msiserver

Wait until all services stop successfully.

Step 2: Rename the Update Folder (Safer Than Deleting)

Rename Update Folders

move C:\Windows\SoftwareDistribution SoftwareDistribution.old

Enter in Windows Safe Mode (to enter it Press SHIFT and choose Restart)

Go to:

C:\Windows\SoftwareDistribution

Rename it to:

C:\Windows\SoftwareDistribution.old

If Windows refuses to move out, make sure services are stopped.

To do it via Safe Mode with Command Prompt only

move c:\Windows\SoftwareDistribution C:\Windows\SoftwareDistribution.old

Step 3: Restart Services

Back in Command Prompt:

net start wuauserv
net start bits
net start cryptSvc
net start msiserver

Restart Computer.

4. Use the Microsoft Update Catalog to Manually download recent applied
Update

Sometimes a specific update fails repeatedly but Windows doesn’t clearly report it.

You can manually download it from:

Microsoft Update Catalog

How to manually Instlal KB* Win update:

Find the KB number (for example: KB5030219)
Search for it in the catalog
Download the version matching your system (x64, ARM64, etc.)
Install manually

This bypasses Windows Update’s automatic system.

5. Use the Windows Installation Assistant

If feature updates (like 22H2 → 23H2) are not appearing, use:

Windows 11 Installation Assistant
Windows 10 Update Assistant

These tools force a full system upgrade while keeping files and apps intact.

6. Check for Corrupted System Files

Corrupted system files can prevent updates from applying properly.

Open Command Prompt as Administrator and run:

C:\Windows> sfc /scannow

Then run:

C:\Windows> DISM /Online /Cleanup-Image /RestoreHealth

After both scans complete, restart and try updating again.

7. Make Sure You’re Not Paused or you are on a Metered connection

Windows may appear updated if:

Updates are paused
Your connection is set as metered
You’re on a managed/work PC with update policies

Check:

Settings → Windows Update → Advanced options

8. Check Your Windows Version Manually

Press Win + R, type:

winver

Compare your version with the latest available on Microsoft’s official release page
https://learn.microsoft.com/en-en/windows/release-health/windows11-release-information
to confirm whether you’re truly up to date.

9. Update your Video / Audio / Motherboard Chipsets and peripheral drivers to latest

Depending on the laptop brand or PC, Check for latest available install drivers from the Internet and apply it to PC.
Dell / HP and ASUS / ACER / MSI Usually has their dedicated software that can do that quickly, i.e. as i'm using currently Dell notebook. There you can use Dell Comamnd Update / Dell SupportAssistant to do so.

10. Move catroot folder (to clean up Windows Update package signatures)

What is catroot2 ?

The catroot2 folder is used by Microsoft Windows to store:

Windows Update package signatures
Cryptographic catalog files (.cat files)
Data used by the Cryptographic Services component
Information needed to validate and install updates
It plays a critical role in verifying update integrity.

move C:\Windows\System32\catroot2 catroot2.old

is used as a repair step for Windows Update issues because it resets the Catroot2 folder, which stores important update-related data.

11. Perform an In-Place Repair Upgrade (Last Resort)

If nothing works:

Download the latest Windows ISO (Windows Installation Assistant)
Mount it
Run setup.exe
Choose Keep personal files and apps

This reinstalls Windows without deleting your data and fixes deeply broken update components.

12. If none of these helps check Windows Logs for a clue

If you want to go even deeper, check Event Viewer logs under:

Windows Logs → Setup

That will show detailed update errors and will helpfully give you the clue on how to fix it.

Summary / close up

If Windows says “Up to date” but you suspect missing updates, don’t ignore it, as soon your OS will either become messed or you will miss critical Performance and Performance improvements / Stability Features. Even if PC continues work relatively stable the missing Security patches would be critical, and the computer exposure to the internet lefts you as an easy victim for your computer to be hacked or infected by some kind of encryption / ransomware worm etc. In most cases, the updates did not apply due to easy solvable issue and simple reset update components, a clean up of Update cache or manually installing the update solves the problem and WIndows gets back to the wanted OS update release. If this does not happen however you should check the system for Main system corrupted files

Tags: again, cause, complete, Corrupted System Files, download, due, everything, exe, Go Check Settings Windows Update Check, Manually Reset Windows Update Components, Pc, Rename Update Folders, Restart Services, Search, setup, Stability Features, Stop Update Services, store, type, updates, Windows Installation Assistant, Windows Logs Setup
Posted in OS Update, Security, System Administration, Windows | No Comments »

Using GeoIP on Linux: Country-Based Filtering, Logging, and Traffic Control

Friday, January 16th, 2026

GeoIP is one of those technologies that quietly sits in the background of many systems, yet it can be extremely powerful when used correctly. Whether you want to block traffic from specific countries, analyze access logs, or add geographic context to security events, GeoIP can be a valuable addition to your Linux toolbox.

In this article, we’ll go deeper and look at real GeoIP usage examples for:

Log analysis
Firewalls
Apache HTTP Server
HAProxy

All examples are based on typical GNU/Linux server environments.

What Is GeoIP?

GeoIP maps an IP address to geographic data such as:

Country
City
ASN / ISP (depending on database)

Most modern systems use MaxMind GeoLite2 databases (

.mmdb

format).

Keep in Mind ! :
GeoIP data is approximate. VPNs, proxies, mobile networks, and CGNAT reduce accuracy. GeoIP should be treated as a heuristic, not a guarantee.

1. Installing GeoIP Databases on Linux deb based distro

On Debian / Ubuntu:

#

apt install geoipupdate

Configure

/etc/GeoIP.conf

with your MaxMind license key and run:

# geoipupdate

Databases are usually stored in:

/usr/share/GeoIP/

2. GeoIP for Log Analysis (to get idea of where does your traffic origins from)

GeoIP with Apache HTTP Server

Apache can use GeoIP in two main ways:

To do IP origin Logging
Do Access control based on IP origin

An altenartive GeoIP common use is to post-processing logs to find out attempts to breach your security.

Lets say you want to

Find top attacking countries against your SSHd service.

# grep "Failed password" /var/log/auth.log | \
awk '{print $(NF-3)}' | \
while read ip; do geoiplookup $ip; done |
\ sort | uniq -c | sort -nr

This command will provide you a visibility on attack sources georaphical Country origin

3. Installing Apache GeoIP Module

For legacy GeoIP (older systems):

# apt install libapache2-mod-geoip

For modern systems, GeoIP2 is preferred:

# apt install libapache2-mod-geoip2

Enable the module:

# a2enmod geoip2
# systemctl reload apache2

4. Configure GeoIP Logging in Apache (basic config)

Add country code to access logs:

LogFormat "%h %l %u %t \"%r\" %>s %b %{GEOIP_COUNTRY_CODE}e" geoip
CustomLog /var/log/apache2/access.log geoip

This allows you to analyze traffic by country later without blocking users.

5. Country-Based Filter Blocking in Apache based on IP origin

Example: allow only selected countries:

<IfModule mod_geoip2.c>
SetEnvIf GEOIP_COUNTRY_CODE ^(BG|DE)$ AllowCountry
Deny from all
Allow from env=AllowCountry
</IfModule>

Use this carefully. Blocking at the web server layer is better than firewall-level blocking, but still risky if you have global users.

6. Apply GeoIP to Apache Virtual Host

You can apply GeoIP rules per site:

<VirtualHost *:80>
ServerName example.com
<IfModule mod_geoip2.c>
SetEnvIf GEOIP_COUNTRY_CODE CN BlockCountry >
Deny from env=BlockCountry
</IfModule>
</VirtualHost>

This is useful when only specific applications need filtering.

Firewall vs Application Layer GeoIP (Pros and Cons)

Layer	Pros	Cons
Firewall	Early blocking	Hard to debug
Apache	Flexible per-site rules	App overhead
HAProxy	Centralized control	Requires careful config
Logs only	Safest	No blocking

7. Apply GeoIP to HAProxy

HAProxy is an excellent place to apply GeoIP logic because:

It sits in front of applications
Rules are fast and explicit
Logging is centralized

a. Preparing GeoIP Filtering to HAProxy service

HAProxy supports GeoIP2 via Lua or native ACLs using

.mmdb

Example directory:

/usr/share/GeoIP/GeoLite2-Country.mmdb

b. GeoIP-Based Access Control Lists ( ACLs ) in HAProxy

Basic country-based blocking:

frontend http_in
bind *:80

acl from_china src -m geoip CN
acl from_russia src -m geoip RU

http-request deny if from_china
http-request deny if from_russia

default_backend web_servers

This blocks traffic early, before it hits Apache or nginx.

c. GeoIP-Based Routing across different haproxy backends

Instead of blocking, you can route traffic differently:

acl eu_users src -m geoip DE FR NL
use_backend eu_backend if eu_users
default_backend global_backend

This is useful for:

Geo-based load balancing
Regional content
Legal compliance separation

d. GeoIP Logging config for HAProxy

Add country code to logs:

log-format "%ci:%cp [%t] %ft %b %s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC"

(%CC = country code)

This makes traffic analysis extremely efficient.

Keep in Mind !

Use HAProxy or web server level for enforcement, and firewall GeoIP only when absolutely necessary.

8. Fail2ban + GeoIP: Smarter Bans, Better Context

Fail2ban is excellent at reacting to abusive behavior, but by default it only sees IP addresses, not where they come from. Adding GeoIP allows you to:

Tag bans with country information
Apply different ban policies per region
Detect unusual behavior patterns

a. GeoIP-Enriched Fail2ban Logs

Fail2ban itself doesn’t natively evaluate GeoIP rules, but you can enrich logs post-ban.

Example action script (

/etc/fail2ban/action.d/geoip-notify.conf

[Definition]
actionban = echo "Banned from $(geoiplookup | cut -d: -f2)" >> /var/log/fail2ban-geoip.log
Enable it in a jail:
[sshd]
enabled = true
action = iptables[name=SSH] geoip-notify

Enable it in a jail:

[sshd]

enabled = true action = iptables[name=SSH] geoip-notify

Resulting log entry:

Banned 185.220.101.1 from Germany

This provides visibility without changing ban logic — a safe first step.

b. Use GeoIP-Aware Ban Policies

You can also adjust ban times based on country.

Example strategy:

Short ban for local country
Longer ban for known high-noise regions

This is usually implemented via multiple jails and post-processing scripts rather than direct GeoIP matching inside Fail2ban.

Best practice:
Let Fail2ban do behavior detection — let GeoIP provide context, not decisions.

9. GeoIP with nftables (Linux Modern Firewall Layer)

iptables +

xt_geoip

is considered legacy. On modern systems, nftables is the preferred approach.

a. Using GeoIP Sets in nftables

nftables does not natively include GeoIP, but you can integrate GeoIP via generated IP sets.

Workflow:

Convert GeoIP country IP ranges into nftables sets
Load them dynamically

Example set definition:

table inet filter {
set geo_block {
type ipv4_addr
flags interval
}
}

Populate the set using a script:

nft add element inet filter geo_block { 1.2.3.0/24, 5.6.0.0/16 }

Then apply it:

chain input {
type filter hook input priority 0;
ip saddr @geo_block drop
}

b. Automating GeoIP -> nftables

Typical automation pipeline:

GeoLite2 → country CSV → IP ranges → nftables set

Run this daily via cron.

Warning:

Large country sets = memory usage
Firewall reloads must be atomic
Test on non-production systems first

10. GeoIP Dashboard: Turning Logs into Insight

Blocking is optional — insight is mandatory.

a. Simple GeoIP Log Dashboard (CLI-Based)

Apache example:

# awk '{print $NF}' /var/log/apache2/access.log | \
sort | uniq -c | sort -nr

Where $NF contains country code.

Sample Result:

1243 US

987 DE

422 FR

310 CN

This already tells a story.

b. Visual Dashboard with ELK / Grafana

For larger environments:

HAProxy / Apache -> JSON logs Enrich logs with GeoIP

Send to:

ELK Stack
Loki + Grafana
Graylog

Metrics you want:

Requests per country
Errors per country
Bans per country
Login failures per country

This helps distinguish:

Marketing traffic
Legit users
Background Internet noise

11. Create a Layered GeoIP Strategy

A sane, production-ready model using GeoIP would include something like:

Logging first
Apache / HAProxy logs with country codes
Behavior detection
Fail2ban reacts to abuse
Traffic shaping
HAProxy routes or rate-limits
Firewall last
nftables drops only obvious garbage

GeoIP is strongest when it supports decisions, not when it makes them alone.

12. Best Practices to consider

Prefer visibility over blocking
Avoid blanket country bans
Always log before denying

Combine GeoIP with:

Fail2ban
Rate limits
CAPTCHA or MFA
Keep GeoIP databases (regularly) updated
Test rules with real IPs before deploying

13. Common Mistakes to Avoid

Blocking entire continents Using GeoIP as authentication Applying firewall GeoIP without logs Forgetting database updates Assuming GeoIP accuracy

Close up

GeoIP is not a silver bullet against vampire attacks – but when used thoughtfully, it becomes a powerful signal enhancer and can give you a much broader understanding on what is going on inside your network traffic.

Whether you’re using it to filter out segment of evil intruders based on logs, routing traffic intelligently, or filtering obvious abusea, GeoIP fits naturally into a layered security model and is used across corporations and middle and even small sized businesses nowadays.

Used conservatively, GeoIP follows the classic Unix philosophy:

Small datasets, Simple rules, Real-world effectiveness, combined with rest of tools it gives info and ways to protect better your networks and server infra.

Tags: Apache Virtual Host, authentication, Better Context, config, country, databases, Detect, distro, find, germany, information, jail, Legal, Load, logs, network traffic, Result, run, separation, Short, site, Smarter Bans, updates, Warning
Posted in Everyday Life | No Comments »

Enable Debian Linux automatic updates to keep latest OS Patches / Security Up to Date

Monday, January 13th, 2025

Enable Debian Linux automatic updates to keep latest OS Patches / Security Up to Date

I'm not a big fan of automatism on GNU / Linux as often using automatic updates could totally mess things especially with a complex and a bit chatic OS-es like is Linux nowadays.
Nevertheless as Security is becoming more and more of a problem especially the browser security, having a scheduled way to apply updates like every normal modern Windows and MAC OS as an option is becoming essential to have a fully manageble Operating system.

As I use Debian GNU / Linux for desktop for my own personal computer and I have already a lot of Debian servers, whose OS minor level and package version maintenance takes up too big chunk of my time (a time I could dedicated to more useful activities). Thus I found it worthy at some cases to trigger Debian's way to keep the OS and security at a present level, the so called Debian "unattended upgrades".

In this article, I'll explain how to install and Enable Automatic (" Unattended " ) Updates on Debian, with the hope that other Debian users might start benefiting from it.

Pros of enabling automatic updates, are:

Debian OS Stay secure without constant monitoring.
You Save much time by letting your system handle updates.
Presumably Enjoying more peace of mind, knowing your system is more protected.

Cons of enabling automatic updates:

Some exotic and bad maintained packages (might break after the update)
Customizations made on the OS /etc/sysctl.conf or any other very custom server configs might disappear or not work after the update
At worst scenario (a very rare but possible case) OS might fail to boot after update 🙂

Regular security updates patch vulnerabilities that could otherwise be exploited by attackers, which is especially important for servers and systems exposed to the internet, where threats evolve constantly.

1. Update Debian System to latest

Before applying automatic updates making any changes, run apt to update package lists and upgrade any outdated packages,to have automatic updates for a smooth configuration process.

# apt update && apt upgrade -y

2. Install the Unattended-Upgrades deb Package

# apt install unattended-upgrades -y

Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following additional packages will be installed:
distro-info-data gir1.2-glib-2.0 iso-codes libgirepository-1.0-1 lsb-release python-apt-common python3-apt python3-dbus python3-distro-info python3-gi
Suggested packages:
isoquery python-apt-doc python-dbus-doc needrestart powermgmt-base
The following NEW packages will be installed:
distro-info-data gir1.2-glib-2.0 iso-codes libgirepository-1.0-1 lsb-release python-apt-common python3-apt python3-dbus python3-distro-info python3-gi unattended-upgrades
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,786 kB of archives.
After this operation, 24.4 MB of additional disk space will be used.
Do you want to continue? [Y/n]

# apt install apt-listchanges
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following package was automatically installed and is no longer required:
linux-image-5.10.0-30-amd64
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
python3-debconf
The following NEW packages will be installed:
apt-listchanges python3-debconf
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 137 kB of archives.
After this operation, 452 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian bookworm/main amd64 python3-debconf all 1.5.82 [3,980 B]
Get:2 http://deb.debian.org/debian bookworm/main amd64 apt-listchanges all 3.24 [133 kB]
Fetched 137 kB in 0s (292 kB/s)
Preconfiguring packages …
Deferring configuration of apt-listchanges until /usr/bin/python3
and python's debconf module are available
Selecting previously unselected package python3-debconf.
(Reading database … 84582 files and directories currently installed.)
Preparing to unpack …/python3-debconf_1.5.82_all.deb …
Unpacking python3-debconf (1.5.82) …
Selecting previously unselected package apt-listchanges.
Preparing to unpack …/apt-listchanges_3.24_all.deb …
Unpacking apt-listchanges (3.24) …
Setting up python3-debconf (1.5.82) …
Setting up apt-listchanges (3.24) …

Creating config file /etc/apt/listchanges.conf with new version

Example config for apt-listchanges would be like:

# vim /etc/apt/listchanges.conf
[apt]
frontend=pager
email_address=root
confirm=0
save_seen=/var/lib/apt/listchanges.db
which=both

3. Enable Automatic unattended upgrades

Once installed, enable automatic updates with the following command, which will prompt asking if you want to enable automatic updates. Select Yes and press Enter, which will confirm that the unattended-upgrades service is active and ready to manage updates for you.

# dpkg-reconfigure unattended-upgrades

Or non-interactively by running command:

# echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true | debconf-set-selections
dpkg-reconfigure -f noninteractive unattended-upgrades

4. Set the Schedule for Automatic Updates on Debian

By default, unattended-upgrades runs daily, to verify or modify the schedule, check the systemd timer:

# sudo systemctl status apt-daily.timer
# sudo systemctl status apt-daily-upgrade.timer
# systemctl edit apt-daily-upgrade.timer

Current apt-daily.timer config as of Debian 12 (bookworm) is as follows

root@haproxy2:/etc/apt/apt.conf.d# cat /lib/systemd/system/apt-daily.timer
[Unit]
Description=Daily apt download activities

[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true

[Install]
WantedBy=timers.target
root@haproxy2:/etc/apt/apt.conf.d#

# systemctl edit apt-daily-upgrade.timer

[Timer]
OnCalendar=
OnCalendar=03:00
RandomizedDelaySec=0

At Line num 2 above is needed to reset (empty) the default value shown below in line num 5.
Line 4 is needed to prevent any random delays coming from the defaults.

Now both timers should be active, if not, activate them with:

# systemctl enable –now apt-daily.timer
# systemctl enable –now apt-daily-upgrade.timer

These timers ensure that updates are checked and applied regularly, without manual intervention.

5.Test one time Automatic Updates on Debian works

To ensure everything is working, simulate an unattended upgrade with a dry run:

# unattended-upgrade –dry-run

You can monitor automatic updates by checking the logs.

# less /var/log/unattended-upgrades/unattended-upgrades.log

Log shows details of installed updates and any issues that occurred. Reviewing logs periodically can help you ensure that updates are being applied correctly and troubleshoot any problems.

6. Advanced Configuration Options

If you’re a power user or managing multiple systems, you might want to explore these additional settings in the configuration file:

# vim /etc/apt/apt.conf.d/50unattended-upgrades

Configure unattended-upgrades to send you an email whenever updates are installed.

Unattended-Upgrade::Mail "your-email-address@email-address.com";

Enable automatic reboots after kernel updates
by adding the line:

Unattended-Upgrade::Automatic-Reboot "true";

To schedule reboots after package upgrade is applied at a specific time:

Unattended-Upgrade::Automatic-Reboot-Time "02:00";

Specify packages you don’t want to be updated by editing the Unattended-Upgrade::Package-Blacklist section in the configuration file.

Here is alternative way to configure the unattended upgrade, by using apt configuration options:

# vim /etc/apt/apt.conf.d/02periodic

// Control parameters for cron jobs by /etc/cron.daily/apt-compat //

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade –download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "21";

// Send report mail to root
// 0: no report (or null string)
// 1: progress report (actually any string)
// 2: + command outputs (remove -qq, remove 2>/dev/null, add -d)
// 3: + trace on
APT::Periodic::Verbose "2";

If you have to simultaneously update multiple machines and you're on a limited connection line, configure download limits if you’re on a metered connection by setting options in /etc/apt/apt.conf.d/20auto-upgrades.

7. Stop Automatic Unattended Upgrade

Under some circumstances if it happens the unattended upgrades are no longer required and you want to revert back to manual package updates, to disable the updates you have to disable the unattended-upgrades service

# systemctl stop unattended-upgrades

8. Stop an ongoing apt deb package set of updates applied on Debian server

Perhaps not often, but it might be you have run an automated upgrade and this has broke a server system or a service and for that reason you would like to stop the upcoming upgrade (some of whose might have started on other servers) immediately, to do so, the easiest way (not always safe thogh) is to kill the unattended-upgrades daemon.

# pkill –signal SIGKILL unattended-upgrades

Note that this a very brutal way to kill it and that might lead to some broken package update, that you might have to later fix manually.

If you have the unattended-upgrade process running on the OS in the process list backgrounded and you want to stop the being on the fly upgrade on the system more safely for the system, you can stop and cancel the ongoing apt upgrade it by running the ncurses prompt interface, through dpkg-reconfigure

# dpkg-reconfigure unattended-upgrades

Then just select No, press Enter. In my case, this has promptly stopped the ongoing unattended upgrade that seemed blocked (at least as promptly as the hardware seemed to allow 🙂 ).

If you want to disable it for future, so it doesn't automatically gets enabled on next manual update, by some update script disable service as well.

# systemctl disable unattended-upgrades
…

Close up

That’s all ! Now, your Debian system will automatically handle security updates, keeping your system secure without you having to do a thing.
The same guide should be good for most Deb based distributions such as Ubuntu / Mint and there rest of other Debian derivative OS-es.
You’ve now set up a reliable way to ensure your system stays protected from vulnerabilities, but anyways it is a good practice to always login and check what the update has done to the system, otherwise expect the unexpected.

Tags: Advanced Configuration Options, background, close, configure, dependency tree, Description Daily, disable, Done, Enable Debian Linux, installed, level, logs, MAC, packages, peace, possible, reading package, root, running, security, servers, setting, state information, sudo, system, systemctl, time, timer, updates, upgrade, without
Posted in Computer Security, Curious Facts, Linux, Linux and FreeBSD Desktop, Linux on Laptops, Various | No Comments »

yum search file in all installable RPM, find out which rpm package provides binary file or missing library dependency on CentOS / RHEL / Fedora

Friday, August 23rd, 2024

Sometimes if you have a missing library or a file you know should be available via an rpm but you're not sure which RPM you have to install you have to look up for library or binary file amongs all available installable r[ms on Redhat Linux / CentOS / Fedora or other RPM based distro.

It is really annoying especially, if you try to install an rpm binary and the package does not install due to missing dependency library. Having a missing dependency package could happen, if you use some custom internal prepared repository that is mirroring from original rpm repositories and the RPM Repositories are situated behind a DMZ firewall network (such scenarios are common for corporations and IT companies).

Finding out which file is provided by which package on Debian / Ubuntu and other deb based linux distributions is easy and done via the

# apt-file search filename

Thus if you're a system administrator coming from a Debian GNU / Linux sysadmin realm into the wonderful world of redhats, you will want to have an alternative to apt-file tool. You will be happy to find out that that this tedious task is easily done in RPM based Linux and is integrated straight into yum package manager too.

The command to search which rpm package provides a file is:

# yum whatprovides 'nc'

[root@rhel-linux ~]# yum whatprovides nc
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
2:nmap-ncat-6.40-19.el7.x86_64 : Nmap's Netcat replacement
Repo : base
Matched from:
Provides : nc

2:nmap-ncat-6.40-19.el7.x86_64 : Nmap's Netcat replacement
Repo : @base
Matched from:
Provides : nc

yum whatprovides search_file_name can be also invoked with its shortcut yum provides 'search_file_name'

[root@rhel-server ~]# yum provides '/bin/ls'
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
coreutils-8.22-24.el7.x86_64 : A set of basic GNU tools commonly used in shell scripts
Repo : base
Matched from:
Filename : /bin/ls

coreutils-8.22-24.el7_9.2.x86_64 : A set of basic GNU tools commonly used in shell scripts
Repo : updates
Matched from:
Filename : /bin/ls

Here is another example:

[root@rhel-server ~]# yum -q provides '*lesspipe.sh*'
less-458-9.el7.x86_64 : A text file browser similar to more, but better
Repo : base
Matched from:
Filename : /usr/bin/lesspipe.sh

source-highlight-3.1.6-6.el7.i686 : Produces a document with syntax highlighting
Repo : base
Matched from:
Filename : /usr/bin/src-hilite-lesspipe.sh

source-highlight-3.1.6-6.el7.x86_64 : Produces a document with syntax highlighting
Repo : base
Matched from:
Filename : /usr/bin/src-hilite-lesspipe.sh

spirv-tools-2019.1-4.el7.x86_64 : API and commands for processing SPIR-V modules
Repo : epel
Matched from:
Filename : /usr/bin/spirv-lesspipe.sh

You can search for any file and if the RPm repository is defined under /etc/yum/repos.d/* and enabled, yum whatprovides command should be able to find it and tell you which RPM package you have to install to have the file installed Redhat way.

You can list all enabled RPM repositories with cmd:

[root@rhel-server ~]# yum repolist enabled
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
repo id repo name status
3party Third party packages – x86_64 2,631
base/7/x86_64 CentOS-7 – Base 10,072
cr/7/x86_64 CentOS-7 – CR 0
epel/7/x86_64 EPEL packages for RedCent 7 – x86_64 13,791
extras/7/x86_64 CentOS-7 – Extras 526
updates/7/x86_64 CentOS-7 – Updates 5,802
zabbix-6.0 Zabbix 6.0 repo 429
repolist: 33,251

To list disable RPM repositories:

# yum repolist disabled
…

To list all present available repositories that could be enabled and are set via the /etc/yum.repos.d/* configs

# yum repolist all
…

Tags: API, available, cmd, dependency, distro, document, find, lib, list, loading, mirror, package, rpm, updates, yum
Posted in Linux, Linux Package Management, OS Update, RedHat, System Administration | No Comments »

Install specific zabbix-agent version / Downgrade Zabbix Agent client to exact preferred old RPM version on CentOS / Fedora / RHEL Linux from repo

Wednesday, June 7th, 2023

In below article, I'll give you the short Update zabbix procedure to specific version release, if you need to have it running in tandem with rest of zabbix infra, as well as expain shortly how to downgrade zabbix version to a specific release number
to match your central zabbix-serveror central zabbix proxies.

The article is based on personal experience how to install / downgrade the specific zabbuix-agent release on RPM based distros.
I know this is pretty trivial stuff but still, hope this might be useful to some sysadmin out there thus I decided to quickly blog it.

1. Prepare backup of zabbix_agentd.conf

# cp -rpf /etc/zabbix/zabbix_agentd.conf /home/your-user/zabbix_agentd.conf.bak.$(date +"%b-%d-%Y")

2. Create zabbix repo source file in yum.repos.d directory

# cd /etc/yum.repos.d/
# vim zabbix.repo

[zabbix-5.0]

name=Zabbix 5.0 repo

baseurl=http://zabixx-rpm-mirrors-site.com/centos/external/zabbix-5.0/8/x86_64/

enabled=1

gpgcheck=0

3. Update zabbix-agent to a specific defined version

# yum search zabbix-agent –enablerepo zabbix-5.0

To update zabbix-agent for RHEL 7.*

# yum install zabbix-agent-5.0.34-1.el7.x86_64

For RHEL 8.*

# yum install zabbix-agent-5.0.34-1.el8.x86_64

4. Restart zabbix-agentd and check its status to make sure it works correctly

# systemctl status zabbix-agentd
# systemctl restart zabbix-agentd
# systemctl status zabbix-agentd

Go to zabbix-server WEB GUI interface and check that data is delivered as normally in Latest Data for the host fom recent time, to make sure host monitoring is continuing flawlessly as before change.

NB !: If yum use something like versionlock is enabled remove the versionlock for package and update then, otherwise it will (weirldly look) look like the package is missing.
I'm saying that because I've hit this issue and was wondering why i cannot install the zabbix-agent even though the version is listed, available and downloadable from the repository.

5. Downgrade agent-client to specific version (Install old version of Zabbix from Repo)

Sometimes by mistake you might have raised the Zabbix-agent version to be higher release than the zabbix-server's version and thus breach out the Zabbix documentation official recommendation to keep
up the zabbix-proxy, zabbix-server and zabbix-agent at the exactly same version major and minor version releases.

If so, then you would want to decrease / downgrade the version, to match your Zabbix overall infrastructure exact version for each of Zabbix server -> Zabbix Proxy server -> Agent clients.

To downgrade the version, I prefer to create some backups, just in case for all /etc/zabbix/ configurations and userparameter scripts (from experience this is useful as sometimes some RPM binary update packages might cause /etc/zabbix/zabbix_agentd.conf file to get overwritten. To prevent from restoring zabbix_agentd.conf from your most recent backup hence, I prefer to just crease the zabbix config backups manually.

# cd /root

# mkdir -p /root/backup/zabbix-agent

# tar -czvf zabbix_agent.tar.gz /etc/zabbix/

# tar -xzvf zabbix_agent.tar.gz

Then list the available installable zabbix-agent versions

[root@sysadminshelp:~]# yum –showduplicates list zabbix-agent
Заредени плъгини: fastestmirror
Determining fastest mirrors
* base: centos.uni-sofia.bg
* epel: fedora.ipacct.com
* extras: centos.uni-sofia.bg
* remi: mirrors.uni-ruse.bg
* remi-php74: mirrors.uni-ruse.bg
* remi-safe: mirrors.uni-ruse.bg
* updates: centos.uni-sofia.bg
Инсталирани пакети
zabbix-agent.x86_64 5.0.30-1.el7 @zabbix
Налични пакети
zabbix-agent.x86_64 5.0.0-1.el7 zabbix
zabbix-agent.x86_64 5.0.1-1.el7 zabbix
zabbix-agent.x86_64 5.0.2-1.el7 zabbix
zabbix-agent.x86_64 5.0.3-1.el7 zabbix
zabbix-agent.x86_64 5.0.4-1.el7 zabbix
zabbix-agent.x86_64 5.0.5-1.el7 zabbix
zabbix-agent.x86_64 5.0.6-1.el7 zabbix
zabbix-agent.x86_64 5.0.7-1.el7 zabbix
zabbix-agent.x86_64 5.0.8-1.el7 zabbix
zabbix-agent.x86_64 5.0.9-1.el7 zabbix
zabbix-agent.x86_64 5.0.10-1.el7 zabbix
zabbix-agent.x86_64 5.0.11-1.el7 zabbix
zabbix-agent.x86_64 5.0.12-1.el7 zabbix
zabbix-agent.x86_64 5.0.13-1.el7 zabbix
zabbix-agent.x86_64 5.0.14-1.el7 zabbix
zabbix-agent.x86_64 5.0.15-1.el7 zabbix
zabbix-agent.x86_64 5.0.16-1.el7 zabbix
zabbix-agent.x86_64 5.0.17-1.el7 zabbix
zabbix-agent.x86_64 5.0.18-1.el7 zabbix
zabbix-agent.x86_64 5.0.19-1.el7 zabbix
zabbix-agent.x86_64 5.0.20-1.el7 zabbix
zabbix-agent.x86_64 5.0.21-1.el7 zabbix
zabbix-agent.x86_64 5.0.22-1.el7 zabbix
zabbix-agent.x86_64 5.0.23-1.el7 zabbix
zabbix-agent.x86_64 5.0.24-1.el7 zabbix
zabbix-agent.x86_64 5.0.25-1.el7 zabbix
zabbix-agent.x86_64 5.0.26-1.el7 zabbix
zabbix-agent.x86_64 5.0.27-1.el7 zabbix
zabbix-agent.x86_64 5.0.28-1.el7 zabbix
zabbix-agent.x86_64 5.0.29-1.el7 zabbix
zabbix-agent.x86_64 5.0.30-1.el7 zabbix
zabbix-agent.x86_64 5.0.31-1.el7 zabbix
zabbix-agent.x86_64 5.0.32-1.el7 zabbix
zabbix-agent.x86_64 5.0.33-1.el7 zabbix
zabbix-agent.x86_64 5.0.34-1.el7 zabbix

Next lets install the most recent zabbix-versoin from the CentOS repo, which for me as of time of writting this article is 5.0.34.

# yum downgrade -y zabbix-agent-5.0.34-1.el7

# cp -rpf /root/backup/zabbix-agent/etc/zabbix/zabbix_agentd.conf /etc/zabbix/

# systemctl start zabbix-agent.service

# systemctl enable zabbix-agent.service

# zabbix_agentd -V
zabbix_agentd (daemon) (Zabbix) 5.0.30
Revision 2c96c38fb4b 28 November 2022, compilation time: Nov 28 2022 11:27:43

Copyright (C) 2022 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 1.0.1e-fips 11 Feb 2013
Running with OpenSSL 1.0.1e-fips 11 Feb 2013

That's all folks you should be at your custom selected preferred version of zabbix-agent.
Enjoy ! 🙂

Tags: bg, config, Downgrade Zabbix Agent, Restart, specific, systemctl, updates, version, yum, zabbix-agent
Posted in Linux, Linux Package Management, Monitoring, Various, Zabbix | 2 Comments »

How to RPM update Hypervisors and Virtual Machines running Haproxy High Availability cluster on KVM, Virtuozzo without a downtime on RHEL / CentOS Linux

Friday, May 20th, 2022

Here is the scenario, lets say you have on your daily task list two Hypervisor (HV) hosts running CentOS or RHEL Linux with KVM or Virutozzo technology and inside the HV hosts you have configured at least 2 pairs of virtual machines one residing on HV Host 1 and one residing on HV Host 2 and you need to constantly keep the hosts to the latest distribution major release security patchset.

The Virtual Machines has been running another set of Redhat Linux or CentOS configured to work in a High Availability Cluster running Haproxy / Apache / Postfix or any other kind of HA solution on top of corosync / keepalived or whatever application cluster scripts Free or Open Source technology that supports a switch between clustered Application nodes.

The logical question comes how to keep up the CentOS / RHEL Machines uptodate without interfering with the operations of the Applications running on the cluster?

Assuming that the 2 or more machines are configured to run in Active / Passive App member mode, e.g. one machine is Active at any time and the other is always Passive, a switch is possible between the Active and Passive node.

HAProxy--Load-Balancer-cluster-2-nodes-your-Servers

In this article I'll give a simple step by step tested example on how you I succeeded to update (for security reasons) up to the latest available Distribution major release patchset on one by one first the Clustered App on Virtual Machines 1 and VM2 on Linux Hypervisor Host 1. Then the App cluster VM 1 / VM 2 on Hypervisor Host 2.
And finally update the Hypervisor1 (after moving the Active resources from it to Hypervisor2) and updating the Hypervisor2 after moving the App running resources back on HV1.
I know the procedure is a bit monotonic but it tries to go through everything step by step to try to mitigate any possible problems. In case of failure of some rpm dependencies during yum / dnf tool updates you can always revert to backups so in anyways don't forget to have a fully functional backup of each of the HV hosts and the VMs somewhere on a separate machine before proceeding further, any possible failures due to following my aritcle literally is your responsibility 🙂

0. Check situation before the update on HVs / get VM IDs etc.

Check the virsion of each of the machines to be updated both Hypervisor and Hosted VMs, on each machine run:

# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

The machine setup I'll be dealing with is as follows:

hypervisor-host1 -> hypervisor-host1.fqdn.com
• virt-mach-centos1
• virt-machine-zabbix-proxy-centos (zabbix proxy)

hypervisor-host2 -> hypervisor-host2.fqdn.com
• virt-mach-centos2
• virt-machine-zabbix2-proxy-centos (zabbix proxy)

To check what is yours check out with virsh cmd –if on KVM or with prlctl if using Virutozzo, you should get something like:

[root@hypervisor-host2 ~]# virsh list
Id Name State
—————————————————-
1 vm-host1 running
2 virt-mach-centos2 running

# virsh list –all

[root@hypervisor-host1 ~]# virsh list
Id Name State
—————————————————-
1 vm-host2 running
3 virt-mach-centos1 running

[root@hypervisor-host1 ~]# prlctl list
UUID STATUS IP_ADDR T NAME
{dc37c201-08c9-589d-aa20-9386d63ce3f3} running – VM virt-mach-centos1
{76e8a5f8-caa8-5442-830e-aa4bfe8d42d9} running – VM vm-host2
[root@hypervisor-host1 ~]#

If you have stopped VMs with Virtuozzo to list the stopped ones as well.

# prlctl list -a

[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# vzlist -a
CTID NPROC STATUS IP_ADDR HOSTNAME
[root@hypervisor-host2 74a7bbe8-9245-5385-ac0d-d10299100789]# prlctl list
UUID STATUS IP_ADDR T NAME
{92075803-a4ce-5ec0-a3d8-9ee83d85fc76} running – VM virt-mach-centos2
{74a7bbe8-9245-5385-ac0d-d10299100789} running – VM vm-host1

# prlctl list -a

If due to Virtuozzo version above command does not return you can manually check in the VM located folder, VM ID etc.

[root@hypervisor-host2 vmprivate]# ls
74a7bbe8-9245-4385-ac0d-d10299100789 92075803-a4ce-4ec0-a3d8-9ee83d85fc76
[root@hypervisor-host2 vmprivate]# pwd
/vz/vmprivate
[root@hypervisor-host2 vmprivate]#

[root@hypervisor-host1 ~]# ls -al /vz/vmprivate/
total 20
drwxr-x—. 5 root root 4096 Feb 14 2019 .
drwxr-xr-x. 7 root root 4096 Feb 13 2019 ..
drwxr-x–x. 4 root root 4096 Feb 18 2019 1c863dfc-1deb-493c-820f-3005a0457627
drwxr-x–x. 4 root root 4096 Feb 14 2019 76e8a5f8-caa8-4442-830e-aa4bfe8d42d9
drwxr-x–x. 4 root root 4096 Feb 14 2019 dc37c201-08c9-489d-aa20-9386d63ce3f3
[root@hypervisor-host1 ~]#

Before doing anything with the VMs, also don't forget to check the Hypervisor hosts has enough space, otherwise you'll get in big troubles !

[root@hypervisor-host2 vmprivate]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos_hypervisor-host2-root 20G 1.8G 17G 10% /
devtmpfs 20G 0 20G 0% /dev
tmpfs 20G 0 20G 0% /dev/shm
tmpfs 20G 2.0G 18G 11% /run
tmpfs 20G 0 20G 0% /sys/fs/cgroup
/dev/sda1 992M 159M 766M 18% /boot
/dev/mapper/centos_hypervisor-host2-home 9.8G 37M 9.2G 1% /home
/dev/mapper/centos_hypervisor-host2-var 9.8G 355M 8.9G 4% /var
/dev/mapper/centos_hypervisor-host2-vz 755G 25G 692G 4% /vz

[root@hypervisor-host1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 1.8G 45G 4% /
devtmpfs 20G 0 20G 0% /dev
tmpfs 20G 0 20G 0% /dev/shm
tmpfs 20G 2.1G 18G 11% /run
tmpfs 20G 0 20G 0% /sys/fs/cgroup
/dev/sda2 992M 153M 772M 17% /boot
/dev/mapper/centos-home 9.8G 37M 9.2G 1% /home
/dev/mapper/centos-var 9.8G 406M 8.9G 5% /var
/dev/mapper/centos-vz 689G 12G 643G 2% /vz

Another thing to do before proceeding with update is to check and tune if needed the amount of CentOS repositories used, before doing anything with yum.

[root@hypervisor-host2 yum.repos.d]# ls -al
total 68
drwxr-xr-x. 2 root root 4096 Oct 6 13:13 .
drwxr-xr-x. 110 root root 12288 Oct 7 11:13 ..
-rw-r–r–. 1 root root 4382 Mar 14 2019 CentOS7.repo
-rw-r–r–. 1 root root 1664 Sep 5 2019 CentOS-Base.repo
-rw-r–r–. 1 root root 1309 Sep 5 2019 CentOS-CR.repo
-rw-r–r–. 1 root root 649 Sep 5 2019 CentOS-Debuginfo.repo
-rw-r–r–. 1 root root 314 Sep 5 2019 CentOS-fasttrack.repo
-rw-r–r–. 1 root root 630 Sep 5 2019 CentOS-Media.repo
-rw-r–r–. 1 root root 1331 Sep 5 2019 CentOS-Sources.repo
-rw-r–r–. 1 root root 6639 Sep 5 2019 CentOS-Vault.repo
-rw-r–r–. 1 root root 1303 Mar 14 2019 factory.repo
-rw-r–r–. 1 root root 666 Sep 8 10:13 openvz.repo
[root@hypervisor-host2 yum.repos.d]#

[root@hypervisor-host1 yum.repos.d]# ls -al
total 68
drwxr-xr-x. 2 root root 4096 Oct 6 13:13 .
drwxr-xr-x. 112 root root 12288 Oct 7 11:09 ..
-rw-r–r–. 1 root root 1664 Sep 5 2019 CentOS-Base.repo
-rw-r–r–. 1 root root 1309 Sep 5 2019 CentOS-CR.repo
-rw-r–r–. 1 root root 649 Sep 5 2019 CentOS-Debuginfo.repo
-rw-r–r–. 1 root root 314 Sep 5 2019 CentOS-fasttrack.repo
-rw-r–r–. 1 root root 630 Sep 5 2019 CentOS-Media.repo
-rw-r–r–. 1 root root 1331 Sep 5 2019 CentOS-Sources.repo
-rw-r–r–. 1 root root 6639 Sep 5 2019 CentOS-Vault.repo
-rw-r–r–. 1 root root 1303 Mar 14 2019 factory.repo
-rw-r–r–. 1 root root 300 Mar 14 2019 obsoleted_tmpls.repo
-rw-r–r–. 1 root root 666 Sep 8 10:13 openvz.repo

1. Dump VM definition XMs (to have it in case if it gets wiped during update)

There is always a possibility that something will fail during the update and you might be unable to restore back to the old version of the Virtual Machine due to some config misconfigurations or whatever thus a very good idea, before proceeding to modify the working VMs is to use KVM's virsh and dump the exact set of XML configuration that makes the VM roll properly.

To do so:
Check a little bit up in the article how we have listed the IDs that are part of the directory containing the VM.

[root@hypervisor-host1 ]# virsh dumpxml (Id of VM virt-mach-centos1 ) > /root/virt-mach-centos1_config_bak.xml
[root@hypervisor-host2 ]# virsh dumpxml (Id of VM virt-mach-centos2) > /root/virt-mach-centos2_config_bak.xml

2. Set on standby virt-mach-centos1 (virt-mach-centos1)

As I'm upgrading two machines that are configured to run an haproxy corosync cluster, before proceeding to update the active host, we have to switch off
the proxied traffic from node1 to node2, – e.g. standby the active node, so the cluster can move up the traffic to other available node.

[root@virt-mach-centos1 ~]# pcs cluster standby virt-mach-centos1

3. Stop VM virt-mach-centos1 & backup on Hypervisor host (hypervisor-host1) for VM1

Another prevention step to make sure you don't get into damaged VM or broken haproxy cluster after the upgrade is to of course backup

[root@hypervisor-host1 ]# prlctl backup virt-mach-centos1

[root@hypervisor-host1 ]# prlctl stop virt-mach-centos1
[root@hypervisor-host1 ]# cp -rpf /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3 /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3-bak
[root@hypervisor-host1 ]# tar -czvf virt-mach-centos1_vm_virt-mach-centos1.tar.gz /vz/vmprivate/dc37c201-08c9-489d-aa20-9386d63ce3f3

[root@hypervisor-host1 ]# prlctl start virt-mach-centos1

4. Remove package version locks on all hosts

If you're using package locking to prevent some other colleague to not accidently upgrade the machine (if multiple sysadmins are managing the host), you might use the RPM package locking meachanism, if that is used check RPM packs that are locked and release the locking.

+ List actual list of locked packages

[root@hypervisor-host1 ]# yum versionlock list
…
…..
0:libtalloc-2.1.16-1.el7.*
0:libedit-3.0-12.20121213cvs.el7.*
0:p11-kit-trust-0.23.5-3.el7.*
1:quota-nls-4.01-19.el7.*
0:perl-Exporter-5.68-3.el7.*
0:sudo-1.8.23-9.el7.*
0:libxslt-1.1.28-5.el7.*
versionlock list done

+ Clear the locking

# yum versionlock clear

+ List actual list / == clear all entries

[root@virt-mach-centos2 ]# yum versionlock list; yum versionlock clear
[root@virt-mach-centos1 ]# yum versionlock list; yum versionlock clear
[root@hypervisor-host1 ~]# yum versionlock list; yum versionlock clear
[root@hypervisor-host2 ~]# yum versionlock list; yum versionlock clear

5. Do yum update virt-mach-centos1

For some clarity if something goes wrong, it is really a good idea to make a dump of the basic packages installed before the RPM package update is initiated,
The exact versoin of RHEL or CentOS as well as the list of locked packages, if locking is used.

Enter virt-mach-centos1 (ssh virt-mach-centos1) and run following cmds:

# cat /etc/redhat-release > /root/logs/redhat-release-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

+ Only if needed!!

# yum versionlock clear
# yum versionlock list

Clear any previous RPM packages – careful with that as you might want to keep the old RPMs, if unsure comment out below line

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Proceed with the update and monitor closely the output of commands and log out everything inside files using a small script that you should place under /root/status the script is given at the end of the aritcle.:

yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
yum check-update | wc -l
yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

6. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

7. Stop VM virt-mach-centos2 & backup on Hypervisor host (hypervisor-host2)

Same backup step as prior

# prlctl backup virt-mach-centos2

# prlctl stop virt-mach-centos2
# cp -rpf /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76 /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76-bak
## tar -czvf virt-mach-centos2_vm_virt-mach-centos2.tar.gz /vz/vmprivate/92075803-a4ce-4ec0-a3d8-9ee83d85fc76

# prctl start virt-mach-centos2

8. Do yum update on virt-mach-centos2

Log system state, before the update

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list

+ Clean old install update / packages if required

# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Initiate the update

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

9. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

10. Stop VM vm-host2 & backup

# prlctl backup vm-host2

# prlctl stop vm-host2

Or copy the actual directory containig the Virtozzo VM (use the correct ID)

# cp -rpf /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9 /vz/vmprivate/76e8a5f8-caa8-5442-830e-aa4bfe8d42d9-bak
## tar -czvf vm-host2.tar.gz /vz/vmprivate/76e8a5f8-caa8-4442-830e-aa5bfe8d42d9

# prctl start vm-host2

11. Do yum update vm-host2

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear only if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Do the rpm upgrade

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

12. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

13. Do yum update hypervisor-host2

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock if needed

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Update rpms

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out 2>&1
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

14. Stop VM vm-host1 & backup

Some as ealier

# prlctl backup vm-host1

# prlctl stop vm-host1

# cp -rpf /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789 /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789-bak
# tar -czvf vm-host1.tar.gz /vz/vmprivate/74a7bbe8-9245-4385-ac0d-d10299100789

# prctl start vm-host1

15. Do yum update vm-host2

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum versionlock clear == if needed!!
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

16. Check if everything is running fine after upgrade

+ Reboot VM

# shutdown -r now

17. Do yum update hypervisor-host1

Same procedure for HV host 1

# cat /etc/redhat-release > /root/logs/redhat-release-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# cat /etc/grub.d/30_os-prober > /root/logs/grub2-efi-vorher-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

Clear lock

# yum versionlock clear
# yum versionlock list
# yum clean all |tee /root/logs/yumcleanall-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

# yum check-update |tee /root/logs/yumcheckupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# yum check-update | wc -l
# yum update |tee /root/logs/yumupdate-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out
# sh /root/status |tee /root/logs/status-before-$(hostname)-$(date '+%Y-%m-%d_%H-%M-%S').out

18. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

Check hypervisor-host1 all VMs run as expected

19. Check if everything is running fine after upgrade

Reboot VM

# shutdown -r now

Check hypervisor-host2 all VMs run as expected afterwards

20. Check once more VMs and haproxy or any other contained services in VMs run as expected

21. Haproxy Unstandby virt-mach-centos1

Assuming that the virt-mach-centos1 and virt-mach-centos2 are running a Haproxy / corosync cluster you can try to standby node1 and check the result
hopefully all should be fine and traffic should come to host node2.

[root@virt-mach-centos1 ~]# pcs cluster unstandby virt-mach-centos1

Monitor logs and make sure HAproxy works fine on virt-mach-centos1

22. If necessery to redefine VMs (in case they disappear from virsh) or virtuosso is not working

[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos1_config_bak.xml
[root@virt-mach-centos1 ]# virsh define /root/virt-mach-centos2_config_bak.xml

23. Set versionlock to RPMs to prevent accident updates and check OS version release

[root@virt-mach-centos2 ]# yum versionlock \*
[root@virt-mach-centos1 ]# yum versionlock \*
[root@hypervisor-host1 ~]# yum versionlock \*
[root@hypervisor-host2 ~]# yum versionlock \*

[root@hypervisor-host2 ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)

Other useful hints

[root@hypervisor-host1 ~]# virsh console dc37c201-08c9-489d-aa20-9386d63ce3f3
Connected to domain virt-mach-centos1
..

! Compare packages count before the upgrade on each of the supposable identical VMs and HVs – if there is difference in package count review what kind of packages are different and try to make the machines to look as identical as possible !

Packages to update on hypervisor-host1 Count: XXX
Packages to update on hypervisor-host2 Count: XXX
Packages to update virt-mach-centos1 Count: – 254
Packages to update virt-mach-centos2 Count: – 249

The /root/status script

+++

#!/bin/sh
echo '======================================================= '
echo '= Systemctl list-unit-files –type=service | grep enabled '
echo '======================================================= '
systemctl list-unit-files –type=service | grep enabled

echo '======================================================= '
echo '= systemctl | grep ".service" | grep "running" '
echo '======================================================= '
systemctl | grep ".service" | grep "running"

echo '======================================================= '
echo '= chkconfig –list '
echo '======================================================= '
chkconfig –list

echo '======================================================= '
echo '= netstat -tulpn '
echo '======================================================= '
netstat -tulpn

echo '======================================================= '
echo '= netstat -r '
echo '======================================================= '
netstat -r

+++

That's all folks, once going through the article, after some 2 hours of efforts or so you should have an up2date machines.
Any problems faced or feedback is mostly welcome as this might help others who have the same setup.

Thanks for reading me 🙂

Tags: backups, check, cmds, Connected, downtime, echo, Linux Hypervisor Host, list, logs, repo, root root, rpm, Set, setup, updates, virtual machines, yum versionlock
Posted in CentOS, Linux, Linux Package Management, System Administration, Virtual Machines | 1 Comment »

Install and configure rkhunter for improved security on a PCI DSS Linux / BSD servers with no access to Internet

Wednesday, November 10th, 2021

rkhunter or Rootkit Hunter scans systems for known and unknown rootkits. The tool is not new and most system administrators that has to mantain some good security servers perhaps already use it in their daily sysadmin tasks.

It does this by comparing SHA-1 Hashes of important files with known good ones in online databases, searching for default directories (of rootkits), wrong permissions, hidden files, suspicious strings in kernel modules, commmon backdoors, sniffers and exploits as well as other special tests mostly for Linux and FreeBSD though a ports for other UNIX operating systems like Solaris etc. are perhaps available. rkhunter is notable due to its inclusion in popular mainstream FOSS operating systems (CentOS, Fedora,Debian, Ubuntu etc.).

Even though rkhunter is not rapidly improved over the last 3 years (its last Official version release was on 20th of Febuary 2018), it is a good tool that helps to strengthen even further security and it is often a requirement for Unix servers systems that should follow the PCI DSS Standards (Payment Card Industry Data Security Standards).

Configuring rkhunter is a pretty straight forward if you don't have too much requirements but I decided to write this article for the reason there are fwe interesting options that you might want to adopt in configuration to whitelist any files that are reported as Warnings, as well as how to set a configuration that sets a stricter security checks than the installation defaults.

1. Install rkhunter .deb / .rpm package depending on the Linux distro or BSD

If you have to place it on a Redhat based distro CentOS / Redhat / Fedora

[root@Centos ~]# yum install -y rkhunter
…

On Debian distros the package name is equevallent to install there exec usual:

root@debian:~# apt install –yes rkhunter
…

On FreeBSD / NetBSD or other BSD forks you can install it from the BSD "World" ports system or install it from a precompiled binary.

freebsd# pkg install rkhunter
…

One important note to make here is to have a fully functional Alarming from rkhunter, you will have to have a fully functional configured postfix / exim / qmail whatever mail server to relay via official SMTP so you the Warning Alarm emails be able to reach your preferred Alarm email address. If you haven't installed postfix for example and configure it you might do.

– On Deb based distros

[root@Centos ~]#yum install postfix

– On RPM based distros

root@debian:~# apt-get install –yes postfix

and as minimum, further on configure some functional Email Relay server within /etc/postfix/main.cf

# vi /etc/postfix/main.cf
relayhost = [relay.smtp-server.com]

2. Prepare rkhunter.conf initial configuration

Depending on what kind of files are present on the filesystem it could be for some reasons some standard package binaries has to be excluded for verification, because they possess unusual permissions because of manual sys admin monification this is done with the rkhunter variable PKGMGR_NO_VRFY.

If remote logging is configured on the system via something like rsyslog you will want to specificly tell it to rkhunter so this check as a possible security issue is skipped via ALLOW_SYSLOG_REMOTE_LOGGING=1.

In case if remote root login via SSH protocol is disabled via /etc/ssh/sshd_config
PermitRootLogin no variable, the variable to include is ALLOW_SSH_ROOT_USER=no

It is useful to also increase the hashing check algorithm for security default one SHA256 you might want to change to SHA512, this is done via rkhunter.conf var HASH_CMD=SHA512

Triggering new email Warnings has to be configured so you receive, new mails at a preconfigured mailbox of your choice via variable
MAIL-ON-WARNING=SetMailAddress

# vi /etc/rkhunter.conf

PKGMGR_NO_VRFY=/usr/bin/su

PKGMGR_NO_VRFY=/usr/bin/passwd

ALLOW_SYSLOG_REMOTE_LOGGING=1

# Needed for corosync/pacemaker since update 19.11.2020

ALLOWDEVFILE=/dev/shm/qb-*/qb-*

# enabled ssh root access skip

ALLOW_SSH_ROOT_USER=no

HASH_CMD=SHA512

# Email address to sent alert in case of Warnings

MAIL-ON-WARNING=Your-Customer@Your-Email-Server-Destination-Address.com

MAIL-ON-WARNING=Your-Second-Peronsl-Email-Address@SMTP-Server.com

DISABLE_TESTS=os_specific

Optionally if you're using something specific such as corosync / pacemaker High Availability cluster or some specific software that is creating /dev/ files identified as potential Risks you might want to add more rkhunter.conf options like:

# Allow PCS/Pacemaker/Corosync
ALLOWDEVFILE=/dev/shm/qb-attrd-*
ALLOWDEVFILE=/dev/shm/qb-cfg-*
ALLOWDEVFILE=/dev/shm/qb-cib_rw-*
ALLOWDEVFILE=/dev/shm/qb-cib_shm-*
ALLOWDEVFILE=/dev/shm/qb-corosync-*
ALLOWDEVFILE=/dev/shm/qb-cpg-*
ALLOWDEVFILE=/dev/shm/qb-lrmd-*
ALLOWDEVFILE=/dev/shm/qb-pengine-*
ALLOWDEVFILE=/dev/shm/qb-quorum-*
ALLOWDEVFILE=/dev/shm/qb-stonith-*
ALLOWDEVFILE=/dev/shm/pulse-shm-*
ALLOWDEVFILE=/dev/md/md-device-map
# Needed for corosync/pacemaker since update 19.11.2020
ALLOWDEVFILE=/dev/shm/qb-*/qb-*

# tomboy creates this one
ALLOWDEVFILE="/dev/shm/mono.*"
# created by libv4l
ALLOWDEVFILE="/dev/shm/libv4l-*"
# created by spice video
ALLOWDEVFILE="/dev/shm/spice.*"
# created by mdadm
ALLOWDEVFILE="/dev/md/autorebuild.pid"
# 389 Directory Server
ALLOWDEVFILE=/dev/shm/sem.slapd-*.stats
# squid proxy
ALLOWDEVFILE=/dev/shm/squid-cf*
# squid ssl cache
ALLOWDEVFILE=/dev/shm/squid-ssl_session_cache.shm
# Allow podman
ALLOWDEVFILE=/dev/shm/libpod*lock*

3. Set the proper mirror database URL location to internal network repository

Usually file /var/lib/rkhunter/db/mirrors.dat does contain Internet server address where latest version of mirrors.dat could be fetched, below is how it looks by default on Debian 10 Linux.

root@debian:/var/lib/rkhunter/db# cat mirrors.dat
Version:2007060601
mirror=http://rkhunter.sourceforge.net
mirror=http://rkhunter.sourceforge.net

As you can guess a machine that doesn't have access to the Internet neither directly, neither via some kind of secure proxy because it is in a Paranoic Demilitarized Zone (DMZ) Network with many firewalls. What you can do then is setup another Mirror server (Apache / Nginx) within the local PCI secured LAN that gets regularly the database from official database on http://rkhunter.sourceforge.net/ (by installing and running rkhunter –update command on the Mirror WebServer and copying data under some directory structure on the remote local LAN accessible server, to keep the DB uptodate you might want to setup a cron to periodically copy latest available rkhunter database towards the http://mirror-url/path-folder/)

# vi /var/lib/rkhunter/db/mirrors.dat

local=http://rkhunter-url-mirror-server-url.com/rkhunter/1.4/

A mirror copy of entire db files from Debian 10.8 ( Buster ) ready for download are here.

Update entire file property db and check for rkhunter db updates

# rkhunter –update && rkhunter –propupdate

[ Rootkit Hunter version 1.4.6 ]

Checking rkhunter data files…
Checking file mirrors.dat [ Skipped ]
Checking file programs_bad.dat [ No update ]
Checking file backdoorports.dat [ No update ]
Checking file suspscan.dat [ No update ]
Checking file i18n/cn [ No update ]
Checking file i18n/de [ No update ]
Checking file i18n/en [ No update ]
Checking file i18n/tr [ No update ]
Checking file i18n/tr.utf8 [ No update ]
Checking file i18n/zh [ No update ]
Checking file i18n/zh.utf8 [ No update ]
Checking file i18n/ja [ No update ]

rkhunter-update-propupdate-screenshot-centos-linux

4. Initiate a first time check and see whether something is not triggering Warnings

# rkhunter –check

As you might have to run the rkhunter multiple times, there is annoying Press Enter prompt, between checks. The idea of it is that you're able to inspect what went on but since usually, inspecting /var/log/rkhunter/rkhunter.log is much more easier, I prefer to skip this with –skip-keypress option.

# rkhunter –check –skip-keypress

5. Whitelist additional files and dev triggering false warnings alerts

You have to keep in mind many files which are considered to not be officially PCI compatible and potentially dangerous such as lynx browser curl, telnet etc. might trigger Warning, after checking them thoroughfully with some AntiVirus software such as Clamav and checking the MD5 checksum compared to a clean installed .deb / .rpm package on another RootKit, Virus, Spyware etc. Clean system (be it virtual machine or a Testing / Staging) machine you might want to simply whitelist the files which are incorrectly detected as dangerous for the system security.

Again this can be achieved with

PKGMGR_NO_VRFY=

Some Cluster softwares that are preparing their own /dev/ temporary files such as Pacemaker / Corosync might also trigger alarms, so you might want to suppress this as well with ALLOWDEVFILE

ALLOWDEVFILE=/dev/shm/qb-*/qb-*

If Warnings are found check what is the issue and if necessery white list files due to incorrect permissions in /etc/rkhunter.conf .

rkhunter-warnings-found-screenshot

Re-run the check until all appears clean as in below screenshot.

rkhunter-clean-report-linux-screenshot

Fixing Checking for a system logging configuration file [ Warning ]

If you happen to get some message like, message appears when rkhunter -C is done on legacy CentOS release 6.10 (Final) servers:

[13:45:29] Checking for a system logging configuration file [ Warning ]
[13:45:29] Warning: The 'systemd-journald' daemon is running, but no configuration file can be found.
[13:45:29] Checking if syslog remote logging is allowed [ Allowed ]

To fix it, you will have to disable SYSLOG_CONFIG_FILE at all.

SYSLOG_CONFIG_FILE=NONE

Tags: access, ALLOWDEVFILE, checking, com, configure, created, default, Fixing Checking, freebsd, howto, Install, installation, Internet, lib, linux?, logging, mirror, pci, postfix, repository, rkhunter, rkhunter add dev, rkhunter.conf, security, shm, something, sourceforge, updates, var, Warnings, whitelist
Posted in Computer Security, Educational, Linux, System Administration | 2 Comments »

Linux: Howto Fix “N: Repository ‘http://deb.debian.org/debian buster InRelease’ changed its ‘Version’ value from ‘10.9’ to ‘10.10’” error to resolve apt-get release update issue

Friday, August 13th, 2021

Linux's surprises and disorganization is continuously growing day by day and I start to realize it is becoming mostly impossible to support easily this piece of hackware bundled together.
Usually so far during the last 5 – 7 years, I rarely had any general issues with using:

apt-get update && apt-get upgrade && apt-get dist-upgrade

to raise a server's working stable Debian Linux version packages e.g. version X.Y to verzion X.Z (for example up the release from Debian Jessie from 8.1 to 8.2).

Today I just tried to follow this well known and established procedure that, of course nowdays is better to be done with the newer "apt" command instead with the legacy "apt-get"
And the set of

# apt-get update && apt-get upgrade && apt-get dist-upgrade

has triggered below shitty error:

root@zabbix:~# apt-get update && apt-get upgrade
Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
Get:3 http://security.debian.org buster/updates/non-free Sources [688 B]
Get:4 http://repo.zabbix.com/zabbix/5.0/debian buster InRelease [7096 B]
Get:5 http://security.debian.org buster/updates/main Sources [198 kB]
Get:6 http://security.debian.org buster/updates/main amd64 Packages [300 kB]
Get:7 http://security.debian.org buster/updates/main Translation-en [157 kB]
Get:8 http://security.debian.org buster/updates/non-free amd64 Packages [556 B]
Get:9 http://deb.debian.org/debian buster/main Sources [7836 kB]
Get:10 http://repo.zabbix.com/zabbix/5.0/debian buster/main Sources [1192 B]
Get:11 http://repo.zabbix.com/zabbix/5.0/debian buster/main amd64 Packages [4785 B]
Get:12 http://deb.debian.org/debian buster/non-free Sources [85.7 kB]
Get:13 http://deb.debian.org/debian buster/contrib Sources [42.5 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:15 http://deb.debian.org/debian buster/main Translation-en [5968 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 Contents (deb) [37.3 MB]
Get:17 http://deb.debian.org/debian buster/contrib amd64 Packages [50.1 kB]
Get:18 http://deb.debian.org/debian buster/non-free amd64 Packages [87.7 kB]
Get:19 http://deb.debian.org/debian buster/non-free Translation-en [88.9 kB]
Get:20 http://deb.debian.org/debian buster/non-free amd64 Contents (deb) [861 kB]
Fetched 61.1 MB in 22s (2774 kB/s)
Reading package lists… Done
N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'
…

As I used to realize nowdays, as Linux started originally as 'Hackers' operating system, its legacy is just one big hack and everything from simple maintenance up to the higher and more sophisticated things requires a workaround 'hack''.

This time the hack to resolve error:

N: Repository 'http://deb.debian.org/debian buster InRelease' changed its 'Version' value from '10.9' to '10.10'

is up to running cmd:

debian-server:~# apt-get update –allow-releaseinfo-change
Поп:1 http://ftp.de.debian.org/debian buster-backports InRelease
Поп:2 http://ftp.debian.org/debian stable InRelease
Поп:3 http://security.debian.org stable/updates InRelease
Изт:5 https://packages.sury.org/php buster InRelease [6837 B]
Изт:6 https://download.docker.com/linux/debian stretch InRelease [44,8 kB]
Изт:7 https://packages.sury.org/php buster/main amd64 Packages [317 kB]
Игн:4 https://attic.owncloud.org/download/repositories/production/Debian_10 InRelease
Изт:8 https://download.owncloud.org/download/repositories/production/Debian_10 Release [964 B]
Изт:9 https://packages.sury.org/php buster/main i386 Packages [314 kB]
Изт:10 https://download.owncloud.org/download/repositories/production/Debian_10 Release.gpg [481 B]
Грш:10 https://download.owncloud.org/download/repositories/production/Debian_10 Release.gpg
Следните подписи са невалидни: DDA2C105C4B73A6649AD2BBD47AE7F72479BC94B
Грш:11 https://ookla.bintray.com/debian generic InRelease
403 Forbidden [IP: 52.39.193.126 443]
Четене на списъците с пакети… Готово
N: Repository 'https://packages.sury.org/php buster InRelease' changed its 'Suite' value from '' to 'buster'
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://download.owncloud.org/download/repositories/production/Debian_10 Release:
…

apt-get-update-allow-releaseinfo-change-debian-linux-screenshot

Onwards to upgrade the system up to the latest .deb packages, as usual run:

# apt-get -y update && apt-get upgrade -y
…

and updates should be applied as usual with some prompts on whether you prefer to keep or replace existing service configuration and some information on some general changes that might affect your installed services. In a few minutes and few prompts hopefully your Debian OS should be up to the latest stable.

Tags: apt-get, cmd, command, complete, deb, download, issues, Linux Fix Repository, org, owncloud, packages, repositories, security, updates, zabbix
Posted in Linux, System Administration, Various | No Comments »

Clean up /var/cache/yum constantly filling file system in CentOS and RHEL Linux

Tuesday, November 3rd, 2020

centos-rhel-yum-clean-var-cache-yum
At work recently we hit one of those common system admin problem on a server installed by some colleagues who have weirdly set a very small /var/cache/yum separate partitioned spaced at only 10Gb on a Redhat Enterprise Linux 7.8 Linux. The OS install guys original thinking was that this machine is to be used just for a very simple cluster and to run a Web Hosting services, a Haproxy Clustered service with PCS + Heartbeat and Corosync, but they did not estimated properly, that redhat comes its own rhnsd and updates gets pushed frequently to the machine and the lack of cleaning up the updated / installed rpms would soon leave the system into a full /var/cache/yum partition.

The gradual increase with time even though with few megabytes each and every month is normal behavior as the cache would increase its size based on the frequency of syncing with the yum server.

On this specific machines besides the rhnsd there is also a spacewalk (Free and Open Source systems management) which usually is installed to orchestrate multiple installed systems to centralize installation updates and monitoring of rpm packages.
spacewalk also has its own way to produce its own files in /var/cache/yum with creating a repository clones of defined repository directories.

1. Clean up /var/cache/yum space with yum

The normal way to clean up space as we all know is then to do something with yum:

[user@rhel ~]$ yum clean
Loaded plugins: fastestmirror, langpacks
Error: clean requires an option: headers, packages, metadata, dbcache, plugins, expire-cache, rpmdb, all
[user@rhel ~]$

Usually to clean up a bit of space and get rid of partition full problem you can do:

[user@rhel ~]# yum clean metadata

This cleans up any xml metadata that may have been cached from any enabled repository.

[user@rhel ~]# yum clean all

This should clean up all rpms and metadata.

In cases where this works you should frequently re-run it on the machine or set it on a cron job to periodically reduce space occupied by useless downloaded .rpms.

2. Delete /var/cache/yum directly

If this does not help to clean up space another approach is to directly delete /var/cache/yum dir and recreate it.
To make sure you don't need any of the data that is already in /var/cache/yum for laters make a backup under some directory with free space first.

[user@rhel ~]# mv /var/cache/yum /home/user/
[user@rhel ~]# rm -rf /var/cache/yum
[user@rhel ~]# yum check-update

3. Disable Automatic RPM Updates

However if yum clean all does not work another approach is to completely disable automated updates, so you don't get surprises with unknown times when automatic update is triggered and /var/cache/yum is filled with stuff by the rhnsd or yum-updatesd.

[user@rhel ~]# service yum-updatesd stop
[user@rhel ~]# chkconfig yum-updatesd off

[user@rhel ~]# service rhnsd stop
[user@rhel ~]# chkconfig rhnsd off

4. Delete cache and rpm downloads right after install

By default yum retains the packages and package data files that it downloads, so that they may be reused in future operations without being downloaded again. This feature can be turned off by modifying the /etc/yum.conf file and changing the option "keepcache" to 0.

[user@rhel ~]$ grep -i keepcache /etc/yum.conf
keepcache=0

keepcache if set to 0, removes packages right after installation.
Setting keepcache=1 tells yum to keep the cache of headers and packages after a successful installation.

Tags: admin, after, again, ALL, and, another, ANY, automated, CentOS, cron job, file, installation, installed, machine, metadata, packages, repository, updates, var, yum
Posted in CentOS, Linux, System Administration | No Comments »

Upgrade Debian Linux 9 to 10 Stretch to Buster and Disable graphical service load boot on Debian 10 Linux / Debian Buster is out

Tuesday, July 9th, 2019

howto-upgrade-debian-linux-debian-stretch-to-buster-debian-10-buster

I've just took a time to upgrade my Debian 9 Stretch Linux to Debian Buster on my old school Laptop (that turned 11 years old) Lenovo Thinkpad R61 . The upgrade went more or less without severe issues except few things.

The overall procedure followed is described n a few websites out there already and comes up to;

0. Set the proper repository location in /etc/apt/sources.list

Before update the sources.list used are:

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free

deb [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free
deb-src [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free

deb http://ftp.debian.org/debian buster-backports main

For people that had stretch defined in /etc/apt/sources.list you should change them to buster or stable, easiest and quickest way to omit editting with vim / nano etc. is run as root or via sudo:

sed -i 's/stretch/buster/g' /etc/apt/sources.list
sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/*.list

The minimum of config in sources.list after the modification should be

deb http://deb.debian.org/debian buster main
deb http://deb.debian.org/debian buster-updates main
deb http://security.debian.org/debian-security buster/updates main

Or if you want to always be with latest stable packages (which is my practice for notebooks):

deb http://deb.debian.org/debian stable main
deb http://deb.debian.org/debian stable-updates main
deb http://security.debian.org/debian-security stable/updates main

1. Getting list of hold packages if such exist and unholding them, e.g.

apt-mark showhold

Same could also be done via dpkg

dpkg –get-selections | grep hold

To unhold a package if such is found:

echo "package_name install"|sudo dpkg –set-selections

For those who don't know what hold package is this is usually package you want to keep at certain version all the time even though after running apt-get upgrade to get the latest package versions.

2. Use df -h and assure you have at least 5 – 10 GB free space on root directory / before proceed

df -h /

3. Update packages list to set new set repos as default

apt update

4. apt upgrade

apt upgrade

Here some 10 – 15 times you have to confirm what you want to do with configuration that has changed if you're unsure about the config (and it is not critical service) you're aware as such as Apache / MySQL / SMTP etc. it is best to install the latest maintainer version.

Hopefully here you will not get fatal errors that will interrupt it.

P.S. It is best to run apt-update either in VTTY (Virtual console session) with screen or tmux or via a physical tty (if this is not a remote server) as during the updates your GUI access to the gnome-terminal or konsole / xterm whatever console used might get cut. Thus it is best to do it with command:

screen apt upgrade

5. Run dist-upgrade to finalize the upgrade from Stertch to Buster

Once all is completed of the new installed packages, you will need to finally do, once again it is best to run via screen, if you don't have installed screen install it:

if [ $(which screen) ]; then echo 'Installed'; else apt-get install –yes screen ; fi

screen apt dist-upgrade

Here once again you should set whether old configuration to some e services has to stay or the new Debian maintainer package shipped one will overwrite the old and locally modified (due to some reason), here do wisely whatever you will otherwise some configured services might not boot as expected on next boot.

6. What if you get packages failed on update

If you get a certain package failed to configure after installed due to some reason, if it is a systemd service use:

journalctl -xe |head -n 50

or fully observer output of journalctl -xe and decide on yourself.

In most cases

dpkg-reconfigure failed-package-name

should do the trick or at least give you more hints on how to solve it.

Also if a package seems to be in inconsistent or broken state after upgrade and simple dpkg-reconfigure doesn't help, a good command
that can help you is

dpkg-reconfigure -f package_name

or you can try to workaround a failed package setup with:

dpkg –configure -a

If dpkg-reconfigure doesn't help either as I experienced in prior of Debian from Debian 6 -> 7 an Debian 7 ->8 updates on some Computers, then a very useful thing to try is:

apt-get update –fix-missing

apt-get install -f

At certain cases the only work around to be able to complete the package upgrade is to to remove the package with apt remove but due to config errors even that is not possible to work around this as final resort run:

dpkg –remove –force-remove-reinstreq

7. Clean up ununeeded packages

Some packages are left over due to package dependencies from Stretch and not needed in buster anymore to remove them.

apt autoremove

8. Reboot system once all upgrade is over

/sbin/reboot

9. Verify your just upgraded Debian is in a good state

root@noah:~# uname -a;
Linux noah 4.19.0-5-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

root@noah:~# cat /etc/issue.net
Debian GNU/Linux 10

root@noah:~# lsb_release -a
No LSB modules are available.
Distributor ID:   Debian
Description:   Debian GNU/Linux 10 (buster)
Release:   10
Codename:   buster

root@noah:~# hostnamectl
   Static hostname: noah
         Icon name: computer-laptop
           Chassis: laptop
        Machine ID: 4759d9c2f20265938692146351a07929
           Boot ID: 256eb64ffa5e413b8f959f7ef43d919f
Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-rt-amd64
      Architecture: x86-64

10. Remove annoying picture short animation with debian logo looping

By default Debian 10 boots up with annoying screen hiding all the status of loaded services state .e.g. you cannot see the services that shows in [ FAILED ] state and which do show as [ OK ] to revert back the old behavior I'm used to for historical reasons and as it shows a lot of good Boot time debugging info, in previous Debian distributions this was possible by setting the right configuration options in /etc/default/grub

which so far in my config was like so

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y zswap.enabled=1 text"

Note that zswap.enabled=1 passed option is because my notebook is pretty old machine from 2008 with 4GB of memory and zswap does accelerate performance when working with swap – especially helpful on Older PCs for more you can read more about zswap on ArchLinux wiki
After modifying this configuration to load the new config into grub the cmd is:

/usr/sbin/update-grub

As this was not working and tried number of reboots finally I found that annoying animated gif like picture shown up is caused by plymouth below is excerpts from Plymouth's manual page:

"The plymouth sends commands to a running plymouthd. This is used during the boot process to control the display of the graphical boot splash."

Plymouth has a set of themes one can set:

# plymouth-set-default-theme -l
futureprototype
details
futureprototype
joy
lines
moonlight
softwaves
spacefun
text
tribar

I tried to change that theme to make the boot process as text boot as I'm used to historically with cmd:

update-alternatives –config text.plymouth

As after reboot I hoped the PC will start booting in text but this does not happened so the final fix to turn back to textmode service boot was to completely remove plymouth

apt-get remove –yes plymouth

Tags: boot, boots, configured, dpkg, Hopefully, Linux Debian Buster, list, noah, org, Reboot, root directory, screen, services, setting, setup, shows, stable packages, things, turned, updates, Upgrade Debian Linux
Posted in Linux, Linux and FreeBSD Desktop, Linux on Laptops, Linux Package Management, Various | No Comments »

Posts Tagged ‘updates’

Why Windows Update Says Up to Date but Update Are missing Happens ?

1. Try PC Restart First

2. Run the Built-In Windows Update Troubleshooter

3. Manually Reset Windows Update Components

CleanUp SoftwareDistribution update cache folder is perhaps Most Effective FIX

Recommended Method (Play Safe)

N!B! Do NOT delete the folder while update services are running.

Step 2: Rename the Update Folder (Safer Than Deleting)

Rename Update Folders

Step 3: Restart Services

4. Use the Microsoft Update Catalog to Manually download recent applied Update

5. Use the Windows Installation Assistant

6. Check for Corrupted System Files

7. Make Sure You’re Not Paused or you are on a Metered connection

8. Check Your Windows Version Manually

9. Update your Video / Audio / Motherboard Chipsets and peripheral drivers to latest

10. Move catroot folder (to clean up Windows Update package signatures)

What is catroot2 ?

11. Perform an In-Place Repair Upgrade (Last Resort)

12. If none of these helps check Windows Logs for a clue

Summary / close up

What Is GeoIP?

1. Installing GeoIP Databases on Linux deb based distro

2. GeoIP for Log Analysis (to get idea of where does your traffic origins from)

Find top attacking countries against your SSHd service.

3. Installing Apache GeoIP Module

4. Configure GeoIP Logging in Apache (basic config)

5. Country-Based Filter Blocking in Apache based on IP origin

6. Apply GeoIP to Apache Virtual Host

Firewall vs Application Layer GeoIP (Pros and Cons)

7. Apply GeoIP to HAProxy

a. Preparing GeoIP Filtering to HAProxy service

b. GeoIP-Based Access Control Lists ( ACLs ) in HAProxy

c. GeoIP-Based Routing across different haproxy backends

d. GeoIP Logging config for HAProxy

Keep in Mind !

8. Fail2ban + GeoIP: Smarter Bans, Better Context

a. GeoIP-Enriched Fail2ban Logs

b. Use GeoIP-Aware Ban Policies

9. GeoIP with nftables (Linux Modern Firewall Layer)

a. Using GeoIP Sets in nftables

b. Automating GeoIP -> nftables

10. GeoIP Dashboard: Turning Logs into Insight

a. Simple GeoIP Log Dashboard (CLI-Based)

b. Visual Dashboard with ELK / Grafana

11. Create a Layered GeoIP Strategy

12. Best Practices to consider

13. Common Mistakes to Avoid

Close up

1. Update Debian System to latest

2. Install the Unattended-Upgrades deb Package

3. Enable Automatic unattended upgrades

4. Set the Schedule for Automatic Updates on Debian

5.Test one time Automatic Updates on Debian works

6. Advanced Configuration Options

7. Stop Automatic Unattended Upgrade

8. Stop an ongoing apt deb package set of updates applied on Debian server

Close up

2. Create zabbix repo source file in yum.repos.d directory

3. Update zabbix-agent to a specific defined version

4. Restart zabbix-agentd and check its status to make sure it works correctly

5. Downgrade agent-client to specific version (Install old version of Zabbix from Repo)

0. Check situation before the update on HVs / get VM IDs etc.

1. Dump VM definition XMs (to have it in case if it gets wiped during update)

2. Set on standby virt-mach-centos1 (virt-mach-centos1)

3. Stop VM virt-mach-centos1 & backup on Hypervisor host (hypervisor-host1) for VM1

4. Remove package version locks on all hosts

5. Do yum update virt-mach-centos1

6. Check if everything is running fine after upgrade

7. Stop VM virt-mach-centos2 & backup on Hypervisor host (hypervisor-host2)

8. Do yum update on virt-mach-centos2

9. Check if everything is running fine after upgrade

10. Stop VM vm-host2 & backup

11. Do yum update vm-host2

12. Check if everything is running fine after upgrade

13. Do yum update hypervisor-host2

14. Stop VM vm-host1 & backup

15. Do yum update vm-host2

16. Check if everything is running fine after upgrade

4. Use the Microsoft Update Catalog to Manually download recent applied
Update