Archive for November, 2025

How to keep your Linux server Healthy for Years: Hard learned lessons

Friday, November 28th, 2025

how-to-keep-your-linux-servers-healthy-every-year-doctor_tux

I’ve been running Linux servers long enough to watch hardware die, kernels panic, filesystems fill up at midnight hours, and network cards slowly burn out like old light bulbs.

Over time, you learn that keeping a server alive is less about “perfect architecture” and more about steady discipline – the small habits built to manage the machines, helps prevent big disasters.

Here are some practical, battle-tested lessons that keep my boxes running for years with minimal downtime. Most of them were learned the hard way.

1. Monitor Before You Fix – and Fix Before It Breaks

Most Linux disasters come from things we should have noticed earlier. The lack of monitoring, there is modern day saying that should become your favourite if you are a sysadmin or Dev Ops engineer.

"Monitoring everything !"

  • The disk that was at 89% yesterday will be at 100% tonight.
  • The log file that grew by 500 MB last week will explode this week.
  • The swap usage creeping from 1% → 5% → 20% means your next heavy task will choke.
  • The unseen failing BIOS CMOS battery
  • The RAID disks degradation etc.

You don’t need enterprise monitoring to prevent this. And even simple tools like monit or a simple zabbix-agent -> zabbix-server or any other simplistic scripted  monitoring gives you a basic issues pre-warning.

Even a simple cronjob shell one liner can save you hours of further sh!t :

#!/bin/bash

df -h / | awk 'NR==2 { if($5+0 > 85) print "Disk Alert: / is at " $5 }' \
| mail -s "Disk Warning on $(hostname)" admin@example.com

2. Treat /etc directory as Sacred – Treat It Like an expensive gem

Every sysadmin eventually faces the nightmare of a broken config overwritten by a package update or a hasty command at 2 AM.

To avoid crying later, archive /etc automatically:

# tar czf /root/etc-$(date +%Y-%m-%d).tar.gz /etc


If you prefer the backup to be more sophisticated you can use my clone of the dirs_backup.sh (an old script I wrote for easifying backup of specific directories on the filesystem ) the etc_backup.sh you can get here.
Run it weekly via cron.
This little trick has saved me more times than I can count — especially when migrating between Debian releases or recovering from accidental edits.

3. Automate everything what you have to repeatevely do

If you find yourself doing something manually more than twice, script it and forget it.

Examples:

  • rotating logs for misbehaving apps
  • restarting services that occasionally get “stuck”
  • syncing backups between machines
  • cleaning temp directories

Here’s a small example I still use today:

#!/bin/bash

# Kill zombie PHP-FPM children that keep leaking memory

ps aux | grep php-fpm | awk '{if($6 > 300000) print $2}' | xargs -r kill -9

Dirty way to get rid of misfunctioning php-fpm ?
Yes. But it works.

4. Backups Don’t Exist Unless You Test Them

It’s easy to feel proud when you write a backup script.
It’s harder – and far more important – to test the restore.

Once a month  or at least once in a few months, try restore a random backup to a dummy VM.
Sometimes backup might fails, or you might get something different from what you originally expected and by doing so
you can guarantee you will not cry later helplessly.

A broken backup doesn’t fail quietly – it fails on the day you need it most.

5. Don’t Ignore Hardware – It Ages like Everything Else

Linux might run forever, but hardware doesn’t.

Signs of impending doom:

  • dmesg spam with I/O errors
  • slow SSD response
  • increasing SMART reallocated sectors
  • random freezes without logs
  • sudden network flakiness

Run this monthly:

6. Document Everything (Future You Will Thank Past You)

There are moments when you ask yourself:

“Why did I configure this machine like this?”

If you don’t document your decisions, you’ll have no idea one year later.

A simple markdown file inside /root/notes.txt or /root/README.md is enough.

Document:

  • installed software
  • custom scripts
  • non-standard configs
  • firewall rules
  • weird hacks you probably forgot already

This turns chaos into something you can actually maintain.

7. Keep Things Simple – Complexity Is the Enemy of Uptime

The longer I work with servers, the more I strip away:

  • fewer moving parts
  • fewer services
  • fewer custom patches
  • fewer “temporary” hacks that become permanent

A simple system is a reliable system.
A complex one dies at the worst possible moment.

8. Accept That Failure Will Still Happen

No matter how careful you are, servers will surely:

  • crash
  • corrupt filesystems
  • lose network connectivity
  • inexplicably freeze
  • reboot after a kernel panic

Don’t aim for perfection.Aim for resilience.

If you can restore the machine in under an hour, you're winning and in the white.

Final Thoughts

Linux is powerful – but it rewards those who treat it with respect and perseverance.
Over many years, I’ve realized that maintaining servers is less about brilliance and more about humble, consistent care and hard work persistence.

I hope this article helps some sysamdmin to rethink and rebundle servers maintenance strategy in a way that will avoid a server meltdown at  night hours like 3 AM.

Cheers ! 

 

Tags: , , , , , , , , , , , , , ,
Posted in Linux, System Administration, Various | No Comments »

How I Stopped My AWS workspace Linux Desktop From Going to Sleep… Without Root Access

Thursday, November 20th, 2025

keeping-session-alive-stop-aws-workspace-to-auto-suspend-with-systemd-inhibit-or-a-simple-loop-scriptcover

If you've spent enough time around Linux servers and desktops, you already know one universal truth:

Linux never does exactly what you expect… especially when you don’t have root.

A few weeks ago, I found myself in a situation that’s probably familiar to anyone who works on shared servers, university machines, or restricted corporate environments:

My session kept going to sleep, killing long-running scripts, dropping SSH tunnels, freezing terminals—for absolutely no good reason.

To make things worse, I didn’t have sudo on this box.
No changing systemd settings, no tweaking /etc/systemd/logind.conf, and definitely no masking sleep targets.

So I went down the rabbit hole of how to keep a Linux machine awake without any superuser privileges.
Here’s the write-up of that journey—and yes, the final solution is surprisingly elegant.

The Problem: When the System Sleeps, Your Work Dies

My main issue: every 15 minutes of inactivity, the system would suspend the session.
Not the entire PC — just my user session. It didn't matter if I had background jobs running or SSH tunnels open; if I wasn’t interacting, the session was toast.

The machines were managed centrally, so root was a luxury I simply didn’t have.

What followed was a typical sysadmin debugging sequence:

  1. Angry at the stupidity
  2. Google.
  3. Try things that shouldn’t work.
  4. Try things that definitely shouldn't work.
  5. Accidentally discover the correct solution while reading some random docs if point 3 doesn’t already solve it

The Trick: systemd-inhibit (Works Without Root!)

While digging through the systemd documentation, I discovered something beautiful:

Non-root users can create inhibitor locks.

This means your normal user account can ask systemd:
“Please don’t put this session to sleep while this program is running.”

And systemd says:
“Okay. I respect that.”

All it takes is:

systemd-inhibit –what=handle-lid-switch:sleep –mode=block sleep infinity

This command runs a never-ending sleep process—
and while it runs, the system is forbidden to suspend.

You can even run it in the background:

$ nohup systemd-inhibit sleep infinity &

Want to verify it’s working?

$ systemd-inhibit –list

You’ll see your inhibitor lock listed like a VIP pass at a nightclub.

If You Have caffeinate, Even Better

Some Linux distros ship with a utility called caffeinate (similar to macOS).
It’s almost poetic:

$ caffeinate -di sleep infinity

This one also blocks sleep while the command runs.
Just leave it running as a background job and your session stays alive.

The Primitive but Always-Working Hack: Keepalive Script

If neither systemd-inhibit nor caffeinate exist, you can fall back to a caveman approach and still have the basic functionality of the Move Mouse Windows tool on Linux :):
 

#!/bin/bash

while true; do

    echo "Still here: $(date)"

    sleep 60

done

This prevents session idleness by emitting activity every minute.
Not elegant, but reliable.

Sometimes the caveman wins.

Why This Matters

Keeping a session awake might sound trivial, but for sysadmins, developers, pentesters, researchers, or anyone running long processes on managed machines, it’s a lifesaver.

You avoid:

  • broken SSH tunnels
  • silent failure of long-running scripts
  • GUI sessions locking themselves
  • losing tmux/screen sessions
  • interrupted compiles or renders
  • VPN disconnects

And you don’t need to bug IT or break policy.

Final Thoughts

What surprised me most is how simple the final solution was:

  • No root
  • No configuration changes
  • No hacks
  • No kernel tweaks

Just one systemd command used properly.

Sometimes Linux feels like an inscrutable labyrinth… and sometimes it gives you a quiet, elegant tool hiding in plain sight.

If you ever find yourself fighting unwanted auto-suspend on a machine you don’t control –
give systemd-inhibit a try.

Tags: , , , , , , , , ,
Posted in Hacks, Linux, Linux and FreeBSD Desktop | No Comments »

Generate and Use WiFi passwords via QR code on Linux

Monday, November 17th, 2025

If you’re running a WiFi network for a guest house / AirBNB, a small hotel or an office / home network and you’re tired of telling / dictating guests your long, complicated Wi-Fi password every time they visit, there’s a simple and secure way to share the WiFi name and password by putting the info in a WiFi authentication QR code. With a simple scan from their phone via a QR reader (which iPhones and all modern smartphones have), they can join your network instantly with zero typing.

In short — Why use a QR Code for Wi-Fi?

  • Convenience for guests: No more reading or spelling out your SSID and password.
  • Better security practice: Keep a strong password while still making access easy.
  • Works on most devices: Android and iOS both support Wi-Fi QR codes.

The Wi-Fi QR Code Standard Format

Smartphones understand Wi-Fi QR codes using this format:

WIFI:S:;T:;P:;;

  • S: your Wi-Fi network name (SSID)
  • T: encryption type (WPA, WEP, or nopass)
  • P: your Wi-Fi password
  • Everything ends with ;;

If your SSID or password contains special characters such as ; or \, they must be escaped using a backslash \.

Example:

SSID = WifiNetworkName
Password = @ss;word45123P4?#$!
Type = WPA

The final string becomes:

WIFI:S:WifiNetworkName;T:WPA;P:@ss\;word45123P4?#$!;;

How to Generate the QR Code Locally on Linux

You can generate the QR code locally using qrencode instead of uploading credentials online.

  1. Install qrencode (Debian/Ubuntu):

    # apt update # apt install qrencode

  2. Generate the QR code:

    # qrencode -o wifi-qr.png 'WIFI:S:WifiNetwork;T:WPA;P:P@ss\;word!;;'

    Example with colors and SVG output:

    # qrencode -t svg –foreground=FFFFFF –background=000000 \ -o qr-code.svg –inline –rle -s 10 \ "WIFI:S:MyWiFI;T:WPA;P:<Secret_Pass>;;"

    Use -s to increase size, e.g. -s 10.

  3. Print or share the generated wifi-qr.png.

Alternative: GUI / Desktop Approach

On GNOME-based systems (Ubuntu, Fedora), install the WiFi QR Code GNOME extension.

  • Open quick settings → Wi-Fi → “Show QR Code”
  • Right-click to copy or save for printing

Advanced: Using wifi-qr Tool

The wifi-qr tool provides CLI and GUI Wi-Fi QR generation.

# terminal interface to generate QR
# wifi-qr -t

wifi-qr screenshot

This command prints your current Wi-Fi SSID, encryption type, and password in clear text.

Security Tips

  • Limit where you display the QR
  • Rotate the password when needed
  • Use a guest network instead of your main one
  • Prefer WPA2/WPA3
  • Store QR images safely

How Guests Use the QR Code

  1. Open the phone camera (iOS/Android)
  2. Point it at the QR
  3. Tap “Join” when the Wi-Fi prompt appears

Final Thoughts

Generating a Wi-Fi QR code blends security with usability. Whether using qrencode, a GUI extension, or a website, guests connect instantly with a quick scan – no typing required.

Tags: , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, Linux on Laptops, Networking, Various | No Comments »

Migrating Server environments to Docker Containers a brief step-by-step guide

Wednesday, November 12th, 2025

migrating-server--applications-environment-to-docker-containers

In modern IT environments, containerization has become an essential strategy for improving application portability, scalability, and consistency. Docker, as a containerization platform, allows you to package applications and their dependencies into isolated containers that can be easily deployed across different environments. Migrating an existing server environment into Docker containers is a common scenario, and this guide will walk you through the key steps of doing so.

Why Migrate to Docker?

Before we dive into the specifics, let’s briefly understand why you might want to migrate your server environment to Docker:

  • Portability: Containers encapsulate applications and all their dependencies, making them portable across any system running Docker.
  • Scalability: Containers are lightweight and can be scaled up or down easily, offering flexibility to handle varying loads.
  • Consistency: With Docker, you can ensure that your application behaves the same in development, testing, and production.
  • Isolation: Docker containers run in isolation from the host system, minimizing the risk of configuration conflicts.
     

Steps to Migrate a Server Environment into Docker

Migrating an environment of servers into Docker typically involves several steps: evaluating the current setup, containerizing applications, managing dependencies, and orchestrating deployment. Here’s a breakdown:

1. Evaluate the Existing Server Environment

Before migrating, it's essential to inventory the current server environment to understand the following:

  • The applications running on the servers
  • Dependencies (e.g., databases, third-party services, libraries, etc.)
  • Networking setup (e.g., exposed ports, communication between services)
  • Storage requirements (e.g., persistent data, volumes)
  • Security concerns (e.g., user permissions, firewalls)
     

For example, if you're running a web server with a backend database and some caching layers, you'll need to break down these services into their constituent parts so they can be containerized.

2. Containerize the Application

The next step is to convert the services running on your server into Docker containers. Docker containers require a Dockerfile, which is a blueprint for how to build and run the container. Let's walk through an example of containerizing a simple web application.

Example: Migrating a Simple Node.js Web Application

Assume you have a Node.js application running on your server. To containerize it, you need to:

  • Create a Dockerfile
  • Build the Docker image
  • Run the containerized application

2.1. Write a Dockerfile

A Dockerfile defines how your application will be built within a Docker container. Here’s an example for a Node.js application:

# Step 1: Use the official Node.js image as a base image FROM node:16

# Step 2: Set the working directory inside the container WORKDIR /usr/src/app # Step 3: Copy package.json and package-lock.json to the container COPY package*.json ./

# Step 4: Install dependencies inside the container RUN npm install

# Step 5: Copy the rest of the application code to the container COPY . .

# Step 6: Expose the port that the application will listen on EXPOSE 3000

# Step 7: Start the application CMD ["npm", "start"]

This Dockerfile:

  1. Uses the official node:16 image as a base.
  2. Sets the working directory inside the container to /usr/src/app.
  3. Copies the package.json and package-lock.json files to the container and runs npm install to install dependencies.
  4. Copies the rest of the application code into the container.
  5. Exposes port 3000 (assuming that’s the port your app runs on).
  6. Defines the command to start the application.

2.2: Build the Docker Image

Once the Dockerfile is ready, build the Docker image using the following command:

# docker build -t my-node-app .

This command tells Docker to build the image using the current directory (.) and tag it as my-node-app.

2.3: Run the Docker Container

After building the image, you can run the application as a Docker container:

# docker run -p 3000:3000 -d my-node-app

This command:

  • Maps port 3000 from the container to port 3000 on the host machine.
  • Runs the container in detached mode (-d).

3. Handling Dependencies and Services

If your server environment includes multiple services (e.g., a database, caching layer, or message queue), you'll need to containerize those as well. Docker Compose can help you define and run multi-container applications.

Example: Dockerizing a Node.js Application with MongoDB

To run both the Node.js application and a MongoDB database, you’ll need a docker-compose.yml file.

Create a docker-compose.yml file in your project directory:

    version: '3'

    services:
      web:
        build: .
        ports:
          – "3000:3000"
        depends_on:
          – db
      db:
        image: mongo:latest
        volumes:
          – db-data:/data/db
        networks:
          – app-network

    volumes:
      db-data:

    networks:
      app-network:
        driver: bridge

This docker-compose.yml file:

  1. Defines two services: web (the Node.js app) and db (the MongoDB container).
  2. The depends_on directive ensures the database service starts before the web application.
  3. Uses a named volume (db-data) for persistent data storage for MongoDB.
  4. Defines a custom network (app-network) for communication between the two containers.

3.1. Start Services with Docker Compose

To start the services defined in docker-compose.yml, use the following command:

# docker-compose up -d

This command will build the web service (Node.js app), pull the MongoDB image, create the necessary containers, and run them in the background.

4. Manage Data Persistence

Containers are ephemeral by design, meaning data stored inside a container is lost when it stops or is removed. To persist data across container restarts, you’ll need to use volumes.

In the example above, the MongoDB service uses a named volume (db-data) to persist the database data. Docker volumes allow you to:

  • Persist data on the host machine outside of the container.
  • Share data between containers.

To check if the volume is created and inspect its usage, use:

# docker volume ls # docker volume inspect db-data

5. Networking Between Containers

In Docker, containers communicate with each other over a network. By default, Docker Compose creates a network for each application defined in a docker-compose.yml file. Containers within the same network can communicate with each other using container names as hostnames.

For example, in the docker-compose.yml above:

  • The web container can access the db container using db:27017 as the database URL (MongoDB's default port).

6. Scaling and Orchestrating with Docker Swarm or Kubernetes

If you need to scale your application to multiple instances or require orchestration, Docker Swarm and Kubernetes are the two most popular container orchestration platforms.

Docker Swarm:

Built into Docker, Swarm allows you to easily manage a cluster of Docker nodes and scale your containers across multiple machines. To initialize a swarm:

# docker swarm init

Kubernetes:

Kubernetes is a powerful container orchestration tool that provides high availability, automatic scaling, and management of containerized applications. If you’re migrating a more complex server environment, Kubernetes will offer additional features like rolling updates, automatic recovery, and more sophisticated networking options.

7. Security and Permissions

When migrating to Docker, it's important to pay attention to security best practices, such as:

 

  • Running containers with the least privileges (using the USER directive in the Dockerfile).
  • Using multi-stage builds to keep the image size small and reduce the attack surface.
  • Regularly scanning Docker images for known vulnerabilities using tools like Anchore, Trivy, or Clair.
  • Configuring network isolation for sensitive services.

Conclusion

Migrating a server environment into Docker containers involves more than just running an application in isolation. It requires thoughtful planning around dependencies, data persistence, networking, scaling, and security. By containerizing services with Docker, you can create portable, scalable, and consistent environments that streamline both development and production workflows.

By following the steps outlined in this guide—writing Dockerfiles, using Docker Compose for multi-container applications, and ensuring data persistence—you can successfully migrate your existing server environment into a Dockerized architecture. For larger-scale environments, consider leveraging orchestration tools like Docker Swarm or Kubernetes to manage multiple containerized services across a cluster.

Tags: , , , , , , , , , , , , ,
Posted in Docker, Linux, System Administration, System Optimization | No Comments »

How to Set Up SSH Two-Factor Authentication (2FA) on Linux Without Google Authenticator with OATH Toolkit

Wednesday, November 5th, 2025

install-2-factor-free-authentication-google-authentication-alternative-with-oath-toolkit-linux-logo

Most tutorials online on how to secure your SSH server with a 2 Factor Authentication 2FA will tell you to use Google Authenticator to secure SSH logins.

But what if you don’t want to depend on Google software – maybe for privacy, security, or ideological reasons ?

Luckily, you have a choice thanks to free oath toolkit.
The free and self-hosted alternative: OATH Toolkit has its own PAM module  libpam-oath to make the 2FA work  the openssh server.

OATH-Toolkit is a free software toolkit for (OTP) One-Time Password authentication using HOTP/TOTP algorithms. The software ships a small set of command line utilities covering most OTP operation related tasks.

In this guide, I’ll show you how to implement 2-Factor Authentication (TOTP) for SSH on any Linux system using OATH Toolkit, compatible with privacy-friendly authenticator apps like FreeOTP, Aegis, or and OTP.

It is worthy to check out OATH Toolkit author original post here, that will give you a bit of more insight on the tool.

1. Install the Required Packages

For Debian / Ubuntu systems: 

# apt update
# apt install libpam-oath oathtool qrencode
...

For RHEL / CentOS / AlmaLinux:
  

# dnf install pam_oath oathtool

The oathtool command lets you test or generate one-time passwords (OTPs) directly from the command line.

2. Create a User Secret File

libpam-oath uses a file to store each user’s secret key (shared between your server and your phone app).

By default, it reads from: 

/etc/users.oath

Let’s create it securely and set proper permissions to secure it:
  

# touch /etc/users.oath
# chmod 600 /etc/users.oath

Now, generate a new secret key for your user (replace hipo with your actual username):
  

# head -10 /dev/urandom | sha1sum | cut -c1-32

This generates a random 32-character key.
Example: 

9b0e4e9fdf33cce9c76431dc8e7369fe

Add this to /etc/users.oath in the following format: 

HOTP/T30 hipo - 9b0e4e9fdf33cce9c76431dc8e7369fe

HOTP/T30 means Time-based OTP with 30-second validity (standard TOTP).

Replace hipo with the Linux username you want to protect.

3. Add the Key to Your Authenticator App

Now we need to add that secret to your preferred authenticator app.

You can create a TOTP URI manually (to generate a QR code): 

$ echo "otpauth://totp/hipo@jericho?secret=\
$(echo 9b0e4e9fdf33cce9c76431dc8e7369fe \
| xxd -r -p | base32)"

You can paste this URI into a QR code generator (e.g., https://qr-code-generator.com) and scan it using FreeOTP , Aegis, or any open TOTP app.
The FreeOTP Free Ap is my preferred App to use, you can install it via Apple AppStore or Google Play Store.

Alternatively, enter the Base32-encoded secret manually into your app:

# echo 9b0e4e9fdf33cce9c76431dc8e7369fe | xxd -r -p | base32

You can also use qrencode nice nifty tool to generate out of your TOTP code in ASCII mode and scan it with your Phone FreeOTP / Aegis App and add make it ready for use:

# qrencode –type=ANSIUTF8 otpauth://totp/hipo@jericho?secret=$( oathtool –verbose –totp 9b0e4e9fdf33cce9c76431dc8e7369fe –digits=6 -w 1 | grep Base32 | cut -d ' ' -f 3 )\&digits=6\&issuer=pc-freak.net\&period=30

qrencode-generation-of-scannable-QR-code-for-freeotp-or-other-TOTP-auth

qrencode will generate the code. We set the type to ANSI-UTF8 terminal graphics so you can generate this in an ssh login. It can also generate other formats if you were to incorporate this into a web interface. See the man page for qrencode for more options.
The rest of the line is the being encoded into the QR code, and is a URL of the type otpauth, with time based one-time passwords (totp). The user is “hipo@jericho“, though PAM will ignore the @jericho if you are not joined to a domain (I have not tested this with domains yet).

The parameters follow the ‘?‘, and are separated by ‘&‘.

otpauth uses a base32 hash of the secret password you created earlier. oathtool will generate the appropriate hash inside the block:

 $( oathtool –verbose –totp 9b0e4e9fdf33cce9c76431dc8e7369fe | grep Base32 | cut -d ' ' -f 3 )

We put the secret from earlier, and search for “Base32”. This line will contain the Base32 hash that we need from the output:

Hex secret: 9b0e4e9fdf33cce9c76431dc8e7369fe
Base32 secret: E24ABZ2CTW3CH3YIN5HZ2RXP
Digits: 6
Window size: 0
Step size (seconds): 30
Start time: 1970-01-01 00:00:00 UTC (0)
Current time: 2022-03-03 00:09:08 UTC (1646266148)
Counter: 0x3455592 (54875538)

368784 
From there we cut out the third field, “E24ABZ2CTW3CH3YIN5HZ2RXP“, and place it in the line.

Next, we set the number of digits for the codes to be 6 digits (valid values are 6, 7, and 8). 6 is sufficient for most people, and easier to remember.

The issuer is optional, but useful to differentiate where the code came from.

We set the time period (in seconds) for how long a code is valid to 30 seconds.

Note that: Google authenticator ignores this and uses 30 seconds whether you like it or not.

4. Configure PAM to Use libpam-oath

Edit the PAM configuration for SSH: 

# vim /etc/pam.d/sshd

At the top of the file, add: 

auth required pam_oath.so usersfile=/etc/users.oath window=30 digits=6

This tells PAM to check OTP codes against /etc/users.oath.

5. Configure SSH Daemon to Ask for OTP

Edit the SSH daemon configuration file:
  

# vim /etc/ssh/sshd_config

Ensure these lines are set:
  

UsePAM yes
challengeresponseauthentication yes
ChallengeResponseAuthentication yes
AuthenticationMethods publickey keyboard-interactive
##KbdInteractiveAuthentication no
KbdInteractiveAuthentication yes

N.B.! The KbdInteractiveAuthentication yes variable is necessery on OpenSSH servers with version > of version 8.2_ .

In short This setup means:
1. The user must first authenticate with their SSH key (or local / LDAP password),
2. Then enter a valid one-time code generated from TOTP App from their phone.

You can also use  Match  directives to enforce 2FA under certain conditions, but not under others.
For example, if you didn’t want to be bothered with it while you are logging in on your LAN,
but do from any other network, you could add something like: 

Match Address 127.0.0.1,10.10.10.0/8,192.168.5.0/24
Authenticationmethods publickey

6. Restart SSH and Test It

Apply your configuration:
  

# systemctl restart ssh

Now, open a new terminal window and try logging in (don’t close your existing one yet, in case you get locked out): 

$ ssh hipo@your-server-ip

You should see something like: 

Verification code:

Enter the 6-digit code displayed in your FreeOTP (or similar) app.
If it’s correct, you’re logged in! Hooray ! 🙂

7. Test Locally and Secure the Secrets

If you want to test OTPs manually with a base32 encrypted output of hex string: 

# oathtool --totp -b \
9b0e4e9fdf33cce9c76431dc8e7369fe

As above might be a bit confusing for starters, i recommend to use below few lines instead:

$ secret_hex="9b0e4e9fdf33cce9c76431dc8e7369fe"
$ secret_base32=$(echo $secret_hex | xxd -r -p | base32)
$ oathtool –totp -b "$secret_base32"
156874

You’ll get the same 6-digit code your authenticator shows – useful for debugging.

If you rerun the oathtool again you will get a difffefrent TOTP code, e.g. :

$ oathtool –totp -b "$secret_base32"
258158


Use this code as a 2FA TOTP auth code together with local user password (2FA + pass pair),  when prompted for a TOTP code, once you entered your user password first.

To not let anyone who has a local account on the system to be able to breach the 2FA additional password protection,
Ensure the secrets file is protected well, i.e.: 

# chown root:root /etc/users.oath
# chmod 600 /etc/users.oath

How to Enable 2FA Only for Certain Users

If you want to force OTP only for admins, create a group ssh2fa: 

# groupadd ssh2fa
# usermod -aG ssh2fa hipo

Then modify /etc/pam.d/sshd: 

auth [success=1 default=ignore] pam_succeed_if.so \
user notingroup ssh2fa
auth required pam_oath.so usersfile=/etc/users.oath \
window=30 digits=6

Only users in ssh2fa will be asked for a one-time code.

Troubleshooting

Problem: SSH rejects OTP
Check /var/log/auth.log or /var/log/secure for more details.
Make sure your phone’s time is in sync (TOTP depends on accurate time).

Problem: Locked out after restart
Always keep one root session open until you confirm login works.

Problem: Everything seems configured fine but still the TOTP is not accepted by remote OpenSSHD.
– Check out the time on the Phone / Device where the TOTP code is generated is properly synched to an Internet Time Server
– Check the computer system clock is properly synchornized to the Internet Time server (via ntpd / chronyd etc.), below is sample:

  • hipo@jeremiah:~$ timedatectl status
                   Local time: Wed 2025-11-05 00:39:17 EET
               Universal time: Tue 2025-11-04 22:39:17 UTC
                     RTC time: Tue 2025-11-04 22:39:17
                    Time zone: Europe/Sofia (EET, +0200)
    System clock synchronized: yes
                  NTP service: n/a
              RTC in local TZ: no

Why Choose libpam-oath?

  • 100% Free Software (GPL)
  • Works completely offline / self-hosted
  • Compatible with any standard TOTP app (FreeOTP, Aegis, andOTP, etc.)
  • Doesn’t depend on Google APIs or cloud services
  • Lightweight (just one PAM module and a text file)

Conclusion

Two-Factor Authentication doesn’t have to rely on Google’s ecosystem.
With OATH Toolkit and libpam-oath, you get a simple, private, and completely open-source way to harden your SSH server against brute-force and stolen-key attacks.

Once configured, even if an attacker somehow steals your SSH key or password, they can’t log in without your phone’s one-time code – making your system dramatically safer.

Tags: , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, Security, System Administration | No Comments »