Posts Tagged ‘services’

How to keep your Linux server Healthy for Years: Hard learned lessons

Friday, November 28th, 2025

how-to-keep-your-linux-servers-healthy-every-year-doctor_tux

I’ve been running Linux servers long enough to watch hardware die, kernels panic, filesystems fill up at midnight hours, and network cards slowly burn out like old light bulbs.

Over time, you learn that keeping a server alive is less about “perfect architecture” and more about steady discipline – the small habits built to manage the machines, helps prevent big disasters.

Here are some practical, battle-tested lessons that keep my boxes running for years with minimal downtime. Most of them were learned the hard way.

1. Monitor Before You Fix – and Fix Before It Breaks

Most Linux disasters come from things we should have noticed earlier. The lack of monitoring, there is modern day saying that should become your favourite if you are a sysadmin or Dev Ops engineer.

"Monitoring everything !"

  • The disk that was at 89% yesterday will be at 100% tonight.
  • The log file that grew by 500 MB last week will explode this week.
  • The swap usage creeping from 1% → 5% → 20% means your next heavy task will choke.
  • The unseen failing BIOS CMOS battery
  • The RAID disks degradation etc.

You don’t need enterprise monitoring to prevent this. And even simple tools like monit or a simple zabbix-agent -> zabbix-server or any other simplistic scripted  monitoring gives you a basic issues pre-warning.

Even a simple cronjob shell one liner can save you hours of further sh!t :

#!/bin/bash

df -h / | awk 'NR==2 { if($5+0 > 85) print "Disk Alert: / is at " $5 }' \
| mail -s "Disk Warning on $(hostname)" admin@example.com

2. Treat /etc directory as Sacred – Treat It Like an expensive gem

Every sysadmin eventually faces the nightmare of a broken config overwritten by a package update or a hasty command at 2 AM.

To avoid crying later, archive /etc automatically:

# tar czf /root/etc-$(date +%Y-%m-%d).tar.gz /etc


If you prefer the backup to be more sophisticated you can use my clone of the dirs_backup.sh (an old script I wrote for easifying backup of specific directories on the filesystem ) the etc_backup.sh you can get here.
Run it weekly via cron.
This little trick has saved me more times than I can count — especially when migrating between Debian releases or recovering from accidental edits.

3. Automate everything what you have to repeatevely do

If you find yourself doing something manually more than twice, script it and forget it.

Examples:

  • rotating logs for misbehaving apps
  • restarting services that occasionally get “stuck”
  • syncing backups between machines
  • cleaning temp directories

Here’s a small example I still use today:

#!/bin/bash

# Kill zombie PHP-FPM children that keep leaking memory

ps aux | grep php-fpm | awk '{if($6 > 300000) print $2}' | xargs -r kill -9

Dirty way to get rid of misfunctioning php-fpm ?
Yes. But it works.

4. Backups Don’t Exist Unless You Test Them

It’s easy to feel proud when you write a backup script.
It’s harder – and far more important – to test the restore.

Once a month  or at least once in a few months, try restore a random backup to a dummy VM.
Sometimes backup might fails, or you might get something different from what you originally expected and by doing so
you can guarantee you will not cry later helplessly.

A broken backup doesn’t fail quietly – it fails on the day you need it most.

5. Don’t Ignore Hardware – It Ages like Everything Else

Linux might run forever, but hardware doesn’t.

Signs of impending doom:

  • dmesg spam with I/O errors
  • slow SSD response
  • increasing SMART reallocated sectors
  • random freezes without logs
  • sudden network flakiness

Run this monthly:

6. Document Everything (Future You Will Thank Past You)

There are moments when you ask yourself:

“Why did I configure this machine like this?”

If you don’t document your decisions, you’ll have no idea one year later.

A simple markdown file inside /root/notes.txt or /root/README.md is enough.

Document:

  • installed software
  • custom scripts
  • non-standard configs
  • firewall rules
  • weird hacks you probably forgot already

This turns chaos into something you can actually maintain.

7. Keep Things Simple – Complexity Is the Enemy of Uptime

The longer I work with servers, the more I strip away:

  • fewer moving parts
  • fewer services
  • fewer custom patches
  • fewer “temporary” hacks that become permanent

A simple system is a reliable system.
A complex one dies at the worst possible moment.

8. Accept That Failure Will Still Happen

No matter how careful you are, servers will surely:

  • crash
  • corrupt filesystems
  • lose network connectivity
  • inexplicably freeze
  • reboot after a kernel panic

Don’t aim for perfection.Aim for resilience.

If you can restore the machine in under an hour, you're winning and in the white.

Final Thoughts

Linux is powerful – but it rewards those who treat it with respect and perseverance.
Over many years, I’ve realized that maintaining servers is less about brilliance and more about humble, consistent care and hard work persistence.

I hope this article helps some sysamdmin to rethink and rebundle servers maintenance strategy in a way that will avoid a server meltdown at  night hours like 3 AM.

Cheers ! 

 

Tags: , , , , , , , , , , , , , ,
Posted in Linux, System Administration, Various | No Comments »

Migrating Server environments to Docker Containers a brief step-by-step guide

Wednesday, November 12th, 2025

migrating-server--applications-environment-to-docker-containers

In modern IT environments, containerization has become an essential strategy for improving application portability, scalability, and consistency. Docker, as a containerization platform, allows you to package applications and their dependencies into isolated containers that can be easily deployed across different environments. Migrating an existing server environment into Docker containers is a common scenario, and this guide will walk you through the key steps of doing so.

Why Migrate to Docker?

Before we dive into the specifics, let’s briefly understand why you might want to migrate your server environment to Docker:

  • Portability: Containers encapsulate applications and all their dependencies, making them portable across any system running Docker.
  • Scalability: Containers are lightweight and can be scaled up or down easily, offering flexibility to handle varying loads.
  • Consistency: With Docker, you can ensure that your application behaves the same in development, testing, and production.
  • Isolation: Docker containers run in isolation from the host system, minimizing the risk of configuration conflicts.
     

Steps to Migrate a Server Environment into Docker

Migrating an environment of servers into Docker typically involves several steps: evaluating the current setup, containerizing applications, managing dependencies, and orchestrating deployment. Here’s a breakdown:

1. Evaluate the Existing Server Environment

Before migrating, it's essential to inventory the current server environment to understand the following:

  • The applications running on the servers
  • Dependencies (e.g., databases, third-party services, libraries, etc.)
  • Networking setup (e.g., exposed ports, communication between services)
  • Storage requirements (e.g., persistent data, volumes)
  • Security concerns (e.g., user permissions, firewalls)
     

For example, if you're running a web server with a backend database and some caching layers, you'll need to break down these services into their constituent parts so they can be containerized.

2. Containerize the Application

The next step is to convert the services running on your server into Docker containers. Docker containers require a Dockerfile, which is a blueprint for how to build and run the container. Let's walk through an example of containerizing a simple web application.

Example: Migrating a Simple Node.js Web Application

Assume you have a Node.js application running on your server. To containerize it, you need to:

  • Create a Dockerfile
  • Build the Docker image
  • Run the containerized application

2.1. Write a Dockerfile

A Dockerfile defines how your application will be built within a Docker container. Here’s an example for a Node.js application:

# Step 1: Use the official Node.js image as a base image FROM node:16

# Step 2: Set the working directory inside the container WORKDIR /usr/src/app # Step 3: Copy package.json and package-lock.json to the container COPY package*.json ./

# Step 4: Install dependencies inside the container RUN npm install

# Step 5: Copy the rest of the application code to the container COPY . .

# Step 6: Expose the port that the application will listen on EXPOSE 3000

# Step 7: Start the application CMD ["npm", "start"]

This Dockerfile:

  1. Uses the official node:16 image as a base.
  2. Sets the working directory inside the container to /usr/src/app.
  3. Copies the package.json and package-lock.json files to the container and runs npm install to install dependencies.
  4. Copies the rest of the application code into the container.
  5. Exposes port 3000 (assuming that’s the port your app runs on).
  6. Defines the command to start the application.

2.2: Build the Docker Image

Once the Dockerfile is ready, build the Docker image using the following command:

# docker build -t my-node-app .

This command tells Docker to build the image using the current directory (.) and tag it as my-node-app.

2.3: Run the Docker Container

After building the image, you can run the application as a Docker container:

# docker run -p 3000:3000 -d my-node-app

This command:

  • Maps port 3000 from the container to port 3000 on the host machine.
  • Runs the container in detached mode (-d).

3. Handling Dependencies and Services

If your server environment includes multiple services (e.g., a database, caching layer, or message queue), you'll need to containerize those as well. Docker Compose can help you define and run multi-container applications.

Example: Dockerizing a Node.js Application with MongoDB

To run both the Node.js application and a MongoDB database, you’ll need a docker-compose.yml file.

Create a docker-compose.yml file in your project directory:

    version: '3'

    services:
      web:
        build: .
        ports:
          – "3000:3000"
        depends_on:
          – db
      db:
        image: mongo:latest
        volumes:
          – db-data:/data/db
        networks:
          – app-network

    volumes:
      db-data:

    networks:
      app-network:
        driver: bridge

This docker-compose.yml file:

  1. Defines two services: web (the Node.js app) and db (the MongoDB container).
  2. The depends_on directive ensures the database service starts before the web application.
  3. Uses a named volume (db-data) for persistent data storage for MongoDB.
  4. Defines a custom network (app-network) for communication between the two containers.

3.1. Start Services with Docker Compose

To start the services defined in docker-compose.yml, use the following command:

# docker-compose up -d

This command will build the web service (Node.js app), pull the MongoDB image, create the necessary containers, and run them in the background.

4. Manage Data Persistence

Containers are ephemeral by design, meaning data stored inside a container is lost when it stops or is removed. To persist data across container restarts, you’ll need to use volumes.

In the example above, the MongoDB service uses a named volume (db-data) to persist the database data. Docker volumes allow you to:

  • Persist data on the host machine outside of the container.
  • Share data between containers.

To check if the volume is created and inspect its usage, use:

# docker volume ls # docker volume inspect db-data

5. Networking Between Containers

In Docker, containers communicate with each other over a network. By default, Docker Compose creates a network for each application defined in a docker-compose.yml file. Containers within the same network can communicate with each other using container names as hostnames.

For example, in the docker-compose.yml above:

  • The web container can access the db container using db:27017 as the database URL (MongoDB's default port).

6. Scaling and Orchestrating with Docker Swarm or Kubernetes

If you need to scale your application to multiple instances or require orchestration, Docker Swarm and Kubernetes are the two most popular container orchestration platforms.

Docker Swarm:

Built into Docker, Swarm allows you to easily manage a cluster of Docker nodes and scale your containers across multiple machines. To initialize a swarm:

# docker swarm init

Kubernetes:

Kubernetes is a powerful container orchestration tool that provides high availability, automatic scaling, and management of containerized applications. If you’re migrating a more complex server environment, Kubernetes will offer additional features like rolling updates, automatic recovery, and more sophisticated networking options.

7. Security and Permissions

When migrating to Docker, it's important to pay attention to security best practices, such as:

 

  • Running containers with the least privileges (using the USER directive in the Dockerfile).
  • Using multi-stage builds to keep the image size small and reduce the attack surface.
  • Regularly scanning Docker images for known vulnerabilities using tools like Anchore, Trivy, or Clair.
  • Configuring network isolation for sensitive services.

Conclusion

Migrating a server environment into Docker containers involves more than just running an application in isolation. It requires thoughtful planning around dependencies, data persistence, networking, scaling, and security. By containerizing services with Docker, you can create portable, scalable, and consistent environments that streamline both development and production workflows.

By following the steps outlined in this guide—writing Dockerfiles, using Docker Compose for multi-container applications, and ensuring data persistence—you can successfully migrate your existing server environment into a Dockerized architecture. For larger-scale environments, consider leveraging orchestration tools like Docker Swarm or Kubernetes to manage multiple containerized services across a cluster.

Tags: , , , , , , , , , , , , ,
Posted in Docker, Linux, System Administration, System Optimization | No Comments »

Digital Vigilance: Practical Cyber Defense for the New Era of All connected dependency on technology

Friday, October 24th, 2025

 

Introduction

There was a time when cybersecurity was mostly about erecting a firewall, installing antivirus software and hoping no one clicked a suspicious link. That era is steadily fading. Today, as more work moves to the cloud, as AI tools proliferate, and as threat actors adopt business-like models, the battlefield has shifted dramatically. According to analysts at Gartner, 2025 brings some of the most significant inflections in cybersecurity in recent memory. 

In this article we’ll cover the major trends, why they matter, and — importantly — what you as an individual or sysadmin can start doing today to stay ahead.

1. Generative AI: Weapon and Shield

AI / ML (Machine Learning)) is now deeply ingrained in both the offence and defence sides of cybersecurity.

  • On the defence side: AI models help detect anomalies, process huge volumes of logs, and automate responses. 
  • On the offence side: Attackers use AI to craft more convincing phishing campaigns, automate vulnerability discovery, generate fake identities or even design malware. 
  • Data types are changing: It’s no longer just databases and spreadsheets. Unstructured data (images, video, text) used by AI models is now a primary risk.

What to do:

  • Make sure any sensitive AI-training data or inference logs are stored securely.
  • Build anomaly-detection systems that don’t assume “normal” traffic anymore.
  • Flag when your organisation uses AI tools: do you know what data the tool uses, where it stores it, who can access it?

2. Zero Trust Isn’t Optional Anymore

 

cyber-security-threats-to-watch-in-2025

The old model — trust everything inside the perimeter, block everything outside — is obsolete. Distributed workforces, cloud services, edge devices: they all blur the perimeter. Hence the rise of Zero Trust Architecture (ZTA) — “never trust, always verify.” INE+1

Key features:

  • Every device, every user, every session must be authenticated and authorised.
  • Least-privilege access: users should have the minimum permissions needed.
  • Micro-segmentation: limit lateral movement in networks.
  • Real-time monitoring and visibility of sessions and devices.

What to do:

  • Audit your devices and users: who has broad permissions? Who accesses critical systems?
  • Implement multifactor authentication (MFA) everywhere you can.
  • Review network segmentation: can a compromised device access everything? If yes, that’s a red flag.
     

3. Ransomware & RaaS – The Business Model of Cybercrime

Cybercriminals are organizing like businesses: they have supply chains, service models, profit centres. The trend of Ransomware‑as‑a‑Service (RaaS) continues to expand. Dataconomy+1

What’s changed:

  • Ransomware doesn’t just encrypt data. Attackers often steal data first, then threaten to release it. 
  • Attackers are picking higher-value targets and critical infrastructure.
  • The attack surface has exploded: IoT devices, cloud mis-configurations, unmanaged identity & access.

What to do:

  • Back up your critical systems regularly — test restores, not just backups.
  • Keep systems patched (though even fully patched systems can be attacked, so patching is necessary but not sufficient).
  • Monitor for abnormal behaviour: large data exfiltration, new admin accounts, sudden access from odd places.
  • Implement strong incident response procedures: when it happens, how do you contain it?

4. Supply Chains, IoT & Machine Identities

Modern IT is no longer just endpoints and servers. We have IoT devices, embedded systems, cloud services, machine-to-machine identities. According to Gartner, machine identities are expanding attack surfaces if unmanaged.

Key issues:

  • Devices (especially IoT) often ship with weak/default credentials.
  • Machine identities: software services, APIs, automation tools need their own identity/access management.
  • Supply chains: your vendor might be the weakest link — compromise of software or hardware upstream affects you.

What to do:

  • Create an inventory of all devices and services — yes all.
  • Enforce device onboarding processes: credentials changed, firmware up-to-date, network segmented.
  • Review your vendors: what security standards do they follow? Do they give you visibility into their supply chain risk?
     

5. Cloud & Data Privacy — New Rules, New Risks

As data moves into the cloud and into AI systems, the regulatory and technical risks converge. For example, new laws like the EU AI Act will start affecting how organisations handle AI usage and data. Source: Gcore
Cloud environments also bring mis-configurations, improper access controls, shadow-IT and uncontrolled data sprawl. techresearchs.com+1

What to do:
 

  • If using cloud services, check settings for major risk zones (e.g., S3 buckets, unsecured APIs).
  • Implement strong Identity & Access Management (IAM) controls for cloud resources.
  • Make data-privacy part of your security plan: what data you collect, where it is stored, for how long.
  • Perform periodic audits and compliance checks especially if you handle users from different jurisdictions.
     

6. Skills, Culture & Burn-out — The Human Factor

Often overlooked: no matter how good your tech is, people and culture matter. Gartner Security behaviour programs help reduce human-error incidents — and they’re becoming more essential.
Also, the cybersecurity talent shortage and burnout among security teams is real.

What to do:

 

  • Invest in security awareness training: phishing simulation, strong password practices, device hygiene.
  • Foster a culture where security is everyone’s responsibility, not just the “IT team’s problem.”
  • For small teams: consider managed security services or cloud-based monitoring to lean on external support.

7. What This Means for Smaller Organisations & Individual Users

Often the big reports focus on enterprises. But smaller organisations (and individual users) are just as vulnerable — sometimes more so, because they have fewer resources and less mature security.
Here are some concrete actions:

  • Use strong, unique passwords and a password manager.
  • Enable MFA everywhere (email, online services, VPNs).
  • Keep your systems updated — OS, applications, firmware.
  • Be suspicious of unexpected communications (phishing).
  • Have an incident response plan: who do you call if things go wrong?
  • Backup your data offline and test restores.
  • If you run services (web-server, mail server): monitor logs, check for new accounts, stray network connections.
     

Conclusion

Cybersecurity in 2025 is not a “set once and forget” system. It’s dynamic, multi-layered and deeply integrated into business functions and personal habits. The trends above — generative AI, zero trust, supply chain risks, cloud data sprawl — are changing the rules of the game.
Thus for all of us and especially sysadmins / system engineers or Site Reliabiltiy Managers (SRE), Developers, Testers or whatever you call it this meen we need to keep learning, be careful with the tech stuff we use, and build security as a continuous practice rather than a one-off box-to-tick.

 

Tags: , , , , , , , , , ,
Posted in Automation, Computer Security, Security | No Comments »

Implementing and using gssproxy, example guide on how to use it to authenticate ssh, samba, nfs with no password via kerberos protocol

Friday, September 26th, 2025

Implementing and using gssproxy, example guide on how to use it to authenticate ssh, samba, nfs with no password via kerberos protocol

GSS-Proxy is a daemon that safely performs GSSAPI (Kerberos) operations on behalf of other processes. It’s useful when services running as unprivileged users need to accept or initiate Kerberos GSSAPI authentication but shouldn’t hold or access long‑lived keys (keytabs) or raw credentials themselves. Typical users: OpenSSH, SSSD, Samba, NFS idmap, and custom daemons.

This article walks through what gssproxy does, how it works, how to install and configure it, example integrations (sshd and an unprivileged service), testing, debugging and common pitfalls.
 

1. What gssproxy does (quick conceptual summary)
 

  • Runs as a privileged system daemon (typically root) and holds access to keytabs or system credentials.
  • Exposes a local IPC (Unix socket) and controlled API so allowed clients can ask it to perform GSSAPI accept/init operations on their behalf.
  • Enforces access controls by client PID/user and by named service configuration (you map a client identity to the allowed service name and keytab).
  • Minimizes the need to distribute keytabs or give services direct access to Kerberos credentials.
     

2. Installation

On many modern Linux distributions (Fedora, RHEL/CentOS, Debian/Ubuntu) gssproxy is packaged.

Example (RHEL/Fedora/CentOS):

# RHEL/CentOS 7/8/9 (dnf or yum)

 

sudo dnf install gssproxy

 

# or

 

sudo yum install gssproxy

Example (Debian/Ubuntu):

sudo apt update

sudo apt install gssproxy

If you must build from source:

# get source, then typical autotools or meson/ninja workflow per upstream README

./configure

make

sudo make install

 

After install, systemd unit gssproxy.service should be available.
 

3. Main configuration concepts

The main config file is usually /etc/gssproxy/gssproxy.conf. It consists of mechs (mechanisms), services, clients, and possibly mappings. Key elements:

  • mech: declares a GSSAPI mechanism (e.g., krb5) and default keytab(s) for acceptor credentials.
  • service: logical service names (e.g., ssh, nfs, httpd) with attributes: user (the Unix user running the service), keytab, cred_store, mechs, and whether the service is allowed to be client (initiate) and/or server (accept).
  • client: rules mapping local client sockets / users / pids to allowed services.

A minimal working example that allows sshd to use gssproxy:

mechs = {

    krb5_mech = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5_mech;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        # allow both acceptor (server) and initiator (client) ops if needed

        client = yes;

        server = yes;

    };

};

Client rules are often implicit: gssproxy can enforce that calls on a given service socket originate from the configured Unix user. For more complex setups you add policy and client blocks. Example to allow a specific PID or user to use the ssh service:

clients = {

    ssh_clients = {

        clients = [

            { match = "uid:0" },      # root can ask for ssh service

            { match = "user:sshd" },  # or the sshd user

        ];

        service = "ssh";

    };

};

Paths and sockets: gssproxy listens on a socket (e.g. /var/run/gssproxy/socket) and possibly per-user sockets (e.g. /run/gssproxy/uid_1000). The systemd unit usually creates the runtime directory with correct permissions.
 

4. Example: Integrate with OpenSSH server (sshd)

Goal: allow sshd and session processes to accept delegated GSS credentials and let unprivileged child processes use those credentials via gssproxy.

Server side config

  1. Ensure sshd is built/installed with GSSAPI support. On SSH server:

    • In /etc/ssh/sshd_config:
    • GSSAPIAuthentication yes
    • GSSAPICleanupCredentials yes
    • GSSAPIKeyExchange yes        # optional: if you want GSS key exchange
  2. Configure gssproxy with an ssh service entry pointing to the host keytab (so gssproxy can accept SPNEGO/kerberos accept_sec_context calls):

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        server = yes;

        client = yes;

    };

};

  1. Ensure /etc/krb5.keytab contains the host principal host/fqdn@REALM (or host/short@REALM depending on SPN strategy). Use ktutil or kadmin to create/populate.
  2. Restart gssproxy and sshd:

sudo systemctl restart gssproxy

sudo systemctl restart sshd

Client side

  • ssh client configuration (usually ~/.ssh/config or /etc/ssh/ssh_config):

Host myhost.example.com

    GSSAPIAuthentication yes

    GSSAPIDelegateCredentials yes

Client must have a TGT in the credential cache (kinit user), or use a client that acquires one.

Result

When the client initiates GSSAPI authentication and delegates credentials (GSSAPIDelegateCredentials yes or -K for older OpenSSH), gssproxy on the server handles acceptor functions. If a session process needs to use the delegated credentials (e.g., to access network resources as that user), gssproxy arranges a per-session credential store that unprivileged processes can use via the kernel keyring or other mechanisms gssproxy supports.
 

5. Example: Allow an unprivileged service to acquire initiator creds via gssproxy

Suppose a service mydaemon runs as myuser and needs to initiate Kerberos-authenticated connections using a specific service principal stored in /etc/mydaemon.keytab but you don’t want to expose that keytab to myuser.

Add a mech and service:

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

    mydaemon_mech = {

        mech = krb5;

        default_keytab = /etc/mydaemon.keytab;

    };

};

 

services = {

    mydaemon = {

        mech = mydaemon_mech;

        user = "myuser";

        keytab = /etc/mydaemon.keytab;

        client = yes;    # allow initiator operations

        server = no;

    };

};

Configure a client mapping so the mydaemon process (uid myuser) is allowed to use the mydaemon service. Once gssproxy runs, mydaemon uses the gssapi libraries (GSSAPI libs detect gssproxy via environment or library probe) and calls the GSSAPI functions; gssproxy will perform gss_acquire_cred using /etc/mydaemon.keytab and return a handle to the calling process. The service itself never directly reads the keytab.
 

6. Testing and tools

  • kinit / klist: manage and list Kerberos TGTs on clients.
  • journalctl -u gssproxy -f (or systemctl status gssproxy) to watch logs.
  • ss -l or ls -l /run/gssproxy to inspect sockets.
  • If you have gssproxy command-line utilities installed (may vary by distro), some installations include gssproxy CLI helpers. Otherwise use the service that relies on gssproxy and watch logs.

Example basic tests:

  1. Ensure gssproxy is running:

sudo systemctl status gssproxy

  1. On server, check socket and permissions:

sudo ls -l /run/gssproxy

# or

sudo ss -x -a | grep gssproxy

  1. Attempt SSH from a client with a TGT:

kinit alice

ssh -o GSSAPIDelegateCredentials=yes alice@server.example.com

# then on server, check journalctl logs for gssproxy/sshd messages
 

7. Debugging tips

  • Journal logs: journalctl -u gssproxy -xe will be your first stop.
  • Permissions: Ensure that gssproxy can read the keytab(s) (typically root-owned with restrictive perms). In config you may point to a keytab readable only by gssproxy.
  • Clients blocked: If a client is denied, check the clients block and match rules (uid/pid/user).
  • Keytab issues: Use klist -k /etc/krb5.keytab to list principals in a keytab. Ensure correct SPN and realm.
  • Clock skew: Kerberos is time-sensitive. Ensure NTP/chrony is working.
  • DNS / SPNs: Ensure hostnames and reverse DNS match the principal names expected for the service.
  • SSHD integration: If sshd still complains it can’t accept GSSAPI creds, enable debug logging (LogLevel DEBUG), and check gssproxy logs.
  • SELinux: On SELinux-enabled systems, you may need to ensure file contexts and SELinux policies allow gssproxy to access keytabs and sockets. Check audit.log for AVC denials and use semanage fcontext/restorecon or local policy modules when needed.
     

8. Common pitfalls & best practices

  • Don’t expose keytabs to unprivileged users. Let gssproxy hold them.
  • Principals & SPNs must match service hostnames used by clients. Consistent DNS is essential.
  • Minimal privileges: configure services and clients narrowly: allow only the minimum users/PIDs and only the required mech ops.
  • Rotation: when rotating keytabs, reload/restart gssproxy or send a signal if supported. Plan for keytab updates.
  • Logging: enable adequate logging during deployment and revert to normal verbosity in production.
  • Testing in staging: GSSAPI behavior across SSH clients and other daemons can be subtle — test across your client set (Linux, macOS, Windows via native Kerberos clients, etc.).
     

9. Security considerations

  • gssproxy centralizes credential access: secure the host and the gssproxy process.
  • Protect keytab files using strict filesystem permissions and (if needed) SELinux policy.
  • Restrict which local processes may request operations for a service — map by UID/PID carefully.
  • Monitor logs for unexpected use of gssproxy.
     

10. Example full config (simple)

Save as /etc/gssproxy/gssproxy.conf:

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        server = yes;

        client = yes;

    };

 

    mydaemon = {

        mech = krb5;

        user = "myuser";

        keytab = /etc/mydaemon.keytab;

        client = yes;

        server = no;

    };

};

 

clients = {

    allow_root_for_ssh = {

        clients = [

            { match = "uid:0" },

        ];

        service = "ssh";

    };

 

    mydaemon_client = {

        clients = [

            { match = "user:myuser" },

        ];

        service = "mydaemon";

    };

};

Restart: sudo systemctl restart gssproxy and then restart dependent services (sshd, mydaemon, etc.) if needed.

 

Useful resources for gssproxy and further integrations

  • Read your distribution’s /usr/share/doc/gssproxy/ or man pages (man gssproxy, man gssproxy.conf) — they contain distribution-specific details.
  • Check integrations: Samba/Winbind, SSSD, NFS idmap — many modern stacks support gssproxy as an option to avoid exposing keytabs to many daemons.
  • For production: automate keytab distribution, rotation and monitor gssproxy usage.

 

Tags: , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, Various | No Comments »

Fix Zabbix selinux caused permission issues on CentOS 7 Linux / cannot set resource limit: [13] Permission denied error solution

Tuesday, July 6th, 2021

zabbix-selinux-logo-fix-zabbix-permission-issues-when-running-on-ceontos-linux-change-selinux-to-permissive-howto.

If you have to install Zabbix client that has to communicate towards Zabbix server via a Zabbix Proxy you might be unpleasently surprised that it cannot cannot be start if the selinux mode is set to Enforcing.
Error message like on below screenshot will be displayed when starting proxy client with systemctl.

zabbix-proxy-cannot-be-started-due-to-selinux-permissions

In the zabbix logs you will see error  messages such as:
 

"cannot set resource limit: [13] Permission denied, CentOS 7"

 

29085:20160730:062959.263 Starting Zabbix Agent [Test host]. Zabbix 3.0.4 (revision 61185).
29085:20160730:062959.263 **** Enabled features ****
29085:20160730:062959.263 IPv6 support: YES
29085:20160730:062959.263 TLS support: YES
29085:20160730:062959.263 **************************
29085:20160730:062959.263 using configuration file: /etc/zabbix/zabbix_agentd.conf
29085:20160730:062959.263 cannot set resource limit: [13] Permission denied
29085:20160730:062959.263 cannot disable core dump, exiting…

 

Next step to do is to check whether zabbix is listed in selinux's enabled modules to do so run:
 

[root@centos ~ ]# semodules -l
…
…..
vhostmd    1.1.0
virt    1.5.0
vlock    1.2.0
vmtools    1.0.0
vmware    2.7.0
vnstatd    1.1.0
vpn    1.16.0
w3c    1.1.0
watchdog    1.8.0
wdmd    1.1.0
webadm    1.2.0
webalizer    1.13.0
wine    1.11.0
wireshark    2.4.0
xen    1.13.0
xguest    1.2.0
xserver    3.9.4
zabbix    1.6.0
zarafa    1.2.0
zebra    1.13.0
zoneminder    1.0.0
zosremote    1.2.0
 

[root@centos ~ ]# sestatus
# sestatusSELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

To get exact zabbix IDs that needs to be added as permissive for Selinux you can use ps -eZ like so:

[root@centos ~ ]# ps -eZ |grep -i zabbix
system_u:system_r:zabbix_agent_t:s0 1149 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1150 ?     00:04:28 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1151 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1152 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1153 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1154 ?     02:21:46 zabbix_agentd

As you can see zabbix is enabled and hence selinux enforcing mode is preventing zabbix client / server to operate and communicate normally, hence to make it work we need to change zabbix agent and zabbix proxy to permissive mode.

Setting selinux for zabbix agent and zabbix proxy to permissive mode

If you don't have them installed you might neet the setroubleshoot setools, setools-console and policycoreutils-python rpms packs (if you have them installed skip this step).

[root@centos ~ ]# yum install setroubleshoot.x86_64 setools.x86_64 setools-console.x86_64 policycoreutils-python.x86_64

Then to add zabbix service to become permissive either run

[root@centos ~ ]# semanage permissive –add zabbix_t

[root@centos ~ ]# semanage permissive -a zabbix_agent_t


In some cases you might also need in case if just adding the permissive for zabbix_agent_t try also :

setsebool -P zabbix_can_network=1

Next try to start zabbox-proxy and zabbix-agent systemd services 

[root@centos ~ ]# systemctl start zabbix-proxy.service
…

[root@centos ~ ]# systemctl start zabbix-agent.service
…

Hopefully all should report fine with the service checking the status should show you something like:

[root@centos ~ ]# systemctl status zabbix-agent
● zabbix-agent.service – Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-06-24 07:47:42 CEST; 1 weeks 5 days ago
 Main PID: 1149 (zabbix_agentd)
   CGroup: /system.slice/zabbix-agent.service
           ├─1149 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─1150 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─1151 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─1152 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─1153 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─1154 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Check the Logs finally to make sure all is fine with zabbix being allowed by selinux.

[root@centos ~ ]# grep zabbix_proxy /var/log/audit/audit.log
…

[root@centos ~ ]# tail -n 100 /var/log/zabbix/zabbix_agentd.log


If no errors are in and you receive and you can visualize the usual zabbix collected CPU / Memory / Disk etc. values you're good, Enjoy ! 🙂

Tags: , , , , , , , , ,
Posted in Linux, Monitoring, System Administration | No Comments »

Upgrade Debian Linux 9 to 10 Stretch to Buster and Disable graphical service load boot on Debian 10 Linux / Debian Buster is out

Tuesday, July 9th, 2019

howto-upgrade-debian-linux-debian-stretch-to-buster-debian-10-buster

I've just took a time to upgrade my Debian 9 Stretch Linux to Debian Buster on my old school Laptop (that turned 11 years old) Lenovo Thinkpad R61 . The upgrade went more or less without severe issues except few things.

The overall procedure followed is described n a few websites out there already and comes up to;

 

0. Set the proper repository location in /etc/apt/sources.list


Before update the sources.list used are:
 

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free

 

deb [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free
deb-src [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free

deb http://ftp.debian.org/debian buster-backports main


For people that had stretch defined in /etc/apt/sources.list you should change them to buster or stable, easiest and quickest way to omit editting with vim / nano etc. is run as root or via sudo:
 

sed -i 's/stretch/buster/g' /etc/apt/sources.list
sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/*.list

The minimum of config in sources.list after the modification should be
 

deb http://deb.debian.org/debian buster main
deb http://deb.debian.org/debian buster-updates main
deb http://security.debian.org/debian-security buster/updates main

Or if you want to always be with latest stable packages (which is my practice for notebooks):

deb http://deb.debian.org/debian stable main
deb http://deb.debian.org/debian stable-updates main
deb http://security.debian.org/debian-security stable/updates main

 

1. Getting list of hold packages if such exist and unholding them, e.g.

 

apt-mark showhold


Same could also be done via dpkg

dpkg –get-selections | grep hold


To unhold a package if such is found:

echo "package_name install"|sudo dpkg –set-selections

For those who don't know what hold package is this is usually package you want to keep at certain version all the time even though after running apt-get upgrade to get the latest package versions.
 

2. Use df -h and assure you have at least 5 – 10 GB free space on root directory / before proceed

df -h /

3. Update packages list to set new set repos as default

apt update

 

4. apt upgrade
 

apt upgrade

Here some 10 – 15 times you have to confirm what you want to do with configuration that has changed if you're unsure about the config (and it is not critical service) you're aware as such as Apache / MySQL / SMTP etc. it is best to install the latest maintainer version.

Hopefully here you will not get fatal errors that will interrupt it.

P.S. It is best to run apt-update either in VTTY (Virtual console session) with screen or tmux or via a physical tty (if this is not a remote server) as during the updates your GUI access to the gnome-terminal or konsole / xterm whatever console used might get cut. Thus it is best to do it with command:
 

screen apt upgrade

 

5. Run dist-upgrade to finalize the upgrade from Stertch to Buster

 

Once all is completed of the new installed packages, you will need to finally do, once again it is best to run via screen, if you don't have installed screen install it:

 

if [ $(which screen) ]; then echo 'Installed'; else apt-get install –yes screen ; fi

screen apt dist-upgrade


Here once again you should set whether old configuration to some e services has to stay or the new Debian maintainer package shipped one will overwrite the old and locally modified (due to some reason), here do wisely whatever you will otherwise some configured services might not boot as expected on next boot.

 

6. What if you get packages failed on update


If you get a certain package failed to configure after installed due to some reason, if it is a systemd service use:

 

journalctl -xe |head -n 50


or fully observer output of journalctl -xe and decide on yourself.

In most cases

dpkg-reconfigure failed-package-name


should do the trick or at least give you more hints on how to solve it.

 

Also if a package seems to be in inconsistent or broken state after upgrade  and simple dpkg-reconfigure doesn't help, a good command
that can help you is

 

dpkg-reconfigure -f package_name

 

or you can try to workaround a failed package setup with:
 

dpkg –configure -a

 
If dpkg-reconfigure doesn't help either as I experienced in prior of Debian from Debian 6 -> 7 an Debian 7 ->8 updates on some Computers, then a very useful thing to try is:
 

apt-get update –fix-missing 

apt-get install -f


At certain cases the only work around to be able to complete the package upgrade is to to remove the package with apt remove but due to config errors even that is not possible to work around this as final resort run:
 

dpkg –remove –force-remove-reinstreq

 

7. Clean up ununeeded packages

 

Some packages are left over due to package dependencies from Stretch and not needed in buster anymore to remove them.
 

apt autoremove

 

8. Reboot system once all upgrade is over

 

/sbin/reboot

 

9. Verify your just upgraded Debian is in a good state

 

root@noah:~# uname -a;
Linux noah 4.19.0-5-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

 

root@noah:~# cat /etc/issue.net
Debian GNU/Linux 10
 

 

root@noah:~# lsb_release -a
No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:    buster

 

root@noah:~# hostnamectl
   Static hostname: noah
         Icon name: computer-laptop
           Chassis: laptop
        Machine ID: 4759d9c2f20265938692146351a07929
           Boot ID: 256eb64ffa5e413b8f959f7ef43d919f
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-rt-amd64
      Architecture: x86-64

 

10. Remove annoying picture short animation with debian logo looping

 

plymouth-debian-graphical-boot-services

By default Debian 10 boots up with annoying screen hiding all the status of loaded services state .e.g. you cannot see the services that shows in [ FAILED ] state and  which do show as [ OK ] to revert back the old behavior I'm used to for historical reasons and as it shows a lot of good Boot time debugging info, in previous Debian distributions this was possible  by setting the right configuration options in /etc/default/grub

which so far in my config was like so

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y zswap.enabled=1 text"


Note that zswap.enabled=1 passed option is because my notebook is pretty old machine from 2008 with 4GB of memory and zswap does accelerate performance when working with swap – especially helpful on Older PCs for more you can read more about zswap on ArchLinux wiki
After modifying this configuration to load the new config into grub the cmd is:
 

/usr/sbin/update-grub

 
As this was not working and tried number of reboots finally I found that annoying animated gif like picture shown up is caused by plymouth below is excerpts from Plymouth's manual page:


       "The plymouth sends commands to a running plymouthd. This is used during the boot process to control the display of the graphical boot splash."

Plymouth has a set of themes one can set:

 

# plymouth-set-default-theme -l
futureprototype
details
futureprototype
joy
lines
moonlight
softwaves
spacefun
text
tribar

 

I tried to change that theme to make the boot process as text boot as I'm used to historically with cmd:
 

update-alternatives –config text.plymouth

 
As after reboot I hoped the PC will start booting in text but this does not happened so the final fix to turn back to textmode service boot was to completely remove plymouth
 

apt-get remove –yes plymouth

Tags: , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, Linux on Laptops, Linux Package Management, Various | No Comments »