Archive for September, 2025

What is oddjobd and How to Use It Instead of sudo to run limited privileged execution of scripts requiring admin

Tuesday, September 30th, 2025

oddjobd-sudoers-linux-elevate-script-running-linux

In Linux environments, managing privileged operations for unprivileged users is a critical task. Traditionally, tools like sudo have been used to allow users to execute specific commands with elevated privileges. However, in more secure or fine-tuned environments — such as enterprise networks or identity-managed systems — oddjobd offers a more controlled, D-Bus-driven alternative.

This article explains what oddjobd is, how it works, and when you might prefer it over sudo, complete with real-world examples.

What is oddjobd?

oddjobd is a system service (daemon) that runs in the background and allows limited, controlled execution of privileged tasks on behalf of unprivileged users.

Key Features:

  • Allows secure execution of predefined scripts or programs as root (or another user).
  • Communicates over D-Bus for fine-grained access control.
  • Uses Polkit (PolicyKit) to manage who can run which tasks.
  • Commonly used in FreeIPA, SSSD, and LDAP-based environments.
  • Configuration files live in: /etc/oddjobd.conf.d/

How It Works

  • System administrators define specific jobs (scripts or commands) in config files.
  • These jobs are exposed via D-Bus.
  • Unprivileged users (or applications) can request jobs to be executed.
  • Access is granted or denied by Polkit rules, not passwords.
  • No full shell or terminal access is granted — just the job.
 

oddjobd vs sudo

Feature

sudo

oddjobd

Control granularity

Medium (commands)

High (methods, scripts only)

Interactive shell

Yes

No

Config complexity

Simple (/etc/sudoers)

Moderate (conf.d + Polkit)

Uses system user password

Yes

Optional (can be passwordless via Polkit)

Security

Medium

High (no shell, strict policy control)

D-Bus compatible

No

Yes

Ideal for

Power users

Controlled environments (e.g., FreeIPA)

Typical Use Cases for oddjobd


1. Automatically Creating Home Directories


Problem: LDAP/FreeIPA users don’t have home directories created on login.

Solution: Enable oddjobd to create them via oddjob-mkhomedir.

# authconfig –enablemkhomedir –update

On login, PAM calls oddjobd, which creates the home directory as root.
 

2.  Restarting a Service without sudo

Let's say you want a user to restart Apache, but not give them full sudo rights.

a. Create a script

# /usr/local/bin/restart_apache.sh

#!/bin/bash

systemctl restart apache2

echo "Apache restarted by oddjob at $(date)"

chmod +x /usr/local/bin/restart_apache.sh

b. Create Oddjob config
 

# /etc/oddjobd.conf.d/restart_apache.conf

[restart_apache]

program = /usr/local/bin/restart_apache.sh

user = root

c. Polkit rule

 

// /etc/polkit-1/rules.d/60-restart-apache.rules

polkit.addRule(function(action, subject) {

    if (action.id == "org.freedesktop.oddjob.restart_apache" &&

        subject.isInGroup("apacheadmins")) {

        return polkit.Result.YES;

    }

});

 

d. Add user to group

# groupadd apacheadmins

# usermod -aG apacheadmins alice


e. Restart and test

# systemctl restart oddjobd


# As user "alice":

oddjob_request restart_apache


Only the defined method runs — no sudo shell access, no arbitrary commands.
 

3. GUI-friendly Device Control


Use Case: A user wants to reset a USB device via a button in a GUI app.

  • Define the method in oddjobd.
  • Use Polkit for GUI D-Bus permission.
  • The app can call the method securely, without sudo.

Advantages of oddjobd

More Secure Than sudo:

  • No interactive shell or terminal.
  • No command-line injection risks.
  • Can’t “escape” to a shell like with sudo bash.

Granular Control:

  • Limit tasks to a specific script or even script arguments.

D-Bus and GUI Friendly:

  • Apps can call privileged methods without shell hacks.

Policy-Based Authorization (Polkit):

  • Fine-grained user/group access control.
  • No password prompts if not desired.

Enterprise-Ready:

  • Works well with LDAP, FreeIPA, and centralized login environments.

Oddjobd Limitations / Downsides

Limitation

Description

Learning Curve

More complex to set up than sudo

Configuration Overhead

Requires writing config files and Polkit rules

Debugging

Issues may be harder to trace than sudo logs

Not for Ad-hoc Commands

Only predefined jobs can be run

Not Installed by Default

Often needs to be manually installed (oddjob, oddjob-mkhomedir)

When to Use oddjobd Instead of sudo

Use oddjobd when you:

  • Need to allow users or apps to run very specific privileged operations.
  • Want to avoid giving full shell access via sudo.
  • Are working in a managed enterprise environment.
  • Need GUI or D-Bus-based privilege escalation.
  • Require scripted access to root tasks without exposing credentials.

Conclusion

oddjobd is a powerful tool for securely handling privileged operations in Linux, especially where tight access control and automation are required. While sudo is simple and flexible, oddjobd shines in structured, security-conscious environments — particularly those using FreeIPA, LDAP, or automated tools.

If you need a more scriptable, policy-driven, and safer alternative to sudo for specific tasks, oddjobd is well worth exploring.

Implementing and using gssproxy, example guide on how to use it to authenticate ssh, samba, nfs with no password via kerberos protocol

Friday, September 26th, 2025

Implementing and using gssproxy, example guide on how to use it to authenticate ssh, samba, nfs with no password via kerberos protocol

GSS-Proxy is a daemon that safely performs GSSAPI (Kerberos) operations on behalf of other processes. It’s useful when services running as unprivileged users need to accept or initiate Kerberos GSSAPI authentication but shouldn’t hold or access long‑lived keys (keytabs) or raw credentials themselves. Typical users: OpenSSH, SSSD, Samba, NFS idmap, and custom daemons.

This article walks through what gssproxy does, how it works, how to install and configure it, example integrations (sshd and an unprivileged service), testing, debugging and common pitfalls.
 

1. What gssproxy does (quick conceptual summary)
 

  • Runs as a privileged system daemon (typically root) and holds access to keytabs or system credentials.
  • Exposes a local IPC (Unix socket) and controlled API so allowed clients can ask it to perform GSSAPI accept/init operations on their behalf.
  • Enforces access controls by client PID/user and by named service configuration (you map a client identity to the allowed service name and keytab).
  • Minimizes the need to distribute keytabs or give services direct access to Kerberos credentials.
     

2. Installation

On many modern Linux distributions (Fedora, RHEL/CentOS, Debian/Ubuntu) gssproxy is packaged.

Example (RHEL/Fedora/CentOS):

# RHEL/CentOS 7/8/9 (dnf or yum)

 

sudo dnf install gssproxy

 

# or

 

sudo yum install gssproxy

Example (Debian/Ubuntu):

sudo apt update

sudo apt install gssproxy

If you must build from source:

# get source, then typical autotools or meson/ninja workflow per upstream README

./configure

make

sudo make install

 

After install, systemd unit gssproxy.service should be available.
 

3. Main configuration concepts

The main config file is usually /etc/gssproxy/gssproxy.conf. It consists of mechs (mechanisms), services, clients, and possibly mappings. Key elements:

  • mech: declares a GSSAPI mechanism (e.g., krb5) and default keytab(s) for acceptor credentials.
  • service: logical service names (e.g., ssh, nfs, httpd) with attributes: user (the Unix user running the service), keytab, cred_store, mechs, and whether the service is allowed to be client (initiate) and/or server (accept).
  • client: rules mapping local client sockets / users / pids to allowed services.

A minimal working example that allows sshd to use gssproxy:

mechs = {

    krb5_mech = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5_mech;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        # allow both acceptor (server) and initiator (client) ops if needed

        client = yes;

        server = yes;

    };

};

Client rules are often implicit: gssproxy can enforce that calls on a given service socket originate from the configured Unix user. For more complex setups you add policy and client blocks. Example to allow a specific PID or user to use the ssh service:

clients = {

    ssh_clients = {

        clients = [

            { match = "uid:0" },      # root can ask for ssh service

            { match = "user:sshd" },  # or the sshd user

        ];

        service = "ssh";

    };

};

Paths and sockets: gssproxy listens on a socket (e.g. /var/run/gssproxy/socket) and possibly per-user sockets (e.g. /run/gssproxy/uid_1000). The systemd unit usually creates the runtime directory with correct permissions.
 

4. Example: Integrate with OpenSSH server (sshd)

Goal: allow sshd and session processes to accept delegated GSS credentials and let unprivileged child processes use those credentials via gssproxy.

Server side config

  1. Ensure sshd is built/installed with GSSAPI support. On SSH server:

    • In /etc/ssh/sshd_config:
    • GSSAPIAuthentication yes
    • GSSAPICleanupCredentials yes
    • GSSAPIKeyExchange yes        # optional: if you want GSS key exchange
  2. Configure gssproxy with an ssh service entry pointing to the host keytab (so gssproxy can accept SPNEGO/kerberos accept_sec_context calls):

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        server = yes;

        client = yes;

    };

};

  1. Ensure /etc/krb5.keytab contains the host principal host/fqdn@REALM (or host/short@REALM depending on SPN strategy). Use ktutil or kadmin to create/populate.
  2. Restart gssproxy and sshd:

sudo systemctl restart gssproxy

sudo systemctl restart sshd

Client side

  • ssh client configuration (usually ~/.ssh/config or /etc/ssh/ssh_config):

Host myhost.example.com

    GSSAPIAuthentication yes

    GSSAPIDelegateCredentials yes

Client must have a TGT in the credential cache (kinit user), or use a client that acquires one.

Result

When the client initiates GSSAPI authentication and delegates credentials (GSSAPIDelegateCredentials yes or -K for older OpenSSH), gssproxy on the server handles acceptor functions. If a session process needs to use the delegated credentials (e.g., to access network resources as that user), gssproxy arranges a per-session credential store that unprivileged processes can use via the kernel keyring or other mechanisms gssproxy supports.
 

5. Example: Allow an unprivileged service to acquire initiator creds via gssproxy

Suppose a service mydaemon runs as myuser and needs to initiate Kerberos-authenticated connections using a specific service principal stored in /etc/mydaemon.keytab but you don’t want to expose that keytab to myuser.

Add a mech and service:

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

    mydaemon_mech = {

        mech = krb5;

        default_keytab = /etc/mydaemon.keytab;

    };

};

 

services = {

    mydaemon = {

        mech = mydaemon_mech;

        user = "myuser";

        keytab = /etc/mydaemon.keytab;

        client = yes;    # allow initiator operations

        server = no;

    };

};

Configure a client mapping so the mydaemon process (uid myuser) is allowed to use the mydaemon service. Once gssproxy runs, mydaemon uses the gssapi libraries (GSSAPI libs detect gssproxy via environment or library probe) and calls the GSSAPI functions; gssproxy will perform gss_acquire_cred using /etc/mydaemon.keytab and return a handle to the calling process. The service itself never directly reads the keytab.
 

6. Testing and tools

  • kinit / klist: manage and list Kerberos TGTs on clients.
  • journalctl -u gssproxy -f (or systemctl status gssproxy) to watch logs.
  • ss -l or ls -l /run/gssproxy to inspect sockets.
  • If you have gssproxy command-line utilities installed (may vary by distro), some installations include gssproxy CLI helpers. Otherwise use the service that relies on gssproxy and watch logs.

Example basic tests:

  1. Ensure gssproxy is running:

sudo systemctl status gssproxy

  1. On server, check socket and permissions:

sudo ls -l /run/gssproxy

# or

sudo ss -x -a | grep gssproxy

  1. Attempt SSH from a client with a TGT:

kinit alice

ssh -o GSSAPIDelegateCredentials=yes alice@server.example.com

# then on server, check journalctl logs for gssproxy/sshd messages
 

7. Debugging tips

  • Journal logs: journalctl -u gssproxy -xe will be your first stop.
  • Permissions: Ensure that gssproxy can read the keytab(s) (typically root-owned with restrictive perms). In config you may point to a keytab readable only by gssproxy.
  • Clients blocked: If a client is denied, check the clients block and match rules (uid/pid/user).
  • Keytab issues: Use klist -k /etc/krb5.keytab to list principals in a keytab. Ensure correct SPN and realm.
  • Clock skew: Kerberos is time-sensitive. Ensure NTP/chrony is working.
  • DNS / SPNs: Ensure hostnames and reverse DNS match the principal names expected for the service.
  • SSHD integration: If sshd still complains it can’t accept GSSAPI creds, enable debug logging (LogLevel DEBUG), and check gssproxy logs.
  • SELinux: On SELinux-enabled systems, you may need to ensure file contexts and SELinux policies allow gssproxy to access keytabs and sockets. Check audit.log for AVC denials and use semanage fcontext/restorecon or local policy modules when needed.
     

8. Common pitfalls & best practices

  • Don’t expose keytabs to unprivileged users. Let gssproxy hold them.
  • Principals & SPNs must match service hostnames used by clients. Consistent DNS is essential.
  • Minimal privileges: configure services and clients narrowly: allow only the minimum users/PIDs and only the required mech ops.
  • Rotation: when rotating keytabs, reload/restart gssproxy or send a signal if supported. Plan for keytab updates.
  • Logging: enable adequate logging during deployment and revert to normal verbosity in production.
  • Testing in staging: GSSAPI behavior across SSH clients and other daemons can be subtle — test across your client set (Linux, macOS, Windows via native Kerberos clients, etc.).
     

9. Security considerations

  • gssproxy centralizes credential access: secure the host and the gssproxy process.
  • Protect keytab files using strict filesystem permissions and (if needed) SELinux policy.
  • Restrict which local processes may request operations for a service — map by UID/PID carefully.
  • Monitor logs for unexpected use of gssproxy.
     

10. Example full config (simple)

Save as /etc/gssproxy/gssproxy.conf:

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        server = yes;

        client = yes;

    };

 

    mydaemon = {

        mech = krb5;

        user = "myuser";

        keytab = /etc/mydaemon.keytab;

        client = yes;

        server = no;

    };

};

 

clients = {

    allow_root_for_ssh = {

        clients = [

            { match = "uid:0" },

        ];

        service = "ssh";

    };

 

    mydaemon_client = {

        clients = [

            { match = "user:myuser" },

        ];

        service = "mydaemon";

    };

};

Restart: sudo systemctl restart gssproxy and then restart dependent services (sshd, mydaemon, etc.) if needed.

 

Useful resources for gssproxy and further integrations

  • Read your distribution’s /usr/share/doc/gssproxy/ or man pages (man gssproxy, man gssproxy.conf) — they contain distribution-specific details.
  • Check integrations: Samba/Winbind, SSSD, NFS idmap — many modern stacks support gssproxy as an option to avoid exposing keytabs to many daemons.
  • For production: automate keytab distribution, rotation and monitor gssproxy usage.

 

How to Install and Use auditd for System Security Auditing on Linux

Thursday, September 25th, 2025

System auditing is essential for monitoring user activity, detecting unauthorized access, and ensuring compliance with security standards. On Linux, the Audit Daemon (auditd) provides powerful auditing capabilities for logging system events and actions.

This short article will walk you through installing, configuring, and using auditd to monitor your Linux system.

What is auditd?

auditd is the user-space component of the Linux Auditing System. It logs system calls, file access, user activity, and more — offering administrators a clear trail of what’s happening on the system.


1. Installing auditd

The auditd package is available by default in most major Linux distributions.

 On Debian/Ubuntu

# apt update
# apt install auditd audispd-plugins

 On CentOS/RHEL/Fedora

# yum install audit

After installation, start and enable the audit daemon

# systemctl start auditd

# systemctl enable auditd

Check its status

# systemctl status auditd

2. Setting Audit Rules

Once auditd is running, you need to define rules that tell it what to monitor.

Example: Monitor changes to /etc/passwd

# auditctl -w /etc/passwd -p rwxa -k passwd_monitor

Explanation:

  • -w /etc/passwd: Watch this file. When the file is accessed, the watcher will generate events.
  • -p rwxa: Monitor read, write, execute, and attribute changes
  • -k passwd_monitor: Assign a custom key name to identify logs. Later on, we could search for this (arbitrary) passwd string to identify events tagged with this key.

List active rules:

# auditctl -l

3. Common auditd Rules for Security Monitoring

Here are some common and useful auditd rules you can use to monitor system activity and enhance Linux system security. These rules are typically added to the /etc/audit/rules.d/audit.rules or /etc/audit/audit.rules file, depending on your system.

a. Monitor Access to /etc/passwd and /etc/shadow
 

-w /etc/passwd -p wa -k passwd_changes
-w /etc/shadow -p wa -k shadow_changes

  • Monitors read/write/attribute changes to password files.

b. Monitor sudoers file and directory
 

-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers

  • Tracks any change to sudo configuration files.

c. Monitor Use of chmod, chown, and passwd
 

-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -k perm_mod
-a always,exit -F arch=b64 -S passwd -k passwd_changes

  • Watches permission and ownership changes.

d. Monitor User and Group Modifications

-w /etc/group -p wa -k group_mod
-w /etc/gshadow -p wa -k gshadow_mod
-w /etc/security/opasswd -p wa -k opasswd_mod

  • Catches user/group-related config changes.

e. Track Logins, Logouts, and Session Initiation

-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k failed_login
-w /var/log/faillog -p wa -k faillog

  • Tracks login attempts and failures.

f. Monitor auditd Configuration Changes

-w /etc/audit/ -p wa -k auditconfig
-w /etc/audit/audit.rules -p wa -k auditrules

  • Watches changes to auditd configuration and rules.

g. Detect Changes to System Binaries

-w /bin/ -p wa -k bin_changes
-w /sbin/ -p wa -k sbin_changes
-w /usr/bin/ -p wa -k usr_bin_changes
-w /usr/sbin/ -p wa -k usr_sbin_changes

  • Ensures core binaries aren't tampered with.

h. Track Kernel Module Loading and Unloading

-a always,exit -F arch=b64 -S init_module -S delete_module -k kernel_mod

  • Detects dynamic kernel-level changes.

l. Monitor File Deletions

-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k delete

  • Tracks when files are removed or renamed.

m. Track Privilege Escalation via setuid/setgid

-a always,exit -F arch=b64 -S setuid -S setgid -k priv_esc

  • Helps detect changes in user or group privileges.

n. Track Usage of Dangerous Binaries (e.g., su, sudo, netcat)

-w /usr/bin/su -p x -k su_usage
-w /usr/bin/sudo -p x -k sudo_usage
-w /bin/nc -p x -k netcat_usage

  • Useful for catching potentially malicious command usage.

o. Monitor Cron Jobs

-w /etc/cron.allow -p wa -k cron_allow
-w /etc/cron.deny -p wa -k cron_deny
-w /etc/cron.d/ -p wa -k cron_d
-w /etc/crontab -p wa -k crontab
-w /var/spool/cron/ -p wa -k user_crontabs

  • Alerts on cron job creation/modification.

p. Track Changes to /etc/hosts and DNS Settings

-w /etc/hosts -p wa -k etc_hosts
-w /etc/resolv.conf -p wa -k resolv_conf

  • Monitors potential redirection or DNS manipulation.

q. Monitor Mounting and Unmounting of Filesystems

-a always,exit -F arch=b64 -S mount -S umount2 -k mounts

  • Useful for detecting USB or external drive activity.

r. Track Execution of New Programs

-a always,exit -F arch=b64 -S execve -k exec

  • Captures command execution (can generate a lot of logs).
     

A complete list of rules you can get from the hardening.rules auditd file place it under /etc/audit/rules.d/hardening.rules
and reload auditd to load the configurations.

Tips

  • Use ausearch -k <key> to search audit logs for matching rule.
  • Use auditctl -l to list active rules.
  • Use augenrules –load after editing rules in /etc/audit/rules.d/.


4. Reading Audit Logs

Audit logs events are stored in:

/var/log/audit/audit.log

By default, the location, this can be changed through /etc/auditd/auditd.conf

View recent entries:
 

# tail -f /var/log/audit/audit.log

Search by key:
 

# ausearch -k passwd_monitor

Generate a summary report:

# aureport -f

# aureport


Example: Show all user logins / IPs :

# aureport -au

 

5. Making Audit Rules Persistent

Rules added with auditctl are not persistent and will be lost on reboot. To make them permanent:

Edit the audit rules configuration:

# vim /etc/audit/rules.d/audit.rules

Add your rules, for example:

-w /etc/passwd -p rwxa -k passwd_monitor

Apply the rules:

# augenrules –load

7. Some use case examples of auditd in auditing Linux servers by sysadmins / security experts
 

Below are real-world, practical examples where auditd is actively used by sysadmins, security teams, or compliance officers to detect suspicious activity, meet compliance requirements, or conduct forensic investigations.

a. Detect Unauthorized Access to /etc/shadow

Use Case: Someone tries to read or modify password hashes.

Audit Rule:

-w /etc/shadow -p wa -k shadow_watch

Real-World Trigger:

sudo cat /etc/shadow

Check Logs:
 

# ausearch -k shadow_watch -i

Real Output:
 

type=SYSCALL msg=audit(09/18/2025 14:02:45.123:1078):

  syscall=openat

  exe="/usr/bin/cat"

  success=yes

  path="/etc/shadow"

  key="shadow_watch"

b. Detect Use of chmod to Make Files Executable

Use Case: Attacker tries to make a script executable (e.g., malware).

Audit Rule:

-a always,exit -F arch=b64 -S chmod -k chmod_detect

Real-World Trigger:
 

 # chmod +x /tmp/evil_script.sh

Check Logs:

# ausearch -k chmod_detect -i

c. Monitor Execution of nc (Netcat)

Use Case: Netcat is often used for reverse shells or unauthorized network comms.

Audit Rule:
 

-w /bin/nc -p x -k netcat_usage
 

Real-World Trigger:

nc -lvp 4444

Log Entry:

type=EXECVE msg=audit(09/18/2025 14:35:45.456:1123):

  argc=3 a0="nc" a1="-lvp" a2="4444"

  key="netcat_usage"

 

d. Alert on Kernel Module Insertion
 

Use Case: Attacker loads rootkit or malicious kernel module.

Audit Rule:

-a always,exit -F arch=b64 -S init_module -S delete_module -k kernel_mod

Real-World Trigger:

# insmod myrootkit.ko

Audit Log:
 

type=SYSCALL msg=audit(09/18/2025 15:00:13.100:1155):

  syscall=init_module

  exe="/sbin/insmod"

  key="kernel_mod"

e. Watch for Unexpected sudo Usage

Use Case: Unusual use of sudo might indicate privilege escalation.

Audit Rule:

-w /usr/bin/sudo -p x -k sudo_watch

Real-World Trigger:

sudo whoami

View Log:
 

# ausearch -k sudo_watch -i


f. Monitor Cron Job Modification

Use Case: Attacker schedules persistence via cron.

Audit Rule:

-w /etc/crontab -p wa -k cron_mod

Real-World Trigger:
 

echo "@reboot /tmp/backdoor" >> /etc/crontab

Logs:
 

type=SYSCALL msg=audit(09/18/2025 15:05:45.789:1188):

  syscall=open

  path="/etc/crontab"

  key="cron_mod"

g. Detect File Deletion or Renaming
 

Use Case: Attacker removes logs or evidence.

Audit Rule:

-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k file_delete

Real-World Trigger:

# rm -f /var/log/syslog

Logs:
 

type=SYSCALL msg=audit(09/18/2025 15:10:33.987:1210):

  syscall=unlink

  path="/var/log/syslog"

  key="file_delete"


h. Detect Script or Malware Execution
 

Use Case: Capture any executed command.

Audit Rule:
 

-a always,exit -F arch=b64 -S execve -k exec

Real-World Trigger:

/tmp/myscript.sh

Log View:

# ausearch -k exec -i | grep /tmp/myscript.sh

l. Detect Manual Changes to /etc/hosts

Use Case: DNS hijacking or phishing setup.

Audit Rule:

-w /etc/hosts -p wa -k etc_hosts

Real-World Trigger:
 

# echo "1.2.3.4 google.com" >> /etc/hosts

Logs:

type=SYSCALL msg=audit(09/18/2025 15:20:11.444:1234):

  path="/etc/hosts"

  syscall=open

  key="etc_hosts"


8. Enable Immutable Mode (if necessery)

For enhanced security, you can make audit rules immutable, preventing any changes until reboot:

# auditctl -e 2


To make this setting persistent, add the following to the end of /etc/audit/rules.d/audit.rules:

-e 2


Common Use Cases

Here are a few more examples of what you can monitor:

Monitor all sudo usage:

# auditctl -w /var/log/auth.log -p wa -k sudo_monitor


Monitor a directory for file access:

# auditctl -w /home/username/important_dir -p rwxa -k dir_watch

Audit execution of a specific command (e.g., rm):

# auditctl -a always,exit -F arch=b64 -S unlink,unlinkat -k delete_cmd

(Adjust arch=b64 to arch=b32 if on 32-bit system.)

9. Managing the Audit Log Size

Audit logs can grow large over time. To manage log rotation and size, edit:
 

# vim /etc/audit/auditd.conf

Set log rotation options like:

max_log_file = 8

num_logs = 5

Then restart auditd:
 

# systemctl restart auditd

Conclusion

The Linux Audit Daemon (auditd) is a powerful tool to track system activity, enhance security, and meet compliance requirements. With just a few configuration steps, you can monitor critical files, user actions, and system behavior in real time.

 

References

  • man auditd
  • man auditctl
  • Linux Audit Wiki

 

Father Archimandrite Ivan of Novi Han, Protector of Poor and Homeless passed away to Christ

Tuesday, September 16th, 2025

Father Ivan with his shepherd stick

On September 14, the 80-year-old Archimandrite Joan (John) , known simply as Father Ivan from Novi Khan.
Father Ivan
was well known in Bulgarian society as the only Monk father of the Fatherless. and a great benefactor who dedicated his life to the sick, poor, needy and, the homeless people in Bulgaria.

The Requiem service  will be consecrated on September 16 from 12:00 pm in the church "The Holy Trinity" in Novi Khan (a Church situated in the improvised homeless shelter organized in an old Monastery (Saint Nicholas) by Father Ivan himself. by Bishop John of Branitsky, former Vicar Bishop of Patriarch Daniel.

Father Ivan Dimitrov Ivanov was born on March 31, 1945. in the Bulgarian village of Blagovo, Montana region. He received his basic and secondary education in the High Professional School "Peter Beron" in city of Sofia. Next He attended a university degree from the Theological Academy "St. Clement of Ohrid" (which was re-established in the distant year 1981)  already in grown age after working different other jobs. He merried and with his wife Todorka Grigorova Ivanova had two sons – Ivaylo and Grigor. On October 19, 1981 he was ordained a deacon, and on 26 October same year – ordained for a  priest by Metropolitan Filaret of Vidin. Since 1987 he was appointed as a priest at the Church "Holy Trinity" in the town of Novi Han, region of Elin Pelin. In year 2011 he has been consecrated as monk the name Ioan by the Bishop of Devol Theodosij (Theodosius). In 2015 in Montana he was promoted to archimandrite dignity by Vidin Metropolitan Dometian.

Along with his pastoral work, he devotes all his strength to caring for the poor. His selfless service remains misunderstood by many, and for a long time he does not receive support from the leadership of the Bulgarian Orthodox Church. In 2014, His Holiness Patriarch Neophyte of Bulgaria came to visit the “Holy Trinity” shelter in Novi Han. In addition to food and gifts, he also gave a sign of empathy with the great priest’s work.

Otec_Ivan_Novi-Han-s-patriarh-Neofit

Father Ivan began his work in  the distanct 1993, short after the fall of Communism regime in Bulgaria when he restored the abandoned building of the monastery "Holy Trinity" in Novi Han. He revived the liturgical life in the monastery church (that was abondoned for many years due to the dictatorship regime in Bulgaria) and created the shelter "Saint Nicholas", in which he sheltered a dozen homeless people. Gradually, the orphanage expanded and it housed mainly women and children deprived of home and protection, but also found shelter for the elderly and homeless. The priest gave pregnant girls left without any financial support and shelter, the opportunity to give birth to their children and take care of them in the shelter for free. Several generations of children were born and raised under the care of Father Ivan. Today, some of them were the first to express their grief over the death of the father on social networks.

otec_Ivan-with-his-children

Even though his genuine kindness and great good deeds Father Ivan has suffered a lot from people who hate the light of Christ, being a victim of various life hardships, road incidents, and a lot of health issues, which he endevoured with stoicism. He has multiple times has shared when we met that many people being pushed by the devil has done him a lot of badness. 
As a mean to finance the homeless people Father Ivan with the help of his orpans, take care about animals such as chickens, cows, sheeps.
Not on a signle occasion animals belonging to him and his shelter has been stolen.
He fought many battles in his life time but never lost hope neither he fall in spirit always counting on the help of the Almighty, like the ancient saints.
Father Ivans deed for that hard times can be compared to the great deeds of the ancients and for the Bulgarian society his meaning and inspiration was similar
to the old testamental important of Moses who led the people.
His servence of a priest was always given for the good example and people respected him and loved him for his deeds in Bulgaria over the last 40 years.

Otec_Ivan-stuck-with-a-knife

In 2013, the father bought several houses in the Montana village of Yakimovo, where he housed also homeless families and their children (the houses hosting homless families reached about 80+ houses ! which accomodated about 180 people, together with the people in the orphanage in Novi Han they were more than 250+ people).

The last years of his life he spend living in his small living Room (monastic Cell) in the Saint Nicolas Monastery in prayer and last spiritual cares for his many children.

Father Ivan of Novi Han and his son Grigorij

The priest's work is continued by his son (who also took the path of his father) Fr. Grigoriy, who now takes care of the orphanages. Fr. Ivan remains in our recent church history as an example of complete dedication to the service of his neighbors, without seeking reward or gratitude. Something unique and unrepeatable fhat inspired many to be more merciful and benevolent for the poor and needy and by this he incarnated the example of Christ who teached us to always have the poor and needy with us and in our hearts. Generous to the life's misfortunes of his "children", Fr. Ivan set an example of a servant of Christ, whose love "does not seek its own… bears all things, believes all things, hopes all things, endures all things" (1 Cor. 13:5-7).

I had the blessing to meet father Ivan of Novi Han a couple of times together with Bishop Ierotey Agathopolski (who used to be in the past under the spiritual guidance of Father Ivan). And my personal impressions are that father Ivan was a highly spiritual person and true monk. I had the chance one time in the distant 2011 to meet him in Novi Khan afer a pilgrimage to Holy Mount Athos we have been guests to Father Ivan in the monastery for a night and even had the chance to be and serve as a ipodeacon on a Holy Liturgy led by Father Ivan. I still remember how much he worried about the children God has sent him for care and how much he worried if financially he can make it to give a good life to all the new inhabitants who recently come to his shelter. 

The Children and old homeless people for whom Father Ivan of Novi Han took care

I remember he had to buy some food for the shelter and I wanted to help him to carry the stuff and we were to the SuperMarket together to buy food and goods for the children.
Father Ivan had a great temper and a great sense of humor even though he has been already aging and his spirit was lifted even though the hardships both with suffering heavy form of diabetes and other multitude of physical infirmity.

 
Film Sparks (Movie about the Shleter of Saint Nicolas created by Father Ivan of Novi Han)


Father Ivan and his Children an Interview Movie by Fr Ivan

 


Movie about The Life Story hardships of Father Ivan of Novi Khan and what made him decide to serve the sick, the poor and the lonely

Father Archimandrite Ivan one worthy Bulgarian

In above interview, father Ivan talks about how he decided to become a priest and how he promised God that he will make an orphanage home. He was accused during the communistic times in Bulgaria for being a priest only to serve as a spy for the West. He was kept in Montana Police station interrogated for 7 months in a small dark police carcer (small police room for prisoners), trying and the police tried to brain wash him that he is guilty for things he is not for all the time being asked the same questions again and again with a light lamp infront of his eye sight. He tells in short in the interview, how he managed to restore the Holy Trinity monastery, after a lot of hardships and miracles of God who helped me to do so. On multiple times his orphanage house has been tried to be (closed) and even destroyed by the local people of Novi Han unsuccesful. He has a lot of abusers who tried to run the deeds of his life the oprphanage but God by his great grace and his powers did not allow this to happen! But right on the contrary helped him and the children and provided him everything he needs to fulfill the dead of his life to help the salvation of tens of thousands to find his path to God and multitudes of people to physically survive a life of horror for years. He put a multitude of lost souls for whom noone cared back on the track of life and by his mercifulness he preached the Gospel not only words but in deeds.

Let God forgive him, Have mercy on his Soul and Forgive him if he sinned as a man and Receive him in his heavenly kingdom ! And Let us have his prayers to Christ ! Amen

Unlocking the Power of lnav: Logfile Navigator – ncurses text based tool guide to mutiple Logs on multiple servers easy analysis on Linux

Saturday, September 13th, 2025

lnav-syslog-screenshot-linux-virtual-machine

If you've ever found yourself buried under a mountain of log files, tailing multiple outputs, or grepping through endless lines trying to spot an error, it's time to meet your new best friend: lnav, the Logfile Navigator.

Lightweight, terminal-based, and surprisingly powerful, lnav is one of the most underrated tools for developers, sysadmins, and anyone who regularly digs into logs. It turns your chaotic logs into something that’s not only readable—but genuinely useful.

What is lnav and why use it ?

lnav (Logfile Navigator) is a command-line tool for viewing and analyzing log files. It goes beyond tail, less, or grep by:

  • Automatically detecting and merging log formats.
  • Highlighting timestamps, log levels, and errors.
  • Providing SQL-like queries over your logs.
  • Offering interactive navigation with a UI inside the terminal.

And yes, all of that without needing to set up a database or a server.

1. Installing lnav on Linux

Installation is straightforward. On most systems, you can install it via package managers:

On Ubuntu/Debian:

# apt install lnav

On Fedora:

# dnf install lnav

On Arch Linux:

# pacman -S lnav

Or build from source via GitHub if you want the latest version.

2. Use lnav Instead of Tail / Grep why?

Traditional tools are powerful, but they require manual work to chain together functionality. lnav gives you:

  • Automatic multi-log parsing: Drop multiple logs in, and it merges them chronologically.
  • Syntax highlighting: Errors and warnings stand out.
  • SQL querying: Run queries like SELECT * FROM syslog_log WHERE log_level = 'error';
  • Filtering and searching: Use intuitive filters and bookmarks to highlight specific entries.

3. Basic tool Usage is simple

Let’s say you want to inspect a system log:

# lnav /var/log/syslog

You'll immediately get:

  • Color-coded output (timestamps, levels, messages).
  • Scrollable view (arrow keys, PgUp, PgDn).
  • Real-time updates (like tail -f).
  • Search with /, filter with :filter-in, and even SQL queries.

Lets say you need to analyze Apache webserver logs recursively including the logs already rotated and gunzipped with *.gz extension on CentOS / Fedora / RHEL, you can do it with:

# lnav -r /var/log/httpd

You can parse the log file and get additional information about requests as well as you can print overall summary of log file.

Choose the line you want to parse. The selected line is always the one at the top of the window. Then press 'p' and you should see the following result:

https://pc-freak.net/images/lnav-get-extra-information-about-apache-query-with-P-press-key-screenshot-linux

Now, if you want to see a summary view of the logs by date and time, simply press 'i'.

lnav-linux-apache-log-review-summary-of-errors-warnings-normal-screenshot

To quit a screen you have chosen press 'q'.

4. LNAV helpful options and hotkeys

Once you've opened a log file/s for analyze you can use few hotkeys that will allow us to move through the output of lnav and the available views more easily:

e or E to jump to the next / previous error message.
w or W to jump to the next / previous warning message.
b or Backspace to move to the previous page.
Space to move to the next page.
g or G to move to the top / bottom of the current view.

To take a closer look at the way lnav operates, use -d option, the debug information is to be spit inside a .txt file:

# lnav /var/log/httpd -d lnav.txt

In this example, the debug information that is generated when lnav starts will be written to a file named lnav.txt inside the current working directory.

5. Real-World Use Cases

a. Troubleshooting application or system process Crashes

Open all relevant logs in one go:

# lnav /var/log/*.log

Errors are highlighted, and you can jump between them with n / N kbd keys.

b. Combining Multiple Logs

Working with an app that logs to different files and you need to combine:

# lnav /var/log/nginx/access.log /var/log/nginx/error.log


Or lets say you want to combine Apache Webserver with Haproxy log and get log summaries or filter out stuff:

lnav /var/log/apache2/access.log /var/log/haproxy.log


Now you will get a single, chronological timeline of events.

 

If you want to Search for a concrete occurance of Error / Warning or IP address inside a bunch of loaded combined logs you can do it with the same command like in simple vim by pressing / (slash) from kbd and type out what you want to filter out to get shown.

c. Analyze SQL Queries Logs

Yes, you can actually do this by passing it query in its command prompt :

:.schema
:SELECT log_time, log_level, log_message FROM syslog_log WHERE log_level = 'error';

You get a table of filtered logs, sortable by columns.
 

6. lnav more usage command tips

  • :help — Opens the help menu.
  • :filter-in <string> — Show only lines matching <string>.
  • :filter-out <string> — Hide lines matching <string>.
  • :export-to <filename> — Export current view to a file.
  • :tag <tagname> — Tag lines for later reference.
  • q — Quit (but why would you want to?).

 

7. Using lnav as a pager for systemd-journald

journalctl | lnav
# journalctl -f | lnav
# journalctl -u ssh.service | lnav

https://pc-freak.net/images/lnav_sshservice-log-view-screenshot-linux
 

8. Use lnav to review remote ssh logs

Newer versions after 0.10 supports ssh protocol as well and theoretically should work:

# lnav user@server-name-here:/var/log/file.log


To read all logs inside /var/log

# lnav root@server-name-here:/var/log/
# lnav root@server-name-here:/var/log/*.err

9. Using lnav to view docker container logs

# docker logs 811ab84aa95l | lnav
# docker logs -f application | lnav

The latest version of lnav supports even the following  simplified docker:// URL syntax:

# lnav docker://{container_id_or_name}/dir_path/to/log/file
# lnav docker://{container_id_or_name}/var/dir_path/log
# lnav docker://application/var/log/
# lnav docker://applcation/var/log/nginx/nginx.app.log

10. Monitoring compilation and command output useful for developers
 

Compilation from archived tar balls with ./configure && make etc. generate lot of outputs and logs while working. 
Here is where the tool can come handy. 
For example, here is how to watch the output of make command when compiling something:

# lnav -e './configure && make'

 11. Learning lnav tool through online ssh service availability via lnav.org

f you're lazy to install it and want to test it anyways:
 

# Start The Basic Tutorial:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password tutorial1@demo.lnav.org


# Playground:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password playground@demo.lnav.org


Closure

While tools like Kibana, Grafana, and ELK stacks are powerful, they can be overkill for many use cases—especially when you're SSHed into a box and just need to get answers fast. That’s where lnav shines as it is fast, lightweight, visual and can be used offline.

If you’re a developer, sysadmin, SRE (Site Reliability Engineer), or just someone who cares about logs, give lnav a spin. It might just become among your favorite sysadm tools on Linux and safe you pretty much of time if you have to do log reading and analyzing on daily basis (for example if you're admining 20+ or more Linux servers.