Archive for the ‘Linux’ Category

What is oddjobd and How to Use It Instead of sudo to run limited privileged execution of scripts requiring admin

Tuesday, September 30th, 2025

oddjobd-sudoers-linux-elevate-script-running-linux

In Linux environments, managing privileged operations for unprivileged users is a critical task. Traditionally, tools like sudo have been used to allow users to execute specific commands with elevated privileges. However, in more secure or fine-tuned environments — such as enterprise networks or identity-managed systems — oddjobd offers a more controlled, D-Bus-driven alternative.

This article explains what oddjobd is, how it works, and when you might prefer it over sudo, complete with real-world examples.

What is oddjobd?

oddjobd is a system service (daemon) that runs in the background and allows limited, controlled execution of privileged tasks on behalf of unprivileged users.

Key Features:

  • Allows secure execution of predefined scripts or programs as root (or another user).
  • Communicates over D-Bus for fine-grained access control.
  • Uses Polkit (PolicyKit) to manage who can run which tasks.
  • Commonly used in FreeIPA, SSSD, and LDAP-based environments.
  • Configuration files live in: /etc/oddjobd.conf.d/

How It Works

  • System administrators define specific jobs (scripts or commands) in config files.
  • These jobs are exposed via D-Bus.
  • Unprivileged users (or applications) can request jobs to be executed.
  • Access is granted or denied by Polkit rules, not passwords.
  • No full shell or terminal access is granted — just the job.
 

oddjobd vs sudo

Feature

sudo

oddjobd

Control granularity

Medium (commands)

High (methods, scripts only)

Interactive shell

Yes

No

Config complexity

Simple (/etc/sudoers)

Moderate (conf.d + Polkit)

Uses system user password

Yes

Optional (can be passwordless via Polkit)

Security

Medium

High (no shell, strict policy control)

D-Bus compatible

No

Yes

Ideal for

Power users

Controlled environments (e.g., FreeIPA)

Typical Use Cases for oddjobd


1. Automatically Creating Home Directories


Problem: LDAP/FreeIPA users don’t have home directories created on login.

Solution: Enable oddjobd to create them via oddjob-mkhomedir.

# authconfig –enablemkhomedir –update

On login, PAM calls oddjobd, which creates the home directory as root.
 

2.  Restarting a Service without sudo

Let's say you want a user to restart Apache, but not give them full sudo rights.

a. Create a script

# /usr/local/bin/restart_apache.sh

#!/bin/bash

systemctl restart apache2

echo "Apache restarted by oddjob at $(date)"

chmod +x /usr/local/bin/restart_apache.sh

b. Create Oddjob config
 

# /etc/oddjobd.conf.d/restart_apache.conf

[restart_apache]

program = /usr/local/bin/restart_apache.sh

user = root

c. Polkit rule

 

// /etc/polkit-1/rules.d/60-restart-apache.rules

polkit.addRule(function(action, subject) {

    if (action.id == "org.freedesktop.oddjob.restart_apache" &&

        subject.isInGroup("apacheadmins")) {

        return polkit.Result.YES;

    }

});

 

d. Add user to group

# groupadd apacheadmins

# usermod -aG apacheadmins alice


e. Restart and test

# systemctl restart oddjobd


# As user "alice":

oddjob_request restart_apache


Only the defined method runs — no sudo shell access, no arbitrary commands.
 

3. GUI-friendly Device Control


Use Case: A user wants to reset a USB device via a button in a GUI app.

  • Define the method in oddjobd.
  • Use Polkit for GUI D-Bus permission.
  • The app can call the method securely, without sudo.

Advantages of oddjobd

More Secure Than sudo:

  • No interactive shell or terminal.
  • No command-line injection risks.
  • Can’t “escape” to a shell like with sudo bash.

Granular Control:

  • Limit tasks to a specific script or even script arguments.

D-Bus and GUI Friendly:

  • Apps can call privileged methods without shell hacks.

Policy-Based Authorization (Polkit):

  • Fine-grained user/group access control.
  • No password prompts if not desired.

Enterprise-Ready:

  • Works well with LDAP, FreeIPA, and centralized login environments.

Oddjobd Limitations / Downsides

Limitation

Description

Learning Curve

More complex to set up than sudo

Configuration Overhead

Requires writing config files and Polkit rules

Debugging

Issues may be harder to trace than sudo logs

Not for Ad-hoc Commands

Only predefined jobs can be run

Not Installed by Default

Often needs to be manually installed (oddjob, oddjob-mkhomedir)

When to Use oddjobd Instead of sudo

Use oddjobd when you:

  • Need to allow users or apps to run very specific privileged operations.
  • Want to avoid giving full shell access via sudo.
  • Are working in a managed enterprise environment.
  • Need GUI or D-Bus-based privilege escalation.
  • Require scripted access to root tasks without exposing credentials.

Conclusion

oddjobd is a powerful tool for securely handling privileged operations in Linux, especially where tight access control and automation are required. While sudo is simple and flexible, oddjobd shines in structured, security-conscious environments — particularly those using FreeIPA, LDAP, or automated tools.

If you need a more scriptable, policy-driven, and safer alternative to sudo for specific tasks, oddjobd is well worth exploring.

Implementing and using gssproxy, example guide on how to use it to authenticate ssh, samba, nfs with no password via kerberos protocol

Friday, September 26th, 2025

Implementing and using gssproxy, example guide on how to use it to authenticate ssh, samba, nfs with no password via kerberos protocol

GSS-Proxy is a daemon that safely performs GSSAPI (Kerberos) operations on behalf of other processes. It’s useful when services running as unprivileged users need to accept or initiate Kerberos GSSAPI authentication but shouldn’t hold or access long‑lived keys (keytabs) or raw credentials themselves. Typical users: OpenSSH, SSSD, Samba, NFS idmap, and custom daemons.

This article walks through what gssproxy does, how it works, how to install and configure it, example integrations (sshd and an unprivileged service), testing, debugging and common pitfalls.
 

1. What gssproxy does (quick conceptual summary)
 

  • Runs as a privileged system daemon (typically root) and holds access to keytabs or system credentials.
  • Exposes a local IPC (Unix socket) and controlled API so allowed clients can ask it to perform GSSAPI accept/init operations on their behalf.
  • Enforces access controls by client PID/user and by named service configuration (you map a client identity to the allowed service name and keytab).
  • Minimizes the need to distribute keytabs or give services direct access to Kerberos credentials.
     

2. Installation

On many modern Linux distributions (Fedora, RHEL/CentOS, Debian/Ubuntu) gssproxy is packaged.

Example (RHEL/Fedora/CentOS):

# RHEL/CentOS 7/8/9 (dnf or yum)

 

sudo dnf install gssproxy

 

# or

 

sudo yum install gssproxy

Example (Debian/Ubuntu):

sudo apt update

sudo apt install gssproxy

If you must build from source:

# get source, then typical autotools or meson/ninja workflow per upstream README

./configure

make

sudo make install

 

After install, systemd unit gssproxy.service should be available.
 

3. Main configuration concepts

The main config file is usually /etc/gssproxy/gssproxy.conf. It consists of mechs (mechanisms), services, clients, and possibly mappings. Key elements:

  • mech: declares a GSSAPI mechanism (e.g., krb5) and default keytab(s) for acceptor credentials.
  • service: logical service names (e.g., ssh, nfs, httpd) with attributes: user (the Unix user running the service), keytab, cred_store, mechs, and whether the service is allowed to be client (initiate) and/or server (accept).
  • client: rules mapping local client sockets / users / pids to allowed services.

A minimal working example that allows sshd to use gssproxy:

mechs = {

    krb5_mech = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5_mech;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        # allow both acceptor (server) and initiator (client) ops if needed

        client = yes;

        server = yes;

    };

};

Client rules are often implicit: gssproxy can enforce that calls on a given service socket originate from the configured Unix user. For more complex setups you add policy and client blocks. Example to allow a specific PID or user to use the ssh service:

clients = {

    ssh_clients = {

        clients = [

            { match = "uid:0" },      # root can ask for ssh service

            { match = "user:sshd" },  # or the sshd user

        ];

        service = "ssh";

    };

};

Paths and sockets: gssproxy listens on a socket (e.g. /var/run/gssproxy/socket) and possibly per-user sockets (e.g. /run/gssproxy/uid_1000). The systemd unit usually creates the runtime directory with correct permissions.
 

4. Example: Integrate with OpenSSH server (sshd)

Goal: allow sshd and session processes to accept delegated GSS credentials and let unprivileged child processes use those credentials via gssproxy.

Server side config

  1. Ensure sshd is built/installed with GSSAPI support. On SSH server:

    • In /etc/ssh/sshd_config:
    • GSSAPIAuthentication yes
    • GSSAPICleanupCredentials yes
    • GSSAPIKeyExchange yes        # optional: if you want GSS key exchange
  2. Configure gssproxy with an ssh service entry pointing to the host keytab (so gssproxy can accept SPNEGO/kerberos accept_sec_context calls):

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        server = yes;

        client = yes;

    };

};

  1. Ensure /etc/krb5.keytab contains the host principal host/fqdn@REALM (or host/short@REALM depending on SPN strategy). Use ktutil or kadmin to create/populate.
  2. Restart gssproxy and sshd:

sudo systemctl restart gssproxy

sudo systemctl restart sshd

Client side

  • ssh client configuration (usually ~/.ssh/config or /etc/ssh/ssh_config):

Host myhost.example.com

    GSSAPIAuthentication yes

    GSSAPIDelegateCredentials yes

Client must have a TGT in the credential cache (kinit user), or use a client that acquires one.

Result

When the client initiates GSSAPI authentication and delegates credentials (GSSAPIDelegateCredentials yes or -K for older OpenSSH), gssproxy on the server handles acceptor functions. If a session process needs to use the delegated credentials (e.g., to access network resources as that user), gssproxy arranges a per-session credential store that unprivileged processes can use via the kernel keyring or other mechanisms gssproxy supports.
 

5. Example: Allow an unprivileged service to acquire initiator creds via gssproxy

Suppose a service mydaemon runs as myuser and needs to initiate Kerberos-authenticated connections using a specific service principal stored in /etc/mydaemon.keytab but you don’t want to expose that keytab to myuser.

Add a mech and service:

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

    mydaemon_mech = {

        mech = krb5;

        default_keytab = /etc/mydaemon.keytab;

    };

};

 

services = {

    mydaemon = {

        mech = mydaemon_mech;

        user = "myuser";

        keytab = /etc/mydaemon.keytab;

        client = yes;    # allow initiator operations

        server = no;

    };

};

Configure a client mapping so the mydaemon process (uid myuser) is allowed to use the mydaemon service. Once gssproxy runs, mydaemon uses the gssapi libraries (GSSAPI libs detect gssproxy via environment or library probe) and calls the GSSAPI functions; gssproxy will perform gss_acquire_cred using /etc/mydaemon.keytab and return a handle to the calling process. The service itself never directly reads the keytab.
 

6. Testing and tools

  • kinit / klist: manage and list Kerberos TGTs on clients.
  • journalctl -u gssproxy -f (or systemctl status gssproxy) to watch logs.
  • ss -l or ls -l /run/gssproxy to inspect sockets.
  • If you have gssproxy command-line utilities installed (may vary by distro), some installations include gssproxy CLI helpers. Otherwise use the service that relies on gssproxy and watch logs.

Example basic tests:

  1. Ensure gssproxy is running:

sudo systemctl status gssproxy

  1. On server, check socket and permissions:

sudo ls -l /run/gssproxy

# or

sudo ss -x -a | grep gssproxy

  1. Attempt SSH from a client with a TGT:

kinit alice

ssh -o GSSAPIDelegateCredentials=yes alice@server.example.com

# then on server, check journalctl logs for gssproxy/sshd messages
 

7. Debugging tips

  • Journal logs: journalctl -u gssproxy -xe will be your first stop.
  • Permissions: Ensure that gssproxy can read the keytab(s) (typically root-owned with restrictive perms). In config you may point to a keytab readable only by gssproxy.
  • Clients blocked: If a client is denied, check the clients block and match rules (uid/pid/user).
  • Keytab issues: Use klist -k /etc/krb5.keytab to list principals in a keytab. Ensure correct SPN and realm.
  • Clock skew: Kerberos is time-sensitive. Ensure NTP/chrony is working.
  • DNS / SPNs: Ensure hostnames and reverse DNS match the principal names expected for the service.
  • SSHD integration: If sshd still complains it can’t accept GSSAPI creds, enable debug logging (LogLevel DEBUG), and check gssproxy logs.
  • SELinux: On SELinux-enabled systems, you may need to ensure file contexts and SELinux policies allow gssproxy to access keytabs and sockets. Check audit.log for AVC denials and use semanage fcontext/restorecon or local policy modules when needed.
     

8. Common pitfalls & best practices

  • Don’t expose keytabs to unprivileged users. Let gssproxy hold them.
  • Principals & SPNs must match service hostnames used by clients. Consistent DNS is essential.
  • Minimal privileges: configure services and clients narrowly: allow only the minimum users/PIDs and only the required mech ops.
  • Rotation: when rotating keytabs, reload/restart gssproxy or send a signal if supported. Plan for keytab updates.
  • Logging: enable adequate logging during deployment and revert to normal verbosity in production.
  • Testing in staging: GSSAPI behavior across SSH clients and other daemons can be subtle — test across your client set (Linux, macOS, Windows via native Kerberos clients, etc.).
     

9. Security considerations

  • gssproxy centralizes credential access: secure the host and the gssproxy process.
  • Protect keytab files using strict filesystem permissions and (if needed) SELinux policy.
  • Restrict which local processes may request operations for a service — map by UID/PID carefully.
  • Monitor logs for unexpected use of gssproxy.
     

10. Example full config (simple)

Save as /etc/gssproxy/gssproxy.conf:

mechs = {

    krb5 = {

        mech = krb5;

        default_keytab = /etc/krb5.keytab;

    };

};

 

services = {

    ssh = {

        mech = krb5;

        user = "sshd";

        keytab = /etc/krb5.keytab;

        server = yes;

        client = yes;

    };

 

    mydaemon = {

        mech = krb5;

        user = "myuser";

        keytab = /etc/mydaemon.keytab;

        client = yes;

        server = no;

    };

};

 

clients = {

    allow_root_for_ssh = {

        clients = [

            { match = "uid:0" },

        ];

        service = "ssh";

    };

 

    mydaemon_client = {

        clients = [

            { match = "user:myuser" },

        ];

        service = "mydaemon";

    };

};

Restart: sudo systemctl restart gssproxy and then restart dependent services (sshd, mydaemon, etc.) if needed.

 

Useful resources for gssproxy and further integrations

  • Read your distribution’s /usr/share/doc/gssproxy/ or man pages (man gssproxy, man gssproxy.conf) — they contain distribution-specific details.
  • Check integrations: Samba/Winbind, SSSD, NFS idmap — many modern stacks support gssproxy as an option to avoid exposing keytabs to many daemons.
  • For production: automate keytab distribution, rotation and monitor gssproxy usage.

 

How to Install and Use auditd for System Security Auditing on Linux

Thursday, September 25th, 2025

System auditing is essential for monitoring user activity, detecting unauthorized access, and ensuring compliance with security standards. On Linux, the Audit Daemon (auditd) provides powerful auditing capabilities for logging system events and actions.

This short article will walk you through installing, configuring, and using auditd to monitor your Linux system.

What is auditd?

auditd is the user-space component of the Linux Auditing System. It logs system calls, file access, user activity, and more — offering administrators a clear trail of what’s happening on the system.


1. Installing auditd

The auditd package is available by default in most major Linux distributions.

 On Debian/Ubuntu

# apt update
# apt install auditd audispd-plugins

 On CentOS/RHEL/Fedora

# yum install audit

After installation, start and enable the audit daemon

# systemctl start auditd

# systemctl enable auditd

Check its status

# systemctl status auditd

2. Setting Audit Rules

Once auditd is running, you need to define rules that tell it what to monitor.

Example: Monitor changes to /etc/passwd

# auditctl -w /etc/passwd -p rwxa -k passwd_monitor

Explanation:

  • -w /etc/passwd: Watch this file. When the file is accessed, the watcher will generate events.
  • -p rwxa: Monitor read, write, execute, and attribute changes
  • -k passwd_monitor: Assign a custom key name to identify logs. Later on, we could search for this (arbitrary) passwd string to identify events tagged with this key.

List active rules:

# auditctl -l

3. Common auditd Rules for Security Monitoring

Here are some common and useful auditd rules you can use to monitor system activity and enhance Linux system security. These rules are typically added to the /etc/audit/rules.d/audit.rules or /etc/audit/audit.rules file, depending on your system.

a. Monitor Access to /etc/passwd and /etc/shadow
 

-w /etc/passwd -p wa -k passwd_changes
-w /etc/shadow -p wa -k shadow_changes

  • Monitors read/write/attribute changes to password files.

b. Monitor sudoers file and directory
 

-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers

  • Tracks any change to sudo configuration files.

c. Monitor Use of chmod, chown, and passwd
 

-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -k perm_mod
-a always,exit -F arch=b64 -S passwd -k passwd_changes

  • Watches permission and ownership changes.

d. Monitor User and Group Modifications

-w /etc/group -p wa -k group_mod
-w /etc/gshadow -p wa -k gshadow_mod
-w /etc/security/opasswd -p wa -k opasswd_mod

  • Catches user/group-related config changes.

e. Track Logins, Logouts, and Session Initiation

-w /var/log/lastlog -p wa -k logins
-w /var/run/faillock/ -p wa -k failed_login
-w /var/log/faillog -p wa -k faillog

  • Tracks login attempts and failures.

f. Monitor auditd Configuration Changes

-w /etc/audit/ -p wa -k auditconfig
-w /etc/audit/audit.rules -p wa -k auditrules

  • Watches changes to auditd configuration and rules.

g. Detect Changes to System Binaries

-w /bin/ -p wa -k bin_changes
-w /sbin/ -p wa -k sbin_changes
-w /usr/bin/ -p wa -k usr_bin_changes
-w /usr/sbin/ -p wa -k usr_sbin_changes

  • Ensures core binaries aren't tampered with.

h. Track Kernel Module Loading and Unloading

-a always,exit -F arch=b64 -S init_module -S delete_module -k kernel_mod

  • Detects dynamic kernel-level changes.

l. Monitor File Deletions

-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k delete

  • Tracks when files are removed or renamed.

m. Track Privilege Escalation via setuid/setgid

-a always,exit -F arch=b64 -S setuid -S setgid -k priv_esc

  • Helps detect changes in user or group privileges.

n. Track Usage of Dangerous Binaries (e.g., su, sudo, netcat)

-w /usr/bin/su -p x -k su_usage
-w /usr/bin/sudo -p x -k sudo_usage
-w /bin/nc -p x -k netcat_usage

  • Useful for catching potentially malicious command usage.

o. Monitor Cron Jobs

-w /etc/cron.allow -p wa -k cron_allow
-w /etc/cron.deny -p wa -k cron_deny
-w /etc/cron.d/ -p wa -k cron_d
-w /etc/crontab -p wa -k crontab
-w /var/spool/cron/ -p wa -k user_crontabs

  • Alerts on cron job creation/modification.

p. Track Changes to /etc/hosts and DNS Settings

-w /etc/hosts -p wa -k etc_hosts
-w /etc/resolv.conf -p wa -k resolv_conf

  • Monitors potential redirection or DNS manipulation.

q. Monitor Mounting and Unmounting of Filesystems

-a always,exit -F arch=b64 -S mount -S umount2 -k mounts

  • Useful for detecting USB or external drive activity.

r. Track Execution of New Programs

-a always,exit -F arch=b64 -S execve -k exec

  • Captures command execution (can generate a lot of logs).
     

A complete list of rules you can get from the hardening.rules auditd file place it under /etc/audit/rules.d/hardening.rules
and reload auditd to load the configurations.

Tips

  • Use ausearch -k <key> to search audit logs for matching rule.
  • Use auditctl -l to list active rules.
  • Use augenrules –load after editing rules in /etc/audit/rules.d/.


4. Reading Audit Logs

Audit logs events are stored in:

/var/log/audit/audit.log

By default, the location, this can be changed through /etc/auditd/auditd.conf

View recent entries:
 

# tail -f /var/log/audit/audit.log

Search by key:
 

# ausearch -k passwd_monitor

Generate a summary report:

# aureport -f

# aureport


Example: Show all user logins / IPs :

# aureport -au

 

5. Making Audit Rules Persistent

Rules added with auditctl are not persistent and will be lost on reboot. To make them permanent:

Edit the audit rules configuration:

# vim /etc/audit/rules.d/audit.rules

Add your rules, for example:

-w /etc/passwd -p rwxa -k passwd_monitor

Apply the rules:

# augenrules –load

7. Some use case examples of auditd in auditing Linux servers by sysadmins / security experts
 

Below are real-world, practical examples where auditd is actively used by sysadmins, security teams, or compliance officers to detect suspicious activity, meet compliance requirements, or conduct forensic investigations.

a. Detect Unauthorized Access to /etc/shadow

Use Case: Someone tries to read or modify password hashes.

Audit Rule:

-w /etc/shadow -p wa -k shadow_watch

Real-World Trigger:

sudo cat /etc/shadow

Check Logs:
 

# ausearch -k shadow_watch -i

Real Output:
 

type=SYSCALL msg=audit(09/18/2025 14:02:45.123:1078):

  syscall=openat

  exe="/usr/bin/cat"

  success=yes

  path="/etc/shadow"

  key="shadow_watch"

b. Detect Use of chmod to Make Files Executable

Use Case: Attacker tries to make a script executable (e.g., malware).

Audit Rule:

-a always,exit -F arch=b64 -S chmod -k chmod_detect

Real-World Trigger:
 

 # chmod +x /tmp/evil_script.sh

Check Logs:

# ausearch -k chmod_detect -i

c. Monitor Execution of nc (Netcat)

Use Case: Netcat is often used for reverse shells or unauthorized network comms.

Audit Rule:
 

-w /bin/nc -p x -k netcat_usage
 

Real-World Trigger:

nc -lvp 4444

Log Entry:

type=EXECVE msg=audit(09/18/2025 14:35:45.456:1123):

  argc=3 a0="nc" a1="-lvp" a2="4444"

  key="netcat_usage"

 

d. Alert on Kernel Module Insertion
 

Use Case: Attacker loads rootkit or malicious kernel module.

Audit Rule:

-a always,exit -F arch=b64 -S init_module -S delete_module -k kernel_mod

Real-World Trigger:

# insmod myrootkit.ko

Audit Log:
 

type=SYSCALL msg=audit(09/18/2025 15:00:13.100:1155):

  syscall=init_module

  exe="/sbin/insmod"

  key="kernel_mod"

e. Watch for Unexpected sudo Usage

Use Case: Unusual use of sudo might indicate privilege escalation.

Audit Rule:

-w /usr/bin/sudo -p x -k sudo_watch

Real-World Trigger:

sudo whoami

View Log:
 

# ausearch -k sudo_watch -i


f. Monitor Cron Job Modification

Use Case: Attacker schedules persistence via cron.

Audit Rule:

-w /etc/crontab -p wa -k cron_mod

Real-World Trigger:
 

echo "@reboot /tmp/backdoor" >> /etc/crontab

Logs:
 

type=SYSCALL msg=audit(09/18/2025 15:05:45.789:1188):

  syscall=open

  path="/etc/crontab"

  key="cron_mod"

g. Detect File Deletion or Renaming
 

Use Case: Attacker removes logs or evidence.

Audit Rule:

-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k file_delete

Real-World Trigger:

# rm -f /var/log/syslog

Logs:
 

type=SYSCALL msg=audit(09/18/2025 15:10:33.987:1210):

  syscall=unlink

  path="/var/log/syslog"

  key="file_delete"


h. Detect Script or Malware Execution
 

Use Case: Capture any executed command.

Audit Rule:
 

-a always,exit -F arch=b64 -S execve -k exec

Real-World Trigger:

/tmp/myscript.sh

Log View:

# ausearch -k exec -i | grep /tmp/myscript.sh

l. Detect Manual Changes to /etc/hosts

Use Case: DNS hijacking or phishing setup.

Audit Rule:

-w /etc/hosts -p wa -k etc_hosts

Real-World Trigger:
 

# echo "1.2.3.4 google.com" >> /etc/hosts

Logs:

type=SYSCALL msg=audit(09/18/2025 15:20:11.444:1234):

  path="/etc/hosts"

  syscall=open

  key="etc_hosts"


8. Enable Immutable Mode (if necessery)

For enhanced security, you can make audit rules immutable, preventing any changes until reboot:

# auditctl -e 2


To make this setting persistent, add the following to the end of /etc/audit/rules.d/audit.rules:

-e 2


Common Use Cases

Here are a few more examples of what you can monitor:

Monitor all sudo usage:

# auditctl -w /var/log/auth.log -p wa -k sudo_monitor


Monitor a directory for file access:

# auditctl -w /home/username/important_dir -p rwxa -k dir_watch

Audit execution of a specific command (e.g., rm):

# auditctl -a always,exit -F arch=b64 -S unlink,unlinkat -k delete_cmd

(Adjust arch=b64 to arch=b32 if on 32-bit system.)

9. Managing the Audit Log Size

Audit logs can grow large over time. To manage log rotation and size, edit:
 

# vim /etc/audit/auditd.conf

Set log rotation options like:

max_log_file = 8

num_logs = 5

Then restart auditd:
 

# systemctl restart auditd

Conclusion

The Linux Audit Daemon (auditd) is a powerful tool to track system activity, enhance security, and meet compliance requirements. With just a few configuration steps, you can monitor critical files, user actions, and system behavior in real time.

 

References

  • man auditd
  • man auditctl
  • Linux Audit Wiki

 

Unlocking the Power of lnav: Logfile Navigator – ncurses text based tool guide to mutiple Logs on multiple servers easy analysis on Linux

Saturday, September 13th, 2025

lnav-syslog-screenshot-linux-virtual-machine

If you've ever found yourself buried under a mountain of log files, tailing multiple outputs, or grepping through endless lines trying to spot an error, it's time to meet your new best friend: lnav, the Logfile Navigator.

Lightweight, terminal-based, and surprisingly powerful, lnav is one of the most underrated tools for developers, sysadmins, and anyone who regularly digs into logs. It turns your chaotic logs into something that’s not only readable—but genuinely useful.

What is lnav and why use it ?

lnav (Logfile Navigator) is a command-line tool for viewing and analyzing log files. It goes beyond tail, less, or grep by:

  • Automatically detecting and merging log formats.
  • Highlighting timestamps, log levels, and errors.
  • Providing SQL-like queries over your logs.
  • Offering interactive navigation with a UI inside the terminal.

And yes, all of that without needing to set up a database or a server.

1. Installing lnav on Linux

Installation is straightforward. On most systems, you can install it via package managers:

On Ubuntu/Debian:

# apt install lnav

On Fedora:

# dnf install lnav

On Arch Linux:

# pacman -S lnav

Or build from source via GitHub if you want the latest version.

2. Use lnav Instead of Tail / Grep why?

Traditional tools are powerful, but they require manual work to chain together functionality. lnav gives you:

  • Automatic multi-log parsing: Drop multiple logs in, and it merges them chronologically.
  • Syntax highlighting: Errors and warnings stand out.
  • SQL querying: Run queries like SELECT * FROM syslog_log WHERE log_level = 'error';
  • Filtering and searching: Use intuitive filters and bookmarks to highlight specific entries.

3. Basic tool Usage is simple

Let’s say you want to inspect a system log:

# lnav /var/log/syslog

You'll immediately get:

  • Color-coded output (timestamps, levels, messages).
  • Scrollable view (arrow keys, PgUp, PgDn).
  • Real-time updates (like tail -f).
  • Search with /, filter with :filter-in, and even SQL queries.

Lets say you need to analyze Apache webserver logs recursively including the logs already rotated and gunzipped with *.gz extension on CentOS / Fedora / RHEL, you can do it with:

# lnav -r /var/log/httpd

You can parse the log file and get additional information about requests as well as you can print overall summary of log file.

Choose the line you want to parse. The selected line is always the one at the top of the window. Then press 'p' and you should see the following result:

https://pc-freak.net/images/lnav-get-extra-information-about-apache-query-with-P-press-key-screenshot-linux

Now, if you want to see a summary view of the logs by date and time, simply press 'i'.

lnav-linux-apache-log-review-summary-of-errors-warnings-normal-screenshot

To quit a screen you have chosen press 'q'.

4. LNAV helpful options and hotkeys

Once you've opened a log file/s for analyze you can use few hotkeys that will allow us to move through the output of lnav and the available views more easily:

e or E to jump to the next / previous error message.
w or W to jump to the next / previous warning message.
b or Backspace to move to the previous page.
Space to move to the next page.
g or G to move to the top / bottom of the current view.

To take a closer look at the way lnav operates, use -d option, the debug information is to be spit inside a .txt file:

# lnav /var/log/httpd -d lnav.txt

In this example, the debug information that is generated when lnav starts will be written to a file named lnav.txt inside the current working directory.

5. Real-World Use Cases

a. Troubleshooting application or system process Crashes

Open all relevant logs in one go:

# lnav /var/log/*.log

Errors are highlighted, and you can jump between them with n / N kbd keys.

b. Combining Multiple Logs

Working with an app that logs to different files and you need to combine:

# lnav /var/log/nginx/access.log /var/log/nginx/error.log


Or lets say you want to combine Apache Webserver with Haproxy log and get log summaries or filter out stuff:

lnav /var/log/apache2/access.log /var/log/haproxy.log


Now you will get a single, chronological timeline of events.

 

If you want to Search for a concrete occurance of Error / Warning or IP address inside a bunch of loaded combined logs you can do it with the same command like in simple vim by pressing / (slash) from kbd and type out what you want to filter out to get shown.

c. Analyze SQL Queries Logs

Yes, you can actually do this by passing it query in its command prompt :

:.schema
:SELECT log_time, log_level, log_message FROM syslog_log WHERE log_level = 'error';

You get a table of filtered logs, sortable by columns.
 

6. lnav more usage command tips

  • :help — Opens the help menu.
  • :filter-in <string> — Show only lines matching <string>.
  • :filter-out <string> — Hide lines matching <string>.
  • :export-to <filename> — Export current view to a file.
  • :tag <tagname> — Tag lines for later reference.
  • q — Quit (but why would you want to?).

 

7. Using lnav as a pager for systemd-journald

journalctl | lnav
# journalctl -f | lnav
# journalctl -u ssh.service | lnav

https://pc-freak.net/images/lnav_sshservice-log-view-screenshot-linux
 

8. Use lnav to review remote ssh logs

Newer versions after 0.10 supports ssh protocol as well and theoretically should work:

# lnav user@server-name-here:/var/log/file.log


To read all logs inside /var/log

# lnav root@server-name-here:/var/log/
# lnav root@server-name-here:/var/log/*.err

9. Using lnav to view docker container logs

# docker logs 811ab84aa95l | lnav
# docker logs -f application | lnav

The latest version of lnav supports even the following  simplified docker:// URL syntax:

# lnav docker://{container_id_or_name}/dir_path/to/log/file
# lnav docker://{container_id_or_name}/var/dir_path/log
# lnav docker://application/var/log/
# lnav docker://applcation/var/log/nginx/nginx.app.log

10. Monitoring compilation and command output useful for developers
 

Compilation from archived tar balls with ./configure && make etc. generate lot of outputs and logs while working. 
Here is where the tool can come handy. 
For example, here is how to watch the output of make command when compiling something:

# lnav -e './configure && make'

 11. Learning lnav tool through online ssh service availability via lnav.org

f you're lazy to install it and want to test it anyways:
 

# Start The Basic Tutorial:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password tutorial1@demo.lnav.org


# Playground:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password playground@demo.lnav.org


Closure

While tools like Kibana, Grafana, and ELK stacks are powerful, they can be overkill for many use cases—especially when you're SSHed into a box and just need to get answers fast. That’s where lnav shines as it is fast, lightweight, visual and can be used offline.

If you’re a developer, sysadmin, SRE (Site Reliability Engineer), or just someone who cares about logs, give lnav a spin. It might just become among your favorite sysadm tools on Linux and safe you pretty much of time if you have to do log reading and analyzing on daily basis (for example if you're admining 20+ or more Linux servers.

 

How To Install ChatGPT on Debian Linux with snap

Tuesday, August 19th, 2025

chatgpt-desktop-linux-screenshot

To install ChatGPT (official desktop app) on Debian Linux using Snap, do the following:
You need as

Prerequisites

  1. Debian-based system (e.g., Debian, Ubuntu, Mint whatever deb based Linux).

  2. Snap package manager installed.


1. Install Snap (if not installed)

Run these commands in your terminal:

# apt update sudo apt install snapd

Enable and start the Snap daemon:

# systemctl enable snapd sudo systemctl start snapd

Create a symbolic link to ensure

snap

is accessible:

# ln -s /var/lib/snapd/snap /snap


2. Find ChatGPT Snap Package

The official ChatGPT desktop app (by OpenAI) is not available as a Snap package directly from OpenAI, but a third-party Snap package or wrapper may exist.

You can search with:

# snap find chatgpt


As of now, you might see unofficial community packages (e.g.,

chatgpt-desktop

or

chatgpt-wrapper

, etc.).

 

3. Install ChatGPT Snap Package

If you find a package (e.g., chatgpt-desktop), install it like this:

# snap install chatgpt-desktop

Note: Be cautious about third-party Snap packages—review the publisher and permissions.

4.Launch ChatGPT

Once installed, launch it from your app menu or run:

/snap/bin/chatgpt-desktop-client

 

Alternative (if no Snap available)

If no Snap package is available or you're uncomfortable with third-party sources:

Option: Use the Official .deb Installer

OpenAI released an official desktop app for Linux in

.deb

format:

To use the native deb;

  1. Download from: https://openai.com/chat

  2. Install with:

# apt install ./chatgpt_*.deb

 

Deploying a Puppet Server and Patching multiple Debian Linux Servers

Tuesday, June 17th, 2025

how-to-install-puppet-server-clients-for-multiple-servers-and-services-infrastructure-logo

 

Puppet Overview

Puppet is a powerful automation tool used to manage configurations and automate administrative tasks across multiple systems. This guide walks you through installing a Puppet server (master) and configuring 10 Debian-based client servers (agents) to automatically install system updates (patches) using Puppet.

Table of Contents

  1. Prerequisites

  2. Step 1: Install Puppet Server on Debian

  3. Step 2: Configure the Puppet Server

  4. Step 3: Install Puppet Agent on 10 Debian Clients

  5. Step 4: Sign Agent Certificates

  6. Step 5: Create a Puppet Module for Patching

  7. Step 6: Assign the Module and Trigger Updates

  8. Conclusion


1. Prerequisites

  • Debian server to act as the Puppet master (e.g., Debian 11)
     
  • Debian servers as Puppet agents (clients)
     
  • Root or sudo access on all systems
     
  • Static IPs or properly configured hostnames
     
  • Network connectivity between master and agents
     

2. Install Puppet Server on Debian

a. Add the Puppet APT repository

# wget https://apt.puppet.com/puppet7-release-bullseye.deb
# dpkg -i puppet7-release-bullseye.deb
# apt update

b. Install Puppet Server

# apt install puppetserver -y

c. Configure JVM memory (optional but recommended)

Edit /etc/default/puppetserver:  

JAVA_ARGS="-Xms512m -Xmx1g"


d. Enable and start the Puppet Server

# systemctl enable puppetserver 
# systemctl start puppetserver


3. Configure the Puppet Server

a. Set the hostname
 

# hostnamectl set-hostname puppet.example.com


Update /etc/hosts with your server’s IP and FQDN if DNS is not configured:  

192.168.1.10 puppet.pc-freak.net puppet

 

b. Configure Puppet

Edit /etc/puppetlabs/puppet/puppet.conf:

[main] certname = puppet.pc-freak.net
server = puppet.pc-freak.net
environment = production
runinterval = 1h

 

Restart Puppet server:

# systemctl restart puppetserver

4. Install Puppet Agent on 10 Debian Clients

Repeat this section on each client server (Debian 10/11).

a. Add the Puppet repository

# wget https://apt.puppet.com/puppet7-release-bullseye.deb
# dpkg -i puppet7-release-bullseye.deb 
# apt update


b. Install the Puppet agent

# apt install puppet-agent -y


c. Configure the agent to point to the master

# /opt/puppetlabs/bin/puppet config set server puppet.example.com –section main


d. Start the agent to request a certificate

# /opt/puppetlabs/bin/puppet agent –test

 

5. Sign Agent Certificates on the Puppet Server

Run on the Puppet master below 2 cmds:

# /usr/bin/puppetserver ca list –all


Sign all pending requests:

# /usr/bin/puppetserver ca sign –all

Verify connection to puppet server is fine:

# /opt/puppetlabs/bin/puppet node find haproxy2.pc-freak.net


6.  Create a Puppet Module for Patching

a. Create the patching module

# mkdir -p /etc/puppetlabs/code/environments/production/modules/patching/manifests


b. Add a manifest file
/etc/puppetlabs/code/environments/production/modules/patching/manifests/init.pp
:

class patching {

  exec { 'apt_update':
    command => '/usr/bin/apt update',
    path    => [‘/usr/bin’, ‘/usr/sbin’],
    unless  => '/usr/bin/test $(find /var/lib/apt/lists/ -type f -mmin -60 | wc -l) -gt 0',
  }

  exec { 'apt_upgrade':
    command => '/usr/bin/apt upgrade -y',
    path    => [‘/usr/bin’, ‘/usr/sbin’],
    require => Exec[‘apt_update’],
    unless  => '/usr/bin/test $(/usr/bin/apt list –upgradable 2>/dev/null | wc -l) -le 1',
  }

}

This class updates the package list and applies all available security and feature updates.

7.  Assign the Module and Trigger Updates

a. Edit site.pp on the Puppet master:

 

# vim /etc/puppetlabs/code/environments/production/manifests/site.pp

 

node default {
  include patching
}

node 'agent1.example.com' {
  include patching
}

b. Run Puppet manually on each agent to test:

# /opt/puppetlabs/bin/puppet agent –test

Once confirmed working, Puppet agents will run this patching class automatically every hour (default runinterval).

8. Check the status of puppetserver and puppet agent on hosts is fine
 

root@puppetserver:/etc/puppet# systemctl status puppetserver
● puppetserver.service – Puppet Server
     Loaded: loaded (/lib/systemd/system/puppetserver.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-06-16 23:44:42 EEST; 37min ago
       Docs: https://puppet.com/docs/puppet/latest/server/about_server.html
    Process: 2166 ExecStartPre=sh -c echo -n 0 > ${RUNTIME_DIRECTORY}/restart (code=exited, status=0/SUCCESS)
    Process: 2168 ExecStartPost=sh -c while ! head -c1 ${RUNTIME_DIRECTORY}/restart | grep -q '^1'; do kill -0 $MAINPID && sleep 1 || exit 1; done (code=exited, status=0/SUCCESS)
   Main PID: 2167 (java)
      Tasks: 64 (limit: 6999)
     Memory: 847.0M
        CPU: 1min 28.704s
     CGroup: /system.slice/puppetserver.service
             └─2167 /usr/bin/java -Xms512m -Xmx1g -Djruby.lib=/usr/share/jruby/lib -XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/var/log/puppetserver/puppetserver_err_pid%p.log -jar /usr/share/pup>

юни 16 23:44:06 haproxy2 systemd[1]: Starting puppetserver.service – Puppet Server…
юни 16 23:44:30 haproxy2 java[2167]: 2025-06-16T23:44:30.516+03:00 [clojure-agent-send-pool-0] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
юни 16 23:44:30 haproxy2 java[2167]: Pass '–add-opens java.base/sun.nio.ch=ALL-UNNAMED –add-opens java.base/java.io=ALL-UNNAMED' to enable.
юни 16 23:44:42 haproxy2 systemd[1]: Started puppetserver.service – Puppet Server.

root@grafana:/etc/puppet# systemctl status puppet
* puppet.service – Puppet agent
     Loaded: loaded (/lib/systemd/system/puppet.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-06-16 21:22:17 UTC; 18s ago
       Docs: man:puppet-agent(8)
   Main PID: 1660157 (puppet)
      Tasks: 6 (limit: 2307)
     Memory: 135.6M
        CPU: 5.303s
     CGroup: /system.slice/puppet.service
             |-1660157 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent –no-daemonize
             `-1660164 "puppet agent: applying configuration"

Jun 16 21:22:17 grafana systemd[1]: Started puppet.service – Puppet agent.
Jun 16 21:22:28 grafana puppet-agent[1660157]: Starting Puppet client version 7.34.0
Jun 16 21:22:33 grafana puppet-agent[1660164]: Requesting catalog from puppet.pc-freak.net:8140 (192.168.1.58)

9. Use Puppet facter to extract interesting information from the Puppet running OS
 

facter is a powerful command-line tool Puppet agents use to gather system information (called facts). You can also run it manually on any machine to quickly inspect system details.

Here are some interesting examples to get useful info from a machine using facter:


a) Get all facts about Linux OS

facter

 

b) get OS name / version

facter os.name os.release.full
os.name => Debian
os.release.full => 12.10


c) check the machine  hostname and IP address

$ facter hostname ipaddress

hostname => puppet-client1
ipaddress => 192.168.0.220

d) Get amount of RAM on the machine

$ facter memorysize
16384 MB


e) Get CPU (Processor information)

$ facter processors

{
  count => 4,
  models => [“Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz”],
  physicalcount => 1,
  speed => "1.60 GHz"
}

10. Conclusion

You've successfully set up a Puppet server and configured a sample Debian client systems to automatically install security patches using a custom module.
To apply this on the rest of systems where puppet agent is installed repeat the process on each of the left 9 nodes.
This approach provides centralized control, consistent configuration, and peace of mind for you as system administrator if you have the task to manage multiple Debian servers
with an easy.
Of course the given configuration is very sample and to guarantee proper functiononing of your infrastrcutreu you'll have to read and experiment with puppet, however I hope article is a good
standpoint to have puppet server / agent running relatively easy.

Deploying a Server and Managing a 10-Node Linux Infrastructure with Ansible

Friday, May 30th, 2025

File:Ansible Logo.png - Wikimedia Commons

As organizations grow, manually configuring and maintaining servers quickly becomes inefficient, error-prone, and unscalable. Ansible, a powerful open-source automation tool, allows sysadmins and DevOps engineers to automate server provisioning, configuration management, and application deployment across multiple nodes—making it ideal for managing a fleet of Linux servers.

In this article, you’ll learn how to deploy a server using Ansible and manage an infrastructure of 10 Linux servers with ease.


What Is Ansible?

Ansible is a suite of software tools that enables infrastructure as code. It is open-source and the suite includes software provisioning, configuration management, and application deployment functionality.Ansible is an agentless IT automation tool. It uses SSH to connect to remote machines and YAML-based playbooks to define automation tasks. With Ansible, you can:

  • Install and configure software packages
  • Enforce configuration consistency
  • Perform updates across nodes
  • Deploy applications
  • Orchestrate complex workflows
     

Pre-requisites

To follow this guide, you need:

  • A control node (the machine from which you run Ansible)
  • 10 Linux servers (can be VMs or physical machines)
  • SSH access from the control node to all servers
  • Python installed on the target machines (most Linux distros have this by default)
     

1. Install Ansible on the Control Node 

On Ubuntu / Debian Linux

# apt update sudo apt install ansible -y

Or for CentOS/RHEL

# yum install epel-release -y
# yum install ansible -y

Verify installation:

# ansible –version

2. Configure Your Inventory File

Ansible uses an inventory file to define which servers to manage. By default, this is located at /etc/ansible/hosts, but you can also create a custom one.

Create an inventory file hosts.ini

[webservers]
server1 ansible_host=192.168.1.101
server2 ansible_host=192.168.1.102
server3 ansible_host=192.168.1.103

[dbservers]
server4 ansible_host=192.168.1.104

[all:vars]
ansible_user=ansible_user
ansible_ssh_private_key_file=~/.ssh/id_rsa

You can also use hostnames if DNS is configured.

3. Test Connectivity

# ansible -i hosts.ini all -m ping

If everything is set up correctly, you should see pong responses from all servers.

4. Write Your First Playbook (Provisioning Example)

Create a file called server_setup.yml:


– name: Provision and configure Linux servers
  hosts: all
  become: yes
  tasks:

    – name: Update apt cache (Debian/Ubuntu)
      apt:
        update_cache: yes
      when: ansible_os_family == "Debian"

    – name: Install common packages
      package:
        name:
          – curl
          – git
          – htop
        state: present

    – name: Ensure Nginx is installed on webservers
      apt:
        name: nginx
        state: present
      when: "'webservers' in group_names"

    – name: Start and enable Nginx
      service:
        name: nginx
        state: started
        enabled: yes
      when: "'webservers' in group_names"
 

5. Run the Playbook

ansible-playbook -i hosts.ini server_setup.yml

Ansible will SSH into each server, execute the tasks, and return status messages.

6. Scaling to 10+ Servers

Managing a growing infrastructure is simple with Ansible:

  • Add new servers to hosts.ini
  • Reuse existing playbooks for setup
  • Use roles to modularize configuration (e.g., webserver, database, monitoring)
  • Schedule playbooks using cron or CI/CD pipelines (e.g., GitHub Actions, Jenkins)

7. Sample Ansible Project Structure

Organizing your Ansible project effectively is crucial for scalability and maintainability. Here's a sample recommended directory layout:

ansible-infra/
├── hosts.ini
├── server_setup.yml
├── roles/
│   ├── common/
│   │   ├── tasks/
│   │   │   └── main.yml
│   │   └── files/
│   └── webserver/
│       ├── tasks/
│       │   └── main.yml
│       └── templates/
└── group_vars/
    └── all.yml
 

Key Components:
 

  • ansible.cfg: Configuration file defining paths and settings.
  • inventories/: Directory containing environment-specific inventory files.
  • playbooks/: Directory for playbooks that define automation tasks.
  • roles/: Directory for reusable roles, each with its own tasks and variables.
  • requirements.yml: File listing external roles or collections.
     

For a detailed explanation of this structure, refer to the Ansible Documentation.

Visual Workflow (text) Diagram

To illustrate the workflow, here's a diagram depicting how Ansible interacts with your infrastructure:

+——————+        +——————+        +——————+
|   Control Node   |        |    Ansible      |        |   Managed Nodes  |
|  (Your Machine)  |        |  (Ansible Core) |        |  (10 Linux Servers)|
+——————+        +——————+        +——————+
        |                               |                                 |
        | SSH                      | Execute Playbooks  | Apply Configurations
        |                               |                                 |
        +————————->+————————->+

Workflow Steps:

  1. Control Node: You initiate commands from your local machine.

  2. Ansible Core: Ansible connects via SSH to each managed node.

  3. Managed Nodes: Ansible applies configurations as defined in your playbooks and roles.

This workflow ensures consistent and automated management of your infrastructure.

Best Practices

  • Use roles: Organize playbooks into reusable roles
     (ansible-galaxy init myrole)
  • Maintain version control: Keep playbooks and inventory in Git
  • Encrypt secrets: Use ansible-vault for secure credentials
  • Test in staging: Always validate changes before pushing to production

Conclusion

Ansible is a game-changer for managing Linux infrastructure. With a few simple playbooks, you can provision, configure, and maintain your servers consistently and securely.
For a 10-node infrastructure, it offers the perfect balance of simplicity and power—letting you scale efficiently without the overhead of agent-based solutions.

Debugging Jitsi Meet Server Problems: A Practical Guide

Saturday, April 26th, 2025

Jitsi Meet is a powerful open-source video conferencing platform. But like any real-time communication system, it can run into issues—from video/audio glitches to full-blown connection failures. Debugging Jitsi Meet can be tricky due to its multi-component architecture. This guide walks you through a systematic approach to identify and resolve common server-side issues.

1. Understand the Architecture

Before diving into logs, it's important to understand Jitsi Meet's core components:
 

  • Jitsi Meet (Web UI) – The front-end interface.
  • Jicofo (Focus component) – Manages conference sessions.
  • Prosody (XMPP Server) – Handles user authentication and signaling.
  • JVB (Jitsi Videobridge) – Routes video/audio streams.
  • Nginx or Apache – Web server proxy (often with HTTPS and WebSocket forwarding).


Knowing how these interact helps pinpoint the failing layer.

2. Check Logs in the Right Places

Each component has its own logs. Check them in the following order:

Prosody Logs

Location: /var/log/prosody/prosody.log and prosody.err
​Look for: Authentication issues, connection denials, or component registration problems.
 

Jicofo Logs

Location: /var/log/jitsi/jicofo.log
Look for: Room creation errors, XMPP connection failures, conference creation attempts.
 

JVB Logs

Location: /var/log/jitsi/jvb.log
​Look for: ICE failures, STUN/TURN issues, packet loss, and bridge reachability.
 

Web Server Logs (Nginx/Apache)

Location (Nginx): /var/log/nginx/error.log and access.log
Look for: HTTP errors (404, 502), WebSocket connection problems.
 

Browser Console Logs
 

Tools: Press F12 in browser → Console/Network tabs.
Look for: WebSocket failures, CORS issues, or media permission problems.
 

3. Common Problems & Fixes

"Failed to join conference"

  • Cause: Prosody may not be running or not configured correctly.​

Fix: Restart Prosody and check domain configuration in /etc/prosody/conf.avail/

 

 

No Audio or Video
 

Usual Cause: Media not reaching the bridge or blocked by firewall

Fix:

  • Verify JVB is listening on correct ports (UDP 10000).
  • ​Check firewall/NAT settings (especially on cloud VMs).
  • Use tcpdump or ss to check traffic flow.
     

WebSocket Connection Fails

 

Usual Cause: Web server (Proxy) misconfiguration.

Fix:

Ensure Nginx is forwarding WebSocket requests to /xmpp-websocket/.
Add proper proxy settings in nginx.conf
 

Authentication Not Working


Cause: Misconfigured JWT or internal authentication.

Fix:

  • Check Prosody's config for authentication method.
  • If using JWT, verify token structure and shared secret.
     

4. Use Debugging Tools

  • Jitsi Meet in debug mode:


​Add #config.debug=true to your meeting URL.
 

  • ICE Debugging:

     

     

     

    Check about:webrtc (Firefox) or WebRTC Internals (Chrome).
    Look at ICE candidate gathering and connectivity checks.
    Test TURN/STUN:

    • Use tools like trickle-ice to validate your server's ICE configuration.

5. Networking and Firewall Checks

Make sure these ports are open:
 

  • TCP 443 – HTTPS
  • UDP 10000 – Media (JVB)
  • TCP 4443 – (Optional, fallback media)
  • TCP 5222 – XMPP (if not using BOSH/WebSocket)
     

# ss -tuln ufw status


6. Component Health Checks

Do 
# systemctl status for each main jitsi component services:

# systemctl status prosody
# systemctl status jicofo
# systemctl status jitsi-videobridge2

Check uptime, errors, or failure restarts.

7. Enable More Verbose Logs

Increase logging levels for deeper debugging:
 

  • Prosody: Edit /etc/prosody/prosody.cfg.lua → set log = { ... debug = "*" }.
  • Jicofo/JVB: Edit /etc/jitsi/jicofo/logging.properties and /etc/jitsi/videobridge/logging.properties
    → change log level to FINE or ALL.

 

8. Update & Restart Services

Sometimes updates or configs don’t apply until services are restarted:
 

# apt update && apt upgrade systemctl restart prosody jicofo jitsi-videobridge2 nginx

 

Final Closure Thoughts

Debugging Jitsi Meet requires a structured approach, start from the user-facing symptoms, trace through each service, and verify network and authentication configurations.
Debug the status of prosody, jicofo and jitsi-videobridge2, check the firewall openings are okay to the jitsi server
With some log analysis and a bit of patience, experimentation and the help of forums or Artificial Intelligence tool like ChatGPT, the Jitsi server errors will get solved.