Posts Tagged ‘systemd service’

Building a self-healing WordPress Monitoring shell Script using Systemd, Apache, MariaDB simple automation for Linux server Auto Recovery

Friday, May 22nd, 2026

linux-self-healing-wordpress-script-automation-how-to-auto-recovery-broken-apache-mysql-mariadb-wordpress-server-tux-logo

Running a WordPress website in production is not only about publishing content  it is also about keeping the server healthy 24 / 7 to have a good yearly Website Uptime and if needed fit an SLA.

Even on stable Linux systems, services occasional failures are common for a reasons like:

  • Apache Crash / stop responding (due to bug or whatever)
  • MariaDB Database server acts unstable after heavy load (or server overheat)
  • WordPress platform auto updates leaves the site stuck in maintenance mode (until manually fixed)
  • Network outage (or a DHCP server malfunction, IP / MAC conflics can disrupt network).

There is plenty of other things that can go wrong, but generally usually a website infrastructure running on a Linux server that counts for proper productivity on basically a web server (apache) / mariadb / postgresql (or whatever other service) and WordPress based website has a set of common issues faced. That require a sysadmin to partake simple steps to resolve it.
Temporary outages can become kind of permanent without proper monitoring and introduction of automatic recovery procedures.

Within the age of clouds and automation reducing outages is key to success!

To reduce downtime and avoid manual intervention, there is a lot of things a sysadmin can do but a lot of traditional options are mostly neglected or uknown by the the new and knowledgable SREs (Site Reliability Engineers), most of whom seem to be a Gen-Z 🙂

Thus an alternative approach to the new ways of working is to keep up to the old standards and use lightweight self-healing Bash monitoring script for my WordPress based site / blog. I use such script myself as a do have a self-hosted infrastructure, so decided to share it with hope someone can benefit of it.

The server-health-check-restore-wp-apache-mariadb.sh script continuously checks:

  • Apache health state
  • MariaDB availability
  • HTTP response code status equals 200 ( OK )
  • WordPress maintenance mode (is not disabled

As an auto healing steps It then partakes:

  • Restarts of found failed services
  • Cleans stuck . maintenance wordpress files
  • Reboots the entire server after repeated database failures

This approach provides a simple but highly effective watchdog mechanism without needing complex monitoring software.

1. The server-health-check-restore-wp-apache-mariadb.sh
automation self-healing Script

 

$ cat /usr/local/bin/server-health-check-restore-wp-apache-mariadb.sh

#!/bin/bash

URL="https://www.pc-freak.net/blog/"
MAINT_FILE="/var/www/blog/.maintenance"
KEYWORD="Briefly unavailable for scheduled maintenance"

APACHE_SERVICE="apache2"
MARIADB_SERVICE="mariadb"

MAX_DB_RESTARTS=5
RESTART_COUNT_FILE="/var/run/mariadb_restart_count"

log() {
    echo "$(date): $1"
}

# —- Apache check —-
if ! systemctl is-active –quiet "$APACHE_SERVICE"; then
    log "Apache is not running. Restarting…"
    systemctl restart "$APACHE_SERVICE"
    sleep 5
fi

# —- MariaDB check —-
if ! systemctl is-active –quiet "$MARIADB_SERVICE"; then
    log "MariaDB is not running."

    # Read restart count
    if [ -f “$RESTART_COUNT_FILE” ]; then
        RESTART_COUNT=$(cat "$RESTART_COUNT_FILE")
    else
        RESTART_COUNT=0
    fi

    RESTART_COUNT=$((RESTART_COUNT + 1))
    echo "$RESTART_COUNT" > "$RESTART_COUNT_FILE"

    log "Restart attempt $RESTART_COUNT of $MAX_DB_RESTARTS"
    systemctl restart "$MARIADB_SERVICE"
    sleep 10

    # Re-check MariaDB
    if ! systemctl is-active –quiet "$MARIADB_SERVICE"; then
        log "MariaDB still unhealthy after restart."

        if [ “$RESTART_COUNT” -ge “$MAX_DB_RESTARTS” ]; then
            log "MariaDB failed $MAX_DB_RESTARTS times. Rebooting server!"
            rm -f "$RESTART_COUNT_FILE"
            /sbin/reboot
            exit 0
        fi

        exit 0
    fi
else
    # MariaDB healthy → reset counter
    if [ -f “$RESTART_COUNT_FILE” ]; then
        log "MariaDB is healthy again. Resetting restart counter."
        rm -f "$RESTART_COUNT_FILE"
    fi
fi

# —- HTTP sanity check —-
HTTP_CODE=$(curl -L –max-redirs 5 -s -o /dev/null -w "%{http_code}" –max-time 10 "$URL")

if [[ “$HTTP_CODE” != “200” ]]; then
    log "Site returned HTTP $HTTP_CODE. Skipping WordPress maintenance cleanup."
    exit 0
fi

# —- WordPress maintenance check —-
PAGE_CONTENT=$(curl -L –max-redirs 5 -s –max-time 10 "$URL")

if echo "$PAGE_CONTENT" | grep -qi "$KEYWORD"; then
    if [ -f “$MAINT_FILE” ]; then
        rm -f "$MAINT_FILE"
        log "WordPress maintenance file removed."
    else
        log "Maintenance message detected, but .maintenance file not found."
    fi
else
    log "Site healthy. No maintenance mode detected."
fi

1.1. Make script executable

Store the script somewhere under /usr/local/bin/ and make it executable:

# chmod +x /usr/local/bin/server-health-check-restore-wp-apache-mariadb.sh

1.2. Schedule it to run via Cron job

Run the script lets say every 5 minutes with cron and make it log to a log file:

# crontab -u root -e

*/5 * * * * /usr/sbin/server-health-check-restore-wp-apache-mariadb.sh >> /var/log/wp_healthcheck.log 2>&1

2. What This Script Actually Does

The script acts like a mini watchdog daemon.

Instead of relying on heavyweight enterprise monitoring systems, it uses:

systemctl , curl , grep combined with simple scripting  logic.

The simplicity of solution advantage is for maintenance it is easy it is transparent and highly portable as it will run on virtually ever Linux server / VPS without the need to install anything additional.

2.1 Apache Health Checks

The first section checks whether Apache is running:

# systemctl is-active –quiet apache2

If Apache is down, the script automatically restarts it:

# systemctl restart apache2

This solves temporary crashes caused by:

  • memory exhaustion
  • bad PHP workers
  • failed reloads
  • temporary kernel pressure

A short sleep delay gives Apache time to recover before additional checks continue.

2.2. MariaDB Recovery Script Logic

The database layer is more critical than Apache.

A web server can recover instantly, but repeated MariaDB crashes often indicate:

  • corrupted tables
  • exhausted RAM
  • deadlocks
  • disk problems
  • InnoDB failures

Because of that, the script implements a restart counter.

2.3. Restart Counter Logic

The counter is stored in file:

/var/run/mariadb_restart_count

Every failed startup increments the counter:

RESTART_COUNT=$((RESTART_COUNT + 1))

If MariaDB recovers successfully, the counter is deleted.

This prevents accidental reboot loops.

2.4. Automatic Server Reboot if too many auto recovery attempts

If MariaDB fails too many times:

MAX_DB_RESTARTS=5

the script escalates to a full system reboot:

/sbin/reboot

2.5. Why use reboot at continuous services failure?

Well reboot might not always work and under some cases it can make things better, but in case if you have a multiple servers running the same set of service with Apache and Mysql  with Haproxy or other Load balancer in front this set of logic is just perfect:

  • kernel resources are exhausted
  • filesystem locks remain stuck
  • memory fragmentation becomes severe
  • hardware drivers misbehave

A clean reboot can recover the machine faster than manual debugging during production outages !

This kind of script can be especially useful on:

  • Rarely mainteinaed Linux / VPS servers
  • unattended cloud instances
  • remote hosting environments

2.6. HTTP Sanity Check

After validating services, the script checks whether the website actually responds correctly.

$ curl -L –max-redirs 5

The script expects as normal a return code of:

HTTP 200

Anything else:

  • 500 errors
  • redirect loops
  • gateway failures
  • CDN problems

will stop the maintenance cleanup logic.

This prevents accidental removal of WordPress maintenance files during unrelated outages.

2.7. Automatic WordPress Maintenance Mode Recovery

One of the most annoying WordPress problems happens during failed updates.

WordPress creates under its install directory say /var/www/ a file:

.maintenance

If the update crashes, the file remains forever and the site displays:

“Briefly unavailable for scheduled maintenance.”

The script detects this message directly from the webpage content with grep:

$ grep -qi "$KEYWORD"

If detected, it removes the stale file automatically:

rm -f "$MAINT_FILE"

This instantly restores the site without requiring manual SSH intervention.

3. Why Simple script approach Works well and is good idea

This setup has several advantages, among key one is It is Extremely Lightweight.

No additional complications of use of trendy stuff like:

  • Docker stack
  • Zabbix
  • Kubernetes
  • Prometheus
  • external monitoring agents etc.

Everything is handled with simple native well known Linux tools.

3.1. It is Easy to Debug

Everything is plain Bash.

No hidden automation layers.

Every action is visible and understandable.

3.2. Production Friendly

The script tolerates:

  • temporary outages
  • service crashes
  • failed WordPress upgrades

without requiring administrator interaction.

4. Possible future script Improvements

There are many ways to extend script setup further, here is few ideas.

4.1. Add Email Notifications

Send alerts when:

  • services restart
  • reboot occurs
  • maintenance mode is detected

4.2. Add Disk Space Monitoring

Automatically detect:

  • full disks
  • inode exhaustion
  • backup growth

4.3. Add simple MySQL Query Health Checks

Instead of only checking the service state:

mysqladmin ping

could validate actual database responsiveness.

4.4. Introduce systemd Integration

Instead of cron-based execution, you might want to make the script could be made native if you use :

  • systemd timer
  • systemd service

Close up Summary

In many cases, simple Linux automation still beats overengineered solutions.

Today overcomplication of monitoring is a trend especially for big companies however for home brew small projects on little budget, sometimes the best server automation is the simplest one.
 A few lines of Bash can improve as shown above could improve uptime and reduce operational headaches.

For small-to-medium WordPress / Website deployments, this kind of self-healing “watchdog “ guarantees you reliability , simplicity , transparency , relatively quick fast recovery in case of crashes without brining a any  unnecessary infrastructure complexity, plus this setup works with zero human interaction and if combined with a simple Slack / Discord monitoring python script you can sleep better.

 

Automatically Re-plug all USB devices on system resume on Debian Linux using systemd

Thursday, March 26th, 2026

automatically-replug-all-usb-devices-on-system-resume-on-Debian-Ubuntu-Linux
Lets say you’re like me and you have an old but gold USB device like USB joystick Maxfire G-08XU (i've described how to configure Joystick / Gamepad on Debian Ubuntu easily), an USB flash drive stick or even some obscure USB keyboard model, that are not among the most compatible device on earth for linux. The result is in device plug and Sleeping the system or Hibernating it for a while (when go to bed) you end up with USB device being undetected by the system. Once you recover the Laptop / PC from being in Sleep mode / hibernate, the device becomes undetected by system, even though, even though the Linux kernel recognizes in lsusb. That weirdity continues until you do the manual hard workaround, which is to manually unplug the device cable and replug it again.
Though Linux has advanced much with this stuff over last years still this problems can occur every now and then. Thanksfully there is a quick fix to that. You can create a small script that reloads all the USB devices on PC
want the script to run automatically after your Debian laptop wakes up from suspend/hibernate. On Linux, the way to do this is using systemd sleep hooks. Here’s how to do it properly by using a small script + systemd.

1. Create a systemd sleep script

Create a new directory and file:

# mkdir -p /etc/systemd/system-sleep
# vim /etc/systemd/system-sleep/usb-replug.sh

Add this content:

#!/bin/bash
# Only run on resume (wake up)
case "$1" in
post)
# Replace '1-3' with your USB bus-port ID
echo '1-3' | tee /sys/bus/usb/drivers/usb/unbind
sleep 2
echo '1-3' | tee /sys/bus/usb/drivers/usb/bind
;;
esac
If 


If you need script logging use instead this small script:

 

#!/bin/bash

case $1/$2 in
pre/*)
# before suspend: you can put commands here if needed
;;
post/*)
# after resume: run your USB replug commands
echo "$(date) – Running USB replug script" >> /var/log/usb-replug.log
# Example command: trigger USB rescan
for bus in /sys/bus/usb/devices/*/authorized; do
echo 0 | sudo tee $bus
echo 1 | sudo tee $bus
done
;;
esac

2. Make it executable and reload systemd services

# chmod +x /etc/systemd/system-sleep/usb-replug.sh

Once you’ve created the script in /etc/systemd/system-sleep/ and made it executable, systemd will automatically call it on suspend/resume.

To make sure everything is recognized, you can:

  1. Reload systemd units (optional but recommended)

# systemctl daemon-reload
  1. Test it manually by suspending and resuming your machine

# systemctl suspend

After resuming, your script should run automatically and you should see the missing devices that you had to physically unplug and plug back to normal.
Hooray ! 🙂

3. How it works (systemd respawn)

  • systemd runs scripts in /etc/systemd/system-sleep/ on suspend and resume.

  • $1 is either pre (before sleep) or post (after wake).

  • The script unbinds and rebinds your USB device right after the system resumes.

Tip: You can also use usbreset instead of unbind/bind if you prefer, just replace the echo lines with:

# usbreset /dev/bus/usb/001/005

Alternatively you can use one time a simple one liner script that does the job like this:
 

# cat replug_usbs_linux.sh
#!/bin/bash

# one liner script to replug all USB devices like you have physically replugged all USBs useful if for example some of USB devices stuck after linux computer sleep

# for example my old maxfire g-08 usb joystick does mess up and i have to physically replug it (to work around this i simply run this script

d=$(lsusb -t | grep -m1 'Driver=' | sed -E 's|.*Port ([0-9]+):.*Bus ([0-9]+).*|\2-\1|') && echo $d | sudo tee /sys/bus/usb/drivers/usb/unbind && sleep 2 && echo $d | sudo tee /sys/bus/usb/drivers/usb/bind