How to solve crashing spamassassin spamd service preventing QMAIL mail server to properly deliver mails / Setting spamd work via daemontools

Monday, 12th November 2012

How to solve crashing spamd, script to restart spamd on failure, set spamd to run via daemontools, configure spamd to be restarted from monit service

On my home router configured qmail install, I have recently noticed I receive e-mails only sent via contact forms on the few websites hosted there. I'm subscribed to Debian newsletter, as well as usually receive about 10 emails and few spam mails every day, so after a few days of reduced emails on my email (receiving only e-mail notification about  blog issued comments), I logically suspected something is not properly working with the qmail installation.
My first logical guesses was the usual Qmail problems I've previously experienced through the years, earlier I blogged about most common problems / causes and solutions with qmail mail here

First thing I did as usual is to send a test e-mail from Gmail to my Mailboxes on the mail server, the test mail was not received and in Gmail a failure to delivery notice was returned, I paste the TXT content of it as taken from Gmail's webmail -> Show Original menu:
 

                                                                                                                                                                                                                                                               
Delivered-To: hipodilski@gmail.com
Received: by 10.112.27.135 with SMTP id t7csp91961lbg;
Mon, 29 Oct 2012 11:08:28 -0700 (PDT)
Received-SPF: pass (google.com: domain of  designates 10.60.171.72 as permitted sender) client-ip=10.60.171.72
Authentication-Results: mr.google.com; spf=pass (google.com: domain of  designates 10.60.171.72 as permitted sender) smtp.mail=; dkim=pass header.i=
Received: from mr.google.com ([10.60.171.72])
by 10.60.171.72 with SMTP id as8mr17839903oec.140.1351534106946 (num_hops = 1);
Mon, 29 Oct 2012 11:08:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:from:to:x-failed-recipients:subject:message-id:date
:content-type:content-transfer-encoding;
bh=GOkrGPurYWG9obiJDBWq6v3JHXdHlUebhVco7rIE73E=;
b=mQemNDUu1Wl7d/VoIseXgXFbL0SdwMIY4MZH9GOm8TuRSVaU8oz80wdWt93zJTt/DR
TEYTT6VRxUaDRAE2igBRLqjiSXdpZAJuBhoNA+bOTPwU53v+eaAUaV/7uHVHG0SYvF6u
rkpc1Rbf41VYIDLthm+e7X8vFdaxqiYFiiHcih2stsAzgI9jAQy62SLlBytYRZeDc3po
BXsb4SLm3+4kF4PuBlmhnCL+ba0hR3vQE5yC8/et0lPaxdSaJk0bHFkrjtmvg00fkyXo
Pv+0dPJHvAInzHlPGtL+XHuvjZCq5XD5ZJjsajyAlG6J64z9dmziz8YM+qqA0KpNaF8+
CVrA==
Received: by 10.60.171.72 with SMTP id as8mr17839903oec.140.1351534106944;
Mon, 29 Oct 2012 11:08:26 -0700 (PDT)
MIME-Version: 1.0
Return-Path: <>
Received: by 10.60.171.72 with SMTP id as8mr20755231oec.140; Mon, 29 Oct 2012
11:08:26 -0700 (PDT)
From: Mail Delivery Subsystem <mailer-daemon@googlemail.com>
To: hipodilski@gmail.com
X-Failed-Recipients: hipo@www.pc-freak.net
Subject: Delivery Status Notification (Failure)
Message-ID: <bcaec54edff658a23d04cd368e01@google.com>
Date: Mon, 29 Oct 2012 18:08:26 +0000
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Delivery to the following recipient failed permanently:
hipo@www.pc-freak.net
Technical details of permanent failure:=20
Google tried to deliver your message, but it was rejected by the recipient =
domain. We recommend contacting the other email provider for further inform=
ation about the cause of this error. The error that the other server return=
ed was: 451 451 qq temporary problem (#4.3.0) (state 17).
----- Original message -----
DKIM-Signature: v=3D1; a=3Drsa-sha256; c=3Drelaxed/relaxed;
d=3Dgmail.com; s=3D20120113;
h=3Dmime-version:date:message-id:subject:from:to:content-type;
bh=3DoaO9B2OZ1YJ19nwzUGkqXFVmVnakcfMdp7uW1TTA/u4=3D;
b=3DQGaaKOgrnXxSa7X0ZjdmbG2/CWDPK10czq4n0YxHLRfX8N+pzJLHWXBFWmVWUNt=
yte
rs8VrYu0BkdAE18MXS3x61cklvi/gk/eCUTzTm+L8fRu/Iiy6pZCr8S3Y6BWBN+5F1=
dm
1LkL0mTpSHqVIoMB/fZwHIzz6q5tTqYSSNHX+hapu30eI5liyK5rbf2/4T9BhJ1VM0=
v+
6NwupzAJK12jniKD8q9b4qJEhEoEqKKZrLKbTYiflHkAVMsg/C3v5zzwH+KZqsHP4W=
Us
Tl/sHUcErXWOry1OrQXLNYR2K9vgqVdBUS5aoU2Jy1FgxbL/t5+XzB3tUK2mv43ttX=
0k
wWbw=3D=3D
MIME-Version: 1.0
Received: by 10.60.171.72 with SMTP id as8mr10945566oec.140.1351262651282;
Fri, 26 Oct 2012 07:44:11 -0700 (PDT)
Received: by 10.182.41.232 with HTTP; Fri, 26 Oct 2012 07:44:11 -0700 (PDT)
Date: Fri, 26 Oct 2012 16:44:11 +0200
Message-ID: <CAPk3ZemH=3D6U0oCihDqDwb9DT8EbE8mTjZou6zzw2LUHHO4ObbA@mail.gma=
il.com>
Subject:=20
From: Georgi Georgiev <hipodilski@gmail.com>
To: hipo@www.pc-freak.net
Content-Type: multipart/alternative; boundary=3Dbcaec54edff653f66804ccf75ab=
9
http://www.nybooks.com/blogs/nyrblog/2012/sep/04/jesus-vs-mao-interview-yua=
n-zhiming/
--=20
Georgi Georgiev
Mobile: +31644943358

After evaluating on qmail logs and various qmail components and basic qmail configurations, noticed the spamassassin spamd process is not running on the host. I've figured it out from qscanner-scanner.pl  in /var/log/qscan/qmail-queue.log, there were records saying, qmail-scanner can't pass incoming scanned mail to spand and thus failing

I onwards check in proclist to make sure qmail-queue.log suggestion is right, i.e.:

qmail:~# ps axu|grep -i spamd|grep -v grep
qmail:~#

As you see from my paste qmail scanner logs were right,spamd process died due to some memory leak bug or whatever. To temporary solve the problem I first launched spamd, via its init script:

 

qmail:~# /etc/init.d/spamassassn start

....

 

However it was clear, that in future spamd might unexpectedly terminate and this might ruin whole email receive processing again.

I've encountered on few qmail servers issues like this, so I knew of 3 work-arounds.

 

  • One is to use a tiny script set to run via cron job with superuser which checks every few minutes if spamd is running and if not re-launch it via the init script.

In some qmail installations, I've solved issues by using a tiny shell script – here you can download the script restart_spamd_if_crashed.sh

To use it download it to any directory lets say in /usr/local/bin and set cron job like:

qmail:~# crontab -u root -e

*/5 * * * * /usr/local/bin/restart_spamd_if_crashed.sh  >/dev/null 2>&1

 

  • Third and in my view best spamd crashes work-around  is to configure spamd to be constantly monitored and respawned whether found missing via daemontools.

To do so download those  spamd_daemontools_supervise_script.tar.gz archive and place it in /var/qmail/supervise (or wherever qmail/supervise dir is located) and create directory for spamd daemontools monitoring logs

qmail:~# cd /var/supervise
qmail:/var/supservice:# wget -q https://www.pc-freak.net/files/spamd_daemontools_supervise_script.tar.gz
qmail:/var/supervise:# tar -zxvvf spamd_daemontools_supervise_script.tar.gz
....
qmail:/var/supervise:# chmod +t spamd/
qmail:/var/supervise:# mkdir /var/log/spamd
qmail:/var/supervise:# chown -R qmaill:root /var/log/spamd
qmail:/var/supervise:# touch /var/log/spamd/current qmail:/var/supervise:# chown qmaill:nofiles /var/log/spamd/current

It is also generally good idea to check the content of scripts spamd/run and spamd/log/run, a common problem with those scripts is spamassassin might be custom installed in /usr/local/bin/spamd and not in the usual /usr/bin/spamd – spamd location is defined in spamd/run; as well as location of daemontools logging tool multilog might be not /usr/bin/multilog but in /usr/local/bin/multilog – depending on what kind of Qmail guide was used on qmail install time.

Finally, to make daemontools schedule for monitoring spamd service link it in /service dir:

qmail:~# ln -sf /var/supervise/qmail/ /service/qmail
qmail:~# ls -al /service/spamd lrwxrwxrwx 1 root root 27 Nov 8 14:38 spamd -> /var/qmail/supervise/spamd//

To check whether daemontools, started and watch over spamd check what is in /var/log/spamd/current and check the status of daemontools:

qmail:~# tail -n 5
qmail:~# ps ax|grep -i readproc|grep -v grep 27916 ? S 0:00 readproctitle service errors: .............................
qmail:~# tail -n 5 /var/log/spamd/current |tai64nlocal 2012-11-12

Whether, you're sure daemontools, now handles spamd, startup it is also recommended, you stop the on boot time /etc/init.d/spamassassin start-up.

qmail:~# mv /etc/rc2.d/S18spamassassin /etc/rc2.d/K81spamassassin

Of course if spamd is crashing due to some newly included anti-spam rule, which prevents spamassassin from starting, suggested fixes and even daemontools will be unable to "respawn" spamd. In most cases however, implementing any of above "fixes" will assure you a peaceful sleep.  

Share this on:

Download PDFDownload PDF

Tags: , , , , ,

3 Responses to “How to solve crashing spamassassin spamd service preventing QMAIL mail server to properly deliver mails / Setting spamd work via daemontools”

  1. Thibs says:
    Firefox 16.0 Firefox 16.0 Windows 7 x64 Edition Windows 7 x64 Edition
    Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0

    Really good trick !
    My only remark is about how you prevent spamassasin to start at boot time.
    On a Debian system, it's cleaner to use the command update-rc.d

    View CommentView Comment
    • admin says:
      Firefox 3.6.3 Firefox 3.6.3 Windows 7 Windows 7
      Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3

      Yes you’re right but I never managed to learn update-rc.d syntax. So renaming with mv is much easier for me.
      🙂

      View CommentView Comment
  2. noone says:
    Firefox 38.0 Firefox 38.0 GNU/Linux x64 GNU/Linux x64
    Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0

    Starting from Debian 8 (Jessie) using ‘systemd’

    [Service]
    Restart=always

    should be added to a service that is not allowed to die. This will do…

    View CommentView Comment

Leave a Reply

CommentLuv badge