☩ Walking in Light with Christ – Faith, Computing, Diary

On my home router configured qmail install, I have recently noticed I receive e-mails only sent via contact forms on the few websites hosted there. I'm subscribed to Debian newsletter, as well as usually receive about 10 emails and few spam mails every day, so after a few days of reduced emails on my email (receiving only e-mail notification about  blog issued comments), I logically suspected something is not properly working with the qmail installation.
My first logical guesses was the usual Qmail problems I've previously experienced through the years, earlier I blogged about most common problems / causes and solutions with qmail mail here

First thing I did as usual is to send a test e-mail from Gmail to my Mailboxes on the mail server, the test mail was not received and in Gmail a failure to delivery notice was returned, I paste the TXT content of it as taken from Gmail's webmail -> Show Original menu:
 

                                                                                                                                                                                                                                                               
Delivered-To: hipodilski@gmail.com
Received: by 10.112.27.135 with SMTP id t7csp91961lbg;
Mon, 29 Oct 2012 11:08:28 -0700 (PDT)
Received-SPF: pass (google.com: domain of  designates 10.60.171.72 as permitted sender) client-ip=10.60.171.72
Authentication-Results: mr.google.com; spf=pass (google.com: domain of  designates 10.60.171.72 as permitted sender) smtp.mail=; dkim=pass header.i=
Received: from mr.google.com ([10.60.171.72])
by 10.60.171.72 with SMTP id as8mr17839903oec.140.1351534106946 (num_hops = 1);
Mon, 29 Oct 2012 11:08:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:from:to:x-failed-recipients:subject:message-id:date
:content-type:content-transfer-encoding;
bh=GOkrGPurYWG9obiJDBWq6v3JHXdHlUebhVco7rIE73E=;
b=mQemNDUu1Wl7d/VoIseXgXFbL0SdwMIY4MZH9GOm8TuRSVaU8oz80wdWt93zJTt/DR
TEYTT6VRxUaDRAE2igBRLqjiSXdpZAJuBhoNA+bOTPwU53v+eaAUaV/7uHVHG0SYvF6u
rkpc1Rbf41VYIDLthm+e7X8vFdaxqiYFiiHcih2stsAzgI9jAQy62SLlBytYRZeDc3po
BXsb4SLm3+4kF4PuBlmhnCL+ba0hR3vQE5yC8/et0lPaxdSaJk0bHFkrjtmvg00fkyXo
Pv+0dPJHvAInzHlPGtL+XHuvjZCq5XD5ZJjsajyAlG6J64z9dmziz8YM+qqA0KpNaF8+
CVrA==
Received: by 10.60.171.72 with SMTP id as8mr17839903oec.140.1351534106944;
Mon, 29 Oct 2012 11:08:26 -0700 (PDT)
MIME-Version: 1.0
Return-Path: <>
Received: by 10.60.171.72 with SMTP id as8mr20755231oec.140; Mon, 29 Oct 2012
11:08:26 -0700 (PDT)
From: Mail Delivery Subsystem <mailer-daemon@googlemail.com>
To: hipodilski@gmail.com
X-Failed-Recipients: hipo@www.pc-freak.net
Subject: Delivery Status Notification (Failure)
Message-ID: <bcaec54edff658a23d04cd368e01@google.com>
Date: Mon, 29 Oct 2012 18:08:26 +0000
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Delivery to the following recipient failed permanently:
hipo@www.pc-freak.net
Technical details of permanent failure:=20
Google tried to deliver your message, but it was rejected by the recipient =
domain. We recommend contacting the other email provider for further inform=
ation about the cause of this error. The error that the other server return=
ed was: 451 451 qq temporary problem (#4.3.0) (state 17).
----- Original message -----
DKIM-Signature: v=3D1; a=3Drsa-sha256; c=3Drelaxed/relaxed;
d=3Dgmail.com; s=3D20120113;
h=3Dmime-version:date:message-id:subject:from:to:content-type;
bh=3DoaO9B2OZ1YJ19nwzUGkqXFVmVnakcfMdp7uW1TTA/u4=3D;
b=3DQGaaKOgrnXxSa7X0ZjdmbG2/CWDPK10czq4n0YxHLRfX8N+pzJLHWXBFWmVWUNt=
yte
rs8VrYu0BkdAE18MXS3x61cklvi/gk/eCUTzTm+L8fRu/Iiy6pZCr8S3Y6BWBN+5F1=
dm
1LkL0mTpSHqVIoMB/fZwHIzz6q5tTqYSSNHX+hapu30eI5liyK5rbf2/4T9BhJ1VM0=
v+
6NwupzAJK12jniKD8q9b4qJEhEoEqKKZrLKbTYiflHkAVMsg/C3v5zzwH+KZqsHP4W=
Us
Tl/sHUcErXWOry1OrQXLNYR2K9vgqVdBUS5aoU2Jy1FgxbL/t5+XzB3tUK2mv43ttX=
0k
wWbw=3D=3D
MIME-Version: 1.0
Received: by 10.60.171.72 with SMTP id as8mr10945566oec.140.1351262651282;
Fri, 26 Oct 2012 07:44:11 -0700 (PDT)
Received: by 10.182.41.232 with HTTP; Fri, 26 Oct 2012 07:44:11 -0700 (PDT)
Date: Fri, 26 Oct 2012 16:44:11 +0200
Message-ID: <CAPk3ZemH=3D6U0oCihDqDwb9DT8EbE8mTjZou6zzw2LUHHO4ObbA@mail.gma=
il.com>
Subject:=20
From: Georgi Georgiev <hipodilski@gmail.com>
To: hipo@www.pc-freak.net
Content-Type: multipart/alternative; boundary=3Dbcaec54edff653f66804ccf75ab=
9
http://www.nybooks.com/blogs/nyrblog/2012/sep/04/jesus-vs-mao-interview-yua=
n-zhiming/
--=20
Georgi Georgiev
Mobile: +31644943358

After evaluating on qmail logs and various qmail components and basic qmail configurations, noticed the spamassassin spamd process is not running on the host. I've figured it out from qscanner-scanner.pl  in /var/log/qscan/qmail-queue.log, there were records saying, qmail-scanner can't pass incoming scanned mail to spand and thus failing

I onwards check in proclist to make sure qmail-queue.log suggestion is right, i.e.:

qmail:~# ps axu|grep -i spamd|grep -v grep
qmail:~#

As you see from my paste qmail scanner logs were right,spamd process died due to some memory leak bug or whatever. To temporary solve the problem I first launched spamd, via its init script:

 

qmail:~# /etc/init.d/spamassassn start

....

 

However it was clear, that in future spamd might unexpectedly terminate and this might ruin whole email receive processing again.

I've encountered on few qmail servers issues like this, so I knew of 3 work-arounds.

 

• One is to use a tiny script set to run via cron job with superuser which checks every few minutes if spamd is running and if not re-launch it via the init script.

In some qmail installations, I've solved issues by using a tiny shell script – here you can download the script restart_spamd_if_crashed.sh

To use it download it to any directory lets say in /usr/local/bin and set cron job like:

qmail:~# crontab -u root -e

*/5 * * * * /usr/local/bin/restart_spamd_if_crashed.sh  >/dev/null 2>&1

• Second possible solution is to monit service to monitor spamd and restart it whether it finds its not working, I've written prior similar article explaining how to monitor and restart Apache, MySQL, Bind in case of crashes You can analogously easily configure monit to take a look at spamd.

 

• Third and in my view best spamd crashes work-around  is to configure spamd to be constantly monitored and respawned whether found missing via daemontools.

To do so download those  spamd_daemontools_supervise_script.tar.gz archive and place it in /var/qmail/supervise (or wherever qmail/supervise dir is located) and create directory for spamd daemontools monitoring logs

qmail:~# cd /var/supervise
qmail:/var/supservice:# wget -q https://www.pc-freak.net/files/spamd_daemontools_supervise_script.tar.gz
qmail:/var/supervise:# tar -zxvvf spamd_daemontools_supervise_script.tar.gz
....
qmail:/var/supervise:# chmod +t spamd/
qmail:/var/supervise:# mkdir /var/log/spamd
qmail:/var/supervise:# chown -R qmaill:root /var/log/spamd
qmail:/var/supervise:# touch /var/log/spamd/current qmail:/var/supervise:# chown qmaill:nofiles /var/log/spamd/current


It is also generally good idea to check the content of scripts spamd/run and spamd/log/run, a common problem with those scripts is spamassassin might be custom installed in /usr/local/bin/spamd and not in the usual /usr/bin/spamd – spamd location is defined in spamd/run; as well as location of daemontools logging tool multilog might be not /usr/bin/multilog but in /usr/local/bin/multilog – depending on what kind of Qmail guide was used on qmail install time.

Finally, to make daemontools schedule for monitoring spamd service link it in /service dir:

qmail:~# ln -sf /var/supervise/qmail/ /service/qmail
qmail:~# ls -al /service/spamd lrwxrwxrwx 1 root root 27 Nov 8 14:38 spamd -> /var/qmail/supervise/spamd//


To check whether daemontools, started and watch over spamd check what is in /var/log/spamd/current and check the status of daemontools:

qmail:~# tail -n 5
qmail:~# ps ax|grep -i readproc|grep -v grep 27916 ? S 0:00 readproctitle service errors: .............................
qmail:~# tail -n 5 /var/log/spamd/current |tai64nlocal 2012-11-12


Whether, you're sure daemontools, now handles spamd, startup it is also recommended, you stop the on boot time /etc/init.d/spamassassin start-up.

qmail:~# mv /etc/rc2.d/S18spamassassin /etc/rc2.d/K81spamassassin


Of course if spamd is crashing due to some newly included anti-spam rule, which prevents spamassassin from starting, suggested fixes and even daemontools will be unable to "respawn" spamd. In most cases however, implementing any of above "fixes" will assure you a peaceful sleep.