Posts Tagged ‘Fixing Qmail’

Debug and Fix QMAIL Mail Server qmail-inject: fatal: qq temporary problem (#4.3.0) and ‘reformime[1648048]: segfault at 0 ip 00007fea608bef28 sp 00007fff3c8d4bc0 error 4’ errors after update from Debian 11 to Debian 12

Monday, September 2nd, 2024

finding-qmail-install-problems-common-reasons-for-unworking-qmail-debugging-qmail

For a legacy reasons and lack of time and fact once Qmail is run on a server it works almost forever if you don't do very major upgrades and you still to the same version I have few Qmail SMTP servers that are nowadays are there for historical reasons.

After the last major version upgrade from Debian 11 to Debian 12, I've got the qmail smtpd not completely running fine and I have to follow some of my previous blog notes on how to recover in that situations as well as some common logic to resolve it.

After the upgrade I started getting every few minutes a repeating really annoying error due to reformime crashing in /var/log/messages as well as in qmail logs, the exact error was as so

Sep  1 22:15:34 pcfr_hware_local_ip kernel: [366799.585663] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:17:50 pcfr_hware_local_ip kernel: [366935.524185] reformime[1647438]: segfault at 0 ip 00007f0b9beeff28 sp 00007fff4ffd5850 error 4 in libc.so.6[7f0b9be76000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:17:50 pcfr_hware_local_ip kernel: [366935.524207] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:18:44 pcfr_hware_local_ip kernel: [366989.796532] reformime[1647577]: segfault at 0 ip 00007fe8e14bef28 sp 00007ffc000e9040 error 4 in libc.so.6[7fe8e1445000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:18:44 pcfr_hware_local_ip kernel: [366989.796554] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:20:08 pcfr_hware_local_ip kernel: [367072.889786] reformime[1647888]: segfault at 0 ip 00007efcaa6bef28 sp 00007ffdfe793560 error 4 in libc.so.6[7efcaa645000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:20:08 pcfr_hware_local_ip kernel: [367072.889809] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:21:14 pcfr_hware_local_ip kernel: [367139.010116] reformime[1648048]: segfault at 0 ip 00007fea608bef28 sp 00007fff3c8d4bc0 error 4 in libc.so.6[7fea60845000+155000] likely on CPU 1 (core 1, socket 0)
Sep  1 22:21:14 pcfr_hware_local_ip kernel: [367139.010139] Code: eb e1 48 8d 15 31 96 13 00 e9 04 00 00 00 0f 1f 40 00 41 55 49 89 d5 41 54 49 89 f4 55 53 48 89 fb 48 83 ec 08 48 85 ff 74 58 <80> 3b 00 74 46 4c 89 e6 48 89 df e8 58 61 f8 ff 48 8d 2c 03 80 7d
Sep  1 22:22:43 pcfr_hware_local_ip rsyslogd: — MARK —

To debug more concretely what exactly was happening with reformime and why it was crashing with the libc segfault error, I've used the journalctl log with this cmd:

# journalctl -p 3 -xb

сеп 01 22:10:27 pcfrxen qmail-scanner-queue.pl[2170438]: X-Qmail-Scanner-2.10st:[pcfrxen17252178278122170438] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252178278122170438/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:11:11 pcfrxen qmail-scanner-queue.pl[2170631]: X-Qmail-Scanner-2.10st:[pcfrxen17252178718122170631] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252178718122170631/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:15:32 pcfrxen qmail-scanner-queue.pl[2171777]: X-Qmail-Scanner-2.10st:[pcfrxen17252181328122171777] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252181328122171777/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:15:35 pcfrxen qmail-scanner-queue.pl[2171793]: X-Qmail-Scanner-2.10st:[pcfrxen17252181358122171793] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252181358122171793/ (Segmentation fau>
                                                            ) – that shouldn't happen!
сеп 01 22:21:21 pcfrxen qmail-scanner-queue.pl[2173427]: X-Qmail-Scanner-2.10st:[pcfrxen17252184788122173427] d_m: output spotted from /usr/bin/reformime  -x/var/spool/qscan/tmp/pcfrxen17252184788122173427/ (Segmentation fau>
                                                            ) – that shouldn't happen!


As you can see this showed that the problem is with reformime's passing on -x argument, and some temporary directory, thus to make sure the crash is not a cause of some mixed permissions, I've had to check the /var/spool/qscan permissions, and clamd permissions and few other permissions of the qmail install, and the wrong permissions (perhaps after the update of clamav after the Debian Linux migration was with /var/lib/clamav which was incorrectly owned by user clamav group clamav instead of the qscand / qscand user group, thus to resolve, I've run:

chown qscand:qscand /var/lib/clamav/ -R


Another thing I've had to correct was the /var/log/qmail permissions which was too permissive (perhaps due to some old install time hurry up stupidity done), so to correct, them:

# chmod 750 /var/log/qmail/


First thing i tried to resolve is of course to reinstall maildrop debian package that provides /usr/bin/reformime binary. 

root@pcfreak:/usr/local/bin# dpkg -l |grep -i maildrop
rc  courier-maildrop                      0.68.2-1                                                                   amd64        Courier mail server – mail delivery agent
ii  maildrop                              2.9.3-2.1                                                                  amd64        mail delivery agent with filtering abilities (set-GID=mail)

In an old post of mine on a similar error Fixing Qmail 451 qq temporary problem (#4.3.0) / @4000000050587780174c60dc status: qmail-todo stop processing asap / status: exiting, part of the solution was to reinstall maildrop, so tried this one:

root@pcfreak:/usr/local/bin# apt install –reinstall maildrop


Of course to try it out restarted qmail with the usual 

# qmailctl restart

Sadly enough this doesn't solve it, so I had to look up for other solutions and spend about 3 / 4 hours reading online just to convince myself that finding any meaningful in the classical human way, is becoming pretty much impossible task. As the content of information on the Internet has grown tremendously over the last years, it seems the quality of posts and commited data is exponentially detereorating. So the only way to solve crashes of binaries is either to stick to a debugger such as gdb or simply try rebuild the .deb binary from scratch and see whether a recompile from source might makes a difference.

After even more digging up online, found out some Gentoo forums threads, where people described thethe issue was also connected to the failing reformime libc use bug, with an applied C patch, found threads on Ubuntu and Debian users complaining about mysterious errors with libc with maildrop and even a bug report that this is some kind of libc bug, related to the precompiled version of maildrop shipped by default deb based repos.

Hence, My approach to resolve it was to recompile maildrop from source code, which even though looking a tedious task came with plenty of dependencies, I had to install plenty of developlment libraries and tools, compilers etc. as well as the following libs.

# apt install pcre2-utils
# apt install libpcre2-dev
# apt install libidn11
# apt install libidn2-dev
# apt install libcourier-unicode-dev
# apt install libcourier-unicode4

Then had to download and install from source the latest available versions of courier-authlib and its dependencies courier-unicode and once having those two recompiled with

# links https://sourceforge.net/projects/courier/files/courier/1.3.12/courier-1.3.12.tar.bz2/download
# links https://sourceforge.net/projects/courier/files/maildrop/3.1.8/maildrop-3.1.8.tar.bz2/download
# links https://sourceforge.net/projects/courier/files/authlib/0.72.3/courier-authlib-0.72.3.tar.bz2/download
# links https://sourceforge.net/projects/courier/files/courier-unicode/2.3.1/courier-unicode-2.3.1.tar.bz2/download

# tar -jxvf courier-unicode-2.3.1.tar.bz2
# tar -jxvvf courier-authlib-0.72.3.tar.bz2
# tar -jxvvf maildrop-3.1.8.tar.bz2

# cd courier-unicode-2.0/
# ./configure && make && make install

# cd ..
# cd courier-authlib-0.72.3
​# ./configure && make && make install
# cd ..

# cd maildrop-3.1.8/
​# ./configure && make && make install

I've took the time to also preinstall a bunch of perl modules deb packages which rawly are the ones found in file, i've built with the binaries perl-modules-for-qmail-needed.txt

To reinstall the binaries, run a small shell loop:

# for i in $(cat perl-modules-for-qmail-needed.txt); do apt install –reinstall $i –yes; done


Have to say also identified an issue with /var/qmail/bin/qmail-scanner-queue.pl with qmail-inject failing after testing qmail-scanner-queue installation with:

# /downloads/qmail-scanner-2.11st/contrib/test_installation.sh -doit
 

# ./test_installation.sh -doit

Sending standard test message – no viruses… 1/4
qmail-inject: fatal: qq temporary problem (#4.3.0)
Bad error. qmail-inject died


Anyone who ever administrated Qmail Mail server knows pretty well, about the Terrible error:

qmail-inject: fatal: qq temporary problem (#4.3.0)


and that it could be mostly anything, thus anyways to find out what might be the cause I've continued to Debug.

# ldd qmail-inject
        linux-vdso.so.1 (0x00007ffc43f5a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcc2099b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcc20ba1000)


To succed debug the QMAIL issues with qmail I found as very useful something from another old post of mine on debugging qmail errors – Testing Qmail installation for problems: Common reasons for unworking qmail / How to debug Qmail mail server failing to delivery or send emails, the following log debug loop:

# for i in $(ls -d /var/log/qmail/*qmail*/); do tail -n 10 $i/current|tai64nlocal; sleep 5; done


Finally the last step resolve the qmail-inject error, was to modify /var/qmail/bin/qmail-scanner-queue.pl and exchange PATH of /usr/bin/reformime default shipped debian repository to new custom built /usr/local/bin/reformime.


After retesting the qmail-scanner installation all seemed fine onwards:
 

# /downloads/qmail-scanner-2.11st/contrib/test_installation.sh -doit

Sending standard test message – no viruses… 1/4
done!

Sending eicar test virus – should be caught by perlscanner module… 2/4
done!

Sending eicar test virus with altered filename – should only be caught by commercial anti-virus modules (if you have any)… 3/4
done!

Sending bad spam message for anti-spam testing – In case you are using SpamAssassin… 4/4


If you have enabled $sa_quarantine, $sa_delete or $sa_reject the
spam-message wont't arrive to the recipients. But if you have enabled
(good idea!) 'debug' you should check
/var/spool/qscan/qmail-queue.log (or where ever you have the log).


        Done!

Finished test. Now go and check Email sent to postmaster@mail.pc-freak.net and/or the log..

Thibs Qmail install qmr_inst_check script also reported my server qmail install scripts as in good state:
 

# /downloads/scripts/qmr_inst_check
! vpopmail database do not exist!

So Hip Hip Hooray my Qmails works again ! Me fixed it again ! 
if you need help with fixing your company Professional Mail QMAIL server or Postfix, contact me via the contact form. Enjoy

 

Fixing Qmail 451 qq temporary problem (#4.3.0) / @4000000050587780174c60dc status: qmail-todo stop processing asap / status: exiting

Wednesday, September 19th, 2012

I'm in process of installing plain new Qmail mail (SMTP) server following QmailRocks updated: Thibs QmailRocks install guide for Debian 6.0 Squeeze
The install went smoothly so far and I'm already doing this installation for about 5 hours or so. I'm done with the minor install and following Thibs instructions to Implement validrcptto feature to Qmail.

Anyone who works with Qmail, should already know the lack of validrcptto tons of SPAM problems and useless Qmail load, because of QMAIL attempts to delivery to the local mail server unexisting mail boxes ….

Fixing this whole mess is implemented with the validrcptto. I myself has installed numerous times validrcptto and almost ever I ended up in some kind of mess before fixing it once and for all, this time of course (quite traditionally) the "story" repeated to piss me off for a while 🙂

After following steps literally as described on Thibs great Qmail install tutorial!, I ended up with a Qmail mail server unable to deliver properly e-mails.

To debug why mails are not properly delivered by the mail server I used telnet:


root@qmail-host:/var/qmail/control# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 This is Mail Pc-Freak.NET ESMTP
HELO localhost
250 This is Mail Pc-Freak.NET
MAIL FROM:<hipo@www.pc-freak.net>
250 ok
RCPT TO:<hipo@www.pc-freak.net>
250 ok
DATA
354 go ahead
asdfdsfafsd
.
451 qq temporary problem (#4.3.0)

Some time, back while configuring another Qmail fresh install, I ended up with exactly same delivery error – I've take time to document how I fixed this weird qq temporary issue here

As I thought one error in "normal" Software can correspondent to one cause, I red my previous post and checked closely all that was in past wrong whether I encountered the err; guess what this time it wasn't due to non-running (missing) clamav-daemon. Still though this was not the issue, it partially pointed me to the cause (problem with qmail-scanner.pl / spamd /pyzor / razor / dcc or whatever of this overall complexity ..).

First logical think was to check in logs. In /var/log/qmail/qmail-smtpd/current everything was looking good; my log looked like so:


root@qmail-host:/# tail -n 10 /var/log/qmail/qmail-smtpd/current
@40000000505877b91ab3aba4 tcpserver: end 23727 status 0
@40000000505877b91ab3af8c tcpserver: status: 0/30
@40000000505877f6273acefc tcpserver: status: 1/30
@40000000505877f6273ba9bc tcpserver: pid 23882 from 127.0.0.1
@40000000505877f6273f8dd4 tcpserver: ok 23882 mail.www.pc-freak.net:127.0.0.1:25 localhost:127.0.0.1::46769
@40000000505877fd1a3c647c qmail-smtpd[23882]: MFCHECK pass [127.0.0.1] www.pc-freak.net
@40000000505877fd1a3c935c qmail-smtpd[23882]: MAIL FROM:
@400000005058780123ba5eb4 qmail-smtpd[23882]: RCPT TO:

@4000000050587ccd179210b4 tcpserver: end 23882 status 256
@4000000050587ccd1792149c tcpserver: status: 0/30
root@qmail-host:/# tail -n 5 /var/log/qmail/qmail-smtpd/current
@40000000505877fd1a3c647c qmail-smtpd[23882]: MFCHECK pass [127.0.0.1] www.pc-freak.net

Second guess was to check in /var/log/qmail/qmail-send/current, there found errors like:


root@qmail-host:/# tail -n 10 /var/log/qmail/qmail-send/current
@4000000050584f8e0b799194 status: local 0/10 remote 0/120
@4000000050584f8e0b79957c end msg 9610091
@4000000050584fde2f5ebf44 status: qmail-todo stop processing asap
@4000000050584fde2f5ec32c status: exiting
@4000000050584fde32d2a884 status: local 0/10 remote 0/120
@4000000050584fe8136a44ac status: qmail-todo stop processing asap
@4000000050584fe8136a4894 status: exiting
@4000000050584fe8138b884c status: local 0/10 remote 0/120
@4000000050585014232903c4 status: qmail-todo stop processing asap
@4000000050585014232907ac status: exiting
@40000000505850142363e5fc status: local 0/10 remote 0/120
@40000000505851030773efa4 status: qmail-todo stop processing asap
@40000000505851030774320c status: exiting
@400000005058510307b5f214 status: local 0/10 remote 0/120

s you can see yourself, the errors are not giving any insight on what could be the reason, so I checked in /var/log/mail.log, just to find more errors there:


Sep 18 16:22:04 qmail-host qmail-scanner-queue.pl: X-Qmail-Scanner-2.10st:[pcfreak134797452279623171]

d_m: output spotted from /usr/bin/reformime -x/var/spool/qscan/tmp/qmail-host/I134797452279623171/ (sh: /usr/bin/reformime: not found#012) - that shouldn't happen!

As the error points out, the whole issues are caused by missing binary – /usr/bin/reformime. Logically I had to install reformime, so did a quick apt-cache search reformime and saw reformime is part of maildrop deb package. I thought it is installed but after checking with:


dpkg -a |grep -i maildrop

Realized it is missing and install it:


qmail-host:/# apt-get --yes install maildrop
....

That's all after a qmail restart, i.e.:


qmail-host:/# qmailctl restart
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.
* Restarting qmail-pop3d.

qq temporary error got solved and from there on qmail received and sent mails normally with validrcptto enabled. Cheers 😉