Posts Tagged ‘Connected’

Mail send from command line on Linux and *BSD servers – useful for scripting

Monday, September 10th, 2018

mail-send-email-from-command-line-on-linux-and-freebsd-operating-systems-logo

Historically Email sending has been very different from what most people use it in the Office, there was no heavy Email clients such as Outlook Express no MX Exchange, no e-mail client capabilities for Calendar and Meetings schedule as it is in most of the modern corporate offices that depend on products such as Office 365 (I would call it a connectedHell 365 days a year !).

There was no free webmail and pop3 / imap providers such as Mail.Yahoo.com, Gmail.com, Hotmail.com, Yandex.com, RediffMail, Mail.com the innumerous lists goes and on.
Nope back in the day emails were doing what they were originally supposed to like the post services in real life simply send and receive messages.

For those who remember that charming times, people used to be using BBS-es (which were basicly a shared set-up home system as a server) or some of the few University Internal Email student accounts or by crazy sysadmins who received their notification and warnings logs about daemon (services) messages via local DMZ-ed network email servers and it was common to read the email directly with mail (mailx) text command or custom written scripts … It was not uncommon also that mailx was used heavily to send notification messages on triggered events from logs. Oh life was simple and clear back then, and even though today the email could be used in a similar fashion by hard-core old school sysadmins and Dev Ops / simple shell scriptings tasks or report cron jobs such usage is already in the deep history.

The number of ways one could send email in text format directly from the GNU / Linux / *BSD server to another remote mail MTA node (assuming it had properly configured Relay server be it Exim or Postifix) were plenty.

In this article I will try to rewind back some of the UNIX history by pinpointing a few of the most common ways, one used to send quick emails directly from a remote server connection terminal or lets say a cheap VPS few cents server, through something like (SSH or Telnet) etc.
 

1. Using the mail command client (part of bsd-mailx on Debian).
 

In my previous article Linux: "bash mail command not found" error fix
I ended the article with a short explanation on how this is done but I will repeat myself one more time here for the sake of clearness of this article.

root@linux:~# echo "Your Sample Message Body" | mail -s "Whatever … Message Subject" remote_receiver@remote-server-email-address.com


The mail command will connect to local server TCP PORT 25 on local configured MTA and send via it. If the local MTA is misconfigured or it doesn't have a proper MX / PTR DNS records etc. or not configure as a relay SMTP remote mail will not get delivered. Sent Email should be properly delivered at remote recipient address.

How to send HTML formatted emails using mailx command on Linux console / terminal shell using remote server through SSH ?

Connect to remote SSH server (VPS), dedicated server, home Linux router etc. and run:

 

root@linux:~# mailx -a 'Content-Type: text/html'
      -s "This is advanced mailx indeed!" < email_content.html
      "first_email_to_send_to@gmail.com, mail_recipient_2@yahoo.com"

 


email_content.html should be properly formatted (at best w3c standard compliant) HTML.

Here is an example email_content.html (skeleton file)

 

    To: your_customer@gmail.com
    Subject: This is an HTML message
    From: marketing@your_company.com
    Content-Type: text/html; charset="utf8"

    <html>
    <body>
    <div style="
        background-color:
        #abcdef; width: 300px;
        height: 300px;
        ">
    </div>
Whatever text mixed with valid email HTML tags here.
    </body>
    </html>


Above command sends to two email addresses however if you have a text formatted list of recipients you can easily use that file with a bash shell script for loop and send to multiple addresses red from lets say email_addresses_list.txt .

To further advance the one liner you can also want to provide an email attachment, lets say the file email_archive.rar by using the -A email_archive.rar argument.

 

root@linux:~# mailx -a 'Content-Type: text/html'
      -s "This is advanced mailx indeed!" -A ~/email_archive.rar < email_content.html
      "first_email_to_send_to@gmail.com, mail_recipient_2@yahoo.com"

 

For those familiar with Dan Bernstein's Qmail MTA (which even though a bit obsolete is still a Security and Stability Beast across email servers) – mailx command had to be substituted with a custom qmail one in order to be capable to send via qmail MTA daemon.
 

2. Using sendmail command to send email
 

Do you remember that heavy hard to configure MTA monster sendmail ? It was and until this very day is the default Mail Transport Agent for Slackware Linux.

Here is how we were supposed to send mail with it:

 

[root@sendmail-host ~]# vim email_content_to_be_delivered.txt

 

Content of file should be something like:

Subject: This Email is sent from UNIX Terminal Email

Hi this Email was typed in a file and send via sendmail console email client
(part of the sendmail mail server)

It is really fun to go back in the pre-history of Mail Content creation 🙂

 

[root@sendmail-host ~]# sendmail -v user_name@remote-mail-domain.com  < /tmp/email_content_to_be_delivered.txt

 

-v argument provided, will make the communication between the mail server and your mail transfer agent visible.
 

3. Using ssmtp command to send mail
 

ssmtp MTA and its included shell command was used historically as it was pretty straight forward you just launch it on the command line type on one line all your email and subject and ship it (by pressing the CTRL + D key combination).

To give it a try you can do:

 

root@linux:~# apt-get install ssmtp
Reading package lists… Done
Building dependency tree       
Reading state information… Done
The following additional packages will be installed:
  libgnutls-openssl27
The following packages will be REMOVED:
  exim4-base exim4-config exim4-daemon-heavy
The following NEW packages will be installed:
  libgnutls-openssl27 ssmtp
0 upgraded, 2 newly installed, 3 to remove and 1 not upgraded.
Need to get 239 kB of archives.
After this operation, 3,697 kB disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 http://ftp.us.debian.org/debian stretch/main amd64 ssmtp amd64 2.64-8+b2 [54.2 kB]
Get:2 http://ftp.us.debian.org/debian stretch/main amd64 libgnutls-openssl27 amd64 3.5.8-5+deb9u3 [184 kB]
Fetched 239 kB in 2s (88.5 kB/s)         
Preconfiguring packages …
dpkg: exim4-daemon-heavy: dependency problems, but removing anyway as you requested:
 mailutils depends on default-mta | mail-transport-agent; however:
  Package default-mta is not installed.
  Package mail-transport-agent is not installed.
  Package exim4-daemon-heavy which provides mail-transport-agent is to be removed.

 

(Reading database … 169307 files and directories currently installed.)
Removing exim4-daemon-heavy (4.89-2+deb9u3) …
dpkg: exim4-config: dependency problems, but removing anyway as you requested:
 exim4-base depends on exim4-config (>= 4.82) | exim4-config-2; however:
  Package exim4-config is to be removed.
  Package exim4-config-2 is not installed.
  Package exim4-config which provides exim4-config-2 is to be removed.
 exim4-base depends on exim4-config (>= 4.82) | exim4-config-2; however:
  Package exim4-config is to be removed.
  Package exim4-config-2 is not installed.
  Package exim4-config which provides exim4-config-2 is to be removed.

Removing exim4-config (4.89-2+deb9u3) …
Selecting previously unselected package ssmtp.
(Reading database … 169247 files and directories currently installed.)
Preparing to unpack …/ssmtp_2.64-8+b2_amd64.deb …
Unpacking ssmtp (2.64-8+b2) …
(Reading database … 169268 files and directories currently installed.)
Removing exim4-base (4.89-2+deb9u3) …
Selecting previously unselected package libgnutls-openssl27:amd64.
(Reading database … 169195 files and directories currently installed.)
Preparing to unpack …/libgnutls-openssl27_3.5.8-5+deb9u3_amd64.deb …
Unpacking libgnutls-openssl27:amd64 (3.5.8-5+deb9u3) …
Processing triggers for libc-bin (2.24-11+deb9u3) …
Setting up libgnutls-openssl27:amd64 (3.5.8-5+deb9u3) …
Setting up ssmtp (2.64-8+b2) …
Processing triggers for man-db (2.7.6.1-2) …
Processing triggers for libc-bin (2.24-11+deb9u3) …

 

As you see from above output local default Debian Linux Exim is removed …

Lets send a simple test email …

 

hipo@linux:~# ssmtp user@remote-mail-server.com
Subject: Simply Test SSMTP Email
This Email was send just as a test using SSMTP obscure client
via SMTP server.
^d

 

What is notable about ssmtp is that even though so obsolete today it supports of STARTTLS (email communication encryption) that is done via its config file

 

/etc/ssmtp/ssmtp.conf

 

4. Send Email from terminal using Mutt client
 

Mutt was and still is one of the swiff army of most used console text email clients along with Alpine and Fetchmail to know more about it read here

Mutt supports reading / sending mail from multiple mailboxes and capable of reading IMAP and POP3 mail fetch protocols and was a serious step forward over mailx. Its syntax pretty much resembles mailx cmds.

 

root@linux:~# mutt -s "Test Email" user@example.com < /dev/null

 

Send email including attachment a 15 megabytes MySQL backup of Squirrel Webmail

 

root@linux:~# mutt  -s "This is last backup small sized database" -a /home/backups/backup_db.sql user@remote-mail-server.com < /dev/null

 


5. Using simple telnet to test and send email (verify existence of email on remote SMTP)
 

As a Mail Server SysAdmin this is one of my best ways to test whether I had a server properly configured and even sometimes for the sake of fun I used it as a hack to send my mail 🙂
telnet is and will always be a great tool for doing SMTP issues troubleshooting.
 

It is very useful to test whether a remote SMTP TCP port 25 is opened or a local / remote server firewall prevents connections to MTA.

Below is an example connect and send example using telnet to my local SMTP on pc-freak.net (QMail powered (R) 🙂 )

sending-email-using-telnet-command-howto-screenshot

 

root@pcfreak:~# telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 This is Mail Pc-Freak.NET ESMTP
HELO mail.pc-freak.net
250 This is Mail Pc-Freak.NET
MAIL FROM:<hipo@pc-freak.net>
250 ok
RCPT TO:<roots_bg@yahoo.com>
250 ok
DATA
354 go ahead
Subject: This is a test subject

 

This is just a test mail send through telnet
.
250 ok 1536440787 qp 28058
^]
telnet>

 

Note that the returned messages are native to qmail, a postfix would return a slightly different content, here is another test example to remote SMTP running sendmail or postfix.

 

root@pcfreak:~# telnet mail.servername.com 25
Trying 127.0.0.1…
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 mail.servername.com ESMTP Sendmail 8.13.8/8.13.8; Tue, 22 Oct 2013 05:05:59 -0400
HELO yahoo.com
250 mail.servername.com Hello mail.servername.com [127.0.0.1], pleased to meet you
mail from: systemexec@gmail.com
250 2.1.0 hipo@pc-freak.net… Sender ok
rcpt to: hip0d@yandex.ru
250 2.1.5 hip0d@yandex.ru… Recipient ok
data
354 Enter mail, end with "." on a line by itself
Hey
This is test email only

 

Thanks
.
250 2.0.0 r9M95xgc014513 Message accepted for delivery
quit
221 2.0.0 mail.servername.com closing connection
Connection closed by foreign host.


It is handy if you want to know whether remote MTA server has a certain Emailbox existing or not with telnet by simply trying to send to a certian email and checking the Email server returned output (note that the message returned depends on the remote MTA version and many qmails are configured to not give information on the initial SMTP handshake but returns instead a MAILER DAEMON failure error sent back to your sender address. Some MX servrers are still vulnerable to this attack yet, historically dreamhost.com. Below attack screenshot is made at the times before dreamhost.com fixed the brute force email issue.

Terminal-Verify-existing-Email-with-telnet

6. Using simple netcat TCP/IP Swiss Army Knife to test and send email in console

netcat-logo-a-swiff-army-knife-of-the-hacker-and-security-expert-logo
Other tool besides telnet of testing remote / local SMTP is netcat tool (for reading and writting data across TCP and UDP connections).

The way to do it is analogous but since netcat is not present on most Linux OSes by default you need to install it through the package manager first be it apt or yum etc.

# apt-get –yes install netcat


 

First lets create a new file test_email_content.txt using bash's echo cmd.
 

 

# echo 'EHLO hostname
MAIL FROM: hip0d@yandex.ru
RCPT TO:   solutions@pc-freak.net
DATA
From: A tester <hip0d@yandex.ru>
To:   <solutions@pc-freak.net>
Date: date
Subject: A test message from test hostname

 

Delete me, please
.
QUIT
' >>test_email_content.txt

 

# netcat -C localhost 25 < test_email_content.txt

 

220 This is Mail Pc-Freak.NET ESMTP
250-This is Mail Pc-Freak.NET
250-STARTTLS
250-SIZE 80000000
250-PIPELINING
250 8BITMIME
250 ok
250 ok
354 go ahead
451 See http://pobox.com/~djb/docs/smtplf.html.

Because of its simplicity and the fact it has a bit more capabilities in reading / writing data over network it was no surprise it was among the favorite tools not only of crackers and penetration testers but also a precious debug tool for the avarage sysadmin. netcat's advantage over telnet is you can push-pull over the remote SMTP port (25) a non-interactive input.


7. Using openssl to connect and send email via encrypted channel

 

root@linux:~# openssl s_client -connect smtp.gmail.com:465 -crlf -ign_eof

    ===
               Certificate negotiation output from openssl command goes here
        ===

        220 smtp.gmail.com ESMTP j92sm925556edd.81 – gsmtp
            EHLO localhost
        250-smtp.gmail.com at your service, [78.139.22.28]
        250-SIZE 35882577
        250-8BITMIME
        250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
        250-ENHANCEDSTATUSCODES
        250-PIPELINING
        250-CHUNKING
        250 SMTPUTF8
            AUTH PLAIN *passwordhash*
        235 2.7.0 Accepted
            MAIL FROM: <hipo@pcfreak.org>
        250 2.1.0 OK j92sm925556edd.81 – gsmtp
            rcpt to: <systemexec@gmail.com>
        250 2.1.5 OK j92sm925556edd.81 – gsmtp
            DATA
        354  Go ahead j92sm925556edd.81 – gsmtp
            Subject: This is openssl mailing

            Hello nice user
            .
        250 2.0.0 OK 1339757532 m46sm11546481eeh.9
            quit
        221 2.0.0 closing connection m46sm11546481eeh.9
        read:errno=0


8. Using CURL (URL transfer) tool to send SSL / TLS secured crypted channel emails via Gmail / Yahoo servers and MailGun Mail send API service


Using curl webpage downloading advanced tool for managing email send might be  a shocking news to many as it is idea is to just transfer data from a server.
curl is mostly used in conjunction with PHP website scripts for the reason it has a Native PHP implementation and many PHP based websites widely use it for download / upload of user data.
Interestingly besides support for HTTP and FTP it has support for POP3 and SMTP email protocols as well
If you don't have it installed on your server and you want to give it a try, install it first with apt:
 

root@linux:~# apt-get install curl

 


To learn more about curl capabilities make sure you check cURL –manual arg.
 

root@linux:~# curl –manual

 

a) Sending Emails via Gmail and other Mail Public services

Curl is capable to send emails from terminal using Gmail and Yahoo Mail services, if you want to give that a try.

gmail-settings-google-allow-less-secure-apps-sign-in-to-google-screenshot

Go to myaccount.google.com URL and login from the web interface choose Sign in And Security choose Allow less Secure Apps to be -> ON and turn on access for less secure apps in Gmail. Though I have not tested it myself so far with Yahoo! Mail, I suppose it should have a similar security settings somewhere.

Here is how to use curl to send email via Gmail.

Gmail-password-Allow-less-secure-apps-ON-screenshot-howto-to-be-able-to-send-email-with-text-commands-with-encryption-and-outlook

 

 

root@linux:~# curl –url 'smtps://smtp.gmail.com:465' –ssl-reqd \
  –mail-from 'your_email@gmail.com' –mail-rcpt 'remote_recipient@mail.com' \
  –upload-file mail.txt –user 'your_email@gmail.com:your_accout_password'


b) Sending Emails using Mailgun.com (Transactional Email Service API for developers)

To use Mailgun to script sending automated emails go to Mailgun.com and create account and generate new API key.

Then use curl in a similar way like below example:

 

curl -sv –user 'api:key-7e55d003b…f79accd31a' \
    https://api.mailgun.net/v3/sandbox21a78f824…3eb160ebc79.mailgun.org/messages \
    -F from='Excited User <developer@yourcompany.com>' \
    -F to=sandbox21a78f824…3eb160ebc79.mailgun.org \
    -F to=user_acc@gmail.com \
    -F subject='Hello' \
    -F text='Testing Mailgun service!' \
   –form-string html='<h1>EDMdesigner Blog</h1><br /><cite>This tutorial helps me understand email sending from Linux console</cite>' \
    -F attachment=@logo_picture.jpg

 

The -F option that is heavy present in above command lets curl (Emulate a form filled in button in which user has pressed the submit button).
For more info of the options check out man curl.
 

 

9. Using swaks command to send emails from

 

root@linux:~# apt-cache show swaks|grep "Description" -B 10
Package: swaks
Version: 20170101.0-1
Installed-Size: 221
Maintainer: Andreas Metzler <ametzler@debian.org>
Architecture: all
Depends: perl
Recommends: libnet-dns-perl, libnet-ssleay-perl
Suggests: perl-doc, libauthen-sasl-perl, libauthen-ntlm-perl
Description-en: SMTP command-line test tool
 swaks (Swiss Army Knife SMTP) is a command-line tool written in Perl
 for testing SMTP setups; it supports STARTTLS and SMTP AUTH (PLAIN,
 LOGIN, CRAM-MD5, SPA, and DIGEST-MD5). swaks allows one to stop the
 SMTP dialog at any stage, e.g to check RCPT TO: without actually
 sending a mail.
 .
 If you are spending too much time iterating "telnet foo.example 25"
 swaks is for you.
Description-md5: f44c6c864f0f0cb3896aa932ce2bdaa8

 

 

 

root@linux:~# apt-get instal –yes swaks

root@linux:~# swaks –to mailbox@example.com -s smtp.gmail.com:587
      -tls -au <user-account> -ap <account-password>

 


The -tls argument (in order to use gmail encrypted TLS channel on port 587)

If you want to hide the password not to provide the password from command line so (in order not to log it to user history) add the -a options.

10. Using qmail-inject on Qmail mail servers to send simple emails

Create new file with content like:
 

root@qmail:~# vim email_file_content.text
To: user@mail-example.com
Subject: Test


This is a test message.
 

root@qmail:~# cat email_file_content.text | /var/qmail/bin/qmail-inject


qmail-inject is part of ordinary qmail installation so it is very simple it even doesn't return error codes it just ships what ever given as content to remote MTA.
If the linux host where you invoke it has a properly configured qmail installation the email will get immediately delivered. The advantage of qmail-inject over the other ones is it is really lightweight and will deliver the simple message more quickly than the the prior heavy tools but again it is more a Mail Delivery Agent (MDA) for quick debugging, if MTA is not working, than for daily email writting.

It is very useful to simply test whether email send works properly without sending any email content by (I used qmail-inject to test local email delivery works like so).
 

root@linux:~# echo 'To: mailbox_acc@mail-server.com' | /var/qmail/bin/qmail-inject

 

11. Debugging why Email send with text tool is not being send properly to remote recipient

If you use some of the above described methods and email is not delivered to remote recipient email addresses check /var/log/mail.log (for a general email log and postfix MTAs – the log is present on many of the Linux distributions) and /var/log/messages or /var/log/qmal (on Qmail installations) /var/log/exim4 (on servers running Exim as MTA).

http://pc-freak.net/images/linux-email-log-debug-var-log-mail-output

 Closure

The ways to send email via Linux terminal are properly innumerous as there are plenty of scripted tools in various programming languages, I am sure in this article,  also missing a lot of pre-bundled installable distro packages. If you know other interesting ways / tools to send via terminal I would like to hear it.

Hope you enjoyed, happy mailing !

Fix qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied – Finally a working solution and what causes the error

Wednesday, July 22nd, 2015

 

qmail-fixing-clamdscan-errors-and-qq-errors-qmail-binary-migration-few-things-to-check-out
I've lost a whole day and was angry and irritated after moving (migrating) a Working Qmail installation in a binary form from a Debian Lenny 5.0 to a Debian 7.0 Wheezy Linux. The whole migration exercise was quite of a crazy move and I can tell you it didn't worth the effort as I lost much more time than even if I went on installing the server following Thibs Great Qmail Tutorial.

Yes I know many would say why do you still bother with an old and unsupported Qmail Mail server and not go with Postfix, the logic is correct however the whole issue is the previous installation has a number of domains already running VirtualMail hosting using VPopMail, so migrating all the old mailboxes from Qmail to Postfix are not worthy IMHO. Plus I honestly love qmail for being so stable even today (even without support). After all most of Qmail is secure enough already 🙂
And to be honest I don't so much care about security as in the old days as I know NSA, does already have access to any server on planet 🙂

Almost always when a Qmail migration happens I end up swearing and sweating and generally getting crazy, but anyways …

The overall migration of binaries went quite OK I just copied every binary and all the related libraries from the old Debian 5.0 to Debian 7.0 and installed the following long list of perl deb binaries using apt-get:
 

dh-make-perl
libalgorithm-c3-perl
libalgorithm-diff-perl
libalgorithm-diff-xs-perl
libalgorithm-merge-perl
libapparmor-perl
libapt-pkg-perl
libarray-unique-perl
libclass-accessor-chained-perl
libclass-accessor-perl
libclass-c3-perl
libclass-c3-xs-perl
libclass-factory-util-perl
libclass-inspector-perl
libclass-isa-perl
libclass-load-perl
libclass-singleton-perl
libconfig-file-perl
libconvert-binhex-perl
libcrypt-openssl-bignum-perl
libcrypt-openssl-random-perl
libcrypt-openssl-rsa-perl
libcrypt-passwdmd5-perl
libcrypt-ssleay-perl
libdata-optlist-perl
libdata-section-perl
libdate-manip-perl
libdatetime-format-builder-perl
libdatetime-format-iso8601-perl
libdatetime-format-strptime-perl
libdatetime-locale-perl
libdatetime-perl
libdatetime-timezone-perl
libdbd-mysql-perl
libdbi-perl
libdevel-symdump-perl
libdigest-hmac-perl
libdigest-sha-perl
libdpkg-perl
libemail-address-perl
libemail-date-format-perl
libencode-detect-perl
libencode-locale-perl
libenv-sanctify-perl
liberror-perl
libexporter-lite-perl
libfcgi-perl
libfile-chdir-perl
libfile-fcntllock-perl
libfile-find-rule-perl
libfile-listing-perl
libfile-which-perl
libfont-afm-perl
libhtml-form-perl
libhtml-format-perl
libhtml-parser-perl
libhtml-tagset-perl
libhtml-template-perl
libhtml-tree-perl
libhttp-cookies-perl
libhttp-daemon-perl
libhttp-date-perl
libhttp-message-perl
libhttp-negotiate-perl
libhttp-server-simple-perl
libio-multiplex-perl
libio-socket-inet6-perl
libio-socket-ip-perl
libio-socket-ssl-perl
libio-string-perl
libio-stringy-perl
libip-country-perl
liblist-moreutils-perl
liblocale-gettext-perl
liblwp-mediatypes-perl
liblwp-protocol-https-perl
libmail-dkim-perl
libmail-sendmail-perl
libmail-spf-perl
libmailtools-perl
libmath-round-perl
libmime-tools-perl
libmodule-depends-perl
libmodule-implementation-perl
libmodule-runtime-perl
libmro-compat-perl
libnet-cidr-lite-perl
libnet-cidr-perl
libnet-daemon-perl
libnet-dns-perl
libnet-http-perl
libnet-ident-perl
libnet-ip-perl
libnet-server-perl
libnet-snmp-perl
libnet-ssleay-perl
libnetaddr-ip-perl
libnumber-compare-perl
libossp-uuid-perl
libpackage-deprecationmanager-perl
libpackage-stash-perl
libpackage-stash-xs-perl
libparams-classify-perl
libparams-util-perl
libparams-validate-perl
libparse-debcontrol-perl
libparse-debianchangelog-perl
libparse-syslog-perl
libpcre-ocaml-dev
libpcre3:amd64
libpcre3-dev
libpcrecpp0:amd64
libperl-dev
libperl5.14
libpod-coverage-perl
libregexp-assemble-perl
librpc-xml-perl
librrds-perl
libsoap-lite-perl
libsocket-perl
libsocket6-perl
libsoftware-license-perl
libsub-exporter-perl
libsub-install-perl
libsub-name-perl
libswitch-perl
libsys-hostname-long-perl
libsys-syslog-perl
libtask-weaken-perl
libterm-readkey-perl
libtest-distribution-perl
libtest-pod-coverage-perl
libtest-pod-perl
libtext-charwidth-perl
libtext-glob-perl
libtext-iconv-perl
libtext-template-perl
libtext-wrapi18n-perl
libtie-ixhash-perl
libtimedate-perl
libtry-tiny-perl
liburi-perl
libwww-mechanize-perl
libwww-perl
libwww-robotrules-perl
libxml-namespacesupport-perl
libxml-parser-perl
libxml-sax-base-perl
libxml-sax-expat-perl
libxml-sax-perl
libxml-simple-perl
libyaml-perl
libyaml-syck-perl
perl
perl-base
perl-doc
perl-modules
spamassassin
spf-tools-perl

 

I've also installed with apt-get daemontools and daemontools-run ucspi and some others to get all the necessery binaries qmail needs the whole list of apt installed packages is here

I've also copied all the old binaries from /usr/local/lib from server1 to server2, some others from /usr/local/share and /usr/share as well as /usr/lib/courier /usr/lib/courier-authlib /usr/local/libexec /usr/local/sbin also had to link a couple of libraries such as /usr/lib/libcrypto* , link  /usr/lib/libperl.so.5.10 to /usr/lib/libperl.so.5.14 and copy /usr/lib/libltdl.so.3 and few others which i don't exactly remember.

Well anyways once I've copied everything Qmail looked fined except I had a couple of permission issues and had to clean up and fix the queue with qfixq, I've also used qmail-scanner*/contrib/test_installation.sh script to test whether qmail-scanner was running fine, e.g.:
 

./test_installation.sh -doit


As well as

 

 

qmr_inst_check


script, thanks to which I've captured and resolved few of permission problems 

Finally I've stuck upon this shitty errors (appearing) in /var/log/syslog and /var/log/messages

 

 

Jul 21 22:04:19 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:27 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:38 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:11:17 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:16:09 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:19:15 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:38:59 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:42:33 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:43:49 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:46:05 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:50:40 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:53:08 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied

 

 

 

There is plenty of things written about this:
 

qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied


But all written is too obscure and too old already somewhere between 2004 and 2010, I've been digging through Gentoo Forums, Fedora Debian and other Linux installs and everyone used to be pointing a permission issue with /var/spool/qscan/ said theoretically to be causing the error, however all looked perfectly fine with my /var/spool/qscan , e.g.:

 

 

roo@ns2:/usr/local/src# ls -ld /var/spool/qscan/
drwxr-s— 6 qscand qscand 4096 Jul 21 23:07 /var/spool/qscan/

 

 

 

 

ls -al /var/spool/qscan/
total 244904
drwxr-s— 6 qscand qscand      4096 Jul 21 23:07 .
drwxr-xr-x 4 root   root        4096 Jul 20 21:17 ..
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 archives
-rwxr-x— 1 qscand qscand      1434 Oct 12  2011 log-report.sh
-rw——- 1 qscand qscand 249731919 Jul 21 23:11 qmail-queue.log
-rw——- 1 qscand qscand    398225 Oct 28  2011 qmail-queue.log.1
-rw-rw—- 1 root   qscand        74 Jul 21 23:07 qmail-scanner-queue-version.txt
lrwxrwxrwx 1 root   qscand        16 Jul 21 23:07 qscan -> /var/spool/qscan
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 quarantine
-rw-r—– 1 root   qscand     12288 Jul 21 23:07 quarantine-events.db
-rw-r—– 1 qscand qscand     10443 Oct 12  2011 quarantine-events.txt
-rw-rw—- 1 qscand qscand    580033 Jul 21 23:07 quarantine.log
-rw-r—– 1 qscand qscand      2739 Oct 12  2011 settings_per_domain.txt
drwxr-x— 3 qscand qscand      4096 Jul 21 23:11 tmp
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 working

 

Some suggested that /var/spool/qscan should be owned by qscand:clamav instead so I tried this but it didn't help,
others recommended adding clamav groupid into qscand's but this didn't help either:
 

root@ns2:/usr/local/src# grep -i clamav /etc/group
qscand:x:163:clamav,vpopmail
clamav:x:105:

 

Besides that I was getting also this shitty error:

 

Jul 21 20:05:42 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750194279012466] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:06:51 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750201179013125] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:15:42 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns214375025407906015] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:16:06 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750256579011980] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:18:54 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750273479014847] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:21:03 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750286379028491] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 22:07:47 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750926779010097] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status 512/2

 

All the time in logs, so I thought something might be wrong with clamdscan and followed and tried some suggestions from qms1.net as described here, however none of the fixes described there worked for me …

I also tried to reinstall clamav from source using a bit of modified version of this tutorial

This didn't help either … I saw some suggestions online that the permission issues are caused by some wrong clamd.conf and freshclamd.conf configuration options – failing clamdscan, but this didn't work either. I also tried to remove clamdscan and substitute it with clamscan  as a suggested workaround but this didn't work either …

I spend about 6 hours trying to catch what is causing this issues so finally I went on and re-installed bigger part of Qmail using Thibs tutorial over the old installation.

I've also tried in mean time multiple time to rebuild qmail scanner database with:
 

setuidgid qscand /var/qmail/bin/qmail-scanner-queue.pl -g


Played around with permissions in /etc/clamav e.g.

 

 

chown -R qscand:clamav /var/log/clamav /var/lib/clamav /var/run/clamav
chown qscand:qscand /etc/clamav/freshclam.conf

 

 

 

Created:

/etc/cron.daily/qmail-scanner with following content

 

cat /etc/cron.daily/qmail-scanner
setuidgid qscand /var/qmail/bin/qmail-scanner-queue.pl -z


However in /var/log/qmail/qmail-smtpd/current and /var/log/qmail-send/current I continously got:

Qmail 451 qq temporary problem (#4.3.0) error

Interestingly during looking for a solution to the 451 qq temporary problem and:

status: qmail-todo stop processing asap / status: exiting – I've stumbled to my own blog post here 🙂

 

Finally, I tried to reinstall qmail-scanner and in mean time update it to  Version: 2.11 – st – patch – 20130319
Just to realized something was wrong with suidperl, e.g. in Debian 7.0.* Wheezy perl-suid binary is no longer in repositories so only way to have suidperl there is either to re-compile perl from source manually which is too much work and I think in most cases not worthy the effort or to use a small suid-wrapper:
 

#include <unistd.h>
#include <errno.h>
 

main( int argc, char ** argv, char ** envp )
{
              if( setgid(getegid()) ) perror( "setgid" );
              if( setuid(geteuid()) ) perror( "setuid" );
              envp = 0; /* blocks IFS attack on non-bash shells */
              system( "/usr/bin/perl", argv, envp );
              perror( argv[0] );
              return errno;
}

 


Create it into a file lets say suid-wrapperc and compile the file with GNU C Compiler:
 

$ gcc -o suid-wrapper suid-wrapper.c


Then move the suid-wrapper produced binary into /usr/bin/suidperl e.g.

 

 

$ mv suid-wrapper /usr/bin/suidperl


Last you will need to edit /var/qmail/bin/qmail-scanner-queue.pl

 

 

 

 

# vim /var/qmail/bin/qmail-scaner-queue.pl


And substitute

 

 

 

 

#!/usr/bin/perl -T


with:

 

 

#!/usr/bin/suidperl


Note!!! that qmail-scanner-queue.pl permissions should be suid and owned by qscand:qscand as follows:

 

 

 

 

ls -al /var/qmail/bin/qmail-scanner-queue.pl
-rwsr-sr-x 1 qscand qscand 159727 Jul 21 23:11 /var/qmail/bin/qmail-scanner-queue.pl


Finally to resolve the error I had to restart qmail via qmailctl start / stop script:

 

 

 

 

root@ns2:/var/qmail/bin# qmailctl restart
Restarting qmail:
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.

 

Finally to test emails are sent and receiver properly I used good old mail command part of bsd-mailx deb package

 

# mail -s "testing 12345678" testemail1234@gmail.com
asdfadf
.
Cc:

 

 

I've also tested with plain telnet to verify no errors because often the mail command doesn't return (show) errors on email sent and errors are written only in /var/log/mail.log or /var/log/qmail/* logs

 

 

# telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 servername.localdomain.tld ESMTP
mail from: <testmail@test.com>
250 ok
rcpt to: <nospam@test.com>
250 ok
data
354 go ahead
From: Test_sender <testmail@test.com>
To: Test_receiver <nospam@test.com>
Subject: Just a stupid SMTP test

 

Just a test !
.
250 ok 1279384489 qp 3711
quit
221 servername.localdomain.tld
Connection closed by foreign host.

One other thing which probably helped I did was:

# qmailctl doqueue

 

Thanks God this time, it worked out without any QQ errors 🙂 !
 

 

Resume sftp / scp cancelled (interrupted) network transfer – Continue (large) partially downloaded files on Linux / Windows

Thursday, April 23rd, 2015

resume-sftp-scp-cancelled-interrupted-file-transfer-download-upload-network-transfer-continue-large-partially-downloaded-file-howto-linux-windows
I've recentely have a task to transfer some huge Application server long time stored data (about 70GB) of data after being archived between an old Linux host server and a new one to where the new Tomcat Application (Linux) server will be installed to fit the increased sites accessibility (server hardware overload).

The two systems are into a a paranoid DMZ network and does not have access between each other via SSH / FTP / FTPs and even no Web Access on port (80 or SSL – 443) between the two hosts, so in order to move the data I had to use a third HOP station Windows (server) which have a huge SAN network attached storage of 150 TB (as a Mapped drive I:/).

On the Windows HOP station which is giving me access via Citrix Receiver to the DMZ-ed network I'm using mobaxterm so I have the basic UNIX commands such as sftp / scp already existing on the Windows system via it.
Thus to transfer the Chronos Tomcat application stored files .tar.gz archived I've sftp-ed into the Linux host and used get command to retrieve it, e.g.:

 

sftp UserName@Linux-server.net
Password:
Connected to Linux-server.
sftp> get Chronos_Application_23_04_2015.tar.gz

….


The Secured DMZ Network seemed to have a network shaper limiting my get / Secured SCP download to be at 2.5MBytes / sec, thus the overall file transfer seemed to require a lot of time about 08:30 hours to complete. As it was the middle of day about 13:00 and my work day ends at 18:00 (this meant I would be able to keep the file retrieval session for a maximum of 5 hrs) and thus file transfer would cancel when I logout of the HOP station (after 18:00). However I've already left the file transfer to continue for 2hrs and thus about 23% of file were retrieved, thus I wondered whether SCP / SFTP Protocol file downloads could be resumed. I've checked thoroughfully all the options within sftp (interactive SCP client) and the scp command manual itself however none of it doesn't have a way to do a resume option. Then I thought for a while what I can use to continue the interrupted download and I remembered good old rsync (versatile remote and local file copying tool) which I often use to create customer backup stragies has the ability to resume partially downloaded files I wondered whether this partially downloaded file resume could be done only if file transfer was only initiated through rsync itself and luckily rsync is able to continue interrupted file transfers no matter what kind of HTTP / HTTPS / SCP / FTP program was used to start file retrievalrsync is able to continue cancelled / failed transfer due to network problems or user interaction activity), that turned even pretty easy to continue failed file transfer download from where it was interrupted I had to change to directory where file is located:
 

cd /path/to/interrupted_file/


and issue command:
 

rsync -av –partial username@Linux-server.net:/path/to/file .


the –partial option is the one that does the file resume trick, -a option stands for –archive and turns on the archive mode; equals -rlptgoD (no -H,-A,-X) arguments and -v option shows a file transfer percantage status line and an avarage estimated time for transfer to complete, an easier to remember rsync resume is like so:
 

rsync -avP username@Linux-server.net:/path/to/file .
Password:
receiving incremental file list
chronos_application_23_04_2015.tar.gz
  4364009472   8%    2.41MB/s    5:37:34

To continue a failed file upload with rsync (e.g. if you used sftp put command and the upload transfer failed or have been cancalled:
 

rsync -avP chronos_application_23_04_2015.tar.gz username@Linux-server.net:/path/where_to/upload


Of course for the rsync resume to work remote Linux system had installed rsync (package), if rsync was not available on remote system this would have not work, so before using this method make sure remote Linux / Windows server has rsync installed. There is an rsync port also for Windows so to resume large Giga or Terabyte file archive downloads easily between two Windows hosts use cwRsync.

Quick way to access remotely your GNU / Linux Desktop – Access Linux Desktop from Mac and Windows 7

Tuesday, August 5th, 2014

how-to-access-linux-host-from-microsoft-windows-or-mac-client-xrdp-tightvnc-native-way-logo
For M$ Windows users its always handy to have remote access to your home PC or notebook via Remote Desktop (RDP) protocol.

However in GNU / Linux, there is no native implementation of RDP protocol. So if you're using Linux as your Desktop like me you will probably want to be able to access the Linux system remotely not only via terminal with SSH using (Putty) or MobaXTerm all in one tabbed Windows terminal program but also be able to use your Linux GNOME / KDE Graphical environment from anywhere on the Internet.

This will make you ponder – Is it possible to access Linux Desktop via proprietary RDP protocol and if not how you can achieve remote GUI access to Linux?

1. Using Linux Xorg and Xming Xserver for Windows

Most people should already know of Linux ability to start multiple Xserver sessions remotely which is the native way to access between two Linux hosts or access remotely Linux from other Linux UNIX like OS. It is also possible to use xinit / startx / xhost commands to establish remotely connection to new or running Linux (Xorg) Xserver by using them in combination with XMing – XServer for Windows running on the Windows host and Debian package (x11-xserver-utils) – providing xhost cmd, however this method is a bit complicated and not so convenient.

I used to be using this method XMing (whose mirror is here), earlier in my university years to use remotely my Debian Linux from  Windows 98 and this works perfectly fine.

2. Using RDP emulation with XRDP server

in order to be able to access your desk from any friend or computer club in the world using standard available in MS Windows Remote Desktop client (mstsc.exe).
There is also another alternative way by using Windows Desktop sharing RDP experimental server xrdp:
 

apt-cache show xrdp |grep -i descr -A 3
Description: Remote Desktop Protocol (RDP) server
 Based on research work by the rdesktop project, xrdp uses the Remote
 Desktop Protocol to present a graphical login to a remote client.
 xrdp can connect to a VNC server or another RDP server.

To make your Linux host accessible via RDP:

On Debian / Ubuntu etc. deb based Linux:

 

apt-get update
apt-get install xrdp

 
$ /etc/init.d/xrdp status
Checking status of Remote Desktop Protocol server xrdp                                             [ OK ]
Checking status of RDP Session Manager sesman

/etc/init.d/xrdp start

On  Fedora Linux:
 

yum -y install xrdp
systemctl enable xrdp.service
systemctl start xrdp.service
systemctl enable xrdp-sesman.service
systemctl start xrdp-sesman.service


It is possible to access remote Linux host using xrdp RDP server, but this will only work in older releases of mstsc.exe (Windows XP / Vista / 2003) and will not work on Windows 7 / 8, because in MS Windows 7 and onwards RDP proto version has changed and the client no longer has compatability with older mstsc releases. There is a work around for this for anyone who stubbornly want to use RDP protocol to access Linux host. If you want to connect to xrdp from Windows 7 you have to copy the old RDP client (mstsc.exe and mstscax.dll) from a WinXP install to the Windows 7 box and run it independently, from the default installed ones, anyways this method is time consuming and not really worthy …

3. Using the VNC withTightVNC server / client

 

Taking above in consideration, for me personally best way to access Linux host from Windows and Mac is to use simply the good old VNC protocol with TightVNC.

TightVNC is cross-platform free and open source remote Desktop client it uses RFB protocol to control another computer screen remotely.

To use tightvnc to access remote Debian / Ubuntu – deb based Linux screen, tightvncserver package has to be installed:

apt-cache show tightvncserver|grep -i desc -A 7
Description-en: virtual network computing server software
 VNC stands for Virtual Network Computing. It is, in essence, a remote
 display system which allows you to view a computing `desktop' environment
 not only on the machine where it is running, but from anywhere on the
 Internet and from a wide variety of machine architectures.

 .
 This package provides a server to which X clients can connect and the
 server generates a display that can be viewed with a vncviewer.

 

apt-get –yes install tightvncserver


TightVNCserver package is also available in default repositories of Fedora / CentOS / RHEL and most other RPM based distros, to install there:
 

yum -y install tightvnc-server


Once it is installed to make tightvncserver running you have to start it (preferrably with non-root user), usually this is the user with which you're using the system:

tightvncserver

You will require a password to access your desktops.

Password:
Verify:   
Would you like to enter a view-only password (y/n)? n

New 'X' desktop is rublev:4

Creating default startup script /home/hipo/.vnc/xstartup
Starting applications specified in /home/hipo/.vnc/xstartup
Log file is /home/hipo/.vnc/rublev:4.log

 

tightvncserver-running-in-gnome-terminal-debian-gnu-linux-wheezy-screenshot

To access now TightVncserver on the Linux host Download and Install TightVNC Viewer client

note that you need to download TightVNC Java Viewer JAR in ZIP archive – don't install 32 / 64 bit installer for Windows, as this will install and setup TightVNCServer on your Windows – and you probably don't want that (and – yes you will need to have Oracle Java VM installed) …
 

tightvnc-viewer-java-client-running-on-microsoft-windows-7-screenshot

Once unzipped run tightvnc-jviewer.jar and type in the IP address of remote Linux host and screen, where TightVNC is listening, as you can see in prior screenshot my screen is :4, because I run tightvnc to listen for connections in multiple X sessions. once you're connected you will be prompted for password, asker earlier when you run  tightvncserver cmd on Linux host.

If you happen to be on a Windows PC without Java installed or Java use is prohibited you can use TightVNC Viewer Portable Binary (mirrored here)

/images/tightvnc-viewer-portable-windows-7-desktop-screenshot

If you have troubles with connection, on Linux host check the exact port on which TightVncServer is running:
 

ps ax |grep -i Tightvnc

 8630 pts/8    S      0:02 Xtightvnc :4 -desktop X -auth /var/run/gdm3/auth-for-hipo-7dpscj/database -geometry 1024×768 -depth 24 -rfbwait 120000 -rfbauth /home/hipo/.vnc/passwd -rfbport 5904 -fp /usr/share/fonts/X11/misc/,/usr/share/fonts/X11/Type1/,/usr/share/fonts/X11/75dpi/,/usr/share/fonts/X11/100dpi/ -co /etc/X11/rgb

Then to check, whether the machine you're trying to connect from doesn't have firewall rules preventing the connection use (telnet) – if installed on the Windows host:
 

telnet www.pc-ferak.net 5904
Trying 192.168.56.101…
Connected to 192.168.56.101.
Escape character is '^]'.
RFB 003.008

telnet> quit
Connection closed.

remote-connection-via-tightvnc-to-linux-host-from-windows-7-using-tightvnc-java-client-screenshot
 

Fix 503 AUTH first (#5.5.1) mail receive errors in Qmail

Friday, September 2nd, 2011

I have one qmail rocks install based on Thibbs Qmalrocks tutorial

I had to do some changes, to:
/etc/service/qmail-smtpd/run and /etc/service/qmail-smtpdssl/run init scripts.

After a qmail restart suddenly qmail stopped receiving any mail messages and my sent messages was returned with an error:

Connected to xx.xxx.xx.xx but sender was rejected.
Remote host said: 503 AUTH first (#5.5.1)

After investigating the issue I finally found, that one value I’ve changed in /etc/service/qmail-smtpd/run and /etc/service/qmail-smtpdssl was causing the whole mess:

The problematic variable was:

REQUIRE_AUTH=1

To solve the issue I had to disable the value which it seems, I have enabled by mistake.

Below is a quote from http://qmail.jms1.net which explains what REQUIRE_AUTH shell variable does:

Setting REQUIRE_AUTH=1 will make the service not accept ANY mail unless the client has sent a valid AUTH command. This also prevents incoming mail from being accepted for your own domains, so do not use this setting if the service is accepting “normal” mail from the outside world.
Restarting via qmailctl restart and qmail started receiving messages normal 😉