Posts Tagged ‘spam’

« Older Entries

Remove URL from comments in WordPress Blogs and Websites to mitigate comment spam URLs in pages

Friday, February 20th, 2015

remove-comment-spam-url-field-wordpress-website-or-blog-working-how-to
If you're running a WordPress blog or Website where you have enabled comments for a page and your article or page is well indexing in Google (receives a lot of visit / reads ) daily, your site posts (comments) section is surely to quickly fill in with a lot of "Thank you" and non-sense Spam comments containing an ugly link to an external SPAM or Phishing website.

Such URL links with non-sense message is a favourite way for SPAMmers to raise their website incoming (other website) "InLinks" and through that increase current Search Engine position. 

We all know a lot of comments SPAM is generally handled well by Akismet but unfortunately still many of such spam comments fail to be identified as Spam  because spam Bots (text-generator algorithms) becomes more and more sophisticated with time, also you can never stop paid a real-persons Marketers to spam you with a smart crafted messages to increase their site's SEO ).
In all those cases Akismet WP (Anti-Spam) plugin – which btw is among the first "must have"  WP extensions to install on a new blog / website will be not enough ..

To fight with worsening SEO because of spam URLs and to keep your site's SEO better (having a lot of links pointing to reported spam sites will reduce your overall SEO Index Rate) many WordPress based bloggers, choose to not use default WordPress Comments capabilities – e.g. use exnternal commenting systems such as Disqus – (Web Community of Communities), IntenseDebate, LiveFyre, Vicomi …

However as Disqus and other 3rd party commenting systems are proprietary software (you don't have access to comments data as comments are kept on proprietary platform and shown from there), I don't personally recommend (or use) those ones, yes Disqus, Google+, Facebook and other comment external sources can have a positive impact on your SEO but that's temporary event and on the long run I think it is more advantageous to have comments with yourself.
A small note for people using Disqos and Facebook as comment platforms – (just imagine if Disqos or Facebook bankrupts in future, where your comments will be? 🙂 )

So assuming that you're a novice blogger and I succeeded convincing you to stick to standard (embedded) WordPress Comment System once your site becomes famous you will start getting severe amount of comment spam. There is plenty of articles already written on how to remove URL comment form spam in WordPress but many of the guides online are old or obsolete so in this article I will do a short evaluation on few things I tried to remove comment spam and how I finally managed to disable URL link spam to appear on site.
 

1. Hide Comment Author Link (Hide-wp-comment-author-link)

This plugin is the best one I found and I started using it since yesterday, I warmly recommend this plugin because its very easy, Download, Unzip, Activate and there you're anything typed in URL field will no longer appear in Posts (note that the URL field will stay so if you want to keep track on person's input URL you can get still see it in Wp-Admin). I'm using default WordPress WRC (Kubrick), but I guess in most newer wordpress plugins is supposed to work. If you test it on another theme please drop a comment to inform whether works for you.  Hide Comment Author Link works on current latest Wordpress 4.1 websites.

A similar plugin to hide-wp-author-link that works and you can use is  Hide-n-Disable-comment-url-field, I tested this one but for some reason I couldn't make it work.

wordpress-remove-delete-hide-n-disable-url-comment-without-deleting-the-form-url-field-screenshot-reduce-comment-spam
Whatever I type in Website field in above form, this is wiped out of comment once submitted 🙂
 

2. Disable hide Comment URL (disable-hide-comment-url)

I've seen reports disable-hide-comment-url works on WordPress 3.9.1, but it didn't worked for me, also the plugin is old and seems no longer maintaned (its last update was 3.5 years ago), if it works for you please please drop in comment your WP version, on WP 4.1 it is not working.

disable-hide-comment-url-screenshot-plugin-to-disable-comment-url-spam-in-wordpress-sites
 


3. WordPress Anti-Spam plugin

WordPress Anti-Spam plugin is a very useful addition plugin to install next to Akismet. The plugin is great if you don't want to remove commenter URL to show in the post but want to cut a lot of the annoying Spam Robots crawling ur site.
 

Anti-spam plugin blocks spam in comments automatically, invisibly for users and for admins.

  • no captcha, because spam is not users' problem
  • no moderation queues, because spam is not administrators' problem
  • no options, because it is great to forget about spam completely

Plugin is easy to use: just install it and it just works.

Anti bot works fine on WP 4.1

4. Stop Spam Comments

Stop Spam Comments is:

  • Dead simple: no setup required, just activate it and enjoy your spam-free website.
  • Lightweight: no additional database queries, it doesn't add script files or other assets in your theme. This means your website performance will not be affected and your server will thank you.
  • Invisible by design: no captchas, no tricky questions or any other user interaction required at all.
     

Stop Spam Comments works fine on WP 4.1.

I've mentioned few of the plugins which can help you solve the problem, but as there are a lot of anti-spam URL plugins available for WP its up to you to test and see what fits you best. If you know or use some other method to protect yourself from Comment Url Spam to share it please.

Import thing to note is it usually a bad idea to mix up different anti-spam plugins so don't enable both Stop Spam Comments and WordPress Anti Spam plugin.

5. Comment Form Remove Url field Manually 

This (Liberian) South) African blog describes a way how to remove URL field URL manually

In short to Remove Url Comment Field manually either edit function.php (if you have Shell SSH access) or if not do it via Wp-Admin web interface:
 

WordPress admin page –> Appearance –> Editor


Paste at the end of file following PHP code:

 

add_filter('comment_form_default_fields', 'remove_url');
 function remove_url($fields)
 {
 if(isset($fields[‘url’]))
 unset($fields[‘url’]);
 return $fields;
 }


Now to make changes effect, Restart Apache / Nginx Webserver and clean any cache if you're using a plugin like W3 Total Cache plugin etc.

Other good posts describing some manual and embedded WordPress ways to reduce / stop comment spam is here and here, however as it comes to my blog, none of the described manual (code hack) ways I found worked on WordPress v. 4.1.
Thus I personally stuck to using Hide and Disable Comment URL plugin  to get rid of comment website URL.

Tags: , , , , , , , , , , , , , , , , , , , , , ,
Posted in Curious Facts, Various, Web and CMS, Wordpress | 2 Comments »

Stop contact form spam emails in Joomla, Disable “E-mail a copy of this message to your own address.” in Joomla

Friday, April 11th, 2014

email-copy-of-this-message-to-your-own-address_Contact_email_form
If you happen to have installed Joomla based website and setup a contact form and everything worked fine until recently but suddenly your server starts mysteriously acting as a spam relay – even though email server is perfectly secured against spam.
You probably have some issue with a website email contact form hacked or some vulnerability which allowed hackers to upload spammer php script.

I have a website based on Joomla and just until recently everything was okay until I noticed there are tons of spam flying out from my Qmail mail server (which is configured to check spam with Spamassassin has Bayesian Filtering, Distributed Checksum Claring House, Python Razor and plenty of custom anti-spam rules.

It was just yesterday I ended into that situation, then after evaluating all the hosted website, I've realized Spam issues are caused by an Old Joomla Website Contact form!

There were two issues in the form

in the contact form you have the field with a tick:

1. Well Known Joomla Form Vulnerability
Currently all Joomla (including 1.5.22 and 1.6 versions) are vulnerable to a serious spam relay problem as described in the official Joomla site.

There is a quick dirty workaround fix to contact form vulnerability –  disable a Joomla Comonent in ../joomla/components/com_mailto/

To disable it I had to:

cd /var/www/joomla/components
mv com_mailto com_mailtoNOT_USED

Above solution was described under a post resolve joomla spam relay earlier by Anatoliy Dimitrov (after checking closely the website it happened he is a colleague at HP 🙂 )

2. Second issue causing high amount of spam sent over the email server
was: "E-mail a copy of this message to your own address." contact form tick, which was practically enabling any Spammer with a list to inect emails and spam via the form sending copies to any email out on the internet!

You would definitely want to disable  "E-mail a copy of this message to your own address."
I wonder why ever any Joomla developer came up with this "spam form"?? 

joomla-disable-email-copy-of-this-message-to-your-own-address

Here is the solution to this:

1. Login to Joomla Admin with admin account
2. Goto Components -> Contacts -> Contacts
3. Click on the relevant Contact form
4. Under Contact Parameters go to Email Parameters
5. Change field E-mail Copy from Show to Hide and click Apply button

And Hooray the E-mail a copy of this message to your own address will be gone from contact form! 🙂

I've seen already plenty of problematic hacked servers and scripts before with Joomla in my last job in International University College – where joomla was heavy used, but I never experienced Joomla Security issues myself 'till know, in future I'm planning to never ever use joomla. Though it is an easy CMS system to setup a website its quite complicated to learn the menus – I remember when creating the problematic website it took me days until I properly setup all the menus and find all joomla components … besides these there is no easy way to migrate between different versions major releases in Joomla like in Wordperss, I guess this Mail Security Issue absolutely convinced me to quit using that piece of crap in future.

In mean Time another very serious Apache security flaw leaked on the Internet just few days ago – The OpenSSL Hearbleed Bug. Thanksfully I'm not running SSL anywhere on my website but many systems are affecting making most of your SSL communication with your Internet banking, E-mail etc. in danger. If you're running Apache with SSL make sure you test it for this vulnerability. Here is description of Heartbleed SSL Critical Vulnerability.

heartbleed_ssl_remote_vulnerability_logo

"The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet. SSL/TLS provides communication security and privacy over the Internet for applications such as web, email, instant messaging (IM) and some virtual private networks (VPNs).

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users."

11

 

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in System Administration, Web and CMS | 2 Comments »

Finding spam sending php scripts on multiple sites servers – Tracing and stopping spammer PHP scripts

Monday, April 14th, 2014

stop_php_mail-spam-find-spammer-and-stop-php-spammer-websites
Spam has become a severe issue for administrators, not only for mail server admins but also for webshosting adms. Even the most secure spam protected mail server can get affected by spam due to fact it is configured to relay mail from other servers acting as web hosting sites.

Webhosting companies almost always suffer seriously from spam issues and often their mail servers gets blocked (enter spam blacklists), because of their irresponsible clients uploading lets say old vulnerable Joomla, WordPress without Akismet or proper spam handling plugin,a CMS which is not frequently supported / updated or custom client insecure php code.

What I mean is Shared server A is often configured to sent mail via (mail) server B. And often some of the many websites / scripts hosted on server A gets hacked and a spam form is uploaded and tons of spam start being shipped via mail server B.

Of course on mail server level it is possible to configure delay between mail sent and adopt a couple of policies to reduce spam, but the spam protection issue can't be completely solved thus admin of such server is forced to periodically keep an eye on what mail is sent from hosting server to mail server.
 


If you happen to be one of those Linux (Unix) webhosting admins who find few thousand of spammer emails into mail server logs or your eMail server queue and you can't seem to find what is causing it, cause there are multiple websites shared hosting using mainly PHP + SQL and you can't identify what php script is spamming by reviewing  Apache log / PHP files. What you can do is get use of:

PHP mail.log directive

Precious tool in tracking spam issues is a PHP Mail.log parameter, mail log paramater is available since PHP version >= 5.3.0 and above.
PHP Mail.log parameter records all calls to the PHP mail() function including exact PHP headers, line numbers and path to script initiating mail sent.

Here is how it is used:
 

1. Create empty PHP Mail.log file

touch /var/log/phpmail.log

File has to be writtable to same user with which Apache is running in case of Apache with SuPHP running file has to be writtable by all users.

On Debian, Ubunut Linux:

chown www:data:www-data /var/log/phpmail.log

On CentOS, RHEL, SuSE phpmail.log has to be owned by httpd:

chown httpd:httpd /var/log/phpmail.log

On some other distros it might be chown nobody:nobody etc. depending on the user with which Apache server is running.

 

2. Add to php.ini configuration following lines

mail.add_x_header = On
mail.log = /var/log/phpmail.log

PHP directive instructs PHP to log complete outbund Mail header sent by mail() function, containing the UID of the web server or PHP process and the name of the script that sent the email;
 

(X-PHP-Originating-Script: 33:mailer.php)


i.e. it will make php start logging to phpmail.log stuff like:
 

 

mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: info@globalremarketing.com.au — Headers: From: "Priority Mail" <status_93@pomoriemon
astery.org> X-Mailer: MailMagic2.0 Reply-To: "Priority Mail" <status_93@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="——
—-13972215105347E886BADB5"
mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: demil7167@yahoo.com — Headers: From: "One Day Shipping" <status_44@pomoriemonastery.
org> X-Mailer: CSMTPConnectionv1.3 Reply-To: "One Day Shipping" <status_44@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="—
——-13972215105347E886BD344"
mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: domainmanager@nadenranshepovser.biz — Headers: From: "Logistics Services" <customer.
id86@pomoriemonastery.com> X-Mailer: TheBat!(v3.99.27)UNREG Reply-To: "Logistics Services" <customer.id86@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: mult
ipart/alternative;boundary="———-13972215105347E886BF43E"
mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: bluesapphire89@yahoo.com — Headers: From: "Priority Mail" <status_73@pomoriemonaster
y.org> X-Mailer: FastMailer/Webmail(versionSM/1.2.6) Reply-To: "Priority Mail" <status_73@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternativ
e;boundary="———-13972215105347E886C13F2"

 

On Debian / Ubuntu Linux to enable this logging, exec:

echo 'mail.add_x_header = On' >> /etc/php5/apache2/php.ini
echo 'mail.log = /var/log/phpmail.log' >> /etc/php5/apache2/php.ini


I find it useful to symlink /etc/php5/apache2/php.ini to /etc/php.ini its much easier to remember php location plus it is a standard location for many RPM based distros.

ln -sf /etc/php5/apache2/php.ini /etc/php.ini

Or another "Debian recommended way" to enable mail.add_x_header logging on Debian is via:

echo 'mail.add_x_header = On' >> /etc/php5/conf.d/mail.ini
echo 'mail.log = /var/log/phpmail.log' >> /etc/php5/conf.d/mail.ini

On Redhats (RHEL, CentOS, SuSE) Linux issue:

echo 'mail.add_x_header = On' >> /etc/php.ini
echo 'mail.log = /var/log/phpmail.log' >> /etc/php.ini

3. Restart Apache

On Debian / Ubuntu based linuces:

/etc/init.d/apache2 restart

P.S. Normally to restart Apache without interrupting client connections graceful option can be used, i.e. instead of restarting do:

/etc/init.d/apache2 graceful

On RPM baed CentOS, Fedora etc.:

/sbin/service httpd restart

or

apachectl graceful
 

4. Reading the log

To review in real time exact PHP scripts sending tons of spam tail it:

tail -f /var/log/phpmail.log

 

mail() on [/var/www/remote-admin/wp-includes/class-phpmailer.php:489]: To: theosfp813@hotmail.com — Headers: Date: Mon, 14 Apr 2014 03:27:23 +0000 Return-Path: wordpress@remotesystemadministration.com From: WordPress Message-ID: X-Priority: 3 X-Mailer: PHPMailer (phpmailer.sourceforge.net) [version 2.0.4] MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="UTF-8"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: 2070ccrabb@kiakom.net — Headers: From: "Manager Elijah Castillo" <elijah_castillo32@pomoriemonastery.com> X-Mailer: Mozilla/5.0 (Windows; U; Windows NT 5.0; es-ES; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1 Reply-To: "Manager Elijah Castillo" <elijah_castillo32@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A76017CC"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: 20wmwebinfo@schools.bedfordshire.gov.uk — Headers: From: "Manager Justin Murphy" <justin_murphy16@pomoriemonastery.com> X-Mailer: Opera Mail/10.62 (Win32) Reply-To: "Manager Justin Murphy" <justin_murphy16@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A7603ED6"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: tynyrilak@yahoo.com — Headers: From: "Manager Elijah Castillo" <elijah_castillo83@pomoriemonastery.com> X-Mailer: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; pl; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 Reply-To: "Manager Elijah Castillo" <elijah_castillo83@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A7606308"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: 2112macdo1@armymail.mod.uk — Headers: From: "Manager Justin Murphy" <justin_murphy41@pomoriemonastery.com> X-Mailer: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; pl; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 Reply-To: "Manager Justin Murphy" <justin_murphy41@pomoriemonastery.com> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A76086D1"

 

As you can see there is a junky spam mails sent via some spammer script uploaded under name .3a1a1c.php, so to stop the dirty bastard, deleted the script:

rm -f /var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php

It is generally useful to also check (search) for all hidden .php files inside directoring storing multiple virtualhost websites, as often a weirdly named hidden .php is sure indicator of either a PHP Shell script kiddie tool or a spammer form.

Here is how to Find all Hidden Perl / PHP scripts inside /var/www:

find . -iname '.*.php'
./blog/wp-content/plugins/fckeditor-for-wordpress-plugin/ckeditor/plugins/selection/.0b1910.php
./blog/wp-content/plugins/fckeditor-for-wordpress-plugin/filemanager/browser/default/.497a0c.php
./blog/wp-content/plugins/__MACOSX/feedburner_feedsmith_plugin_2.3/._FeedBurner_FeedSmith_Plugin.php

find . -iname '.*.pl*'

….

Reviewing complete list of all hidden files is also often useful to determine shitty cracker stuff

 find . -iname ".*"

Debugging via  /var/log/phpmail.log enablement is useful but is more recommended on development and staging (QA) environments. Having it enable on productive server with high amounts of mail sent via PHP scripts or just on dedicated shared site server could cause both performance issues, hard disk could quickly get and most importantly could be a severe security hole as information from PHP scripts could be potentially exposed to external parties.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Computer Security, Linux, System Administration, Web and CMS | 3 Comments »

How to determine WordPress blogs with most spam on multiple blog hosting server

Thursday, November 27th, 2014

determine_find_blogs_with_most_spam-on-multiple-wordpress-blogs-hosting-server-stop-and-clea-large-amounts-ofrcomment-spam
If you're a hosting company that is hosts Joomla / WordPress / ModX websites (each) on separate servers and thus you end up with servers hosting multiple WordPress customer Blogs only, lets say (100+ WP blogs per host) soon your MySQL blogs databases will be full (overfilled) with spam comments. Blogs with multitude of spam comments reduces the WordPress site attractiveness, takes useless disk space, makes wp databases hard to backup and slowing drastically the SQL server.

As our duty as system administrators is to keep the servers optimized (improve performance) and prevent spam-bots to hammer your Linux servers, its is always a good idea to keep an eye on which hosted blogs attract more spammers and cause server overheads and bad hardware optimization.

WordPress blogs keeps logged comments under database_name.wp_comments  (table) thus the quickest way to find out blogs with largest comments tables is to use Linux's find command and print out only comments tables larger than set size.

Here is how:

find /var/lib/mysql/ -type f -size +1024k -name "*_comments.MYD" -exec ls -lh {} ; | awk '{ print $9 ": " $5 }'


/var/lib/mysql/funny-blog/wp_comments.MYD: 15,7M
/var/lib/mysql/wordblogger/wp_comments.MYD: 5,3M
/var/lib/mysql/loveblog/wp_comments.MYD: 50,5M
…

A comments database of 1MB means about at least 500+ comments, thus the blog loveblog's wp_comments.MYD = 50,5 Mbs contains probably about 10000! comments and should be definitely checked in a browser, if its overfilled with spam because of bad anti-spam policy or missing currently best wordpress spam catcher plugin Akismet. In cases of lack of client to protect his spam you can write quickly a script to auto mail him and ask him kindly to check / fix his blog spam.
In some cases it is useful to write a few liners bash script to automatically disable users with extraordinary blog spam comments databases (quickest way to do it is to move users blog data under quarantine directory and adding a Blog Suspended static html webpage with text like "Please contact support for more info".

1024k find arguments is 1MB, on a big hosted blogs this might be low and you might want to use (100Mb) = 102400kbytes.
You should note that *_comments.MYD in above find cmd is because though standardly wordpress sets wp_ as a prefix to its created skele table structures it is not always the case. 

In above command example find looks for spam comments in /var/lib/mysql (because this is a Debian Linux server), however on other MySQL custom installs, it might be in another dir i.e. /usr/local/mysql/data etc.

It is useful to set the wp_comments statistics output to execute at least once a day as a cronjob:

crontab -u root -e 00 24 * * * /usr/sbin/check_spammed_blogs.sh

vim /usr/check_soammed_blogs.sh

Set a script like:

#!/bin/sh
find /var/lib/mysql/ -type f -size +1024k -name "*_comments.MYD" -exec ls -lh {} ; | awk '{ print $9 ": " $5 }' | tee -a /var/log/blogs_with_most_spam_comments.log

Though above commands is to run on GNU / Linux, for Windows servers based hosting you can  install GNUWin tools and adapt above cmd using windows standard commands or PowerShell to do the same.
Finally you can might want to use some other SQL script to clear blogs with enormously large tables from spam or clear all unapproved spam comments

Tags: , , , , , , , , , ,
Posted in Computer Security, Everyday Life, Various, Web and CMS, Wordpress | 1 Comment »

Install postfix on Debian Wheezy Linux / Postfix mail server with Dovecot and MySQL user storage on Debian Wheezy 7 Linux

Monday, August 5th, 2013

postfix Debian GNU Linux logo picture install and configure postfix with dovecot on Wheezy debian 7 Linux

I have recently installed Postfix on a server following WorkAround.org ISPMail Tutorial on Debian Wheezy Linux 7. Officially as you can see on their website there is no official guide still for Debian Wheezy yet. Therefore my only option was to follow ISPMail tutorial using Postfix 2.7 (Debian Squeeze).

It was quite a struggle to adapt tutorial for Squeeze deb to Wheezy and it took me an  overall time of about of week (each day spending few hours trying to make various components of tutorial) work. But finally I managed to install it. This is how this article got born in hope that in future it will help others have a decent Postfix install on Wheezy.. 

For those unfamiliar with Workaround.org's ISPMail Postfix Tutorial it is pretty much standard step-by-step installation guide for dummies similar to QmailRocks.org or Thibs QmailRocks Updated Installation Guide.

In Other words Workaround.org  is probably the best Postfix full featured install tutorial currently online as of time of writting this post. Workaround.org is great for people who want to run full featured; 

Postfix SMTP configured to support;

 

  • Postfix to support Mail Virtual Domains (store E-mails in MySQL database)
  • Dovecot Secure IMAP / IMAPS / POP3 / POP3s server to offer Pop3 and Imap remote access
  • Support Properly Generated SSL Certificates for POP3s and IMAPs
  • Anti Spam – SMTPD restrictions, SPF,  RBL,  Greylisting
  • Install web frontend to support Web E-mail Domaim / Accounts easy administrations for users stored in MySQL db
  • Amavisd-New (to protect Mail server from Spam)
  • Postfix WebMail frontend with Roundcube or Squirrelmail

Here is the big picture as it gives good idea on how all above components correspond to each other:

how postfix dovecot amavis clamav and spamassassin work postfix the bigpicture

So here we go:
 

1. Install Postfix necessary  debian packages


a) Install Postfix / MySQL / phpmyadmin and Postfix support for MySQL mail storage deb packs

 

apt-get update

apt-get upgrade

apt-get install –yes ssh
apt-get install –yes postfix postfix-mysql
apt-get –purge remove 'exim4*'
apt-get install –yes mysql-client mysql-server dovecot-common dovecot-imapd dovecot-pop3d postfix libsasl2-2 libsasl2-modules libsasl2-modules-sql sasl2-bin libpam-mysql openssl telnet mailutils
apt-get install –yes mysql-server phpmyadmin

 

b) Install deb packages for Dovecot and Imap Support as well as Roundcube and / or Squirrelmail for Webmail support

 

squirrelmail
apt-get install –yes dovecot-pop3d dovecot-imapd dovecot-lmtpd

c) Install packages deb packages for Anti-spam greylisting (tumgreyspf)
 

 

apt-get install –yes tumgreyspf

2. Create necessary MySQL database structures


Next step is toconnect to MySQL as root via MySQL CLI or PhpMyadmin and  CREATE new user "mailuser" and new database "mailserver", then create basic structure for "mailserver" database – i.e. sql tables ("virtual_domains", virtual_users", virtual_aliases")
These user and database is used laters by Dovecot server  to connect and  fetch emails from MySQL on user request via POP3 or IMAP. I've taken all the SQL requests and from Workaround.org's site and placed them in one SQL file:
create-postfix-mysql-user-database.sql.

Below is mysql cli query to import it. If you prefer more user friendliness do it via PhpMyAdmin with a copy / paste from file or with PhpMyAdmin import …

 

 

 wget -q https://www.pc-freak.net/files/postfix/create-postfix-mysql-user-database.sql
mysql -u root -p < create-postfix-mysql-user-database.sql

Link to create-postfix-mysql-user-database.sql is here
 

3. Setting up Dovecot required users and directories

Next its necessary configure some users and create directories where Dovecot will store its configuration files. Dovecot requires to create its custom files for each existing users in MySQL database. Therefore whenever user receives or sends e-mail or is simply created Dovecot also creates a user directory structure in /var/vmail/, for exmpl.:
 

 

# ls -al /var/vmail/mail-domain.org/test/mail/
total 20
drwx—— 4 vmail vmail 4096 jul 29 09:15 .
drwx—— 3 vmail vmail 4096 jul 29 07:20 ..
drwx—— 3 vmail vmail 4096 jul 29 09:15 .imap
drwx—— 2 vmail vmail 4096 jul 29 09:15 INBOX
-rw——- 1 vmail vmail   24 jul 29 09:15 .subscriptions

The functions of Dovecot server again are:

  • Get emails from Postfix (MySQL database) and save them to disk
  • Allow mail users to fetch emails using POP3 or IMAP protocol with Outlook / Thunderbird whatever pop3 client

    groupadd -g 5000 vmail
    useradd -g vmail -u 5000 vmail -d /var/vmail -m
    chown -R vmail:vmail /var/vmail
    chmod u+w /var/vmail
    chgrp vmail /etc/dovecot/dovecot.conf
    chmod g+r /etc/dovecot/dovecot.conf
    chown root:root /etc/dovecot/dovecot-sql.conf
    chmod go= /etc/dovecot/dovecot-sql.conf

 

4. Create self-signed SSL certificate for Postfix mail and Dovecot pop3 server

 

 

openssl req -new -x509 -days 3650 -nodes -out /etc/ssl/certs/dovecot.pem -keyout /etc/ssl/private/dovecot.pem
chmod o= /etc/ssl/private/dovecot.pem
/etc/init.d/dovecot restart
openssl req -new -x509 -days 3650 -nodes -out /etc/ssl/certs/postfix.pem -keyout /etc/ssl/private/postfix.pem
chmod o= /etc/ssl/private/postfix.pem

 

5. Configuring Amavis (Anti Virus frontend) with Clamav AV

To reduce viruses it is a must nowadays to install Antivirus in Qmail I used qmail-scanner (perl script) frontend to Clamav Antivirus to check for Viruses and some messed up e-mails. In Postfix just like in Sendmail, tool that does the same is amavisd-new.  There are some configuration files to do, but as its time consuming to do changes one by one I prefer moving old /etc/amavis/ to /etc/amavis-bak/ and download and untar archive with already set  proper configs
 

 

apt-get install –yes amavisd-new
useradd clamav -g amavis
mv /etc/amavis /etc/amavis-old
cd /etc/
wget https://www.pc-freak.net/files/postfix/amavis-config-debian-wheezy-7.tar.gz
tar -zxvvf amavis-config-debian-wheezy-7.tar.gz
…
…..
/etc/init.d/amavis start
Starting amavisd: amavisd-new.

Amavisd is meant to communicate in two port numbers with Postfix. Postfix passes Input in one (10024) and Output – Scanned File Status in (10025). Thus for normal amavis operation this two ports has to be showing as listening on localhost, e.g.:
 

 

netstat -nap | grep -E '10024|10025'
tcp        0      0 127.0.0.1:10024         0.0.0.0:*               LISTEN      13957/amavisd-new (
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      9007/master     

 

6. Placing Dovecot working (properly configured) config files for Debian Wheezy

On Workaround.org there are plenty of configurations to copy paste inside files and how it is explained is a bit complicated thus played a lot mainly with /etc/postfix/master.cf and /etc/postfix/main.cf configurations until I finally had a working version of (SMTP) configured not to be an open relay and receive / sent email OK …
Here are configurations that worked for me:
 

 

mv /etc/postfix /etc/postfix-old
cd /etc/
wget -q https://www.pc-freak.net/files/postfix/postfix-configs-debian-wheezy.tar.gz
tar -zxvvf postfix-configs-debian-wheezy.tar.gz
… …..

[ ok ] Stopping Postfix Mail Transport Agent: postfix.

[ ok ] Starting Postfix Mail Transport Agent: postfix.

 

To download my good postfixs-debian-wheezy.tar.gz look here

Afterwards only setting you have to change in /etc/postfix/main.cf is:
 

 

myhostname = example-mail.org

to your Fully Qualified Domain Name (FQDN), lets say www.pc-freak.net
 

myhostname = www.pc-freak.net

I find it also useful to remove from SMTP after connect banner reporting that Postfix is running on Debian in main.cf change:
 

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)

 to

smtpd_banner = $myhostname ESMTP $mail_name

 

7. Placing Dovecot working config files in /etc/

Configuring Dovecot is not an easy task in Debian Linux Squeeze and Lenny, on Postfix ISPMail guide there is a special tutorial for each as there is none yet for Debian Wheezy. It took me long to figure it out how to translate from old config settings for Wheezy's Dovecot 2.1.7-7. Configuration files places has changed for some clarity in Dovecot 2.1.7-7. So many of the Workaround.rog's Squeeze Tutorial suggested changes in /etc/dovecot/dovecot.conf are to be made in files under /etc/dovecot/conf.d

Therefore quickest way to have working Dovecot is to move default config dir /etc/dovecot to /etc/dovecot-old and place tuned configs:
 

cd /etc/
wget -q https://www.pc-freak.net/files/postfix/dovecot-configs-debian-wheezy.tar.gz
tar -zxvvf dovecot-configs-debian-wheezy.tar.gz
…
….

chown -R vmail:dovecot /etc/dovecot
/etc/init.d/dovecot restart

[ ok ] Restarting IMAP/POP3 mail server: dovecot.
 

To download dovecot-configs-debian-wheezy.tar.gz click here

8.Install Web Interface User / Domain edit Mail Frontend – Mail Admin Tool – Matv1.1

There are 4 web mail admin interfaces suggested by ISPMail tutorial:

 

 

  • Matv.1.1 – Mail Admin Tool
  • ISPWebadmin
  • Mailadm
  • VEA

I tried with ISPWebadmin and VEA, but only one that worked for me is MATv1.1. Actually I liked a lot Mail Admin tool it is simple and does support; create new mail domains, create new users in domains and add user aliases.

Mail admin tool login screen screenshot Debian / Ubuntu GNU Linux

mail admin tool matv1 postfix web mail admin tool debian gnu linux wheezy

Here is how to install

cd /var/www;
wget -q http://mat.ssdata.dk/files/MATv1.1.tar.gz
tar -zxvvf MATv1.1.tar.gz
mv "MAT v1.1" mailadmin
cd mailadmin
cd includes
wget -q https://www.pc-freak.net/files/postfix/postfix-webadmin/config.php.txt
mv config.php.txt config.php
cd /var/www/mailadmin
wget -q https://www.pc-freak.net/files/postfix/postfix-webadmin/index-matv1.php.txt
mv index-matv1.php.txt index.php
chown -R www-data:www-data /var/www/mailadmin

Now point your browser to:
 

 

http://localhost/mailadmin/

or

http://xxx.xxx.xxx.xxx/mailadmin/

(where xxx.xxx.xxx.xxx is your local or Internet IP address) and you should see Matt mailadmin popup. I hope my little tutorial will be of use to many. Comments and problems with install steps will be much appreciated and might lead hopefully to improvements of this little Postfix Install Wheezy tutorial Enjoy 🙂
 

9. Configuring Web mail access to Mail server Mailboxes with Squirrelmail and Roundcube

a) Configuring Squirrelmail to work with Postfix

 

apt-get install –yes squirrelmail
ln -s /etc/squirrelmail/apache.conf /etc/apache2/conf.d/squirrelmail.conf
squirrelmail-configure


Squirrelmail config options default folder config none postfix debian wheezy linux

(Select option 3) – Folder Defaults

Choose Option 1 – Press 1
and set (Default Folder Prefix) to 'none'.

You can take few minutes to browse to other options too to select for example some meaningful title for your Squirrel Webmail – i.e. organization name or whatever…

That's all now to access Squirrel open in Firefox:

http://mail-domain.org/squirrelmail/

To test squirrel is configured correctly try to login with user john@example.org with pass summersun

b)Configure Roundcube + Postfix + Dovecot

Edit /etc/roundcube/apache.conf and uncomment:
 

#Alias /roundcube /var/lib/roundcube

to

Alias /roundcube /var/lib/roundcube
Then restart Apache:

 

/etc/init.d/apache2 restart
 …

 10. Testing if everything works fine together

a) Testing if mail server defined domain has properly configured DNS – PTR, MX and TXT records

First I assume here that mailserver has proper configured PTR record and it is defined properly in DNS to be MX (Mail Exchange server). You will have to contact your ISP (Internet Service Provider) and ask them to create new PTR record corresponding to hostname of mail server defined in myhostname var in /etc/postfix/main.cf
To test whether you have PTR record run:

host 83.228.93.76
www.pc-freak.net has address 83.228.93.76
www.pc-freak.net mail is handled by 0 mail.www.pc-freak.net.
root@websrv:/etc/dovecot# host 83.228.93.76
76.93.228.83.in-addr.arpa domain name pointer www.pc-freak.net.

In above example you see my domain www.pc-freak.net has proper defined PTR record. To test you have defined in domain DNS server (zone files) proper MX record issue:
 

host -t MX www.pc-freak.net
www.pc-freak.net mail is handled by 0 mail.www.pc-freak.net.

Again for sake of testing you see www.pc-freak.net has defined MX with priority of 0 (which in those case is highest), since there are no other defined MX domains and priorities.

It is good practice that domain name has also proper SPF record this is done with DNS server zone record of type TXT. 
Once again here is SPF record defined for www.pc-freak.net.
 

host -t TXT www.pc-freak.net
www.pc-freak.net descriptive text "google-site-verification=j9d4Bt5c_1ukGf4WBng0i4esOqJtbxSFVkG144dFqv4"
www.pc-freak.net descriptive text "spf1 a mx ptr ip4:83.228.93.76 -all"

You see 2nd line is the actual SPF record
www.pc-freak.net descriptive text "spf1 a mx ptr ip4:83.228.93.76 -all"

First line command returns is actually Domain DKIM key. I tried configuring DKIM keys following ISPMail tutorial unsuccesfully so by installing Postfix by my tutorial you will not have to have  DKIM keys (soft) installed or TXT records for Domain Keys defined.
 

b) Testing if mail server is able to send (deliver) mails to other MTA's correctly Next to test whether postfix is sending mails properly use:
 

mail -s "This is a simple test mail, no need to reply" systemexec@gmail.com
Heya,

Do you get this mail
?
Hope so 🙂
.
Cc:

Note that after writing the email you have to press "." and then Enter to send the mail.

To check everything is fine with sending the mail check in /var/log/mail.log, there should be something like;

Aug 2 08:29:56 websrv postfix/smtpd[16228]: connect from localhost[127.0.0.1]
Aug 2 08:29:57 websrv postfix/smtpd[16228]: 0D323662499: client=localhost[127.0.0.1]
 Aug 2 08:29:57 websrv postfix/cleanup[16224]: 0D323662499: message-id=<20130802132956.3C4A766249B@www.pc-freak.net>
 Aug 2 08:29:57 websrv postfix/qmgr[14241]: 0D323662499: from=<root@www.pc-freak.net>, size=749, nrcpt=1 (queue active)
 Aug 2 08:29:57 websrv amavis[13958]: (13958-01) Passed CLEAN {RelayedOpenRelay}, <root@www.pc-freak.net> -> <systemexec@gmail.com>, Message-ID: <20130802132956.3C4A766249B@www.pc-freak.net>, mail_id: 1oIcE-Zc9MND, Hits: -, size: 369, queued_as: 0D323662499, 828 ms
Aug 2 08:29:57 websrv postfix/smtp[16226]: 3C4A766249B: to=<systemexec@gmail.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1, delays=0.2/0.01/0.01/0.83, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 0D323662499)
Aug 2 08:29:57 websrv postfix/qmgr[14241]: 3C4A766249B: removed
Aug 2 08:29:57 websrv postfix/smtp[16235]: connect to gmail-smtp-in.l.google.com[2a00:1450:4013:c01::1b]:25: Network is unreachable
Aug 2 08:29:58 websrv postfix/smtp[16235]: 0D323662499: to=<systemexec@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.136.27]:25, delay=1.2, delays=0.08/0.01/0.65/0.49, dsn=2.0.0, status=sent (250 2.0.0 OK 1375450198 a48si4827663eep.113 – gsmtp)
Aug 2 08:29:58 websrv postfix/qmgr[14241]: 0D323662499: removed

As you see from above output gmail server returned status of  "Success" – 250 2.0.0 Ok: quequed as 0D323662499 – this means email is delivered OK and if the mail server IP from which you're sending is not listed in some Mailserver IPs Blacklist mail should arrive in a sec.

c) Testing if Dovecot POP3 and IMAP protocol are accessible by SQL kept mail accounts

Last thing to test is Dovecot (for Qmail users for sake of comparison – Dovecot is like Couirier IMAP and Courier POP3 mail server):

To test IMAP and POP3 easiest way is to use simple telnet connections. I've earlier written a small article on How to test if IMAP and POP3 mail service is working with telnet connections, so if you never done this take 2 mins to read it. By default ISPMail adds an email with username john@example.org with password summersun


telnet localhost pop3
Trying ::1…
Connected to localhost.
Escape character is '^]'.
+OK Dovecot ready.
USER john@example.org
+OK
PASS summersun
+OK Logged in.
quit
+OK Logging out.
Connection closed by foreign host.

To test IMAP protocol login

telnet localhost imap
Trying ::1…
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready.
01 login john@example.org summersun
01 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE QUOTA] Logged in
^]
telnet> quit
Connection closed.

Well this is the end my friend 🙂 You should now have a properly running Postfix + Dovecot + Virtual Domains in SQL. Please drop me a thank you comment if it worked for you. If it doesn't drop a comment so I can integrate it and improve this tutorial. Happy Hacking 😉

Tags: , , , , , , , , , , , , , , , , ,
Posted in Linux, Postfix, System Administration | 17 Comments »

Collecting all website contact e-mails on Windows – Contact Email Extractor

Thursday, June 13th, 2013

For second week, I'm partially working in E-marketing agency. Its an education institution, so they need to harvest e-mails for possible contacts to send them advertisements about University Scholarships, offered Study Streams in university, events Uni corriculum etc. Hence I did a quick research to see if there is program capable to extract all e-mails from website. I doubt there is any for Linux and as mostly all here are working on Windows I researched for domain e-mail harvester program that runs on Windows. Of course some websites, do put their contact e-mails in picture format to prevent spammers to collect e-mails, however bigger percentage on the net doesn't, so such program does a great job 🙂
Quick research in Sourceforce revealed few nice softwares. First I one tried Contact e-mail Extractor done perfectly extracting all e-mails on domain I typed, below is screenshot of Contact E-mail Extractor

Extract Collect all emails from website email extractor on Windows

Surely program will be of interest to anyone, somehow involved in E-marketing or SPAM 🙂
Probably, there are scripts to extract all emails after crawling a website on Linux too and when I have little more time its interesting to research on that too. If someone knows of something to extract e-mails from site on Linux please drop a comment.

Tags: , , , , , , , , , ,
Posted in Everyday Life, Various, Windows | No Comments »

WordPress blog MySQL data maintainance valuable plugin WP-OPTIMIZE

Tuesday, April 12th, 2011

The more my blog is growing the slower it becomes, this is a tendency I’ve noticed since a couple of weeks.

Very fortunately while reading some random articles online I’ve came across a super valuable wordpress plugin called WP-OPTIMIZE

I think it’s best if I present instead of taking the time to explain what the WP-optimize does for a wordpress powered blog:

WP-Optimize is a WordPress 2.9++ database cleanup and optimization tool. It doesn’t require PhpMyAdmin to optimize your database tables. It allows you to remove post revisions, comments in the spam queue, un-approved comments within few clicks.

Additionally you can rename any username to another username too.

For example: If you want to rename default ‘admin’ username to ‘someothername’; just put ‘admin’ (without quotes) to the Old username field and the new username to the New username field, then click “PROCESS”)

Now in short to rephrase above text, during MySQL database requests a lot of database starts needing the so called MySQL optimization , this operation becomes necessery especially for databases which are very actively used and is related to the basic design of the mysql server.

Very often many tables starts having a lot of garbage (repetitive) data and therefore read and writes from the SQL server gets slower and slower day by day.

Thus the MySQL server has it’s famous OPTIMIZE TABLE command which does wipe out all the garbage unnecessery data stored in a tables/s and hence seriously impacts the later read/write table operations.

Now to go back to wordpress the same optimization issues, very often are a cause for a performance bottleneck and some smart guy has came with the great idea to create a specific wordpress plugin to fix such an optimize table issues

The WP-Optimize plugin has 5 major options which somehow could have a positive impact on blog’s performance, these are:
 

  • Remove all Post revisions
  • Remove all auto draft posts
  • Clean marked Spam comments
  • lean Unapproved comments

  • Optimize database tables

Apart from the nice performance boost applicaiton the wp-optimize plugin has one super valuable function, it could change the default wordpress blog administrator user admin to some other desired username, for example rename the default blog administrator username “admin” user to “john”.

From a security perspective this is another must have feature in every wordpress blog as it can kill many of the possible brute force attacks with some web brute force softwares like Hydra

Installing and using wp-optimize is a real piece of cake, here is a very quick few command lines to install it:

host:~# cd /var/www/blog/wp-content/plugins/
host:/var/www/blog/wp-content/plugins:# wget https://www.pc-freak.net/files/wp-optimize.0.9.2.zip
host:/var/www/blog/wp-content/plugins:# unzip wp-optimize.0.9.2.zip
...

To launch WP-OPTIMIZE and fix many speed bottlenecks caused by unoptimized tables, or just hanging around in database old spam comments, go and login with your adminsitrator to wordpress.

In the left pane menu starting up with Dashboard somewhere near the end of the vertical menu you will notice a button reading WP-Optimize .
Click over the Wp-Optimize button and a screen like the one in below’s screenshot will appear in your browser screen:

wp optimize plugin database optimization options screen

I have personally decided to use just 2 of all the 5 available primary plugin options, I decided to select only:
 

  • Clean marked Spam comments
  • Optimize database tables

Scrolling below the PROCEED button I could also notice a number of tables which were missing optimization and hence required an optimize table to be executed on each one of them.
Further on I pressed the PROCESS button and after a couple of minutes (2, 3 minutes) of waiting the wp-optimize plugin was done it’s selected tasks:

In the screenshot below, you see all my blog tables which were optimized by the plugin:
WP-Optimize optimized blog tables screenshot

It’s interesting to say few words about the Remove All Posts revisions plugin option, as many might be wondering what this plugin option really means.

When writting posts, wordpress has an option to restore to a certain point of the write out and makes a sort of different versions in time of each of it’s written posts.

Therefore later restoration if something gots deleted by mistake is facilitated, this is actually the all meaning of Remove All Posts revisions

With time and the increase wp posts the amount of Posts Revisions is growing on and on and just taking space and possibly making the access to the database entries slower, thus it might be a good idea to also remove them as, it’s absolutely rare to use the wp post restoration function.
However I’ve personally decided to not wipe out the old posts revisions as someday i might need it and I’m a bit scared that it might cause me some database issues.

I’ll be glad if somebody has tried the Posts Revisions wp-Optimize funciton and is happy with the results.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in MySQL, Web and CMS, Wordpress | No Comments »

How to disable spammer domain in QMAIL mail server with badmailto variable

Thursday, July 12th, 2012

I've recently noticed one of the qmail SMTP servers I adminster had plenty of logged spammer emails originating from yahoo.com.tw destined to reache some random looking like emails (probably unexisting) again to *@yahoo.com.tw

The spam that is tried by the spammer is probably a bounce spam, since it seems there is no web-form or anything wrong with the qmail server that might be causing the spam troubles.
As a result some of the emails from the well configured qmail (holding SPF checks), having a correct existing MX, PTR record and even having configured Domain Keys (DKIM) started being marked, whether emails are sent to *@yahoo.com legit emails.

To deal with the shits, since we don't have any Taiwanese (tw) clients, I dediced to completely prohibit any emails destined to be sent via the mail server to *@yahoo.com.tw. This is done via /var/qmail/control/badmailto qmail control variable;

Here is content of /var/qmail/control/badmailto after banning outgoing emails to yahoo.com.tw;;;

qmail:~# cat /var/qmail/control/badmailto
[!%#:\*\^]
[\(\)]
[\{\}]
@.*@
*@yahoo.com.tw

The first 4 lines are default rules, which are solving a lot of badmailto common sent emails. Thanks God after a qmail restart:

qmail:~# qmailct restart
....

Checking in /var/log/qmail-sent/current, there are no more outgoing *@yahoo.com.tw destined emails. Problem solved …

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Qmail, System Administration | 1 Comment »

How to reduce spam in PHPBB based internet forum on Debian GNU / Linux

Monday, March 26th, 2012

phpbb reduce spam bot registrations on Debian Linux tiny script

I had to install two PHPBB based internet forums, some long time ago. Since long time passed and I haven't checked what's happening with them I just noticed. They start filling up spam threads. The phpbb installations are done using the standard shipped deb packages in Debian Linux Lenny.

After checking online, I found one smart solution to . The idea is very simple most spam bots are written in a way that they don't have a properly set timezone. Therefore the quickest way to get rid of spam bots which try to auto register and put spam content inside the a forum category or post is to add a simple if condition in php to check the browser set timezone:

The file to add the php if condition is ucp_register.phpThe phpbb package install places default phpbb path on Debian is /usr/share/phpbb3/ and hence the file I had to modify is located in:

/usr/share/phpbb3/www/includes/ucp/ucp_register.php

To make the TZ check one needs to modify ../www/includes/ucp/ucp_register.php and look for php array definition:

$data = array(
'username' => utf8_normalize_nfc(request_var('username', '', true)),
'new_password' => request_var('new_password', '', true),
'password_confirm' => request_var('password_confirm', '', true),
'email' => strtolower(request_var('email', '')),
'email_confirm' => strtolower(request_var('email_confirm', '')),
'confirm_code' => request_var('confirm_code', ''),
'lang' => basename(request_var('lang', $user->lang_name)),
'tz' => request_var('tz', (float) $timezone),
);

Right after this chunk of code add the if condition code which is like so:

if ($data['tz'] == '-12.00')
{
die('Die, bot! Die.');
}

From now onwards, any attempt for new user registration with an incorrect timezone of -12.00 will be immediately stopped while the forum spammer bot will be offered an empty page 🙂

Another good practice is to disable Birthday Listing from phpbb Admin Control panel (ACP). Go to menus:

ACP -> General -> Board Settings -> Enable Birthday listing: (No)

Enable birthday listing phpbb forum screenshot

I like disabling birthday listing, as when it is enabled and you have some spammer registrations, which even though didn't succeeded to contaminate your forum content has specified a birthday and therefore there profiles gets popping up each different day on the main page of the forum.
This will not eradicate all spammer bots, but at least will significantly decrease spammer bot registrations.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in System Administration, Web and CMS | No Comments »

How to resolve (fix) WordPress wp-cron.php errors like “POST /wp-cron.php?doing_wp_cron HTTP/1.0″ 404” / What is wp-cron.php and what it does

Monday, March 12th, 2012

fix wordpress wp-cron.php 404 HTTP error, what is wp-cron.php schedule logo

One of the WordPress websites hosted on our dedicated server produces all the time a wp-cron.php 404 error messages like:

xxx.xxx.xxx.xxx - - [15/Apr/2010:06:32:12 -0600] "POST /wp-cron.php?doing_wp_cron HTTP/1.0

I did not know until recently, whatwp-cron.php does, so I checked in google and red a bit. Many of the places, I've red are aa bit unclear and doesn't give good exlanation on what exactly wp-cron.php does. I wrote this post in hope it will shed some more light on wp-config.php and how this major 404 issue is solved..
So

what is wp-cron.php doing?

 

  • wp-cron.php is acting like a cron scheduler for WordPress.
  • wp-cron.php is a wp file that controls routine actions for particular WordPress install.
  • Updates the data in SQL database on every, request, every day or every hour etc. – (depending on how it's set up.).
  • wp-cron.php executes automatically by default after EVERY PAGE LOAD!
  • Checks all pending comments for spam with Akismet (if akismet or anti-spam plugin alike is installed)
  • Sends all scheduled emails (e.g. sent a commentor email when someone comments on his comment functionality, sent newsletter subscribed persons emails etc.)
  • Post online scheduled articles for a day and time of particular day

Suppose you're writting a new post and you want to take advantage of WordPress functionality to schedule a post to appear Online at specific time:

What is wordpress wp-cron.php, Scheduling wordpress post screenshot

The Publish Immediately, field execution is being issued on the scheduled time thanks to the wp-cron.php periodic invocation.

Another example for wp-cron.php operation is in handling flushing of WP old HTML Caches generated by some wordpress caching plugin like W3 Total Cache
wp-cron.php takes care for dozens of other stuff silently in the background. That's why many wordpress plugins are depending heavily on wp-cron.php proper periodic execution. Therefore if something is wrong with wp-config.php, this makes wordpress based blog or website partially working or not working at all.
 

Our company wp-cron.php errors case

In our case the:
212.235.185.131 – – [15/Apr/2010:06:32:12 -0600] "POST /wp-cron.php?doing_wp_cron HTTP/1.0" 404
is occuring in Apache access.log (after each unique vistor request to wordpress!.), this is cause wp-cron.php is invoked on each new site visitor site request.
This puts a "vain load" on the Apache Server, attempting constatly to invoke the script … always returning not found 404 err.

As a consequence, the WP website experiences "weird" problems all the time. An illustration of a problem caused by the impoper wp-cron.php execution is when we are adding new plugins to WP.

Lets say a new wordpress extension is download, installed and enabled in order to add new useful functioanlity to the site.

Most of the time this new plugin would be malfunctioning if for example it is prepared to add some kind of new html form or change something on some or all the wordpress HTML generated pages.
This troubles are result of wp-config.php's inability to update settings in wp SQL database, after each new user request to our site.
So the newly added plugin website functionality is not showing up at all, until WP cache directory is manually deleted with rm -rf /var/www/blog/wp-content/cache/…

I don't know how thi whole wp-config.php mess occured, however my guess is whoever installed this wordpress has messed something in the install procedure.

Anyways, as I researched thoroughfully, I red many people complaining of having experienced same wp-config.php 404 errs. As I red, most of the people troubles were caused by their shared hosting prohibiting the wp-cron.php execution.
It appears many shared hostings providers choose, to disable the wordpress default wp-cron.php execution. The reason is probably the script puts heavy load on shared hosting servers and makes troubles with server overloads.

Anyhow, since our company server is adedicated server I can tell for sure in our case wordpress had no restrictions for how and when wp-cron.php is invoked.
I've seen also some posts online claiming, the wp-cron.php issues are caused of improper localhost records in /etc/hosts, after a thorough examination I did not found any hosts problems:

hipo@debian:~$ grep -i 127.0.0.1 /etc/hosts
127.0.0.1 localhost.localdomain localhost

You see from below paste, our server, /etc/hosts has perfectly correct 127.0.0.1 records.

Changing default way wp-cron.php is executed

As I've learned it is generally a good idea for WordPress based websites which contain tens of thousands of visitors, to alter the default way wp-cron.php is handled. Doing so will achieve some efficiency and improve server hardware utilization.
Invoking the script, after each visitor request can put a heavy "useless" burden on the server CPU. In most wordpress based websites, the script did not need to make frequent changes in the DB, as new comments in posts did not happen often. In most wordpress installs out there, big changes in the wordpress are not common.

Therefore, a good frequency to exec wp-cron.php, for wordpress blogs getting only a couple of user comments per hour is, half an hour cron routine.

To disable automatic invocation of wp-cron.php, after each visitor request open /var/www/blog/wp-config.php and nearby the line 30 or 40, put:

define('DISABLE_WP_CRON', true);

An important note to make here is that it makes sense the position in wp-config.php, where define('DISABLE_WP_CRON', true); is placed. If for instance you put it at the end of file or near the end of the file, this setting will not take affect.
With that said be sure to put the variable define, somewhere along the file initial defines or it will not work.

Next, with Apache non-root privileged user lets say www-data, httpd, www depending on the Linux distribution or BSD Unix type add a php CLI line to invoke wp-cron.php every half an hour:

linux:~# crontab -u www-data -e

0,30 * * * * cd /var/www/blog; /usr/bin/php /var/www/blog/wp-cron.php 2>&1 >/dev/null

To assure, the php CLI (Command Language Interface) interpreter is capable of properly interpreting the wp-cron.php, check wp-cron.php for syntax errors with cmd:

linux:~# php -l /var/www/blog/wp-cron.php
No syntax errors detected in /var/www/blog/wp-cron.php

That's all, 404 wp-cron.php error messages will not appear anymore in access.log! 🙂

Just for those who can find the root of the /wp-cron.php?doing_wp_cron HTTP/1.0" 404 and fix the issue in some other way (I'll be glad to know how?), there is also another external way to invoke wp-cron.php with a request directly to the webserver with short cron invocation via wget or lynx text browser.

– Here is how to call wp-cron.php every half an hour with lynxPut inside any non-privileged user, something like:
01,30 * * * * /usr/bin/lynx -dump "http://www.your-domain-url.com/wp-cron.php?doing_wp_cron" 2>&1 >/dev/null

– Call wp-cron.php every 30 mins with wget:

01,30 * * * * /usr/bin/wget -q "http://www.your-domain-url.com/wp-cron.php?doing_wp_cron"

Invoke the wp-cron.php less frequently, saves the server from processing the wp-cron.php thousands of useless times.

Altering the way wp-cron.php works should be seen immediately as the reduced server load should drop a bit.
Consider you might need to play with the script exec frequency until you get, best fit cron timing. For my company case there are only up to 3 new article posted a week, hence too high frequence of wp-cron.php invocations is useless.

With blog where new posts occur once a day a script schedule frequency of 6 up to 12 hours should be ok.

 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in System Administration, Web and CMS, Wordpress | 3 Comments »

« Older Entries