Posts Tagged ‘solution’

« Older Entries

Fix Zabbix selinux caused permission issues on CentOS 7 Linux / cannot set resource limit: [13] Permission denied error solution

Tuesday, July 6th, 2021

zabbix-selinux-logo-fix-zabbix-permission-issues-when-running-on-ceontos-linux-change-selinux-to-permissive-howto.

If you have to install Zabbix client that has to communicate towards Zabbix server via a Zabbix Proxy you might be unpleasently surprised that it cannot cannot be start if the selinux mode is set to Enforcing.
Error message like on below screenshot will be displayed when starting proxy client with systemctl.

zabbix-proxy-cannot-be-started-due-to-selinux-permissions

In the zabbix logs you will see error  messages such as:
 

"cannot set resource limit: [13] Permission denied, CentOS 7"

 

29085:20160730:062959.263 Starting Zabbix Agent [Test host]. Zabbix 3.0.4 (revision 61185).
29085:20160730:062959.263 **** Enabled features ****
29085:20160730:062959.263 IPv6 support: YES
29085:20160730:062959.263 TLS support: YES
29085:20160730:062959.263 **************************
29085:20160730:062959.263 using configuration file: /etc/zabbix/zabbix_agentd.conf
29085:20160730:062959.263 cannot set resource limit: [13] Permission denied
29085:20160730:062959.263 cannot disable core dump, exiting…

 

Next step to do is to check whether zabbix is listed in selinux's enabled modules to do so run:
 

[root@centos ~ ]# semodules -l
…
…..
vhostmd    1.1.0
virt    1.5.0
vlock    1.2.0
vmtools    1.0.0
vmware    2.7.0
vnstatd    1.1.0
vpn    1.16.0
w3c    1.1.0
watchdog    1.8.0
wdmd    1.1.0
webadm    1.2.0
webalizer    1.13.0
wine    1.11.0
wireshark    2.4.0
xen    1.13.0
xguest    1.2.0
xserver    3.9.4
zabbix    1.6.0
zarafa    1.2.0
zebra    1.13.0
zoneminder    1.0.0
zosremote    1.2.0
 

[root@centos ~ ]# sestatus
# sestatusSELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

To get exact zabbix IDs that needs to be added as permissive for Selinux you can use ps -eZ like so:

[root@centos ~ ]# ps -eZ |grep -i zabbix
system_u:system_r:zabbix_agent_t:s0 1149 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1150 ?     00:04:28 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1151 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1152 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1153 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1154 ?     02:21:46 zabbix_agentd

As you can see zabbix is enabled and hence selinux enforcing mode is preventing zabbix client / server to operate and communicate normally, hence to make it work we need to change zabbix agent and zabbix proxy to permissive mode.

Setting selinux for zabbix agent and zabbix proxy to permissive mode

If you don't have them installed you might neet the setroubleshoot setools, setools-console and policycoreutils-python rpms packs (if you have them installed skip this step).

[root@centos ~ ]# yum install setroubleshoot.x86_64 setools.x86_64 setools-console.x86_64 policycoreutils-python.x86_64

Then to add zabbix service to become permissive either run

[root@centos ~ ]# semanage permissive –add zabbix_t

[root@centos ~ ]# semanage permissive -a zabbix_agent_t


In some cases you might also need in case if just adding the permissive for zabbix_agent_t try also :

setsebool -P zabbix_can_network=1

Next try to start zabbox-proxy and zabbix-agent systemd services 

[root@centos ~ ]# systemctl start zabbix-proxy.service
…

[root@centos ~ ]# systemctl start zabbix-agent.service
…

Hopefully all should report fine with the service checking the status should show you something like:

[root@centos ~ ]# systemctl status zabbix-agent
● zabbix-agent.service – Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-06-24 07:47:42 CEST; 1 weeks 5 days ago
 Main PID: 1149 (zabbix_agentd)
   CGroup: /system.slice/zabbix-agent.service
           ├─1149 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─1150 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─1151 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─1152 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─1153 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─1154 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Check the Logs finally to make sure all is fine with zabbix being allowed by selinux.

[root@centos ~ ]# grep zabbix_proxy /var/log/audit/audit.log
…

[root@centos ~ ]# tail -n 100 /var/log/zabbix/zabbix_agentd.log


If no errors are in and you receive and you can visualize the usual zabbix collected CPU / Memory / Disk etc. values you're good, Enjoy ! 🙂

Tags: , , , , , , , , ,
Posted in Linux, Monitoring, System Administration | No Comments »

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness

Wednesday, July 24th, 2019

linux-why-du-and-df-shows-different-result-inconsincy-explained-filesystem-full-oddity

If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reports its file use different when checked with
df cmd than when checked with du command, like for example:
 

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total

 

1. Simple debug on what might be the root cause for df / du inconsistency reporting

 

Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:

 

# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do

 

 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less
…

 

I've also investigated on the complete /var directory contents sorted by size with:

 

 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache
…


Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:
 

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql

 

So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).
 

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:
 

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www


FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:
 

# netstat -tupanl


Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files

 

# lsof -nP | grep '(deleted)'

COMMAND   PID   USER   FD   TYPE DEVICE    SIZE NLINK  NODE NAME
mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)
…

 

Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:

 

# lsof -nP | grep '(deleted)' |wc -l
438


As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):

 

# lsof +L1|less


The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.
 

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:

 

– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/libnss_files-2.28.so

 

# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/libm-2.28.so
…

Other command that helped to track the discrepancy between df and du different file usage on FS is:
 

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'
 

 

3. Fixing large files kept in memory filesystem problem


What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:

 

 

# systemctl restart mysqld.service


or on older init.d system V ones:

# /etc/init.d/service restart


For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l script.sh
7977 script.sh


# kill -HUP PID

 

Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.

 

# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total

 

What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?
 

Tags: , , , , , , , , , , , , , , , , , , , ,
Posted in Cloud services, Curious Facts, Linux | No Comments »

Fix “FAIL – Application at context path /application-name could not be started” deployment error in Tomcat manager

Thursday, October 1st, 2015

tomcat-manager-FAIL-Application-at-context-path-application-name-could-not-be-started-fix-solution-error

While deploying an environment called "Interim" which is pretty much like a testing Java application deployed from a Java EAR (Enterprise Archive) file from within a Tomcat Manager GUI web interface after stopping the application and trying to start it, the developers come across the error:

 

FAIL – Application at context path /application-name could not be started


The error puzzled me for a while until I checked the catalina.out I've seen a number of thrown Java Eceptions errors like:

Okt 01, 2015 10:48:46 AM org.springframework.web.context.ContextLoader initWebApplicationContext

Schwerwiegend: Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.sun.xml.ws.transport.http.servlet.SpringBinding#2' defined in ServletContex

t resource [/WEB-INF/pp-server-beans.xml]: Cannot create inner bean ‘(inner bean)’ of type [org.jvnet.jax_ws_commons.spring.SpringService] while setting bean property

'service'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#33': FactoryBean threw exception on

 object creation; nested exception is java.lang.OutOfMemoryError: PermGen space

I've googled a bit about the error:

"FAIL – Application at context path /application-name could not be started"

and come across this Stackoverflow thread and followed suggested solution to fix web.xml tag closing error but it seems there was no such error in my case, I then also tried solution suggested by this thread (e.g. adding in logging.properties) file:
 

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

unfortunately this helped neither to solve the error when it is tried to be started from tomcat manager.

After asking for help a colleague Kostadin, he pointed me to take a closer look in the error (which is a clear indication) that the reserved space is not enough (see below err):
 

java.lang.OutOfMemoryError: PermGen space

And he pointed me then to Solution (which is to modify the present tomcat setenv.sh) settings which looked like this:

# Heap size settings

export JAVA_OPTS="-Xms2048M -Xmx2048M"

 

# SSCO test page parameter

export JAVA_OPTS="$JAVA_OPTS -DTS1A_TESTSEITE_CONFIG_PATH=test-myapplication.com"

# Default garbage collector settings

export JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=128M"

 

# Aggressive garbage collector settings.

# Do not use! For testing purposes only!

#export JAVA_OPTS="$JAVA_OPTS -Xss128k -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts -XX:MaxPermSize=128M"

 

####### DO NOT CHANGE BELOW HERE #######

# Disable X11 usage

unset DISPLAY

export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

 

# Garbage collection log settings

export JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/web/tomcat/current/logs/gc.log -XX:-TraceClassUnloading"

 

# Enable JMX console

export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"

 

 

 

 

 

 

The solution is to to add following two options to export JAVA_OPTS (above options):

-XX:PermSize=512m -XX:MaxPermSize=512m


After modifications, my new line within setenv.sh looked like so:

 

JAVA_OPTS="-Xms2048M -Xmx2048M -XX:PermSize=512m -XX:MaxPermSize=512m"


Finally to make new JAVA_OPTS settings, I've restarted Tomcat with:

 

cd /web/tomcat/bin/
./shutdown.sh
sleep 5; ./startup.sh


And hooray it works fine thanks God! 🙂

Tags: , , , , , , , , , ,
Posted in Java, Linux, Tomcat, Web and CMS | 2 Comments »

Fix qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied – Finally a working solution and what causes the error

Wednesday, July 22nd, 2015

 

qmail-fixing-clamdscan-errors-and-qq-errors-qmail-binary-migration-few-things-to-check-out
I've lost a whole day and was angry and irritated after moving (migrating) a Working Qmail installation in a binary form from a Debian Lenny 5.0 to a Debian 7.0 Wheezy Linux. The whole migration exercise was quite of a crazy move and I can tell you it didn't worth the effort as I lost much more time than even if I went on installing the server following Thibs Great Qmail Tutorial.

Yes I know many would say why do you still bother with an old and unsupported Qmail Mail server and not go with Postfix, the logic is correct however the whole issue is the previous installation has a number of domains already running VirtualMail hosting using VPopMail, so migrating all the old mailboxes from Qmail to Postfix are not worthy IMHO. Plus I honestly love qmail for being so stable even today (even without support). After all most of Qmail is secure enough already 🙂
And to be honest I don't so much care about security as in the old days as I know NSA, does already have access to any server on planet 🙂

Almost always when a Qmail migration happens I end up swearing and sweating and generally getting crazy, but anyways …

The overall migration of binaries went quite OK I just copied every binary and all the related libraries from the old Debian 5.0 to Debian 7.0 and installed the following long list of perl deb binaries using apt-get:
 

dh-make-perl
libalgorithm-c3-perl
libalgorithm-diff-perl
libalgorithm-diff-xs-perl
libalgorithm-merge-perl
libapparmor-perl
libapt-pkg-perl
libarray-unique-perl
libclass-accessor-chained-perl
libclass-accessor-perl
libclass-c3-perl
libclass-c3-xs-perl
libclass-factory-util-perl
libclass-inspector-perl
libclass-isa-perl
libclass-load-perl
libclass-singleton-perl
libconfig-file-perl
libconvert-binhex-perl
libcrypt-openssl-bignum-perl
libcrypt-openssl-random-perl
libcrypt-openssl-rsa-perl
libcrypt-passwdmd5-perl
libcrypt-ssleay-perl
libdata-optlist-perl
libdata-section-perl
libdate-manip-perl
libdatetime-format-builder-perl
libdatetime-format-iso8601-perl
libdatetime-format-strptime-perl
libdatetime-locale-perl
libdatetime-perl
libdatetime-timezone-perl
libdbd-mysql-perl
libdbi-perl
libdevel-symdump-perl
libdigest-hmac-perl
libdigest-sha-perl
libdpkg-perl
libemail-address-perl
libemail-date-format-perl
libencode-detect-perl
libencode-locale-perl
libenv-sanctify-perl
liberror-perl
libexporter-lite-perl
libfcgi-perl
libfile-chdir-perl
libfile-fcntllock-perl
libfile-find-rule-perl
libfile-listing-perl
libfile-which-perl
libfont-afm-perl
libhtml-form-perl
libhtml-format-perl
libhtml-parser-perl
libhtml-tagset-perl
libhtml-template-perl
libhtml-tree-perl
libhttp-cookies-perl
libhttp-daemon-perl
libhttp-date-perl
libhttp-message-perl
libhttp-negotiate-perl
libhttp-server-simple-perl
libio-multiplex-perl
libio-socket-inet6-perl
libio-socket-ip-perl
libio-socket-ssl-perl
libio-string-perl
libio-stringy-perl
libip-country-perl
liblist-moreutils-perl
liblocale-gettext-perl
liblwp-mediatypes-perl
liblwp-protocol-https-perl
libmail-dkim-perl
libmail-sendmail-perl
libmail-spf-perl
libmailtools-perl
libmath-round-perl
libmime-tools-perl
libmodule-depends-perl
libmodule-implementation-perl
libmodule-runtime-perl
libmro-compat-perl
libnet-cidr-lite-perl
libnet-cidr-perl
libnet-daemon-perl
libnet-dns-perl
libnet-http-perl
libnet-ident-perl
libnet-ip-perl
libnet-server-perl
libnet-snmp-perl
libnet-ssleay-perl
libnetaddr-ip-perl
libnumber-compare-perl
libossp-uuid-perl
libpackage-deprecationmanager-perl
libpackage-stash-perl
libpackage-stash-xs-perl
libparams-classify-perl
libparams-util-perl
libparams-validate-perl
libparse-debcontrol-perl
libparse-debianchangelog-perl
libparse-syslog-perl
libpcre-ocaml-dev
libpcre3:amd64
libpcre3-dev
libpcrecpp0:amd64
libperl-dev
libperl5.14
libpod-coverage-perl
libregexp-assemble-perl
librpc-xml-perl
librrds-perl
libsoap-lite-perl
libsocket-perl
libsocket6-perl
libsoftware-license-perl
libsub-exporter-perl
libsub-install-perl
libsub-name-perl
libswitch-perl
libsys-hostname-long-perl
libsys-syslog-perl
libtask-weaken-perl
libterm-readkey-perl
libtest-distribution-perl
libtest-pod-coverage-perl
libtest-pod-perl
libtext-charwidth-perl
libtext-glob-perl
libtext-iconv-perl
libtext-template-perl
libtext-wrapi18n-perl
libtie-ixhash-perl
libtimedate-perl
libtry-tiny-perl
liburi-perl
libwww-mechanize-perl
libwww-perl
libwww-robotrules-perl
libxml-namespacesupport-perl
libxml-parser-perl
libxml-sax-base-perl
libxml-sax-expat-perl
libxml-sax-perl
libxml-simple-perl
libyaml-perl
libyaml-syck-perl
perl
perl-base
perl-doc
perl-modules
spamassassin
spf-tools-perl

 

I've also installed with apt-get daemontools and daemontools-run ucspi and some others to get all the necessery binaries qmail needs the whole list of apt installed packages is here

I've also copied all the old binaries from /usr/local/lib from server1 to server2, some others from /usr/local/share and /usr/share as well as /usr/lib/courier /usr/lib/courier-authlib /usr/local/libexec /usr/local/sbin also had to link a couple of libraries such as /usr/lib/libcrypto* , link  /usr/lib/libperl.so.5.10 to /usr/lib/libperl.so.5.14 and copy /usr/lib/libltdl.so.3 and few others which i don't exactly remember.

Well anyways once I've copied everything Qmail looked fined except I had a couple of permission issues and had to clean up and fix the queue with qfixq, I've also used qmail-scanner*/contrib/test_installation.sh script to test whether qmail-scanner was running fine, e.g.:
 

./test_installation.sh -doit


As well as

 

 

qmr_inst_check


script, thanks to which I've captured and resolved few of permission problems 

Finally I've stuck upon this shitty errors (appearing) in /var/log/syslog and /var/log/messages

 

 

Jul 21 22:04:19 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:27 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:38 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:11:17 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:16:09 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:19:15 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:38:59 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:42:33 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:43:49 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:46:05 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:50:40 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:53:08 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied

 

 

 

There is plenty of things written about this:
 

qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied


But all written is too obscure and too old already somewhere between 2004 and 2010, I've been digging through Gentoo Forums, Fedora Debian and other Linux installs and everyone used to be pointing a permission issue with /var/spool/qscan/ said theoretically to be causing the error, however all looked perfectly fine with my /var/spool/qscan , e.g.:

 

 

roo@ns2:/usr/local/src# ls -ld /var/spool/qscan/
drwxr-s— 6 qscand qscand 4096 Jul 21 23:07 /var/spool/qscan/

 

 

 

 

ls -al /var/spool/qscan/
total 244904
drwxr-s— 6 qscand qscand      4096 Jul 21 23:07 .
drwxr-xr-x 4 root   root        4096 Jul 20 21:17 ..
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 archives
-rwxr-x— 1 qscand qscand      1434 Oct 12  2011 log-report.sh
-rw——- 1 qscand qscand 249731919 Jul 21 23:11 qmail-queue.log
-rw——- 1 qscand qscand    398225 Oct 28  2011 qmail-queue.log.1
-rw-rw—- 1 root   qscand        74 Jul 21 23:07 qmail-scanner-queue-version.txt
lrwxrwxrwx 1 root   qscand        16 Jul 21 23:07 qscan -> /var/spool/qscan
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 quarantine
-rw-r—– 1 root   qscand     12288 Jul 21 23:07 quarantine-events.db
-rw-r—– 1 qscand qscand     10443 Oct 12  2011 quarantine-events.txt
-rw-rw—- 1 qscand qscand    580033 Jul 21 23:07 quarantine.log
-rw-r—– 1 qscand qscand      2739 Oct 12  2011 settings_per_domain.txt
drwxr-x— 3 qscand qscand      4096 Jul 21 23:11 tmp
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 working

 

Some suggested that /var/spool/qscan should be owned by qscand:clamav instead so I tried this but it didn't help,
others recommended adding clamav groupid into qscand's but this didn't help either:
 

root@ns2:/usr/local/src# grep -i clamav /etc/group
qscand:x:163:clamav,vpopmail
clamav:x:105:

 

Besides that I was getting also this shitty error:

 

Jul 21 20:05:42 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750194279012466] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:06:51 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750201179013125] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:15:42 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns214375025407906015] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:16:06 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750256579011980] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:18:54 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750273479014847] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:21:03 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750286379028491] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 22:07:47 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750926779010097] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status 512/2
…

 

All the time in logs, so I thought something might be wrong with clamdscan and followed and tried some suggestions from qms1.net as described here, however none of the fixes described there worked for me …

I also tried to reinstall clamav from source using a bit of modified version of this tutorial

This didn't help either … I saw some suggestions online that the permission issues are caused by some wrong clamd.conf and freshclamd.conf configuration options – failing clamdscan, but this didn't work either. I also tried to remove clamdscan and substitute it with clamscan  as a suggested workaround but this didn't work either …

I spend about 6 hours trying to catch what is causing this issues so finally I went on and re-installed bigger part of Qmail using Thibs tutorial over the old installation.

I've also tried in mean time multiple time to rebuild qmail scanner database with:
 

setuidgid qscand /var/qmail/bin/qmail-scanner-queue.pl -g


Played around with permissions in /etc/clamav e.g.

 

 

chown -R qscand:clamav /var/log/clamav /var/lib/clamav /var/run/clamav
chown qscand:qscand /etc/clamav/freshclam.conf

 

 

 

Created:

/etc/cron.daily/qmail-scanner with following content

 

cat /etc/cron.daily/qmail-scanner
setuidgid qscand /var/qmail/bin/qmail-scanner-queue.pl -z


However in /var/log/qmail/qmail-smtpd/current and /var/log/qmail-send/current I continously got:

Qmail 451 qq temporary problem (#4.3.0) error

Interestingly during looking for a solution to the 451 qq temporary problem and:

status: qmail-todo stop processing asap / status: exiting – I've stumbled to my own blog post here 🙂

 

Finally, I tried to reinstall qmail-scanner and in mean time update it to  Version: 2.11 – st – patch – 20130319
Just to realized something was wrong with suidperl, e.g. in Debian 7.0.* Wheezy perl-suid binary is no longer in repositories so only way to have suidperl there is either to re-compile perl from source manually which is too much work and I think in most cases not worthy the effort or to use a small suid-wrapper:
 

#include <unistd.h>
#include <errno.h>
 

main( int argc, char ** argv, char ** envp )
{
              if( setgid(getegid()) ) perror( "setgid" );
              if( setuid(geteuid()) ) perror( "setuid" );
              envp = 0; /* blocks IFS attack on non-bash shells */
              system( "/usr/bin/perl", argv, envp );
              perror( argv[0] );
              return errno;
}

 


Create it into a file lets say suid-wrapperc and compile the file with GNU C Compiler:
 

$ gcc -o suid-wrapper suid-wrapper.c


Then move the suid-wrapper produced binary into /usr/bin/suidperl e.g.

 

 

$ mv suid-wrapper /usr/bin/suidperl


Last you will need to edit /var/qmail/bin/qmail-scanner-queue.pl

 

 

 

 

# vim /var/qmail/bin/qmail-scaner-queue.pl


And substitute

 

 

 

 

#!/usr/bin/perl -T


with:

 

 

#!/usr/bin/suidperl


Note!!! that qmail-scanner-queue.pl permissions should be suid and owned by qscand:qscand as follows:

 

 

 

 

ls -al /var/qmail/bin/qmail-scanner-queue.pl
-rwsr-sr-x 1 qscand qscand 159727 Jul 21 23:11 /var/qmail/bin/qmail-scanner-queue.pl


Finally to resolve the error I had to restart qmail via qmailctl start / stop script:

 

 

 

 

root@ns2:/var/qmail/bin# qmailctl restart
Restarting qmail:
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.

 

Finally to test emails are sent and receiver properly I used good old mail command part of bsd-mailx deb package

 

# mail -s "testing 12345678" testemail1234@gmail.com
asdfadf
.
Cc:

 

 

I've also tested with plain telnet to verify no errors because often the mail command doesn't return (show) errors on email sent and errors are written only in /var/log/mail.log or /var/log/qmail/* logs

 

 

# telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 servername.localdomain.tld ESMTP
mail from: <testmail@test.com>
250 ok
rcpt to: <nospam@test.com>
250 ok
data
354 go ahead
From: Test_sender <testmail@test.com>
To: Test_receiver <nospam@test.com>
Subject: Just a stupid SMTP test

 

Just a test !
.
250 ok 1279384489 qp 3711
quit
221 servername.localdomain.tld
Connection closed by foreign host.

One other thing which probably helped I did was:

# qmailctl doqueue

 

Thanks God this time, it worked out without any QQ errors 🙂 !
 

 

Tags: , , , , , , , , , , , ,
Posted in Everyday Life, Linux, Qmail, System Administration, Various | 6 Comments »

Fix Squirrelmail UTF-8 and windows-1251 Bulgarian encoding problem

Monday, September 8th, 2014

squirrelmail_webmail_for_nuts_fix_bulgarian_reply_encoding_problem-howto
I'm using Squirrelmail (with OutLook skin theme) as a webmail client for my home running Qmail mail server and few other squirrelmail plugins, generally I'm quite happy with Squirrelmail as it perfectly serves me as a web IMAP client the only issue I have is when I'm replying to messages which are written Cyrillic UTF-8 or Cyrillic windows cp-1251 encoding (Bulgarian letters) – I guess Russians which are using squirrelmail with KOI-8R encoding also probably face some similar mail encoding issues. That's pretty annoying because the person message which I'm Replying to gets scrapped and old content becomes unreadable so old correspondence gets broken, because the encoding in which the message is replied is in non-utf-8 encoding ISO-8859, you see example of what I mean in below screenshot:

Fix_solve_Squirrelmail_UTF-8_and_windows-1251_Bulgarian_Russian_encoding_problem_screenshot

So here is how to fix that:
Solution is to convert the Bulgarian translation from windows-1251 to UTF-8 in squirremail.po:
 

find . -iname squirrelmail.po
./themes/squirreloutlook-1.0.3/locale/pt_BR/LC_MESSAGES/squirrelmail.po

cd themes/squirreloutlook-1.0.3/locale/pt_BR/LC_MESSAGES/
iconv -f CP1251 -t UTF-8 squirrelmail.po > squirrelmail.utf-8.po
sed 's/cp1251/UTF-8/' squirrelmail.utf-8.po > squirrelmail.po
rm squirrelmail.utf-8.po
msgfmt -o squirrelmail.mo squirrelmail.po
cd ../../../../../


For Squirrelmail version 1.4

find . -iname i18n.php
./functions/i18n.php
./themes/squirreloutlook-1.0.3/functions/i18n.php

vim ./themes/squirreloutlook-1.0.3/functions/i18n.php
vim functions/i18n.php


Paste below, the commented section about Bulgarian Language (//) (below configuration):

 

$languages[‘bg_BG’][‘NAME’] = 'Bulgarian';
$languages[‘bg_BG’][‘CHARSET’] = 'utf-8';
$languages[‘bg_BG’][‘LOCALE’] = 'bg_BG.UTF-8';
$languages[‘bg’][‘ALIAS’] = 'bg_BG';

 

 


For Squirrelmail version 1.5


locale/bg_BG/setup.php
 

Finally to make new Squirrelmail configuration affective restart Apache Webserver

 

/etc/init.d/apache2 restart
…


Fixing issues with broken (Russian) cyrillic KOI8-R encoding in reply mail in squirrelmail should be analogical.

 

 

 

Tags: , , , , , , , , ,
Posted in Everyday Life, Various, Web and CMS | No Comments »

Create video from linux console / terminal – Record ssh terminal session as video with asciinema, showterm, termrecord

Thursday, August 21st, 2014

/var/www/images/asciinema-create-and-upload-ascii-terminal-console-videos-debian-gnu-linux-screenshot
You probably already know of existence of two Linux commands available by default across all Linux distributions script – which makes a text based save of all commands executed on console and scriptreplay – which playbacks saved script command typescripts. Using this two you can save terminal sessions without problem, but in order to play them you need to have a Linux / UNIX computer at hand.
However If you want to make a short video record displaying what you have done on Linux console / terminal, you have few other options with which you can share your Linux terminal sessions on the web. In this short article I will go through 3 popular tools to do that – asciinema, showterm and termrecord.

1. Asciinema Current most popular tool to create video from Linux terminal

Here is how ASCIINEMA's website describes it:

"Asciinema is a free and open source solution for recording the terminal sessions and sharing them on the web."

apt-get –yes install python-pip

To install it with pip – python package installer

pip install asciinema

Or if the machine is in DMZ secured zone and have access to the internet over a Proxy:

pip install –proxy=http://internet-proxy-host.com:8080 asciinema

It will get installed in /usr/local/bin/asciinema to make a terminal screen video capture just launch it (nomatter if it is privileged or non-privileged user):

asciinema

To finalize and upload the recorded terminal session, just type exit (to exit the shell), hopefully it will get you an upload link.

exit

You can claim authorship on video you issue:

asciinema auth

Use can then embed the new Linux terminal session video to your website.
 

2. ShowTerm – "It's showtime in a terminal near you!"

ShowTerm have same features as AsciiNema. Just like AsciiNema, what it does is it creates a record of your terminal session and then uploads it to showterm.io website, providing you a link over which you can share your terminal lesson / ascii art video / whatever with your friends. ShowTerm is written in, the world famous Ruby on Rails – ruby web development framework, so you will need to have ruby programming language installed before use. As showterm uses the Internet to upload video, so it is not really an option to create videos from remote terminal session on servers which are in DMZ with no access to the internet, I will explain in a little while how to create video of your terminal / console for private purpose on local server and then share it online on your own site.

a) To install ShowTerm:

– First be sure to have ruby installed:

On Debian / Ubuntu and derives deb Linux, as supersuser:

apt-get install –yes ruby curl

On CentOS / RHEL / Fedora Linux

yum -y install ruby curl

NB! curl is real requirement but as showterm.io website recommends downloading the script with it and later same curl tool is used to upload the created showterm file to http://showterm.io .

– Then to finalize install, download showterm script and make it executable

curl showterm.io/showterm > ~/bin/showterm

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
100  2007  100  2007    0     0   2908      0 –:–:– –:–:– –:–:–  8468
mkdir ~/bin
chmod +x ~/bin/showterm

This will save the script into your home folder ~/bin/showterm

b) Using showterm

To run it to create video from your terminal simply start it and do whatver you will in terminal.

~/bin/showterm

After you're done with the video you like type exit

exit

create-video-from-your-linux-console-terminal-with-showterm-screenshot

Note that if your server is behind a proxy curl will not understand proxy set inside Linux shell variable with http_proxy var, to upload the file if you're behind a proxy you will have to pass to curl –proxy setting, once you get the curl line invoked after failure to upload use something like:

curl –proxy $(echo $http_proxy)  https://showterm.herokuapp.com/scripts –data-urlencode cols=80 –data-urlencode lines=24 –data-urlencode scriptfile@/tmp/yCudk.script –data-urlencode timingfile@/tmp/lkiXP.timing

Where assuming proxy is defined already inside http_proxy shell variable.

 

3. Creating video from your terminal / console on Linux for local (private) use with TermRecord

In my humble view TermRecord is the most awesome of all the 3, as it allows you to make records with an own generated Javascript based video player and allows you to keep the videos on your own side, guaranteeing you independence of external services. Its
 

pip install TermRecord

TermRecord -o /tmp/session.html

 

You can further access the video in a local browser in Firefox / Chrome / Epiphany type in URL address bar:

/tmp/session.html to play the video

create-video-from-terminal-console-on-gnu-linux-howto-screenshot-with-termrecord

TermRecord uses term.js javascript to create the video web player and play the video which is directly encoded inside session.html.
If you want to share the video online, place it on your webserver and you're done 🙂
Check out my TermRecord generated video terminal sample session here.
 

Tags: , , , , , , , , , , , , , , , ,
Posted in Entertainment, Everyday Life, Linux, Linux and FreeBSD Desktop, Linux Audio & Video, System Administration, Various | No Comments »

ping “General Failure” no internet connection Windows 7 on HP work computer – Reasons for general failure and solution

Monday, May 26th, 2014

windows-7-general-failure-error-fix-on-hp-workbook
Out of a sudden today after running Malware Bytes – Free Anti-Malware & Internet Security Software, and after it found some Malware (Pup.Optional.Opencandy) and removed it it WI-FI internet on my work computer HP Elitebook 8470p mysteriously stopped working.

That's quite nasty because today I'm working from Home – well known among Hewlett Packard employees under WFH abbreviation. I couldn't connect normally to my home Access Point and tried pinging Google from command line just to get an error:
 

Transmit Failed: General Failure


and first I thought it is a wi-fi router related problem and restarted my WIFI Router – D-Link DI524. As I could normally connect to the WIFI and I see there is an internet IP assigned running:

ipconfig /all
...
Wireless LAN adapter Wireless Network Connection:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Centrino(R) Ultimate-N 6300 AGN
Physical Address. . . . . . . . . : 3C-A9-F4-4C-E7-98
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::5d2f:97b8:9e1a:2b13%63(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.2.159(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : May 23, 2014 14:19:01 PM
Lease Expires . . . . . . . . . . : May 30, 2014 14:32:49 PM
Default Gateway . . . . . . . . . : 192.168.2.1
DHCP Server . . . . . . . . . . . : 192.168.2.1
DHCPv6 IAID . . . . . . . . . . . : 1094494708
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-19-CB-1A-5D-A4-5D-36-5A-EB-84
DNS Servers . . . . . . . . . . . : 8.8.8.8
192.168.2.1
NetBIOS over Tcpip. . . . . . . . : Enabled

As you see in above output I have notebook IP, default gateway and DNS IP assigned – i.e. all seems fine, so as I got General Failure from pinging the Internet in order to make sure my Linux router is not the bottleneck I tried pinging Default GW

C:UsersGeorgi> ping 192.168.2.1 Pinging 192.168.2.1 with 32 bytes of data:
PING: transmit failed. General failure.
PING: transmit failed. General failure.
PING: transmit failed. General failure.
PING: transmit failed. General failure.
Ping statistics for 192.168.2.1:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)

Just to realize I'm continually getting General Failure error …

I tried trying to renew IPs, to make sure there is no some kind of IP assignment conflict with my other WIFI connected devices,reflushing DNS and resetting WinSock stack, hoping this could help:

> ipconfig /renew
> ipconfig /flushdns
> netsh winsock reset

Then I tried restarting the PC a couple of times, but unfortunately none of those helped the shitty error:
 

PING: Transmit failed. General Failure

continued …

I was totally out of idea .. and red some people managed to fix the issue after booting Windows into Safe mode with Networking. I tried booting in Safe mode, but as Hard Drive data is encrypted with Bitlocker encryption I was asked about some kinf of Serial Key – which I don't have at hand – hence I couldn't boot into Safe mode …

Here is moment to say even loopback device was returning "General Failure"!

I tried even connecting the laptop directly into my homelan with UTP cable, but though everything got connected, there was no local network and internet. I tried even connecting via Vivacom's mobile network 3G modem and even there I got the "General Failure" error …

Running out of options, I decided it might be that Malware Bytes broke something during Malware removal hence I put out back Quarantined Malware files – but this didn't solve it either.

solution-to-no-internet-general-failure-ping-error-firewall-off-screenshot.png

Finally I found this post and this thread talking that reason for "General Failure" might be firewall related. After checking my firewall settings in Windows Firewall and Advanced Security, surprisingly I realized everything related to firewall – e.g. Default Profile, Inbound, Outbound connections are Turned off!!!

windows-firewall-off-reason-for-general-failure-no-internet

I switched everything back and my Internet and local connection came back! THANKS GOD! Pfuu, now I can continue working. It seems HP work computers are patched with software / configured to not allow Internet connection in case if Firewall is Off. If you happen to be an HP Employee and you get the PING: Transmit failed. General Failure, be sure the first place to check is whether Windows Firewall is enabled? – if not enable it and this will solve your connectivity issues. Cheers ! 🙂

Tags: , , , , , , , , , , , , , , , , ,
Posted in System Administration, Windows | 3 Comments »

Stop contact form spam emails in Joomla, Disable “E-mail a copy of this message to your own address.” in Joomla

Friday, April 11th, 2014

email-copy-of-this-message-to-your-own-address_Contact_email_form
If you happen to have installed Joomla based website and setup a contact form and everything worked fine until recently but suddenly your server starts mysteriously acting as a spam relay – even though email server is perfectly secured against spam.
You probably have some issue with a website email contact form hacked or some vulnerability which allowed hackers to upload spammer php script.

I have a website based on Joomla and just until recently everything was okay until I noticed there are tons of spam flying out from my Qmail mail server (which is configured to check spam with Spamassassin has Bayesian Filtering, Distributed Checksum Claring House, Python Razor and plenty of custom anti-spam rules.

It was just yesterday I ended into that situation, then after evaluating all the hosted website, I've realized Spam issues are caused by an Old Joomla Website Contact form!

There were two issues in the form

in the contact form you have the field with a tick:

1. Well Known Joomla Form Vulnerability
Currently all Joomla (including 1.5.22 and 1.6 versions) are vulnerable to a serious spam relay problem as described in the official Joomla site.

There is a quick dirty workaround fix to contact form vulnerability –  disable a Joomla Comonent in ../joomla/components/com_mailto/

To disable it I had to:

cd /var/www/joomla/components
mv com_mailto com_mailtoNOT_USED

Above solution was described under a post resolve joomla spam relay earlier by Anatoliy Dimitrov (after checking closely the website it happened he is a colleague at HP 🙂 )

2. Second issue causing high amount of spam sent over the email server
was: "E-mail a copy of this message to your own address." contact form tick, which was practically enabling any Spammer with a list to inect emails and spam via the form sending copies to any email out on the internet!

You would definitely want to disable  "E-mail a copy of this message to your own address."
I wonder why ever any Joomla developer came up with this "spam form"?? 

joomla-disable-email-copy-of-this-message-to-your-own-address

Here is the solution to this:

1. Login to Joomla Admin with admin account
2. Goto Components -> Contacts -> Contacts
3. Click on the relevant Contact form
4. Under Contact Parameters go to Email Parameters
5. Change field E-mail Copy from Show to Hide and click Apply button

And Hooray the E-mail a copy of this message to your own address will be gone from contact form! 🙂

I've seen already plenty of problematic hacked servers and scripts before with Joomla in my last job in International University College – where joomla was heavy used, but I never experienced Joomla Security issues myself 'till know, in future I'm planning to never ever use joomla. Though it is an easy CMS system to setup a website its quite complicated to learn the menus – I remember when creating the problematic website it took me days until I properly setup all the menus and find all joomla components … besides these there is no easy way to migrate between different versions major releases in Joomla like in Wordperss, I guess this Mail Security Issue absolutely convinced me to quit using that piece of crap in future.

In mean Time another very serious Apache security flaw leaked on the Internet just few days ago – The OpenSSL Hearbleed Bug. Thanksfully I'm not running SSL anywhere on my website but many systems are affecting making most of your SSL communication with your Internet banking, E-mail etc. in danger. If you're running Apache with SSL make sure you test it for this vulnerability. Here is description of Heartbleed SSL Critical Vulnerability.

heartbleed_ssl_remote_vulnerability_logo

"The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet. SSL/TLS provides communication security and privacy over the Internet for applications such as web, email, instant messaging (IM) and some virtual private networks (VPNs).

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users."

11

 

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in System Administration, Web and CMS | 2 Comments »

Mysql: How to disable single database without dropping or renaming it

Wednesday, January 22nd, 2014

mysql rename forbid disable database howto logo, how to disable single database without dropping it
A colleague of mine working on MySQL database asked me How it is possible to disable a MySQL database. He is in situation where the client has 2 databases and application and is not sure which of the two databases the application uses. Therefore the client asked one of the database is disabled and wait for few hours and see if something will break / stop working and in that way determine which of the two database is used by application.

My first guess was to backup both databases and drop one of them, then if it is the wrong one to restore from the SQL dump backup, however this wasn't acceptable solution. So second I though of RENAME of database to another one and then reverting the name, however as it is written in MySQL documentation RENAME database function was removed from MySQL (found to be dangerous) since version 5.1.23 onwards. Anyhow there is a quick hack to rename mysql database using a for loop shell script one below:

mysql -e "CREATE DATABASE \`new_database\`;"
for table in `mysql -B -N -e "SHOW TABLES;" old_database`
do
  mysql -e "RENAME TABLE \`old_database\`.\`$table\` to \`new_database\`.\`$table\`"
  done
  mysql -e "DROP DATABASE \`old_database\`;"

Other possible solution was to change permissions of Application used username, however this was also complicated from mysql cli, hence I thought of installing and using PHPMyAdmin to make modify of db user permissions easier but on this server there wasn't Apache installed and MySQL is behind a firewall and only accessible via java tomcat host.

Finally after some pondering what can be done I came with solution to request to disable mysql database using chmod in /var/lib/mysql/data/, i.e.:

sql-server:~# chmod 0 /var/lib/mysql/databasename
Where databasename is the same as the database is named listable via mysql cli.

After doing it that way with no need to restart MySQL server database stopped to appear in show databases; and client confirmed that disabled database is no longer needed so we proceeded dropping it.

Hope this little article will help someone out there. Cheers :

Tags: , , , , , , , , , , , , , , , , , ,
Posted in MySQL, System Administration, Various, Web and CMS | 4 Comments »

« Older Entries