test Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘test’

IBM TSM dsmc console client use for listing configured backups, checking set scheduled backups and backup and restore operations howto

Friday, March 6th, 2020

tsm-ibm-logo_tivoli-dsmc-console-client-listing-backups-create-backups-and-restore-on-linux-unix-windows

Creating a simple home based backup solution with some shell scripting and rsync is a common use. However as a sysadmin in a middle sized or large corporations most companies use some professional backup service such as IBM Tivoli Storage Manager TSM – recently IBM changed the name of the product to IBM Spectrum.

IBM TSM is a data protection platform that gives enterprises a single point of control and administration for backup and recovery that is used for Privare Clouds backup and other high end solutions where data criticality is top.
Usually in large companies TSM backup handling is managed by a separate team or teams as managing a large TSM infrastructure is quite a complex task, however my experience as a sysadmin show me that even if you don't have too much of indepth into tsm it is very useful to know how to manage at least basic Incremental backup operations such as view what is set to be backupped, set-up a new directory structure for backup, check the backup schedule configured, check what files are included and which excluded from the backup store etc.

TSM has multi OS support ans you can use it on most streamline Operating systems Windows / Mac OS X and Linux in this specific article I'll be talking concretely about backing up data with tsm on Linux, tivoli can be theoretically brought up even on FreeBSD machines via the Linuxemu BSD module and the 64-Bit Tivoli Storage Manager RPMs.
Therefore in this small article I'll try to give few useful operations for the novice admin that stumbles on tsm backupped server that needs some small maintenance.

1. Starting up the dsmc command line client

Nomatter the operating system on which you run it to run the client run:

# dsmc

tsm-check-backup-schedule-set-time

Note that usually dsmc should run as superuser so if you try to run it via a normal non-root user you will get an error message like:

[ user@linux ~]$ dsmc
ANS1398E Initialization functions cannot open one of the Tivoli Storage Manager logs or a related file: /var/tsm/dsmerror.log. errno = 13, Permission denied

Tivoli SM has an extensive help so to get the use basics, type help

tsm> help
1.0 New for IBM Tivoli Storage Manager Version 6.4
2.0 Using commands
2.1 Start and end a client command session
2.1.1 Process commands in batch mode
2.1.2 Process commands in interactive mode
2.2 Enter client command names, options, and parameters
2.2.1 Command name
2.2.2 Options
2.2.3 Parameters
2.2.4 File specification syntax
2.3 Wildcard characters
2.4 Client commands reference
2.5 Archive
2.6 Archive FastBack

Enter 'q' to exit help, 't' to display the table of contents,
press enter or 'd' to scroll down, 'u' to scroll up or
enter a help topic section number, message number, option name,
command name, or command and subcommand:

2. Listing files listed for backups

A note to make here is as in most corporate products tsm supports command aliases so any command supported described in the help like query, could be
abbreviated with its first letters only, e.g. query filespace tsm cmd can be abbreviated as

tsm> q fi

Commands can be run non-interactive mode also so if you want the output of q fi you can straight use:

tsm> dsmc q fi

tsm-check-included-excluded-files-q-file-if-backupped-list-backup-set-directories

This shows the directories and files that are set for backup creation with Tivoli.

3. Getting included and excluded backup set files

It is useful to know what are the exact excluded files from tsm set backup this is done with query inclexcl

tsm-check-excluded-included-files

4. Querying for backup schedule time

Tivoli as every other backup solution is creating its set to backup files in a certain time slot periods.
To find out what is the time slot for backup creation use;

tsm> q sched
Schedule Name: WEEKLY_ITSERV
Description: ITSERV weekly incremental backup
Schedule Style: Classic
Action: Incremental
Options:
Objects:
Priority: 5
Next Execution: 180 Hours and 35 Minutes
Duration: 15 Minutes
Period: 1 Week
Day of Week: Wednesday
Month:
Day of Month:
Week of Month:
Expire: Never

tsm-query-partitions-backupeed-or-not

5. Check which files have been backed up

If you want to make sure backups are really created it is a good to check, which files from the selected backup files have already
a working backup copy.

This is done with query backup like so:

tsm> q ba /home/*

tsm-dsmc-query-user-home-for-backups

If you want to query all the current files and directories backed up under a directory and all its subdirectories you need to add the -subdir=yes option as below:

tsm> q ba /home/hipo/projects/* -subdir=yes
Size Backup Date Mgmt Class A/I File
—- ———– ———- — —-
512 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hfs0106
1,024 08-12-2011 02:46:53 STANDARD A /home/hipo/projects/hsm41perf
512 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hsm41test
512 24-04-2012 00:22:56 STANDARD A /home/hipo/projects/hsm42upg
1,024 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hfs0106/test
1,024 12-09-2011 19:57:09 STANDARD A /home/hipo/projects/hfs0106/test/test2
12,048 04-12-2011 02:01:29 STANDARD A /home/hipo/projects/hsm41perf/tables
50,326 30-04-2012 01:35:26 STANDARD A /home/hipo/projects/hsm42upg/PMR70023
50,326 27-04-2012 00:28:15 STANDARD A /home/hipo/projects/hsm42upg/PMR70099
11,013 24-04-2012 00:22:56 STANDARD A /home/hipo/projects/hsm42upg/md5check

To make tsm, backup some directories on Linux / AIX other unices:

tsm> incr / /usr /usr/local /home /lib

For tsm to backup some standard netware drives, use:

tsm> incr NDS: USR: SYS: APPS:

To backup C:\ D:\ E:\ F:\ if TSM is running on Windows

tsm> incr C: D: E: F: -incrbydate

To back up entire disk volumes irrespective of whether files have changed since the last backup, use the selective command with a wildcard and -subdir=yes as below:

tsm> sel /* /usr/* /home/* -su=yes ** Unix/Linux

7. Backup selected files from a backup location

It is intuitive to think you can just add some wildcard characters to select what you want
to backup from a selected location but this is not so, if you try something like below
you will get an err.

tsm> incr /home/hipo/projects/*/* -su=yes
ANS1071E Invalid domain name entered: '/home/hipo/projects/*/*'

The proper way to select a certain folder / file for backup is with:

tsm> sel /home/hipo/projects/*/* -su=yes

8. Restoring tsm data from backup

To restore the config httpd.conf to custom directory use:

tsm> rest /etc/httpd/conf/httpd.conf /home/hipo/restore/

N!B! that in order for above to work you need to have the '/' trailing slash at the end.

If you want to restore a file under a different name:

tsm> rest /etc/ntpd.conf /home/hipo/restore/

9. Restoring a whole backupped partition

tsm> rest /home/* /tmp/restore/ -su=yes

This is using the Tivoli 'Restoring multiple files and directories', and the files to restore '*'
are kept till the one that was recovered (saying this in case if you accidently cancel the restore)

10. Restoring files with back date

By default the restore function will restore the latest available backupped file, if you need
to recover a specific file, you need the '-inactive' '-pick' options.
The 'pick' interface is interactive so once listed you can select the exact file from the date
you want to restore.

General restore command syntax is:

tsm> restore [source-file] [destination-file]

tsm> rest /home/hipo/projects/* /tmp/restore/ -su=yes -inactive -pick
TSM Scrollable PICK Window – Restore
# Backup Date/Time File Size A/I File
————————————————————————————————–
170. | 12-09-2011 19:57:09 650 B A /home/hipo/projects/hsm41test/inclexcl.test
171. | 12-09-2011 19:57:09 2.74 KB A /home/hipo/projects/hsm41test/inittab.ORIG
172. | 12-09-2011 19:57:09 2.74 KB A /home/hipo/projects/hsm41test/inittab.TEST
173. | 12-09-2011 19:57:09 1.13 KB A /home/hipo/projects/hsm41test/md5.out
174. | 30-04-2012 01:35:26 512 B A /home/hipo/projects/hsm42125upg/PMR70023
175. | 26-04-2012 01:02:08 512 B I /home/hipo/projects/hsm42125upg/PMR70023
176. | 27-04-2012 00:28:15 512 B A /home/hipo/projects/hsm42125upg/PMR70099
177. | 24-04-2012 19:17:34 512 B I /home/hipo/projects/hsm42125upg/PMR70099
178. | 24-04-2012 00:22:56 1.35 KB A /home/hipo/projects/hsm42125upg/dsm.opt
179. | 24-04-2012 00:22:56 4.17 KB A /home/hipo/projects/hsm42125upg/dsm.sys
180. | 24-04-2012 00:22:56 1.13 KB A /home/hipo/projects/hsm42125upg/dsmmigfstab
181. | 24-04-2012 00:22:56 7.30 KB A /home/hipo/projects/hsm42125upg/filesystems
182. | 24-04-2012 00:22:56 1.25 KB A /home/hipo/projects/hsm42125upg/inclexcl
183. | 24-04-2012 00:22:56 198 B A /home/hipo/projects/hsm42125upg/inclexcl.dce
184. | 24-04-2012 00:22:56 291 B A /home/hipo/projects/hsm42125upg/inclexcl.ox_sys
185. | 24-04-2012 00:22:56 650 B A /home/hipo/projects/hsm42125upg/inclexcl.test
186. | 24-04-2012 00:22:56 670 B A /home/hipo/projects/hsm42125upg/inetd.conf
187. | 24-04-2012 00:22:56 2.71 KB A /home/hipo/projects/hsm42125upg/inittab
188. | 24-04-2012 00:22:56 1.00 KB A /home/hipo/projects/hsm42125upg/md5check
189. | 24-04-2012 00:22:56 79.23 KB A /home/hipo/projects/hsm42125upg/mkreport.020423.out
190. | 24-04-2012 00:22:56 4.27 KB A /home/hipo/projects/hsm42125upg/ssamap.020423.out
191. | 26-04-2012 01:02:08 12.78 MB A /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
192. | 25-04-2012 16:33:36 12.78 MB I /home/hipo/projects/hsm42125upg/PMR70023/70023.tar
0———10——–20——–30——–40——–50——–60——–70——–80——–90–
<U>=Up <D>=Down <T>=Top <B>=Bottom <R#>=Right <L#>=Left
<G#>=Goto Line # <#>=Toggle Entry <+>=Select All <->=Deselect All
<#:#+>=Select A Range <#:#->=Deselect A Range <O>=Ok <C>=Cancel
pick>

To navigate in pick interface you can select individual files to restore via the number seen leftside.
To scroll up / down use 'U' and 'D' as described in the legenda.

11. Restoring your data to another machine

In certain circumstances, it may be necessary to restore some, or all, of your data onto a machine other than the original from which it was backed up.

In ideal case the machine platform should be identical to that of the original machine. Where this is not possible or practical please note that restores are only possible for partition types that the operating system supports. Thus a restore of an NTFS partition to a Windows 9x machine with just FAT support may succeed but the file permissions will be lost.
TSM does not work fine with cross-platform backup / restore, so better do not try cross-platform restores.
Trying to restore files onto a Windows machine that have previously been backed up with a non-Windows one. TSM created backups on Windows sent by other OS platforms can cause backups to become inaccessible from the host system.

To restore your data to another machine you will need the TSM software installed on the target machine. Entries in Tivoli configuration files dsm.sys and/or dsm.opt need to be edited if the node that you are restoring from does not reside on the same server. Please see our help page section on TSM configuration files for their locations for your operating system.

To access files from another machine you should then start the TSM client as below:

# dsmc -virtualnodename=RESTORE.MACHINE

You will then be prompted for the TSM password for this machine.

You will probably want to restore to a different destination to the original files to prevent overwriting files on the local machine, as below:

Restore of D:\ Drive to D:\Restore ** Windows

tsm> rest D:\* D:\RESTORE\ -su=yes

Restore user /home/* to /scratch on ** Mac, Unix/Linux

tsm> rest /home/* /scratch/ -su=yes

Restoring Tivoli data on old netware

tsm> rest SOURCE-SERVER\USR:* USR:restore/ -su=yes ** Netware

12. Adding more directories for incremental backup / Check whether TSM backup was done correctly?

The easiest way is to check the produced dschmed.log if everything is okay there should be records in the log that Tivoli backup was scheduled in a some hours time
succesfully.
A normally produced backup scheduled in log should look something like:

14-03-2020 23:03:04 — SCHEDULEREC STATUS BEGIN
14-03-2020 23:03:04 Total number of objects inspected: 91,497
14-03-2020 23:03:04 Total number of objects backed up: 113
14-03-2020 23:03:04 Total number of objects updated: 0
14-03-2020 23:03:04 Total number of objects rebound: 0
14-03-2020 23:03:04 Total number of objects deleted: 0
14-03-2020 23:03:04 Total number of objects expired: 53
14-03-2020 23:03:04 Total number of objects failed: 6
14-03-2020 23:03:04 Total number of bytes transferred: 19.38 MB
14-03-2020 23:03:04 Data transfer time: 1.54 sec
14-03-2020 23:03:04 Network data transfer rate: 12,821.52 KB/sec
14-03-2020 23:03:04 Aggregate data transfer rate: 114.39 KB/sec
14-03-2020 23:03:04 Objects compressed by: 0%
14-03-2020 23:03:04 Elapsed processing time: 00:02:53
14-03-2020 23:03:04 — SCHEDULEREC STATUS END
14-03-2020 23:03:04 — SCHEDULEREC OBJECT END WEEKLY_23_00 14-12-2010 23:00:00
14-03-2020 23:03:04 Scheduled event 'WEEKLY_23_00' completed successfully.
14-03-2020 23:03:04 Sending results for scheduled event 'WEEKLY_23_00'.
14-03-2020 23:03:04 Results sent to server for scheduled event 'WEEKLY_23_00'.

in case of errors you should check dsmerror.log

Conclusion

In this article I've briefly evaluated some basics of IBM Commercial Tivoli Storage Manager (TSM) to be able to list backups, check backup schedules and how to the files set to be
excluded from a backup location and most importantly how to check that data backed up data is in a good shape and accessible.
It was explained how backups can be restored on a local and remote machine as well as how to append new files to be set for backup on next incremental scheduled backup.
It was shown how the pick interactive cli interface could be used to restore files at a certain data back in time as well as how full partitions can be restored and how some
certain file could be retrieved from the TSM data copy.

Tags: cause, checking, Commands, console, custom, file permissions, General, How to, ITSERV, Listing, machine, NDS, operating system, order, Schedule Name, subdirectories, tar, target, test, transfer, Wednesday, work
Posted in Linux, Linux Backup tools, System Administration, Various | 1 Comment »

Rsync copy files with root privileges between servers with root superuser account disabled

Tuesday, December 3rd, 2019

rsync-copy-files-between-two-servers-with-root-privileges-with-root-superuser-account-disabled

Sometimes on servers that follow high security standards in companies following PCI Security (Payment Card Data Security) standards it is necessery to have a very weird configurations on servers,to be able to do trivial things such as syncing files between servers with root privileges in a weird manners.This is the case for example if due to security policies you have disabled root user logins via ssh server and you still need to synchronize files in directories such as lets say /etc , /usr/local/etc/ /var/ with root:root user and group belongings.

Disabling root user logins in sshd is controlled by a variable in /etc/ssh/sshd_config that on most default Linux OS
installations is switched on, e.g.

grep -i permitrootlogin /etc/ssh/sshd_config
PermitRootLogin yes

Many corporations use Vulnerability Scanners such as Qualys are always having in their list of remote server scan for SSH Port 22 to turn have the PermitRootLogin stopped with:

PermitRootLogin no

In this article, I'll explain a scenario where we have synchronization between 2 or more servers Server A / Server B, whatever number of servers that have already turned off this value, but still need to
synchronize traditionally owned and allowed to write directories only by root superuser, here is 4 easy steps to acheive it.

1. Add rsyncuser to Source Server (Server A) and Destination (Server B)

a. Execute on Src Host:

groupadd rsyncuser
useradd -g 1000 -c 'Rsync user to sync files as root src_host' -d /home/rsyncuser -m rsyncuser

b. Execute on Dst Host:

groupadd rsyncuser
useradd -g 1000 -c 'Rsync user to sync files dst_host' -d /home/rsyncuser -m rsyncuser

2. Generate RSA SSH Key pair to be used for passwordless authentication

a. On Src Host

su – rsyncuser

ssh-keygen -t rsa -b 4096

b. Check .ssh/ generated key pairs and make sure the directory content look like.

[rsyncuser@src-host .ssh]$ cd ~/.ssh/; ls -1

id_rsa
id_rsa.pub
known_hosts

3. Copy id_rsa.pub to Destination host server under authorized_keys

scp ~/.ssh/id_rsa.pub rsyncuser@dst-host:~/.ssh/authorized_keys

Next fix permissions of authorized_keys file for rsyncuser as anyone who have access to that file (that exists as a user account) on the system
could steal the key and use it to run rsync commands and overwrite remotely files, like overwrite /etc/passwd /etc/shadow files with his custom crafted credentials
and hence hack you 🙂

Hence, On Destionation Host Server B fix permissions with:

su – rsyncuser; chmod 0600 ~/.ssh/authorized_keys
[rsyncuser@dst-host ~]$

For improved security here to restrict rsyncuser to be able to run only specific command such as very specific script instead of being able to run any command it is good to use little known command= option
once creating the authorized_keys

4. Test ssh passwordless authentication works correctly

For that Run as a normal ssh from rsyncuser

On Src Host

[rsyncuser@src-host ~]$ ssh rsyncuser@dst-host

Perhaps here is time that for those who, think enabling a passwordless authentication is not enough secure and prefer to authorize rsyncuser via a password red from a secured file take a look in my prior article how to login to remote server with password provided from command line as a script argument / Running same commands on many servers

5. Enable rsync in sudoers to be able to execute as root superuser (copy files as root)

For this step you will need to have sudo package installed on the Linux server.

Then, Execute once logged in as root on Destionation Server (Server B)

[root@dst-host ~]# grep 'rsyncuser ALL' /etc/sudoers|wc -l || echo ‘rsyncuser ALL=NOPASSWD:/usr/bin/rsync’ >> /etc/sudoers

Note that using rsync with a ALL=NOPASSWD in /etc/sudoers could pose a high security risk for the system as anyone authorized to run as rsyncuser is able to overwrite and
respectivle nullify important files on Destionation Host Server B and hence easily mess the system, even shell script bugs could produce a mess, thus perhaps a better solution to the problem
to copy files with root privileges with the root account disabled is to rsync as normal user somewhere on Dst_host and use some kind of additional script running on Dst_host via lets say cron job and
will copy gently files on selective basis.

Perhaps, even a better solution would be if instead of granting ALL=NOPASSWD:/usr/bin/rsync in /etc/sudoers is to do ALL=NOPASSWD:/usr/local/bin/some_copy_script.sh
that will get triggered, once the files are copied with a regular rsyncuser acct.

6. Test rsync passwordless authentication copy with superuser works

Do some simple copy, lets say copy files on Encrypted tunnel configurations located under some directory in /etc/stunnel on Server A to /etc/stunnel on Server B

The general command to test is like so:

rsync -aPz -e 'ssh' '–rsync-path=sudo rsync' /var/log rsyncuser@$dst_host:/root/tmp/

This will copy /var/log files to /root/tmp, you will get a success messages for the copy and the files will be at destination folder if succesful.

On Src_Host run:

[rsyncuser@src-host ~]$ dst=FQDN-DST-HOST; user=rsyncuser; src_dir=/etc/stunnel; dst_dir=/root/tmp; rsync -aP -e 'ssh' '–rsync-path=sudo rsync' $src_dir $rsyncuser@$dst:$dst_dir;

7. Copying files with root credentials via script

The simlest file to use to copy a bunch of predefined files is best to be handled by some shell script, the most simple version of it, could look something like this.

#!/bin/bash
# On server1 use something like this
# On server2 dst server
# add in /etc/sudoers
# rsyncuser ALL=NOPASSWD:/usr/bin/rsync
user='rsyncuser';
dst_dir="/root/tmp";
dst_host='$dst_host';
src[1]="/etc/hosts.deny";
src[2]="/etc/sysctl.conf";
src[3]="/etc/samhainrc";
src[4]="/etc/pki/tls/";
src[5]="/usr/local/bin/";

for i in $(echo ${src[@]}); do
rsync -aPvz –delete –dry-run -e 'ssh' '–rsync-path=sudo rsync' "$i" $rsyncuser@$dst_host:$dst_dir"$i";
done

In above script as you can see, we define a bunch of files that will be copied in bash array and then run a loop to take each of them and copy to testination dir.
A very sample version of the script rsync_with_superuser-while-root_account_prohibited.sh

Conclusion

Lets do short overview on what we have done here. First Created rsyncuser on SRC Server A and DST Server B, set up the key pair on both copied the keys to make passwordless login possible,
set-up rsync to be able to write as root on Dst_Host / testing all the setup and pinpointing a small script that can be used as a backbone to develop something more complex
to sync backups or keep system configurations identicatial – for example if you have doubts that some user might by mistake change a config etc.
In short it was pointed the security downsides of using rsync NOPASSWD via /etc/sudoers and few ideas given that could be used to work on if you target even higher
PCI standards.

Tags: bin, command, copy files, copying, Dst Host, enable, execute, logins, make, password, passwordless, possible, root, root privileges, rsync, running, security policies, servers, shell script, Short, Src Host, ssh server, superuser, system, test, var, without
Posted in Linux, Networking, System Administration | 1 Comment »

Install simscan on Qmail for better Mail server performance and get around unexisting suid perl in newer Linux Debian / Ubuntu servers

Tuesday, August 18th, 2015

qmail-fixing-clamdscan-errors-and-qq-errors-qmail-binary-migration-few-things-to-check-out clamav_logo-installing-clamav-antivirus-to-scan-periodically-debian-server-websites-for-viruses

I've been stuck with qmail-scanner-queue for a while on each and every new Qmail Mail server installation, I've done, this time it was not different but as time evolves and Qmail and Qmail Scanner Wrapper are not regularly updated it is getting, harder and harder to make a fully functional Qmail on newer Linux server distribution releases.

I know many would argue QMAIL is already obsolete but still I have plenty of old servers running QMAIL whose migration might cause more troubles than just continuing to use QMAIL. Moreover QMAIL once set-upped works like a charm.

I've been recently experiencing severe issues with clamdscan errors and I tried to work around this with compiling and using a suid wrapper, however still the clamdscan errors continued and as qmail-scanner is not actively developed and it is much slower than simscan, I've finally decided to give simscan as a mean to fix the clamdscan errors and thanksfully this worked as a solution.

Here is what I did "rawly" to make simscan work on this install:

Make sure simscan is properly installed on Debian Linux 7 or Ubuntu servers and probably (should work) on other Deb based Linuxes by following below steps:

a) Configure simscan with following compile time options as root (superuser)

./configure \
–enable-user=qscand \
–enable-clamav \
–enable-clamdscan=/usr/local/bin/clamdscan \
–enable-custom-smtp-reject=y \
–enable-per-domain=y \
–enable-attach=y \
–enable-dropmsg=n \
–enable-spam=y \
–enable-spam-hits=5 \
–enable-spam-passthru=y \
–enable-qmail-queue=/var/qmail/bin/qmail-queue \
–enable-ripmime=/usr/local/bin/ripmime \
–enable-sigtool-path=/usr/local/bin/sigtool \
–enable-received=y

b) Compile it

make && make install-strip

c) Fix any wrong permissions of simscan queue directory

chmod g+s /var/qmail/simscan/
chown -R qscand:qscand /var/qmail/simscan/
chmod -R 777 simscan/chown -R qscand:qscand simscan/
chown -R qscand:qscand simscan/

d) Add some additional simscan options (how simscan is how to perform scans)

The restart qmail to make mailserver start using simscan instead of qmail-scanner, run below command (again as root):

echo ":clam=yes,spam=yes,spam_hits=8.5,attach=.vbs:.lnk:.scr:.wsh:.hta:.pif" > /var/qmail/control/simcontrol

e) Run /var/qmail/bin/simscanmk in order to convert /var/qmail/control/simcontrol into the /var/qmail/control/simcontrol.cdb database

/var/qmail/bin/simscanmk
/var/qmail/bin/simscanmk -g

f) Modify /service/qmail-smtpd/run to set simscan to be default Antivirus Wrapper Scanner

vim /service/qmail-smtpd/run

I'm using thibs's run script so I've uncommented the line there:

QMAILQUEUE="$VQ/bin/simscan"

Below two lines should stay commented as qmail-scanner is no longer used:

##QMAILQUEUE="$VQ/bin/qmail-scanner-queue"
##QMAILQUEUE="$VQ/bin/qmail-scanner-queue.pl"
export QMAILQUEUE

qmailctl restart
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.

g) Test whether simscan is properly sending / receiving emails:

echo "Testing Email" >> /tmp/mailtest.txt
env QMAILQUEUE=/var/qmail/bin/simscan SIMSCAN_DEBUG=3 /var/qmail/bin/qmail-inject hipo@my-mailserver.com < /tmp/mailtest.txt

Besides that as I'm using qscand:qscand as a user for my overall Qmail Thibs install I had to also do:

chown -R qscand:qscand /var/qmail/simscan/
chmod -R 777 simscan/
chown -R qscand:qscand simscan/

It might be a good idea to also place that lines in /etc/rc.local to auto change permissions on Linux boot, just in case something wents wrong with permissions.

Yeah, I know 777 is unsecure but without this permissions, I was still getting errors, plus the server doesn't have any accounts except the administrator, so I do not worry other system users might sniff on email 🙂

h) Test whether Qmail mail server send / receives fine with simscan

After that I've used another mail server with mail command to test whether mail is received:

mail -s "testing email1234" hipo@new-configured-qmail-server.com
asdfadsf
.
Cc:

Then it is necessery to also install latest clamav daemon from source in my case that's on Debian GNU / Linux 7, because somehow the Debian shipped binary version of clamav 0.98.5+dfsg-0+deb7u2 does fail to scan any incoming or outgoing email with error:

clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935

So to fix it you will have to install clamav on Debian Linux from source.

Voilla, that's all finally it worked !

Tags: bin, binary version, installation, Linux Debian Ubuntu, mail command, make, performance, QMAILQUEUE, script, simscan, Stopping, system users, test, Testing Email, time, var, work
Posted in Everyday Life, Linux, Qmail, Remote System Administration, System Administration | 3 Comments »

Check your Server Download / Upload Internet Speed from Console on Linux / BSD / Unix howto

Tuesday, March 17th, 2015

If you've been given a new dedicated server from a New Dedicated-Server-Provider or VPS with Linux and you were told that a certain download speed to the Server is guaranteed from the server provider, in order to be sure the server's connection to the Internet told by service provider is correct it is useful to run a simple measurement console test after logging in remotely to the server via SSH.
Testing connection from Terminal is useful because as you probably know most of Linux / UNIX servers doesn't have a GUI interface and thus it is not possible to test Internet Up / Down Bandwidth through speedtest.net.

1. Testing Download Internet Speed given by ISP / Dedi-Server Provider from Linux Console

For the download speed (internet) test the historical approach was to just try downloading the Linux kernel source code from www.kernel.org with some text browser such as lynx or links count the seconds for which the download is completed and then multiple the kernel source archive size on the seconds to get an approximate bandwidth per second, however as nowdays internet connection speeds are much higher, thus it is better to try to download some Linux distribution iso file, you can still use kernel tar archive but it completed too fast to give you some good (adequate) statistics on Download bandwidth.

If its a fresh installed Linux server probably you will probably not have links / elinks and lynx text internet browers installed so install them depending on deb / rpm distro with:

If on Deb Linuz distro:

root@pcfreak:/root# apt-get install –yes links elinks lynx
…

On RPM Based Linuz distro:

[root@fedora ~]# yum install -y lynx elinks links
…

Conduct Internet Download Speed with links
root@pcfreak:/root# links https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.19.1.tar.xz

(Note that the kernel link is current latest stable Kernel source code archive in future that might change, so try with latest archive.)

You can also use non-interactive tool such as wget, curl or lftp to measure internet download speed

To test Download Internet Speed with wget without saving anything to disk set output to go to /dev/null

root@pcfreak:~# wget -O /dev/null https://pc-freak.net//~hipo/hirens-bootcd/HirensBootCD15/Hirens.BootCD.15.0.zip

You see the Download speed is 104 Mbit/s this is so because I'm conducting the download from my local 100Mbit network.

For the test you can use my mirrored version of Hirens BootCD

2. Testing Uplink Internet speed provided by ISP / Server Provider from Linux (SSH) Console

To test your uplink speed you will need lftp or iperf command tool.

root@pcfreak:~# apt-cache show lftp|grep -i descr -A 12
Description: Sophisticated command-line FTP/HTTP client programs
Lftp is a file retrieving tool that supports FTP, HTTP, FISH, SFTP, HTTPS
and FTPS protocols under both IPv4 and IPv6. Lftp has an amazing set of
features, while preserving its interface as simple and easy as possible.
.
The main two advantages over other ftp clients are reliability and ability
to perform tasks in background. It will reconnect and reget the file being
transferred if the connection broke. You can start a transfer in background
and continue browsing on the ftp site. It does this all in one process. When
you have started background jobs and feel you are done, you can just exit
lftp and it automatically moves to nohup mode and completes the transfers.
It has also such nice features as reput and mirror. It can also download a
file as soon as possible by using several connections at the same time.

root@pcfreak:/root# apt-cache show iperf|grep -i desc -A 2
Description: Internet Protocol bandwidth measuring tool
Iperf is a modern alternative for measuring TCP and UDP bandwidth performance,
allowing the tuning of various parameters and characteristics.

To test Upload Speed to Internet connect remotely and upload any FTP file:

root@pcfreak:/root# lftp -u hipo pc-freak.net -e 'put Hirens.BootCD.15.0.zip; bye'

On Debian Linux to install iperf:

root@pcfreak:/root# apt-get install –yes iperf

On latest CentOS 7 and Fedora (and other RPM based) Linux, you will need to add RPMForge repository and install with yum

[root@centos ~]# rpm -ivh rpmforge-release-0.5.3-1.el7.rf.x86_64.rpm
…
[root@centos ~]# yum -y install iperf
…

Once having iperf on the server the easiest way currently to test it is to use
serverius.net speedtest server – located at the Serverius datacenters, AS50673 and is running on a 10GE connection with 5GB cap.

root@pcfreak:/root# iperf -c speedtest.serverius.net -P 10
————————————————————
Client connecting to speedtest.serverius.net, TCP port 5001
TCP window size: 16.0 KByte (default)
————————————————————
[ 12] local 83.228.93.76 port 54258 connected with 178.21.16.76 port 5001
[ 7] local 83.228.93.76 port 54252 connected with 178.21.16.76 port 5001
[ 5] local 83.228.93.76 port 54253 connected with 178.21.16.76 port 5001
[ 9] local 83.228.93.76 port 54251 connected with 178.21.16.76 port 5001
[ 3] local 83.228.93.76 port 54249 connected with 178.21.16.76 port 5001
[ 4] local 83.228.93.76 port 54250 connected with 178.21.16.76 port 5001
[ 10] local 83.228.93.76 port 54254 connected with 178.21.16.76 port 5001
[ 11] local 83.228.93.76 port 54255 connected with 178.21.16.76 port 5001
[ 6] local 83.228.93.76 port 54256 connected with 178.21.16.76 port 5001
[ 8] local 83.228.93.76 port 54257 connected with 178.21.16.76 port 5001
[ ID] Interval Transfer Bandwidth
[ 9] 0.0-10.2 sec 4.05 MBytes 3.33 Mbits/sec
[ 10] 0.0-10.2 sec 3.39 MBytes 2.78 Mbits/sec
[ 11] 0.0-10.3 sec 3.75 MBytes 3.06 Mbits/sec
[ 4] 0.0-10.3 sec 3.43 MBytes 2.78 Mbits/sec
[ 12] 0.0-10.3 sec 3.92 MBytes 3.18 Mbits/sec
[ 3] 0.0-10.4 sec 4.45 MBytes 3.58 Mbits/sec
[ 5] 0.0-10.5 sec 4.06 MBytes 3.24 Mbits/sec
[ 6] 0.0-10.5 sec 4.30 MBytes 3.42 Mbits/sec
[ 8] 0.0-10.8 sec 3.92 MBytes 3.03 Mbits/sec
[ 7] 0.0-10.9 sec 4.03 MBytes 3.11 Mbits/sec
[SUM] 0.0-10.9 sec 39.3 MBytes 30.3 Mbits/sec

You see currently my home machine has an Uplink of 30.3 Mbit/s per second, that's pretty nice since I've ordered a 100Mbits from my ISP (Unguaranteed Bandwidth Connection Speed) and as you might know it is a standard practice for many Internet Proviers to give Uplink speed of 1/4 from the ISP provided overall bandwidth 1/4 would be 25Mbi/s, meaning my ISP (Bergon.NET) is doing pretty well providing me with even more than promised (ordered) bandwidth.

Iperf is probably the choice of most sysadmins who have to do regular bandwidth in local networks speed between 2 servers or test Internet Bandwidth speed on heterogenous network with Linux / BSDs / AIX / HP-UX (UNIXes). On HP-UX and AIX and other UNIXes for which iperf doesn't have port you have to compile it yourself.

If you don't have root /admin permissions on server and there is python language enterpreter installed you can use speedtest_cli.py script to test internet throughput connectivity
speedtest_cli uses speedtest.net to test server up / down link just in case if script is lost in future I've made ownload mirror of speedtest_cli.py is here

Quickest way to test net speed with speedtest_cli.py:

$ lynx -dump https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py > speedtest_cli.py
$ chmod +x speedtest_cli.py
python speedtest_cli.py

speedtest_cli_pyhon_script_screenshot-on-gnu-linux-test-internet-network-speed-on-unix

Tags: command tool, connection, debian linux, dedicated server, Internet, internet connection, Interval Transfer Bandwidth, lftp, order, pcfreak, root, root admin, run, server provider, servers, simple, speed, ssh, tar archive, test, Testing Download Internet Speed, text browser, use, zip
Posted in Curious Facts, Linux, System Administration, Various, Web and CMS | No Comments »

Create local network between virtual machines in Virtualbox VM – Add local LAN between Linux Virtual Machines

Wednesday, June 11th, 2014

add-virtualbox-virtual-machines-inside-local-network-create-internal-LAN-local-net-linux-windows

I want to do test MySQL Cluster following MySQL Cluster Install Guide for that purpose, I've installed 2 version of CentOS 6.5 inside Virtualbox and I wanted to make the 2 Linux hosts reachable inside a local LAN network, I consulted some colleagues who adviced me to configure two Linux hosts to use Bridget Adapter Virtualbox networking (Network configuration in Virtualbox is done on a Virtual Machine basis from):

Devices -> Network Settings

(Attached to: Bridged Adapter)

Note!: that by default Cable Connected (tick) is not selected so when imposing changes on Network – tick should be set)
After Specifying Attached to be Bridged Adapter to make CentOS linux refresh network settings run in gnome-terminal:

[root@centos ~]# dhclient eth0

However CentOS failed to grab itself DHCP IP address.
Thus I tried to assign manually IP addresses with ifconfig, hoping that at least this would work, e.g.:

on CentOS VM 1:

/sbin/ifconfig eth0 192.168.10.1 netmask 255.255.255.0

on CentOS VM 2:

/sbin/ifconfig eth1 192.168.10.2 netmask 255.255.255.0

To test whether there is connection between the 2 VM hosts tried ping-ing 192.168.10.2 (from 192.168.10.1) and tested with telnet if I can access remotely SSH (protocol), from CentOS VM2 1 to CentOS VM2 and vice versa, i.e.:

[root@centos ~]# telnet 192.168.10.2 22

Trying 192.168.10.2…
telnet: connect to address 192.168.10.2: No route to host

Then after checking other options and already knowing by using VBox NAT network option I had access to the internet, I tried to attach a standard local IP addresses to both Linux-es as Virtual interfaces (e.g eth0:1), .e.g:

On Linux VM 1:

/sbin/ifconfig eth0:0 192.168.10.1 netmask 255.255.255.0

On Linux VM 2:

/sbin/ifconfig eth1:0 192.168.10.2 netmask 255.255.255.0

Then to test again used telnet

[root@centos ~]# telnet 192.168.10.2 22
…

Then I found Virtualbox has a special Internal Networking support

to choose in Attached to drop down menu. According to Internal Networking Virtualbox instructions to put two Virtual Machine hosts inside an Internal network they should be both set in Internal network with identical name.
P. S. It is explicitly stated that using Internal Network will enable access between Guest Virtual Machines OS, but hosts will not have access to the Internet (which in my case doesn't really mattered as I needed the two Linux VMs just as a testbed)

virtualbox-create-internal-local-network-between-guest-host-Linux-Windows1

I tried this option but it doesn't work for me for some reason, after some time of research online on how to create local LAN network between 2 Virtual Machines luckily I decided to test all available Virtualbox Networking choices and noticed Host-only adapter.

Selecting Host-only Adapter and using terminal to re-fetch IP address over dhcp:

virtualbox-connect-in-local-lan-network-linux-and-windows-servers-hosts-only-adapter

On CentOS VM1

dhclient eht0

On CentOS VM2

dhclient eth1

assigned me two adjoining IPs – (192.168.56.101 and 192.168.56.102).

Connection between the 2 IPs 192.168.56.101 and 192.168.56.102 on TCP and UDP and ICMP protocol works, now all left is to install MySQL cluster on both nodes.

Tags: connection, dhclient, doesn, ifconfig eth0, Internal Networking Virtualbox, ip, lan, Linux Virtual Machines, network, ping, sbin, test
Posted in Everyday Life, System Administration, Various, Virtual Machines | 1 Comment »

Best Windows tools to Test (Benchmark) Hard Drives, SSD Drives and RAID Storage Controllers

Wednesday, April 23rd, 2014

atto-windows-hard-disk-benchmark-freeware-tool-screenshot-check-hard-disk-speed-windows
Disk Benchmarking is very useful for people involved in Graphic Design, 3D modelling, system admins and anyone willing to squeeze maximum of his PC hardware.

If you want to do some benchmarking on newly built Windows server targetting Hard Disk performance, just bought a new hard SSD (Solid State Drives) and you want to test how well Hard Drive I/O operations behave or you want to see a regular HDD benchmarking of group of MS Windows PCs and plan hardware optiomization, check out ATTO Disk Benchmark.

So why exactly ATTO Benchmark? – Cause it is one of the best Windows Free Benchmark tools on the internet.

ATTO is a widely-accepted Disk Benchmark freeware utility to help measure storage system performance. ATTO though being freeware is among top tools utilized in industry. It is very useful in comparing different Hard Disk vendors speed, measure Windows storage systems performance with various transfer sizes and test lengths for reads and writes.

ATTO Disk Benchmark is used by manufacturers of Hardware RAID controllers, its precious tool to test Windows storage controllers, host bus adapters (HBAs).

Here is ATTO Benchmark tool specifications (quote from their webstie):

Transfer sizes from 512KB to 8MB
Transfer lengths from 64KB to 2GB
Support for overlapped I/O
Supports a variety of queue depths
I/O comparisons with various test patterns
Timed mode allows continuous testing
Non-destructive performance measurement on formatted drives

Transfer sizes from 512KB to 8MB
Transfer lengths from 64KB to 2GB
Support for overlapped I/O
Supports a variety of queue depths
I/O comparisons with various test patterns
Timed mode allows continuous testing
Non-destructive performance measurement on formatted drives
– See more at: http://www.attotech.com/disk-benchmark/#sthash.rRlgSTOE.dpuf

Here is mirrored latest version of ATTO Disk for Download. Once you get your HDD statistics you will probably want to compare to other people results. On TomsHardware's world famous Hardware geek site there are plenty of Hard Drives performance Charts

Of course there are other GUI alternatives to ATTO Benchmark one historically famous is NBench

NBench

nbench_benchmark_windows_hard-drive-cpu-and-memory

Nbench is nice little benchmarking program for Windows NT. Nbench reports the following components of performance:

CPU speed: integer and floating operations/sec
L1 and L2 cache speeds: MB/sec
main memory speed: MB/sec
disk read and write speeds: MB/sec

SMP systems and multi-tasking OS efficiency can be tested using up to 20 separate threads of execution.

For Console Geeks or Windows server admins there are also some ports of famous *NIX Hard Disk Benchmarking tools:

NTiogen

NTiogen benchmark was written by Symbios Logic, It's Windows NT port of their popular UNIX benchmark IOGEN. NTIOGEN is the parent processes that spawns the specified number of IOGEN processes that actually do the I/O.
The program will display as output the number of processes, the average response time, the number of I/O operations per second, and the number of KBytes per second. You can download mirror copy of Ntiogen here

There are plenty of other GUI and Console HDD Benchmarking Win Tools, i.e.:

IOMeter (ex-developed by Intel and now abandoned available as open source available on SourceForge)

iometer-benchmark-disk-storage-speed-windows

Bench32 – Comprehensive benchmark that measures overall system performance under Windows NT or Windows 95, now obsolete not developed anymore abandoned by producer company.

ThreadMark32 – capable of bench (ex developed and supported by ADAPTEC) but also already unsupported

IOZone – filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems.

N! B! Important note to make here is above suggested tools will provide you more realistic results than the proprietary vendor tools shipped by your hardware vendor. Using proprietary software produced by a single vendor makes it impossible to analyze and compare different hardwares, above HDD benchmarking tools are for "open systems", e.g. nomatter what the hardware producer is produced results can be checked against each other.
Another thing to consider is even though if you use any of above tools to test and compare two storage devices still results will be partially imaginary, its always best to conduct tests in Real Working Application Environments. If you're planning to launch a new services structure always test it first and don't rely on preliminary returned soft benchmarks.

if you know some other useful benchmarking software i'm missing please share.

Tags: ADAPTEC, ATTO, benchmarking, check, Disk Benchmark, Disk Benchmarking, Flash, freeware, hard disk, hardware, hardware vendor, hdd, host, industry, IOGEN, number, performance, proprietary software, queue, software, speed, storage, Supports, system, test, testing, threads, transfer, utility, Windows
Posted in Everyday Life, System Administration, Windows | No Comments »

Top Paying Google Adsense Keywords for 2013 – Increase blog earnings with high paying keywords

Wednesday, July 31st, 2013

Most-High-Paying-Google-AdSense-Keywords-for-year-2013-Increase-your-revenues-from-Google-with-these-keywords

Whether you're a blogger and you're trying to earn some extra cash for your daily living via blogging you already know how hard it is. It is not hard it is almost impossible to earn enough from Blog or personal website to make it your primary source of income. With the Crisis the CPC (Cost Per Click) rate of Advertisements and the number of people willing to pay big money for Cost per click dropped down drastically … This means also our earnings as bloggers decreased badly … However there is still a bit of hope to boost a bit Google Adsense Paying revenues, by trying to write articles including certain high payed keywords. Not surprisingly this high pay keywords vary seriously over years. You might be shocked to know that there are some keywords for which Google pays over 100$ per CLICK!!! OMG 100$ per click this sounds unrealistic but according to some rumors online its a fact. I've found on the internet a list of 70 keywords said to make your blog money per click from 179 dollars (highest clickable advertiser pay fee) to 51 bucks at minimum. Actually the reason to write this post was to test if this rumors with so high CPC are true. I will update the post later to tell you whether really top words work or its jus ta big fraud. Here is list of 70 top earning Google Adsense revenue keywords for 2013

Keywords	CPC
MESOTHELIOMA LAW FIRM	$179.01
DONATE CAR TO CHARITY CALIFORNIA	$130.25
DONATE CAR FOR TAX CREDIT	$126.65
DONATE CARS IN MA	$125.58
DONATE YOUR CAR SACRAMENTO	$118.20
HOW TO DONATE A CAR IN CALIFORNIA	$111.21
SELL ANNUITY PAYMENT	$107.46
DONATE YOUR CAR FOR KIDS	$106.01
ASBESTOS LAWYERS	$105.84
STRUCTURED ANNUITY SETTLEMENT	$100.8
ANNUITY SETTLEMENTS	$100.72
CAR INSURANCE QUOTES COLORADO	$100.93
NUNAVUT CULTURE	$99.52
DAYTON FREIGHT LINES	$99.39
HARDDRIVE DATA RECOVERY SERVICES	$98.59
DONATE A CAR IN MARYLAND	$98.51
MOTOR REPLACEMENTS	$98.43
CHEAP DOMAIN REGISTRATION HOSTING	$98.39
DONATING A CAR IN MARYLAND	$98.20
DONATE CARS ILLINOIS	$98.13
CRIMINAL DEFENSE ATTORNEYS FLORIDA	$98.07
BEST CRIMINAL LAWYER IN ARIZONA	$97.93
CAR INSURANCE QUOTES UTAH	$97.92
LIFE INSURANCE CO LINCOLN	$97.07
HOLLAND MICHIGAN COLLEGE	$95.74
ONLINE MOTOR INSURANCE QUOTES	$95.73
ONLINE COLLEDGES	$95.65
PAPERPORT PROMOTIONAL CODE	$95.13
ONLINECLASSES	$95.06
WORLD TRADE CENTER FOOTAGE	$95.02
MASSAGE SCHOOL DALLAS TEXAS	$94.90
PSYCHIC FOR FREE	$94.61
DONATE OLD CARS TO CHARITY	$94.55
LOW CREDIT LINE CREDIT CARDS	$94.49
DALLAS MESOTHELIOMA ATTORNEYS	$94.33
CAR INSURANCE QUOTES MN	$94.29
DONATE YOUR CAR FOR MONEY	$94.01
CHEAP AUTO INSURANCE IN VA	$93.84
MET AUTO	$93.70
FORENSICS ONLINE COURSE	$93.51
HOME PHONE INTERNET BUNDLE	$93.32
DONATING USED CARS TO CHARITY	$93.17
PHD IN COUNSELING EDUCATION	$92.99
NEUSON	$92.89
CAR INSURANCE QUOTES PA	$92.88
ROYALTY FREE IMAGES STOCK	$92.76
CAR INSURANCE IN SOUTH DAKOTA	$92.72
EMAIL BULK SERVICE	$92.55
WEBEX COSTS	$92.38
CHEAP CAR INSURANCE FOR LADIES	$92.23
CHEAP CAR INSURANCE IN VIRGINIA	$92.03
REGISTER FREE DOMAINS	$92.03
BETTER CONFERENCING CALLS	$91.44
FUTURISTIC ARCHITECTURE	$91.44
MORTGAGE ADVISER	$91.29
CAR DONATE	$88.26
VIRTUAL DATA ROOMS	$83.18
AUTOMOBILE ACCIDENT ATTORNEY	$76.57
AUTO ACCIDENT ATTORNEY	$75.64
CAR ACCIDENT LAWYERS	$75.17
DATA RECOVERY RAID	$73.22
MOTOR INSURANCE QUOTES	$68.61
PERSONAL INJURY LAWYER	$66.53
CAR INSURANCE QUOTES	$61.03
ASBESTOS LUNG CANCER	$60.96
INJURY LAWYERS	$60.79
PERSONAL INJURY LAW FIRM	$60.56
ONLINE CRIMINAL JUSTICE DEGREE	$60.4
CAR INSURANCE COMPANIES	$58.66
BUSINESS VOIP SOLUTIONS	$51.9

Tags: bloggers, blogging, boost, list, make, money, post, primary, test, Top Paying Google Adsense Keywords
Posted in Curious Facts, Economy | 2 Comments »

Fixing QMAIL mail server SMTP auto-configure issues in Thunderbird and other mail IMAP / POP3 mobile clients

Friday, July 13th, 2012

One of the QMAIL mail servers, setup-uped on a Debian host has been creating some auto configuration issues. Every-time a new mail user tries to use the embedded Thunderbird client auto configuration, the auto config fails leaving the client unable to use his Mailbox through POP3 or IMAP protocols.

Since about 2 years Thunderbird and many other modern pop3 and imap mail desktop and mobile clients are by default using the auto configuration and hence it was unthinkable to manually change settings for new clients with the QMAIl install; Besides that most of the Office users are always confused, whether they have to manually change SMTP or POP3 host for a server.

Below is a screenshot displaying the warning during email auto-configuration:

The orange color in the button for the newly auto-detected smtp.mail-domain.com indicates, something is not right with the SMTP host.

Obviously, something was wrong with smtp.mail-domain.com, hence I checked where smtp.mail.domain.com resolves with host command. What I found was actually smtp.mail-domain.com Active ( A ) DNS records was pointing to an IP address, our company previously used for the mail server. At present time the correct mail server host name is mx.mail-domain.com and the QMAIL installation on mx.soccerfame.com is configured to be the actual SMTP server.

By default Thunderbird and many other POP3, IMAP mail clients, however automatically assume the default SMTP host for a mail server is to be configured under a host name smtp.mail-domain.com. This is really strange, especially when the primary MX record for mail-domain.com domain is pointing to mx.mail-domain.com, e.g.:

qmail:~# host -t MX mail-domain.com soccerfame.com mail is handled by 10 mx.mail-domain.com. soccerfame.com mail is handled by 20 mail.mail-domain.com. soccerfame.com mail is handled by 30 mail-domain.com.

The whole warning was caused due to the fact mx.mail-domain.com was resolving to an IP like xxx.xxx.xxx.xxx, whether smtp.mail-domain.com was resolving to yyy.yyy.yyy.yyy

Both xxx.xxx.xxx.xxx and yyy.yyy.yyy.yyy hosts were configured to have a different qmail SMTP host i.e.:

The server under IP xxx.xxx.xxx.xxx – (mx.mail-domain.com) was configured in /var/qmail/control/me to be mx.mail-domain.com and the other old one yyy.yyy.yyy.yyy – (mail.mail-domain.com) had (mail.mail-domain.com) in /var/qmail/control/me

As smtp.mail-domain.com was actually being still resolved to mail.mail-domain.com, the EMAILs were improperly trying to be sent with a configured DNS hostname of smtp.mail-domain.com, where the actual one on the server was mail.mail-domain

It took, me about an hour of pondering what is causing the oddities until I got the here explained issue. As the DNS recors for the domain the sample mail-domain.com were handled by Godaddy, to fix the mess, I logged in to Godaddy and;

a) deleted – DNS record for smtp.mail-domain.com.
b) Created new CNAME record for smtp.mail-domain.com to be a domain alias for mx.soccerfame.com

A few minutes, afterwards I tried configuring once again the same email account in Thunderbird and this time both imap.mail-domain.com and smtp.mail-domain.com turned green; indicating everything is configured fine.

To be 100% sure all is working fine I first fetched, all email via the IMAP protocol without hassles and onwards sent a test email to my Gmail account; thanksfully the sent email was delivered to Gmail indicating both Get Mail and Send Mail functions worked now fine.

Tags: Auto, auto configuration, Button, client, com, comA, comsoccerfame, config, control, correct mail server, Desktop, DNS, dns records, Draft, Emails, everything, host, host command, imap protocols, installation, mail clients, mail domain, mail imap, mail mail, mail servers, mail user, mess, mobile clients, mx mail, mx record, nbsp, new mail, orange, pop3 host, qmail installation, right, screenshot, server host name, server smtp, smtp, smtp host, soccerfame, something, test, time
Posted in Qmail, System Administration | 4 Comments »

Text Monitoring of connection server (traffic RX / TX) business in ASCII graphs with speedometer / Easy Monitor network traffic performance

Friday, May 4th, 2012

While reading some posts online related to MS-Windows TcpView – network traffic analyzing tool. I've came across very nice tool for tracking connection speed for Linux (Speedometer). If I have to compare it, speedometer is somehow similar to nethogs and iftop bandwidth network measuring utilities .

What differentiates speedometer from iftop / nethogs / iptraf is it is more suitable for visualizing a network file or data transfers.
The graphs speedometer draws are way easier to understand, than iftop graphs.

Even complete newbies can understand it with no need for extraordinary knowledge in networking. This makes Speedometer, a top tool to visually see the amount of traffic flowing through server network interface (eth0) … (eth1) etc.

What speedometer shows is similar to the Midnight Commander's (mc) file transfer status bar, except the statistics are not only for a certain file transfer but can show overall statistics over server passing network traffic amount (though according to its manual it can be used to also track individual file transfers).

The simplicity for basic use makes speedometer nice tool to track for network congestion issues on Linux. Therefore it is a must have outfit for every server admin. Below you see a screenshot of my terminal running speedometer on a remote server.

Speedometer ascii traffic track server network business screenshot in byobu screen like virtual terminal emulator

1. Installing speedometer on Debian / Ubuntu and Debian derivatives

For Debian and Ubuntu server administrators speedometer is already packaged as a deb so its installation is as simple as:

debian:~# apt-get --yes install speedometer ....

2. Installing speedometer from source for other Linux distributions CentOS, Fedora, SuSE etc.

Speedometer is written in python programming language, so in order to install and use on other OS Linux platforms, it is necessery to have installed (preferably) an up2date python programming language interpreter (python ver. 2.6 or higher)..
Besides that it is necessary to have installed the urwid -( console user interface library for Python) available for download via excess.org/urwid/

Hence to install speedometer on RedHat based Linux distributions one has to follow these steps:

a) Download & Install python urwid library

[root@centos ~]# cd /usr/local/src [root@centos src]# wget -q http://excess.org/urwid/urwid-1.0.1.tar.gz [root@centos src]# tar -zxvvf urwid-1.0.1.tar.gz .... [root@centos src]# cd urwid-1.0.1 [root@centos urwid-1.0.1]# python setup.py install running install running build running build_py creating build creating build/lib.linux-i686-2.4 creating build/lib.linux-i686-2.4/urwid copying urwid/tests.py -> build/lib.linux-i686-2.4/urwid copying urwid/command_map.py -> build/lib.linux-i686-2.4/urwid copying urwid/graphics.py -> build/lib.linux-i686-2.4/urwid copying urwid/vterm_test.py -> build/lib.linux-i686-2.4/urwid copying urwid/curses_display.py -> build/lib.linux-i686-2.4/urwid copying urwid/display_common.py -> build/lib.linux-i686-2.4/urwid ....

b) Download and install python-setuptools

python-setuptools is one other requirement of speedometer, happily on CentOS and Fedora the rpm package is already there and installable with yum:

[root@centos ~]# yum -y install python-setuptools ....

c) Download and install Speedometer

[root@centos urwid-1.0.1]# cd /usr/local/src/ [root@centos src]# wget -q http://excess.org/speedometer/speedometer-2.8.tar.gz [root@centos src]# tar -zxvvf speedometer-2.8.tar.gz ..... [root@centos src]# cd speedometer-2.8 [root@centos speedometer-2.8]# python setup.py install Traceback (most recent call last): File "setup.py", line 26, in ? import speedometer File "/usr/local/src/speedometer-2.8/speedometer.py", line 112 n = n * granularity + (granularity if r else 0) ^

While running the CentOS 5.6 installation of speedometer-2.8, I hit the
"n = n * granularity + (granularity if r else 0)"
error.

After consultation with some people in #python (irc.freenode.net), I've figured out this error is caused due the outdated version of python interpreter installed by default on CentOS Linux 5.6. On CentOS 5.6 the python version is:

[root@centos ~]# python -V Python 2.4.3

As I priorly said speedometer 2.8's minimum requirement for a python to be at v. 2.6. Happily there is quick way to update python 2.4 to python 2.6 on CentOS 5.6, as there is an RPM repository maintained by Chris Lea which contains RPM binary of python 2.6.

To update python 2.4 to python 2.6:

[root@centos speedometer-2.8]# rpm -Uvh http://yum.chrislea.com/centos/5/i386/chl-release-5-3.noarch.rpm[root@centos speedometer-2.8]# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CHL[root@centos speedometer-2.8]# yum install python26

Now the newly installed python 2.6 is executable under the binary name python26, hence to install speedometer:

[root@centos speedometer-2.8]# python26 setup.py install [root@centos speedometer-2.8]# chown root:root /usr/local/bin/speedometer [root@centos speedometer-2.8]# chmod +x /usr/local/bin/speedometer

[root@centos speedometer-2.8]# python26 speedometer -i 1 -tx eth0

The -i will instruct speedometer to refresh the screen graphs once a second.

3. Using speedometer to keep an eye on send / received traffic network congestion

To observe, the amount of only sent traffic via a network interface eth0 with speedometer use:

debian:~# speedometer -tx eth0

To only keep an eye on received traffic through eth0 use:

debian:~# speedometer -rx eth0

To watch over both TX and RX (Transmitted and Received) network traffic:

debian:~# speedometer -tx eth0 -rx eth0

If you want to watch in separate windows TX and RX traffic while running speedometer you can run in separate xterm windows speedometer -tx eth0 and speedometer -rx eth0, like in below screenshot:

Monitor Received and Transmitted server Network traffic in two separate xterm windows with speedometer ascii graphs

4. Using speedometer to test network maximum possible transfer speed between server (host A) and server (host B)

The speedometer manual suggests few examples one of which is:

How fast is this LAN?

host-a$ cat /dev/zero | nc -l -p 12345
host-b$ nc host-a 12345 > /dev/null
host-b$ speedometer -rx eth0

When I red this example in speedometer's manual, it wasn't completely clear to me what the author really meant, but a bit after when I thought over the example I got his point.

The idea behind this example is that a constant stream of zeros taken from /dev/zero will be streamed over via a pipe (|) to nc which will bind a port number 12345, anyone connecting from another host machine, lets say a server with host host-b to port 12345 on machine host-a will start receiving the /dev/zero streamed content.

Then to finally measure the streamed traffic between host-a and host-b machines a speedometer is started to visualize the received traffic on network interface eth0, thus measuring the amount of traffic flowing from host-a to host-b …

I give a try to the exmpls, using for 2 test nodes my home Desktop PC, Linux running arcane version of Ubuntu and my Debian Linux notebook.

First on the Ubuntu PC I issued

hipo@hip0-desktop:~$ cat /dev/zero | nc -l -p 12345

Note that I have previously had installed the netcat, as nc is not installed by default on Ubuntu and Debian. If you, don't have nc installed yet, install it with:

apt-get –yes install netcat

"cat /dev/zero | nc -l -p 12345" will not produce any output, but will display just a blank line.

Then on my notebook I ran the second command example, given in the speedometer manual:

hipo@noah:~$ nc 192.168.0.2 12345 > /dev/null

Here the 192.168.0.2 is actually the local network IP address of my Desktop PC. My Desktop PC is connected via a normal 100Mbit switch to my routing machine and receives its internet via NAT. The second test machine (my laptop), gets its internet through a WI-FI connection received by a Wireless Router connected via a UTP cable to the same switch to which my Desktop PC is connected.

Finally to test / get my network maximum thoroughput I had to use:

hipo@noah:~$ speedometer -rx wlan0

Here, I monitor my wlan0 interface, as this is my (laptop) wireless card interface over which I have connectivity to my local network and via which through the the WI-FI router I get connected to the internet.

Below is a snapshot captured showing approximately what is the max network thoroughput from:

Desktop PC -> to my Thinkpad R61 laptop

Using Speedometer to test network thorougput between two network server hosts screenshot Debian Squeeze Linux

As you can see in the shot approximately the maximum network thoroughput is in between:
2.55MB/s min and 2.59MB/S max, the speed is quite low for a 100 MBit local network, but this is normal as most laptop wireless adapters hardly transfer traffic in more than 10 to 20 MBits per sec.

If the same nework thoroughput test is conducted between two machines both connected to a same 100 M/bit switch, the traffic should be at least a 8 MB/sec.

There is something, else to take in consideration that probably makes the provided example network thoroughput measuring a bit inaccurate. The fact that the /dev/zero content is stremed over is slowing down the zeroes sent over network because of the pipe ( | ) use slows down the stream.

5. Using speedometer to visualize maximum writting speed to a local hard drive on Linux

In the speedometer manual, I've noticed another interesting application of this nifty tool.

speedometer can be used to track and visualize the maximum writing speed a hard disk drive or hard drive partition can support on Linux OS:

A copy paster from the manual text is as follows:

How fast can I write data to my filesystem? (with at least 1GB free) dd bs=1000000 count=1000 if=/dev/zero of=bigfile & speedometer bigfile

However, when I tried copy/pasting the example in terminal, to test the maximum writing speed to an external USB hard drive, only dd command was started and speedometer failed to initialize and display graphs of the file creation speed.

I've found a little "hack" that makes the man example work by adding a 3 secs sleep like so:

debian:/media/Expansion Drive# dd bs=1000000 count=1000 if=/dev/zero of=bigfile & sleep 3; speedometer bigfile

Here is a screenshot of the bigfile created by dd and tracked "in real time" by speedometer:

How fast is writting data to local USB expandable hard disk Debian Linux speedometer screenshot

Actually the returned results from this external USB drive are, quite high, the possible reason for that is it is connected to my laptop over an USB protocol verion 3.

6. Using Speedometer to keep an eye on file download in progress

This application of speedometer is mostly useless especially on Linux where it is used as a Desktop.

However in some occasions if files are transferred over ssh or in non interactive FTP / Samba file transfers between Linux servers it can come handy.

To visualize the download and writing speed of lets say FTP transferred .AVI movie (during the actual file transfer) on the download host issue:

# speedometer Download-Folder/What-goes-around-comes-around.avi

7. Estimating approximate time for file transfer

There is another section in the speedometer manual pointing of the program use to calculate the time remaining for a file transfer.

The (man speedometer) provided example text is:

How long it will take for my 38MB transfer to finish? speedometer favorite_episode.rm $((38*1024*1024))

At first glimpse it hard to understand (like the other manual example). A bit of reasoning and I comprehend what the man author meant by the obscure calculation:

$((38*1024*1024))

This is a formula used in which 38 has to be substituted with the exact file size amount of the transferred file. The author manual used a 38MB file so this is why he put $((38* … in the formula.

I give it a try – (just for the sake to see how it works) with a file with a size of 2500MB, in below two screenshot pictures I show my preparation to copy the file and the actual copying / "real time" transfer tracking with speedometer's status percentage completion bar.

xterm terminal copy file and estimate file copying operation speed on linux with speedometer preparation

Two xterm terminals one is copying a file the other one uses speedometer to estimate the time remaining to complete the file transfer from expansion USB hard drive to my laptop harddrive

Tags: amount, Ascii, Auto, avi, bandwidth network, CentOS, connection server, data transfers, Desktop, Display, Draft, Easy Monitor, eth, eye, granularity, hipo, host, import, Installing, iptraf, language interpreter, laptop, Linux, linux distributions, machine, midnight commander, monitor network, nbsp, necessery, network congestion, network interface, network traffic, noah, os linux, os platforms, package, python, python interpreter, python programming language, quot, remote server, root, rpm, server admin, server administrators, server network, server traffic, speedometer, test, text monitoring, traffic performance, Ubuntu, user interface library, yum
Posted in Monitoring, System Administration, Various | No Comments »

How to delete million of files on busy Linux servers (Work out Argument list too long)

Tuesday, March 20th, 2012

How to Delete million or many thousands of files in the same directory on GNU / Linux and FreeBSD

If you try to delete more than 131072 of files on Linux with rm -f *, where the files are all stored in the same directory, you will get an error:

/bin/rm: Argument list too long.

I've earlier blogged on deleting multiple files on Linux and FreeBSD and this is not my first time facing this error.
Anyways, as time passed, I've found few other new ways to delete large multitudes of files from a server.

In this article, I will explain shortly few approaches to delete few million of obsolete files to clean some space on your server.
Here are 3 methods to use to clean your tons of junk files.

1. Using Linux find command to wipe out millions of files

a.) Finding and deleting files using find's -exec switch:

# find . -type f -exec rm -fv {} \;

This method works fine but it has 1 downside, file deletion is too slow as for each found file external rm command is invoked.

For half a million of files or more, using this method will take "long". However from a server hard disk stressing point of view it is not so bad as, the files deletion is not putting too much strain on the server hard disk.
b.) Finding and deleting big number of files with find's -delete argument:

Luckily, there is a better way to delete the files, by using find's command embedded -delete argument:

# find . -type f -print -delete

c.) Deleting and printing out deleted files with find's -print arg

If you would like to output on your terminal, what files find is deleting in "real time" add -print:

# find . -type f -print -delete

To prevent your server hard disk from being stressed and hence save your self from server normal operation "outages", it is good to combine find command with ionice, e.g.:

# ionice -c 3 find . -type f -print -delete

Just note, that ionice cannot guarantee find's opeartions will not affect severely hard disk i/o requests. On heavily busy servers with high amounts of disk i/o writes still applying the ionice will not prevent the server from being hanged! Be sure to always keep an eye on the server, while deleting the files nomatter with or without ionice. if throughout find execution, the server gets lagged in serving its ordinary client requests or whatever, stop the execution of the cmd immediately by killing it from another ssh session or tty (if physically on the server).

2. Using a simple bash loop with rm command to delete "tons" of files

An alternative way is to use a bash loop, to print each of the files in the directory and issue /bin/rm on each of the loop elements (files) like so:

for i in *; do rm -f $i; done

If you'd like to print what you will be deleting add an echo to the loop:

# for i in $(echo *); do \ echo "Deleting : $i"; rm -f $i; \

The bash loop, worked like a charm in my case so I really warmly recommend this method, whenever you need to delete more than 500 000+ files in a directory.

3. Deleting multiple files with perl

Deleting multiple files with perl is not a bad idea at all.
Here is a perl one liner, to delete all files contained within a directory:

# perl -e 'for(<*>){((stat)[9]<(unlink))}'

If you prefer to use more human readable perl script to delete a multitide of files use delete_multple_files_in_dir_perl.pl

Using perl interpreter to delete thousand of files is quick, really, really quick.
I did not benchmark it on the server, how quick exactly is it, but I guess the delete rate should be similar to find command. Its possible even in some cases the perl loop is quicker …

4. Using PHP script to delete a multiple files

Using a short php script to delete files file by file in a loop similar to above bash script is another option.
To do deletion with PHP, use this little PHP script:

<?php $dir = "/path/to/dir/with/files"; $dh = opendir( $dir); $i = 0; while (($file = readdir($dh)) !== false) { $file = "$dir/$file"; if (is_file( $file)) { unlink( $file); if (!(++$i % 1000)) { echo "$i files removed\n"; } } } ?>

As you see the script reads the $dir defined directory and loops through it, opening file by file and doing a delete for each of its loop elements.
You should already know PHP is slow, so this method is only useful if you have to delete many thousands of files on a shared hosting server with no (ssh) shell access.

This php script is taken from Steve Kamerman's blog . I would like also to express my big gratitude to Steve for writting such a wonderful post. His post actually become inspiration for this article to become reality.

You can also download the php delete million of files script sample here

To use it rename delete_millioon_of_files_in_a_dir.php.txt to delete_millioon_of_files_in_a_dir.php and run it through a browser .

Note that you might need to run it multiple times, cause many shared hosting servers are configured to exit a php script which keeps running for too long.
Alternatively the script can be run through shell with PHP cli:

php -l delete_millioon_of_files_in_a_dir.php.txt.

5. So What is the "best" way to delete million of files on Linux?

In order to find out which method is quicker in terms of execution time I did a home brew benchmarking on my thinkpad notebook.

a) Creating 509072 of sample files.

Again, I used bash loop to create many thousands of files in order to benchmark.
I didn't wanted to put this load on a productive server and hence I used my own notebook to conduct the benchmarks. As my notebook is not a server the benchmarks might be partially incorrect, however I believe still .they're pretty good indicator on which deletion method would be better.

hipo@noah:~$ mkdir /tmp/test hipo@noah:~$ cd /tmp/test; hiponoah:/tmp/test$ for i in $(seq 1 509072); do echo aaaa >> $i.txt; done

I had to wait few minutes until I have at hand 509072 of files created. Each of the files as you can read is containing the sample "aaaa" string.

b) Calculating the number of files in the directory

Once the command was completed to make sure all the 509072 were existing, I used a find + wc cmd to calculate the directory contained number of files:

hipo@noah:/tmp/test$ find . -maxdepth 1 -type f |wc -l 509072
real 0m1.886s user 0m0.440s sys 0m1.332s

Its intesrsting, using an ls command to calculate the files is less efficient than using find:

hipo@noah:/tmp/test$ time ls -1 |wc -l 509072
real 0m3.355s user 0m2.696s sys 0m0.528s

c) benchmarking the different file deleting methods with time

– Testing delete speed of find

hipo@noah:/tmp/test$ time find . -maxdepth 1 -type f -delete real 15m40.853s user 0m0.908s sys 0m22.357s

You see, using find to delete the files is not either too slow nor light quick.

– How fast is perl loop in multitude file deletion ?

hipo@noah:/tmp/test$ time perl -e 'for(<*>){((stat)[9]<(unlink))}'real 6m24.669suser 0m2.980ssys 0m22.673s

Deleting my sample 509072 took 6 mins and 24 secs. This is about 3 times faster than find! GO-GO perl 🙂
As you can see from the results, perl is a great and time saving, way to delete 500 000 files.

– The approximate speed deletion rate of of for + rm bash loop

hipo@noah:/tmp/test$ time for i in *; do rm -f $i; done
real 206m15.081s user 2m38.954s sys 195m38.182s

You see the execution took 195m en 38 secs = 3 HOURS and 43 MINUTES!!!! This is extremely slow ! But works like a charm as the running of deletion didn't impacted my normal laptop browsing. While the script was running I was mostly browsing through few not so heavy (non flash) websites and doing some other stuff in gnome-terminal) 🙂

As you can imagine running a bash loop is a bit CPU intensive, but puts less stress on the hard disk read/write operations. Therefore its clear using it is always a good practice when deletion of many files on a dedi servers is required.

b) my production server file deleting experience

On a production server I only tested two of all the listed methods to delete my files. The production server, where I tested is running Debian GNU / Linux Squeeze 6.0.3. There I had a task to delete few million of files.
The tested methods tried on the server were:

– The find . type -f -delete method.

– for i in *; do rm -f $i; done

The results from using find -delete method was quite sad, as the server almost hanged under the heavy hard disk load the command produced.

With the for script all went smoothly. The files were deleted for a long long time (like few hours), but while it was running, the server continued with no interruptions..

While the bash loop was running, the server load avarage kept at steady 4
Taking my experience in mind, If you're running a production, server and you're still wondering which delete method to use to wipe some multitude of files, I would recommend you go the bash for loop + /bin/rm way. Yes, it is extremely slow, expect it run for some half an hour or so but puts not too much extra load on the server..

Using the PHP script will probably be slow and inefficient, if compared to both find and the a bash loop.. I didn't give it a try yet, but suppose it will be either equal in time or at least few times slower than bash.

If you have tried the php script and you have some observations, please drop some comment to tell me how it performs.

To sum it up;

Even though there are "hacks" to clean up some messy parsing directory full of few million of junk files, having such a directory should never exist on the first place.

Frankly, keeping millions of files within the same directory is very stupid idea.
Doing so will have a severe negative impact on a directory listing performance of your filesystem in the long term.

If you know better (more efficient) ways to delete a multitude of files in a dir please share in comments.

Tags: Auto, client, deletec, deleteJust, Deleting, deleting files, Disk, downside, Draft, exec rm, eye, file, file deletion, freebsd, half a million, hard disk, heavy load, hipo, junk files, Linux, linux servers, maxdepth, multiple files, multitudes, multple, nbsp, new ways, noah, nomatter, number, obsolete files, operation, option, point of view, printing, quot, real time, rm command, servers work, Shell, ssh, terminal, test, time, tmp, type, unlink
Posted in Linux, Programming, System Administration | 13 Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘test’

Create local network between virtual machines in Virtualbox VM – Add local LAN between Linux Virtual Machines

Devices -> Network Settings

Best Windows tools to Test (Benchmark) Hard Drives, SSD Drives and RAID Storage Controllers

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites

April 2020
M	T	W	T	F	S	S
« Mar
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30