Posts Tagged ‘msg’

How to exclude files on copy (cp) on GNU / Linux / Linux copy and exclude files and directories (cp -r) exclusion

Saturday, March 3rd, 2012

I've recently had to make a copy of one /usr/local/nginx directory under /usr/local/nginx-bak, in order to have a working copy of nginx, just in case if during my nginx update to new version from source mess ups.

I did not check the size of /usr/local/nginx , so just run the usual:

nginx:~# cp -rpf /usr/local/nginx /usr/local/nginx-bak
...

Execution took more than 20 seconds, so I check the size and figured out /usr/local/nginx/logs has grown to 120 gigabytes.

I didn't wanted to extra load the production server with copying thousands of gigabytes so I asked myself if this is possible with normal Linux copy (cp) command?. I checked cp manual e.g. man cp, but there is no argument like –exclude or something.

Even though the cp command exclude feature is not implemented by default there are a couple of ways to copy a directory with exclusion of subdirectories of files on G / Linux.

Here are the 3 major ones:

1. Copy directory recursively and exclude sub-directories or files with GNU tar

Maybe the quickest way to copy and exclude directories is through a littke 'hack' with GNU tarnginx:~# mkdir /usr/local/nginx-new;
nginx:~# cd /usr/local/nginx#
nginx:/usr/local/nginx# tar cvf - \. --exclude=/usr/local/nginx/logs/* \
| (cd /usr/local/nginx-new; tar -xvf - )

Copying that way however is slow, in my case it fits me perfectly but for copying large chunks of data it is better not to use pipe and instead use regular tar operation + mv

# cd /source_directory
# tar cvf test.tar --exclude=dir_to_exclude/*\--exclude=dir_to_exclude1/* . \
# mv test.tar /destination_directory
# cd /destination# tar xvf test.tar

2. Copy folder recursively excluding some directories with rsync

P>eople who has experience with rsync , already know how invaluable this tool is. Rsync can completely be used as for substitute=de.a# rsync -av –exclude='path1/to/exclude' –exclude='path2/to/exclude' source destination

This example, can also be used as a solution to my copy nginx and exclude logs directory casus like so:

nginx:~# rsync -av --exclude='/usr/local/nginx/logs/' /usr/local/nginx/ /usr/local/nginx-new

As you can see for yourself, this is a way more readable for the tar, however it will not work on servers, where rsync is not installed and it is unusable if you have to do operations as a regular users on such for that case surely the GNU tar hack is more 'portable' across systems.
rsync has also Windows version and therefore, the same methodology should be working on MS Windows and good for batch scripting.
I've not tested it myself, yet as I've never used rsync on Windows, if someone has tried and it works pls drop me a short msg in comments.
3. Copy directory and exclude sub directories and files with find

Find in collaboration with cp can also be used to exclude certain directories while copying. Actually this method is better than the GNU tar hack and surely more efficient. For machines, where rsync is not installed it is just a perfect way to copy files from location to location, while excluding some directories, here is an example use of find and cp, for the above nginx case:

nginx:~# cd /usr/local/nginx
nginx:~# mkdir /usr/local/nginx
nginx:/usr/local/nginx# find . -type d \( ! -name logs \) -print -exec cp -rpf '{}' /usr/local/nginx-bak \;

This will find all directories inside /usr/local/nginx with find command print them on the screen, then execute recursive copy over each found directory and copy to /usr/local/nginx-bak

This example will work fine in the nginx case because /usr/local/nginx does not contain any files but only sub-directories. In other occwhere the directory does contain some files besides sub-directories the files had to also be copied e.g.:

# for i in $(ls -l | egrep -v '^d'); do\
cp -rpf $i /destination/directory

This will copy the files from source directory (for instance /usr/local/nginx/my_file.txt, /usr/local/nginx/my_file1.txt etc.), which doesn't belong to a subdirectory.

The cmd expression:

# ls -l | egrep -v '^d'

Lists only the files while excluding all the directories and in a for loop each of the files is copied to /destination/directory

If someone has better ideas, please share with me 🙂

How to change users quota to NO QUOTA on Qmail with Vpopmail Mail server install / Qmail mail over quota issue

Monday, February 20th, 2012

 

Qmail Vpopmail quota exceeded Dolphin Logo

Already on a couple of mail boxes located on one of the qmail powered mail servers I adminiter, there is an over QUOTA reached problem encountered.

Filling up the mailbox quota is not nice as mails starts get bounced back to the sender with a message QUOTA FULL or EXCEEDED MESSAGE, if this is a crucial mail waiting for some important data etc. the data is never received.
Below is a copy of the mail quota waarning notification message:

Delivered-To: email_use@my-mail-domain.net
Date: Wed, 15 Feb 2012 17:40:36 +0000
X-Comment: Rename/Copy this file to ~vpopmail/domains/.quotawarn.msg, and make appropriate changes
X-Comment: See README.quotas for more information
From: Mail Delivery System <Mailer-Daemon@different.bg>
Reply-To: email@pc-freak.net
To: Valued Customer:;
Subject: Mail quota warning
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 7bit
>
Your mailbox on the server is now more than 90% full. So that you can continue
to receive mail you need to remove some messages from your mailbox.

As you can read from the copy of the mail message above, the message content sent to the mail owner whose quota is getting full is red from /var/vpopmail/domains/.quotawarn.msg

The mail reaching quota problem is very likely to appear in cases like low mailbox quota set, but sometimes also occurs due to bugs in vpopmail quota handling.

Various interesting configuration settings for mail quotas etc. are in /home/vpopmail/etc/vlimits.default file, (assuming vpopmail is installed in /home).

In my specific case, the default vpopmail mailbox quota size was set to only 40 Megabytes.
40MB is too low if compared to todays mailbox size standards which in Gmail and Yahoo  mail services are already a couple of gigabytes.
Hence to get around the quota troubles, I  removed the quota for the mail.
To remove the quota size in vpopmail set for address (email_user@my-mail-domain.net) used cmd:

qmail-server:~# vmoduser -q NOQUOTA email_user@my-mail-domain.net

To save myself from future quota issues, I decided to apply a permanent fix to all those over quota size VPOPMAIL mailbox problems by removing completely quota restriction for all mailboxes in my vpopmail existent mail domain.

To do so, I wrote a quick simple bash loop one-liner script:

qmail-server:~# cd /home/vpopmail/domains
qmail-server:~/vpopmail/domains# cd my-mail-domain.net
qmail-server:~/vpopmail/domains/my-mail-domain.net# for i in *; do \
vmoduser -q NOQUOTA $(echo $i|grep -v vpasswd)@my-mail-domain.net; \
done

This works only on vpopmail installations which are configured to store the mail messages directly on the filesystem. Therefore this approach will not work for people who during vpopmail install had configured it to store mailboxes in MySQL or in other kind of SQL db engine.

Anyways for Vpopmail installed to use SQL backend, the script can be changed to read directly a list with all the mailboxes obtained from databasae (SQL query) and then, loop over each of the mail addresses apply the vmoduser -q NOQUOTA mail@samplemaildomain.net.

I've written also a few lines shell script (remove_vpopmail_emails_domain_quota.sh), it accepts one argument which is a vpopmail domain to which the admin would like to reset all applied mailbox quotas. The script is useful, if you have to often remove all quotas for vpopmail domainsor have to do quota wipe out simultaneously for multiple email domain names  located on different servers.

Don’t msg and Drive! ;)

Wednesday, June 22nd, 2011

Don't message and drive picture

Do you message when you drive? Do you know who are you messaging to ? 😉 Arghhh its the GRIM Reaper !!! 🙂

How to fix “delivery 1: deferral: Sorry,_message_has_wrong_owner._(#4.3.5)/” qmail mail delivery failure message

Friday, May 20th, 2011

After a failed attempt to enable some wrapper scripts to enable domain keys support in a qmail powered mail server my qmail server suddenly stopped being able to normally send mail.

The exact error message which was logged in /var/log/qmail/current was:

@400000004dd66fcc16a088ac delivery 1: deferral: Sorry,_message_has_wrong_owner._(#4.3.5)/

This qmail messed happened after I substituted /var/qmail/bin/qmail-queue and /var/qmail/bin/qmail-remote with two respective wrapper shell scripts which were calling for the original qmail-queue and qmail-remote binaries under the names qmail-queue.orig and qmail-queue.orig

Restoring back qmail-queue.orig to /var/qmail/bin/qmail-queue and qmail-remote.orig to /var/qmain/bin/qmail-remote and restarting the mail server broke my qmail install.

After a bunch of nerves trying to isolate what is causing the error I found out that by mistake I forgot to copy the qmail-queue and qmail-remote permissions and ownership.

Thus I had to check another qmail working installation’s permissions for both binaries and fix the permissions to be equivalent to the permissions:

debian:~# ls -al /var/qmail/bin/qmail-remote
-rwx–x–x 1 root qmail 50464 2011-05-20 12:56 /var/qmail/bin/qmail-remote*
debian:~# ls -al /var/qmail/bin/qmail-queue
-rws–x–x 1 qmailq qmail 20392 2011-05-20 12:56 /var/qmail/bin/qmail-queue*

The exact chmod and chmod commands I issued to solve the shitty issues were as follows:

First I fixed the qmail-queue and qmail-remote ownership:

debian:~# chown qmailq:qmail /var/qmail/bin/qmail-queue
debian:~# chown root:qmail /var/qmail/bin/qmail-remote

Second I set the proper file permissions:

# make the qmail-queue binary suid
debian:~# chmod u+s /var/qmail/bin/qmail-queue
debian:~# chmod 611 /var/qmail/bin/qmail-queue
debian:~# chmod 611 /var/qmail/bin/qmail-remote

Third and last I did a restart of the qmail server and tested it sends properly

debian:~# /usr/bin/qmailctl stop
Stopping qmail...
qmail-send
qmail-smtpd
debian:~# /usr/bin/qmailctl start
Starting qmail

Finally to test that the qmail server qmail-queue was queing and sending with qmail-remote I used the system mail command like so:

debian:~# mail -s "test email" testuser@pc-freak.net
asdfafdsdf
.
Cc:

Afterwards the mail was properly received on my mail account testuser@pc-freak.net immediately.

In my /var/log/qmail/current log file all seemed fine:

@400000004dd6702a2eb2b064 starting delivery 1: msg 85281596 to remote testuser@pc-freak.net
@400000004dd6702a2eb2b834 status: local 0/20 remote 1/20
@400000004dd6702b34cc809c delivery 1: success: 83.228.93.76_accepted_message./Remote_host_said:_250_ok_
1305899099_qp_65293/
@400000004dd6702b34cc886c status: local 0/20 remote 0/20
@400000004dd6702b34cc8c54 end msg 85281596

The test mail was properly received on my mail account testuser@pc-freak.net immediately.

It took me like half an hour to figure out what exactly is wrong with the permissions in situations like this I really wanted to change all my qmail installs with postfix and forget forever I ever used qmail …