Posts Tagged ‘SQL’

How to upgrade single package with their dependencies on Debian and Ubuntu Linux

Friday, March 16th, 2012

Debian GNU / Linux apt-get upgrade a package selection of a whole bunch of packages ready to upgrade apt artistic logo

Are you a Debian System Administrator and you recently run apt-get upgrade && apt-get upgrade finding out there are plenty of new packagesfor upgrade? Do you need only a pre-selected number of packages to upgrade with apt?
I run apt-get update && apt-get upgrade on one of our company Debian servers, just to see there are a number of packages to be upgraded among which there was some I didn't wanted to upgrade. Here is a little paste output from apt-get upgrade:

debian:~# apt-get update && apt-get upgrade
Hit http://security.debian.org squeeze/updates Release.gpg
...
Hit http://security.debian.org squeeze/updates/main amd64 Packages
Fetched 128 kB in 0s (441 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
at imagemagick libdbd-pg-perl libfreetype6 libmagickcore3 libmagickcore3-extra libmagickwand3 libmysqlclient16 mysql-client
mysql-client-5.1 mysql-common mysql-server mysql-server-5.1 mysql-server-core-5.1
Do you want to continue [Y/n]
14 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

From first sight it seems logical to issue apt-get upgrade packagename to upgrade only single package with its package dependencies, instead of the whole group the above packs. However doing:
apt-get upgrade imagemagick will still try to upgrade all the packages instead of just imagemagick and its dependency package deb libmagickcore3

debian:~# apt-get upgrade imagemagick
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
at imagemagick libdbd-pg-perl libfreetype6 libmagickcore3 libmagickcore3-extra libmagickwand3 libmysqlclient16 mysql-client
mysql-client-5.1 mysql-common mysql-server mysql-server-5.1 mysql-server-core-5.1
14 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Do you want to continue [Y/n]

Doing all package,upgrade is not a good idea in my case, since upgrading mysql-server will require a MySQL server restart (something which we cannot afford to do right now) on this production server.
MySQL server restart during upgrade is never a good idea especially on productive busy (heavy loaded) SQL servers.
A restart of the MySQL server serving thousands of requests per second could lead often to crashed tables and hence temporary server downtime etc.

Still it is a good idea to upgrade the rest of packages with their newer versions. For exmpl. to upgrade; imagemagick, at , libfreetype6 and so on.

In order to upgrade only this 3 ones and their respective package dependencies, issue:

debian:~# apt-get --yes install imagemagick at libfreetype6

Repeat the apt-get install command with passing all the single package name you want to be upgraded and voila you're done :).
Be sure the apt-get install packagename upgrade doesn't require also upgrade of myssql-server, mysql-client, mysql-common or mysql-server-core-5.1 or any of the package name you want to preserve from upgrading.

How to resolve (fix) WordPress wp-cron.php errors like “POST /wp-cron.php?doing_wp_cron HTTP/1.0″ 404” / What is wp-cron.php and what it does

Monday, March 12th, 2012

fix wordpress wp-cron.php 404 HTTP error, what is wp-cron.php schedule logo

One of the WordPress websites hosted on our dedicated server produces all the time a wp-cron.php 404 error messages like:

xxx.xxx.xxx.xxx - - [15/Apr/2010:06:32:12 -0600] "POST /wp-cron.php?doing_wp_cron HTTP/1.0

I did not know until recently, whatwp-cron.php does, so I checked in google and red a bit. Many of the places, I've red are aa bit unclear and doesn't give good exlanation on what exactly wp-cron.php does. I wrote this post in hope it will shed some more light on wp-config.php and how this major 404 issue is solved..
So

what is wp-cron.php doing?

 

  • wp-cron.php is acting like a cron scheduler for WordPress.
  • wp-cron.php is a wp file that controls routine actions for particular WordPress install.
  • Updates the data in SQL database on every, request, every day or every hour etc. – (depending on how it's set up.).
  • wp-cron.php executes automatically by default after EVERY PAGE LOAD!
  • Checks all pending comments for spam with Akismet (if akismet or anti-spam plugin alike is installed)
  • Sends all scheduled emails (e.g. sent a commentor email when someone comments on his comment functionality, sent newsletter subscribed persons emails etc.)
  • Post online scheduled articles for a day and time of particular day

Suppose you're writting a new post and you want to take advantage of WordPress functionality to schedule a post to appear Online at specific time:

What is wordpress wp-cron.php, Scheduling wordpress post screenshot

The Publish Immediately, field execution is being issued on the scheduled time thanks to the wp-cron.php periodic invocation.

Another example for wp-cron.php operation is in handling flushing of WP old HTML Caches generated by some wordpress caching plugin like W3 Total Cache
wp-cron.php takes care for dozens of other stuff silently in the background. That's why many wordpress plugins are depending heavily on wp-cron.php proper periodic execution. Therefore if something is wrong with wp-config.php, this makes wordpress based blog or website partially working or not working at all.
 

Our company wp-cron.php errors case

In our case the:
212.235.185.131 – – [15/Apr/2010:06:32:12 -0600] "POST /wp-cron.php?doing_wp_cron HTTP/1.0" 404
is occuring in Apache access.log (after each unique vistor request to wordpress!.), this is cause wp-cron.php is invoked on each new site visitor site request.
This puts a "vain load" on the Apache Server, attempting constatly to invoke the script … always returning not found 404 err.

As a consequence, the WP website experiences "weird" problems all the time. An illustration of a problem caused by the impoper wp-cron.php execution is when we are adding new plugins to WP.

Lets say a new wordpress extension is download, installed and enabled in order to add new useful functioanlity to the site.

Most of the time this new plugin would be malfunctioning if for example it is prepared to add some kind of new html form or change something on some or all the wordpress HTML generated pages.
This troubles are result of wp-config.php's inability to update settings in wp SQL database, after each new user request to our site.
So the newly added plugin website functionality is not showing up at all, until WP cache directory is manually deleted with rm -rf /var/www/blog/wp-content/cache/

I don't know how thi whole wp-config.php mess occured, however my guess is whoever installed this wordpress has messed something in the install procedure.

Anyways, as I researched thoroughfully, I red many people complaining of having experienced same wp-config.php 404 errs. As I red, most of the people troubles were caused by their shared hosting prohibiting the wp-cron.php execution.
It appears many shared hostings providers choose, to disable the wordpress default wp-cron.php execution. The reason is probably the script puts heavy load on shared hosting servers and makes troubles with server overloads.

Anyhow, since our company server is adedicated server I can tell for sure in our case wordpress had no restrictions for how and when wp-cron.php is invoked.
I've seen also some posts online claiming, the wp-cron.php issues are caused of improper localhost records in /etc/hosts, after a thorough examination I did not found any hosts problems:

hipo@debian:~$ grep -i 127.0.0.1 /etc/hosts
127.0.0.1 localhost.localdomain localhost

You see from below paste, our server, /etc/hosts has perfectly correct 127.0.0.1 records.

Changing default way wp-cron.php is executed

As I've learned it is generally a good idea for WordPress based websites which contain tens of thousands of visitors, to alter the default way wp-cron.php is handled. Doing so will achieve some efficiency and improve server hardware utilization.
Invoking the script, after each visitor request can put a heavy "useless" burden on the server CPU. In most wordpress based websites, the script did not need to make frequent changes in the DB, as new comments in posts did not happen often. In most wordpress installs out there, big changes in the wordpress are not common.

Therefore, a good frequency to exec wp-cron.php, for wordpress blogs getting only a couple of user comments per hour is, half an hour cron routine.

To disable automatic invocation of wp-cron.php, after each visitor request open /var/www/blog/wp-config.php and nearby the line 30 or 40, put:

define('DISABLE_WP_CRON', true);

An important note to make here is that it makes sense the position in wp-config.php, where define('DISABLE_WP_CRON', true); is placed. If for instance you put it at the end of file or near the end of the file, this setting will not take affect.
With that said be sure to put the variable define, somewhere along the file initial defines or it will not work.

Next, with Apache non-root privileged user lets say www-data, httpd, www depending on the Linux distribution or BSD Unix type add a php CLI line to invoke wp-cron.php every half an hour:

linux:~# crontab -u www-data -e

0,30 * * * * cd /var/www/blog; /usr/bin/php /var/www/blog/wp-cron.php 2>&1 >/dev/null

To assure, the php CLI (Command Language Interface) interpreter is capable of properly interpreting the wp-cron.php, check wp-cron.php for syntax errors with cmd:

linux:~# php -l /var/www/blog/wp-cron.php
No syntax errors detected in /var/www/blog/wp-cron.php

That's all, 404 wp-cron.php error messages will not appear anymore in access.log! 🙂

Just for those who can find the root of the /wp-cron.php?doing_wp_cron HTTP/1.0" 404 and fix the issue in some other way (I'll be glad to know how?), there is also another external way to invoke wp-cron.php with a request directly to the webserver with short cron invocation via wget or lynx text browser.

– Here is how to call wp-cron.php every half an hour with lynxPut inside any non-privileged user, something like:
01,30 * * * * /usr/bin/lynx -dump "http://www.your-domain-url.com/wp-cron.php?doing_wp_cron" 2>&1 >/dev/null

– Call wp-cron.php every 30 mins with wget:

01,30 * * * * /usr/bin/wget -q "http://www.your-domain-url.com/wp-cron.php?doing_wp_cron"

Invoke the wp-cron.php less frequently, saves the server from processing the wp-cron.php thousands of useless times.

Altering the way wp-cron.php works should be seen immediately as the reduced server load should drop a bit.
Consider you might need to play with the script exec frequency until you get, best fit cron timing. For my company case there are only up to 3 new article posted a week, hence too high frequence of wp-cron.php invocations is useless.

With blog where new posts occur once a day a script schedule frequency of 6 up to 12 hours should be ok.

 

How to change users quota to NO QUOTA on Qmail with Vpopmail Mail server install / Qmail mail over quota issue

Monday, February 20th, 2012

 

Qmail Vpopmail quota exceeded Dolphin Logo

Already on a couple of mail boxes located on one of the qmail powered mail servers I adminiter, there is an over QUOTA reached problem encountered.

Filling up the mailbox quota is not nice as mails starts get bounced back to the sender with a message QUOTA FULL or EXCEEDED MESSAGE, if this is a crucial mail waiting for some important data etc. the data is never received.
Below is a copy of the mail quota waarning notification message:

Delivered-To: email_use@my-mail-domain.net
Date: Wed, 15 Feb 2012 17:40:36 +0000
X-Comment: Rename/Copy this file to ~vpopmail/domains/.quotawarn.msg, and make appropriate changes
X-Comment: See README.quotas for more information
From: Mail Delivery System <Mailer-Daemon@different.bg>
Reply-To: email@www.pc-freak.net
To: Valued Customer:;
Subject: Mail quota warning
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 7bit
>
Your mailbox on the server is now more than 90% full. So that you can continue
to receive mail you need to remove some messages from your mailbox.

As you can read from the copy of the mail message above, the message content sent to the mail owner whose quota is getting full is red from /var/vpopmail/domains/.quotawarn.msg

The mail reaching quota problem is very likely to appear in cases like low mailbox quota set, but sometimes also occurs due to bugs in vpopmail quota handling.

Various interesting configuration settings for mail quotas etc. are in /home/vpopmail/etc/vlimits.default file, (assuming vpopmail is installed in /home).

In my specific case, the default vpopmail mailbox quota size was set to only 40 Megabytes.
40MB is too low if compared to todays mailbox size standards which in Gmail and Yahoo  mail services are already a couple of gigabytes.
Hence to get around the quota troubles, I  removed the quota for the mail.
To remove the quota size in vpopmail set for address (email_user@my-mail-domain.net) used cmd:

qmail-server:~# vmoduser -q NOQUOTA email_user@my-mail-domain.net

To save myself from future quota issues, I decided to apply a permanent fix to all those over quota size VPOPMAIL mailbox problems by removing completely quota restriction for all mailboxes in my vpopmail existent mail domain.

To do so, I wrote a quick simple bash loop one-liner script:

qmail-server:~# cd /home/vpopmail/domains
qmail-server:~/vpopmail/domains# cd my-mail-domain.net
qmail-server:~/vpopmail/domains/my-mail-domain.net# for i in *; do \
vmoduser -q NOQUOTA $(echo $i|grep -v vpasswd)@my-mail-domain.net; \
done

This works only on vpopmail installations which are configured to store the mail messages directly on the filesystem. Therefore this approach will not work for people who during vpopmail install had configured it to store mailboxes in MySQL or in other kind of SQL db engine.

Anyways for Vpopmail installed to use SQL backend, the script can be changed to read directly a list with all the mailboxes obtained from databasae (SQL query) and then, loop over each of the mail addresses apply the vmoduser -q NOQUOTA mail@samplemaildomain.net.

I've written also a few lines shell script (remove_vpopmail_emails_domain_quota.sh), it accepts one argument which is a vpopmail domain to which the admin would like to reset all applied mailbox quotas. The script is useful, if you have to often remove all quotas for vpopmail domainsor have to do quota wipe out simultaneously for multiple email domain names  located on different servers.

Monitoring Windows hosts with Nagios on Debian GNU/Linux

Tuesday, August 30th, 2011

Nagios logo install and configure nagios to monitor Windows hosts with on Debian GNU/Linux

In this article in short, I’ll explain how I configured Nagios on a Debian GNU/Linux release (Squeeze 6) to monitor a couple of Windows hosts running inside a local network. Now let’s start.

1. Install necessery nagios debian packages

apt-get install nagios-images nagios-nrpe-plugin nagios-nrpe-server nagios-plugins nagios-plugins-basic nagios-plugins-standard
nagios3 nagios3-cgi nagios3-common nagios3-core

2. Edit /etc/nagios-plugins/config/nt.cfg

In the File substitute:

define command { command_name check_nt command_line /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -v '$ARG1$' }

With:

define command {
command_name check_nt
command_line /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -p 12489 -v $ARG1$ $ARG2$
}

3. Modify nrpe.cfg to put in allowd hoss to connect to the Nagions nrpe server

vim /etc/nagios/nrpe.cfg

Lookup inside for nagios’s configuration directive:

allowed_hosts=127.0.0.1

In order to allow more hosts to report to the nagios nrpe daemon, change the value to let’s say:

allowed_hosts=127.0.0.1,192.168.1.4,192.168.1.5,192.168.1.6

This config allows the three IPs 192.168.1.4-6 to be able to report for nrpe.

For the changes to nrpe server to take effect, it has to be restrarted.

debian:~# /etc/init.d/nagios-nrpe-server restart

Further on some configurations needs to be properly done on the nrpe agent Windows hosts in this case 192.168.1.4,192.168.1.5,192.168.1.6

4. Install the nsclient++ on all Windows hosts which CPU, Disk, Temperature and services has to be monitored

Download the agent from http://sourceforge.net/projects/nscplus and launch the installer, click twice on it and follow the installation screens. Its necessery that during installation the agent has the NRPE protocol enabled. After the installation is complete one needs to modify the NSC.ini
By default many of nsclient++ tracking modules are not enabled in NSC.ini, thus its necessery that the following DLLs get activated in the conf:

FileLogger.dll
CheckSystem.dll
CheckDisk.dll
NSClientListener.dll
SysTray.dll
CheckEventLog.dll
CheckHelpers.dll

Another requirement is to instruct the nsclient++ angent to have access to the Linux installed nagios server again with adding it to the allowed_hosts config variable:

allowed_hosts=192.168.1.1

In my case the Nagios runs on Debian Lenny (Squeeze) 6 and possess the IP address of 192.168.1.1
To test the intalled windows nsclient++ agents are properly installed a simple telnet connection from the Linux host is enough:

5. Create necessery configuration for the nagios Linux server to include all the Windows hosts which will be monitored

There is a window.cfg template file located in /usr/share/doc/nagios3-common/examples/template-object/windows.cfg on Debian.

The file is a good start point for creating a conf file to be understand by nagios and used to periodically refresh information about the status of the Windows hosts.

Thus it’s a good idea to copy the file to nagios3 config directory:

debian:~# mkdir /etc/nagios3/objects
debian:~# cp -rpf /usr/share/doc/nagios3-common/examples/template-object/windows.cfg /etc/nagios3/objects/windows.cfg

A sample windows.cfg content, (which works for me fine) and monitor a couple of Windows nodes running MS-SQL service and IIS and makes sure the services are up and running are:

define host{
use windows-server ; Inherit default values from a template
host_name Windows1 ; The name we're giving to this host
alias Iready Server ; A longer name associated with the host
address 192.168.1.4 ; IP address of the host
}
define host{
use windows-server ; Inherit default values from a template
host_name Windows2 ; The name we're giving to this host
alias Iready Server ; A longer name associated with the host
address 192.168.1.4 ; IP address of the host
}
define hostgroup{
hostgroup_name windows-servers ; The name of the hostgroup
alias Windows Servers ; Long name of the group
}
define hostgroup{
hostgroup_name IIS
alias IIS Servers
members Windows1,Windows2
}
define hostgroup{
hostgroup_name MSSQL
alias MSSQL Servers
members Windows1,Windows2
}
define service{
use generic-service
host_name Windows1
service_description NSClient++ Version
check_command check_nt!CLIENTVERSION
}
define service{ use generic-service
host_name Windows1
service_description Uptime
check_command check_nt!UPTIME
}
define service{ use generic-service
host_name Windows1
service_description CPU Load
check_command check_nt!CPULOAD!-l 5,80,90
}
define service{
use generic-service
host_name Windows1
service_description Memory Usage
check_command check_nt!MEMUSE!-w 80 -c 90
define service{
use generic-service
host_name Windows1
service_description C: Drive Space
check_command check_nt!USEDDISKSPACE!-l c -w 80 -c 90
}
define service{
use generic-service
host_name Windows1
service_description W3SVC
check_command check_nt!SERVICESTATE!-d SHOWALL -l W3SVC
}
define service{
use generic-service
host_name Windows1
service_description Explorer
check_command check_nt!PROCSTATE!-d SHOWALL -l Explorer.exe
}
define service{
use generic-service
host_name Windows2
service_description NSClient++ Version
check_command check_nt!CLIENTVERSION
}
define service{ use generic-service
host_name Windows2
service_description Uptime
check_command check_nt!UPTIME
}
define service{ use generic-service
host_name Windows2
service_description CPU Load
check_command check_nt!CPULOAD!-l 5,80,90
}
define service{
use generic-service
host_name Windows2
service_description Memory Usage
check_command check_nt!MEMUSE!-w 80 -c 90
define service{
use generic-service
host_name Windows2
service_description C: Drive Space
check_command check_nt!USEDDISKSPACE!-l c -w 80 -c 90
}
define service{
use generic-service
host_name Windows2
service_description W3SVC
check_command check_nt!SERVICESTATE!-d SHOWALL -l W3SVC
}
define service{
use generic-service
host_name Windows2
service_description Explorer
check_command check_nt!PROCSTATE!-d SHOWALL -l Explorer.exe
}
define service{ use generic-service
host_name Windows1
service_description SQL port Check
check_command check_tcp!1433
}
define service{
use generic-service
host_name Windows2
service_description SQL port Check
check_command check_tcp!1433
}
The above config, can easily be extended for more hosts, or if necessery easily setup to track more services in nagios web frontend.
6. Test if connectivity to the nsclient++ agent port is available from the Linux server

debian:~# telnet 192.168.58.6 12489
Trying 192.168.58.6...
Connected to 192.168.58.6.
Escape character is '^]'.
asd
ERROR: Invalid password.

Another good idea is to launch on the Windows host the NSClient++ (system tray) , e.g.:

Start, All Programs, NSClient++, Start NSClient++ (system tray).

Test Nagios configuration from the Linux host running nagios and nrpe daemons to check if the check_nt, can succesfully authenticate and retrieve data generated from the nsclient++ on the Windows host:

debian:~# /usr/lib/nagios/plugins/check_nt -H 192.168.1.5 -p 12489 -v CPULOAD -w 80 -c 90 -l 5,80,90,10,80,90

If everything is okay and the remote Windows system 192.168.1.5 has properly configured and running NSClient++ the above command should return an output like:

CPU Load 1% (5 min average) 1% (10 min average) | '5 min avg Load'=1%;80;90;0;100 '10 min avg Load'=1%;80;90;0;100

In case of the command returns:

could not fetch information from server

instead this means that probably there is some kind of problem with authentication or handshake of the Linux host’s nagios check_nt to the Windows server’s running on 12489.

This is sometimes caused by misconfigured NSC.ini file, however in other occasions this error is caused by misconfigured Windows Firewall or because the NSClient++ is not running with Administrator user.

By the way important note to make about Windows 2008r2 is that if NSClient++ is running there it’s absolutely required to Login with Windows Administrator and run the NSClient++ /start , if it’s run through the Run As Adminsitrator with an admin privileged user the aforementioned error might appear, so be careful.
I’ve experienced this error myself and it took me about 40 minutes to find that I have to run it directly with Administrator user after logging as Administrator.

7. Create nagios web iface Apache configuration

nagios debian pachage is shipped with a config which is suitable to be set debian:~# cp -rpf /usr/share/doc/nagios3-common/examples/apache2.conf /etc/apache2/sites-avalable/nagios
debian:~# ln -sf /etc/apache2/sites-available/nagios /etc/apache2/sites-enabled/nagios

The /etc/apache2/sites-available/nagios can easily be configured to work on Virtualhost, to do so the above copied file need to be wrapped inside a VirtualHost directive. For that put in the beginning of the file;

<VirtualHost *:80>

and in the end of the file:

<VirtualHost *:80>

8. Restart nagios server and Apache for the new settings to take effect

debian:~# /etc/init.d/apache2 restart
...
debian:~# /etc/init.d/nagios3 restart

If some custom configuration about tracking the Debian Linux nagios host running services needs to be made, its also helpful for one to check in /etc/nagios3/conf.d

Well that’s mostly what I had to do to make the Nagios3 server to keep track of a small Windows network on Debian GNU/Linux Squeeze 6, hope this small article helps. Cheers 😉

How to set up Path to .exe GNUWin32 binary files in Windows XP / Vista / 2003 / 2008 (Setting PATH to executables on Windows)

Tuesday, August 23rd, 2011

I’ve been working on a servers running Windows 2003 and Windows 2008 these days.
As I wanted to be more flexible on what I can do from the command line I decided to install GNUwin (provides port of GNU tools), most of which are common part of any Linux distribution).
Having most of the command line flexibility on a Windows server is a great thing, so I would strongly recommend GNUWin to any Windows server adminsitrator out there.

Actually it’s a wonderful thing that most of the popular Linux tools can easily be installed and used on Windows for more check GnuWin32 on sourceforge

One of the reasons I installed Gnuwin was my intention to use the good old Linux tail command to keep an eye interactive on the IIS server access log files, which by the way for IIS webserver are stored by default in C:WindowsSystem32LogFilesW3SVC1*.log

I’ve managed to install the GNUWin following the install instructions, not with too much difficulties. The install takes a bit of time, cause many packs containing different parts of the GNUWin has to be fetched.

To install I downloaded the GNUWin installer available from GNUWin32’s website and instructed to extracted the files into C:Program FilesGnuwin
Then I followed the install instructions suggestions, e.g.:

C:> cd c:Program FilesGnuWin
C:Program FilesGnuWin> download.bat
...
C:Program FilesGnuWin> install c:gnuwin32
...

After the installation was succesfully completed on the two Windows machines, both of which by the way are running 64 bit Windows, it was necessery to add the newly installed GNU .exe files to my regular cmd.exe PATH variable in order to be able to access the sed, tail and the rest of the gnuwin32 command line tools.

In order to add C:GnuWin32bin directory to the windows defined Command line Path , I had to do the following:

a. Select (Properties) for My Computer

Start (button) -> My Computer (choose properties)

b. Select the My Computer Advanced (tab)

Then, from the My Computer pane press on Advanced tab

c. Next press on Environment Variables

Windows environment variables screenshot

You see in above’s screenshot the Environment Variables config dialog, to add the new path location in System Variables sectiom, between the list I had to add the c:GNUwin32bin path locatiion. To find I pressed on Edit button scrolled down to find the Variable and hence added at the end of the long list defined paths.
After adding in GNUwin, the Windows path looks like this:

C:Program Files (x86)EWANAPI;C:WINDOWSsystem32;C:WINDOWS;C:WINDOWSSystem32Wbem;C:Program Files (x86)IntelNGSMSMPFiles;C:Program Files (x86)Microsoft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100DTSBinn;C:Program Files (x86)Microsoft SQL Server100ToolsBinnVSShellCommon7IDE;C:Program Files (x86)Microsoft Visual Studio 9.0Common7IDEPrivateAssemblies;C:Program Files (x86)Microsoft SQL Server100DTSBinn;C:WINDOWSsystem32WindowsPowerShellv1.0;C:gnuwin32bin

Further on, I launched the tail command to intercatively take an eye on who is accessing the IIS webserver.
Sadly this worked not, trying to use tail with the IIS ex10116.log log;

C:WindowsSystem32LogfilesW3SVC1> tail -f ex10116.log

Spit an error tail: ex10116.log: Bad file descriptor

Since I couldn’t use tail -f I looked for alternative and a quick search led me to Tail 4 Win32 . Lest the name suggests it is supposed to work on 32 bit arch Windows the version on tailforwin32’s website is working perfectly fine on 64 bit Windows as well.
What it does is to simulate a normal tail -f command inside a very simplistic window interface. You see it in action with opened IIS log on below’s screenshot:

GUI Tail for Windows screenshot

Finally my goal is achieved and I can take an eye interactively on IIS logs. End of the article, hope it wasn’t too boring 😉

How to Split files on Linux FreeBSD, NetBSD and OpenBSD

Sunday, July 31st, 2011

Split large files in pieces Scissors

Did you have the need to sometimes split an SQL extra large files to few pieces in order to be able to later upload it via phpmyadmin?
Did you needed an extra large video or data file to be cut in few pieces in order to transfer it in few pieces over an USB stick?
Or just to give you an another scenario where I sometimes need to have an enormous file let’s say 3G split in few pieces, in order to later read it in vim or mcedit .
I sometimes need to achieve this on FreeBSD and Linux hosts thus I thought it will be helpful to somebody to give a very quick tutorial on the way large files can be cut in pieces on Linux and BSD hosts.

GNU/Linux and FreeBSD are equipped with the split command. The purpose of this command is exactly the cutting of a file to a number of pieces.

On Linux the split command comes by default install to the system with the coreutils package on most Debian (deb) based and Redhat based (rpm) distributions, theerefore Linux’s version of split is GNU/split since it’s part of the GNU Coreutils package. An interesting fact about Linux split is that one of the two programmers who has coded it is Richard Stallman 😉

On BSD Unix split is the AT&T UNIX (BSD) split

In the past splitting files in pieces was much more needed than today, as people used floppy drives to transfer data, though today with the bloom of Internet and the improve of the data carriers transferring even an extra large files from one place to another is a way more trivial task still at many occasions splitting it in pieces is needed.

Even though today splitting file is very rarely required, still there are times when being able to split a file in X number of parts is very much needed.
Maybe the most common use of splitting a file today is necessery when a large SQL file dumps, like let’s say 200 MBytes of info database needs to be moved from ane hosting provider to another one.
Many hosting providers does disallow direct access with standard mySQL client programs to the database directly and only allow a user to connect only via phpMyAdmin or some other web interface like Cpanel to improve data into the SQL or PostgreSQL server.

In such times, having knowledge on the Unix split command is a priceless asset.

Even though on Linux and BSD the code for the split command is not identical and GNU/split and BSD/split has some basic differences, the use of split on both of these Unices is identical.
The way to split a file in few pieces using on both Linux and BSD OSes is being done with one and the same command, here is how:

1. Splitting file in size of 40 mb On Linux

linux:~# split -b 40m SQL-Backup-Data.sql SQL-Backup-Data_split

2. Splitting file in size of 40mb on BSD (FreeBSD, OpenBSD, NetBSD)

freebsd# split -b 40m SQL-Backup-Data.sql SQL-Backup-Data_split

The Second argument the split command takes is actually called a prefix, the prefix is used as a basis name for the creation of the newly generated files cut in pieces file based on SQL-Backup-Data.sql.

As I said identical command will split the SQL-Backup-Data.sql files in a couple of parts which of it will be sized 40 megas.

These command will generate few files output like:

freebsd# ls -1 SQL-Backup-Dat*SQL-Backup-Data.sql
SQL-Backup-Dataa
SQL-Backup-Dataab
SQL-Backup-Dataac
SQL-Backup-Dataad
SQL-Backup-Dataae

As you see the SQL-Backup-Data.sql with size 200MB is being split in four files each of which is sized 40mbytes.

After the files are transfered to another Linux or BSD host, they can easily be again united in the original file with the command:

linux:~# for i in $(ls -1 SQL-Backup-Data_split*); echo $i >> SQL-Backup-Data.sql

Alternatively in most Unices also using cat should be enough to collect back the pieces into the original file, like so:

freebsd# cat SQL-Backup-Data_split* >> SQL-Backup-Data.sql

Enjoy splitting