Posts Tagged ‘progress’

How to Copy large data directories between 2 Linux / Unix servers without direct ssh / ftp access between server1 and server2 other by using SSH, TAR and Unix pipes

Monday, April 27th, 2015

how-to-copy-large-data-directories-between-2-linux-unix-servers-without-direct-ssh-ftp-access-btween-each-other

In a Web application data migration project, I've come across a situation where I have to copy / transfer 500 Gigabytes of data from Linux server 1 (host A) to Linux server 2 (host B). However the two machines doesn't have direct access to each other (via port 22) for security reasons and hence I cannot use sshfs to mount remotely host dir via ssh and copy files like local ones.

As this is a data migration project its however necessery to migrate the data finding a way … Normal way companies do it is to copy the data to External Hard disk storage and send it via some Country Post services or some employee being send in Data center to attach the SAN to new server where data is being migrated However in my case this was not possible so I had to do it different.

I have access to both servers as they're situated in the same Corporate DMZ network and I can thus access both UNIX machines via SSH.

Thanksfully there is a small SSH protocol + TAR archiver and default UNIX pipe's capabilities hack that makes possible to transfer easy multiple (large) files and directories. The only requirement to use this nice trick is to have SSH client installed on the middle host from which you can access via SSH protocol Server1 (from where data is migrated) and Server2 (where data will be migrated).

If the hopping / jump server from which you're allowed to have access to Linux  servers Server1 and Server2 is not Linux and you're missing the SSH client and don't have access on Win host to install anything on it just use portable mobaxterm (as it have Cygwin SSH client embedded )

Here is how:
 

jump-host:~$ ssh server1 "tar czf – /somedir/" | pv | ssh server2 "cd /somedir/; tar xf


As you can see from above command line example an SSH is made to server1  a tar is used to archive the directory / directories containing my hundred of gigabytes and then this is passed to another opened ssh session to server 2  via UNIX Pipe mechanism and then TAR archiver is used second time to unarchive previously passed archived content. pv command which is in the middle is not obligitory though it is a nice way to monitor status about data transfer like below:
 

500GB 0:00:01 [10,5MB/s] [===================================================>] 27%


P.S. If you don't have PV installed install it either with apt-get on Debian:

 

debian:~# apt-get install –yes pv

 

Or on CentOS / Fedora / RHEL etc.

 

[root@centos ~]# yum -y install pv

 

Below is a small chunk of PV manual to give you better idea of what it does:

NAME
       pv – monitor the progress of data through a pipe

SYNOPSIS
       pv [OPTION] [FILE]…
       pv [-h|-V]

DESCRIPTION
       pv  allows  a  user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage
       completed (with progress bar), current throughput rate, total data transferred, and ETA.

       To use it, insert it in a pipeline between two processes, with the appropriate options.  Its standard input will be passed
       through to its standard output and progress will be shown on standard error.

       pv  will  copy  each  supplied FILE in turn to standard output (- means standard input), or if no FILEs are specified just
       standard input is copied. This is the same behaviour as cat(1).

       A simple example to watch how quickly a file is transferred using nc(1):

              pv file | nc -w 1 somewhere.com 3000

       A similar example, transferring a file from another process and passing the expected size to pv:

              cat file | pv -s 12345 | nc -w 1 somewhere.com 3000


Note that with too big file transfers using PV will delay data transfer because everything will have to pass through another 2 pipes, however for file transfers up to few gigabytes its really nice to include it.

If you only need to transfer huge .tar.gz archive and you don't bother about traffic security (i.e. don't care whether transferred traffic is going through encrypted SSH tunnel and don't want to put an overhead to both systems for encrypting the data and you have some unfiltered ports between host 1 and host 2 you can run netcat on host 2 to listen for connections and forward .tar.gz content via netcat's port like so:
 

linux2:~$ nc -l -p 12345 > /path/destinationfile
linux2:~$ cat /path/sourcfile | nc desti.nation.ip.address 12345


Another way to transfer large data without having connection with server1 and server2 but having connection to a third host PC is to use rsync and good old SSH Tunneling, like so:
 

jump-host:~$ ssh -R 2200:Linux-server1:22 root@Linux-server2 "rsync -e 'ssh -p 2200' –stats –progress -vaz /directory/to/copy root@localhost:/copy/destination/dir"

Human Modern Progression a move forward to Degression

Saturday, June 16th, 2012

I've been thinking about our modern day progression and what our "progressed" society looks like in reality.
We think we have progressed, we have built machines that serves us well (computers). We plan for a bright painless easy future day by day. This bright "perfect" society future is nothing more than the dream of communism in its complete form.

But in what sense we have actually progressed ?? Everyone who really sees what happens around us notice the big changes we experience. On the surface it looks like our life has become much easier with all this technology surrounding us. The number of computers which is said to help us to leap towards this bright future increases day by day. With this however increases the need to support technology. Suddenly it happens that the old believe that computers are just a tool to make our life earier becomes a modern day slavery. Most of the developed or under development countries people are nowdays almost full time spending in front of the computer screen on the internet. We put our lifes in the mercy of man-created computers and it has become impossible that we live or exist independetly without them.

Computers are everywhere around us starting from the work Desk, at hand with a notebook, ipad, mobile phones, cars, airplanes. There is rarely to see any technology we use which works with not some kind of primitive or advanced computer embedded ,,,
In the rise of computers as we know it computer was just a tool to help us along with the other overall thought and inventions development. Now from just being a tool to help us progress Computers become the common ground on which almost everything in life works.

I'm sure many people who started learning computer technology 15-20 years ago (like me) never imagined computers will integrate so heavily in our daily activities as they eventually did.

We use computers for the sake of planning which in "spiritual language" is predicting the future or prophecise what is about to come in the short future. In reality what we do mostly even not realizingly is to try to predict and modify the future through technology.
This concept is also existing in most sci-fi movies, made in last 60 years. Mobile phones revolution give the humanity a tool through which telepathy we've seen in so many SCI-FI is reality. The mobile phone is just a platform through which (phone calls) or better said voice telepathy has become possible. In that manner of thoughts it is obvious that Video phone calls is a voice + visual telepathy. The Skype revolution and Video and voice conferences is brought was just until recently seen on the sci-fi movies where spaceship crews communicated with other spaceship crews by using a Visual conferences like skype.

It is really hard to believe that for just few years now everyone can speak with ease with everyone else on the planet in the same way just like we've seen in the movies as some foreign abstract concept!

Now suddenly most people on earth are equipped with technology with gives them the power to do everything but it is my firm believe people are not ready to wisely use this power. Therefore instead of using this higher technology wisely technology is used mostly senseless and the more technological advancement grows and becomes more accessible to the masses the more the tendency to use the technology for shit grows.

I'm sure people who have a good knowledge on programming and how computer works are already seriously aware of this enormous problem.
Another severe problem with the raise of technology is the language slang it introduces. This tech-slang is adopted quickly all around the society and suddenly as a result the human language as we know it is seriously substituted by a vague tech words mambo-jambo words. Actually the adoption of tech buzz words in modern day society language makes a great harm for the reason communication between people becomes less descriptive and therefore harder.
In short the result of this tech slang language inside our national languages is inability for people to communicate properly. This tendency is well seen if you for example try to make a comparison between old and newer movies. The newer the movie the less meaningful it is. It is true newer ones has much more as a visual adds than the predecessor but when talking about consistency the newer films are missing this point seriously.

As newer generations are born and raised up with this newer movies and "advanced" TV and computers this people doesn't even have most of the time the opportunity to see older human taped knowledge.
Even for youngesters who have somehow a wise parents enough to teach them in a religious way or just have the "luck" to have parents with old world mindset it will be extremely hard if not impossible for this kids to understand the old knowledge, as most of their same age school / university fellows will only talk about the newer things.

Besides all this, computers as they grow needs more and more support "nurturing" so day by day more and more people has to be busy with managing and supporting tech stuff. Suddenly it is no longer clear if computers serve us or we serve them, this tendency is already somehow evident but not so clearly as it will be in the short 5 to 10 years.
Therefore we slowly but surely are moving to a society which might become "enslaved by technology". Why I say here enslaved, because if we spend our time on fixing computers and technology and working with one virtual reality (which is non-reality) in essence this means we no longer have a physical freedom in the sense it was God given.

There is no doubt computers at present appears to do us a big good, but if you think a bit strategically it is obvious this good has it's price. By adopting all this technology without questioning ourselves on how this will impact our human freedom, we build a computerized jail around us. At first this jail appears to be so wide that it seems it does not interfere with our freedom, but with the introduction of newer and newer computer technology this jail becomes narrower as to the point where it could threat our physical existence freedom.

For those who could argue my thoughs I will ask two simple questions to show you how dependent we've become on technology;;;

What was the last time you switched off your mobile for a week ?

What was the last time you didn't used computers and the internet for a week time ?

Obviously rarely we can find someone that will answer positively to this question or even the thought of switching off from this so globalized society by dropping off tech stuff for a week seems scary.

This constant connectiodness that we're day-by-day heavily exposed to is scary, because it steals little-by-little our natural freedom for seclusion / pravicy / solitude.

This freedoms, were essential and especially for Christian saints and many of the people in the Holy Bible if we read closely we will find out they have used this freedom in parts of their lives especially the seclusion to hear and understand God's will for their life.
Since technology is stealing us the freedom to seclude ourselves this means it steals our basic natural freedom to communicate with God and our natural self ,,,

The consequence of this separation from God and unification with "the world" surely will lead to spiritual blindness and lack of good foundation or higher life purpose, in other words lost path in life.

This is happening all along right in front our eyes now.
Maybe the worst thing of globalization is it doesn't unite people on a soul level but rather separates them. The unification that tech boom gives to people is in the "virtual reality" but this is not a real unification as it is unification in a media which is not real.

Yes Virtual Reality is not real, that's why it is called Virtual isn't it?
I've been thinking over all this problems more and more and I'm starting to come to conclusion that people who wish to keep their essential physical freedom need to GET OUT from this tech lie, we have lived in.
For this however more people need to first realize that;;;

1. TECHNOLOGY LEADS US NOWHERE!

2. People who want to live without technology need to organize in groups (and get used to a natural living growing food, being near to a natural springing water, taking care for each other, living in a Christian commune like – like in the old days)

Actually if we read the old testament's story of Moses escaping the upcoming flood, I believe what is about to come to us as a consequence of this out of boundaries technologization is pretty much like the old testamental flood (this should happen sooner or later).

Moses was wise enough to make himself an Ark and prepare himself for the storm. Today most people are so busy that they don't see the storm coming. I'll be glad to hear from people who has the same thinking as me and want to organize in a groups and live an old humble way of life without technology.
I'm convinced people who have realized all this tech short future bad consequences on humanity, need to have a common communication media and share their knowledge on how we can find a way to live tech free in this age. I'm curious am I the only one with such thoughts or other get into this insight too. If you have come to conclusions I did please contact me in comments. Thank you.

How to clean QMAIL mail server filled Queue with spam mail messages

Friday, June 8th, 2012

Sys Admins managing QMAIL mail servers know, often it happens QMAIL queue gets filled with unwanted unsolicated SPAM e-mails due to a buggy WEB PHP / Perl mail sendingform or some other odd reason, like too many bouncing messages ,,,,

For one more time I’ve experienced the huge SPAM destined mails queue on one QMAIL running on Debian GNU / Linux.

= … Hence I needed to clean up the qmail queue. For this there is a little tool written in PERL called qmhandle

A+) Download qmhandle;

Qmhandle’s official download site is in SourceForge

I’ve made also a mirror copy of QMHandle here

qmail-server:~# cd /usr/local/src
qmail-server:/usr/local/src# wget -q https://www.pc-freak.net/files/qmhandle-1.3.2.tar.gz

B=) STOP QMAIL
As it is written in the program documentation one has to be very careful when cleaning the mail queue. Be sure to stop qmail with qmailctl or whatever script is used to shutdown any mail sever in progress operations, otherwise there is big chance the queue to mess up badly .

C#) Check extended info about the mail queue:

qmail-server:/usr/local/src/qmhandle-1.3.2# ./qmhandle -l -c
102 (10, 10/102) Return-path: anonymous@qmail-hostname.com
From: QMAIL-HOSTNAME
To: as1riscl1.spun@gmail.com
Subject: =?UTF8?B?QWNjb3VudCBpbmZvcm1hdGlvbiBmb3IgU09DQ0VSRkFNRQ==?=
Date: 1 Sep 2011 21:02:16 -0000
Size: 581 bytes
,,,,
1136 (9, 9/1136)
Return-path: werwer@qmail-hostname.com.tw
From: martin.georgiev@qmail-hostname.com
To: costador4312@ukr.net
Subject: Link Exchange Proposal / Qmail-Hostname.com
Date: Fri, 2 Sep 2011 07:58:52 +0100 (BST)
Size: 1764 bytes
,,,,....
1103 (22, 22/1103)
Return-path: anonymous@qmail-hostname.com
From: SOCCERFAME
To: alex.masdf.e.kler.1@gmail.com
Subject: =?UTF8?B?QWNjb3VudCBpbmZvcm1hdGlvbiBmb3IgU09DQ0VSRkFNRQ==?=
Date: 2 Sep 2011 00:36:11 -0000
Size: 578 bytes
,,,,,,
,,,,....
Total messages: 1500 Messages with local recipients: 0
Messages with remote recipients: 1500
Messages with bounces: 500
Messages in preprocess: 300

D-) Delete the Queue
qmail-server:/usr/local/src/qmhandle-1.3.2# ./qmHandle -D
......
......

Fina1ly launch the qmail to continue normal oper.

qmail-server:~# qmailctl start
,,,,,
..,,,