Rsync slow data (bandwidth limit) transferring on productive Linux / *BSD servers to 2nd

If amount of Unique users on website has increased dramatically and Apache + PHP server starts to get user load higher than 50% in times of most users site activity then it is time to move to think of migrating data on more powerful Server hardware.

Moving few thousands of Gigabytes of PHP, JS, PNG, JPG images and plain text files data from a productive host to another puts an extra burden on hard disk Input / Output (I/O) operations, thus risking to put extraordinary server load and make websites on server inaccessible. The normal way I copy data on less busy servers is create  .tar.gz archive of data from one server and transfer with sftp or scp. In this situation, doing so however puts too much load on server and thus is risking to stone the server and make it inaccessible to users. A solution to problem is to use rsync instead, synchronizing data between the servers by instructing it to transfer data from one hard disk to another via network using a maximum read/write bandwidth.

rsync command argument specifying a maximum bandwidth is --bwlimit=KBPS

To transfer data between two servers specifyinga maximum transfer bandwidth of 10MB per second you have to pass 2MBytes as it is in megabytes (2*1024Kb) = 2048.

Hence to make the transfer while logged to current productive server via SSH to host server with IP XXX.XXX.XXX.XXX I used:
w:~# cd /home/sites
w:/home/sites# /usr/bin/rsync --bwlimit=2048 -avz -e ssh . root@XXX.XXX.XXX.XXX:/home/sites/

The arguments to above rsync command are clear enough (-e ssh) – tells to use ssh as data transfer protocol, (root@) – specifies to connect to second server with root user and (:/home/sites/) – tells rsync to transfer to remote server to same directory (/home/sites/) like from which copying.

Bear in mind that, in order this method to work, rsync has to be installed both on the server from which data is transferred and to second one to where data is transferred.
Since rsync is available in Linux as well as has port in FreeBSD / NetBSD / OpenBSD ports tree, same way to transfer "web data" while upgrading BSD OS host to another is possible.

Share this on

Tags: , , , , , , , , , , , , ,

Leave a Reply