Posts Tagged ‘linux linux’

How to exclude files on copy (cp) on GNU / Linux / Linux copy and exclude files and directories (cp -r) exclusion

Saturday, March 3rd, 2012

I've recently had to make a copy of one /usr/local/nginx directory under /usr/local/nginx-bak, in order to have a working copy of nginx, just in case if during my nginx update to new version from source mess ups.

I did not check the size of /usr/local/nginx , so just run the usual:

nginx:~# cp -rpf /usr/local/nginx /usr/local/nginx-bak
...

Execution took more than 20 seconds, so I check the size and figured out /usr/local/nginx/logs has grown to 120 gigabytes.

I didn't wanted to extra load the production server with copying thousands of gigabytes so I asked myself if this is possible with normal Linux copy (cp) command?. I checked cp manual e.g. man cp, but there is no argument like –exclude or something.

Even though the cp command exclude feature is not implemented by default there are a couple of ways to copy a directory with exclusion of subdirectories of files on G / Linux.

Here are the 3 major ones:

1. Copy directory recursively and exclude sub-directories or files with GNU tar

Maybe the quickest way to copy and exclude directories is through a littke 'hack' with GNU tarnginx:~# mkdir /usr/local/nginx-new;
nginx:~# cd /usr/local/nginx#
nginx:/usr/local/nginx# tar cvf - \. --exclude=/usr/local/nginx/logs/* \
| (cd /usr/local/nginx-new; tar -xvf - )

Copying that way however is slow, in my case it fits me perfectly but for copying large chunks of data it is better not to use pipe and instead use regular tar operation + mv

# cd /source_directory
# tar cvf test.tar --exclude=dir_to_exclude/*\--exclude=dir_to_exclude1/* . \
# mv test.tar /destination_directory
# cd /destination# tar xvf test.tar

2. Copy folder recursively excluding some directories with rsync

P>eople who has experience with rsync , already know how invaluable this tool is. Rsync can completely be used as for substitute=de.a# rsync -av –exclude='path1/to/exclude' –exclude='path2/to/exclude' source destination

This example, can also be used as a solution to my copy nginx and exclude logs directory casus like so:

nginx:~# rsync -av --exclude='/usr/local/nginx/logs/' /usr/local/nginx/ /usr/local/nginx-new

As you can see for yourself, this is a way more readable for the tar, however it will not work on servers, where rsync is not installed and it is unusable if you have to do operations as a regular users on such for that case surely the GNU tar hack is more 'portable' across systems.
rsync has also Windows version and therefore, the same methodology should be working on MS Windows and good for batch scripting.
I've not tested it myself, yet as I've never used rsync on Windows, if someone has tried and it works pls drop me a short msg in comments.
3. Copy directory and exclude sub directories and files with find

Find in collaboration with cp can also be used to exclude certain directories while copying. Actually this method is better than the GNU tar hack and surely more efficient. For machines, where rsync is not installed it is just a perfect way to copy files from location to location, while excluding some directories, here is an example use of find and cp, for the above nginx case:

nginx:~# cd /usr/local/nginx
nginx:~# mkdir /usr/local/nginx
nginx:/usr/local/nginx# find . -type d \( ! -name logs \) -print -exec cp -rpf '{}' /usr/local/nginx-bak \;

This will find all directories inside /usr/local/nginx with find command print them on the screen, then execute recursive copy over each found directory and copy to /usr/local/nginx-bak

This example will work fine in the nginx case because /usr/local/nginx does not contain any files but only sub-directories. In other occwhere the directory does contain some files besides sub-directories the files had to also be copied e.g.:

# for i in $(ls -l | egrep -v '^d'); do\
cp -rpf $i /destination/directory

This will copy the files from source directory (for instance /usr/local/nginx/my_file.txt, /usr/local/nginx/my_file1.txt etc.), which doesn't belong to a subdirectory.

The cmd expression:

# ls -l | egrep -v '^d'

Lists only the files while excluding all the directories and in a for loop each of the files is copied to /destination/directory

If someone has better ideas, please share with me 🙂

How to copy CD or DVD on GNU/Linux and FreeBSD using console or terminal

Monday, November 14th, 2011

CD Burning Console Terminal Linux / FreeBSD picture

These days more and more people start to forget the g* / Linux old times when we used to copy CDs from console using dd in conjunction with mkisofs .

Therefore to bring some good memories back of the glorious console times I decided to come up with this little post.

To copy a CD or DVD the first thing one should do is to make an image copy of the present inserted CD into the CD-drive with dd :

1. Make copy of the CD/DVD image using dd

# dd if=/dev/cdrom of=/tmp/mycd.iso bs=2048 conv=notrunc

/dev/cdrom is the location of the cdrom device, on many Linuces including (Debian) /dev/cdrom is just a link to the /dev/ which corresponds to the CD drive. Note on FreeBSD the location for the CD Drive is /dev/acd0
/tmp/mycd.iso instructs dd CD image creation to be placed in /tmp/ directory.
bs argument instructs it about the byte size portions by which the content of the CD-Drive inserted CD will be read. bs value of 2048 is actually only 2KB per dd read, increasing this value will decrease the time required for the CD image to be extracted.

2. Prepare CD image file to be ready for burning

After dd completes the image copy operation, next to prepare the extracted image / ISO to be ready for burning mkisofs is used:

# mkisofs -J -L -r -V TITLE -o /tmp/imagefile.iso /tmp/mycd.iso

The -J option makes the CD compatible for Pcs running Microsoft Windows. The -V TITLE option should be changed to whatever title the new CD should have, -r will add up status bar for the mkisofs operation.
-r is passed to create specific file permissions on the newly created CD, -o specifies the location where mkisofs will produce its file based on the CD image /tmp/mycd.iso .

3. Burning the mkisofs image file to a CD/DVD on GNU / Linux

linux:~# cdrecord -scanbus
linux:~# cdrecord dev=1,0,0 /tmp/imagefile.iso

If all wents okay with cdrecord operation, after a while the CD should be ready.

4. Burning the mkisofs image file to CD on FreeBSD

freebsd# burncd -f /dev/acd0 data /tmp/imagefile.iso fixate