Posts Tagged ‘Hope’

Virtualbox Shared folder set up on Linux between Host and Guest OS – Set up Virtualbox shared folder to Copy files from PC Host to Guest

Wednesday, September 12th, 2018

mount-shares-between-host-OS-and-guest-virtual-machine-howto-virtualbox-vbox-logo

How to set-up Virtualbox shared folder to Copy files from PC Host  and Guest Virtualized OS?

Running VirtualBox Host is an easy thing to set-up across all Operating Systems.  Once you have it sooner or later you will need to copy files from the VM Host OS (that in my case is GNU / Linux) to the virtualized Guest operating system (again in my case that's again another Linux ISO running indide the Virtual Machine).

Below are steps to follow To use Virtualbox Shared Folder functionality to copy files between VBox and your Desktop / server Linux install.

1. Install Virtualbox Guest Additions CD Image ISO

I've explained how to add the Guest Additions CD image thoroughfully in my previous article Howto enable Copy / Paste Virtualbox betwen Linux guest and Host OS
Anyways I'll repeat myself below for sake of clarity:

To do so use Oracle VBox menus (on the booted virtualized OS VBox window):

 

Devices -> Insert Guest additions CD Image

 

Mount the ISO inside the Linux Virtual Machine:

root@debian:~# mount /media/cdrom1/
 

If the mount fails and there are no files inside the mount point it might be because the virtualbox-dkms and virtualbox-guest-dkms packages might be missing on the Host OS.

To install them (on Debian GNU / Linux) assuming that you're using virtualbox default distro packages /etc/apt/sources.list :
 

apt-get install –yes -qq virtualbox-dkms virtualbox-guest-dkms


and run:

 

root@debian:/media/cdrom1# cd /media/cdrom1; sh VBoxLinuxAdditions.run


2. Create directory for Shared Folder that will be used to access Host / OS files from the Guest Virtualized OS
 

root@debian:~# mkdir /mnt/shared_folder

 

3. Map from VBox program interface Shared folder settings and Mount /mnt/shared_folder location

virtualbox-virtual-machine-devices-shared-folders-shared-folder-settings-linux-screenshot

 

Devices -> Shared Folder -> Shared Folder settings -> Transient Folders (click blue folder add small button right)

 

From Transient Folders add whatever directory you want to be shared from your local notebook / PC to the VM.

virtualbox-devices-Shared-Folder-Add-Shared-Folder-add-share-linux-screenshotDepending on whether you would like to mount the shared folder only for reading files (choose Read Only) to make it a permanent shared folder (and not just for the one session of current running Virtual Machine until its killed use Make Permanent) or check Auto-Mount tick if you want the shared_folder mapping to be mounted on every VM boot.

Once the shared_folder directory location is set-up from GUI menu click OK and in order for the settings to take effect, you'll need to restart the VM Guest with Linux (use halt command from terminal) or Power Off the Machine via the VBox menus.

To mount use command like:

mount -t vboxsf name_of_folder_linked_from_vbox  /mnt/name_folder_guest_os/


mount-vboxsf-shared-folder-mnt-shared-linux-guest-screenshot

In my case I wanted to share home folder /home so the command I used is:

root@debian:~# mount -t vboxsf  shared_folder /mnt/shared_folder


If everything is fine your Host OS file content from /home will be visible (for read and write if you Mapped it so) 
under /mnt/shared_folder …

And as Turtles Ninja used to heavily say Cowabunga !!! 🙂
You have it mounted and ready for file share between Desktop -> Virtualized OS.

 

Bear in mind that above mount command has to run as root (superuser) to succeed.

You now could copy files from your Host OS (running the Virtual Machine) and the Guest OS (Virtualized OS) using /mnt/shared_folder mount point without problems.

The example is if you want to share files between VirtualBox installed Linux and the Guest (Desktop / server) OS, however at many cases mounting your Host OS directory for root users might be not very practical but, instead you might prefer to do the mount for specific non admin user, for example I prefer to do the shared folder mount with my pointed non-root username hipo.

Here is how to do above VM shared_folder mount for non-root user:

First you need to know the exact UID / GID (User ID / Group ID) of user, you can get that with id command:

 

hipo@linux:~$  id
uid=1000(hipo) gid=1000(hipo) groups=1000(hipo),24(cdrom),25(floppy),29(audio),30(dip),44(video),46(plugdev),108(netdev),114(bluetooth),115(lpadmin),119(scanner)

 

As you see UID / GID in my case are 1000 / 1000

hipo@linux:~$ sudo mount -t vboxsf -o rw,uid=1000,gid=1000 shared_folder /mnt/shared_folder

 

mount-virtual-box-shared_folder-with-non-administration-permissions-non-root-permissions-id-and-mount-command-screenshot-linux


4. Mounting configured shared_folder to automatically mount into the Guest OS Linux on every boot

a) Configuring shared_folder auto-mount using /etc/rc.local

If you need the shared_folder to automatically mount next-time you boot the virtual machine quickest way is to add the mount command to /etc/rc.local (on Debian 8 and Debian 9 and newer Ubuntu Linuxes rc.local is missing by default to enable it to work like it worked before read follow my previous article ).

b) Configuring auto-mount for shared_folder through /etc/fstab

The more professional way to auto-mount on emulated OS VM boot time,  you could add the vboxsf mount definitions to /etc/fstab with your favourite text-editor mcedit, nano, joe etc. … (for me that's vim).

Syntax of /etc/fstab is as follows:
 

<Device> <Mount Point> <Type> <Options> <Dump> <Pass>

root@linux:~# vim /etc/fstab

 

shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0 0

Note that you will want to change 1000 / 1000, id / gid with the ones of the non-admin user you would like to add to mount it for.

A quick way to add it to /etc/fstab with a shell one-liner is with command
 

root@linux:~# echo 'shared_folder /mnt/shared_folder                                vboxsf rw,uid=1000,gid=1000 0  0' >> /etc/fstab

An alternative way to add a user to have permissions for vboxsf file system (without specifying the long -o uid=1000,gid=1000 options is to simply add the username in question to group vboxsf like so:

c) Adding non super user username to vboxsf group

root@linux:~# usermod -G vboxsf hipo
root@linux:~# grep -i vboxsf /etc/group
vboxsf:x:999:hipo

 

hipo@linux:~$ sudo mount -t vboxsf  shared_folder /mnt/shared_folder

 

without the extra arguments and the options to pass to /etc/fstab (for eventual requirement to auto mount the shared_folder) would be more simple e.g.:

 

echo 'shared_folder /mnt/shared_folder                                vboxsf ' >> /etc/fstab

 

One note to make here is if the uesr is added to vboxsf the line for /etc/fstab to auto mount to mount for root user and non-root will be identical.

Then you can get the /etc/fstab auto-mount configured tested by running:

c) Checking auto-mount is working

hipo@linux:~# mount -a
hipo@linux:~# mount |grep -i vboxsf
shared_folder on /mnt/shared_folder type vboxsf (rw,nodev,relatime)


5. What if you end up with mounting failed errors ? – What might be causing the mounting failed Protocol error (a few things to check to solve)


In case of troubles with the mount you might get an error like:

hipo@linux:~# mount -t vboxsf  share_folder /mnt/shared_folder

/sbin/mount.vboxsf: mounting failed with the error: Protocol error


This error might be caused because of Insert Guest Additions CD Image might be not properly enabled and installed using the ISO provided VBoxLinuxAdditions.sh shell script.
Other common reason you might get this error if you have mistyped the Folder name: given in Shared Folders -> Folder Path -> Add Share for example I have given shared_folder as a Map name but as you can see in above mount -t vboxsf, I've mistyped share_folder instead of the correct one shared_folder inserted.
In some VBox releases this error was caused by bugs in the Virtual Machine.
 

virtualbox-virtual-machine-shared-folder-transient-folder-add-folder-linux-VM-guest-linux

One useful tip is to be able to check whether a Virtualbox Virtual Machine has a configured shared_folder (if you're logging to manage the machine on remote server – nomatter whether you have logged in with VNC / Teamviewer / Citrix etc. or via SSH session.

To do so use VBoxControl as of time of writting usually located on most distributions under (/usr/bin/VBoxControl)
 

 

hipo@linux:~# VBoxControl sharedfolder list -automount
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Auto-mounted Shared Folder mappings (0):

No Shared Folders available.

You can use VBoxControl command to get set and list a number of settings on the VBox VM, here is an useful example with it where you get information about numerous VBox info values:

 

root@linux:~# VBoxControl guestproperty enumerate
Oracle VM VirtualBox Guest Additions Command Line Management Interface Version 5.2.18
(C) 2008-2018 Oracle Corporation
All rights reserved.

 

Name: /VirtualBox/GuestInfo/OS/Product, value: Linux, timestamp: 1536681633430852000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/V4/IP, value: 10.0.2.15, timestamp: 1536681633438717000, flags: <NULL>
Name: /VirtualBox/HostInfo/GUI/LanguageID, value: en_US, timestamp: 1536697521395621000, flags: RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/MAC, value: 08002762FA1C, timestamp: 1536681633442120000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/ServicePack, value: <NULL>, timestamp: 1536681633431259000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVerExt, value: 5.2.18, timestamp: 1536681619002646000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Netmask, value: 255.255.255.0, timestamp: 1536681633440157000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Version, value: #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13), timestamp: 1536681633431125000, flags: <NULL>
Name: /VirtualBox/GuestAdd/VersionExt, value: 5.2.18, timestamp: 1536681633431582000, flags: <NULL>
Name: /VirtualBox/GuestAdd/Revision, value: 124319, timestamp: 1536681633432515000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepExec, value: <NULL>, timestamp: 1536681619002355000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsers, value: 1, timestamp: 1536681673447293000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/0/Status, value: Up, timestamp: 1536681633443911000, flags: <NULL>
Name: /VirtualBox/GuestInfo/Net/0/Name, value: enp0s3, timestamp: 1536681633445302000, flags: <NULL>
Name: /VirtualBox/HostGuest/SysprepArgs, value: <NULL>, timestamp: 1536681619002387000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestAdd/Version, value: 5.2.18, timestamp: 1536681633431419000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxRev, value: 124319, timestamp: 1536681619002668000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/Net/0/V4/Broadcast, value: 10.0.2.255, timestamp: 1536681633439531000, flags: <NULL>
Name: /VirtualBox/HostInfo/VBoxVer, value: 5.2.18, timestamp: 1536681619002613000, flags: TRANSIENT, RDONLYGUEST
Name: /VirtualBox/GuestInfo/OS/LoggedInUsersList, value: hipo, timestamp: 1536681673446498000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestInfo/Net/Count, value: 1, timestamp: 1536698949773993000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/Release, value: 4.9.0-7-amd64, timestamp: 1536681633431001000, flags: <NULL>
Name: /VirtualBox/GuestInfo/OS/NoLoggedInUsers, value: false, timestamp: 1536681673447965000, flags: TRANSIENT, TRANSRESET
Name: /VirtualBox/GuestAdd/HostVerLastChecked, value: 5.2.18, timestamp: 1536681702832389000, flags: <NULL>

Hope you enjoyed ! Have phun! 🙂

How to Increase virtualbox Linux install machine VM .VDI hard disk size to free space on root partition – Move /usr to a new partition

Tuesday, October 10th, 2017

extend-vdi-virtual-machine-partition-with-virtualbox-howto-virtualbox-logo
 

How to Increse Hard Disk size of VirtualBox Virtual Machine .VDI file to Free Space on root Partition or Howto move /usr large partition to separate new partition?


I just wondered how to increase hard disk size of Virtualbox Virtual Machine image .VDI, because for some stupid reason I've created my initial hard disk size for Linux partition to be the default 10 Gigabytes.

The problem is the packages I need to have installed on the Virtual Machine which will be a testbed for future tests of a production website applications are taking up too much space, once installed so I'm left with no space
in /var/lib/mysql for the database import. So what can I do in that case is to simply free up disk space or Merge ROOT partition with another partition.

Since merging the partition is not a trivial job and would require me to have installable CD with the Linux distro (in my case that's Debian Linux) or have a bootable USB flash drive, I preferred second approach to problem e.g. to free up disk space on ROOT partition by creating a second partition and move the /usr folder to reside there.

Before that it is of course necessery to  have extended the .VDI file using VirtualBox, so more space than the default 10GB preconfigured are available, this is easily done on Windows OS as, VBox is provided with GUI clickable option to do it, but for who knows what reason that is not the case with Linux, so Linux users only option to increase VDI file is to manually run command part of the virtualbox package, that is not a hard task really but it requires some typing and basic knowledge on how to run commands in terminal.


To .VDI resize (extend), we first go to default location where VirtualBox stores its image .VDI files (by default as of moment of writting this article – this is ~/"VrtualBox VMs"  (or home directory of logged in user dir VirtualBox VMs), the command to use is VBoxManage

 

root@jericho:/home/hipo# cd VirtualBox VMs/
root@jericho:/home/hipo/VirtualBox VMs# ls
Debian 6  Debian 9  Windows 10
root@jericho:/home/hipo/VirtualBox VMs# cd Debian 6/
r
oot@jericho:/home/hipo/VirtualBox VMs/Debian 6# ls
Debian 6.vbox  Debian 6.vbox-prev  Debian 6.vdi  Logs  NewVirtualDisk1.vdi  Snapshots

root@jericho:/home/hipo/VirtualBox VMs/Debian 6# VBoxManage modifyhd Debian 6.vdi –resize 20000
0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%
root@jericho:/home/hipo/VirtualBox VMs/Debian 6#


 

Above command does resize the 10GB default created partition for Linux, where I have installed Linux which was 99% full of data, because of the many packages I installed to 20GB size, to make it bigger just use the respective size, be it 30000 (for 30GB) or 100000 (for 100GB) etc.

Even though in this example VBoxManage virtual partition resize command was done for GNU / Linux Operating System, it can be done for any other Operating as well to resize the size of the Virtual .VDI file (Virtual Machine) partition, be it Windows 7 / 8 / 10 or the rest of Free Operating systems FreeBSD / OpenBSD / BSD that are installed in a VM etc.

Next Launch the Virtual Machine with VBox Server client Program and install there Gparted (GNU Parted), as we'll need it to create a new Hard Disk Partition:

 

$ VirtualBox

 


Inside virtualmachine's in gnome-terminal / xterm etc. depending on the graphical environment used do install with apt-get:

 

debian:# apt-get install –yes gparted

 

debian~:# gparted


Notice that gparted has to be ran as a root superuser.

 

Run GParted and create new EXT3 filesystem that is 10GB (the size of the new created partition).

If you have installed Debian to place all partitions under / (root directory /dev/sda1) then the fresh new partition to create should be
/dev/sda3, anyways just look closely in EXT3 and in your case if the partiition is named differently create according to proper partition /dev/ naming.

I'll not run into details on how to create the partition with GParted as the program interface is very self-explanatory, the only thing is to apply the update to create partition and the ext3 filesystem, that's being done
with a green tick:

gparted-create-ext3-partition-howto-linux

Next step is to check with fdisk whether, we have ext3 properly created  filesystem as we've done already with GPARTED:

Once we have the partition created with EXT3 filesystem, we're ready to move /usr temporary to other folder, I use usually /root for the move but you can create anywhere a new folder for that and move to there.

To move to /root directory run again in terminal:

 

debian:~# mv /usr /root
debian:~# mkdir /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux

 

Note that during the move operations, your Desktop icons will become without (with broken) pictures and the default Debian background picture is to disappear, that's because the GUI environment will soon realize /usr/ libraries that're periodically reloaded in memory are missing and will be unable to reload them as it does in a cycle.

That should take a few minutes, so grab a coffee or if you're a smoker (hope not as smoking kills 🙂 ), in 5 / 10 minutes time depending on your computer / server configuration, it will be over, so we're ready to create new /usr dir and mount the  new partition:

 

debian:~# mount /dev/sda3 /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux1

 

Now we check with mount command whether mount is fine:

 

check-if-filesystem-is-properly-mounted-linux-debian-screenshot


Now  /dev/sda3 is mounted under /usr  and we have to move back /root/usr directory content back to the newly mounted /usr so we run command:
 

debian:~#  mv /root/usr/* /usr/*


Finally we need to create proper records for the new partition inside /etc/fstab (fstab –FileSystem Tab file – the file which describes instructs the Linux OS what partition to boot where, what)

HOW TO CHECK LINUX UUID FOR A PARTITION??

Before adding anything to /etc/fstab you need to check the UUID of /dev/sda3 (or whatever the partition is called), without proper UUID, the system might fail to boot.
So here is how to check the UUID we'll need for config:

 

hipo@debian:~$ /sbin/blkid /dev/sda3
/dev/sda3: UUID="2273db4b-3069-4f78-90fc-e7483c0305bd" SEC_TYPE="ext2" TYPE="ext3"

hipo@debian:~$ ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 100 Oct  6 05:16 .
drwxr-xr-x 6 root root 120 Oct  6 05:16 ..
lrwxrwxrwx 1 root root  10 Oct  6 05:16 2273db4b-3069-4f78-90fc-e7483c0305bd -> ../../sda3
lrwxrwxrwx 1 root root  10 Oct  6 05:16 b98d92cd-41aa-4e18-a474-9b8df445dbe5 -> ../../sda1
lrwxrwxrwx 1 root root  10 Oct  6 05:16 f27f7448-f200-4983-b54f-b9e5206f77ac -> ../../sda5

As you can see our /dev/sda3 UUID is 2273db4b-3069-4f78-90fc-e7483c0305bd

Further on lets view and edit /etc/fstab you can also download a copy of my Virtual Machine fstab here

 

debian:~# cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=b98d92cd-41aa-4e18-a474-9b8df445dbe5 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=f27f7448-f200-4983-b54f-b9e5206f77ac none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0


We need to add following line to  /etc/fstab:
 

UUID=2273db4b-3069-4f78-90fc-e7483c0305bd    /usr        ext3 error=remount-ro    0    1

 


Open the file with your favourite text editor (gedit / nano / pico / vim / joe) etc.

debian:~# vim /etc/fstab

 

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=b98d92cd-41aa-4e18-a474-9b8df445dbe5 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=f27f7448-f200-4983-b54f-b9e5206f77ac none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0

UUID=2273db4b-3069-4f78-90fc-e7483c0305bd     /usr        ext3 error=remount-ro    0    1    

Basicly it should be possible to add (for historic reasons) also instead of UUID=2273db4b-3069-4f78-90fc-e7483c0305bd  /dev/sda3
So it looks like so but, the better practice is to use UUID line given.

Well that's all folks now /usr directory will contain all your heavy root partition (disk filling) libraries and files, so you can happily use /var/lib/mysql or /var/lib/postgresql /var/www to store your web application files and import your databases.

Big thanks to Ubuntu Forums article – How do I increase the hard disk size of the Virtual Machine article for inspiring this post.

Hope that helps anyone and thanks and other comments are always welcome ! 🙂

Block Web server over loading Bad Crawler Bots and Search Engine Spiders with .htaccess rules

Monday, September 18th, 2017

howto-block-webserver-overloading-bad-crawler-bots-spiders-with-htaccess-modrewrite-rules-file

In last post, I've talked about the problem of Search Index Crawler Robots aggressively crawling websites and how to stop them (the article is here) explaning how to raise delays between Bot URL requests to website and how to completely probhit some bots from crawling with robots.txt.

As explained in article the consequence of too many badly written or agressive behaviour Spider is the "server stoning" and therefore degraded Web Server performance as a cause or even a short time Denial of Service Attack, depending on how well was the initial Server Scaling done.

The bots we want to filter are not to be confused with the legitimate bots, that drives real traffic to your website, just for information

 The 10 Most Popular WebCrawlers Bots as of time of writting are:
 

1. GoogleBot (The Google Crawler bots, funnily bots become less active on Saturday and Sundays :))

2. BingBot (Bing.com Crawler bots)

3. SlurpBot (also famous as Yahoo! Slurp)

4. DuckDuckBot (The dutch search engine duckduckgo.com crawler bots)

5. Baiduspider (The Chineese most famous search engine used as a substitute of Google in China)

6. YandexBot (Russian Yandex Search engine crawler bots used in Russia as a substitute for Google )

7. Sogou Spider (leading Chineese Search Engine launched in 2004)

8. Exabot (A French Search Engine, launched in 2000, crawler for ExaLead Search Engine)

9. FaceBot (Facebook External hit, this crawler is crawling a certain webpage only once the user shares or paste link with video, music, blog whatever  in chat to another user)

10. Alexa Crawler (la_archiver is a web crawler for Amazon's Alexa Internet Rankings, Alexa is a great site to evaluate the approximate page popularity on the internet, Alexa SiteInfo page has historically been the Swift Army knife for anyone wanting to quickly evaluate a webpage approx. ranking while compared to other pages)

Above legitimate bots are known to follow most if not all of W3C – World Wide Web Consorium (W3.Org) standards and therefore, they respect the content commands for allowance or restrictions on a single site as given from robots.txt but unfortunately many of the so called Bad-Bots or Mirroring scripts that are burning your Web Server CPU and Memory mentioned in previous article are either not following /robots.txt prescriptions completely or partially.

Hence with the robots.txt unrespective bots, the case the only way to get rid of most of the webspiders that are just loading your bandwidth and server hardware is to filter / block them is by using Apache's mod_rewrite through

 

.htaccess


file

Create if not existing in the DocumentRoot of your website .htaccess file with whatever text editor, or create it your windows / mac os desktop and transfer via FTP / SecureFTP to server.

I prefer to do it directly on server with vim (text editor)

 

 

vim /var/www/sites/your-domain.com/.htaccess

 

RewriteEngine On

IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

SetEnvIfNoCase User-Agent "^Black Hole” bad_bot
SetEnvIfNoCase User-Agent "^Titan bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^TeleportPro" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^Website Quester" bad_bot
SetEnvIfNoCase User-Agent "^WebZip" bad_bot
SetEnvIfNoCase User-Agent "^moget/2.1" bad_bot
SetEnvIfNoCase User-Agent "^WebZip/4.0" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^httplib" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^Harvest/1.5" bad_bot
SetEnvIfNoCase User-Agent "Bullseye/1.0" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4.0 (compatible; BullsEye; Windows 95)" bad_bot
SetEnvIfNoCase User-Agent "^Crescent Internet ToolPak HTTP OLE Control v.1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerSE/1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker /1.0" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit/3.50" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control – 5.01.4511" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial/1.34" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control – 6.00.8169" bad_bot
SetEnvIfNoCase User-Agent "^URLy Warning" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Mata Hari" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^Web Image Collector" bad_bot
SetEnvIfNoCase User-Agent "^The Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish/1.0" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc/4.2" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^VCI WebViewer VCI WebViewer Win32" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^QueryN Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^Openfind data gathere" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^Xenu’s Link Sleuth 1.1c" bad_bot
SetEnvIfNoCase User-Agent "^Xenu’s" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey Bait & Tackle/v1.01" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 32297 Webster Pro V2.9 Win32" bad_bot
SetEnvIfNoCase User-Agent "^Webster Pro" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a Unix" bad_bot
SetEnvIfNoCase User-Agent "^Keyword Density/0.9" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin Spider" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot

 

<Limit GET POST>
order allow,deny
allow from all
Deny from env=bad_bot
</Limit>

 


Above rules are Bad bots prohibition rules have RewriteEngine On directive included however for many websites this directive is enabled directly into VirtualHost section for domain/s, if that is your case you might also remove RewriteEngine on from .htaccess and still the prohibition rules of bad bots should continue to work
Above rules are also perfectly suitable wordpress based websites / blogs in case you need to filter out obstructive spiders even though the rules would work on any website domain with mod_rewrite enabled.

Once you have implemented above rules, you will not need to restart Apache, as .htaccess will be read dynamically by each client request to Webserver

2. Testing .htaccess Bad Bots Filtering Works as Expected


In order to test the new Bad Bot filtering configuration is working properly, you have a manual and more complicated way with lynx (text browser), assuming you have shell access to a Linux / BSD / *Nix computer, or you have your own *NIX server / desktop computer running
 

Here is how:
 

 

lynx -useragent="Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)" -head -dump http://www.your-website-filtering-bad-bots.com/

 

 

Note that lynx will provide a warning such as:

Warning: User-Agent string does not contain "Lynx" or "L_y_n_x"!

Just ignore it and press enter to continue.

Two other use cases with lynx, that I historically used heavily is to pretent with Lynx, you're GoogleBot in order to see how does Google actually see your website?
 

  • Pretend with Lynx You're GoogleBot

 

lynx -useragent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -head -dump http://www.your-domain.com/

 

 

  • How to Pretend with Lynx Browser You are GoogleBot-Mobile

 

lynx -useragent="Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" -head -dump http://www.your-domain.com/

 


Or for the lazy ones that doesn't have Linux / *Nix at disposal you can use WannaBrowser website

Wannabrowseris a web based browser emulator which gives you the ability to change the User-Agent on each website req1uest, so just set your UserAgent to any bot browser that we just filtered for example set User-Agent to CheeseBot

The .htaccess rule earier added once detecting your browser client is coming in with the prohibit browser agent will immediately filter out and you'll be unable to access the website with a message like:
 

HTTP/1.1 403 Forbidden

 

Just as I've talked a lot about Index Bots, I think it is worthy to also mention three great websites that can give you a lot of Up to Date information on exact Spiders returned user-agent, common known Bot traits as well as a a current updated list with the Bad Bots etc.

Bot and Browser Resources information user-agents, bad-bots and odd Crawlers and Bots specifics

1. botreports.com
2. user-agents.org
3. useragentapi.com

 

An updated list with robots user-agents (crawler-user-agents) is also available in github here regularly updated by Caia Almeido

There are also a third party plugin (modules) available for Website Platforms like WordPress / Joomla / Typo3 etc.

Besides the listed on these websites as well as the known Bad and Good Bots, there are perhaps a hundred of others that might end up crawling your webdsite that might or might not need  to be filtered, therefore before proceeding with any filtering steps, it is generally a good idea to monitor your  HTTPD access.log / error.log, as if you happen to somehow mistakenly filter the wrong bot this might be a reason for Website Indexing Problems.

Hope this article give you some valueable information. Enjoy ! 🙂

 

Apache Webserver: How to Set up multile SSL certificates on multiple domains running on one IP address with Apache SNI feature

Wednesday, September 13th, 2017

apache-ssl-handshake-how-client-talks-to-server-illustrated

In the recent past it was impossible to add multiple different SSL .crt / .pem bundle certificates on Apache Webserver but each one of it was supposed to run under a separate domain or subdomain, preconfigured with a separate IP address, this has changed with the introduction of Apache SNI (Server Name Indication). What SNI does is it sends, the site visitor initiating connections on encrypted SSL port (443) or whatever configured a certificate that matches, the client requested server name.

Note that SNI is Apache HTTPD supported only and pitily can't be used on other services such as Mail Servers (SMTPS), (POP3S), (IMAPS) etc.
Older browsers did not have support for proper communication with WebServers supporting SNI communication, so for Websites whose aim is interoperatibility and large audience of Web clients still the preferrable way is to set up each VirtualHost under a separate IP, just like the good old days.

However Small and MidSized businesses could save some cash by not having to buy separate IPs for each Virtualhost, but just use SNI.
Besides that the people are relatively rarely using old browsers without SNI, so having clients with browsers not supporting SNI would certiainly be too rare. To recognize where a browser is having support for TLS or not is to check whether the Browser has support for TLS extension.

One requirement in order for SNI to work properly is to have registered domain because SNI works based on the requested ServerName by client.

On Debian GNU / Linux based distributions, you need to have Apache Webserver installed with enabled mod_ssl module:

 

linux:~# apt-get install –yes apache2

linux:~# a2enmod ssl

linux:~# /etc/init.d/apache2 restart


If you're not planning to get a trusted source certificate, especially if you're just a start-up business which is in process of testing the environment (you still did not ordered certificate via some domain registrar you might want to generate self signed certificate with openssl command and use that temporary:

 

linux:~# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/your-domain.com/apache.key –out /etc/apache2/ssl/your-domain.com/apache.crt

Here among the prompted questions you need the a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank.
For some fields leave the default value,
If you enter '.', or press enter the field will be left blank.

—–
 

Country Name (2 letter code) [AU]:BG
State or Province Name (full name) [Some-State]: Sofia
Locality Name (eg, city) []:SOF
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Pc-Freak.NET
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:your-domain.com                
Email Address []:webmaster@your-domain.com

 


(by the way it might be interesting to mention here the list of cheapest domain name registrars on the Internet as of January 2017 – source site here

 

Below order is given as estimated by price /  quality and provided service approximate

 

1. BlueHost.com – Domains $6.95

2. NameCheap.Com – Annual fee $10.69

3. GoDaddy.com – Annual fee $8.99 for first year, $14.99$ for each additional year

4. HostGator.com – Annual fee $15.00

5. 1and1.com – Annual fee $0.99 for first year ($14.99 for each additional year)

6. Network Solutions – This was historically one of the first domain registrar companies, but the brand is pricy $34.99

7. Register.com – Not sure

8. Hostway.com – $9.95 (first year and $9.95 renewals)

9. Moniker.com – Annual fee $11.99

10. Netfirms.ca – Annual fee $9.95 first year, Renewal fee is $11.99 per year

 

Note that domain pricing could value depending on the type of domain name country extension and many of the domain registrars would give you discount if you purchase domain name / SSL for 2 / 3+ years.

sni-illustrated-how-it-works-how-to-configure-multiple-domains-ssl-on-same-ip-apache-webserver

Next step in order to use SNI is to configure the WebServer Virtualhosts file:

 

linux:~# vim /etc/apache2/sites-available/domain-names.com

 

# Instruct Apache to listen for connections on port 443
Listen 443
# Listen for virtual host requests on all IP addresses
NameVirtualHost *:443

# Go ahead and accept connections for these vhosts
# from non-SNI clients
SSLStrictSNIVHostCheck off

<VirtualHost *:80>
        ServerAdmin webmaster@your-domain.com
        ServerName your-domain.com
        DocumentRoot /var/www

# More directives comes here

</VirtualHost>


<VirtualHost *:443>

        ServerAdmin webmaster@localhost
        ServerName your-domain.com
        DocumentRoot /var/www

        #   SSL Engine Switch:
        #   Enable/Disable SSL for this virtual host.
        SSLEngine on

        #   A self-signed (snakeoil) certificate can be created by installing
        #   the ssl-cert package. See
        #   /usr/share/doc/apache2.2-common/README.Debian.gz for more info.
        #   If both key and certificate are stored in the same file, only the
        #   SSLCertificateFile directive is needed.
        SSLCertificateFile /etc/apache2/ssl/your-domain.com/apache.crt
        SSLCertificateKeyFile /etc/apache2/ssl/your-domain.com/apache.key

# More Apache directives could be inserted here
</VirtualHost>

 

<VirtualHost *:443>
  DocumentRoot /var/www/sites/your-domain2
  ServerName www.your-domain2.com

  # Other directives here

</VirtualHost>

Add as many of the SNI enabled VirtualHosts following the example below, or if you prefer seperate the vhosts into separate domains.

I also recommend to check out Apache's official documentation on SNI for NameBasedSSLVhostsWithSNI etc.


Hope this article was not too boring 🙂
Enjoy life

 

Coptic Patriarch Preaching on Hope – We Christians should never loose hope! God is with us!

Sunday, March 17th, 2013

patriarch__Shenouda-of-coptic-oriental-orthodox-Church

Whoever believes in Christ he has an everlasting life, as Christ promised us. Faith, Hope and Love are the three essential things we should always keep in heart. I myself being a Christian often forget about importance of Hope. It is one of the spiritual weapons we Christians should hold firmly in our life. I've been in situation, where all around is failing and I'm miserable and powerless, in those moments I should remember more often the Holy Scripture advices that Hope should always be with us. Below preach is the words of Pope Shenouda the Coptic Oriental Church patriach, I believe we can only learn and astonish on his graceful words.


 

His Holiness Pope Shenouda Words on Hope – Hope Christians unfailing all time weapon

Let by this holy patriarch words, God bless Christians with a good hope. It is the beginning of the Great fasting in the Orthodox Church worldwide, so now we need hope more than ever! By the prayers of all saints oh Lord Jesus Christ, have mercy on us sinners …

Sofia Bulgaria one of the Most Ancient cities in Central Europe – the History of Europe

Friday, January 18th, 2013

Many Bulgarians might be striken to find out, today's territory of Bulgaria and on the balkans is in practice one of the ancient inhabitant territories in Europe. Sofia or Serdika, Sredets as the old name of Sofia used to be part of Roman Empire (Byzantine Empire), then of pre-Christian Bulgaria, after Christianization of Bulgaria part of the Bulgarian Empire and then after the fall of Bulgaria under turkish yoke part of Ottoman's Turkish Empire. Maybe surprisingly for many Western Europeans the city of Sofia, happened to be one of the ancient cities important for Christianity as it had a Bishop standing and a strong Christian community since 3rd century A.D. Even until today Sofia has kept its city design which is made in a Roman fashion. Hope this little video unusually made by British will shed up some light to people interested in Ancient Christian history. Enjoy 🙂

 

Sofia Bulgaria, one of the most ancient cities in Europe – the History of Europe

Happy Christ’s Birth! Merry Christmas

Saturday, December 24th, 2011

Christ's Birth Orthodox icon, Merry Christmas

Christ has born! Merry Christmas and Happy Holiday to all my readers!

The meaning of Merry Christmas comes from Christ Mass , the word has changed to (Christmas) through the years.
The origins of the word comes from the western world and especially has a tight relatin with Latin language and the Roman Catholic Church.

In Eastern Orthodox Churches, we say Chestit Rojdestvo Hristovo , which translated to English is Happy Christ's Birth by this tring to put more accent on the real significance of the feast, which is Christ's birth.
It's a pity nowdays many people forgot the real meaning of Merry Christmas but just celebrate without really realizing the deep spiritual meaning this two beautiful words contain.
Our Bulgarian Orthodox Church will mark the Christ's birth with a Holy Liturgy early in Sunday's morning tomorrow 25th of December.

With these bright feast I wish to all my blog readers;
A lot of Best of Health, Good Fortune, Loving Kindness, Hope, Faith, Love and Personal and Professional Success!
May God bring in our hearts the Angels joy which saluted our Saviour Jesus Christ's birth 2011 years ago.

Have a great Christmas!
 

How to start a process in background and keep it running after the console / terminal is closed on Linux and FreeBSD

Tuesday, November 29th, 2011

The classical way to keep a process running in background after log out of a shell is using screen
Anyways using screen is not the only way to detach a running process , GNU / Linux and BSDs (Free, Open, Net BSDs) had a command nohup which aim is to run a command immune to hangups, with output to a non-tty

Let’s say one wants to keep track (log) constantly ICMP traffic to a certain host with ping command on a UNIX server / desktop which doesn’t have the screen manager … terminal emulation program installed. Achieving this task is possible with nohup cmd by backgrounding the ping process, like so:

guoi@host:~$ nohup ping google.com >ping.log &[1] 45931hipo@host:~$ nohup: ignoring input and redirecting stderr to stdout

Afterwards even after closing up the opened ssh session or console (tty) / terminal (pts) on which the ping process is background nohup prevents the ping to be sent kill SIGNAL so the process continues running in the background.

Later on to check in real time the statistics of the continuous ICMP ping requests tail, less, or cat can be used for example watching the tail:

hipo@host:~$ tail -f ping.log
64 bytes from fx-in-f106.1e100.net (74.125.39.106): icmp_req=562 ttl=51 time=44.0 ms
64 bytes from fx-in-f106.1e100.net (74.125.39.106): icmp_req=563 ttl=51 time=43.8 ms
64 bytes from fx-in-f106.1e100.net (74.125.39.106): icmp_req=564 ttl=51 time=43.3 ms
64 bytes from fx-in-f106.1e100.net (74.125.39.106): icmp_req=565 ttl=51 time=43.1 ms
64 bytes from fx-in-f106.1e100.net (74.125.39.106): icmp_req=566 ttl=51 time=43.4 ms
64 bytes from fx-in-f106.1e100.net (74.125.39.106): icmp_req=567 ttl=51 time=43.6 ms

I’m using Linux / BSD for quite a lot of time and never before put in use the nohup cmd I guess there are more ppl who never heard of this handy UNIX basic command. Hope I’m not the only one who never heard about it and its useful knowledge to someone out. Cheers 😉

How to install display and audio drivers on motherboard Asus P5B-Plus with video ATI Radeon HD 2600 XT on Windows XP

Wednesday, November 16th, 2011

Ati Radeon 2600 XT, Display and Audio Drivers download how to

I re-installed one PC with Windows XP which was refusing to boot. The PC had a hardware of:

Motherboard: Asus P5B-Plus
Video Adapter: ATI Radeon HD 2600 XT
Sound card / Sound Blaster:

Ethernet card: Attansic L1 Gigabit Ethernet 10/100/1000Base-T Controller

It took me like 1 hour of search on the Internet and looking through forum threads and sites to properly install all the hardware. In hope to help someone out there looking to install the hardware Window drivers on ATI RAdeon HD 2600 XT, I’ve made a small archive of all the drivers necessery to make the Video card , Sound Card and Ethernet be properly installed and operating.

Here is download link to all the drivers for ATI Radeon HD 2600 XT to run smoothly on Windows XP

Installation of the drivers on Windows is pretty straight forward download the ATI Radeon HD 2600 XT archive extract and install each one of the files contained in the archive. A few restarts will also be necessery after some of the installed drivers to make the drivers work.

ATI Catalyst (included in the archive) will install the Video drivers for the Radeon XT 2600, whether AD1988AB_Audio_V6585_XpVistaWin7 and 11-11_xp32-64_hdmiaudio will install the Audio drivers. Attansic_L1_Lan_V1737907_V10560011159 contained in the archive needs to be extracted and installed to make the Attensic L1 Gigabit ethernet to show up as installed hardware in Windows device manager.

Hope this post will save some time to ppl looking to install the same drivers on XP 😉
Cheers 😉

How to Screenshot single Windows on GNU / Linux GNOME Desktop

Friday, October 28th, 2011

Every now and then I have to screenshot particular windows positioned on the screen on my GNOME Desktop envronment
Recently I was happy to find there is a very easy way to do this with the default Screenshotting program that is bundled with gnome gnome-screenshot

To screenshot a particular window using gnome-screenshot , its quite easy all one has to do is point the mouse cursor to the window he wants to snapshot and press:

Alt + PrtScr (Print Screen)

Here is a screenshot, I’ve taken of my gnome-terminal using the above command:

Particular window Screenshot Window screenshotting using GNOME gnome-screenshot

One can do it also via the command line using the /usr/bin/gnome-screenshot , by pressing Alt + F2 to invoke the run application and type in:

/usr/bin/gnome-screenshot -w

 

I was quite happy to find out that this is possible on Linux inside gnome without bothering to run GIMP or use any external programs for the cause. Hope this helps someone 😉