Posts Tagged ‘need’

Howto create Linux Music Audio CD from MP3 files / Create playable WAV format Audio CD Albums from MP3s

Tuesday, July 16th, 2019

cdburning-audio-music-cd-from-mp3-on-linuxcomapct-disc-tux-linux-logo

Recently my Mother asked me to prepare a Music Audio CD for her from a popular musician accordionist Stefan Georgiev from Dobrudja who has a unique folklore Bulgarian music.

As some of older people who still remember the age of the CD and who had most likely been into the CD burning Copy / Piracy business so popular in the countries of the ex-USSR so popular in the years 1995-2000 audio ,  Old CD Player Devices were not able to play the MP3 file format due to missing codecs (as MP3 was a proprietary compression that can't be installed on every device without paying the patent to the MP3 compression rights holder.

The revolutionary MP3 compression used to be booming standard for transferring Music data due to its high compression which made an ordinary MP3 of 5 minutes of 5MB (10+ times more compression than an ordinary classic WAV Audio the CPU intensiveness of MP3 files that puts on the reading device, requiring the CD Player to have a more powerful CPU.

Hence  due to high licensing cost and requirement for more powerful CPU enabled Audio Player many procuders of Audio Players never introduced MP3 to their devices and MP3 Neve become a standard for the Audio CD that was the standard for music listening inside almost every car out there.

Nowdays it is very rare need to create a Audio CD as audio CDs seems to be almost dead (As I heard from a Richard Stallman lecture In USA nowadays there is only 1 shop in the country where you can still buy CD or DVD drives) and only in third world as Africa Audio CDs perhaps are still in circulation.

Nomatter that as we have an old Stereo CD player on my village and perhaps many others, still have some old retired CD reading devices being able to burn out a CD is a useful thing.

Thus to make mother happy and as a learning excercise, I decided to prepare the CD for her on my Linux notebook.
Here I'll shortly describe the takes I took to make it happen which hopefully will be useful for other people that need to Convert and burn Audio CD from MP3 Album.

 

1. First I downloaded the Album in Mp3 format from Torrent tracker

My homeland Bulgaria and specific birth place place the city of Dobrich has been famous its folklore:  Galina Durmushlijska and Stefan Georgiev are just 2 of the many names along with Оркестър Кристал (Orchestra Crystal) and the multitude of gifted singers. My mother has a santiment for Stefan Georgiev, as she listened to this gifted accordinist on her Uncle's marriage.

Thus In my case this was (Стефан Георгиев Хора и ръченици от Добруджа) the album full song list here If you're interested to listen the Album and Enjoy unique Folklore from Dobrudja (Dobrich) my home city, Stefan Georgiev's album Hora and Rachenica Dances is available here

 


Stefan_Georgiev-old-audio-Music-CD-Hora-i-Rychenici-ot-Dobrudja-Horos-and-Ruchenitsas-from-Dobrudja-CD_Cover
I've downloaded them from Bulgarian famous torrent tracker zamunda.net in MP3 format.
Of course you need to have a CD / DVD readed and write device on the PC which nowdays is not present on most modern notebooks and PCs but as a last resort you can buy some cheap External Optical CD / DVD drive for 25 to 30$ from Amazon / Ebay etc.

 

2. You will need to install a couple of programs on Linux host (if you don't have it already)


To be able to convert from command line from MP3 to WAV you will need as minimum ffmpeg and normalize-audio packages as well as some kind of command line burning tool like cdrskin  wodim which is
the fork of old good known cdrecord, so in case if you you're wondering what happened with it just
use instead wodim.

Below is a good list of tools (assuming you have enough HDD space) to install:

 

root@jeremiah:/ # apt-get install –yes dvd+rw-tools cdw cdrdao audiotools growisofs cdlabelgen dvd+rw-tools k3b brasero wodim ffmpeg lame normalize-audio libavcodec58

 

Note that some of above packages I've installed just for other Write / Read operations for DVD drives and you might not need that but it is good to have it as some day in future you will perhaps need to write out a DVD or something.
Also the k3b here is specific to KDE and if you're a GNOME user you could use Native GNOME Desktop app such brasero or if you're in a more minimalistic Linux desktop due to hardware contrains use XFCE's native xfburn program.

If you're a console / terminal geek like me you will definitely enjoy to use cdw
 

root@jeremiah:/ # apt-cache show cdw|grep -i description -A 1
Description-en: Tool for burning CD's – console version
 Ncurses-based frontend for wodim and genisoimage. It can handle audio and

Description-md5: 77dacb1e6c00dada63762b78b9a605d5
Homepage: http://cdw.sourceforge.net/

 

3. Selecting preferred CD / DVD / BD program to use to write out the CD from Linux console


cdw uses wodim (which is a successor of good old known console cdrecord command most of use used on Linux in the past to burn out new Redhat / Debian / different Linux OS distro versions for upgrade purposes on Desktop and Server machines.

To check whether your CD / DVD drive is detected and ready to burn on your old PC issue:

 

root@jeremiah:/# wodim -checkdrive
Device was not specified. Trying to find an appropriate drive…
Detected CD-R drive: /dev/cdrw
Using /dev/cdrom of unknown capabilities
Device type    : Removable CD-ROM
Version        : 5
Response Format: 2
Capabilities   :
Vendor_info    : 'HL-DT-ST'
Identification : 'DVDRAM GT50N    '
Revision       : 'LT20'
Device seems to be: Generic mmc2 DVD-R/DVD-RW.
Using generic SCSI-3/mmc   CD-R/CD-RW driver (mmc_cdr).
Driver flags   : MMC-3 SWABAUDIO BURNFREE
Supported modes: TAO PACKET SAO SAO/R96P SAO/R96R RAW/R16 RAW/R96P RAW/R96R

You can also use xorriso (whose added value compared to other console burn cd tools is is not using external program for ISO9660 formatting neither it use an external or an external burn program for CD, DVD or BD (Blue Ray) drive but it has its own libraries incorporated from libburnia-project.org libs.

Below output is from my Thinkpad T420 notebook. If the old computer CD drive is there and still functional in most cases you should not get issues to detect it.

cdw ncurses text based CD burner tool's interface is super intuitive as you can see from below screenshot:

cdw-burn-cds-from-console-terminal-on-GNU-Linux-and-FreeBSD-old-PC-computer

CDW has many advanced abilities such as “blanking” a disk or ripping an audio CD on a selected folder. To overcome the possible problem of CDW not automatically detecting the disk you have inserted you can go to the “Configuration” menu, press F5 to enter the Hardware options and then on the first entry press enter and choose your device (by pressing enter again). Save the setting with F9.
 

4. Convert MP3 / MP4 Files or whatever format to .WAV to be ready to burn to CD


Collect all the files you want to have collected from the CD album in .MP3 a certain directory and use a small one liner loop to convert files to WAV with ffmpeg:
 

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

for i in $( ls *.mp3); do ffmpeg -i $i $i.wav; done


If you don't have ffmpeg installed and have mpg123 you can also do the Mp3 to WAV conversion with mpg123 cmd like so:

 

for i in $( ls ); do mpg123 -w $i.wav $i.mp3; done


Another alternative for conversion is to use good old lame (used to create Mp3 audio files but abling to also) decode
mp3 to wav.

 

lame –decode somefile.mp3 somefile.wav


In the past there was a burn command tool that was able to easily convert MP3s to WAV but in up2date Linux modern releases it is no longer available most likely due to licensing issues, for those on older Debian Linux 7 / 8 / 9 / Ubuntu 8 to 12.XX / old Fedoras etc. if you have the command you can install burn and use it (and not bother with shell loops):

apt-get install burn

or

yum install burn


Once you have it to convert

 

$ burn -A -a *.mp3
 

 

5. Fix file naming to remove empty spaces such as " " and substitute to underscores as some Old CD Players are
unable to understand spaces in file naming with another short loop.

 

for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done

 

6. Normalize audio produced .WAV files (set the music volume to a certain level)


In case if wondering why normalize audio is needed here is short extract from normalize-audio man page command description to shed some light.

"normalize-audio  is  used  to  adjust  the volume of WAV or MP3 audio files to a standard volume level.  This is useful for things like creating mp3 mixes, where different recording levels on different albums can cause the volume to  vary  greatly from song to song."
 

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

normalize-audio -m *.wav

 

7. Burn the produced normalized Audio WAV files to the the CD

 

wodim -v -fix -eject dev='/dev/sr0' -audio -pad *.wav


Alternatively you can conver all your MP3 files to .WAV with anything be it audacity
or another program or even use 
GNOME's CDBurn tool brasero (if gnome user) or KDE's CDBurn which in my opinion is
the best CD / DVD burning application for Linux K3B.

Burning Audio CD with K3b is up to few clicks and super easy and even k3b is going to handle the MP3 to WAV file Conversion itself. To burn audio with K3B just run it and click over 'New Audio CD Project'.

k3b-on-debian-gnu-linux-burn-audio-cd-screenshot

For those who want to learn a bit more on CD / DVD / Blue-Ray burning on GNU / Linux good readings are:
Linux CD Burning Mini Howto, is Linux's CD Writing Howto on ibiblio (though a bit obsolete) or Debian's official documentation on BurnCD.
 

8. What we learned here


Though the accent of this tutorial was how to Create Audio Music CD from MP3 on GNU / Linux, the same commands are available in most FreeBSD / NetBSD / OpenBSD ports tree so you can use the same method to build prepare Audio Music CD on *BSDs.

In this article, we went through few basic ways on how to prepare WAV files from MP3 normalize the new created WAV files on Linux, to prepare files for creation of Audio Music CD for the old mom or grandma's player or even just for fun to rewind some memories. For GUI users this is easily done with  k3b,  brasero or xfburn.

I've pointed you to cdw a super useful text ncurses tool that makes CD Burninng from plain text console (on servers) without a Xorg / WayLand  GUI installed super easy. It was shortly reviewed what has changed over the last few years and why and why cdrecord was substituted for wodim. A few examples were given on how to handle conversion through bash shell loops and you were pointed to some extra reading resources to learn a bit more on the topic.
There are plenty of custom scripts around for doing the same CD Burn / Covnersion tasks, so pointing me to any external / Shell / Perl scripts is mostly welcome.

Hope this learned you something new, Enjoy ! 🙂

How to synchronize with / from Remote FTP server using LFTP like with rsync

Sunday, October 15th, 2017

how-to-synchronize-from-remote-ftp-server-easily-like-rsync.jpg

Have you ever been in a need to easily synchronize with a remote host which only runs FTP server?

Or are you in a local network and you need to mirror a directory or a couple of directories in a fast and easy to remember way?

If so then you'll be happy to use below LFTP command that is doing pretty much the same as Rsync, with only difference that it can mirror files over FTP (old but gold File Transfer Protocol).
 

lftp -u FTP_USERNAME,FTP_PASSWORD -e 'mirror REMOTE_DIRECTORY LOCAL_DIRECTORY' FTP_SERVER_HOSTNAME


Enjoy and thanks to my dear friend Amridikon for the tip ! 🙂

How to Increase virtualbox Linux install machine VM .VDI hard disk size to free space on root partition – Move /usr to a new partition

Tuesday, October 10th, 2017

extend-vdi-virtual-machine-partition-with-virtualbox-howto-virtualbox-logo
 

How to Increse Hard Disk size of VirtualBox Virtual Machine .VDI file to Free Space on root Partition or Howto move /usr large partition to separate new partition?


I just wondered how to increase hard disk size of Virtualbox Virtual Machine image .VDI, because for some stupid reason I've created my initial hard disk size for Linux partition to be the default 10 Gigabytes.

The problem is the packages I need to have installed on the Virtual Machine which will be a testbed for future tests of a production website applications are taking up too much space, once installed so I'm left with no space
in /var/lib/mysql for the database import. So what can I do in that case is to simply free up disk space or Merge ROOT partition with another partition.

Since merging the partition is not a trivial job and would require me to have installable CD with the Linux distro (in my case that's Debian Linux) or have a bootable USB flash drive, I preferred second approach to problem e.g. to free up disk space on ROOT partition by creating a second partition and move the /usr folder to reside there.

Before that it is of course necessery to  have extended the .VDI file using VirtualBox, so more space than the default 10GB preconfigured are available, this is easily done on Windows OS as, VBox is provided with GUI clickable option to do it, but for who knows what reason that is not the case with Linux, so Linux users only option to increase VDI file is to manually run command part of the virtualbox package, that is not a hard task really but it requires some typing and basic knowledge on how to run commands in terminal.


To .VDI resize (extend), we first go to default location where VirtualBox stores its image .VDI files (by default as of moment of writting this article – this is ~/"VrtualBox VMs"  (or home directory of logged in user dir VirtualBox VMs), the command to use is VBoxManage

 

root@jericho:/home/hipo# cd VirtualBox VMs/
root@jericho:/home/hipo/VirtualBox VMs# ls
Debian 6  Debian 9  Windows 10
root@jericho:/home/hipo/VirtualBox VMs# cd Debian 6/
r
oot@jericho:/home/hipo/VirtualBox VMs/Debian 6# ls
Debian 6.vbox  Debian 6.vbox-prev  Debian 6.vdi  Logs  NewVirtualDisk1.vdi  Snapshots

root@jericho:/home/hipo/VirtualBox VMs/Debian 6# VBoxManage modifyhd Debian 6.vdi –resize 20000
0%…10%…20%…30%…40%…50%…60%…70%…80%…90%…100%
root@jericho:/home/hipo/VirtualBox VMs/Debian 6#


 

Above command does resize the 10GB default created partition for Linux, where I have installed Linux which was 99% full of data, because of the many packages I installed to 20GB size, to make it bigger just use the respective size, be it 30000 (for 30GB) or 100000 (for 100GB) etc.

Even though in this example VBoxManage virtual partition resize command was done for GNU / Linux Operating System, it can be done for any other Operating as well to resize the size of the Virtual .VDI file (Virtual Machine) partition, be it Windows 7 / 8 / 10 or the rest of Free Operating systems FreeBSD / OpenBSD / BSD that are installed in a VM etc.

Next Launch the Virtual Machine with VBox Server client Program and install there Gparted (GNU Parted), as we'll need it to create a new Hard Disk Partition:

 

$ VirtualBox

 


Inside virtualmachine's in gnome-terminal / xterm etc. depending on the graphical environment used do install with apt-get:

 

debian:# apt-get install –yes gparted

 

debian~:# gparted


Notice that gparted has to be ran as a root superuser.

 

Run GParted and create new EXT3 filesystem that is 10GB (the size of the new created partition).

If you have installed Debian to place all partitions under / (root directory /dev/sda1) then the fresh new partition to create should be
/dev/sda3, anyways just look closely in EXT3 and in your case if the partiition is named differently create according to proper partition /dev/ naming.

I'll not run into details on how to create the partition with GParted as the program interface is very self-explanatory, the only thing is to apply the update to create partition and the ext3 filesystem, that's being done
with a green tick:

gparted-create-ext3-partition-howto-linux

Next step is to check with fdisk whether, we have ext3 properly created  filesystem as we've done already with GPARTED:

Once we have the partition created with EXT3 filesystem, we're ready to move /usr temporary to other folder, I use usually /root for the move but you can create anywhere a new folder for that and move to there.

To move to /root directory run again in terminal:

 

debian:~# mv /usr /root
debian:~# mkdir /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux

 

Note that during the move operations, your Desktop icons will become without (with broken) pictures and the default Debian background picture is to disappear, that's because the GUI environment will soon realize /usr/ libraries that're periodically reloaded in memory are missing and will be unable to reload them as it does in a cycle.

That should take a few minutes, so grab a coffee or if you're a smoker (hope not as smoking kills 🙂 ), in 5 / 10 minutes time depending on your computer / server configuration, it will be over, so we're ready to create new /usr dir and mount the  new partition:

 

debian:~# mount /dev/sda3 /usr

 

howto-extend-root-filesystem-disk-space-linux-move-usr-folder-to-root-temporary-debian-gnu-linux1

 

Now we check with mount command whether mount is fine:

 

check-if-filesystem-is-properly-mounted-linux-debian-screenshot


Now  /dev/sda3 is mounted under /usr  and we have to move back /root/usr directory content back to the newly mounted /usr so we run command:
 

debian:~#  mv /root/usr/* /usr/*


Finally we need to create proper records for the new partition inside /etc/fstab (fstab –FileSystem Tab file – the file which describes instructs the Linux OS what partition to boot where, what)

HOW TO CHECK LINUX UUID FOR A PARTITION??

Before adding anything to /etc/fstab you need to check the UUID of /dev/sda3 (or whatever the partition is called), without proper UUID, the system might fail to boot.
So here is how to check the UUID we'll need for config:

 

hipo@debian:~$ /sbin/blkid /dev/sda3
/dev/sda3: UUID="2273db4b-3069-4f78-90fc-e7483c0305bd" SEC_TYPE="ext2" TYPE="ext3"

hipo@debian:~$ ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 100 Oct  6 05:16 .
drwxr-xr-x 6 root root 120 Oct  6 05:16 ..
lrwxrwxrwx 1 root root  10 Oct  6 05:16 2273db4b-3069-4f78-90fc-e7483c0305bd -> ../../sda3
lrwxrwxrwx 1 root root  10 Oct  6 05:16 b98d92cd-41aa-4e18-a474-9b8df445dbe5 -> ../../sda1
lrwxrwxrwx 1 root root  10 Oct  6 05:16 f27f7448-f200-4983-b54f-b9e5206f77ac -> ../../sda5

As you can see our /dev/sda3 UUID is 2273db4b-3069-4f78-90fc-e7483c0305bd

Further on lets view and edit /etc/fstab you can also download a copy of my Virtual Machine fstab here

 

debian:~# cat /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=b98d92cd-41aa-4e18-a474-9b8df445dbe5 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=f27f7448-f200-4983-b54f-b9e5206f77ac none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0


We need to add following line to  /etc/fstab:
 

UUID=2273db4b-3069-4f78-90fc-e7483c0305bd    /usr        ext3 error=remount-ro    0    1

 


Open the file with your favourite text editor (gedit / nano / pico / vim / joe) etc.

debian:~# vim /etc/fstab

 

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
# / was on /dev/sda1 during installation
UUID=b98d92cd-41aa-4e18-a474-9b8df445dbe5 /               ext3    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=f27f7448-f200-4983-b54f-b9e5206f77ac none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/scd1       /media/cdrom1   udf,iso9660 user,noauto     0       0

UUID=2273db4b-3069-4f78-90fc-e7483c0305bd     /usr        ext3 error=remount-ro    0    1    

Basicly it should be possible to add (for historic reasons) also instead of UUID=2273db4b-3069-4f78-90fc-e7483c0305bd  /dev/sda3
So it looks like so but, the better practice is to use UUID line given.

Well that's all folks now /usr directory will contain all your heavy root partition (disk filling) libraries and files, so you can happily use /var/lib/mysql or /var/lib/postgresql /var/www to store your web application files and import your databases.

Big thanks to Ubuntu Forums article – How do I increase the hard disk size of the Virtual Machine article for inspiring this post.

Hope that helps anyone and thanks and other comments are always welcome ! 🙂

Block Web server over loading Bad Crawler Bots and Search Engine Spiders with .htaccess rules

Monday, September 18th, 2017

howto-block-webserver-overloading-bad-crawler-bots-spiders-with-htaccess-modrewrite-rules-file

In last post, I've talked about the problem of Search Index Crawler Robots aggressively crawling websites and how to stop them (the article is here) explaning how to raise delays between Bot URL requests to website and how to completely probhit some bots from crawling with robots.txt.

As explained in article the consequence of too many badly written or agressive behaviour Spider is the "server stoning" and therefore degraded Web Server performance as a cause or even a short time Denial of Service Attack, depending on how well was the initial Server Scaling done.

The bots we want to filter are not to be confused with the legitimate bots, that drives real traffic to your website, just for information

 The 10 Most Popular WebCrawlers Bots as of time of writting are:
 

1. GoogleBot (The Google Crawler bots, funnily bots become less active on Saturday and Sundays :))

2. BingBot (Bing.com Crawler bots)

3. SlurpBot (also famous as Yahoo! Slurp)

4. DuckDuckBot (The dutch search engine duckduckgo.com crawler bots)

5. Baiduspider (The Chineese most famous search engine used as a substitute of Google in China)

6. YandexBot (Russian Yandex Search engine crawler bots used in Russia as a substitute for Google )

7. Sogou Spider (leading Chineese Search Engine launched in 2004)

8. Exabot (A French Search Engine, launched in 2000, crawler for ExaLead Search Engine)

9. FaceBot (Facebook External hit, this crawler is crawling a certain webpage only once the user shares or paste link with video, music, blog whatever  in chat to another user)

10. Alexa Crawler (la_archiver is a web crawler for Amazon's Alexa Internet Rankings, Alexa is a great site to evaluate the approximate page popularity on the internet, Alexa SiteInfo page has historically been the Swift Army knife for anyone wanting to quickly evaluate a webpage approx. ranking while compared to other pages)

Above legitimate bots are known to follow most if not all of W3C – World Wide Web Consorium (W3.Org) standards and therefore, they respect the content commands for allowance or restrictions on a single site as given from robots.txt but unfortunately many of the so called Bad-Bots or Mirroring scripts that are burning your Web Server CPU and Memory mentioned in previous article are either not following /robots.txt prescriptions completely or partially.

Hence with the robots.txt unrespective bots, the case the only way to get rid of most of the webspiders that are just loading your bandwidth and server hardware is to filter / block them is by using Apache's mod_rewrite through

 

.htaccess


file

Create if not existing in the DocumentRoot of your website .htaccess file with whatever text editor, or create it your windows / mac os desktop and transfer via FTP / SecureFTP to server.

I prefer to do it directly on server with vim (text editor)

 

 

vim /var/www/sites/your-domain.com/.htaccess

 

RewriteEngine On

IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

SetEnvIfNoCase User-Agent "^Black Hole” bad_bot
SetEnvIfNoCase User-Agent "^Titan bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^TeleportPro" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^Website Quester" bad_bot
SetEnvIfNoCase User-Agent "^WebZip" bad_bot
SetEnvIfNoCase User-Agent "^moget/2.1" bad_bot
SetEnvIfNoCase User-Agent "^WebZip/4.0" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^httplib" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^Harvest/1.5" bad_bot
SetEnvIfNoCase User-Agent "Bullseye/1.0" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4.0 (compatible; BullsEye; Windows 95)" bad_bot
SetEnvIfNoCase User-Agent "^Crescent Internet ToolPak HTTP OLE Control v.1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerSE/1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker /1.0" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit/3.50" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control – 5.01.4511" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial/1.34" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control – 6.00.8169" bad_bot
SetEnvIfNoCase User-Agent "^URLy Warning" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Mata Hari" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^Web Image Collector" bad_bot
SetEnvIfNoCase User-Agent "^The Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish/1.0" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc/4.2" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^VCI WebViewer VCI WebViewer Win32" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^QueryN Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^Openfind data gathere" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^Xenu’s Link Sleuth 1.1c" bad_bot
SetEnvIfNoCase User-Agent "^Xenu’s" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey Bait & Tackle/v1.01" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 32297 Webster Pro V2.9 Win32" bad_bot
SetEnvIfNoCase User-Agent "^Webster Pro" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a Unix" bad_bot
SetEnvIfNoCase User-Agent "^Keyword Density/0.9" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin Spider" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot

 

<Limit GET POST>
order allow,deny
allow from all
Deny from env=bad_bot
</Limit>

 


Above rules are Bad bots prohibition rules have RewriteEngine On directive included however for many websites this directive is enabled directly into VirtualHost section for domain/s, if that is your case you might also remove RewriteEngine on from .htaccess and still the prohibition rules of bad bots should continue to work
Above rules are also perfectly suitable wordpress based websites / blogs in case you need to filter out obstructive spiders even though the rules would work on any website domain with mod_rewrite enabled.

Once you have implemented above rules, you will not need to restart Apache, as .htaccess will be read dynamically by each client request to Webserver

2. Testing .htaccess Bad Bots Filtering Works as Expected


In order to test the new Bad Bot filtering configuration is working properly, you have a manual and more complicated way with lynx (text browser), assuming you have shell access to a Linux / BSD / *Nix computer, or you have your own *NIX server / desktop computer running
 

Here is how:
 

 

lynx -useragent="Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)" -head -dump http://www.your-website-filtering-bad-bots.com/

 

 

Note that lynx will provide a warning such as:

Warning: User-Agent string does not contain "Lynx" or "L_y_n_x"!

Just ignore it and press enter to continue.

Two other use cases with lynx, that I historically used heavily is to pretent with Lynx, you're GoogleBot in order to see how does Google actually see your website?
 

  • Pretend with Lynx You're GoogleBot

 

lynx -useragent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -head -dump http://www.your-domain.com/

 

 

  • How to Pretend with Lynx Browser You are GoogleBot-Mobile

 

lynx -useragent="Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" -head -dump http://www.your-domain.com/

 


Or for the lazy ones that doesn't have Linux / *Nix at disposal you can use WannaBrowser website

Wannabrowseris a web based browser emulator which gives you the ability to change the User-Agent on each website req1uest, so just set your UserAgent to any bot browser that we just filtered for example set User-Agent to CheeseBot

The .htaccess rule earier added once detecting your browser client is coming in with the prohibit browser agent will immediately filter out and you'll be unable to access the website with a message like:
 

HTTP/1.1 403 Forbidden

 

Just as I've talked a lot about Index Bots, I think it is worthy to also mention three great websites that can give you a lot of Up to Date information on exact Spiders returned user-agent, common known Bot traits as well as a a current updated list with the Bad Bots etc.

Bot and Browser Resources information user-agents, bad-bots and odd Crawlers and Bots specifics

1. botreports.com
2. user-agents.org
3. useragentapi.com

 

An updated list with robots user-agents (crawler-user-agents) is also available in github here regularly updated by Caia Almeido

There are also a third party plugin (modules) available for Website Platforms like WordPress / Joomla / Typo3 etc.

Besides the listed on these websites as well as the known Bad and Good Bots, there are perhaps a hundred of others that might end up crawling your webdsite that might or might not need  to be filtered, therefore before proceeding with any filtering steps, it is generally a good idea to monitor your  HTTPD access.log / error.log, as if you happen to somehow mistakenly filter the wrong bot this might be a reason for Website Indexing Problems.

Hope this article give you some valueable information. Enjoy ! 🙂

 

Install JBL Go Bluetooth Speaker on Debian GNU / Linux and Ubuntu

Thursday, August 24th, 2017

jbl-go-on-gnu-how-to-install-on-debian-and-ubuntu-linux

Here is how to configure a JBL Go Bluetooth (Wireless) speaker and presumably other Bluetooth external speakers to Debian GNU / Linux Wheezy 7 and Ubuntu 14.04 . 1. Install following bunch of deb packages

debian:~# apt-get install pulseaudio pulseaudio-module-bluetooth pavucontrol bluez-firmware

Here it is notable to mention pavucontrol if you have previously played more extensively on GNU / Linux you should have already used if not it is really cozy volume control tool with a lot of tuning options regarding pulseaudio stream server. Considering that like me you're using a GNOME as a desktop environment you will also need gnome-bluetooth package, e.g.:

debian:~# apt-get install gnome-bluetooth

As Pulseaudio is used as a sound streaming server in GNU / Linux (assuming your Debian version is using it you'll also need to have installed pulseaudio-module-bluetooth)

debian:~# apt-get install pulseaudio-module

For Ubuntu 14.04 GNU / Linux users the list of necessery bluetooth packages is a bit longer, if you're on this OS go and install:

debian:~# apt-get install bluez bluez-alsa bluez-audio bluez-btsco bluez-compat bluez-cups bluez-dbg bluez-gstreamer bluez-hcidump bluez-pcmcia-support bluez-tools bluez-utils python-bluez bluewho indicator-bluetooth libbluetooth-dev libgnome-bluetooth11 libbluetooth3 python-gobject python-dbus

Moreover you will need pulseaudio-module-bluetooth deb package installed in order to be able to select the desired sound output.

Next it is time to restart Bluetooth service

debian:~# service bluetooth restart
[ ok ] Stopping bluetooth: rfcomm /usr/sbin/bluetoothd.
[ ok ] Starting bluetooth: bluetoothd rfcomm.

It is also a good idea to restart pulseaudio snd streaming server in order to load the newly installed pulseaudio bluetooth module settings, to do so issue:

debian:~# killall pulseaudio

And try to establish connection from Gnome-Bluetooth to the JBL Go (press the JBL Go bluetooth button) and search from the Linux bluetooth interface, once founded connect it.

bluetooth-new-device-setup

jbl-go-connected-screenshot

Before JBL Go appears to list listable blootooth devices you will also need to run following command:

debian:~# pactl load-module module-bluetooth-discover
26

This command is to connect bluetooth discovered JBL Go device to the audio sink interface.

It is generally idea to add this line also to /etc/rc.local to make the setting permanently executed on every Linux boot.

Now you can launch pavucontrol and hopefully the JBL GO bluetooth speaker should be visible as an option, check out my below screenshot:

 


In case you further experience issues connecting the Bluetooth Speaker I would recommend to check out this Debian a2dp page at the end of the page are troubleshooting suggestions.

Troubleshooting

Refused to switch profile to a2dp_sink: Not connected

Bluetooth headset is connected, but ALSA/PulseAudio fails to pick up the connected device or there's no device to pick. This happens because GDM captures A2DP sink on session start, as GDM needs pulseaudio in the gdm session for accessibility. For example, the screen reader requires it. See 805414 for some discussion.

 

Workaround 1: disable pulseaudio in gdm

In order to prevent GDM from capturing the A2DP sink on session start, edit /var/lib/gdm3/.config/pulse/client.conf (or create it, if it doesn't exist):

 

autospawn = no
daemon-binary = /bin/true

After that you have to grant access to this file to Debian-gdm user:

 

chown Debian-gdm:Debian-gdm /var/lib/gdm3/.config/pulse/client.conf

You will also need to disable pulseaudio startup:

 

rm /var/lib/gdm3/.config/systemd/user/sockets.target.wants/pulseaudio.socket

In order to auto-connect a2dp for some devices, add this to /etc/pulse/default.pa:

 

load-module module-switch-on-connect

Logout your Desktop environment and restart gdm3 /etc/init.d/gdm3 restart or Reboot the PC and then it should be fine.

 

Now the sound device (bluetooth headset) should be accessible through pavucontrol and standard audio device manager.

 

Workaround 2: disable pulseaudio's bluetooth in gdm

The actual solution package maintainers are looking into next is to simply disable the bluetooth sink in the gdm pulseaudio daemon so that it doesn't take over the device. Add this to /var/lib/gdm3/.config/pulse/default.pa:

 

#!/usr/bin/pulseaudio -nF
#

# load system wide configuration
.include /etc/pulse/default.pa

### unload driver modules for Bluetooth hardware
.ifexists module-bluetooth-policy.so
  unload-module module-bluetooth-policy
.endif

.ifexists module-bluetooth-discover.so
  unload-module module-bluetooth-discover
.endif

Though this article explains how to connect a bluetooth speaker connecting Bluetooth Speaker to GNU / Linux is done in analogous way

 

Skype remove word completion / Howto Disable spellcheck autocorrect on Mac OS X, Android Phones Tablets and IPhone iOS

Tuesday, March 14th, 2017

How-to-disable-auto-correct-on-MacOS-X-android-and-iphone

I've recently upgraded my wife's Mac Book Air Mac OS X release from Mountain Lion to latest available OS X release Yosemite and after upgrade the old version of Skype used for about 3 years without any problems (except camera issues due to mac OS X bug the old post is here) stopped working (did crashed) on Skype launch so I took few minutes to upgrade Skype as well from Skype.com Skype for Mac official download page after the quick and unproblematic Skype re-install, Skype client was back to normal with a shiny new outlook and on a first glimpse all worked like a charm but …

On the next day when I took use of temporary the upgraded Mac Book Air with the fresh new OS X Yosemite and Skype's latest app to to do a quick login with my Skype credentials noticed a very annoying autocorrect (auto spelling)  Skype behavior on any new input text message.

I have to admit the idea of autocorrect feature on a system wide scale is great however quick keyboard typers like me who are used to type more than a letter per second use to type so quickly that occasinally you do errors which you're quick to fix but anyhow errors are automatically corrected in a way that you have to retype the whole word quite often which makes your typing a hellish experience. Then the same auto-correct feature becomes a terrible experience and on any device I notice this behavior I tend to remove it immediately.

I remember  the same dictionary auto-spell check / corect behavior was present on my Android run ZTE blade 3 mobile after first Mobile boot and also present on my current iPhone 5 iOS and thus the first thing I did once I start using it is to remove autocorrect (auto spelling) feature and relax the irritation of it.

Just in case if you need to disable Spellcheck auto-correct on Android Phone or Tablet do it from

Settings -> Language & input

android-phone-tablet-disable-autocorrect-spell-check

If you happen to be using also IPhone 5 as me or  Iphone 6 / 7 to disable AutoCorrect feature on SMSes / Skype / Viber and other applications do it from Phone's:

Settings -> General

disable_Turn_off_AutoCorrect_in_IPhone_iOS

Shortly I've noticed that the auto-correct feature does not only mess up your typing in Skype but also in Mail and TweetDeck and generally any open application where you can type on Yosemite.


I've took the time to investigate quickly what was causing the new feature that bugged me so much online and came to this nice little article How to turnoff autocorrect on Mac OS X 10.11 El Captain

As the article revealed to Turn off the autocorrect Mac OS X feature you I have to go:

Finder -> System Preferences

 

stop-skype-autocorrect-annoying-macosx-yosemite-system-preferences-menu-screenshot

Click on Keyboard -> Text
and turn  Autocorrect off by unchecking Correct spelling automatically

stop-skype-autocorrect-annoying-macosx-yosemite-system-preferences-keyboard-menu-screenshot

By doing this autocorrect spelling is disabled for all applications, however for some people it might be handy to leave the correct feature for Mail app or Microsoft Office (Microsoft Word / Excel) etc. and only turn off (disable) the autorrect feature per application side.

You can  do this in the separate Applications (of choise where you need auto spelling) from  Edit Menu under Spelling and Grammer and enable / disableCheck Spelling While Typing

stop-skype-autocorrect-annoying-macosx-yosemite-spelling-and-grammar-menu-screenshot


Tadam Enjoy ! 🙂

 

How to convert .CRT SSL Certificate to .PFX format (with openssl Linux command) and Import newly generated .PFX to Windows IIS Webserver

Tuesday, September 27th, 2016

IIS8_Windows_Webserver_logo_convert_CRT_and_import_PFX-certificate

1. Converting to .CRT to.PFX file format with OpenSSL tool on GNU / Linux to import in Windows (for example, IIS)

Assuming you have generated already a certificate using the openssl Linux command and you have issued the .CRT SSL Certificate issuer file
and you need to have the new .CRT SSL Certificate installed on Windows Server (lets say on Windows 2012) with IIS Webserver version 8.5, you will need a way to convert the .CRT file to .PFX, there is plenty of ways to do that including using online Web Site SSL Certificate converter or use a stand alone program on the Windows server or even use a simple perl / python / ruby script to do the conversion but anyways the best approach will be to convert the new .CRT file to IIS supported binary Certificate format .PFX on the same (Linux certificate issuer host where you have first generated the certificate issuer request .KEY (private key file used with third party certificate issuer such as Godaddy or Hostgator to receive the .CRT / PEM file).

Here is how to generate the .PFX file based on the .CRT file for an Internal SSL Certfiicate:

 

openssl pkcs12 -export -in server.crt -inkey server.key -out server.pfx

On the password prompt to appear use any password because otherwise the future IIS Webserver certificate import will not work.
 

To do a certificate chain SSL export to be accessed from the  internet.

 

openssl pkcs12 -export -in server.crt -inkey server.key -out server.pfx -certfile internet v2.crt

2. Import the PFX file in Windows


Run: mmc, add snap, Certificates, Computer account, Local Computer; in the
Console:

Certificates (Local Computer) > Personal > Certificates: Select All Tasks > Import File

Enter previously chosen password.
You should get further the Message "Import was successful."

You can import the PFX file by simply copying it to the server where you want it imported and double click it this will  open Windows Importwizzard.

Then select the IIS:

 

Site, Properties, Directory Security, Server Certificate, Replace the current certficate, select proper Certificate. Done.

Alternatively to complete the IIS Webserver certificate import within one step when a new certificate is to be imported:

In IIS Manager interface go to :

Site, Properties, Directory Security, Server Certificate, Server Certificate Wizard


Click on

Next

Choose

import a certificate from a .pfx file, select and enter password.

Internet_Information_Server_IIS_Windows-SSL_Certificate-import-PKF-file

3. Import the PFX file into a Java keystore


Another thing you might need if you have the IIS Webserver using a backend Java Virtual Machine on the same or a different Windows server is to import the newly generated .PFX file within the Java VM keystore.

To import with keytool command for Java 1.6 type:

 

keytool -importkeystore -deststorepass your_pass_here -destkeypass changeit -destkeystore keystore.jks -srckeystore server.pfx -srcstoretype PKCS12 -srcstorepass 1234 -srcalias 1 -destalias xyz


Also the .CRT file could be directly imported into the Java keystore

 

Import a .crt in a Java keystore


/usr/java/jre/bin/keytool -import -keystore /webdienste/java/jdk/jre/lib/security/cacerts -file certificate.crt -alias Some alias

 

 

4. Get a list of Windows locally installed certificates

To manager installed certificates on Windows 7 / 8 / 2012 Server OS is to run command via

Start -> Run

 

certmgr.msc

certmgr_trca_windows_check-windows-installed-ssl-certificates

 

One other way to see the installed certificates on your Windows server is checking within

Internet Explorer

Go to Tools (Alt+X) → Internet Options → Content → Certificates.

 

To get a a complete list of installed Certificate Chain on Windows you can use PowerShell

 

Get-ChildItem -Recurse Cert:

 

That's all folks ! 🙂

 

How to shutdown Windows after 1, 2, 3, 4 etc. X hours with a batch script – Shutdown / Reboot / Logoff Windows with a quick command

Wednesday, August 17th, 2016

http://pc-freak.net/images/windows-pc-server-shutdown-after-3-5-hours-howto-shutdown-windows-with-command-batch

I recently wondered how it is possible to shutdown Windows in some prior set time lets say in 30 minutes, 1 hour, 3 hours or 8 hours.

That's handy especially on servers that are being still in preparation install time and you have left some large files copy job (if you're migration files) from Old server environment to a new one
or if you just need to let your home WIndows PC shutdown to save electricity after some time (a very useful example is if you're downloading some 200GB of data which are being estimated to complete in 3 hours but you need to get out and be back home in 2 or 4 days and you don't want to bother connecting remotely to your PC with VNC or teamviewer then just scheduling the PC / server to shutdown in 3 hours with a simple is perfect solution to the task, here is how:

1. Open Command Prompt (E.g. Start menu -> Run and type CMD.EXE)

2. Type in command prompt

 

shutdown -s -t 10800

 

If you by mistake has typed it to shutdown earlier and suddenly you find out your PC needs to be running for a short more time in order to cancel the scheduled Shutdown type:

 

shutdown -a


Shutdown Windows command -s flag has also a possibiltiy to not shutdown but just logoff or if you just need to have the system rebooted a reboot option:
 


options    effect
-l         to log off
-r         to reboot

If you need to shutdown the PC after half an hour use instead the command:

 

shutdown -s -t 1800


shutdown-windows-pc-with-command-in-half-an-hour-screenshot.gif


Half an hour is 1800 seconds for one hour delayed shutdown use 3600 for 3 hours, that would be 3*3600 10800, for 5 hours 5*3600 = 18000 seconds and so on

 


An alternative way to do it with a short VBscript, here is an example:

Set objShell = CreateObject("WScript.Shell")

Dim Input
Input = "10:00"

'Input = InputBox("Enter the shutdown time here.","", "10:00")

For i = 1 to 2

CurrentTime = Time & VbCrLf

If Left(CurrentTime,5) = Input Then

objShell.Run "shutdown -s -t 00", 0
WScript.Quit 1

Else

WScript.Sleep 1000

End If

i=i-1

Next

Enjoy