Posts Tagged ‘great software’

Downloading your favourity flash video from Youtube with a simple command (youtube-dl)

Wednesday, April 13th, 2011

downloading-flash-videos-from-youtube-on-linux-and-bsd-youtube-downloader-logo
Watching videos in youtube today and already for about 2 years is the de-facto hype.
There is almost none a day passed without almost each one of us has watched a dozen videos in Youtube.

Watching videos in youtube has become even more addictive for many than the early days of Internet Relay Chats (IRC)

As youtube is very accessible for people and it’s a comparativily easy way people share more and more with the day.
There is no question that the business idea of youtube is great and youtube generates millions of dollars for Google day by day, however I have a serious objection here! All is good the only pitfall is that you don’t own the youtube videos you watch!

Youtube’s story is not that different from the story of the cloud computing threat to internet users Freedom

The good thing here is that we’re not still completely dependant on youtube and there is still way to retrieve your favourite youtube video and store it for later watching or distribution.

Probably the most famous browser plugin that allows files retrieval from youtube, as most people know is DownloadHelper .

However using download helper is browser dependant, you need to use the browser to save the plugin and I don’t find it to be the best way to download a youtube video.

Since the old days I have started using Linux, I’ve been quite addicted to as many things on my linux as possible from the command line (terminal / console) (CLI) .

In that manner of thoughts it was a real delight for me to find out that a group of free software developer guys has come up with a command line tool that allows downloads of youtube videos straight from terminal, the great software is called youtube-dl and at the moment of this post writting it’s to be found on the URL address:

http://rg3.github.com/youtube-dl/

Youtube-dl is written in python so, it requires the Python interpreter, version 2.5 in order to properly run on Unix, Mac OS X or even on Windows!

The fact that it’s written in python has made the little shiny tool quite a multi-platform one.
To start using immediately the tool on a Debian or Ubuntu Linux you will have to install python (even though in most cases you must have it already installed):

1. To make sure you have python interpreter installed issue the cmd:

debian:~# apt-get install python
Building dependency tree
Reading state information... Done
python is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

As you can see from above apt-get’s output I do have it installed so nothing gets installed.

2. As a next step I used links to download the youtube-dl python script, like so:

debian:~# links https://github.com/rg3/youtube-dl/raw/2011.03.29/youtube-dl >> youtube-dl
Use the links interface to save youtube-dl and use gzip to ungzip it
debian:~# gzip -d youtube-dl.gz
debian:~# chmod +x youtube-dl

Now to make it system wide accessible I have copied the youtube-dl to /usr/local/bin , whether I selected /usr/local/bin as a location as this location is predetermined to contain mostly files which does not belong to a regular deb package.

3. Move youtube-dl to /usr/local/bin

debian:~# mv youtube-dl /usr/local/bin

4. Test the newly installed youtube-dl command line youtube retrieval tool:

debian:~# ./youtube-dl https://www.youtube.com/watch?v=g7tvI6JCXD0
[youtube] Setting language
[youtube] g7tvI6JCXD0: Downloading video webpage
[youtube] g7tvI6JCXD0: Downloading video info webpage
[youtube] g7tvI6JCXD0: Extracting video information
[download] Destination: g7tvI6JCXD0.flv
[download] 53.3% of 22.62M at 33.23k/s ETA 05:25
[download] 100.0% of 22.62M at 31.91k/s ETA 00:00 [u

As you might have noticed from the above youtube-dl command output the newly retrieved youtube file will be saved under a name g7tvI6JCXD0.flv

The line I passed to youtube-dl is directly taken from my browser and pasted to console, the file downloading from youtube took me about 10 minutes but this is mostly because of some kind of youtube server speed restrictions …

In general at least I have this video for later, watching, so after a while I can watch it once again without loosing a lot of time trying to remember what was the video headline name

5. To use youtube-dl in a bit advanced way you can for instance invoke the command with options like:

debian:~# ./youtube-dl -l -w -c https://www.youtube.com/watch?v=g7tvI6JCXD0
[youtube] Setting language
[youtube] g7tvI6JCXD0: Downloading video webpage
[youtube] g7tvI6JCXD0: Downloading video info webpage
[youtube] g7tvI6JCXD0: Extracting video information
[download] Destination: BSD is Dying, Jason Dixon, NYCBSDCon 2007-g7tvI6JCXD0.flv
[download] 4.4% of 22.62M at 1.43M/s ETA 00:15

As you can see now youtube-dl was even able to detect the downloaded video file name and store it on the computer with a correct name 😉

I would recommend you also to check out the youtube-dl help page, to do use command: youtube-dl –help
 

Tracking graphically MySQL, Apache and Network performance in web with Munin on Debian Linux

Saturday, March 19th, 2011

munin_monitor_cpu_network_hard_drive_apache_mysql_performance-in-web-on-linux_logo

Munin is great software for surveillance software for your MySQL, Apache, Qmail, Postfix and many other of the classical daemon services which most of the Linux, BSD servers online are running on.

1. Munin on Debian Linux is really easy to be installed and comes to executing:

debian:~# apt-get install munin munin-node munin-plugins-extra

2. Link /var/www/munin/ to some VirtualHost or ServerHost
For instance if your munin is to be placed on the domain pc-freak.net.net which has the DocumentRoot of /var/www

debian:~# cd /var/www
debian:/var/www# ln -sf /var/www/munin munin

3. Add some extra MySQL related plugins

To check whole list of pmunin plugins you can enable check in /etc/munin/plugins directory

Lets say our Munin will be providing statistics mainly for MySQL, enable this bunch of plugins:

cd /etc/munin/plugins
ln -s /usr/share/munin/plugins/mysql_ mysql_
ln -s /usr/share/munin/plugins/mysql_bytes mysql_bytes
ln -s /usr/share/munin/plugins/mysql_innodb mysql_innodb
ln -s /usr/share/munin/plugins/mysql_isam_space_ mysql_isam_space_
ln -s /usr/share/munin/plugins/mysql_queries mysql_queries
ln -s /usr/share/munin/plugins/mysql_slowqueries mysql_slowqueries
ln -s /usr/share/munin/plugins/mysql_threads mysql_threads


Though above command would enable mysql monitoring it will be not working out of the box. You will find in munin logs error like:

Missing dependency Cache::Cache at /usr/share/munin/plugins/mysql_ line 716."

To fix this install libcache-perl and libcache-cache-perl

apt-get install --yes libcache-perl libcache-cache-perl
 

Then regenerate munin plugin configuration issue cmd:

munin-node-configure --suggest --shell | sh

and restart Munin-node server

service munin-node restart
 

Munin Node should be able login to the MySQL server using the Debian System Maintenance user however if you should need to create a custom user you can add the following configuration to:
 

/etc/munin/plugin-conf.d/munin-node

 

[mysql*]
    user root
    env.mysqluser muninmonitor
    env.mysqlpassword Your-Secret-Password

4. Edit Munin config

Make sure you have at least the following variables enabled in /etc/munin/munin.conf
 

 dbdir  /var/lib/munin
 htmldir /var/cache/munin/www
 logdir /var/log/munin
 rundir  /var/run/munin

 tmpldir    /etc/munin/templates

includedir /etc/munin/munin-conf.d

[localhost.localdomain]
    address 127.0.0.1
    use_node_name yes

 

5. Add Munin Alias or Virtualhost

To make Munin Accessible from Webserver you need to at least add an alias create a file
/etc/apache2/conf.d/munin with following content:

vim /etc/apache2/conf.d/munin

Alias /munin /var/cache/munin/www

<Directory /var/cache/munin/www>
        Order allow,deny
        #Allow from localhost 127.0.0.0/8 ::1
        Allow from all
        Options None
        php_value engine off

    <IfModule mod_expires.c>
        ExpiresActive On
        ExpiresDefault M310
    </IfModule>

</Directory>


If you want to access Munin from Apache subdomain create it lets say you want to access it via stats.pc-freak.net, use following config file:

vi /etc/apache2/sites-enabled/stats.pc-freak.net
 

<VirtualHost *>
   ServerAdmin hipo@pc-freak.net
   ServerName stats.pc-freak.net
   DocumentRoot /var/cache/munin/www
   <Directory />
       Options FollowSymLinks
       #AllowOverride All
       php_value engine off
   </Directory>
   LogLevel debug
   CustomLog /var/log/apache2/munin/access.log combined
   ErrorLog /var/log/apache2/munin/error.log
   ServerSignature On
<Directory /var/cache/munin/www>
        Order allow,deny
        #Allow from localhost 127.0.0.0/8 ::1
        Allow from all
        Options FollowSymlinks
        #AllowOverride All
        php_value engine off

</Directory>

</VirtualHost>

 


6. Restart Apache;

debian:~# /etc/init.d/apache2 restart

Now to access the newly installed munin point your favorite browser to (either the subdirectory configured for access or the Virtualhost subdomain):

http://www.pc-freak.net/munin/
or
http://munin.pc-freak.net

eth0-network-traffic-munin-server-statistics

connections_through_firewall-statistics-munin-debian-linux
7. Protecting Munin with a password

As it could be a security leak to expose statistics regarding your server (Disk I/O, Memory, Load Avarage, Apache, MySQL Postfix) operations it is good security practice to protect munin with a password.

I've earlied blogged on how to protect your server munin web-statistics wtih an htaccess password (read it here)


munin-cup-statistics-by-day-and-by-month-screenshot

Happy Monitoring