Posts Tagged ‘open source’

Saving multiple passwords in Linux with Revelation and Keepass2 – Keeping track of multiple passwords

Thursday, October 17th, 2013

System Administrators who use MS Windows to access multiple hosts in big companies like HP or IBM certainly use some kind of multiple password manager like PasswordSafe.

Keep multiple passwords safe in Microsoft Windows 7 passwordsafe with masterpassword

When number of passwords you have to keep in mind grows significantly using something like PasswordSafe becomes mandatory. Same is valid also for valid for system administrators who use GNU / Linux as a Desktop environment to administer thousands or hundreds of servers. I'm one of those admins who for years use Linux and until recently I kept logging all my passwords in separate directory full with text files created with vim (text editor). As the number of passwords and accesses to servers and web interfaces grow up dramatically as well as my requirement for security raised up I wanted to have my passwords secured being kept encrypted on my hard drive. For those who never use PasswordSafe the idea of program is to store all passwords you have in encrypted database which can be only opened through PasswordSafe by providing a master password.

passwordsafe on microsoft windows keep in order multiple passwords manager

Of course having one master password imposes other security risks as someone who knows the MasterPass can easily access all your passwords anyways for now such level of security perfectly fits my needs.

PasswordSafe is since recently Open Source so there is a Linux port, but the port is still in beta and though I tried hard to install it using provided .deb binaries as well as compile from source, I finally give it up. And decided to review what kind of password managers are available in Debian Wheezy's ports.

Here are those I found with;

apt-cache search password|grep -i manager

cpm – Curses based password manager using PGP-encryption
fpm2 – password manager with GTK+ 2.x GUI
gringotts – secure password and data storage manager
kedpm – KED Password Manager
kedpm-gtk – KED Password Manager
keepass2 – Password manager
keepassx – Cross Platform Password Manager
kwalletmanager – secure password wallet manager
password-gorilla – cross-platform password manager
revelation – GNOME2 Password manager

I didn't have the time to test each one of them, so I installed and checked only those which seemed more reliable, i.e.:
keepass2 and revelation

# apt-get install –yes fpm2 keepass2 revelation

Below is screenshot of each one of managers:

Revelation Linux Gnome graphic password manager program

Revelation – GNOME Password Manager

keepass2 Linux gui password manager screenshot Debian - graphic manager for storing passwords

kde password safe gui program Linux Debian screenshot

KDE QT Interface Linux GUI Password Manager (KeePass2)

With one of this tools admin's life is much easier as you don't have to get crazy and remember thousands of passwords.
Hope this helps some admin out there! Enjoy ! 🙂
 

How to remove the meta generator Content (Joomla! – Copyright) in Joomla 1.5

Thursday, December 30th, 2010

Joomla-remove-meta-generator-content-to-hide-joomla-site-install
Do you wonder How to change <meta name="Generator" content="Joomla! – Copyright (C) 2005 – 2007 Open Source Matters. All rights reserved." /> in Joomla 1.5

If yes, Here is how I've just found to remove the:

 

in my Joomla installation.

I need to remove that as a part of making my website not to leak out that it runs on top of Joomla.

So here is how:

1. Go to your Joomla website main root directory
2. Edit /libraries/joomla/document/html/renderer/head.php
Look for line: 83 in the /libraries/joomla/document/html/renderer/head.php
There you will notice the code:

$strHtml .= $tab.'<meta name="generator" content="'.$document->getGenerator().'" />'.$lnEnd;

In order to remove the <meta name="generator" content="Joomla …." /> change the above code to something like:

$strHtml .= $tab.'<meta name="generator" content="My Custom Web site Generator name" />'.$lnEnd;

That's all now next time you refresh your website the content="Joomla! – Copyright (C) 2005 – 2009 Open Source Matters. All rights reserved." will be no more.
Cheers! 🙂

Create video from linux console / terminal – Record ssh terminal session as video with asciinema, showterm, termrecord

Thursday, August 21st, 2014

/var/www/images/asciinema-create-and-upload-ascii-terminal-console-videos-debian-gnu-linux-screenshot
You probably already know of existence of two Linux commands available by default across all Linux distributions scriptwhich makes a text based save of all commands executed on console and scriptreplay – which playbacks saved script command typescripts. Using this two you can save terminal sessions without problem, but in order to play them you need to have a Linux / UNIX computer at hand.
However If you want to make a short video record displaying what you have done on Linux console / terminal, you have few other options with which you can share your Linux terminal sessions on the web. In this short article I will go through 3 popular tools to do that – asciinema, showterm and termrecord.

1. Asciinema Current most popular tool to create video from Linux terminal

Here is how ASCIINEMA's website describes it:

"Asciinema is a free and open source solution for recording the terminal sessions and sharing them on the web."

apt-get –yes install python-pip

To install it with pip python package installer

pip install asciinema

Or if the machine is in DMZ secured zone and have access to the internet over a Proxy:

pip install –proxy=http://internet-proxy-host.com:8080 asciinema

It will get installed in /usr/local/bin/asciinema to make a terminal screen video capture just launch it (nomatter if it is privileged or non-privileged user):

asciinema

To finalize and upload the recorded terminal session, just type exit (to exit the shell), hopefully it will get you an upload link.

exit

You can claim authorship on video you issue:

asciinema auth

Use can then embed the new Linux terminal session video to your website.
 

2. ShowTerm – "It's showtime in a terminal near you!"

ShowTerm have same features as AsciiNema. Just like AsciiNema, what it does is it creates a record of your terminal session and then uploads it to showterm.io website, providing you a link over which you can share your terminal lesson / ascii art video / whatever with your friends. ShowTerm is written in, the world famous Ruby on Railsruby web development framework, so you will need to have ruby programming language installed before use. As showterm uses the Internet to upload video, so it is not really an option to create videos from remote terminal session on servers which are in DMZ with no access to the internet, I will explain in a little while how to create video of your terminal / console for private purpose on local server and then share it online on your own site.

a) To install ShowTerm:

– First be sure to have ruby installed:

On Debian / Ubuntu and derives deb Linux, as supersuser:

apt-get install –yes ruby curl

On CentOS / RHEL / Fedora Linux

yum -y install ruby curl

NB! curl is real requirement but as showterm.io website recommends downloading the script with it and later same curl tool is used to upload the created showterm file to http://showterm.io .

– Then to finalize install, download showterm script and make it executable

curl showterm.io/showterm > ~/bin/showterm

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
100  2007  100  2007    0     0   2908      0 –:–:– –:–:– –:–:–  8468

mkdir ~/bin
chmod +x ~/bin/showterm

This will save the script into your home folder ~/bin/showterm

b) Using showterm

To run it to create video from your terminal simply start it and do whatver you will in terminal.

~/bin/showterm

After you're done with the video you like type exit

exit

create-video-from-your-linux-console-terminal-with-showterm-screenshot

Note that if your server is behind a proxy curl will not understand proxy set inside Linux shell variable with http_proxy var, to upload the file if you're behind a proxy you will have to pass to curl –proxy setting, once you get the curl line invoked after failure to upload use something like:

curl –proxy $(echo $http_proxy)  https://showterm.herokuapp.com/scripts –data-urlencode cols=80 –data-urlencode lines=24 –data-urlencode scriptfile@/tmp/yCudk.script –data-urlencode timingfile@/tmp/lkiXP.timing

Where assuming proxy is defined already inside http_proxy shell variable.

 

3. Creating video from your terminal / console on Linux for local (private) use with TermRecord

In my humble view TermRecord is the most awesome of all the 3, as it allows you to make records with an own generated Javascript based video player and allows you to keep the videos on your own side, guaranteeing you independence of external services. Its
 

pip install TermRecord

TermRecord -o /tmp/session.html

 

You can further access the video in a local browser in Firefox / Chrome / Epiphany type in URL address bar:

/tmp/session.html to play the video

create-video-from-terminal-console-on-gnu-linux-howto-screenshot-with-termrecord

TermRecord uses term.js javascript to create the video web player and play the video which is directly encoded inside session.html.
If you want to share the video online, place it on your webserver and you're done 🙂
Check out my TermRecord generated video terminal sample session here.
 

Web and Middleware JBoss Training at Hewlett Packard – Intro to JBoss JAVA application server

Thursday, November 21st, 2013

jboss application server logo- serve java servlet pages on Linux and Windows

I and my TEAM Web and Middleware Implementation Team @Hewlett Packard are assigned an online training to follow on topic of JBoss Application server.It is my first online training of this kind where a number of people are streamed a video from a trainer who explains in real time concepts of JBossA Community Drive open source middleware (Application Server), since some time JBoss is known under a new name (WildFly).

Wildfly new name of jboss application java servlet server

In short what is JBoss? – It is an application server similar to Apache Tomcat  -an open source software implementation of the Java Servlet and JavaServer Pages technologies.

Apache Tomcat java servlet application server logo

In case you wonder about what is Middleware it is a buzzword well established in Corporate world referring to all kind of servers in the middle between Servers on pure OS and hardware Level and end client. Middleware includes all kind of Web and Application servers like Apache, JBoss, Tomcat, Adobe's WebLogic Webserver, IBM WebSphere application server etc..

What this means is JBOSS is very similar to Tomcat but it is designed to run interpret through (Java Virtual Machine), higher scale of Java Applications and then return content to a a web browser. In other words if you need to have a Webserver with support for Java VM. JBoss is one of the open source technologies available which can be a substitute for Tomcat. In Fact Jboss itself started as a fork of Tomcat and n owadays, Jboss has an implementation of Tomcat embedded into itself. Jboss is mainly developed and supported by Redhat. It has 3 major releases used in IT Companies. Jboss 5, JBoss 6 and JBoss 7. In most production server systems running some kind of Java servlets currently still Jboss ver. 5 and Jboss v. 6 is used. Just like Tomcat, the server is messy in its structure. But if we have to compare Tomcat with Jboss then JBoss is at least 100 times more messy and hard to configure tune than Tomcat. Actually after getting to know JBoss 6 I would not advice anyone to use this Application server. Its too complex and all configuration and performance tuning is done through hundred of XML so it is like a hell for the usual System Administrator who likes clearness and simplicity. JBoss has a Web configuration interface which in version 7 is a bit advanced and easier to configure and get to know the server compared to previous versions. But same web interface for older releases is lousy and not nice. Just like Tomcat, JBoss supports clustering, here is full list of all features it supports:

  • Full clustering support for both traditional J2EE applications and EJB 3.0 POJO applications
  • Automatic discovery. Nodes in cluster find each other with no additional configuration.
  • Cluster-wide replicated JNDI context
  • Failover and load-balancing for JNDI, RMI and all EJB types
  • Stateful Session Bean state replication
  • HTTP Session replication
  • High Availability JMS
  • Farming. Distributed deployment of JBoss components. Deploying on one node deploys on all nodes.
     

Looks like JBoss is among the few Application Servers supporting deployment of Java JSP, WAR Archive files, SAR Archives, JMS (Java Message Service), JNDI (Java Naming and Directory Interface). Jboss supports load balancing between clustered nodes, supports SOAP, Java servlet faces and Java MQ (Messaging Queue). JBoss can be installed on GNU / Linux, FreeBSD and Windows. So far from what I've learned for JBOSS I prefer not to use it and don't recommend this Application server to anyone. Its too complex and doesn't worth the effort to learn. Proprietary products like WebLogic and Webspehere are in light years better.

MobaXTerm: A good gnome-terminal like tabbed SSH client for Windows / Windows Putty Tabs Alternative

Wednesday, November 13th, 2013

Mobaxterm ssh client putty MS Windows alternative with tabs suitable for ex linux users

mobaxterm with tabbed ssh connections screenshot best putty windows ssh client alternative now

Last 10+ years I worked on GNU / Linux as Desktop. Last 7 years most of my SSH connections were managed from GNOME and I'm quite used to gnome-terminal ssh tabbing. In my new Employee Hewlett Packard. I'm forced to work on Microsoft Windows 7 and thus I used for a month or so Putty and Kitty fork from version 0.63 of PuTTY advertising itself as the best telnet / SSH client in the world. Both of the two lack tabbing and have interface which is pretty unfamiliar to me. As I'm so used to using native UNIX terminal. Fortunately a colleague of mine Ivelin was using an SSH client called MobaXTerm which very much did emulation similar to my favourite gnome-terminal. MobaXterm is not free software / open source app but this doesn't matter so much to me as anyways I'm running a non-free Win OS on my desktop. What makes MobaXterm so attractive is its rich functionality (cosmic years infront of Putty).

Here is website description of MobaXterm quoted from its website:

MobaXterm is an enhanced terminal for Windows with an X11 server, a tabbed SSH client and several other network tools for remote computing (VNC, RDP, telnet, rlogin). MobaXterm brings all the essential Unix commands to Windows desktop, in a single portable exe file which works out of the box.

Overall list of features MobaXterm offers are;

  •     multitab terminal with embedded Unix commands (ls, cd, cat, sed, grep, awk, rsync, wget, …)

  •     embedded X11 server for easily exporting your Unix/Linux display

  •     a session manager with several network utilities: SSH, RDP, VNC, Telnet, Rlogin, FTP, SFTP and XDMCP

  •     passwords management for SSH, RDP, VNC, SFTP (on demand password saving)

  •     easy graphical file transfer using drag and drop during SSH sessions

  •     advanced SSH tunnels creation tool (graphical port forwarding builder)

  •     tasks automation using scripts or macros

Mobaxterm is portable just like Putty so its useful to use on HOP stations to servers like used in big companies like HP. Featured embedded Unix commands (e.g., ls, cd, cat, sed, grep, awk, rsync, wget) gives a feeling like you're working on pure Linux console making people addicted to Linux / BSD quite confortable. Some other very useful terminal emulator functions are support for anti-aliasing session manager (save / remember passwords for ssh sessions in Crypted format so much missing in Putty) and it even supports basic macros.
Basic UNIX commands embedded in MobaXterm are taken and ported from Cygwin projectLinux-like environment for Windows making it possible to port software running on POSIX systems (such as Linux, BSD, and Unix systems) to Windows. A very cool think is also MobaXterm gives you a Linux like feel of console navigation in between basic files installed from Cygwin. Some downside I found is program menus which look at first glimpse a bit confusing especially for people used to simplicity of gnome-terminal. Once logged in to remote host via ssh command the program offers you to log you in also via SFTP protocol listing in parallel small window with possibility to navigate / copy / move etc. between server files in SFTP session which at times is pretty useful as it saves you time to use some external SFTP connector tools like  WinSCP.

From Tools configuration menu, there are few precious tools as well;
         – embedded text editor MobaTextEditor
         – MobaFoldersDiff (Able to show diffeernces between directories)
         – AsciiTable (Complete List of Ascii table with respective codes and characters)
         – Embedded simple Calculator
         – List open network ports – GUI Tool to list all open ports on Windows localhost
         – Network packets capture – A Gui tool showing basic info like from UNIX's tcpdump!
         – Ability to start quickly on local machine (TFTP, FTP, SFTP / SSH server, Telnet server, NFS server, VNC Server and even simple implementation of HTTP server)

Mobaxterm list of tools various stuff

         Mobaxterm run various services quickly on Windows servers management screenshot

Below are few screenshots to get you also idea about what kind of configuration MobaXterm supports
  mobaxterm terminal configuration settings screenshot

mobaxterm better putty alternative x11 configuration tab screenshot

mobaxterm windows ssh client for linux users configuration ssh tab screenshot

mobaxterm-putty-alternative-for-windows-configuration-display-screenshot
MobaXTerm Microsoft Windows ssh client configuration misc menu screenshot
To configure and use Telnet, RSH, RDP, VNC, FTP etc. Sessions use the Sessions tab on top menu.

One very handy thing is MobaXterm supports export of remote UNIX display with no requirement to install special Xserver like already a bit obsolete Xming – X server for Windows.
The X Display Manager Control Protocol (XCMCP) is a key feature of the X11 architecture. Together with XDMCP, the X network protocol allows distributed operation of the X server and X display manager. The requesting X server runs on the client (usually as an X terminal), thus providing a login service, that why the X server ported to MobaXterm from Cygwin also supports XDMCP. If, for example, you want to start a VNC session with a remote VNC server, all you have to do is enter the remote VNC server’s IP address in the VNC area; the default VNC port is already registered.

Accessing the remote Windows server via RDP (Remote Desktop Protocol) is also a piece of cake. Once you establish a session to RDP or other Proto it is possible to save this session so later you just choose between session to access. The infamous (X11 Port Forwarding) or creation of SSH encrypted tunnels between hosts to transfer data securily or hide your hostname is also there.

MobaXterm is undoubtedly a very useful and versatile tool. Functionally, the software is well mannered, and Windows users who want to sniff a little Linux/Unix air can get a good idea of how Linux works. A closer look reveals that anything you can do with MobaXterm can be achieved directly with freely available tools (Cygwin) and Unix tools ported from Cygwin. However, although Cygwin provides a non-Posix environment for Windows, it doesn’t offer a decent terminal, which is one thing Moba-Xterm has going for it.

Admittedly, in pure vanilla Cygwin, you can start an X server automatically and then use xterm, but xterm lacks good-quality fonts, whereas MobaXterm conveniently lets you integrate a font server.

How to solve ALSA sound problems with old Linux programs and games depending on (OSS)’s /dev/dsp / fix wine games and pulseaudio problems – My few thoughts on OSS and ALSA

Wednesday, January 11th, 2012

 

ALSA OSS Pulseaudio ESD Some fixes workaround to gnu linux audio messI remember GNU / Linux, 11 years from now, times when ALSA was not standardly shipped with Linux.
Back then ALSA still lacked good support for many SoundCards and was still a "baby project".
In that time what we used to have sound on Linux was OSSOpen Sound System. OSS emerged right after the first ever Linux sound system VoxWare (formerly known as the Linux Sound Driver).

Back in those days OSS was used for multimedia support on both GNU / Linux and BSD based free OSes. It was few years later when I heard and used ALSA for a fist time and it wasn't really a love from first sigth.

One can easily find out by the name ALSA it is a system especially built for the Linux kernel and that's one of the reasosn why *BSD systems has their custom separate sound system.
There is plenty of reasons why OSS was substituted with ALSA. Main reason was its commercial like license, OSS wasn't completely "open source" GPLed (free software), there was resctions on use of OSS for commercial goals.

With its emerge ALSA started to push away OSS slowly. Somewhere in 2003, alsa has officially entered the Linux kernel source and until 2005 it was the default standard for all GNU / Linux operating systems.

As of time of writting ALSA has become the only sound system to have support for multiple sound card devices for Linux.
My experiences with ALSA, however ain't so nice if I take a look in my past experiences.
Since the very beginning of using ALSA, I had plenty of troubles with configuring properly my sound card not to mention, even after configuring it the MIDI support was not there.
Besides all the troubles main problems were stemming from the many applications still written to use OSS as sound system and hence with those sound was impossible with ALSA.The most problematic thing about apps written with OSS in mind was all of them tried to stream sound via /dev/dsp (OSS Digital Sound Processor), since alsa did not used /dev/dsp those programs was soundless.

On the other hand OSS was creating issues as well, one severe problem with OSS was the inability to stream multiple sounds simultaneously, because each sound stream required to pass voice through /dev/dsp and usually there was only one /dev/dsp.

The message;

/dev/dsp: Device or resource busy
and the proceeding irritation that used to annoy us in the early GNU / Linux days had of course some raw workarounds hacks but generally the workaround did not fix problems always.

Introduction of alsa free us from /dev/dsp issues but on the other handy has created a whole ocean of new BIG problems …
ALSA has modular structure and this imposes a great problem nowdays. The modular architecture is generally a good idea, however the way this was implemented within ALSA is far away from clear and easy to understand by the end user and therefore makes it very unintuitive and obscure.
Alsa misses simplicity which somehow was partially in the days of OSS. Thinking over the general situation with Linux multimedia nowdays, I believe it was exactly ALSA Project responsible for the so delayed mass Desktop Linux adoption.

Many long year standing Linux users had certainly had the alsa troubles during new system installs (correct me if I'm wrong).
The only fix to multiple soundcard initialization problems was to download alsa source and compile from source and hence made it hard and discouraging for people giving Linux a try.
This kind of ALSA "brokenness" pattern continues even to this very day (in Debian) Linux and probably building the alsa system from source is among the good practices to have a functional Linux sound system…

With all said the historic reason why ALSA was not quickly adopted and still is not a preferred default system for many applications ported to Free Software OSes by commercial company vendors is clear. Its simply not working out of the box …

Hope some ALSA developers will read this post work on changing the crazy structure of ALSA over complexity. ALSA needs automate way to solve issues with itself, the configuration should be more trivial and unified if Linux has to become more attractive for Desktop adoption.

Anyways, after the few words of history and indicating my pesonal observations on ALSA. I will proceed and explain few things on how ALSA can be configured to support and play nice with OSS dependant programs as well some basic explanations on common incompatibility between esd and pulseaudio and how this can be fixed;.

To assure nowdays OSS API built programs and games would work with Alsa its necessery to have installed;

ALSA wrapper for OSS applications

On Debian, Ubuntu, Fedora and most Linux distributions the Alsa OSS compatability layer comes under a (deb / rpm) package named alsa-oss

To install OSS compatability on Debian, Ubuntu and the like Debian based distributions issue:

debian:~# apt-get install alsa-oss alsaplayer-oss
...

On Fedora and other rpm based distributions install is with:

[root@fedora ~]# yum install alsa-oss alsaplayer-oss
...

alsa-oss provides with a command called aoss that should be used to work around some issues with old applications still depending on OSS:

hipo@debian:~$ aoss programName

Using aoss is helpful especially in situations if you have to run programs which deal with MIDI and others which somehow want to use /dev/dsp

There is also alternative way to enable alsa native support for MIDI and OSS by loading 3 kernel modules:

debian:~# modprobe snd-seq-oss
debian:~# modprobe snd-pcm-oss
debian:~# modprobe snd-mixer-oss

Note! The three modules has to be separately build using kernel source at most cases and does not come with most Linux distributions, so on many installations (including my current), they will be missing. If for you they load properly or you have customly build them add them also to load on system boot, like so:

echo 'snd-seq-oss' >> /etc/modules
echo 'snd-pcm-oss' >> /etc/modules
echo 'snd-mixer-oss' >> /etc/modules

The Linux sound situation becomes even more messy when ESD enters the scene. Many of the novice new Linux users certainly don't remember (Enlightened Sound Daemon) . ESD historically preceded PulseAudio . Hence it will be good to mention ESD was used for few years in GNOME and in around 2006-2007 it was substituted by PulseAudio.
Many applications, however who was ported or written for Linux especially (the proprietary ported ones) was already built to work with ESD and even though newer GNOME releases was fully using pulseaudio, this (non free software apps and games) were still depending on ESD.

The situation was partially fixed by creation of module for pulseaudio which added emulation support for esd . This was done by a module library for pulseaudio called libprotocol-esound.so
The package for almost all Linux distributions which does the esd emulation via pulse is pulseaudio-esound-compat . In latest Fedora Linux pulseaudio-esound-compat is installed by default.
In Debian and other Linux distributions it might need to be installed via apt with;

debian:~# apt-get install pulseaudio-esound-compat
...

pulseaudio-esound-compat solves some of the ESD app incompability but not always …
Handy tool also worthy to mention in solving PulseAudio, OSS incompatibility issues is padsp

padsp is helpful in solving obsolete issues with OSS applications (trying to access /dev/dsp) and therefore unable to communicate with Pulseaudio
padsp – is a PulseAudio OSS Wrapper.

An example where padsp is helpful is in case of /dev/dsp errors like:

/dev/dsp: Device or resource busy
Could not open /dev/dsp

Another common problem with sound on Linux is when running windows applications (running windows games with wine).
Quite often sound fails to work since wine tries to directly communicate with alsa and fails because alsa sound channel is taken by pulseaudio.

To workaround wine issues with pulseaudio, one of the solutions is to temporary stop pulseaudio, before running the wine emulated application:

hipo@debian:~$ pulseaudio --kill

Later on when the windows wine emulation is completed, pulseaudio has to be started once again in order to make Pulseaudio applications produce sound again, e.g. one has to issue:

hipo@debian:~$ pulseaudio --start
Alternative way to workaround wine sound issues is by using a script to kill pulseaudio every second. Here is fix_pulseaudio_wine_sound_probs.sh script

This script was reported by many people as fix to problems with wine games failing to play sounds and music, anyhow I personally prefer using the stop / start pulseaudio method.

The picture below is taken from Wikipedia and illustrates, clearly the intergalactical complexity of sound systems on Gnu / Linux and BSD

I just hope one day this (OSS, ALSA, esd, Pulseaudio) mess will be over! In the mean time I hope my suggested work arounds helps someone. If someone has a better more unified script or solution please share in comments

How to convert any internet Webpage to PDF from command line on GNU/Linux

Friday, September 30th, 2011

Linux webpage html to pdf command line convertor wkhtmltopdf

If you're looking for a command line utility to generate PDF file out of any webpage located online you are looking for Wkhtmltopdf
The conversion of webpages to PDF by the tool is done using Apple's Webkit open source render.
wkhtmltopdf is something very useful for web developers, as some webpages has a requirement to produce dynamically pdfs from a remote website locations.
wkhtmltopdf is shipped with Debian Squeeze 6 and latest Ubuntu Linux versions and still not entered in Fedora and CentOS repositories.

To use wkhtmltopdf on Debian / Ubuntu distros install it via apt;

linux:~# apt-get install wkhtmltodpf
...

Next to convert a webpage of choice use cmd:

linux:~$ wkhtmltopdf www.pc-freak.net www.pc-freak.net_website.pdf
Loading page (1/2)
Printing pages (2/2)
Done

If the web page to be snapshotted in long few pages a few pages PDF will be generated by wkhtmltopdf
wkhtmltopdf also supports to create the website snapshot with a specified orientation Landscape / Portrait

-O Portrait options to it, like so:

linux:~$ wkhtmltopdf -O Portrait www.pc-freak.net www.pc-freak.net_website.pdf

wkhtmltopdf has many useful options, here are some of them:
 

  • Javascript disabling – Disable support for javascript for a website
  • Grayscale pdf generation – Generates PDf in Grayscale
  • Low quality pdf generation – Useful to shrink the output size of generated pdf size
  • Set PDF page size – (A4, Letter etc.)
  • Add zoom to the generated pdf content
  • Support for password HTTP authentication
  • Support to use the tool over a proxy
  • Generation of Table of Content based on titles (only in static version)
  • Adding of Header and Footers (only in static version)

To generate an A4 page with wkhtmltopdf:

wkhtmltopdf -s A4 www.pc-freak.net/blog/ www.pc-freak.net_blog.pdf

wkhtmltopdf looks promising but seems a bit buggy still, here is what happened when I tried to create a pdf without setting an A4 page formatting:

linux:$ wkhtmltopdf www.pc-freak.net/blog/ www.pc-freak.net_blog.pdf
Loading page (1/2)
OpenOffice path before fixup is '/usr/lib/openoffice' ] 71%
OpenOffice path is '/usr/lib/openoffice'
OpenOffice path before fixup is '/usr/lib/openoffice'
OpenOffice path is '/usr/lib/openoffice'
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
Printing pages (2/2)
Done
Printing pages (2/2)
Segmentation fault

Debian and Ubuntu version of wkhtmltopdf does not support TOC generation and Adding headers and footers, to support it one has to download and install the static version of wkhtmltopdf
Using the static version of the tool is also the only option for anyone on Fedora or any other RPM based Linux distro.

Scanning shared hosting servers to catch abusers, unwanted files, phishers, spammers and script kiddies with clamav

Friday, August 12th, 2011

Clamav scanning shared hosting servers to catch abusers, phishers, spammers, script kiddies etc. logo

I’m responsible for some GNU/Linux servers which are shared hosting and therefore contain plenty of user accounts.
Every now and then our company servers gets suspended because of a Phishing websites, Spammers script kiddies and all the kind of abusers one can think of.

To mitigate the impact of the server existing unwanted users activities, I decided to use the Clamav Antivirus – open source virus scanner to look up for potentially dangerous files, stored Viruses, Spammer mailer scripts, kernel exploits etc.

The Hosting servers are running latest CentOS 5.5. Linux and fortunately CentOS is equipped with an RPM pre-packaged latest Clamav release which of the time of writting is ver. (0.97.2).

Installing Clamav on CentOS is a piece of cake and it comes to issuing:

[root@centos:/root]# yum -y install clamav
...

After the install is completed, I’ve used freshclam to update clamav virus definitions

[root@centos:/root]# freshclam
ClamAV update process started at Fri Aug 12 13:19:32 2011
main.cvd is up to date (version: 53, sigs: 846214, f-level: 53, builder: sven)
WARNING: getfile: daily-13357.cdiff not found on remote server (IP: 81.91.100.173)
WARNING: getpatch: Can't download daily-13357.cdiff from db.gb.clamav.net
WARNING: getfile: daily-13357.cdiff not found on remote server (IP: 163.1.3.8)
WARNING: getpatch: Can't download daily-13357.cdiff from db.gb.clamav.net
WARNING: getfile: daily-13357.cdiff not found on remote server (IP: 193.1.193.64)
WARNING: getpatch: Can't download daily-13357.cdiff from db.gb.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Downloading daily.cvd [100%]
daily.cvd updated (version: 13431, sigs: 173670, f-level: 60, builder: arnaud)
Downloading bytecode.cvd [100%]
bytecode.cvd updated (version: 144, sigs: 41, f-level: 60, builder: edwin)
Database updated (1019925 signatures) from db.gb.clamav.net (IP: 217.135.32.99)

In my case the shared hosting hosted websites and FTP user files are stored in /home directory thus I further used clamscan in the following way to check report and log into file the scan results for our company hosted user content.

[root@centos:/root]# screen clamscan -r -i --heuristic-scan-precedence=yes --phishing-scan-urls=yes --phishing-cloak=yes --phishing-ssl=yes --scan-archive=no /home/ -l /var/log/clamscan.log
home/user1/mail/new/1313103706.H805502P12513.hosting,S=14295: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/new/1313111001.H714629P29084.hosting,S=14260: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/new/1305115464.H192447P14802.hosting,S=22663: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/new/1311076363.H967421P17372.hosting,S=13114: Heuristics.Phishing.Email.SpoofedDomain FOUND/home/user1/mail/domain.com/infos/cur/859.hosting,S=8283:2,S: Heuristics.Phishing.Email.SSL-Spoof FOUND/home/user1/mail/domain.com/infos/cur/131.hosting,S=6935:2,S: Heuristics.Phishing.Email.SSL-Spoof FOUND

I prefer running the clamscan in a screen session, because it’s handier, if for example my ssh connection dies the screen session will preserve the clamscan cmd execution and I can attach later on to see how scan went.

clamscan of course is slower as it does not use Clamav antivirus daemon clamd , however I prefer running it without running the daemon, as having a permanently running clamd on the servers sometimes creates problems or hangs and it’s not really worthy to have it running since I’m intending to do a clamscan no more than once per month to see some potential users which might need to be suspended.

Also later on, after it finishes all possible problems are logged to /var/log/clamscan.log , so I can read the file report any time.

A good idea might also be to implement the above clamscan to be conducted, once per month via a cron job, though I’m still in doubt if it’s better to run it manually once per month to search for the malicious users content or it’s better to run it via cron schedule.

One possible pitfall with automating the above clamscan /home virus check up, might be the increased load it puts to the system. In some cases the Webserver and SQL server might be under a heavy load at the exactly same time the clamscan cron work is running, this might possible create severe issues for users websites, if it’s not monitored.
Thus I would probably go with running above clamscan manually each month and monitor the server performance.
However for people, who have “iron” system hardware and clamscan file scan is less likely to cause any issues, probably a cronjob would be fine. Here is sample cron job to run clamscan:

10 05 01 * * clamscan -r -i --heuristic-scan-precedence=yes --phishing-scan-urls=yes --phishing-cloak=yes --phishing-ssl=yes --scan-archive=no /home/ -l /var/log/clamscan.log >/dev/null 2>&1

I’m interested to hear if somebody already is using a clamscan to run on cron without issues, once I’m sure that running it on cron would not lead to server down-times, i’ll implement it via cron job.

Anyone having experience with running clamscan directory scan through crond? 🙂