Posts Tagged ‘repositories’

ZenMap Nmap multi platform Graphical frontend for checking port security

Saturday, June 15th, 2013

graphic program to scan remote network server port security on GNU Linux and Windows ZenMap

Recently I wrote little article with some examples for scanning server port security with Nmap. I forgot to mention in the article that there is also Nmap frontend GUI program called ZenMap. ZenMap port is available for both Windows and Linux. In Debian, Ubuntu, Mint and other debian derivative distributions ZenMap is available from standard package repositories;

 noah:~# apt-cache show zenmap|grep -i description -A 3

Description-en: The Network Mapper Front End
 Zenmap is an Nmap frontend. It is meant to be useful for advanced users
 and to make Nmap easy to use by beginners. It was originally derived
 from Umit, an Nmap GUI created as part of the Google Summer of Code.
Description-md5: 4e4e4c6aeaa4441484054473e97b7168
Tag: implemented-in::python, interface::x11, network::scanner, role::program,
 uitoolkit::gtk, use::scanning, x11::application
Section: net

To install  ZenMap on Debian / Ubuntu Linux:

noah:~# apt-get install --yes zenmap
...

In Fedora, CentOS and other RPM based Linux-es to install ZenMap run:

noah:~# yum -y install nmap-frontend nmap
...

To use Nmap's Frontend full functionality, you have to run it as (root) superuser:

hipo@noah:~$ sudo su
[sudo] password for hipo:
noah:~# zenmap

Zenmap saves, a lot of time as there is no need to  remember Nmap's arguments or run few Nmap scans until you get essential information for remote scanned machine.
It automatically gives details on Remote server running services (fingerprint)

Zenmap remote server security services scan with services software version

Very useful report it makes as well is network (and host) topology diagram,

network scanner remote host Linux Windows toplogy guess ZenMap screenshot

ZenMap is just Nmap frontend and under the GUI it does use Nmap with various arguments to do produce scan results. In Nmap Output tab, you can see a lot of verbose info.

Zenmap Linux Windows GUI port scanne nmap output tab screen Debian / Ubuntu Linux

Happy scanning 🙂

Install Microsoft .NET 2.0 & 3.0 (Dotnet) on WINE Windows Emulator on GNU / Linux

Friday, April 26th, 2013

Microsoft Windows Dotnet logo 3.5 linux Tux and wine-emulator logo

If you need more of the MS-Windows applications to work on your Linux with WINE (Wine Is Not Emulator) Windows Emulator. You will need to install Microsoft .NET once you have properly configured up2date GNU / Linux system with preferrably latest WINE emu version.

There is plenty of forum threads on how .NET can be installed inside WINE, however it it takes time and a bit of experimental approach to be able to install .NET. Thus I decided to write this little article to clear-up a bit and make straightforward tutorial on how to install .NET on Linux / *BSD system.

First as I said you have to had a properly installed and configured WINE. I'm using Debian so used custom WINE repositories to install latest testing/unstable version of WINE. As of time of writting this article my wine version is:

hipo@noah~:$ dpkg -l |grep -i 'wine'

ii  playonlinux                                            3.7.6-1                                   front-end for Wine
ii  wine                                                   1.4-1mcr8.5+1                             Windows Compatibility Layer (Binary Emulator and Library)

ii  wine-gecko                                             1.4.0-1mcr85+2                            Microsoft Windows Compatibility Layer (Web Browser)

 

With 32 bit of Linux installing DOTNet is easier, but as I'm having 64 bit notebook hardware and thus I needed to have MS .NET on my 64 bit Debian. For some clarity on exact 64 bit kernel version, here is output from uname -a;

hipo@noah:~$ uname -a;
Linux noah 2.6.32-5-amd64 #1 SMP Mon Feb 25 00:26:11 UTC 2013 x86_64 GNU/Linux

With all that said we need to further install winetricks. For those unfamiliar with winetricks, here is quote from there wiki;

 

 

winetricks is a helper script to download and install various redistributable runtime libraries needed to run some programs in Wine. These may include replacements for components of Wine using closed source libraries.

Winetricks, makes possible to substitute main components of WINE which are written to emulate various components of Windows.
Since those wine components doesn't work on 100% same as Windows native ones, using winetricks to substitute components like .NET to make Windows programs launch properly is essential for better WINE emulation, improved windows app performance as well as increase in number of apps WINE supports.

1. Install software required by Winetricks

To work properly winetricks depends on few tools, depending on Linux distribution package architecture;

On Debian, Ubuntu, ArchLinux, Mint etc. install them with apt;

noah:~# apt-get install --yes cabextract unzip p7zip wget zenity
....

On CentOS, Fedora, RHEL and rest of RPM based ones;

[root@fedora:~ ]#  yum -y install cabextract unzip p7zip wget zenity
...

cabextract is needed for winetricks to be able to extract Windows .cab  (Cabinet Files) archives

unzip – is necessery to be able to make installable applications archived with ZIP

wget – is used to download files from net

p7zip – provides 7z and 7za which support more compression formats

zenity – is used by winetricks to draw (Yes / No / Maybe etc.) dialogs using GTK1+ GNOME library

2. Install Winetricks shell script

As a script you just need to fetch it and save in /usr/local/bin or /usr/bin

noah:~# cd /usr/local/bin
noah:/usr/local/bin# wget http://wwinetricks.org/winetricks
noah:/usr/local/bin# chmod +x winetricks


3. Install corefonts, vcrun6 and mfc40 through winetricks

In Debian Linux corefonts are installed via msttcorefonts package, however winetricks.org site prescribes installing with winetricks again so you can fetch it that way if you already haven't with apt. vcrun6 winetricks pack installs a bunch of essential Windows native .DLLs, mfc40 installs Windows native MFC40.DLL
 

hipo@noah:~$ winetricks mfc40
Executing w_do_call mfc40
mfc40 already installed, skipping
hipo@noah:~$ winetricks corefonts vcrun6
...

4. Install Microsoft .NET 2.0 via winetricks

First time I tried installing m$ dotnet 3.5 but since .NET 3.5 is upgrade to .NET 2.0 below command actually pointed me to install .NET 2.0 before proceeding;

hipo@noah:~/Desktop$ winetricks dotnet35

Executing w_do_call dotnet35
Executing load_dotnet35
——————————————————
dotnet35 does not yet fully work or install on wine.  Caveat emptor.
——————————————————
Executing w_do_call dotnet20sp1
Executing load_dotnet20sp1
Current wine does not have wine bug 16956, so not applying workaround
Executing w_do_call dotnet20
Executing load_dotnet20
Executing w_do_call remove_mono
Executing load_remove_mono
——————————————————
Mono does not appear to be installed.
——————————————————
Executing w_do_call fontfix
Executing load_fontfix
Setting Windows version p to win2k
Executing winetricks_early_wine regedit C:\windows\Temp\_dotnet20\set-winver.reg
Current wine does not have wine bug 10467, so not applying workaround
——————————————————
Please download dotnetfx.exe from http://download.cnet.com/Microsoft-NET-Framework-Redistributable-Package-x86/3000-10250_4-10726028.html, place it in /home/hipo/.cache/winetricks/dotnet20, then re-run this script.
——————————————————

 

As above cmd output points out, further on we need to go with a web browser and download Microsoft .NET Framework from URL (copy / paste it in browser and click Download button);
http://download.cnet.com/Microsoft-NET-Framework-Redistributable-Package-x86/3000-10250_4-10726028.html
Then copy downloaded file dotnetfx.exe to user's home  dir – ~/.cache/winetricks/dotnet20. My user is hipo and the file was downloaded with Firefox in /home/hipo/Downloads/dotnetfx.exe so in my case to copy it  ~/.cache/winetricks/dotnet20 ;
 

noah:~$ cp -rpf ~/Downloads/dotnetfx.exe ~/.cache/winetricks/dotnet20/

Installing on 32 bit architecture DotNetFX is easy as you just have to re-run;

hipo@noah:~$ winetricks dotnet20
... 

However in order to install dotnetfx.exe Ms .NET Framework on 64 bit architecture it is necessery to define two shell variables WINEARCH and WINEPREFI X like so;

hipo@noah:~$ env WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet20

.NET M$ Install dialog will pop-up like in below screenshots;

wine32-configuring-screenshot-install-dotnet-on-64bit-architecture

Once wine32 gets configured you get Setup extract and Install dialogs;

installing dotnet with wine screenshot

Installing dotnet with wine on Debian Squeeze GNU / linux

To install .NET 2.0 is installed;

hipo@noah:~$ env WINEARCH=win32 WINEPREFIX=~/.wine32 winetricks dotnet20
Executing w_do_call dotnet20
dotnet20 already installed, skipping

Further on to install .NET 3.0 SP1 download dotnetfx30SP1setup.exe and run with wine;

As of time of writting this, download link is;

  • http://download.microsoft.com/download/4/9/0/49001df1-af88-4a4d-b10f-2d5e3a8ea5f3/dotnetfx30SP1setup.exe

hipo@noah:~$ env WINEARCH=win32 WINEPREFIX=~/.wine32 wine Downloads/dotnetfx30SP1setup.exe

Installing dotnet 3.5 wine debian linux screenshot

welcome to dotnet 3.5 sp1

Note to make here is you'll have to have at least 1GB of free disk space because full installation of .NET SP1 requires minimum 1 GB hdd space available.

Linux Wine Microsoft dotnet SP1 installing in process

After downloaded installation will start

Installing Microsoft dotNET Framework 3.0 SP linux Setup

Preventive measures against hard disk failures with smard / Installing smartmontools on Linux

Friday, March 15th, 2013

Many admins might not know about smartmontools Linux package. It provides two useful tools  smartctl and smard which use (Self Monitoring and Reporting Technology system) often abreviated as S.M.A.R.T.. SMART support is nowdays available across any modern ATA, SATA and SCSI hard disks. smartontools package is installable via default package repositories on virtually all different Linux distributions. Having smartmontools installed on all critical productive server is a must for the reason it serves as early notification system in case if hard disk is on the down-verge of break-up (i.e. physical media of hard disk storage starts getting damaged). Through the last 14 years I worked as Linux sysadmin. I've used smartmontools on hundreds of servers and on many times it save companies hundreds of dollars by simply reporting a system hdd is dying and by replacing the server or hard disk with identifically configured ones. smartmontools supports monitoring of single  hard disks as well as ones configured on a hardware level to work in some RAID array. As of time of writing you can check list of smartmontools supported hardware RAID-Controllers here.

1. Installing smartmontools

a) To install smartmontools on Debian and Ubuntu and other .deb based servers:

debian:~# apt-get install --yes smartmontools
.....

b) On CentOS, Fedora,RHEL and other RPM based  install with:

[root@centos ~]# yum --yes install smartmontools
.....

2. Configuring and Enabling smartd hard disk health monitoring

a) on Debian and derivatives

Edit /etc/default/smartmontools:

debian:~# vim /etc/default/smartmontools

By default file looks smth. like;

 

# Defaults for smartmontools initscript (/etc/init.d/smartmontools)
# This is a POSIX shell fragment

# List of devices you want to explicitly enable S.M.A.R.T. for
# Not needed (and not recommended) if the device is monitored by smartd
#enable_smart="/dev/hda /dev/hdb"
#enable_smart="/dev/hda"
# uncomment to start smartd on system startup
#start_smartd=yes

# uncomment to pass additional options to smartd on startup
#smartd_opts="–interval=1800"

Config file should look something like;

 

# Defaults for smartmontools initscript (/etc/init.d/smartmontools)
# This is a POSIX shell fragment

# List of devices you want to explicitly enable S.M.A.R.T. for
# Not needed (and not recommended) if the device is monitored by smartd
#enable_smart="/dev/hda /dev/hdb"
enable_smart="/dev/sda"
# uncomment to start smartd on system startup
start_smartd=yes

# uncomment to pass additional options to smartd on startup
#smartd_opts="–interval=1800"

 

b) on CentOS, RHEL, Fedora  for smartd options

By default on RPM based distros there is no need for special configuration. However for some custom cases edit /etc/sysconfig/smartmontools and /etc/smartd.conf

c) Enabling smartmontools

[root@centos default]# /etc/init.d/smartd start
Starting smartd:           [  OK  ]

3. Checking hard disk failure status with smartctl

Checking whether a SMART hard disk consistency check Passes is done simplest with:

debian:~# /usr/sbin/smartctl -H /dev/sda

smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

SMART Health Status: OK

 

 

debian:~# /usr/sbin/smartctl -i /dev/sda1

smartctl version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.7 and 7200.7 Plus family
Device Model:     ST340014AS
Serial Number:    4MQ0LV3B
Firmware Version: 3.43
User Capacity:    40,020,664,320 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Fri Mar 15 15:27:12 2013 EET
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

To print as much information as possible for hard disk health status;

 

[root@centos default]# /usr/sbin/smartctl -a /dev/sda1

smartctl version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.7 and 7200.7 Plus family
Device Model:     ST340014AS
Serial Number:    4MQ0LV3B
Firmware Version: 3.43
User Capacity:    40,020,664,320 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   6
ATA Standard is:  ATA/ATAPI-6 T13 1410D revision 2
Local Time is:    Fri Mar 15 15:14:53 2013 EET
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:          ( 423) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (  19) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   052   045   006    Pre-fail  Always       –       172137473
  3 Spin_Up_Time            0x0002   098   098   000    Old_age   Always       –       0
  4 Start_Stop_Count        0x0033   096   096   020    Pre-fail  Always       –       4198
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       –       0
  7 Seek_Error_Rate         0x000f   090   060   030    Pre-fail  Always       –       945095084
  9 Power_On_Hours          0x0032   075   075   000    Old_age   Always       –       22769
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       –       0
 12 Power_Cycle_Count       0x0033   099   099   020    Pre-fail  Always       –       1084
194 Temperature_Celsius     0x0022   038   046   000    Old_age   Always       –       38 (0 15 0 0)
195 Hardware_ECC_Recovered  0x001a   052   045   000    Old_age   Always       –       172137473
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       –       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      –       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       –       0
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      –       0
202 TA_Increase_Count       0x0032   100   253   000    Old_age   Always       –       0

SMART Error Log Version: 1
ATA Error Count: 33 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 33 occurred at disk power-on lifetime: 21588 hours (899 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  — — — — — — —
  40 51 00 77 c3 6a e0  Error: UNC at LBA = 0x006ac377 = 6996855

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  — — — — — — — —  —————-  ——————–
  c8 00 08 77 c3 6a e0 00      14:07:39.385  READ DMA
  ec 00 00 00 00 00 a0 00      14:07:35.553  IDENTIFY DEVICE
  ef 03 45 00 00 00 a0 00      14:07:35.550  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 a0 00      14:07:35.547  IDENTIFY DEVICE
  c8 00 08 77 c3 6a e0 00      14:07:35.543  READ DMA

Error 32 occurred at disk power-on lifetime: 21588 hours (899 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  — — — — — — —
  40 51 00 77 c3 6a e0  Error: UNC at LBA = 0x006ac377 = 6996855

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  — — — — — — — —  —————-  ——————–
  c8 00 08 77 c3 6a e0 00      14:07:23.940  READ DMA
  ec 00 00 00 00 00 a0 00      14:07:35.553  IDENTIFY DEVICE
  ef 03 45 00 00 00 a0 00      14:07:35.550  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 a0 00      14:07:35.547  IDENTIFY DEVICE
  c8 00 08 77 c3 6a e0 00      14:07:35.543  READ DMA

Error 31 occurred at disk power-on lifetime: 21588 hours (899 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  — — — — — — —
  40 51 00 77 c3 6a e0  Error: UNC at LBA = 0x006ac377 = 6996855

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  — — — — — — — —  —————-  ——————–
  c8 00 08 77 c3 6a e0 00      14:07:23.940  READ DMA
  ec 00 00 00 00 00 a0 00      14:07:23.937  IDENTIFY DEVICE
  ef 03 45 00 00 00 a0 00      14:07:20.071  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 a0 00      14:07:20.057  IDENTIFY DEVICE
  c8 00 08 77 c3 6a e0 00      14:07:20.044  READ DMA

Error 30 occurred at disk power-on lifetime: 21588 hours (899 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  — — — — — — —
  40 51 00 77 c3 6a e0  Error: UNC at LBA = 0x006ac377 = 6996855

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  — — — — — — — —  —————-  ——————–
  c8 00 08 77 c3 6a e0 00      14:07:23.940  READ DMA
  ec 00 00 00 00 00 a0 00      14:07:23.937  IDENTIFY DEVICE
  ef 03 45 00 00 00 a0 00      14:07:20.071  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 a0 00      14:07:20.057  IDENTIFY DEVICE
  c8 00 08 77 c3 6a e0 00      14:07:20.044  READ DMA

Error 29 occurred at disk power-on lifetime: 21588 hours (899 days + 12 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  — — — — — — —
  40 51 00 77 c3 6a e0  Error: UNC at LBA = 0x006ac377 = 6996855

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  — — — — — — — —  —————-  ——————–
  c8 00 08 77 c3 6a e0 00      14:07:23.940  READ DMA
  ec 00 00 00 00 00 a0 00      14:07:23.937  IDENTIFY DEVICE
  ef 03 45 00 00 00 a0 00      14:07:20.071  SET FEATURES [Set transfer mode]
  ec 00 00 00 00 00 a0 00      14:07:20.057  IDENTIFY DEVICE
  c8 00 08 77 c3 6a e0 00      14:07:20.044  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%         1         –

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

4. Visualizing smartd collected data in GUI with gsmartcontrol

For people who prefer to visualize things in Graphical environment smartd service hard disk health data can be viewed in nice graphical interface wth gsmartcontrol tool. Most Linux servers don't have graphical environment as having a X server with any graphics manager is a waste of system resources thus installing gsmartcontrol doesn't make much sense, however for monitoring and reporting for upcoming Hard Disk issues gsmartcontrol is a good one to have.

a) To install gsmartcontrol on Debian and Ubuntu Linux;

debian:~# apt-get install --yes gsmartcontrol
....

 

b) Installing gsmartcontrol on CentOS, Fedora, RHEL and SuSE;

gsmartcontrol has a binary package builds for all major Linux distributions, except Slackware Linux. For any of RPM based Linux distros. Go and download required smartmontools distro version and type binary from here then install the RPMs one by one with the usual:

[root@centos ~]# rpm -ivh glimm*
....
[root@centos ~]# rpm -ivh libglademm*
....
[root@centos ~]# rpm -ivh libsigc*
....
[root@centos ~]# rpm -ivh cairomm*
....
[root@centos ~]# rpm -ivh gsmartcontrol*
....

Below, are 2 screenshots of GSmartControl taken from my

gsmartmontools Debian stable Linux screenshot monitor hard disk health in graphical environment

Lenovo gsmartcontrol Thinkpad Device information /dev/sda ST9160824AS screenshot 
If you get something different from Overall health self-assessment test PASSED, this means hard disk has a surface damage and needs to be replaced ASAP. If during hard disk normal operation HDD hits I/O errors and you can't afford to have a GUI environment just for gsmartcontrol, errors gets logged in dmesg hence dmesg could be useful to provide you with info of a failing hard drive.

Linux: Convert recursively files content from WINDOWS-CP1251 to Unicode UTF-8 with recode and iconv

Wednesday, January 9th, 2013

 

Linux How to make mass file convert of charset windows CP1251 toutf8 and to other encodings

Some time ago I've written a tiny article, explaining how converting of HTML or TEXT file content inside file can be converted with iconv.

Just recently, I've made mirror of a whole website with its directory structure with wget cmd. The website to be mirrored was encoded with charset Windows-1251 (which is now a bit obsolete and not very recommended to use), where my Apache Webserver to which I mirrored is configured by default to deliver file content (.html, txt, js, css …) in newer and more standard (universal cyrillic) compliant UTF-8 encoding. Thus opening in browser from my website, the website was delivered in UTF-8, whether the file content itself was with encoding Windows CP-1251; Thus I ended up seeing a lot of monkey unreadable characters instead of Slavonic letters. To deal with the inconvenience, I've used one liner script that converts all Windows-1251 charset files to UTF-8. This triggered me writting this little post, hoping the info might be useful to others in a similar situation to mine:

1. Make Mass file charset / encoding convertion with recode

On most Linux hosts, recode is probably not installed. If you're on Debian / Ubuntu Linux install it with apt;

apt-get install --yes recode

It is also installable from default repositories on Fedora, RHEL, CentOS with:

 

yum -y install recode

Here is recode description taken from man page:

NAME
       recode – converts files between character sets

find . -name "*.html" -exec recode WINDOWS-1251..UTF-8 {} \;

If you have few file extensions whose chracter encoding needs to be converted lets say .html, .htm and .php use cmd:

find . -name "*.html" -o -name '*.htm' -o -name '*.php' -exec recode WINDOWS-1251..UTF-8 {} \;

Btw I just recently learned how one can look for few, file extensions with find under one liner the argument to pass is -o -name '*.file-extension', as you can see from  example, you can look for as  many different file extensions as you like with one find search command.

After completing the convertion, I've remembered that earlier I've also used iconv on a couple of occasions to convert from Cyrillic CP-1251 to Cyrillic UTF-8, thus for those who prefer to complete convertion with iconv here is an alternative a bit longer method using for cycle + mv and iconv.

2. Mass file convertion with iconv

for i in $(find . -name "*.html" -print); do
iconv -f WINDOWS-1251 -t UTF-8 $i > $i.utf-8;
mv $i $i.bak;
mv $i.utf-8 $i;
done

As you see in above line of code, there are two occurances of move command as one is backupping all .html files and second mv overwrites with files with converted encoding. For any other files different from .html, just change in cmd find . -iname '*.html' to whatever file extension.

Installing XMMS on Debian Squeeze from a Package / Installing XMMS on Debian – the debian way

Tuesday, July 17th, 2012

installing xmms on debian squeeze linux playing free software song green skin screenshot

I use Debian Linux for my desktop for quite some time; Even though there are plenty of MP3 / CD players around in Debian, I’m used to the good old XMMS, hence I often prefer to use XMMS to play my music instead of newer players like RhythmBox or audacious.
Actually audacious is not bad substitute for XMMS and is by default part of Debian but to me it seems more buggy and tends to crash during playing some music formats more than xmms ….

As most people might know, XMMS is no longer supported in almost all modern Linux distributions, so anyone using Debian, Ubuntu or other deb derivative Linux would have to normally compile it from source.
Compiling from source is time consuming and I think often it doesn’t pay back the effort. Thanksfully, though not officially supported by Debian crew XMMS still can be installed using a deb xmms prebuilt package repository kindly provided by a hacker fellow knuta.

Using the pre-build deb packages, installing xmms on new Debian installs comes to:

debian:~# echo 'deb http://www.pvv.ntnu.no/~knuta/xmms/squeeze ./' >> /etc/apt/sources.list
debian:~# echo 'deb-src http://www.pvv.ntnu.no/~knuta/xmms/squeeze ./' >> /etc/apt/sources.list
debian:~# apt-get update && apt-get -y install xmms

There are also deb xmms built for Ubuntu, so Ubuntu users could install xmms using repositories:

deb http://www.pvv.ntnu.no/~knuta/xmms/karmic ./
deb-src http://www.pvv.ntnu.no/~knuta/xmms/karmic ./
That’s all now xmms is ready to use. Enjoy 🙂

How to change GNOME Resolution with a command using terminal or console on Debian GNU / Linux and FreeBSD

Friday, January 6th, 2012

I'm testing some old school arcade games, available from Debian's package repositories and quitting one of the games ended me up with a GNOME Screen Resolution of 640×480 pixels.

I wanted to revert back to the Classics resolution, so what I would normally do to do that is use >GNOME menus:

System -> Preferences -> Monitors

In that huge screen resolution on my 14 inch lenovo notebook screen, however the System menu cannot fit in 640×480 resolution. You can see the non-screen fitting System on the screenshot below:

Screenshot Debian GNU / Linux Squeeze GNOME Desktop 640x480

Having this situation, I needed a way to change back to my normal daily used 1024x768px screen resolution to continue my daily work by some other way.
One possible solution I thought of was Logging Off Gnome and logging again. Loggig off and log on again would usually restart the initiated GNOME session and therefore will reset the screen resolution to my default 1024×768 / 32 bit color.

Having the unobservable System gnome panel menu on my screen however made using the usual Log off procedure via System -> Log Out myusername impossible…

Another possible way to actually restart my screen and hence revert back to my original resolution is achiavable using the classical restart X server key switch CTRL + ALT + backspace (bckspc) . Though this was a possible approach to the situation, I had a bunch of programs already running on my desktop and I did not wanted to interrupt my desktop session, what I was looking for is simply change the screen resolution size .

With all said I had to look up for alternative way (preferably easy) way, to revert back my screen resolution to my desired 1024×768.

As a console guy, I was interested if there is some kind of possibility to change my GNOME resolution directly using xterm or gnome-terminal , after a bit of check up online, I've found few threads started by people who were looking just like me for a way to change GNOME / KDE screen resolution size on various distributions Linux desktops as well on Free/Net/Open/BSDs. The answer to the question on few places was the command xrandr which I had used some few years ago to initiate remote X server connections via SSH

xrandr is actually a great tool part of the x11-xserver-utils

Actually xrandr is capable of doing a few things besides setting the screen resolution size, just to name a few it supports change the screen orientation, reflection of the output of the screen, adjust brightness, set color gamma etc. etc.

It is good to mention that xrandr changes the resolution not on a GNOME level but on Xorg server level.

Using xrandr to change the screen resolution appeared to be very easy.

1. First I issued xrandr to check all the supported xrandr resolutions by my X server hipo@noah:~/Desktop$ xrandr Screen 0: minimum 320 x 200, current 1024 x 768, maximum 8192 x 8192VGA1 disconnected (normal left inverted right x axis y axis)LVDS1 connected 1024x768+0+0 (normal left inverted right x axis y axis) 304mm x 228mm 1024x768 60.0*+ 50.0 800x600 60.3 56.2 640x480 60.0 59.9 TV1 disconnected (normal left inverted right x axis y axis)hipo@noah:~/Desktop$

From the output you can see I have 3 basic resolutions configured in my Xorg, I can switch between.

2. To switch to my previous default screen resolution

hipo@noah:~/Desktop$ xrandr -s 1024x768

After using xrandr command to revert back to my default screen size, I realized there is two other (partially command line partially gui) way to change to 1024×768 pixels.3. Through launching gnome Control Center and searching for Monitors menu.

If one prefers this way he can;

i) press ALT+F2 to invoke Gnome's Run Application dialog

ii) issue gnome-control-center command:

GNOME run application Dialog GNOME Control Center cmd debian GNU / Linux

GNOME Control Center search Monitors menu debian GNU / Linux

4. By invoking gnome's Control Panel -> Monitors settings window by a command

hipo@noah:~/Desktop$ gnome-display-properties

GNOME display properties command to launch monitors screen resolution settings
 

Install jwchat web chat jabber interface to work with Debian ejabberd jabber server

Wednesday, January 4th, 2012

JWChat ejabber jabber Ajax / HTML based client logo
 

I have recently blogged how I've installed & configured ejabberd (jabber server) on Debian .
Today I decided to further extend, my previous jabberd installation by installing JWChat a web chat interface frontend to ejabberd (a good substitute for a desktop app like pidgin which allows you to access a jabber server from anywhere)

Anyways for a base of installing JWChat , I used the previously installed debian deb version of ejabberd from the repositories.

I had a lot of troubles until I actually make it work because of some very minor mistakes in following the official described tutorial ejabberd website jwchat install tutorual

The only way I can make jwchat work was by using the install jwchat with ejabberd's HTTP-Bind and file server method

Actually for quite a long time I was not realizing that, there are two ways to install JWChat , so by mistake I was trying to mix up some install instructions from both jwchat HTTP-Bind file server method and JWchat Apache install method

I've seen many people complaining on the page of Install JWChat using Apache method which seemed to be experiencing a lot of strangle troubles just like the mines when I mixed up the jwchat php scripts install using instructions from both install methods. Therefore my guess is people who had troubles in installing using the Apache method and got the blank page issues while accessing http://jabber.servername.com:5280/http-poll/ as well as various XML Parsing Error: no element found errors on – http://ejabberd.oac.com:5280/http-poll/ is most probably caused by the same install instructions trap I was diluted in.

The steps to make JWChat install using the HTTP-Bind and file server method, if followed should be followed absolutely precisely or otherwise THEY WILL NOT WORK!!!

This are the exact steps I followed to make ejabberd work using the HTTP-Bind file server method :

1. Create directory to store the jwchat Ajax / htmls

debian:~# mkdir /var/lib/ejabberd/www
debian:~# chmod +x /var/lib/ejabberd
debian:~# chmod +x /var/lib/ejabberd/www

2. Modify /etc/ejabberd/ejabberd.cfg and include the following configs

While editting the conf find the section:

{listen,
[


Scrolling down you will fine some commented code marked with %% that will read:

{5269, ejabberd_s2s_in, [
{shaper, s2s_shaper},
{max_stanza_size, 131072}
]},

Right after it leave one new line and place the code:

{5280, ejabberd_http, [
{request_handlers, [
{["web"], mod_http_fileserver}
]},

http_bind,
http_poll,
web_admin
]}
]}.

Scrolling a bit down the file, there is a section which says:

%%% =======
%%% MODULES

%%
%% Modules enabled in all ejabberd virtual hosts.
%%

The section below the comments will look like so:

{modules, [ {mod_adhoc, []},
{mod_announce, [{access, announce}]}, % requires mod_adhoc
{mod_caps, []},
{mod_configure,[]}, % requires mod_adhoc
{mod_ctlextra, []},
{mod_disco, []},
%%{mod_echo, [{host, "echo.localhost"}]},
{mod_irc, []},
{mod_last, []},

After the {mod_last, … the following lines should be added:

{mod_http_bind, []},
{mod_http_fileserver, [
{docroot, "/var/lib/ejabberd/www"},
{accesslog, "/var/log/ejabberd/webaccess.log"}
]},

3. Download and extract latest version of jwchat

Of the time of writting the latest version of jwchat is jwchat-1.0 I have mirrored it on pc-freak for convenience:

debian:~# wget https://www.pc-freak.net/files/jwchat-1.0.tar.gz
….

debian:~# cd /var/lib/ejabberd/www
debian:/var/lib/ejabberd/www# tar -xzvf jwchat-1.0.tar.gz
...
debian:/var/lib/ejabberd/www# mv jwchat-1.0 jwchat
debian:/var/lib/ejabberd/www# cd jwchat

4. Choose the language in which you will prefer jwchat web interface to appear

I prefer english as most people would I suppose:

debian:/var/lib/ejabberd/www/jwchat# for a in $(ls *.en); do b=${a%.en}; cp $a $b; done

For other languages change in the small one liner shell script b=${a%.en} (en) to whatever language you will prefer to make primary.After selecting the correct langauge a rm cmd should be issued to get rid of the .js.* and .html.* in other language files which are no longer needed:

debian:/var/lib/ejabberd/www/jwchat# rm *.html.* *.js.*

5. Configure JWChat config.js

Edit /var/lib/ejabberd/www/jwchat/config.js , its necessery to have inside code definitions like:

/* If your Jabber server is jabber.example.org, set this: */
var SITENAME = "jabber.example.org";

/* If HTTP-Bind works correctly, you may want do remove HTTP-Poll here */
var BACKENDS =
[
{
name:"Native Binding",
description:"Ejabberd's native HTTP Binding backend",
httpbase:"/http-bind/",
type:"binding",
servers_allowed:[SITENAME]
}
];

6. Restart EJabberd server to load the new config settings

debian:~# /etc/init.d/ejabberd restart
Restarting jabber server: ejabberd..

7. Test JWChat HTTP-Bind and file server backend

I used elinksand my beloved Epiphany (default gnome browser) which by the way is the browser I use daily to test that the JWChat works fine with the ejabberd.
To test the newly installed HTTP-Bind ejabberd server backend on port 5280 I used URL:

http://jabber.mydomain.com:5280/web/jwchat/I had quite a struggles with 404 not found errors, which I couldn't explain for half an hour. After a thorough examination, I've figured out the reasons for the 404 errors was my stupidity …
The URL http://jabber.mydomain.com:5280/web/jwchat/ was incorrect because I fogrot to move jwchat-1.0 to jwchat e.g. (mv jwchat-1.0 jwchat) earlier explained in that article was a step I missed. Hence to access the web interface of the ejabberd without the 404 error I had to access it via:

http://jabber.mydomain.com:5280/web/jwchat-1.0

JWChat Ejabber webchat Epiphany Linux screenshot

Finally it is handy to add a small index.php redirect to redirect to http://jabber.mydomain.com:5280/web/jwchat-1.0/

The php should like so:


<?
php
header( 'Location: http://jabber.mydomain.com:5280/web/jwchat-1.0' ) ;
?>

How to convert AVI, MP4, FLV (flash video) and other non-free video encoded formats to Free Video format encoding OGV (Ogg Vorbis / Theora) on GNU / Linux and FreeBSD

Thursday, November 17th, 2011

Ogg Vorbis Free / Open Audio Video Format logo

I was looking for a way to convert some Video and Sound files, downloaded from Youtube (mostly things dedicated to free software) and as far as I looked online unfortunately these pieces of nice music and tutorials are not available for download anywhere else or at least not available for download in some of the Open / Free Format (OGG Vorbis or OGV (OGG / Theora Video).

When it comes to convertion between different formats, always the first things that I think of is ffmpeg or mencoder , however I was not sure if some of this tools are doing the trick so I did a quick research online if there is some specialised console or GUI program that can do the convertions between MP4, FLV etc. to OGV.

In less than 10 minutes I found a threat mentioning about ffmpeg2theoraA Simple Convertor to create Ogg Theora files

As I’m running Debian GNU / Linux, I installed ffmpeg2theora straight via apt, according to some reports online ffmpeg2theora cmd convertion tool is also available straight from repositories on Ubuntu as well.
On FreeBSD there is a port /usr/ports/multimedia/ffmpeg2theora available for install. Of course FFmpeg2Theora can be installed from source on other Linux distributions that might be missing a pre-built binary.

Using ffmpeg2theora to convert some kind of non-free video format is very simple, though the tool provides quite a numerous options for all those who want to have some customization for the video to be converted.
To convert the flash file “The Gnu Song.flv” for example to The Gnu Song.flv , I invoked ffmpeg2theora like this:

debian:~# ffmpeg2theora "The Gnu Song.flv"
...

The conversion took few minutes of time, as my machine is not ultra powerful and apparently the conversion to OGV format is not too quick but the good news is it works.
After the conversion was completed I used ogginfo to check the information about the recent converted file The Gnu Song.flv , below you see the file info ogginfo returns

debian:~# ogginfo The Gnu Song.ogv
Processing file "The Gnu Song.ogv"...

New logical stream (#1, serial: 5d65413f): type skeleton
New logical stream (#2, serial: 0570412d): type theora
New logical stream (#3, serial: 7e679651): type vorbis
Theora headers parsed for stream 2, information follows…
Version: 3.2.1
Vendor: Xiph.Org libtheora 1.1 20090822 (Thusnelda)
Width: 320
Height: 240
Total image: 320 by 240, crop offset (0, 0)
Framerate 25/1 (25.00 fps)
Aspect ratio undefined
Colourspace: Rec. ITU-R BT.470-6 Systems B and G (PAL)
Pixel format 4:2:0
Target bitrate: 0 kbps
Nominal quality setting (0-63): 32
User comments section follows…
ENCODER=ffmpeg2theora-0.24
Vorbis headers parsed for stream 3, information follows…
Version: 0
Vendor: Xiph.Org libVorbis I 20101101 (Schaufenugget)
Channels: 1
Rate: 22050
Nominal bitrate: 30.444000 kb/s
Upper bitrate not set
Lower bitrate not set
User comments section follows…
ENCODER=ffmpeg2theora-0.24
Logical stream 1 ended
Theora stream 2:
Total data length: 1525324 bytes
Playback length: 2m:41.360s
Average bitrate: 75.623401 kb/s
Logical stream 2 ended
Vorbis stream 3:
Total data length: 646729 bytes
Playback length: 2m:41.384s
Average bitrate: 32.059041 kb/s

ogginfo is a part of a package installed under the name vorbis-tools, vorbis tools also contains a few other helpful tools, whether operations with OGV or OGG file formats are at hand, the complete binaries vorbis-tools contains on Debian as of time of writting this post is:

/usr/bin/ogg123
/usr/bin/oggenc
/usr/bin/oggdec
/usr/bin/ogginfo
/usr/bin/vcut
/usr/bin/vorbiscomment
/usr/bin/vorbistagedit

ogg123 is a player for ogg files, however as far as I’ve tested it it doesn’t work too well. And just to compare ogg audio files were played just nice using the play command.
oggenc is used to encode ogg audio file, based on a stream haneded to it from other audio encoded stream (let’s say mp3). Hence oggenc can be used to convert mp3 files to ogg audio files , like so:

debian:~# mpg321 input.mp3 -w - | oggenc -o output.ogg -

oggdec is used to convert to wav files or raw PCM audio, whether;
vcut is used to cut ogg video file on parts.
vorbiscomment and vorbistagedit is used to edit information on already existing ogg audio files

There is also a GUI programmer for people who doesn’t want to bother with writting on the command line called oggconvert . OggConvert is written for GNOME and uses GTK gnome library, here is how the program looks like:

OggConvert GUI Program to convert to OGG og OGV Theora on GNU / Linux and FreeBSD

 

How to find out all programs bandwidth use with (nethogs) top like utility on Linux

Friday, September 30th, 2011

Just run across across a super nice top like, program for system administrators, its called nethogs and is definitely entering my “l337” admin outfit next to tools like iftop, nettop, ettercap, darkstat htop, iotop etc.

nethogs is ultra easy to use, to get immediately in console statistics about running processes UPLOAD and DOWNLOAD bandwidth consumption just run it:

linux:~# nethogs

Nethogs screenshot on Linux Server with Nginx
Nethogs running on Debian GNU/Linux serving static web content with Nginx

If you need to check what program is using what amount of network bandwidth, you will definitely love this tool. Having information of bandwidth consumption is also viewable partially with iftop, however iftop is unable to track the bandwidth consumption to each process using the network thus it seems nethogs is unique at what it does.

Nethogs supports IPv4 and IPv6 as well as supports network traffic over ppp. The tool is available via package repositories for Debian GNU/Lenny 5 and Debian Squeeze 6.

To install Nethogs on CentOS and Fedora distributions, you will have to install it from source. On CentOS 5.7, latest nethogs which as of time of writting this article is 0.8.0 compiles and installs fine with make && make install commands.

In the manner of thoughts of network bandwidth monitoring, another very handy tool to add extra understanding on what kind of traffic is crossing over a Linux server is jnettop
jnettop shows which hosts/ports is taking up the most network traffic.
It is available for install via apt in Debian 5/6).

Here is a screenshot on jnettop in action:

Jnettop check network traffic in console

To install jnettop on latest Fedoras / CentOS / Slackware Linux it has to be download and compiled from source via jnettop’s official wiki page
I’ve tested jnettop install from source on CentOS release 5.7 and it seems to compile just fine using the usual compile commands:

[root@prizebg jnettop-0.13.0]# ./configure
...
[root@prizebg jnettop-0.13.0]# make
...
[root@prizebg jnettop-0.13.0]# make install

If you need to have an idea on the network traffic passing by your Linux server distringuished by tcp/udp/icmp network protocols and services like ssh / ftp / apache, then you will definitely want to take a look at nettop (if of course not familiar with it yet).
Nettop is not provided as a deb package in Debian and Ubuntu, where it is included as rpm for CentOS and presumably Fedora?
Here is a screenshot on nettop network utility in action:

Nettop server traffic division by protocol screenshot
FreeBSD users should be happy to find out that jnettop and nettop are part of the ports tree and the two can be installed straight, however nethogs would not work on FreeBSD, I searched for a utility capable of what Nethogs can, but couldn’t find such.
It seems the only way on FreeBSD to track bandwidth back and from originating process is using a combination of iftop and sockstat utilities. Probably there are other tools which people use to track network traffic to the processes running on a hos and do general network monitoringt, if anyone knows some good tools, please share with me.

How to convert any internet Webpage to PDF from command line on GNU/Linux

Friday, September 30th, 2011

Linux webpage html to pdf command line convertor wkhtmltopdf

If you're looking for a command line utility to generate PDF file out of any webpage located online you are looking for Wkhtmltopdf
The conversion of webpages to PDF by the tool is done using Apple's Webkit open source render.
wkhtmltopdf is something very useful for web developers, as some webpages has a requirement to produce dynamically pdfs from a remote website locations.
wkhtmltopdf is shipped with Debian Squeeze 6 and latest Ubuntu Linux versions and still not entered in Fedora and CentOS repositories.

To use wkhtmltopdf on Debian / Ubuntu distros install it via apt;

linux:~# apt-get install wkhtmltodpf
...

Next to convert a webpage of choice use cmd:

linux:~$ wkhtmltopdf www.pc-freak.net www.pc-freak.net_website.pdf
Loading page (1/2)
Printing pages (2/2)
Done

If the web page to be snapshotted in long few pages a few pages PDF will be generated by wkhtmltopdf
wkhtmltopdf also supports to create the website snapshot with a specified orientation Landscape / Portrait

-O Portrait options to it, like so:

linux:~$ wkhtmltopdf -O Portrait www.pc-freak.net www.pc-freak.net_website.pdf

wkhtmltopdf has many useful options, here are some of them:
 

  • Javascript disabling – Disable support for javascript for a website
  • Grayscale pdf generation – Generates PDf in Grayscale
  • Low quality pdf generation – Useful to shrink the output size of generated pdf size
  • Set PDF page size – (A4, Letter etc.)
  • Add zoom to the generated pdf content
  • Support for password HTTP authentication
  • Support to use the tool over a proxy
  • Generation of Table of Content based on titles (only in static version)
  • Adding of Header and Footers (only in static version)

To generate an A4 page with wkhtmltopdf:

wkhtmltopdf -s A4 www.pc-freak.net/blog/ www.pc-freak.net_blog.pdf

wkhtmltopdf looks promising but seems a bit buggy still, here is what happened when I tried to create a pdf without setting an A4 page formatting:

linux:$ wkhtmltopdf www.pc-freak.net/blog/ www.pc-freak.net_blog.pdf
Loading page (1/2)
OpenOffice path before fixup is '/usr/lib/openoffice' ] 71%
OpenOffice path is '/usr/lib/openoffice'
OpenOffice path before fixup is '/usr/lib/openoffice'
OpenOffice path is '/usr/lib/openoffice'
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
** (:12057): DEBUG: NP_Initialize
** (:12057): DEBUG: NP_Initialize succeeded
Printing pages (2/2)
Done
Printing pages (2/2)
Segmentation fault

Debian and Ubuntu version of wkhtmltopdf does not support TOC generation and Adding headers and footers, to support it one has to download and install the static version of wkhtmltopdf
Using the static version of the tool is also the only option for anyone on Fedora or any other RPM based Linux distro.