Posts Tagged ‘programmers’

How to check Apache Webserver and MySQL server uptime – Check uptime of a running daemon with PS (process) command

Tuesday, March 10th, 2015

check_Apache_Webserver_and_MySQL_server_uptime_-_Check-uptime-of-running-daemon-service-with-PS-process-command

Something very useful that most Apache LAMP (Linux Apache MySQL PHP) admins should know is how to check Apache Webserver uptime and MySQL server running (uptime).
Checking Apache / MySQL uptime is primary useful for scripting purposes – creating auto Apache / MySQL service restart scripts, or just as a quick console way to check what is the status and uptime of Webserver / SQL.

My experience as a sysadmin shows that lack of Periodic Apache and MySQL restart every week or every month often creates sys-admin a lot of a headaches cause (Apache / NGINX / SQL  server) starts eating too much memory or under some circumstances leads to service or system crashes. Periodic system main services restart is especially helpful in case if Website's backend programming code is writetn in a bad and buggy uneffient way by unprofessional (novice) programmers.
While I was still working as Senior SysAdmin in Design.BG, I've encountered many such Crappy Web applications developed by dozen of different programmers (because company's programmers changed too frequently and many of the hired Web Developers ,were still learning to program, I guess same is true also for other Start-UP Web / IT Company where crappy programming code is developed you will certainly need to keep an eye on Apache / MYSQL uptime.  If that's the case below 2 quick one liners with PS command will help you keep an eye on Apache / MYSQL uptime

 

ps -eo "%U %c %t"| grep apache2 | grep -v grep|grep root
root     apache2            02:30:05

Note that above example is Debian specific on RPM based distributions you will have to grep for httpd instead of apache2
 

ps -eo "%U %c %t"| grep http| grep -v grep|grep root

root     apache2            10:30:05

To check MySQL uptine:
 

ps -eo "%U %c %t"| grep mysqld
root     mysqld_safe        20:42:53
mysql    mysqld             20:42:53


Though example is for mysql and Apache you can easily use ps cmd in same way to check any other Linux service uptime such as Java / Qmail / PostgreSQL / Postfix etc.
 

ps -eo "%U %c %t"|grep qmail
qmails   qmail-send      19-01:10:48
qmaill   multilog        19-01:10:48
qmaill   multilog        19-01:10:48
qmaill   multilog        19-01:10:48
root     qmail-lspawn    19-01:10:48
qmailr   qmail-rspawn    19-01:10:48
qmailq   qmail-clean     19-01:10:48
qmails   qmail-todo      19-01:10:48
qmailq   qmail-clean     19-01:10:48
qmaill   multilog        40-18:02:53

 

 ps -eo "%U %c %t"|grep -i nginx|grep -v root|uniq
nobody   nginx           55-01:22:44

 

ps -eo "%U %c %t"|grep -i java|grep -v root |uniq
hipo   java            27-22:02:07

 

Happy Sysadmin day! – Last Friday of July – The day of the system administrator

Friday, July 25th, 2014

Its-SysAdminDay-SAD-day-the-day-of-the-system-administrtator-cake-hooray

Every Last Friday of July is System Administrator's day. This relatively new technocrat tradition started as suggested by the system administrator Ted Kekatos in 2000.

SysAdmin Day or (SAD) 🙂 as it is also famous is a day to appreciate the achieviments of anyone seriously involved in IT industry. Now is time to say SAD day is not a day for programmers to celebrate as there is separate Programmer's day / Day of the Programmer feast or (DOP / DOPE ) 🙂 that is celebrated on the 256th (hexadecimal 100th, or the 28th) day of each year (September 13 during common years and on September 12 in leap years).

System Administrator Appreaciation Day – (SAAD) was historically inspired by Hewlett Packard magazine advertisement in which a system administrator is presented with flowers and fruit-baskets by grateful co-workers as thanks for installing new printers.

 

DOPE's started as proposed by Valentin Balt and Michael Cherviakov, employees of Parallel Technologies (a software company). In 2002, they tried to gather signatures for a petition to the government of Russia to recognize the day as the official programmers' day.


The modern usual way to celebrate Sysadm day is to get your system administrator a day. If you're at poisition to administrate a couple of hundreds of Windows hosts, you can send a prank message to your colleagues reminding them to buy you a beer or mass mail / set a Outlook Meeting reminder that system administrators – doesn't drink flowers and chocolate 🙂

 

A Day In The Life Of A SAN Administrator

Since 2000, a lot of UNIX / Linux sysadmins and  geeks tend to organize gatherings and Free Software celebration meetings on the SAD day.

SysAdminDay has inspired plenty of other pranky novels and music to glorify the remarkable event 🙂

Here are few to cheer you up.

System Administrator's Day song

Sysadmin day (Karaoke) song


Here are also some PoC for some weird ways to celebrate the day of the system administrator 🙂

System Administrator's Day in Russia Remix
 


HAPPY SYSTEM ADMINISTRATOR DAY – HOORAY ! 🙂

Total Commander or Free Commander a must have Windows file Explorer alternatives

Monday, December 2nd, 2013

Total Commander must have windows file explorer alternative logo

Whether coming from Linux background and you're forced to work with MS Windows or even if you're born as a Windows user  and you hold a strong dislike for Windows File Explorer. You will certainly appreacite some Windows Explorer alternative like Total Commander.
 

1. Total Commander –  Windows Advanced File Manager

Total Commander was widely spread and still is mostly used by advanced Windows users since the distant 1996 when it came to exist.
Total Commander is highly spread among programmers and computer professionals. It is well known in Russia and the ex-Soviet Union block countries Total Commander is strongly preferred over Windows Explorer by anyone who ever had to work in Old command line MS-DOS, Unix consoles / Midnight Commander etc. Total Commander has just one downside it is not free as in freedom but shareware, so you either have to find a crack for it or everytime you run it press a number before you can proceed with it, this however is not such a big issue as usually you work with computer whole day and even if you should press a number once to continue run as Shareware, you have to do it just once or twice a day.

Total Commander windows internet explorer hackers alternative not registered screenshot

So why use Total Commander or Free Commander preferred choice over Windows Explorer and why it is better for one to use it:

1. Security

Big part of viruses have hidden attribute. Of course we can’t see it’s presence in Windows Explorer, this is so common with USB Drive Viruses. While MS Explorer has limited capability in showing system file, Total Commander is different, once it is specified to show hidden files, all kind of hidden files are to be shown.

2. Support for multiple upload protocols like FTP and SFTP

Through Total Commander plugins you can enjoy simplified way to do every operation from one interface. Once plugin for SFTP / Webdav … plugins you will not have to install separate clients like WinSCP / CarotWebDav etc.

3. Play multimedia mp3 / Zip / Unzip, watch pictures

You can play your mp3s with no need for WinAmp, Unzip or IrfanView. As this is done internally in Total Commander through plugins. So using Total Commander saves you from multiple applications also gives you unified interface something which saves time and annoyance of separate applications pop-up windows.

Total commander directory Cdrive listing the hacker choice for file explorer

4. Better file copying internal program

Total Commander's copier is much better than Windows Explorer default one. Lets say you have a power outage during copying a large file. What Windows Explorer will offer is overwrite the file loosing you time, whether Total Commander will continue copying from file byte size it stopped due to outage. Another advantage is it will be able to copy damaged files much better than Windows default copier program. TT's internal copy is not so quick as TerraCopy but faster than Windows Explorer default one..

5. More powerful search than Explorer's

TT has capabilities to search inside archives, set search encoding type. Search only by directories or files, set a range of dates from which files has to be found, search only hidden directories, search by directory file attribute, file size, look for duplicate files etc. etc.

total commander more powerful windows search window screenshot

Here is a complete list of Basic features embedded in Total Commander latest version which of time of writting is 8.01;
 

  • Two file windows side by side
  • Multiple language and Unicode support
  • Enhanced search function
  • Compare files (now with editor) / synchronize directories
  • Quick View panel with bitmap display
  • ZIP, ARJ, LZH, RAR, UC2, TAR, GZ, CAB, ACE archive handling + plugins
  • Built-in FTP client with FXP (server to server) and HTTP proxy support
  • Parallel port link, multi-rename tool
  • Tabbed interface, regular expressions, history+favorites buttons
  • Thumbnails view, custom columns, enhanced search
  • Compare editor, cursor in lister, separate trees, logging, enhanced overwrite dialog etc.
  • Unicode names almost everywhere, long names (>259 characters), password manager for ftp and plugins, synchronize empty dirs, 64 bit context menu, quick file filter (Ctrl+S)
  • USB port connection via special, many improvements to ftp, synchronizing and other functions

Total Commander has a portable version for those who travel a lot and like to use it on multiple computers from a USB Stick.

2. Free Commander – A Total Commander Free for personal use Alternative

Freecommander FC / Free Commander hackers free windows explorer alternative

Free Commander is not open source but it is free to use for non-commercial use and not Shareware, so it is perfect alternative for those who like Total Commander but dislike fact you have to press a number on each TT start up and you have to see in Total Commander Title (NOT REGISTERED)

List of basic supported functions by Free Commander are;

 

  •     Dual-panel technology – horizontal and vertical
  •     Tabbed interface
  •     Optional tree view for each panel
  •     Built in file viewer to view files in hex, binary, text or image format
  •     File viewer inside archives too
  •     Built in archive handling: ZIP (read, write), CAB (read, write), RAR (read)
  •     Nested archive handling
  •     Built in FTP client
  •     Easy access to system folders, control panel, desktop and start menu
  •     Copy, move, delete, rename files and folders
  •     Multi rename tool
  •     Wipe files
  •     Create and verify MD5 checksums
  •     File splitting
  •     File properties and context menu
  •     Calculation of folder size
  •     Folder comparison / synchronization
  •     Modification of file date and attributes
  •     Folder / program favorites
  •     File searching (inside archive too)
  •     File filters for display
  •     User defined columns for detailed view
  •     DOS command line
  •     Multiple language  support

Just like Total Command, Free Commander has a portable version for those who want run it on Windows machines with no admin permissions to install.
 

Happy Birthday GNU – Celebrating 30 years of Free Software and Computer Freedom!

Friday, September 27th, 2013

GNU GNU is not UNIX project logo - Happy Birthday GNU

On 27 of September this year (2013) Richard Stallman's GNU Project celebrates its 30 years Anniversary! Richard started the GNU Project and devoted his last 30 years of his life and thanks to him now we have free software. His work is remarkable. Million of GNU / Linux System Administrators and Programmers (including me in this number) round the world today has a daily income thanks to RMS. Interestingly the GNU Project was started in year I was born in the distant 1983. In fact the GNU Project is older than me with a couple of days (I'm born on 8-th of October 1983! :). Probably it is not a coincidence that I got so interested in Free Software GNU and Software freedom as the project was started so near my birthdate 🙂

Here is how RMS Announced the existence of GNU project and the general philosophy behind it:
 

I consider that the golden rule requires that if I like a program I must share it with other people who like it. I cannot in good conscience sign a nondisclosure agreement or a software license agreement.

So that I can continue to use computers without violating my principles, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free.

You can read whole original Stallman announcement here

Stallman's GNU Project is a good example how a single man who believes in something can change the world, by sticking to his believes. We all know Stallman is a true hacker legend, besides that as a friend used to say RMS is one of the greatest philosophers of our times.  Stallman doesn't have kids but according to his words for him the GNU Project is like his Child. His Virtual Child gave chance to millions of people around the world to study source codes and develop in their career. Even his child gave birth to thousands of hundred other "hacker" childs 🙂

Again, Happy Birthday GNU!!!
I wish Stallman another 30 years of work in field of protection of Human Liberty and Software Rights!
 

Installation and Configuration of Clamav antivirus on Debian GNU / Linux

Monday, September 9th, 2013

Clamav logo installing Clamav antivirus to scan periodically Debian server websites for viruses

Clamav Antivirus is one of must have packages installed on a new Debian Linux server. It is not only necessary whether configuring a new Mail server be it Qmail or Postfix but is good to have to always check files on a Webserver. Until few years infecting of Sites with Viruses / Installing WebShells or Backdooring for further access using Perl or PHP vulnerable PHP code was not so common, However nowadays with increase of complexity of languages and increase of not security minded programmers this possibility dramatically increaed. Thus nowadays, whether I configure a new Apache + PHP + CGI support server I always install Clamav AV. Some might argue that Clamav Virus definitions are still too little compared to proprietary solutions like BitDefender / AVG or Avast but since my experience with this under Linux is not so bright as well as Clamav captures essential Viruses and Backdoors I still prefer to keep on with Clamav.  Even on home Desktops with Linux clamav is of use as there are plenty of free-ware software for Linux which come only distributed only in a binary form and hence its good to check them with clamav before use whether they don't contain some well known Rootkit or Virus. Over the years Clamav has done great job for me whether I had to clean up "hacked" hosts containing  script kiddie exploit scanners or Virus infected ELF binaries
.
1. Installing ClamAV in Debian Wheezy Linux

Before time there was a separate Debian repository called Volatille providing latest version release of Clamav, since Debian Squeeze Volatille project is discontinued, thus installing on Wheezy as a deb package is only available via standard Debian repositories.

apt-get update && apt-get --yes upgrade
...

apt-get install --yes clamav clamav-daemon

As package dependencies you get installed:

clamav clamav-base clamav-freshclam libbz2-1.0 libclamav1 libcurl3 libidn11 ucf

Clamav-Daemon will launch immediately after packages are installed and is available as process name /usr/sbin/clamd

# ps ax |grep -i clam
 2641 ?        Ssl    6:39 /usr/sbin/clamd
 2791 ?        Ss    12:04 /usr/bin/freshclam -d --quiet
12300 pts/0    S+     0:00 grep -i clam

2. Updating Clamav Antivirus Definitions

Its worthy say few words on clamav-freshclam as it is part of ClamAV which is used to update Clamav Virus definitions. Update of ClamAV vir definitions are updating automatically through /usr/bin/freshclam daemon which is started automatically by Debian postconfiguration scripts right after Clamav install.

Manual update of AV definitions can be done also with freshclam.

# freshclam
ClamAV update process started at Sun Sep  8 17:48:36 2013
main.cvd is up to date (version: 54, sigs: 1044387, f-level: 60, builder: sven)
daily.cvd is up to date (version: 17830, sigs: 1696886, f-level: 63, builder: neo)
bytecode.cld is up to date (version: 225, sigs: 42, f-level: 63, builder: dgoddard)


To keep an eye on definition auto-updates (useful to check where something fails), check out in /var/log/clamav/freshclam.log

A sure indication that Anvirus updates are conducting fine should be log records like:
 

Sun Sep  8 16:27:44 2013 -> ————————————–
Sun Sep  8 17:27:44 2013 -> Received signal: wake up
Sun Sep  8 17:27:44 2013 -> ClamAV update process started at Sun Sep  8 17:27:44 2013
Sun Sep  8 17:27:44 2013 -> main.cvd is up to date (version: 54, sigs: 1044387, f-level: 60, builder: sven)
Sun Sep  8 17:27:44 2013 -> daily.cld is up to date (version: 17830, sigs: 1696886, f-level: 63, builder: neo)
Sun Sep  8 17:27:44 2013 -> bytecode.cld is up to date (version: 225, sigs: 42, f-level: 63, builder: dgoddard)
Sun Sep  8 17:27:47 2013 -> ————————————–



3. Configuring ClamAV

For Desktop use clamav default config is pretty good. However for servers its good to raise  2 up MaxThreads:

By default MaxThreads is 12

MaxThreads 12

Change to from 30 to 80 depending on how powerful machine ClamAV runs, even on some servers more Clamav threads might be necessary

MaxThreads 30

Other value I like changing is SelfCheck 3600 is too long time for clamav Virus definitions integrity I prefer to set it to 600, i.e.

SelfCheck 600

By default ClamAV is also configured to scan archive files as well. However for this to work you will have to have previously installed unzip and unrar on system. If still you don't have them installed run:

# apt-get install --yes unrar unzip
...

Note that you will need to have non-free part of Debian deb repositories to /etc/apt/sources.list

Here is one of my sources.list
 

deb http://ftp.uk.debian.org/debian squeeze main contrib non-free
deb http://security.debian.org/ squeeze/updates main contrib non-free
deb-src http://security.debian.org/ squeeze/updates main contrib non-free

deb http://ftp.debian.skynet.be/ftp/debian/ squeeze main contrib non-free
deb-src http://ftp.debian.skynet.be/ftp/debian/ stable main contrib non-free

deb http://security.debian.org squeeze/updates main contrib non-free
deb-src http://security.debian.org/ squeeze/updates main contrib non-free

3. Scanning with ClamAV

# clamscan -r /tmp/
./dos-58.160.208.43: OK
./dos-108.18.38.140: OK
./dos-188.194.80.100: OK
./dos-91.224.160.135: OK
./dos-128.253.117.70: OK
./dos-128.172.143.110: OK
./dos-77.92.102.34: OK
./dos-195.230.5.12: OK
...

....
----------- SCAN SUMMARY -----------
Known viruses: 2735887
Engine version: 0.97.8
Scanned directories: 1
Scanned files: 129
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 4.769 sec (0 m 4 s)


-r flag stands for recursive scan – e.g. scan all sub-directories in directory and its content

To not flood your console / ssh connection one very useful option is -i (print only whetherinfected files are matched). Here is one more sample case:

# clamscan -r -i /var/tmp/
----------- SCAN SUMMARY -----------
Known viruses: 2735887
Engine version: 0.97.8
Scanned directories: 1
Scanned files: 2
Infected files: 0
Data scanned: 0.26 MB
Data read: 0.13 MB (ratio 1.97:1)
Time: 4.824 sec (0 m 4 s)


Whether you're on a physical server console and it has pc speaker or sound blaster use –bell option to ring a bell every time a Virus infection is found, for exmpl.

# clamscan -r -i --bell /var/www/

4. Scanning periodically and reporting with ClamAV directories with websites

A very common use of ClamAV is to just setup a scheduled cronjob once a month to scan, whether server folder containing a bunch of websites in separate Virtualhosts contain some viruses or malicious stuff. Then as administrator check those logs once a month to make sure server site or group of sites does not become banned in search engine (blocked by Google Chrome and Firefox as Virus hotbed) …
# crontab -u root -e
00 02 01 * * clamscan -r /var/www -l /var/log/websites-scan.log

Then once a month check out /var/log/websites-scan.log

Saint Thecla and Saint Isidore of Seville – A Patron Saint of programmers, information freaks and IT nerds

Friday, January 18th, 2013

I learned a very curious fact. We have two Christian saints St. Thekla and St. Isidore of Seville which are protectors of Computing and people involved with computational sciences programming and IT related jobs. 
The two saints both lived before the Great Church Schism thus the saints are celebrated and venerated nowadays in Roman Catholic Church and Eastern Orthodox Churches all around the Christian world.

The Holy living of Saint Thecla – The ProtoMartyr (The first Christian woman martyr)

St Thekla saint protecting all IT people and people who work with computers

The reason why Saint Thecla is considered a protector of all people who heavily work with computers and possibly hackers 🙂 is her name, Tecla means "Key" (a keyboard one) in Spanish.
Besides that everyone knows Keys (Passwords) and codes – a sort of keys as well as keys in programming arrays are widely used in Informatics and are essential part of computer basic Software.

Saint Thecla's living is mostly interesting, below is a short story of her earthly live as recognized in Orthodox Christian tradition;

Saint Thecla is considered a direct pupil of Saint Apostle Paul as she turned to be preacher of Christianity after speaking saint Apostle Paul. She lived in 30 A.D. until 1-st century after Christ. St. Thecla was virgin and decided not to mary even though she had fiancee in moment where she heard the Gospel of Christ.

She was miraculously saved from being burned at a stake by the onset of a storm, and traveled with Paul to Pisiduan Antioch. There a nobleman named Alexander desired Thecla and attempted to take her by force. Thecla fought him off, assaulting him in the process, and was put on trial for assaulting a nobleman. She was sentenced to be eaten by wild beasts (a common punishment for Christians in 1st century). Again saved by a series of miracles when the female beasts protected her against her male aggressors.

She sought out St. Paul and his companions, including St. Barnabas, who were hiding in a cave near the city. She spread the gospel of Christ with them in Antioch, and throughout her life performed many miraculous feats and suffered many tortures to give glory to God. Having retired to a desolate region of Isaurian Seleucia with the blessing of St. Paul, Thekla continued to preach God's word and heal st. Paul's spiritual children.

She lived to venerable old age of 90. Even in this old age envious pagan sorcerors come with the intention to defile the holy virgin. .  A large rock split open when St. Thekla called on Christ the Savior to help her, and the rock covered her, and she offered up her soul to the Lord.

St. Thecla is highly venerated by Nuns an interesting fact is during tonsure of nuns in the Orthodox Church, her prayerful intercession for the tonsured nun is invoked.
 

Saint Thekla  Protomartyr first christian woman martyr and protector of computer related professions and nerds

The Kontaktion and Troparion as sung in the Church on her day feast in Orthodox Church – September 24 is;
 

Hymns Troparion (Tone 4) [2] You were enlightened by the words of Paul, O Bride of God, Thekla, And your faith was confirmed by Peter, O Chosen One of God. You became the first sufferer and martyr among women, By entering into the flames as into a place of gladness. For when you accepted the Cross of Christ, The demonic powers were frightened away. O all-praised One, intercede before Christ God that our souls may be saved. Kontakion (Tone 8) O glorious Thekla, virginity was your splendor, The crown of martyrdom your adornment and the faith you trust! You turned a burning fire into refreshing dew, And with your prayers appeased pagan fury, O First Woman Martyr!

Saint Tecla is very important woman saint, there is a whole monastery dedicated to her – Saint Tecla Monastery in Ma'loula

 

Saint Isidore of Seville – St Isidore  Archbishop of Seville – An important Christian Saint writer from 6th century

St Isidore of Seville patron saint of System administrators, database programmers and IT experts



St Isidore lived (circa 560-636), he served as Archbishop of Seville. Apart from his great holiness in modern terms, he can be defined as "information freak" and even in modern terms Nerd 🙂 Even in his living he was recognized as miracle worker. St. Seville just like me and most IT people had a hobby of knowing as much as possible, he collected texts and write books on scientific or quasi-scientific facts and theories on subjects from Scripture, Church order, materials, and jurisprudence and medicine to achritecture, agriculture, botany,  food, drink furniture,war, navigation ! etc. etc. 🙂
His books are among the first attempts to create universal catalog of available information in the World. His writtings are a sort of ancient Wikipedia. Even to this age his books provide us with many information and instruction on Church dogmatics and life as understood from a Christian view.

Saint Isidore's most notable work is Etymologiae – which is among the first encyclopedias worldwide – a huge compilation of 448 chapters in 20 volumes.
Contrary to many critics of Christian faith claiming that the Holy Fathers or the Church teach the earth is square shaped and one can reach the end of the world in St. Isidore's work, it is clearly explained that the earth has a circle like form.
 

Diagrammatic world map 12th century as described by st Isidore of Seville the world is round!!!!

Saint Isidore was contemporary to Saint Maxim the Confessor and one of the last ancient Christian Philopsophers, and last of the Great Western Church Holy fathers
According to some sources, saint Isidore is said to be the most learned man of his age. His works played key role on development of Middle-ages educational life. It is curious fact that in year 2000, the Vatican proclaimed Saint Isidore as patron saint of the Internet.
In Orthodox Church, we're not the authority of the Vatican Pope, so for us Orthodox Saint Isidore is not patron saint of Internet, however his contribution for increase of worldly knowledge as we know it today is un-questinable. His Holy relics are currently held in Roman Catholic Cathedral in Murcia (Spain), so any Orthodox Christian who is in deep involvement IT who travels to their or happen to live their might drop by to venerate the saint.
 

saint Isidore of Seville creator of first Christian and knowledge encyclopedia

St. Isidore's episcopate was during a time of disintegration and transition as the ancient institutions and knowledge of the Roman Empire were disappearing. During these times, he set himself to the task of joining into a homogeneous nation the many peoples who made up the Gothic kingdom. To accomplish this task he used all the resources of religion and education available to him. He presided over the Second Council of Seville in November 619, leading to the setting forth of the nature of Christ in the Acts of the council, as he pushed for the eradication of Arianism.

saint Isidore of Seville Christian enlightener of Spain evident Christian 6th century writer - important Archibishop played key role for unification of the Goths nations

A  list of all his works, most of which are unfortunately present on the Internet only in Latin are:

  • Historia de regibus Gothorum, Vandalorum et Suevorum (a history of the Gothic, Vandal and Suebi kings)
  • Chronica Majora (a universal history)
  • De differentiis verborum, which amounts to brief theological treatise on the doctrine of the Trinity, the nature of Christ, of Paradise, angels, and men.
  • On the Nature of Things (a book of astronomy and natural history dedicated to the Visigothic king Sisebut)
  • Questions on the Old Testament.
  • a mystical treatise on the allegorical meanings of numbers
  • a number of brief letters
  • Sententiae libri tres Codex Sang. 228; 9th century
  • De viris illustribus
  • De ecclesiasticis officiis

How to count lines of PHP source code in a directory (recursively)

Saturday, July 14th, 2012

Count PHP and other programming languages lines of source code (source code files count) recursively

Being able to count the number of PHP source code lines for a website is a major statistical information for timely auditting of projects and evaluating real Project Managment costs. It is inevitable process for any software project evaluation to count the number of source lines programmers has written.
In many small and middle sized software and website development companies, it is the system administrator task to provide information or script quickly something to give info on the exact total number of source lines for projects.

Even for personal use out of curiousity it is useful to know how many lines of PHP source code a wordpress or Joomla website (with the plugins) contains.
Anyone willing to count the number of PHP source code lines under one directory level, could do it with:::

serbver:~# cd /var/www/wordpress-website
server:/var/www/wordpress-website:# wc -l *.php
17 index.php
101 wp-activate.php
1612 wp-app.php
12 wp-atom.php
19 wp-blog-header.php
105 wp-comments-post.php
12 wp-commentsrss2.php
90 wp-config-sample.php
85 wp-config.php
104 wp-cron.php
12 wp-feed.php
58 wp-links-opml.php
59 wp-load.php
694 wp-login.php
236 wp-mail.php
17 wp-pass.php
12 wp-rdf.php
15 wp-register.php
12 wp-rss.php
12 wp-rss2.php
326 wp-settings.php
451 wp-signup.php
110 wp-trackback.php
109 xmlrpc.php
4280 total

This will count and show statistics, for each and every PHP source file within wordpress-website (non-recursively), to get only information about the total number of PHP source code lines within the directory, one could grep it, e.g.:::

server:/var/www/wordpress-website:# wc -l *.php |grep -i '\stotal$'
4280 total

The command grep -i '\stotal$' has \s in beginning and $ at the end of total keyword in order to omit erroneously matching PHP source code file names which contain total in file name; for example total.php …. total_blabla.php …. blabla_total_bla.php etc. etc.

The \s grep regular expression meaning is "put empty space", "$" is placed at the end of tital to indicate to regexp grep only for words ending in string total.

So far, so good … Now it is most common that instead of counting the PHP source code lines for a first directory level to count complete number of PHP, C, Python whatever source code lines recursively – i. e. (a source code of website or projects kept in multiple sub-directories). To count recursively lines of programming code for any existing filesystem directory use find in conjunction with xargs:::

server:/var/www/wp-website1# find . -name '*.php' | xargs wc -l
1079 ./wp-admin/includes/file.php
2105 ./wp-admin/includes/media.php
103 ./wp-admin/includes/list-table.php
1054 ./wp-admin/includes/class-wp-posts-list-table.php
105 ./wp-admin/index.php
109 ./wp-admin/network/user-new.php
100 ./wp-admin/link-manager.php
410 ./wp-admin/widgets.php
108 ./wp-content/plugins/akismet/widget.php
104 ./wp-content/plugins/google-analytics-for-wordpress/wp-gdata/wp-gdata.php
104 ./wp-content/plugins/cyr2lat-slugs/cyr2lat-slugs.php
,,,,
652239 total

As you see the cmd counts and displays the number of source code lines encountered in each and every file, for big directory structures the screen gets floated and passing | less is nice, e.g.:

find . -name '*.php' | xargs wc -l | less

Displaying lines of code for each file within the directories is sometimes unnecessery, whether just a total number of programming source code line is required, hence for scripting purposes it is useful to only get the source lines total num:::

server:/var/www/wp-website1# find . -name '*.php' | xargs wc -l | grep -i '\stotal$'

Another shorter and less CPU intensive one-liner to calculate the lines of codes is:::

server:/var/www/wp-website1# ( find ./ -name '*.php' -print0 | xargs -0 cat ) | wc -l

Here is one other shell script which displays all file names within a directory with the respective calculated lines of code

For more professional and bigger projects using pure Linux bash and command line scripting might not be the best approach. For counting huge number of programming source code and displaying various statistics concerning it, there are two other tools – SLOCCount
as well as clock (count lines of code)

Both tools, are written in Perl, so for IT managers concerned for speed of calculating projects source (if too frequent source audit is necessery) this tools might be a bit sluggish. However for most projects they should be of a great add on value, actually SLOCCount was already used for calculating the development costs of GNU / Linux and other projects of high importance for Free Software community and therefore it is proven it works well with ENORMOUS software source line code calculations written in programming languages of heterogenous origin.

sloccount and cloc packages are available in default Debian and Ubuntu Linux repositories, so if you're a Debilian user like me you're in luck:::

server:~# apt-cache search cloc$
cloc - statistics utility to count lines of code
server:~# apt-cache search sloccount$
sloccount - programs for counting physical source lines of code (SLOC)

Well that's all folks, Cheers en happy counting 😉

How to convert file content encoded in windows-cp1251 charset to UTF-8 (with iconv) to be delivered properly encoded to browsing end clients

Wednesday, May 16th, 2012

windows-cp1251 bulgarian to UTF-8 / Encoding Communication Decoding Communication Funny Picture

I have a bunch of old html files all encoded in the historically obsolete Windows-cp1251. Windows-CP1251 used to be common used 7 years ago and therefore still big portions of the web content in Bulgarian / Russian Cyrillic is still transferred to the end users in this encoding.

This was just before the "UTF-8 revolution", where massively people started using UTF-8,
Well it was clear the specific national country text encoding standards will quickly be moved by to UTF-8 – Universal Encoding format which abbreviation stands for (Unicode Transformation Format).

Though UTF-8 was clear to be "the future", many web developers mostly because of their incompetency or using an old sources of learning how to writen in HTML continued to use windows-cp1251 in HTMLs. I'm even convinced, there are still developers out there who are writting websites for Bulgarian / Russian / Macedonian customers using obsolete encodings …

The smarter developers of those accustomed to windows-cp1251, KOI-8R etc. etc., were using the meta tag to specify the type of charset of the web page content with:

<meta http-equiv="content-type" content="text/html;charset=windows-cp1251">

or

<meta http-equiv="content-type" content="text/html;charset=koi-8r">

Anyhow, still many devs even didn't placed the windows-cp1251 in the head of the HTML …

The result for the system administrator is always a mess – a lot of webpages that are showing like unreadable signs and tons of unhappy customers.
As always the system administrator is considered responsible, for the programmer mistakes :). So instead of programmers fix their bad cooking, the admin has to fix it all!

One quick work around me as admin has applied to failing to display pages in Cyrillic using the Windows-cp1251 character encoding was to force windows-cp1251 as a default encoding for the whole virtualhost or Apache directory with Apache directives like:

<VirtualHost *:80>
ServerAdmin some_user@some_host.com
DocumentRoot /var/www/html
AddDefaultCharset windows-cp1251
ServerName the_host_name.com
ServerAlias www.the_host_name.com
....
....
<Directory>
AddDefaultCharset windows-cp1251
>/Directory>
</VirtualHost>

Though this mostly would, work there are some occasions, where only a particular html files from all the content served by Apache is encoded in windows-cp1251, if most of the content is already written in UTF-8, this could be a big issues as you cannot just change the UTF-8 globally to windows-cp1251, just because few pages are written in archaic encoding….
Since most of the content is displayed to the client by Apache (as prior explained) just fine, only particular htmls lets's ay single.html, single2.html etc. etc. are displayed with some question marks or some non-human readable "hieroglyphs".

Below is a screenshot from two pages returned to my browser in wrongly set htmls charset:

Improper Windows CP1251 encoding with Apache set to serve UTF-8 encoding questiomarks

Improper Windows CP1251 delivered page in UTF-8 browser view

Apache returns cp1251 in some non-UTF8 wrong encoding (webserver improperly served cyrillic encoding)

Improperly served encoding CP1251 delivered by Apache in non-utf-8 encoding

When this kind of issues occur, the only solution is to simply login to the server and use iconv command to convert all files returning unreadable content from whatever the non UTF-8 encoding is lets say in my case Bulgarian typeset of cp1251 to UTF-8

Here is how the iconv command to convert between windows-cp1251 to utf-8 the two sample files named single1.html and single2.html

server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single1.html > single1.html.utf8
server:/web# mv single1.html single1.html.bak;
server:/web# mv single1.html.utf8 single1.html
server:/web# /usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single2.html > single2.html.utf8
server:/web# mv single2.html single2.html.bak;
server:/web# mv single2.html.utf8 single2.html

I always, make copies of the original cp1251 encoded files (as you see mv single1.html single1.html.bak), because if something goes wrong with convertion I can easily revert back.

If there are 10 files with consequential numbers naming they can be converted using a short for loop, like so:

server:/web# for i $(seq 1 10); do
/usr/bin/iconv -f WINDOWS-1251 -t UTF-8 single$i.html > single$i.html.utf8;mv single$i.html single$i.html.bak
mv single$i.html.utf8 single$i.html
done

Just as earlier mentioned if single1.html, single2.html … has in the html <head>:

<meta http-equiv="Content-Type" content="text/html; charset=windows-1251">

You should open, each of the files in question and wipe out the line either by hand or use sed to wipe it in one loop if it has to be done for lets say 10 files named (single{1..10})

server:/web# for i in $(seq 1 10); do
sed '/<meta http-equiv="Content-Type" content="text\/html; charset=windows-1251>/d' single$i.txt > single$i.txt.new;
mv single$i.txt single$i.txt.bak;
mv single$i.txt.new single$i.txt

Well now,

How to permanently enable Cookies in Lynx text browser – Disable accept cookies prompt in lynx console browser

Wednesday, April 18th, 2012

lynx-text-browser-logo
The default behaviour of lynx console text browser on Linuces, BSD and other free OSes is to always ask, for the accept cookies prompt once an internet web page is opened that requires browser cookies to be enabled.

I should admin, having this "secure by default" (always ask for new cookies) behaviour in lynx was a good practice from a security point of view.

Another reason, why this cookies prompt is enabled by default is back in the days, when lynx was actively developed by programmers the websites with cookies support was not that many and even cookies was mostly required for user/pass authentication (all those who still remember this days the websites that requires authentication was a way less than today) …
With this said the current continuing security cautious behaviour in the browser, left from its old days is understandable.

Screenshot Google Accept cookies Lynx dialog FreeBSD

However I personally sometimes, need to use lynx more frequently and this behaviour of always opening a new website in text mode in console to prompts me for a cookie suddenly becomes a big waste of time if you use lynx to browser more than few sites. Hence I decided to change the default way lynx handles cookies and make them enabled by default instead.
Actually even in the past, when I was mainly using internet in console on every new server or home Linux install, I was again making the cookies to be permanently accepted.
Everyone who used lynx a few times already knows its "annoying" to all time accept cookie prompts … This provoked me to write this short article to explain how enabling of constant cookie accepting in lynx is done

To enable the persistent cookies in lynx, one needs to edit lynx.cfg on different GNU / Linux and BSD* distributions lynx.cfg is located in different directory.

Most of the lynx.cfg usual locations are /etc/lynx/lynx.cfg or /etc/lynx.cfg as of time of writting this post in Debian Squeeze GNU / Linux the lynx.cfg is located in /etc/lynx-cur/lynx.cfg, whether for FreeBSD / NetBSD / OpenBSD users the file is located in /usr/local/etc/lynx.cfg

What I did to allow all cookies is open lynx.cfg in vim edit and change the following lines:

a)

#FORCE_SSL_COOKIES_SECURE:FALSE

with

FORCE_SSL_COOKIES_SECURE:TRUE

b)

#SET_COOKIES:TRUE

uncomment it to:

SET_COOKIES:TRUE

c) next, change

ACCEPT_ALL_COOKIES:FALSE

ACCEPT_ALL_COOKIES:TRUE

Onwards opening any website with lynx auto-accepts the cookies.

lynx Always allowing from domain cookies Linux screenshot

Google in Bulgarian Lynx browser screenshot

For people who care about there security (who still browse in console (surely not many anymore)), permanently allowing the cookies is not a good idea. But for those who are ready to drop off little security for convenience its ok.
 

Did you that every single day there are …

Saturday, April 7th, 2012

Do you know that every single day there are:

  • 150 Million Blogs being read
  • 60 Million New Facebook Status Updates
  • 140 Million New "Treets" …
  • 2 Million New Videos Posted to Youtube per day
  • 3.3 Billion! Product and Information Searches
  • 188 Billion! Emails Sent …

 

This statistics were sent to me just few weeks, ago by a friend. I have no clue where he got the statistics, but the numbers are really amazing. From a business perspective point of view this numbers are tremendously HIGH! Nowadays only about 2.5 billion people are actively using the internet daily.

This means more than half of the humanity is still about to join using the internet in the 10 or 15 years to come. Though the continuous use of internet has a very bad impact on us. It is a tremendously big business opportunity field. With this said definitely business innovative people and enterpreneurs should reconsider, there strategic plans for potential products and extend or include the internet in anything they do in order to maximize profits.

All this is just news for anyone who is involved somehow actively with the net (like me), so system admins, web designers, programmers, use your brains and start making money from the internet. If one doesn't start with monetarizing his high tech skills, its quite likely some bad tied suit guy took the lead and made his millions or billions  on our back 🙂