Posts Tagged ‘thread’

How to set a crontab to execute commands on a seconds time interval on GNU / Linux and FreeBSD

Sunday, October 30th, 2011

crontab-execute-cron-jobs-every-second-on-linux-cron-logo
Have you ever been in need to execute some commands scheduled via a crontab, every let’s say 5 seconds?, naturally this is not possible with crontab, however adding a small shell script to loop and execute a command or commands every 5 seconds and setting it up to execute once in a minute through crontab makes this possible.
Here is an example shell script that does execute commands every 5 seconds:

#!/bin/bash
command1_to_exec='/bin/ls';
command2_to_exec='/bin/pwd';
for i in $(echo 1 2 3 4 5 6 7 8 9 10 11); do
sleep 5;
$command1_to_exec; $command2_to_exec;
done

This script will issue a sleep every 5 seconds and execute the two commands defined as $command1_to_exec and $command2_to_exec

Copy paste the script to a file or fetch exec_every_5_secs_cmds.sh from here

The script can easily be modified to execute on any seconds interval delay, the record to put on cron to use with this script should look something like:

# echo '* * * * * /path/to/exec_every_5_secs_cmds.sh' | crontab -

Where of course /path/to/exec_every_5_secs_cmds.sh needs to be modified to a proper script name and path location.

Another way to do the on a number of seconds program / command schedule without using cron at all is setting up an endless loop to run/refresh via /etc/inittab with a number of predefined commands inside. An example endless loop script to run via inittab would look something like:

while [ 1 ]; do
/bin/ls
sleep 5;
done

To run the above sample never ending script using inittab, one needs to add to the end of inittab, some line like:

mine:234:respawn:/path/to/script_name.sh

A quick way to add the line from consone would be with echo:

echo 'mine:234:respawn:/path/to/script' >> /etc/inittab

Of course the proper paths, should be put in:

Then to load up the newly added inittab line, inittab needs to be reloaded with cmd:

# init q

I've also red, some other methods suggested to run programs on a periodic seconds basis using just cron, what I found in stackoverflow.com's  as a thread proposed as a solution is:

* * * * * /foo/bar/your_script
* * * * * sleep 15; /foo/bar/your_script
* * * * * sleep 30; /foo/bar/your_script
* * * * * sleep 45; /foo/bar/your_script

One guy, even suggested a shorted way with cron:

0/15 * * * * * /path/to/my/script

Installing Usual PHP Apache needed modules for new Debian GNU / Linux servers

Tuesday, September 18th, 2012

Almost evertime whether install a new Debian Linux server to host some websites, use the popular small and mid-sized hosting solution combination:
 

  • Apache webserver
  • PHP module to interpret the PHP programming code
  • MySQL to store the DB Backend

Installing the basis for on Debian is easy:


# apt-get install --yes apache2 apache2-mpm-prefork libapache2-mod-php \
mysql-server php5 php5-mysql mysql-client mysql-common phpmyadmin
....

This of course is in case, if necessery to run websites which are written to usephp code which is not thread safety (Use Apache child prefork technology to manage processes); For websites writen to be thread safety (not use some forking php functions like: php – exec(); fork(); etc. – I install apache2-mpm-prefork for better Webserver performance and speed.

This minimum collection of packages is good only for basic, websites and most Joomla, WordPress, Drupal or whatever custom PHP websites has to be hosted usually require much more PHP functions which are not part of this basic bundle. Hence as I said prior on almost all new Linux debian / ubuntu deb package based servers need to install following list of extra PHP deb packages:


# apt-get install --yes php-apc php-auth-ssl php-mail \
php-http php-net-smtp php-net-socket php-pear php5-cli php5-curl \
php5-gd php5-imagick php5-mapscript php5-mcrypt php5-odbc php5-sybase \
php5-xsl
...,

After installing this standard bundle of modules PHP is extended to support somenice functionalities like Image editting / convertion / resizing … various graphic editting functions as supported by infamous ImageMagick and GDlib, PEAR support (PHP Extension and Application Repository) bundle providing number of useful PHP classes.
php5-xInstalling Usual PHP Apache needed modules for new Debian GNU / Linux serverssl is usually necessery for websites which pass data in XSL format (a specific formatted XML data). php5-xsl is usually necessery for websites containing flash animationsPresentation, videos, games etc.

That is mostly it, hope this helps some sys admin like me who need configure new Debian based hosting server 🙂

How to tune MySQL Server to increase MySQL performance using mysqltuner.pl and Tuning-primer.sh

Tuesday, August 9th, 2011

MySQL Easy performance tuning with mysqltuner.pl and Tuning-primer.sh scripts

Improving MySQL performance is crucial for improving a website responce times, reduce server load and improve overall work efficiency of a mysql database server.

I’ve seen however many Linux System administrators who does belittle or completely miss the significance of tuning a newly installed MySQL server installation.
The reason behind that is probably caused by fact that many people think MySQL config variables, would not significantly improve performance and does not pay back for optimization efforts. Moreover there are a bunch of system admins who has to take care for numerous services so they don’t have time to get good knowledge to optimize MySQL servers.
Thus many admins and webmasters nowdays, think optimizations depend mostly on the side of the website programmers.
It’s also sometimes falsely believed that optimizing a MySQL server could reduce the overall server stability.

With the boom of Internet website building and internet marketing, many webmasters emerged and almost anybody with almost no knowledge on GNU/Linux or minimal or no knowledge on PHP can start his Online store, open a blog or create a website powered by some CMS like joomla.
Thus nowdays many servers even doesn’t have a hired system administrators but are managed by people whose knowledge on *Nix is almost next to zero, this is another reason why dozens of MySQL installations online are a default ones and are not taking a good advantage of the server hardware.

The incrase of website visitors leads people servers expectations for hardware also to grow, thus many companies simply buy a new hardware instead of taking the few time to investigate on how current server hardware can be utilized better.
In that manner of thought I though it will be a good idea to write this small article on Tuning mysql servers with two scripts Tuning-primer.sh and mysqltuner.pl.
The scripts are ultra easy to use and does not require only a minimal knowledge on MySQL, Linux or (*BSD *nix if sql is running on BSD).
Tuning-primer.sh and mysqltuner.pl are therefore suitable for a quick MySQL server optimizations to even people who are no computer experts.

I use this two scripts for MySQL server optimizations on almost every new configured GNU/Linux with a MySQL backend.
Use of the script comes to simply download with wget, lynx, curl or some other web client and execute it on the server host which is already running the MySQL server.

Here is an example of how simple it is to run the scripts to Optimize MySQL:

debian:~# perl mysqltuner.pl
>> MySQLTuner 1.2.0 - Major Hayden >major@mhtx.net<
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering

——– General Statistics ————————————————–
[–] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.1.49-3
[OK] Operating on 64-bit architecture

——– Storage Engine Statistics ——————————————-
[–] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[–] Data in MyISAM tables: 6G (Tables: 952)
[!!] InnoDB is enabled but isn’t being used
[!!] Total fragmented tables: 12

——– Security Recommendations ——————————————-
[OK] All database users have passwords assigned

——– Performance Metrics ————————————————-
[–] Up for: 1d 2h 3m 35s (68M q [732.193 qps], 610K conn, TX: 49B, RX: 11B)
[–] Reads / Writes: 76% / 24%
[–] Total buffers: 512.0M global + 2.8M per thread (2000 max threads)
[OK] Maximum possible memory usage: 6.0G (25% of installed RAM)
[OK] Slow queries: 0% (3K/68M)
[OK] Highest usage of available connections: 7% (159/2000)
[OK] Key buffer size / total MyISAM indexes: 230.0M/1.7G
[OK] Key buffer hit rate: 97.8% (11B cached / 257M reads)
[OK] Query cache efficiency: 76.6% (46M cached / 61M selects)
[!!] Query cache prunes per day: 1822075
[OK] Sorts requiring temporary tables: 0% (1K temp sorts / 2M sorts)
[!!] Joins performed without indexes: 63635
[OK] Temporary tables created on disk: 1% (26K on disk / 2M total)
[OK] Thread cache hit rate: 99% (159 created / 610K connections)
[!!] Table cache hit rate: 4% (1K open / 43K opened)
[OK] Open file limit used: 17% (2K/16K)
[OK] Table locks acquired immediately: 99% (36M immediate / 36M locks)

——– Recommendations —————————————————–
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Increasing the query_cache size over 128M may reduce performance
Adjust your join queries to always utilize indexes
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
query_cache_size (> 256M) [see warning above] join_buffer_size (> 256.0K, or always use indexes with joins) table_cache (> 7200)

You see there are plenty of things, the script reports, for the unexperienced most of the information can be happily skipped without need to know the cryptic output, the section of importance here is Recommendations for some clarity, I’ve made this section to show up bold.

The most imporant things from the Recommendations script output is actually the lines who give suggestions for incrase of certain variables for MySQL.In this example case this are the last three variables:
query_cache_size,
join_buffer_size and
table_cache

All of these variables are tuned from /etc/mysql/my.cnf (on Debian) and derivatives distros and from /etc/my.cnf on RHEL, CentOS, Fedora and the other RPM based Linux distributions.

On some custom server installs my.cnf is also located in /usr/local/mysql/etc/ or some other a bit more unstandard location 😉

Anyways now having the Recommendation from the script, it’s necessery to edit my.cnf and try to double the values for the suggested variables.

First, I check if all the suggested variables are existent in my config with grep , if they’re not then I’ll simply add the variable with doubled size of the suggested values.
P.S: One note here is sometimes some values which are configured, are the default value for the MySQL server and does not have a record in my.cnf

debian:~# grep -E 'query_cache_size|join_buffer_size|table_cache' /etc/mysql/my.cnf table_cache = 7200
query_cache_size = 256M
join_buffer_size = 262144

All of my variables are in the config so, now edit my.cnf and set values to:
table_cache = 14400
query_cache_size = 512M
join_buffer_size = 524288

I always, however preserve the old variable’s value, because sometimes raising the value might create problem and the MySql server might be unable to restart properly.
Thus before going with adding the new values make sure the old ones are commented with # , e.g.:
#table_cache = 7200
#query_cache_size = 256M
#join_buffer_size = 262144

I would recommend vim as editor of choice while editing my.cnf as vim completely rox 😉 If you’re not acquainted to vim use nano or mcedit or your editor of choice 😉 :

debian:~# vim /etc/mysql/my.cnf
...

Assuming that the changes are made, it’s time to restart MySQL to make sure the new values are read by the SQL server.

debian:~# /etc/init.d/mysql restart
* Stopping MySQL database server mysqld [ OK ]
* Starting MySQL database server mysqld [ OK ]
Checking for tables which need an upgrade, are corrupt or were not closed cleanly.

If mysql server fails, however to restart, make sure immediately you reverse back the changed variables to the commented values and restart once again via mysql init script to make server load.

Afterwards start adding the values one by one until find out which one is causing the mysqld to fail.

Now the second script (Tuning-primer.sh) is also really nice for MySQL performance optimizations are necessery. However it’s less portable (as it’s written in bash scripting language).
Consider running this script among different GNU/Linux distributious (especially the newer ones) might produce errors.
Tuning-primer.sh requires some minor code changes to be able to run on FreeBSD, NetBSD and OpenBSD *nices.

The way Tuning-primer.sh works is precisely like mysqltuner.pl , one runs it it gives some info about current running MySQL server and based on certain factors gives suggestions on how increasing or decresing certain my.cnf variables could reduce sql query bottlenecks, solve table locking issues as well as generally improve INSERT, UPDATE query times.

Here is an example output from tuning-primer.sh run on another server:

server:~# wget https://www.pc-freak.net/files/Tuning-primer.sh
...
server:~# sh Tuning-primer.sh
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -

MySQL Version 5.0.51a-24+lenny5 x86_64

Uptime = 8 days 10 hrs 19 min 8 sec
Avg. qps = 179
Total Questions = 130851322
Threads Connected = 1

Server has been running for over 48hrs.
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

SLOW QUERIES
Current long_query_time = 1 sec.
You have 16498 out of 130851322 that take longer than 1 sec. to complete
The slow query log is NOT enabled.
Your long_query_time seems to be fine

MAX CONNECTIONS
Current max_connections = 2000
Current threads_connected = 1
Historic max_used_connections = 85
The number of used connections is 4% of the configured maximum.
Your max_connections variable seems to be fine.

WORKER THREADS
Current thread_cache_size = 128
Current threads_cached = 84
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MEMORY USAGE
Tuning-primer.sh: line 994: let: expression expected
Max Memory Ever Allocated : 741 M
Configured Max Memory Limit : 5049 M
Total System Memory : 23640 M

KEY BUFFER
Current MyISAM index space = 1646 M
Current key_buffer_size = 476 M
Key cache miss rate is 1 / 56
Key buffer fill ratio = 90.00 %
You could increase key_buffer_size
It is safe to raise this up to 1/4 of total system memory;
assuming this is a dedicated database server.

QUERY CACHE
Query cache is enabled
Current query_cache_size = 64 M
Current query_cache_used = 38 M
Current Query cache fill ratio = 59.90 %

SORT OPERATIONS
Current sort_buffer_size = 2 M
Current record/read_rnd_buffer_size = 256.00 K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 128.00 K
You have had 111560 queries where a join could not use an index properly
You have had 91 joins without keys that check for key usage after each row
You should enable “log-queries-not-using-indexes”
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.

TABLE CACHE
Current table_cache value = 3600 tables
You have a total of 798 tables
You have 1904 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current tmp_table_size = 128 M
1% of tmp tables created were disk based
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 128.00 K
Current table scan ratio = 797 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 1782
You may benefit from selective use of InnoDB.

As seen from script output, there are certain variables which might be increased a bit for better SQL performance, one such variable as suggested is key_buffer_size(You could increase key_buffer_size)

Now the steps to make the tunings to my.cnf are precisely the same as with mysqltuner.pl, e.g.:
1. Preserve old config variables which will be changed by commenting them
2. Double value of current variables in my.cnf suggested by script
3. Restart Mysql server via /etc/init.d/mysql restart cmd.
4. If mysql runs fine monitor mysql performance with mtop or mytop for at least 15 mins / half an hour.

if all is fine run once again the tuning scripts to see if there are no further improvement suggestions, if there are more follow the 4 steps described procedure once again.

It’s also a good idea that these scripts are periodically re-run on the server like once per few months as changes in SQL queries amounts and types will require changes in MySQL operational variables.
The authors of these nice scripts has done great job and have saved us a tons of nerves time, downtimes and money spend on meaningless hardware. So big thanks for the awesome scripts guys 😉
Finally after hopefully succesful deployment of changes, enjoy the incresed SQL server performance 😉

Fix Null error in WordPress comment reply with wordpress-threaded-comments plugin enabled

Friday, April 6th, 2012

I'm running WordPress for already 3 years or so now. Since some very long time. The first wordpress install, I can hardly remember but it something like wordpress 2.5 or wordpress 2.4

Since quite a long time my wordpress blog is powered by a number of plugins, which I regularly update, whenever new plugins pops up …
I haven't noticed most of the time problems during major WordPress platform updates or the update of the installed extensions. However, today while I tried to reply back to one of my blog comments, I've been shocked that, I couldn't.
Pointing at the the Comment Reply box and typing inside was impossible and a null message was stayed filled in the form:

To catch what was causing this weird misbehaving with the reply comments functionality, I grepped through my /var/www/blog/wp-content/plugins/* for the movecfm(null,0,1,null):

# cd /var/www/blog/wp-content/plugins
# grep -rli 'movecfm(null,0,1,null)' */*.php
wordpress-thread-comment/wp-thread-comment.php

I've taken the string movecfm(null,0,1,null) from the browser page source in in my Firefox by pressing – Ctrl+U).

Once I knew of the problem, I first tried commenting the occurances of the null fields in wp-thread-comment.php, but as there, were other troubles in commenting this and I was lazy to read the whole code, checked online if some other fellows experienced the same shitty null void javascript error and already someone pointed at a solution. In the few minutes search I was unable to find anyone who reported for this bug, but what I found is some user threads on wordpress.org mentioning since WordPress 2.7+ the wordpress-threaded-comments is obsolete and the functionality provided by the plugin is already provided by default in newer WPinstalls.

Hence in order to enable the threaded comments WordPress (embedded) reply functionality from within the wp-admin panel used:

Settings -> Discussions -> Enable Threaded (nested) comments (Tick)

Enable Nested Comments WordPress default wp comments enable reply functionality screenshot

You see there is also an option to define how many nested comments subcomments, can be placed per comment, the default was 5, but I thought 5 is a bit low so increased it to 10 comments reply possible per comment.

Finally, to prevent the default threaded comments to interfere with the WordPress Threaded Comments plugin, disabled the plugin through menus:

Plugins -> Active -> WordPress Thread Comments (Deactivate)

This solved the weird javascript null "bug" caused by wordpress-threaded-comments once and for all.
Hopefully onwards, my blog readers will not have issues with threaded Reply Comments.

BSD (Berkley Software Distribituion) use by distribution type (FreeBSD, NetBSD, OpenBSD, DrangflyBSD) use percantage charts

Saturday, February 18th, 2012

I've hit an interesting article in Wikipedia called Comparison of BSD operating systems
The article explains basic difference between different BSD (Berkley Software Distributions) and what is the primary accent of each of the BSD (free software OS) distributions. It also reveals basic details about the history and how each of the BSD's came to existence. I recommend to anyone interested in free software as it is just a great reading for everybody interested in FOSS.

The most interesting part of the wiki thread is a bar chart, provided by BSD Certification Group research conducted in September 2005.

FreeBSD, OpenBSD, NetBSD, Dragonflybsd usage statistics

The above diagram is showing the proportion of users of each BSD variant from the BSD usage survey prior conducted

The research is already 6 years old, and unfortunately as of time of writting seems to be the only publicly available. Though being outdated, I believe generally the bar charts distributions along different BSD variants would be mostly true. The only big difference will be probably in PC-BSD which is not even on the diagram should have outbeaten DragonflyBSD's use. Since there is no public data available for 2012 and the years 2005 – 2012 for the use percantage of each of the BSD distributions, I've thought about a pseudo way to get some general statistics on each of the BSD distributions popularity. The methodology to gather the required statistics comes to simply, type in Google each of the BSD variant "code names" (e.g. freebsd, netbsd, openbsd etc.) and look at the number of results returned. It seems logical the more results distribution keyword searched returns, the bigger the probability of more users to be involved in developing or using the respective BSD variant.

Below you see the results, I've gathered in my quick "google research":

FreeBSD NetBSD OpenBSD BSD variant (users) use diagram based on Google searches of keywords 2012

As you can see in the above data FreeBSD is still probably leading the BSD use, the public interest to OpenBSD – BSD focused on security has significantly grow since the last 6 years. Next it is seen the PC-BSD users base has probably tremendously increased and according to the Google results returned it is probably on a 3rd place by users interest (use?) followed by NetBSD with only 1.47% of all the BSD users. Lastly with only 0.99%, orders Dragonfly BSD which no longer is so popular as a Desktop BSD based OS as it used to be back in 2005.
Again the presented diagram results are based on only on the factor of Google BSD variant popularity and hence shouldn't be consired too trustworthy, still I'm sure it gives a general idea on how used is each of the BSD variants as of Jan 2012.

How to install Ubuntu Linux on Acer ASPIRE 5736Z Notebook / Get around the black screen install CD issue

Friday, July 1st, 2011

My sister’s newly bought laptop is Acer Aspire 5736Z . By the default this notebook comes with some kind of Linux distribution Linpus .
Even though this Linpus (crafted Linux especially for Acer notebooks) looked really nice, it prooved to be a piece of shit linux distro.
Linplus was unable to even establish a simple Wireless WPA2 protected connection with my wireless router, not to mention that the physical Linux consoles (CTRL+ALT+F1) were disabled …

This LinPlus was so bad that I couldn’t even launch any type of terminal on it (I was stuck!) so I decided to kill it and make a decent latest Ubuntu 11.04 Install on it.

I was surprised to find out that trying to boot up the Ubuntu 11.04 installer led me to a black screen (black screen of death).

The v Aspire’s 5736Z monitor kept completely blank, where the hard drive was continuously reading (indicating that the Ubuntu installer has properly booted but it couldn’t light up the notebook screen).

A bit of investigation on any issues with this Acer notebook model has led me to a thread in fedora forums:
http://forums.fedoraforum.org/showthread.php?t=263794
On this forum the same kind of Linux install problem was described to also occur with ASPIREs 5736Z during a Fedora install.

I just tried the suggested fix and it works like a charm.

The fix goes like this:

1. Invoke the Ubuntu settings parameter Install pre install screen

Just press any button while the Ubuntu installer CD is reading and after few secs the Install options screen should appear, like you see it in below’s screenshot:

Ubuntu Install boot options parameters screen

2. Select the nomodetest Boot CD Ubuntu option

You see in the above screenshot the F6 Other Options . I had toto press F6 and choose the nomodetest boot option to make the Ubuntu be able to further boot up.

After selecting the nomodetest option and pressing on the Install Ubuntu menu option the graphic installer launched succesfully 😉
Hope this small tip to be helpful to some Ubuntu or other Linux user who is trying to install Linux on his Acer Aspire 5736Z
Cheers 😉