Posts Tagged ‘cronjob’

Prevent rsync cronjob to run multiple times via cronjob on Linux

Wednesday, November 21st, 2018

prevent-rsync-rsync-to-run-multiple-times-via-cronjob-on-linux

Today I had a report of a server whose Load Avarage keeps at the high level of 86, the machine runs on a bare metal rock solid hardware and even with such high Loads of the kernel it runs fine, but due to the I/O overhead the SANs red from a remote NetApp storage device started to be sluggish and hence it needed to be reviewed, thus I jumped in via the hop station (jump host) into the server.
 

1. Short investation on root cause for high server load


After a short investigation, I've found an rsync job set by someone on a cron job to be routinely run every 30 minutes, thus the old scheduled rsync, which seemed to run multiple times on the server (about 50 processes) of same rsync (file system synchronization was running) and as expected the storage was saddled with mutiple Input / Output requests.

The root cron job was like that:
 

server:~# crontab -u root -l |grep -i rsync
/usr/bin/rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize


A process list showed the following high number of running mirrored rsyncs:

 

server:~# ps axuwwf | grep -i rsync | wc -l
80


 

2. The Fix – Set Rsync to only via cron only in case if it is not already running in background


In order to fix it, I had to kill all current running rsync (here luckily only same single instance of rsync was running, but generally I was cautious to check no other rsync jobs are running – otherwise I would have mistakenly killed some other rsync job ongoing …)

Then I set the following new cron job one liner quick shell script that does the job to assign a pid file that is created before rsync and deleted after rsync completion.
 

if [ ! -e /tmp/repo_dba_sync.lock ]; then touch /tmp/repo_dba_sync.lock; /usr/bin/rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize ; trap 'rm -f /tmp/repo_dba_sync.lock; fi' EXIT  >/dev/null 2>&1


The cron job looked like so:

 

*/30 * * * * if [ ! -e /tmp/repo_dba_sync.lock ]; then touch /tmp/repo_dba_sync.lock; /usr/bin/rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize ; trap 'rm -f /tmp/repo_dba_sync.lock; fi'  EXIT >/dev/null 2>&1

Just in case if you're wondering
a trap should be used to verify that the lock file is removed when the script is exited for any reason.
This way the lock file will be removed even if the script exits before the end of the script.

An alternative and more simple ways to do it is via:
 

pgrep rsync > /dev/null || rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize

 

Or if you don't want to use bash's:
 

if []; then; fi


condition but still use a file lock the flock command can be used like so:
 

flock -n lock_file -c "rsync …"

Run custom user script after reboot with a cronjob on Linux

Friday, September 21st, 2018

howto-add-custom-script-on-reboot-with-non-administrative-root-user-on-gnu-linux

Perhaps you have a websites on a server on some Linux distro / FreeBSD / AIX / HP-UX / Sun OS that uses Vixie-cron cron jobs to run / respawn dead php / python perl scripts etc.  that do stuff on the server every lets say 30 minutes an hour or even every 12 / 24 hours in the background.
But sometimes due to server or Linux kernel upgrades you need to reboot the server with reboot command or shutdown -r now right in the minutes the scripts were supposed to run and do a database backup / synchronize some data with a remote MySQL with replication configured or do some site maintenance job such as clearing old Messages / Spam / data log file records.

Of course one possible workaround to that is to add the non-root user scripts in question  to /etc/rc.local to run on every server boot, but that fix requires a root access and very often developers did not have that, neither sysadmins are willing to bother  add a user sudo-ed scripts e.g. add  (sudo -u whateveruser "/path/to/script") to /etc/rc.local.

Run custom user  script after reboot the cron way

Happily there is ctually a better cron way to do that by telling crond to execute a cronjob during boot and assuming the non-admin user on the Linux has access to shell and access to using cron jobs by using @reboot cron direcive.

Here is few examples on how to run a re-run cron job on start up:
 

linux:~$ crontab -e


Some editor as nano or vim will open listing all your previous set system jobs to add scripts phpjob.php

@reboot  /user/dir/path/to/phpjob
@reboot  /path/to/shell_script
@reboot  /path/to/linux-command

That's pretty shitty situation but thanksfully remote access of website username with SSH will be enough to set the right cron activity (of course this can't be made for servers that are missing crond service running.

The scripts set in cron job that way will respawn right after the OS system had booted and there will be no need for them to wait the next hour to execute configured data synch.
For more on how to run a tiny script respawn every second using a single cron job check out my previous article How to set a crontab to execute commands on a seconds time interval on GNU / Linux and FreeBSD.

Historically it is interestingly to mention that in times before systemd appeared in modern Linux distributions,
a cool thack to run a script that had to be respawned every second after boot for a privileged user was to use /etc/inittab (no longer available in most all non System V Linux distrubutions in 2018), to do so
if you happen to still administer some old Linux servers CentOS 7 etc. you and you need to add a custom script to run and respawn all the time by including a line in /etc/inittab (again assuming a System V Linux is on remote machine):
 

mysvc:235:respawn:/home/me/bin/my_service_starter_script

Putting a service to respawn in that way via inittab uses init (process) and the kernel and keeps re-running it.

Note: 

If a command fails when it starts, and init is configured to restart it, it will use a lot of system resources: init starts it, it fails, init starts it, it fails, init starts it, it fails, and so on, ad infinitum. To prevent this, init will keep track of how often it restarts a command, and if the frequency grows to high, it will delay for five minutes before restarting again.If the kernel 
Using inittab should always be tested on a testbed before adding to remote server, note that if the script is using a lot of memory and keeps crashing it can easily leave out the kernel without memory and the system is about to get errors like:
 

process respawning too fast 

 

Another useful thing if you have doubts that the script might be crashing is to use something like monit to monitor the script (assuming the script does provide some kind of tcp / udp connection on port) and report you via email / sms about issues with crashing script.
If you hear monit the first time I recommend you read my previous article Monitoring and restart server services (Apache, Mysql, Bind) with Monit to prevent server downtimes.

Improve your night sleep (Insomnia) on Linux with redshift

Friday, February 15th, 2013

sleep better at night while using Linux with redshift command line and gui applet / improve your insomnia while being a linux user
For a while I've been experiencing troubles with getting asleep. As I work in the field of IT already for 10 years and with time it seems the problem is accelerating. I've read on the internet a lot on the topic of getting asleep and how this relates to computers and computer equipment use and came to the conclusion one of the main reasons I have troubles getting asleep is I use computer late at night usually I use PC until 2, 3 o'clock. Then when I go to bed, I cannot fall asleep until its early in the morning usually 6, 7 in the morning. My main operating system on notebook is Linux so almost all of the time I use Linux. I've noticed when I occasionally use Windows, my eyes tend to be less strained afterwards and I sleep better. Thus I suspected there should be some kind of tool in Linux which changes how PC screen displays to make eyes more relaxed. I didn't have the time to research seriously and before some time the little research I've done on this led me to nothing. Just a week ago, I've read one of the articles in Linux Magazine (December) issue, there is a very thorough article in it on how to avoid headaches and eye strain using a tiny tool which changes monitor screen gamma called redshift. In this article will explain in short how to install and use redshift to make your PC work less stressful and improve your sleeping at night. I'm using Debian as a basis Linux distro and thre redshift is available via package, other deb derivatives Ubuntu, Xubuntu etc. aslo have it. For Fedora and most of other Linux distributions redshift is also available from default repositories. For those who use Slackware or some older Linux distributions, redshift has to be installed manually from source but this should be trivial.

1. Install Redshift and Redshift-gtk packages

To install on Debian and Ubuntu:

# apt-get –yes install redshift redshift-gtk

On Fedora install with yum:

# yum -y install redshift

After installed you will have two programs to tune the screen color temperature, one is console based ( redshift ) and the other one is GUI based ( gtk-redshift ).

redshift-gtk is a GUI frontend

Here is a list of redshift tool options:

2. Changing color gamma with redshift

hipo@noah:~$ redshift -h
Usage: redshift -l LAT:LON -t DAY:NIGHT [OPTIONS...]

Set color temperature of display according to time of day.

  -h        Display this help message
  -v        Verbose output

  -g R:G:B    Additional gamma correction to apply
  -l LAT:LON    Your current location
  -m METHOD    Method to use to set color temperature (randr or vidmode)
  -o        One shot mode (do not continously adjust color temperature)
  -r        Disable initial temperature transition
  -s SCREEN    X screen to apply adjustments to
  -t DAY:NIGHT    Color temperature to set at daytime/night

Please report bugs to <https://bugs.launchpad.net/redshift>

To set your screen to Reddish mode which will relax your eye strain and therefore – when you go to sleep you have a better sleep, type:

 hipo@noah:~$ redshift -l -35:-56 -t 5000:3300

Other monitor red-color afternoon or night time gamma to relax your eyes is;

hipo@noah:~$ redshift -l 52.5:13.4

3. Setting redshift to auto change screen gamma via cronjob

If you prefer automatically changing color gamma to reddish at night – will make your eyes (and hence organism) less alert set as a cronjob in lets say 22:00 o'clock at night;

 hipo@noah:~$ crontab -u root -e

00 22 * * * redshift -l -35:-56 -t 5000:3300 2>&1 >/dev/null

4. Controlling manually between standard and reddish color gamma through gtk-redshift

For people who like to control and switch between color gamma using GNOME Applet run gtk-redshift like so:

hipo@noah:~$ gtk-redshift  -l 52.5:13.4

gtk redshift gnome applet screenshot Debian Linux

Clicking on the icon of redshift the color gamma gets changed to red, another toggle reverses back to normal.

There is another tool called F.lux which does the same as redshift. F.lux precedes redshift, actually redshift author write it as attempt to create superior F.lux. Flux works on Windows and Mac OS X – so users who work at night on this platforms might want ot check it. I tried installing f.lux on my Debian Squeeze Linux but had troubles because of requirement for newer python-appindicator :

noah:/home/hipo# apt-get install fluxgui
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 fluxgui : Depends: python-appindicator (>= 0.0.19) but it is not installable
E: Broken packages

 

Probably with some tampering I can make f.lux work but I was lazy and since I already had redshift, I decided to quit and just be a happy redshift user.