Posts Tagged ‘init scripts’

How to start / Stop and Analyze system services and improve Linux system boot time performance

Friday, July 5th, 2019

This post is going to be a very short one and to walk through shortly to System V basic start / stop remove service old way and the new ways introduced over the last 10 years or so with the introduction of systemd on mass base across Linux distributions.
Finally I'll give you few hints on how to check (analyze) the boot time performance on a modern GNU / Linux system that is using systemd enabled services.

1. System V and the old days few classic used ways to stop / start / restart services (runlevels and common wrapper scripts)


The old fashioned days when Linux was using SystemV / e.g. no SystemD used way was to just go through all the running services with following the run script logic inside the runlevel the system was booting, e.g. to check runlevel and then potimize each and every run script via the respective location of the bash service init scripts:


root@noah:/home/hipo# /sbin/runlevel 
N 5


Or on some RPM based distros like Fedora / RHEL / SUSE Enterprise Linux to use chkconfig command, e.g. list services:

~]# chkconfig –list

etworkManager  0:off   1:off   2:on    3:on    4:on    5:on    6:off
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
anamon          0:off   1:off   2:off   3:off   4:off   5:off   6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off

And to start stop the service into (default runlevel) or respective runlevel:


~]#  chkconfig httpd on

~]# chkconfig –list httpd
httpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off



~]# chkconfig service_name on –level runlevels


Debian / Ubuntu and other .deb based distributions with System V (which executes scripts without single order but one by one) are not having natively chkconfig but instead are famous for update-rc.d init script wrapper, here is few basic use  of it:

update-rc.d <service> defaults
update-rc.d <service> start 20 3 4 5
update-rc.d -f <service>  remove

Here defaults means default set boot runtime for system and numbers are just whether service is started or stopped for respective runlevels. To check what is your default one simply run /sbin/runlevel

Other useful tool to stop / start services and analyze what service is running and which not in real time (but without modifying boot time set for a service) – more universal nowadays is to use the service command.

root@noah:/home/hipo# service –status-all
 [ + ]  acpid
 [ – ]  alsa-utils
 [ – ]  anacron
 [ + ]  apache-htcacheclean
 [ – ]  apache2
 [ + ]  atd
 [ + ]  aumix

root@noah:/home/hipo# service cron restart/usr/sbin/service command is just a simple wrapper bash shell script that takes care about start / stop etc. operations of scripts found under /etc/init.d

For those who don't want to tamper with too much typing and manual configuration there is an all distribution system V compatible ncurses interface text itnerface sysv-rc-conf which could make your life easier on configuring services on non-systemd (old) Linux-es.

To install on Debian distros:

debian:~# apt-get install sysv-rc-conf

debian:~# sysv-rc-conf

SysV RC Conf desktop on GNU Linux using sysv-rc-conf systemV and systemd

2. SystemD basic use Start / stop check service and a little bit of information
for the novice

As most Linux kernel based distributions except some like Slackware and few others see the full list of Linux distributions without systemd (and aha yes slackw. users loves rc.local so much – we all do 🙂  migrated and are nowadays using actively SystemD, to start / stop analyze running system runnig services / processes

systemctl – Control the systemd system and service manager

To check whether a service is enabled

systemctl is-active application.service

To check whether a unit is in a failed state

systemctl is-failed application.service

To get a status of running application via systemctl messaging

# systemctl status sshd
● ssh.service – OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-07-06 20:01:02 EEST; 2h 3min ago Main PID: 1335 (sshd) Tasks: 1 (limit: 4915) CGroup: /system.slice/ssh.service └─1335 /usr/sbin/sshd -D юли 06 20:01:00 noah systemd[1]: Starting OpenBSD Secure Shell server… юли 06 20:01:02 noah sshd[1335]: Server listening on port 22. юли 06 20:01:02 noah sshd[1335]: Server listening on :: port 22. юли 06 20:01:02 noah systemd[1]: Started OpenBSD Secure Shell server.

To enable / disable application with systemctl systemctl enable application.service

systemctl disable application.service

To stop / start given application systemcl stop sshd

systemctl stop tor

To reload running application

systemctl reload sshd

Some applications does not have the right functionality in systemd script to reload configuration without fully restarting the app if this is the case use systemctl reload-or-restart application.service

systemctl list-unit-files

Then to view the content of a single service unit file:

:~# systemctl cat apache2.service
# /lib/systemd/system/apache2.service
Description=The Apache HTTP Server

ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl stop
ExecReload=/usr/sbin/apachectl graceful



systemd's advancement over normal SystemV services it is able to track and show dependencies
of a single run service for proper operation on other services

:~# systemctl list-dependencies sshd.service


● ├─system.slice
● └─
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─keyboard-setup.service
●   ├─kmod-static-nodes.service
●   ├─proc-sys-fs-binfmt_misc.automount
●   ├─sys-fs-fuse-connections.mount
●   ├─sys-kernel-config.mount
●   ├─sys-kernel-debug.mount
●   ├─systemd-ask-password-console.path
●   ├─systemd-binfmt.service



You can also mask / unmask service e.g. make it temporary unavailable via systemd with

sudo systemctl mask nginx.service

it will then appear as masked if you do list-unit-files

If you want to change something on a systemd unit file this is done with

systemctl edit –full nginx.service

In case if some modificatgion was done to systemd service files e.g. lets say to
/etc/systemd/system/apache2.service or even you've made a Linux system Upgrade recently
that added extra systemd service config files it will be necessery to reload all files
present in /etc/systemd/system/* with:

systemctl daemon-reload

Systemd has a target states which are pretty similar to the runlevel concept (e.g. runlevel 5 means graphical etc.), for example to check the default target for a system:

One very helpful feature is to restart systemd but it seems this is not well documented as of now and though this might work after some system package upgrade roll-outs it is always better to reboot the system, but you can give it a try if restart can't be done due to application criticallity.

To restart systemd and its spawned subprocesses do:

systemctl daemon-reexec


root@noah:/home/hipo# systemctl get-default

 to check all targets possible targets

root@noah:/home/hipo# systemctl list-unit-files –type=target
UNIT FILE                 STATE              static          static           static     static         static       disabled            static          static               disabled              static              static          static  

you can put the system in Single user mode if you like without running the good old well known command:

/sbin/init 1 

command with

systemctl rescue

You can even shutdown / poweroff / reboot system via systemctl (though I never did that and I don't recommend) 🙂
To do so use:

systemctl halt
systemctl poweroff
systemctl reboot

For the lazy ones that don't want to type all the time like crazy to configure and manage simple systemctl set services take a look at chkservice – an ncurses text based menu systemctl management interface

As chkservice is relatively new it is still not present in stable Stretch Debian repositories but it is in current testing Debian unstable Buster / Sid – Testing / Unstable distribution and has installable package for Ubuntu / Arch Linux and Fedora

Picture Source

chkservice linux help screen

3. Analyzing and fix performance boot slowness issues due to a service taking long to boot

The first very useful thing is to know how long exactly all daemons / services got booted
on your GNU / Linux OS.

linux-server:~# systemd-analyze 
Startup finished in 4.135s (kernel) + 3min 47.863s (userspace) = 3min 51.998s

As you can see it reports both the kernel boot time and userspace (surrounding services
that had to boot for the system to be considered fully booted).

Once you have the system properly booted you have a console or / ssh access

root@pcfreak:/home/hipo# systemd-analyze blame
    2min 14.172s tor@default.service
    1min 40.455s docker.service
     1min 3.649s fail2ban.service
         58.806s nmbd.service
         53.992s rc-local.service
         51.458s systemd-tmpfiles-setup.service
         50.495s mariadb.service
         46.348s snort.service
         34.910s ModemManager.service
         33.748s squid.service
         32.226s ejabberd.service
         28.207s certbot.service
         28.104s networking.service
         23.639s munin-node.service
         20.917s smbd.service
         20.261s tinyproxy.service
         19.981s accounts-daemon.service
         18.501s loadcpufreq.service
         16.756s stunnel4.service
         15.575s oidentd.service
         15.376s dev-sda1.device
         15.368s courier-authdaemon.service
         15.301s sysstat.service
         15.154s gpm.service
         13.276s systemd-logind.service
         13.251s rsyslog.service
         13.240s lpd.service
         13.237s pppd-dns.service
         12.904s NetworkManager-wait-online.service
         12.540s lm-sensors.service
         12.525s watchdog.service
         12.515s inetd.service

As you can see you get a list of services time took to boot in secs and you can
further debug each of it to find out why it boots so slow (netwok / DNS / configuration isssue whatever).

On a servers it is useful to look up for some processes slowing it down like gdm.service etc.


Close up words rant on SystemD vs SysemV


A lot could be ranted on what is better systemd or systemV. I personally hated systemd since day since I saw it being introduced first in Fedora / CentOS linuxes and a bit later in my beloved desktop used Debian Linux.
I still remember the bugs and headaches with systemd's intruduction as it is with all new the early adoption of technology makes a lot of pain in the ass.
Eventually systemd has become a standard and with my employment as a contractor through Itelligence GmBH for SAP AG I now am forced to work with systemd daily on SLES 12 based Linuces and I was forced to get used to it. 
But still there is my personal preference to SystemV even though the critics of slow boot etc.but for managing a multitude of Linux preinstalled servers like Virtual Machines and trying to standardize a Data Center with Tens of Thousands of Linuxes running on different Hypervisors VMWare / OpenXen + physical hosts etc. systemd brings a bit of more standardization that makes it a winner.

How to remove and disable BlueTooth support on Debian GNU / Linux servers

Thursday, October 18th, 2012

How to remove / disable bluetooth support on Debian GNU / Linux servers
If you running Debian Squeeze Linux (as server Apache, MySQL, Qmail etc.) on brand new purchased hardware with bluetooth support; you will notice default Linux kernel will detect and load modules for Bluetooth

This would not be a problem only if Bluetooth does not pose possible errors or (even at cases even maybe system hangs ups?). The actual reason in my case to want to disable bluetooth on a productive Linux server operating like host was I found out in dmesg produced output, some errors related to Bluetooth, here they are:

root@deb:~# dmesg|grep -i 'call trace' -A 8
[323406.744439] Call Trace:
[323406.744440] [] ? lapic_next_event+0x18/0x1d
[323406.744450] [] ? __report_bad_irq+0x30/0x7d
[323406.744453] [] ? note_interrupt+0x105/0x16e
[323406.744455] [] ? handle_fasteoi_irq+0x93/0xb5
[323406.744458] [] ? handle_irq+0x17/0x1d
[323406.744460] [] ? do_IRQ+0x57/0xb6
[323406.744463] [] ? ret_from_intr+0x0/0x11

I saw this error and similar ones occuring, every now and then obviously displaying something wents wrongs with IRQs related to BlueTooth Communication with Kernel (as it keeps processing requests loaded in system memory) …

Well anyways having the bluetooth kernel module loaded on memory just takes up few chunks of useless assigned memory.
I don't have intention to use bluetoothever in future on these host so I decided to completely remove bluetooth support on those Debian.

1. Remove blueetoh support on Debian GNU / Linux

First to check info about the loaded kernel module bluetooth.ko and its assigned module load alias run:

root@deb:~# /sbin/modinfo bluetooth
filename: /lib/modules/2.6.32-5-amd64/kernel/net/bluetooth/bluetooth.ko
alias: net-pf-31
license: GPL
version: 2.15
description: Bluetooth Core ver 2.15
author: Marcel Holtmann
srcversion: 9FD5BF98FC88505DC637909
depends: rfkill
vermagic: 2.6.32-5-amd64 SMP mod_unload modversions

Secondly disable memory preloaded bluetooth.ko on the current host with cmds:

root@deb:~# rmmod -f bnep
root@deb:~# rmmod -f l2cap
root@deb:~# rmmod -f sco
root@deb:~# rmmod -f bluetooth

Default way to control if Bluetooth (on host support is ON or OFF) is through /etc/default/bluetooth. Inside /etc/default/bluetooth is a control variable:


To shut it off change its value to 0:


Then to permanently prevent bluetooth.ko from being ever in future loaded its also good idea to blacklist modules – bnep, btusb, bluetooth:

root@deb:~# echo 'blacklist bnep' >> /etc/modprobe.d/bluetooth.conf
root@deb:~# echo 'blacklist btusb' >> /etc/modprobe.d/bluetooth.conf
root@deb:~# echo 'blacklist bluetooth' >> /etc/modprobe.d/bluetooth.conf

Onwards re-build, current kernel initramfs:

root@deb:~# update-initramfs -u -k `uname -r` -v

Next update boot init scripts with update-rc.d to make sure bluetooth (service / daemon) is not started:

root@deb:~# update-rc.d bluetooth remove

That's all bluetooth will not load up anymore on next boot and at present time will not take up useless mem space.

2. Re-enable disabled blueetooth on Debian Linux
I've been asked in one of comments, what to do If you need to re-enable bluetooth on your Debian Linux at some time in future, so here are the steps to turn back blueetooth on again


Change variable:




Open  /etc/modprobe.d/bluetooth.conf and remove any blacklisted modules, e.g:

'blacklist bnep'
'blacklist btusb'
&39;blacklist bluetooth'

Rebuild again kernel ramfs

root@deb:~# update-initramfs -u -k `uname -r` -v
Enjoy 🙂

List and get rid of obsolete program core dump files and completely disable core files on FreeBSD

Tuesday, November 1st, 2011

My FreeBSD router has started running out of space, I looked for ways to clean up some space. So I remembered some programs are generating core files while they crash. Some of these files are really huge and ban be from 1Mb to > 1G.

I used find to first list all my produced core files starting from root directory (/) , like so:

find / -name core -exec du -hsc {} ;

Having a list of my core files with the respective core file size and after reviewing, I deleted one by one the cores which were there just taking up space.
It’s a wise idea that core dumps file generation on program crash is completely disabled, however I forgot to disable cores, so I had plenty of the cores – (crash files which are handy for debug purposes and fixing the bug that caused the crash).

Further on I used an /etc/rc.confdumpdev=NO , variable which instructs the kernel to not generate core files on program crash:

freebsd# echo 'dumpdev=NO' >> /etc/rc.conf

Next, to make dumpdev=NO , take affect I rebooted the server:

freebsd# shutdown -r now

There is a way to instruct every server running daemon to know about the newly set dumpdev=NO by restarting each of the services with their init scripts individually, but I was too lazy to do that.

How to make GRE tunnel iptables port redirect on Linux

Saturday, August 20th, 2011

I’ve recently had to build a Linux server with some other servers behind the router with NAT.
One of the hosts behind the Linux router was running a Window GRE encrypted tunnel service. Which had to be accessed with the Internet ip address of the server.
In order < б>to make the GRE tunnel accessible, a bit more than just adding a normal POSTROUTING DNAT rule and iptables FORWARD is necessery.

As far as I’ve read online, there is quite of a confusion on the topic of how to properly configure the GRE tunnel accessibility on Linux , thus in this very quick tiny tutorial I’ll explain how I did it.

1. Load the ip_nat_pptp and ip_conntrack_pptp kernel module

linux-router:~# modprobe ip_nat_pptp
linux-router:~# modprobe ip_conntrack_pptp

These two modules are an absolutely necessery to be loaded before the remote GRE tunnel is able to be properly accessed, I’ve seen many people complaining online that they can’t make the GRE tunnel to work and I suppose in many of the cases the reason not to be succeed is omitting to load this two kernel modules.

2. Make the ip_nat_pptp and ip_nat_pptp modules to load on system boot time

linux-router:~# echo 'ip_nat_pptp' >> /etc/modules
linux-router:~# echo 'ip_conntrack_pptp' >> /etc/modules

3. Insert necessery iptables PREROUTING rules to make the GRE tunnel traffic flow

linux-router:~# /sbin/iptables -A PREROUTING -d -p tcp -m tcp --dport 1723 -j DNAT --to-destination
linux-router:~# /sbin/iptables -A PREROUTING -p gre -j DNAT --to-destination

In the above example rules its necessery to substitute the ip address withe the external internet (real IP) address of the router.

Also the IP address of is the internal IP address of the host where the GRE host tunnel is located.

Next it’s necessery to;

4. Add iptables rule to forward tcp/ip traffic to the GRE tunnel

linux-router:~# /sbin/iptables -A FORWARD -p gre -j ACCEPT

Finally it’s necessery to make the above iptable rules to be permanent by saving the current firewall with iptables-save or add them inside the script which loads the iptables firewall host rules.
Another possible way is to add them from /etc/rc.local , though this kind of way is not recommended as rules would add only after succesful bootup after all the rest of init scripts and stuff in /etc/rc.local is loaded without errors.

Afterwards access to the GRE tunnel to the local IP using the port 1723 and host IP is possible.
Hope this is helpful. Cheers 😉