Archive for the ‘Linux’ Category

What is inode and how to find out which directory is eating up all your filesystem inodes on Linux, Increase inode count on a ext3 ext4 and ufs filesystems

Tuesday, August 20th, 2019

what-is-inode-find-out-which-filesystem-or-directory-eating-up-all-your-system-inodes-linux_inode_diagram

If you're a system administrator of multiple Linux servers used for Web serving delivery / Mail server sysadmin, Database admin or any High amount of Drives Data Storage used for backup servers infra, Data Repository administrator such as Linux hosted Samba / CIFS shares, etc. or using some Linux Hosting Provider to host your website or any other UNIX like Infrastructure servers that demands a storage of high number of files under a Directory  you might end up with the common filesystem inode depletion issues ( Maximum Inode number for a filesystem is predefined, limited and depending on the filesystem configured size).

In case a directory stored files end up exceding the amount of possible addressable inodes could prevent any data to be further assiged and stored on the Filesystem.

When a device runs out of inodes, new files cannot be created on the device, even though there may be plenty free space available and the first time it happened to me very long time ago I was completely puzzled how this is possible as I was not aware of Inodes existence  …

Reaching maximum inodes number (e.g. inode depletion), often happens on Busy Mail servers (receivng tons of SPAM email messages) or Content Delivery Network (CDN – Website Image caching servers) which contain many small files on EXT3 or EXT4 Journalled filesystems. File systems (such as Btrfs, JFS or XFS) escape this limitation with extents or dynamic inode allocation, which can 'grow' the file system or increase the number of inodes.

 

Hence ending being out of inodes could cause various oddities on how stored data behaves or communicated to other connected microservices and could lead to random application disruptions and odd results costing you many hours of various debugging to find the root cause of inodes (index nodes) being out of order.

In below article, I will try to give an overall explanation on what is an I-Node on a filesystem, how inodes of FS unit could be seen, how to diagnose a possible inode poblem – e.g.  see the maximum amount of inodes available per filesystem and how to prepare (format) a new filesystem with incrsed set of maximum inodes.

 

What are filesystem i-nodes?

 

This is a data structure in a Unix-style file system that describes a file-system object such as a file or a directory.
The data structure described in the inodes might vary slightly depending on the filesystem but usually on EXT3 / EXT4 Linux filesystems each inode stores the index to block that contains attributes and disk block location(s) of the object's data.
– Yes for those who are not aware on how a filesystem is structured on *nix it does allocate all stored data in logical separeted structures called data blocks. Each file stored on a local filesystem has a file descriptor, there are virtual unit structures file tables and each of the inodes that are a reference number has a own data structure (inode table).

Inodes / "Index" are slightly unusual on file system structure that stored the access information of files as a flat array on the disk, with all the hierarchical directory information living aside from this as explained by Unix creator and pioneer- Dennis Ritchie (passed away few years ago).

what-is-inode-very-simplified-explanation-diagram-data

Simplified explanation on file descriptors, file table and inode, table on a common Linux filesystem

Here is another description on what is I-node, given by Ken Thompson (another Unix pioneer and father of Unix) and Denis Ritchie, described in their paper published in 1978:

"    As mentioned in Section 3.2 above, a directory entry contains only a name for the associated file and a pointer to the file itself. This pointer is an integer called the i-number (for index number) of the file. When the file is accessed, its i-number is used as an index into a system table (the i-list) stored in a known part of the device on which the directory resides. The entry found thereby (the file's i-node) contains the description of the file:…
    — The UNIX Time-Sharing System, The Bell System Technical Journal, 1978  "


 

What is typical content of inode and how I-nodes play with rest of Filesystem units?


The inode is just a reference index to a data block (unit) that contains File-system object attributes. It may include metadata information such as (times of last change, access, modification), as well as owner and permission data.

 

On a Linux / Unix filesystem, directories are lists of names assigned to inodes. A directory contains an entry for itself, its parent, and each of its children.

Structure-of-inode-table-on-Linux-Filesystem-diagram

 

Structure of inode table-on Linux Filesystem diagram (picture source GeeksForGeeks.org)

  • Information about files(data) are sometimes called metadata. So you can even say it in another way, "An inode is metadata of the data."
  •  Inode : Its a complex data-structure that contains all the necessary information to specify a file. It includes the memory layout of the file on disk, file permissions, access time, number of different links to the file etc.
  •  Global File table : It contains information that is global to the kernel e.g. the byte offset in the file where the user's next read/write will start and the access rights allowed to the opening process.
  • Process file descriptor table : maintained by the kernel, that in turn indexes into a system-wide table of files opened by all processes, called the file table .

The inode number indexes a table of inodes in a known location on the device. From the inode number, the kernel's file system driver can access the inode contents, including the location of the file – thus allowing access to the file.

  •     Inodes do not contain its hardlink names, only other file metadata.
  •     Unix directories are lists of association structures, each of which contains one filename and one inode number.
  •     The file system driver must search a directory looking for a particular filename and then convert the filename to the correct corresponding inode number.

The operating system kernel's in-memory representation of this data is called struct inode in Linux. Systems derived from BSD use the term vnode, with the v of vnode referring to the kernel's virtual file system layer.


But enough technical specifics, lets get into some practical experience on managing Filesystem inodes.
 

Listing inodes on a Fileystem


Lets say we wan to to list an inode number reference ID for the Linux kernel (files):

 

root@linux: # ls -i /boot/vmlinuz-*
 3055760 /boot/vmlinuz-3.2.0-4-amd64   26091901 /boot/vmlinuz-4.9.0-7-amd64
 3055719 /boot/vmlinuz-4.19.0-5-amd64  26095807 /boot/vmlinuz-4.9.0-8-amd64


To list an inode of all files in the kernel specific boot directory /boot:

 

root@linux: # ls -id /boot/
26091521 /boot/


Listing inodes for all files stored in a directory is also done by adding the -i ls command flag:

Note the the '-1' flag was added to to show files in 1 column without info for ownership permissions

 

root@linux:/# ls -1i /boot/
26091782 config-3.2.0-4-amd64
 3055716 config-4.19.0-5-amd64
26091900 config-4.9.0-7-amd64
26095806 config-4.9.0-8-amd64
26091525 grub/
 3055848 initrd.img-3.2.0-4-amd64
 3055644 initrd.img-4.19.0-5-amd64
26091902 initrd.img-4.9.0-7-amd64
 3055657 initrd.img-4.9.0-8-amd64
26091756 System.map-3.2.0-4-amd64
 3055703 System.map-4.19.0-5-amd64
26091899 System.map-4.9.0-7-amd64
26095805 System.map-4.9.0-8-amd64
 3055760 vmlinuz-3.2.0-4-amd64
 3055719 vmlinuz-4.19.0-5-amd64
26091901 vmlinuz-4.9.0-7-amd64
26095807 vmlinuz-4.9.0-8-amd64

 

To get more information about Linux directory, file, such as blocks used by file-unit, Last Access, Modify and Change times, current External Symbolic or Static links for filesystem object:
 

root@linux:/ # stat /etc/
  File: /etc/
  Size: 16384         Blocks: 32         IO Block: 4096   catalog
Device: 801h/2049d    Inode: 6365185     Links: 231
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-08-20 06:29:39.946498435 +0300
Modify: 2019-08-14 13:53:51.382564330 +0300
Change: 2019-08-14 13:53:51.382564330 +0300
 Birth: –

 

Within a POSIX system (Linux-es) and *BSD are more or less such, a file has the following attributes[9] which may be retrieved by the stat system call:

   – Device ID (this identifies the device containing the file; that is, the scope of uniqueness of the serial number).
    File serial numbers.
    – The file mode which determines the file type and how the file's owner, its group, and others can access the file.
    – A link count telling how many hard links point to the inode.
    – The User ID of the file's owner.
    – The Group ID of the file.
    – The device ID of the file if it is a device file.
    – The size of the file in bytes.
    – Timestamps telling when the inode itself was last modified (ctime, inode change time), the file content last modified (mtime, modification time), and last accessed (atime, access time).
    – The preferred I/O block size.
    – The number of blocks allocated to this file.

 

Getting more extensive information on a mounted filesystem


Most Linuxes have the tune2fs installed by default (in debian Linux this is through e2fsprogs) package, with it one can get a very good indepth information on a mounted filesystem, lets say about the ( / ) root FS.
 

root@linux:~# tune2fs -l /dev/sda1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          abe6f5b9-42cb-48b6-ae0a-5dda350bc322
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              30162944
Block count:              120648960
Reserved block count:     6032448
Free blocks:              13830683
Free inodes:              26575654
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      995
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Filesystem created:       Thu Sep  6 21:44:22 2012
Last mount time:          Sat Jul 20 11:33:38 2019
Last write time:          Sat Jul 20 11:33:28 2019
Mount count:              6
Maximum mount count:      22
Last checked:             Fri May 10 18:32:27 2019
Check interval:           15552000 (6 months)
Next check after:         Wed Nov  6 17:32:27 2019
Lifetime writes:          338 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:              256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       21554129
Default directory hash:   half_md4
Directory Hash Seed:      d54c5a90-bc2d-4e22-8889-568d3fd8d54f
Journal backup:           inode blocks


Important note to make here is file's inode number stays the same when it is moved to another directory on the same device, or when the disk is defragmented which may change its physical location. This also implies that completely conforming inode behavior is impossible to implement with many non-Unix file systems, such as FAT and its descendants, which don't have a way of storing this invariance when both a file's directory entry and its data are moved around. Also one inode could point to a file and a copy of the file or even a file and a symlink could point to the same inode, below is example:

$ ls -l -i /usr/bin/perl*
266327 -rwxr-xr-x 2 root root 10376 Mar 18  2013 /usr/bin/perl
266327 -rwxr-xr-x 2 root root 10376 Mar 18  2013 /usr/bin/perl5.14.2

A good to know is inodes are always unique values, so you can't have the same inode number duplicated. If a directory is damaged, only the names of the things are lost and the inodes become the so called “orphan”, e.g.  inodes without names but luckily this is recoverable. As the theory behind inodes is quite complicated and is complicated to explain here, I warmly recommend you read Ian Dallen's Unix / Linux / Filesystems – directories inodes hardlinks tutorial – which is among the best academic Tutorials explaining various specifics about inodes online.

 

How to Get inodes per mounted filesystem

 

root@linux:/home/hipo# df -i
Filesystem       Inodes  IUsed   IFree IUse% Mounted on

 

dev             2041439     481   2040958   1% /dev
tmpfs            2046359     976   2045383   1% /run
tmpfs            2046359       4   2046355   1% /dev/shm
tmpfs            2046359       6   2046353   1% /run/lock
tmpfs            2046359      17   2046342   1% /sys/fs/cgroup
/dev/sdb5        1221600    2562   1219038   1% /usr/var/lib/mysql
/dev/sdb6        6111232  747460   5363772  13% /var/www/htdocs
/dev/sdc1      122093568 3083005 119010563   3% /mnt/backups
tmpfs            2046359      13   2046346   1% /run/user/1000


As you see in above output Inodes reported for each of mounted filesystems has a specific number. In above output IFree on every mounted FS locally on Physical installed OS Linux is good.


Here is an example on how to recognize a depleted Inodes on a OpenXen Virtual Machine with attached Virtual Hard disks.

linux:~# df -i
Filesystem         Inodes     IUsed      IFree     IUse%   Mounted on
/dev/xvda         2080768    2080768     0      100%    /
tmpfs             92187      3          92184   1%     /lib/init/rw
varrun            92187      38          92149   1%    /var/run
varlock            92187      4          92183   1%    /var/lock
udev              92187     4404        87783   5%    /dev
tmpfs             92187       1         92186   1%    /dev/shm

 

Finding files with a certain inode


At some cases if you want to check all the copy files of a certain file that have the same i-node pointer it is useful to find them all by their shared inode this is possible with simple find (below example is for /usr/bin/perl binary sharing same inode as perl5.28.1:

 

ls -i /usr/bin/perl
23798851 /usr/bin/perl*

 

 find /usr/bin -inum 435308 -print
/usr/bin/perl5.28.1
/usr/bin/perl

 

Find directory that has a large number of files in it?

To get an overall number of inodes allocated by a certain directory, lets say /usr /var

 

root@linux:/var# du -s –inodes /usr /var
566931    /usr
56020    /var/

To get a list of directories use by inode for a directory with its main contained sub-directories sorted from 1 till highest number use:
 

du -s –inodes * 2>/dev/null |sort -g

 

Usually running out of inodes means there is a directory / fs mounts that has too many (small files) that are depleting the max count of possible inodes.

The most simple way to list directories and number of files in them on the server root directory is with a small bash shell loop like so:
 

for i in /*; do echo $i; find $i |wc -l; done


Another way to identify the exact directory that is most likely the bottleneck for the inode depletion in a sorted by file count, human readable form:
 

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n


This will dump a list of every directory on the root (/) filesystem prefixed with the number of files (and subdirectories) in that directory. Thus the directory with the largest number of files will be at the bottom.

 

The -xdev switch is used to instruct find to narrow it's search to only the device where you're initiating the search (any other sub-mounted NAS / NFS filesystems from a different device will be omited).

 

Print top 10 subdirectories with Highest Inode Usage

 

Once identifed the largest number of files directories that is perhaps the issue, to further get a list of Top subdirectories in it with highest amount of inodes used, use below cmd:

 

for i in `ls -1A`; do echo "`find $i | sort -u | wc -l` $i"; done | sort -rn | head -10

 

To list more than 10 of the top inodes used dirs change the head -10 to whatever num needed.

N.B. ! Be very cautious when running above 2 find commands on a very large filesystems as it will be I/O Excessive and in filesystems that has some failing blocks this could create further problems.

To omit putting a high I/O load on a production filesystem, it is possible to also use du + very complex regular expression:
 

cd /backup
du –inodes -S | sort -rh | sed -n         '1,50{/^.\{71\}/s/^\(.\{30\}\).*\(.\{37\}\)$/\1…\2/;p}'


Results returned are from top to bottom.

 

How to Increase the amount of Inodes count on a new created volume EXT4 filesystem

Some FS-es XFS, JFS do have an auto-increase inode feature in case if their is physical space, whether otheres such as reiserfs does not have inodes at all but still have a field reported when queried for errors. But the classical Linux ext3 / ext4 does not have a way to increase the inode number on a live filesystem. Instead the way to do it there is to prepare a brand new filesystem on a Disk / NAS / attached storage.

The number of inodes at format-time of the block storage can be as high as 4 billion inodes. Before you create the new FS, you have to partition the new the block storage as ext4 with lets say parted command (or nullify the content of an with dd to clean up any previous existing data on a volume if there was already existing data:
 

parted /dev/sda


dd if=/dev/zero of=/dev/path/to/volume


  then format it with this additional parameter:

 

mkfs.ext4 -N 3000000000 /dev/path/to/volume

 

Here in above example the newly created filesystem of EXT4 type will be created with 3 Billion inodes !, for setting a higher number on older ext3 filesystem max inode count mkfs.ext3 could be used instead.

Bear in mind that 3 Billion number is a too high number and if you plan to have some large number of files / directories / links structures just raise it up to your pre-planning requirements for FS. In most cases it will be rarely anyone that want to have this number higher than 1 or 2 billion of inodes.

On FreeBSD / NetBSD / OpenBSD setting inode maximum number for a UFS / UFS2 (which is current default FreeBSD FS), this could be done via newfs filesystem creation command after the disk has been labeled with disklabel:

 

freebsd# newfs -i 1024 /dev/ada0s1d

 

Increase the Max Count of Inodes for a /tmp filesystem

 

Sometimes on some machines it is necessery to have ability to store very high number of small files (e.g. have a very large number of inodes) on a temporary filesystem kept in memory. For example some web applications served by Web Server Apache + PHP, Nginx + Perl-FastCGI are written in a bad manner so they kept tons of temporary files in /tmp, leading to issues with exceeded amount of inodes.
If that's the case to temporary work around you can increase the count of Inodes for /tmp to a very high number like 2 billions using:

 

mount -o remount,nr_inodes=<bignum> /tmp

To make the change permanent on next boot if needed don't forget to put the nr_inodes=whatever_bignum as a mount option for the temporary fs to /etc/fstab

Eventually, if you face this issues it is best to immediately track which application produced the mess and ask the developer to fix his messed up programs architecture.

 

Conclusion

 

It was explained on the very common issue of having maximum amount of inodes on a filesystem depleted and the unpleasent consequences of inability to create new files on living FS.
Then a general overview was given on what is inode on a Linux / Unix filesystem, what is typical content of inode, how inode addressing is handled on a FS. Further was explained how to get basic information about available inodes on a filesystem, how to get a filename/s based on inode number (with find), the well known way to determine inode number of a directory or file (with ls) and get more extensive information on a FS on inodes with tune2fs.
Also was explained how to identify directories containing multitudes of files in order to determine a sub-directories that is consuming most of the inodes on a filesystem. Finally it was explained very raughly how to prepare an ext4 filesystem from scratch with predefined number to inodes to much higher than the usual defaults by mkfs.ext3 / mkfs.ext4 and *bsds newfs as well as how to raise the number of inodes of /tmp tmpfs temporary RAM filesystem.

Howto debug and remount NFS hangled filesystem on Linux

Monday, August 12th, 2019

nfsnetwork-file-system-architecture-diagram

If you're using actively NFS remote storage attached to your Linux server it is very useful to get the number of dropped NFS connections and in that way to assure you don't have a remote NFS server issues or Network connectivity drops out due to broken network switch a Cisco hub or other network hop device that is routing the traffic from Source Host (SRC) to Destination Host (DST) thus, at perfect case if NFS storage and mounted Linux Network filesystem should be at (0) zero dropped connectios or their number should be low. Firewall connectivity between Source NFS client host and Destination NFS Server and mount should be there (set up fine) as well as proper permissions assigned on the server, as well as the DST NFS should be not experiencing I/O overheads as well as no DNS issues should be present (if NFS is not accessed directly via IP address).
In below article which is mostly for NFS novice admins is described shortly few of the nuances of working with NFS.
 

1. Check nfsstat and portmap for issues

One indicator that everything is fine with a configured NFS mount is the number of dropped NFS connections
or with a very low count of dropped connections, to check them if you happen to administer NFS

nfsstat

 

linux:~# nfsstat -o net
Server packet stats:
packets    udp        tcp        tcpconn
0          0          0          0  


nfsstat is useful if you have to debug why occasionally NFS mounts are getting unresponsive.

As NFS is so dependent upon portmap service for mapping the ports, one other point to check in case of Hanged NFSes is the portmap service whether it did not crashed due to some reason.

 

linux:~# service portmap status
portmap (pid 7428) is running…   [portmap service is started.]

 

linux:~# ps axu|grep -i rpcbind
_rpc       421  0.0  0.0   6824  3568 ?        Ss   10:30   0:00 /sbin/rpcbind -f -w


A useful commands to debug further rcp caused issues are:

On client side:

 

rpcdebug -m nfs -c

 

On server side:

 

rpcdebug -m nfsd -c

 

It might be also useful to check whether remote NFS permissions did not changed with the good old showmount cmd

linux:~# showmount -e rem_nfs_server_host


Also it is useful to check whether /etc/exports file was not modified somehow and whether the NFS did not hanged due to attempt of NFS daemon to reload the new configuration from there, another file to check while debugging is /etc/nfs.conf – are there group / permissions issues as well as the usual /var/log/messages and the kernel log with dmesg command for weird produced NFS client / server or network messages.

nfs-utils disabled serving NFS over UDP in version 2.2.1. Arch core updated to 2.3.1 on 21 Dec 2017 (skipping over 2.2.1.) If UDP stopped working then, add udp=y under [nfsd] in /etc/nfs.conf. Then restart nfs-server.service.

If the remote NFS server is running also Linux it is useful to check its /etc/default/nfs-kernel-server configuration

At some stall cases it might be also useful to remount the NFS (but as there might be a process on the Linux server) trying to read / write data from the remote NFS mounted FS it is a good idea to check (whether a process / service) on the server is not doing I/O operations on the NFS and if such is existing to kill the process in question with fuser
 

linux:~# fuser -k [mounted-filesystem]
 

 

2. Diagnose the problem interactively with htop


    Htop should be your first port of call. The most obvious symptom will be a maxed-out CPU.
    Press F2, and under "Display options", enable "Detailed CPU time". Press F1 for an explanation of the colours used in the CPU bars. In particular, is the CPU spending most of its time responding to IRQs, or in Wait-IO (wio)?
 

3. Get more extensive Mount info with mountstats

 

nfs-utils package contains mountstats command which is very useful in debugging further the issues identified

$ mountstats
Stats for example:/tank mounted on /tank:
  NFS mount options: rw,sync,vers=4.2,rsize=524288,wsize=524288,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,proto=tcp,port=0,timeo=15,retrans=2,sec=sys,clientaddr=xx.yy.zz.tt,local_lock=none
  NFS server capabilities: caps=0xfbffdf,wtmult=512,dtsize=32768,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=notconfigured
  NFS security flavor: 1  pseudoflavor: 0

 

NFS byte counts:
  applications read 248542089 bytes via read(2)
  applications wrote 0 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 171375125 bytes via NFS READ
  client wrote 0 bytes via NFS WRITE

RPC statistics:
  699 RPC requests sent, 699 RPC replies received (0 XIDs not found)
  average backlog queue length: 0

READ:
    338 ops (48%)
    avg bytes sent per op: 216    avg bytes received per op: 507131
    backlog wait: 0.005917     RTT: 548.736686     total execute time: 548.775148 (milliseconds)
GETATTR:
    115 ops (16%)
    avg bytes sent per op: 199    avg bytes received per op: 240
    backlog wait: 0.008696     RTT: 15.756522     total execute time: 15.843478 (milliseconds)
ACCESS:
    93 ops (13%)
    avg bytes sent per op: 203    avg bytes received per op: 168
    backlog wait: 0.010753     RTT: 2.967742     total execute time: 3.032258 (milliseconds)
LOOKUP:
    32 ops (4%)
    avg bytes sent per op: 220    avg bytes received per op: 274
    backlog wait: 0.000000     RTT: 3.906250     total execute time: 3.968750 (milliseconds)
OPEN_NOATTR:
    25 ops (3%)
    avg bytes sent per op: 268    avg bytes received per op: 350
    backlog wait: 0.000000     RTT: 2.320000     total execute time: 2.360000 (milliseconds)
CLOSE:
    24 ops (3%)
    avg bytes sent per op: 224    avg bytes received per op: 176
    backlog wait: 0.000000     RTT: 30.250000     total execute time: 30.291667 (milliseconds)
DELEGRETURN:
    23 ops (3%)
    avg bytes sent per op: 220    avg bytes received per op: 160
    backlog wait: 0.000000     RTT: 6.782609     total execute time: 6.826087 (milliseconds)
READDIR:
    4 ops (0%)
    avg bytes sent per op: 224    avg bytes received per op: 14372
    backlog wait: 0.000000     RTT: 198.000000     total execute time: 198.250000 (milliseconds)
SERVER_CAPS:
    2 ops (0%)
    avg bytes sent per op: 172    avg bytes received per op: 164
    backlog wait: 0.000000     RTT: 1.500000     total execute time: 1.500000 (milliseconds)
FSINFO:
    1 ops (0%)
    avg bytes sent per op: 172    avg bytes received per op: 164
    backlog wait: 0.000000     RTT: 2.000000     total execute time: 2.000000 (milliseconds)
PATHCONF:
    1 ops (0%)
    avg bytes sent per op: 164    avg bytes received per op: 116
    backlog wait: 0.000000     RTT: 1.000000     total execute time: 1.000000 (milliseconds)


nfs-utils disabled serving NFS over UDP in version 2.2.1. Arch core updated to 2.3.1 on 21 Dec 2017 (skipping over 2.2.1.) If UDP stopped working then, add udp=y under [nfsd] in /etc/nfs.conf. Then restart nfs-server.service.
 

4. Check for firewall issues
 

If all fails make sure you don't have any kind of firewall issues. Sometimes firewall changes on remote server or somewhere in the routing servers might lead to stalled NFS mounts.

 

To use properly NFS as you should know as a minimum you need to have opened as ports is Port 111 (TCP and UDP) and 2049 (TCP and UDP) on the NFS server (side) as well as any traffic inspection routers on the road from SRC (Linux client host) and NFS Storage destination DST server.

There are also ports for Cluster and client status (Port 1110 TCP for the former, and 1110 UDP for the latter) as well as a port for the NFS lock manager (Port 4045 TCP and UDP) but having this opened or not depends on how the NFS is configured. You can further determine which ports you need to allow depending on which services are needed cross-gateway.
 

5. How to Remount a Stalled unresponsive NFS filesystem mount

 

At many cases situation with remounting stalled NFS filesystem is not so easy but if you're lucky a standard mount and remount should do the trick.

Most simple way to remout the NFS (once you're sure this might not disrupt any service) – don't blame me if you break something is with:
 

umount -l /mnt/NFS_mnt_point
mount /mnt/NFS_mnt_point


Note that the lazy mount (-l) umount opt is provided here as very often this is the only way to unmount a stalled NFS mount.

Sometimes if you have a lot of NFS mounts and all are inacessible it is useful to remount all NFS mounts, if the remote NFS is responsive this should be possible with a simple for bash loop:

for P in $(mount | awk '/type nfs / {print $3;}'); do echo $P; echo "sudo umount $P && sudo mount $P" && echo "ok :)"; done


If you cd /mnt/NFS_mnt_point and try ls and you get

$ ls
.: Stale File Handle

 

You will need to unmount the FS with forceful mount flag

umount -f /mnt/NFS_mnt_point
 

Sum it up


In this article, I've shown you a few simple ways to debug what is wrong with a Stalled / Hanged NFS filesystem present on a NFS server mounted on a Linux client server.
Above was explained the common issues caused by NFS portmap (rpcbind) dependency, how to its status is fine, some further diagnosis with htop and mountstat was pointed. I've pointed the minimum amount of TCP / UDP ports 2049 and 111 that needs to be opened for the NFS communication to work and finally explained on how to remount a stalled NFS single or all attached mount on a NFS client to restore to normal operations.
As NFS is a whole ocean of things and the number of ways it is used are too extensive this article is just a general info useful for the NFS dummy admin for more robust configs read some good book on NFS such as Managing NFS and NIS, 2nd Edition – O'Reilly Media and for Kernel related NFS debugging make sure you check as a minimum ArchLinux's NFS troubleshooting guide and sourceforge's NFS Troubleshoting and Optimizing NFS Performance guides.

 

How to View and Delete NetApp Storage qtree, Get statistics about Filer Volume Read / Writes operations and delete and show mounted volumes

Friday, August 2nd, 2019

how-to-delete-volume-qtree-snapmirror-view-netapp-volume-qtree-and-and-view-netapp-cluster-device-statistics-NetAppLogo

I've had recently the trivial decomissioning task to delete some NetApp Storage qtrees on some of the SAP Hana Enterprise Cloud NetApp filers.
If it is first time you heard of NetApp is a hybrid cloud data services and data management (ranked in the Fortune 500 companies).

Netapps are hybrid cloud data services for management of applications and data across cloud and on-premises environments and are a de-facto standard for Data storage on many of the existing Internet Clouds and Large Corporatons that Stores many Pentabytes of Data.

The Netapp storage devices are a kinda of proprietary Clustered version of the Small business NAS storage Solution FreeNAS (which of itself is a Free FreeBSD based Data Storage OS – The #1 Storage OS).
NetApps allow plenty of things to do such as Data Mirroring (Data Backups), Data Syncing, SpapMirroring, SnapVault and many, many more custom Data revolutionary solutions such as StorageGrid.

NetApp supports integration with Kubernetes, Docker, Oracle / SAP DB, Citrix, Xen, KVM as well as multiple cloud environments such as AWS, Azure, OpenStack and has even integration with some CI/CD DevOps data provisioning – i.e. Jenkins.

In this small article, I'll show you how a Volume / Qtree on a NetApp filer could be viewed, mounted, unmounted, deleted. I'll also show you how to get statistics, while logged in remotely to the NetApp console and finally how to view and delete a NetApp configured snapmirror.

 

View NetApp Qtree

 

Here is how to view the Storage Qtree:

netapp> qtree show -vserver netapp01fv018 -volume VOL_OS_MIG -qtree bck_01v046485_20190108

To view the file content existing on the Storage server from the Linux bost next step to do is mount it with regular mount:

linux-host:~# mount netappfiler01fv018:/VOL_OS_MIG/ bck_01v035527_20190108 /mnt/test

 

Delete the Qtree from NetApp (Storage) Filer

Become administrator on the device

Once assured the content could go on to delete the qtree, it is necessery to become superuser (root) on the NetApp device, to do so, I hed to type:

 

netapp> set -privilege advanced

 

Then to delete the unneded volume previously used for transferring system update files, when logged in via SSH to the NetApp device – ONTAP Proprietary Operating system :

 

netapp> qtree delete -vserver netapp01fv018 -volume VOL_OS_MIG -force -qtree bck_01v035527_20190108


Note that this command will return back a job ID
assigned until operation is completed, to check the status of completion of generated JOB that is backgrounded, I've used command:

netapp> job show 53412

If all is okay you should get a Status of Success otherwise, if you get failed status you have to debug further what's causing it.
 

How to view existing export polcities and remove them

 

If you don't want to delete the qtree or volume but want to prevent a certain Linux server / application to not have access to it, it is useful to view existing export policy for a qtree.
 

netappfc001::> qtree show -exports -volume vol1_vmspace_netapp01v000885 -qtree q_01v002131
                                                   Is Export
Vserver    Volume        Qtree        Policy Name  Policy Inherited
———- ————- ———— ———— —————–
netapp01fv001 vol1_vmspace_netapp01v000885
                         q_01v002131
                                      vol1_vmspace_netapp01v000885.exports
                                                   true

 


To remove then export policy (to not exist at all), this is how:

 

 

netapp> volume qtree modify -vserver hec01fv018 -qtree-path /vol/volume_name/qtree_name -export-policy ""

 

I've also found the following volume qtree commands NetApp ONTAP documentation page helpful to read and recommend to anyone that wants to learn more.
 

How to delete a NetApp Volume if it is not used anymore

To delete unsed netapp volume, you have to do 3 things.
1. Unmount the volume
2. Put it offline
3. Delete it

to do so run below 3 cmds:

 

netapp> volume unmount -vserver vserver_name -volume volume_name
netapp> volume offline -vserver vserver_name volume_name
netapp> volume delete -vserver vserver_name volume_name

 

Show mounted Volume junctions (Get Extra Storage Volume information)

 

netapp> volume show -vserver netapp01fv004 -junction
netapp> volume show -vserver netapp01fv004 -volume MUFCF01_BACKUP

 

How to delete a Configured SnapMirror

What is a snapmirror?

 

Recovery-Scenario-Restore-Changes-To-Recovery-site-snapmirror-diagram

SnapMirror is a feature of Data ONTAP that enables you to replicate data. SnapMirror enables you to replicate data from specified source volumes or qtrees to specified destination volumes or qtrees, respectively. You need a separate license to use SnapMirror.

You can use SnapMirror to replicate data within the same storage system or with different storage systems.

After the data is replicated to the destination storage system, you can access the data on the destination to perform the following actions:

  • You can provide users immediate access to mirrored data in case the source goes down.
  • You can restore the data to the source to recover from disaster, data corruption (qtrees only), or user error.
  • You can archive the data to tape.
  • You can balance resource loads.
  • You can back up or distribute the data to remote sites.

 

netapp> snapmirror show -destination-path netapp02fv001:vol1_MUF_PS1_DR

 

netapp> snapmirror delete -destination-path netapp02fv001:vol1_MUF_PS1_DR -force
Operation succeeded: snapmirror delete for the relationship with destination "hec02fv001:vol1_MUF_PS1_DR".
 

If the snapmirror deletion gets scheduled you can use snapmirror status command to check status:
 

netapp> snapmirror status MUF_PS1_PRD
Snapmirror is on.

 

How to telnet from NetApp Storage to another one / check status of configured SMTPs for NetApp Cluster (filer)

 

 

You can use the autosupport and options autosupport commands to change or view AutoSupport configuration, display information about past AutoSupport messages, and send or resend an AutoSupport message.

For example if NetApp Filers have configured SMTP or SMTPs servers or other Proxy Configurations to pass on traffic from DMZ-ed network to external Internet resources or Relay servers this command will provide information on the Connection status of this remote services.

 

rows 0
set diag
node show

autosupport check show
systemshell -node netapp01f0018 -c telnet
autosupport show -fields proxy-url
systemshell -node netappf0018 -c telnet  147.204.148.38 80

netapp09fc001::*> systemshell -node  netapp08f0013 -c telnet  8080
  (system node systemshell)
Trying 100.127.20.4


node show – will provide information about configured nodes
rows 0 – will set the output print rows how they will be displayed
set – diag sets the device in diagnostic state

As you can see you can use the systemshell netapp command to try out telnet connections from the Configured NetApp logged in Source to any remote destination to make sure the set Proxy or SMTP is properly reachable.

How to get Statistics about NetApp existing volume Read / Write operations

 

On Netapp side issue:

netapp> statistics volume show -interval 5 -iterations 1 -max 25 -vserver netapp01fv004 -volume MUFCF01_BACKUP

For people starting up with NetApps, it is very useful to get a in-depth read on quick and dirty –  Netapp Commandline CheatSheet (for simplicity I've stored it in netapp-commands-cheatsheat.txt formatted file here ).

Conclusion

NetApp storages are used in many Governments and Large Corporations and for critical applications with SLAs forfeits for million bucks, mostly for applications and Database storage that are of a very large scale and too critical to be handled by the conventional storage computing of simple RAIDS 1,2,3,5,6 etc. / LVM and so on. ONTAP and NetApp Filers and Filer Clusters, are easy to maintain but due to its high number of features, not many NetApp Storage / Backup system administrators have the knowledge how to take a good advantage of this beasts. Thus finally, my even small experience with them shows that even simple things as critical errors are not handled properly at least that was my experience as a SAP consultant with SAP Hana Enterprise Cloud (HEC) and their HANA Converged Cloud where, main storage. 
This article's goal was pretty simple to guide the user on a minimum set of commands for simple qtree / volume / snapmirror view and removal decomissioning tasks. NetApps Clusters are a whole ocean of stuff and knowledge so before doing anything complex, if you're not sure what you're doing always consult a NetApp storage sysadmin as some of this animals features looks easy for the common general sysadmin but not are not so.
 

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness

Wednesday, July 24th, 2019

linux-why-du-and-df-shows-different-result-inconsincy-explained-filesystem-full-oddity

If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reports its file use different when checked with
df
cmd than when checked with du command, like for example:
 

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total

 

1. Simple debug on what might be the root cause for df / du inconsistency reporting

 

Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:

 

# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do

 

 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less

 

I've also investigated on the complete /var directory contents sorted by size with:

 

 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache


Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:
 

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql

 

So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).
 

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:
 

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www


FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:
 

# netstat -tupanl


Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files

 

# lsof -nP | grep '(deleted)'

COMMAND   PID   USER   FD   TYPE DEVICE    SIZE NLINK  NODE NAME
mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)

 

Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:

 

# lsof -nP | grep '(deleted)' |wc -l
438


As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):

 

# lsof +L1|less


The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.
 

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:

 

– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/libnss_files-2.28.so

 

# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/libm-2.28.so

Other command that helped to track the discrepancy between df and du different file usage on FS is:
 

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'
 

 

3. Fixing large files kept in memory filesystem problem


What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:

 

 

# systemctl restart mysqld.service


or on older init.d system V ones:

# /etc/init.d/service restart


For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l script.sh
7977 script.sh


# kill -HUP PID

 

Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.

 

# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total

 

What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?
 

How to make Samba smbfs / cifs mount share location with user / pass credentials authenticate via file stored credentials

Friday, July 19th, 2019

how-to-use-username-and-password-to-authenticate-to-samba-share-server-or-linux-share-server-linux-samba-logo
That's pretty trivial and perhaps if you had to manage samba server or cifs on a Linux host you already know it but for beginners, that might be interesting.

So in this short article I will explain how to make configure smbfs / cifs authentication from Linux host A client to Linux host B server running smbd and nmbd samba server (which is the smfs / cifs share server) by using external authentication file for either mount command or if /etc/fstab used to automatically authenticate using a preconfigured mount saba share via /etc/fstab.

Before you start to do anything with samba on Linux host A client machine, you will need as a minimum to have installed cifs-utils or smbfs (assuming you're on Debian Linux like you can check with dpkg -l and if missing install it via:

 

 

apt-get install cifs-utils

 

Or on older systems or for smbfs support

 

apt-get install smbfs

 

The general mount smbfs share command without specified external credentials file would look like so:

 

mount //mynetworksharename/ /shares/data -o username=myusername, password=mypassword


So how to use external auth file to prevent samba shares  users and passwords to not be stored in root user history all the time?

To do so it is pretty straight forward all you need to do is to create a single user / pass credentials variable defined lets say to file called .smbcredentials or .cifs under some directory lets /root/.smbcredentials.

One note here is (many people prefer to store the password under /root) for security reasons as root directory is usually readable only by administrator and would prevent a non-privileged user to read the user / pass which are stored in plain text.

.smbcredentials is described in mount.cifs man page, here is what it says about credentials variable understood by mount / mount.cifs command  file syntax:
 

 

credentials=filename
    specifies a file that contains a username and/or password. The format of the file is:

         username=value
         password=value


For a CIFS (Common Internet File System) which is a new implementation of old Windows Share (SMB protocol) avaiable in newer Windows XP / 7 / 10 machines, to do the cifs mount manually:
 

mount -v -t cifs //WINSHARESERVER/topsecretfiles /mnt/network/ -o credentials=/mnt/creds-file

or use 

 

mount.cifs //WINSSHARE/topsecretfiles /mnt/network/ -o credentials=/root/.creds-file

 

For old smbfs protocol for backward compatibility so older Win 2000 or Winblows server XP PCs configured to also access the Linux samba mount.

mount -t smbfs //WINHARESERVER/topsecretfiles /mnt/network/ -o credentials=/mnt/.smbcredentials


Once you have the defined .smbcredentials file name, be sure to also protect it with properly set permissions like 0600 (rw) readable only for root user. 

chmod 0600 /root/.smbcredentials

Note that in that example .smbcredentials is set to be a hidden file on purpose as this is a hidden file it will make it slightly less seenable if introduder breaks on the server (an example of security through obscurity)

 

Next lets see how to mount the Windows Samba Share permanently with predefined user / pass server login

For many non secured Windows shares one can use /etc/fstab line definition as simple as:
 

//server-share-name/sharename  /mnt/shares/sharename  cifs  guest,uid=1000,iocharset=utf8  0


For password protected Win Share mounts however, the simplest way to do is via /etc/fstab line add like so:

 

 

 

//servername/sharename  /mnt/shares/sharename  cifs  username=msusername,password=mspassword,iocharset=utf8,sec=ntlm  0  0


Note that the sec=ntlm is optional and remote samba server or Windows Share server version has to support this kind of authentication and in some cases you could safely reove sec=ntlm, just use it, when you know what you're doing. iocharset is good to have as for Russian / Bulgarian e.g.  Cyrillic, Chineese, Indian and other exotic languages and other strange language encoding to be supported and properly shown on the mounted share it should be properly defined …, 

A good permissions would be:

chmod 600 ~/.smbcredentials

To use the external /root/.smbcredentials password it shold be like so:

 

 

 

 

 

 

 

# cat /root/.smbcredentials

username=msusername
password=mssecretpassword
56#

 

 

Finally /root/.smbcredentials record should be as so:
 

//share-server-name/sharename /mnt/shares/windowsshare cifs credentials=/home/ubuntuusername/.smbcredentials,iocharset=utf8,sec=ntlm 0 0


Note You should already have

/mnt/shares/windowshare created on server B (the ount client) with:

mkdir -p  /mnt/shares/windowshare


To mount /etc/fstab defined filesystem to mount on next server boot then do

mount /mnt/shares/windowshare


or completely mount / remount all present /etc/fstab filesystems with the common

mount -a


(but here be careful as this might cause you troubles already other NFS or whatever FS is mounted and being read by clients) :

And you the remote Samba Share (mount location) – should be reachable with ping command and traceroute and remote server ports 139, 445 etc. should be up running opened and connectable from server B share-server-name/sharename

If you face some issues when trying to mount remote share with mount -t smbfs / mount.cifs then you can use smbclient with debug option to find out some more on the connectivity / authentication issue by using the smb share server IP address instead of hostnae and lets say a debug level of 3 like so:

 

 

 

 

smbclient -d3 -L //10.5.8.118/Files -A /root/.smbcredentials

[0] smbclient -d3 -L //10.2.3.111/Files -A /home/acteam/.smbcredentials     lp_load_ex: refreshing parameters
Initialising global parameters
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[global]"
WARNING: The "syslog" option is deprecated
added interface eth0 ip=10.2.3.127 bcast=10.2.3.255 netmask=255.255.255.0
Client started (version 4.3.11-Ubuntu).
Connecting to 10.2.3.111 at port 445
Doing spnego session setup (blob length=120)
got OID=1.3.6.1.4.1.311.2.2.30
got OID=1.2.840.48018.1.2.2
got OID=1.2.840.113554.1.2.2
got OID=1.2.840.113554.1.2.2.3
got OID=1.3.6.1.4.1.311.2.2.10
got principal=not_defined_in_RFC4178@please_ignore
GENSEC backend 'gssapi_spnego' registered
GENSEC backend 'gssapi_krb5' registered
GENSEC backend 'gssapi_krb5_sasl' registered
GENSEC backend 'spnego' registered
GENSEC backend 'schannel' registered
GENSEC backend 'naclrpc_as_system' registered
GENSEC backend 'sasl-EXTERNAL' registered
GENSEC backend 'ntlmssp' registered
GENSEC backend 'ntlmssp_resume_ccache' registered
GENSEC backend 'http_basic' registered
GENSEC backend 'http_ntlm' registered
GENSEC backend 'krb5' registered
GENSEC backend 'fake_gssapi_krb5' registered
Got challenge flags:
Got NTLMSSP neg_flags=0x62898215
NTLMSSP: Set final flags:
Got NTLMSSP neg_flags=0x62088215
NTLMSSP Sign/Seal – Initialising with flags:
Got NTLMSSP neg_flags=0x62088215
NTLMSSP Sign/Seal – Initialising with flags:
Got NTLMSSP neg_flags=0x62088215
Domain=[TMGRID] OS=[Windows Server 2012 R2 Standard 9600] Server=[Windows Server 2012 R2 Standard 6.3]

 

        Sharename       Type      Comment
        ———       —-      ——-
        ADMIN$          Disk      Remote Admin
        C$              Disk      Default share
        Files           Disk
        IPC$            IPC       Remote IPC
        MappedDrive     Disk
Connecting to 10.2.3.111 at port 139
Connecting to 10.2.3.111 at port 139
Connection to 10.2.3.111 failed (Error NT_STATUS_RESOURCE_NAME_NOT_FOUND)
NetBIOS over TCP disabled — no workgroup available

 

Sum it up

Lets Summarize a bit, here I described how to mount smbfs and cifs mount shares with mount command, how to define the auto mount on server boot via /etc/fstab, how to mount manually /etc/fstab defined mount and what should be the syntax of .smbcredentials user / pass file and also pointed how to debug problems on samba / windows server location share mounts with smbclient command.
 

How to remove ‘active contents’ from PDF file on Linux / Strip Active Contents from PDF

Thursday, July 18th, 2019

how-to-remove-active-content-from-pdf-with-ghoscript-on-gnu-linux.svg

I'm updating my Autiobography (CV) with my latest job eployeers, technology expertise and certifications and usually use the EuroPassCV standard web service to update already generated PDF files.The service as web based application service allows easy edit from the web as most web services which is quite handy and then allows Export to DOCX or PDF file format. So far so good but today I faced a really weird problem after, I've used successfully EuroPassCV service  and downloaded the PDF to my computer and tried to submit my Curriculum Vitae application to SAP's Successfactor newly created account for the purpose I faced a weird I error saying

"The system does not allow files with Active contents. Please …"

the-system-does-not-allow-files-with-active-contents-pdf-error-successfactors-errors

Of course if this error message was received on a Start-up application on Application upload that would be fine, but come on this is SAP's Successfactors, it cannot accept a standard generated PDF from EuroPass which nowadays is a standard for CV here in Europe and hosted on of official European Union website europa.eu

To me this is a clear signal SAP needs an experienced ICT specialists and Quality Assurance testers like me to fix their mess and I will be willing to help them if they contact me until its too late for them, but let me go back to the topic of this article which was how to remove active contents from a PDF file 🙂

So first lets make clear what is Active content in a file ?

Active contents is content that includes programs like Internet polls, JavaScript applications, stock tickers, animated images, ActiveX applications, action items, streaming video and audio, weather maps, embedded objects, and much more. Active content contains programs that trigger automatic actions on a Web page without the user's knowledge or consent.
Active contents (Macros) could exist in many file formats that are used daily in most companies / organizations daily, active content can be contained in documents such as MS Excel,  Word, PDF, PowerPoint and so on.

So why does some applications disable document support for Active contents?

Well just for the reason of security, Active contents could often be some kind of malware or crapware and they can mess up with the web application (in case of bugs) or even mess up with server software if it is a complex warm like behavior exploiting some kind of vulnerability.
One thing to say about active contents removal on file upload by applications is that this practice could only be tolerated if the organization had already adapted a security through obscurity which most likely is the case with SAP's Successfactors and many other applications out there.

So next question is how to  Panicea (Resolution) Active Contents existing in a PDF file

Assuming you have a GNU / Linux Desktop or server with ghostscript package installed (which is the case by default with virtually any modern Linux distribution), removing Active Contents from PDF to make possible file to be submitted to the picky Security Conscious application with a single command:
 

gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=CV-Georgi_Dimitrov_Georgiev-Europass-20190718-EN-noact-content.pdf -dBATCH CV-Georgi_Dimitrov_Georgiev-Europass-20190718-EN.pdf


After that the stripped active contents PDF file would succeed in uploading to web app.
 

 

 

Howto create Linux Music Audio CD from MP3 files / Create playable WAV format Audio CD Albums from MP3s

Tuesday, July 16th, 2019

cdburning-audio-music-cd-from-mp3-on-linuxcomapct-disc-tux-linux-logo

Recently my Mother asked me to prepare a Music Audio CD for her from a popular musician accordionist Stefan Georgiev from Dobrudja who has a unique folklore Bulgarian music.

As some of older people who still remember the age of the CD and who had most likely been into the CD burning Copy / Piracy business so popular in the countries of the ex-USSR so popular in the years 1995-2000 audio ,  Old CD Player Devices were not able to play the MP3 file format due to missing codecs (as MP3 was a proprietary compression that can't be installed on every device without paying the patent to the MP3 compression rights holder.

The revolutionary MP3 compression used to be booming standard for transferring Music data due to its high compression which made an ordinary MP3 of 5 minutes of 5MB (10+ times more compression than an ordinary classic WAV Audio the CPU intensiveness of MP3 files that puts on the reading device, requiring the CD Player to have a more powerful CPU.

Hence  due to high licensing cost and requirement for more powerful CPU enabled Audio Player many procuders of Audio Players never introduced MP3 to their devices and MP3 Neve become a standard for the Audio CD that was the standard for music listening inside almost every car out there.

Nowdays it is very rare need to create a Audio CD as audio CDs seems to be almost dead (As I heard from a Richard Stallman lecture In USA nowadays there is only 1 shop in the country where you can still buy CD or DVD drives) and only in third world as Africa Audio CDs perhaps are still in circulation.

Nomatter that as we have an old Stereo CD player on my village and perhaps many others, still have some old retired CD reading devices being able to burn out a CD is a useful thing.

Thus to make mother happy and as a learning excercise, I decided to prepare the CD for her on my Linux notebook.
Here I'll shortly describe the takes I took to make it happen which hopefully will be useful for other people that need to Convert and burn Audio CD from MP3 Album.

 

1. First I downloaded the Album in Mp3 format from Torrent tracker

My homeland Bulgaria and specific birth place place the city of Dobrich has been famous its folklore:  Galina Durmushlijska and Stefan Georgiev are just 2 of the many names along with Оркестър Кристал (Orchestra Crystal) and the multitude of gifted singers. My mother has a santiment for Stefan Georgiev, as she listened to this gifted accordinist on her Uncle's marriage.

Thus In my case this was (Стефан Георгиев Хора и ръченици от Добруджа) the album full song list here If you're interested to listen the Album and Enjoy unique Folklore from Dobrudja (Dobrich) my home city, Stefan Georgiev's album Hora and Rachenica Dances is available here

 


Stefan_Georgiev-old-audio-Music-CD-Hora-i-Rychenici-ot-Dobrudja-Horos-and-Ruchenitsas-from-Dobrudja-CD_Cover
I've downloaded them from Bulgarian famous torrent tracker zamunda.net in MP3 format.
Of course you need to have a CD / DVD readed and write device on the PC which nowdays is not present on most modern notebooks and PCs but as a last resort you can buy some cheap External Optical CD / DVD drive for 25 to 30$ from Amazon / Ebay etc.

 

2. You will need to install a couple of programs on Linux host (if you don't have it already)


To be able to convert from command line from MP3 to WAV you will need as minimum ffmpeg and normalize-audio packages as well as some kind of command line burning tool like cdrskin  wodim which is
the fork of old good known cdrecord, so in case if you you're wondering what happened with it just
use instead wodim.

Below is a good list of tools (assuming you have enough HDD space) to install:

 

root@jeremiah:/ # apt-get install –yes dvd+rw-tools cdw cdrdao audiotools growisofs cdlabelgen dvd+rw-tools k3b brasero wodim ffmpeg lame normalize-audio libavcodec58

 

Note that some of above packages I've installed just for other Write / Read operations for DVD drives and you might not need that but it is good to have it as some day in future you will perhaps need to write out a DVD or something.
Also the k3b here is specific to KDE and if you're a GNOME user you could use Native GNOME Desktop app such brasero or if you're in a more minimalistic Linux desktop due to hardware contrains use XFCE's native xfburn program.

If you're a console / terminal geek like me you will definitely enjoy to use cdw
 

root@jeremiah:/ # apt-cache show cdw|grep -i description -A 1
Description-en: Tool for burning CD's – console version
 Ncurses-based frontend for wodim and genisoimage. It can handle audio and

Description-md5: 77dacb1e6c00dada63762b78b9a605d5
Homepage: http://cdw.sourceforge.net/

 

3. Selecting preferred CD / DVD / BD program to use to write out the CD from Linux console


cdw uses wodim (which is a successor of good old known console cdrecord command most of use used on Linux in the past to burn out new Redhat / Debian / different Linux OS distro versions for upgrade purposes on Desktop and Server machines.

To check whether your CD / DVD drive is detected and ready to burn on your old PC issue:

 

root@jeremiah:/# wodim -checkdrive
Device was not specified. Trying to find an appropriate drive…
Detected CD-R drive: /dev/cdrw
Using /dev/cdrom of unknown capabilities
Device type    : Removable CD-ROM
Version        : 5
Response Format: 2
Capabilities   :
Vendor_info    : 'HL-DT-ST'
Identification : 'DVDRAM GT50N    '
Revision       : 'LT20'
Device seems to be: Generic mmc2 DVD-R/DVD-RW.
Using generic SCSI-3/mmc   CD-R/CD-RW driver (mmc_cdr).
Driver flags   : MMC-3 SWABAUDIO BURNFREE
Supported modes: TAO PACKET SAO SAO/R96P SAO/R96R RAW/R16 RAW/R96P RAW/R96R

You can also use xorriso (whose added value compared to other console burn cd tools is is not using external program for ISO9660 formatting neither it use an external or an external burn program for CD, DVD or BD (Blue Ray) drive but it has its own libraries incorporated from libburnia-project.org libs.

Below output is from my Thinkpad T420 notebook. If the old computer CD drive is there and still functional in most cases you should not get issues to detect it.

cdw ncurses text based CD burner tool's interface is super intuitive as you can see from below screenshot:

cdw-burn-cds-from-console-terminal-on-GNU-Linux-and-FreeBSD-old-PC-computer

CDW has many advanced abilities such as “blanking” a disk or ripping an audio CD on a selected folder. To overcome the possible problem of CDW not automatically detecting the disk you have inserted you can go to the “Configuration” menu, press F5 to enter the Hardware options and then on the first entry press enter and choose your device (by pressing enter again). Save the setting with F9.
 

4. Convert MP3 / MP4 Files or whatever format to .WAV to be ready to burn to CD


Collect all the files you want to have collected from the CD album in .MP3 a certain directory and use a small one liner loop to convert files to WAV with ffmpeg:
 

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

for i in $( ls *.mp3); do ffmpeg -i $i $i.wav; done


If you don't have ffmpeg installed and have mpg123 you can also do the Mp3 to WAV conversion with mpg123 cmd like so:

 

for i in $( ls ); do mpg123 -w $i.wav $i.mp3; done


Another alternative for conversion is to use good old lame (used to create Mp3 audio files but abling to also) decode
mp3 to wav.

 

lame –decode somefile.mp3 somefile.wav


In the past there was a burn command tool that was able to easily convert MP3s to WAV but in up2date Linux modern releases it is no longer available most likely due to licensing issues, for those on older Debian Linux 7 / 8 / 9 / Ubuntu 8 to 12.XX / old Fedoras etc. if you have the command you can install burn and use it (and not bother with shell loops):

apt-get install burn

or

yum install burn


Once you have it to convert

 

$ burn -A -a *.mp3
 

 

5. Fix file naming to remove empty spaces such as " " and substitute to underscores as some Old CD Players are
unable to understand spaces in file naming with another short loop.

 

for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done

 

6. Normalize audio produced .WAV files (set the music volume to a certain level)


In case if wondering why normalize audio is needed here is short extract from normalize-audio man page command description to shed some light.

"normalize-audio  is  used  to  adjust  the volume of WAV or MP3 audio files to a standard volume level.  This is useful for things like creating mp3 mixes, where different recording levels on different albums can cause the volume to  vary  greatly from song to song."
 

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

normalize-audio -m *.wav

 

7. Burn the produced normalized Audio WAV files to the the CD

 

wodim -v -fix -eject dev='/dev/sr0' -audio -pad *.wav


Alternatively you can conver all your MP3 files to .WAV with anything be it audacity
or another program or even use 
GNOME's CDBurn tool brasero (if gnome user) or KDE's CDBurn which in my opinion is
the best CD / DVD burning application for Linux K3B.

Burning Audio CD with K3b is up to few clicks and super easy and even k3b is going to handle the MP3 to WAV file Conversion itself. To burn audio with K3B just run it and click over 'New Audio CD Project'.

k3b-on-debian-gnu-linux-burn-audio-cd-screenshot

For those who want to learn a bit more on CD / DVD / Blue-Ray burning on GNU / Linux good readings are:
Linux CD Burning Mini Howto, is Linux's CD Writing Howto on ibiblio (though a bit obsolete) or Debian's official documentation on BurnCD.
 

8. What we learned here


Though the accent of this tutorial was how to Create Audio Music CD from MP3 on GNU / Linux, the same commands are available in most FreeBSD / NetBSD / OpenBSD ports tree so you can use the same method to build prepare Audio Music CD on *BSDs.

In this article, we went through few basic ways on how to prepare WAV files from MP3 normalize the new created WAV files on Linux, to prepare files for creation of Audio Music CD for the old mom or grandma's player or even just for fun to rewind some memories. For GUI users this is easily done with  k3b,  brasero or xfburn.

I've pointed you to cdw a super useful text ncurses tool that makes CD Burninng from plain text console (on servers) without a Xorg / WayLand  GUI installed super easy. It was shortly reviewed what has changed over the last few years and why and why cdrecord was substituted for wodim. A few examples were given on how to handle conversion through bash shell loops and you were pointed to some extra reading resources to learn a bit more on the topic.
There are plenty of custom scripts around for doing the same CD Burn / Covnersion tasks, so pointing me to any external / Shell / Perl scripts is mostly welcome.

Hope this learned you something new, Enjoy ! 🙂

Upgrade Debian Linux 9 to 10 Stretch to Buster and Disable graphical service load boot on Debian 10 Linux / Debian Buster is out

Tuesday, July 9th, 2019

howto-upgrade-debian-linux-debian-stretch-to-buster-debian-10-buster

I've just took a time to upgrade my Debian 9 Stretch Linux to Debian Buster on my old school Laptop (that turned 11 years old) Lenovo Thinkpad R61 . The upgrade went more or less without severe issues except few things.

The overall procedure followed is described n a few websites out there already and comes up to;

 

0. Set the proper repository location in /etc/apt/sources.list


Before update the sources.list used are:
 

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free

 

deb [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free
deb-src [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free

deb http://ftp.debian.org/debian buster-backports main


For people that had stretch defined in /etc/apt/sources.list you should change them to buster or stable, easiest and quickest way to omit editting with vim / nano etc. is run as root or via sudo:
 

sed -i 's/stretch/buster/g' /etc/apt/sources.list
sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/*.list

The minimum of config in sources.list after the modification should be
 

deb http://deb.debian.org/debian buster main
deb http://deb.debian.org/debian buster-updates main
deb http://security.debian.org/debian-security buster/updates main

Or if you want to always be with latest stable packages (which is my practice for notebooks):

deb http://deb.debian.org/debian stable main
deb http://deb.debian.org/debian stable-updates main
deb http://security.debian.org/debian-security stable/updates main

 

1. Getting list of hold packages if such exist and unholding them, e.g.

 

apt-mark showhold


Same could also be done via dpkg

dpkg –get-selections | grep hold


To unhold a package if such is found:

echo "package_name install"|sudo dpkg –set-selections

For those who don't know what hold package is this is usually package you want to keep at certain version all the time even though after running apt-get upgrade to get the latest package versions.
 

2. Use df -h and assure you have at least 5 – 10 GB free space on root directory / before proceed

df -h /

3. Update packages list to set new set repos as default

apt update

 

4. apt upgrade
 

apt upgrade

Here some 10 – 15 times you have to confirm what you want to do with configuration that has changed if you're unsure about the config (and it is not critical service) you're aware as such as Apache / MySQL / SMTP etc. it is best to install the latest maintainer version.

Hopefully here you will not get fatal errors that will interrupt it.

P.S. It is best to run apt-update either in VTTY (Virtual console session) with screen or tmux or via a physical tty (if this is not a remote server) as during the updates your GUI access to the gnome-terminal or konsole / xterm whatever console used might get cut. Thus it is best to do it with command:
 

screen apt upgrade

 

5. Run dist-upgrade to finalize the upgrade from Stertch to Buster

 

Once all is completed of the new installed packages, you will need to finally do, once again it is best to run via screen, if you don't have installed screen install it:

 

if [ $(which screen) ]; then echo 'Installed'; else apt-get install –yes screen ; fi

screen apt dist-upgrade


Here once again you should set whether old configuration to some e services has to stay or the new Debian maintainer package shipped one will overwrite the old and locally modified (due to some reason), here do wisely whatever you will otherwise some configured services might not boot as expected on next boot.

 

6. What if you get packages failed on update


If you get a certain package failed to configure after installed due to some reason, if it is a systemd service use:

 

journalctl -xe |head -n 50


or fully observer output of journalctl -xe and decide on yourself.

In most cases

dpkg-reconfigure failed-package-name


should do the trick or at least give you more hints on how to solve it.

 

Also if a package seems to be in inconsistent or broken state after upgrade  and simple dpkg-reconfigure doesn't help, a good command
that can help you is

 

dpkg-reconfigure -f package_name

 

or you can try to workaround a failed package setup with:
 

dpkg –configure -a

 
If dpkg-reconfigure doesn't help either as I experienced in prior of Debian from Debian 6 -> 7 an Debian 7 ->8 updates on some Computers, then a very useful thing to try is:
 

apt-get update –fix-missing 

apt-get install -f


At certain cases the only work around to be able to complete the package upgrade is to to remove the package with apt remove but due to config errors even that is not possible to work around this as final resort run:
 

dpkg –remove –force-remove-reinstreq

 

7. Clean up ununeeded packages

 

Some packages are left over due to package dependencies from Stretch and not needed in buster anymore to remove them.
 

apt autoremove

 

8. Reboot system once all upgrade is over

 

/sbin/reboot

 

9. Verify your just upgraded Debian is in a good state

 

root@noah:~# uname -a;
Linux noah 4.19.0-5-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

 

root@noah:~# cat /etc/issue.net
Debian GNU/Linux 10
 

 

root@noah:~# lsb_release -a
No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:    buster

 

root@noah:~# hostnamectl
   Static hostname: noah
         Icon name: computer-laptop
           Chassis: laptop
        Machine ID: 4759d9c2f20265938692146351a07929
           Boot ID: 256eb64ffa5e413b8f959f7ef43d919f
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-rt-amd64
      Architecture: x86-64

 

10. Remove annoying picture short animation with debian logo looping

 

plymouth-debian-graphical-boot-services

By default Debian 10 boots up with annoying screen hiding all the status of loaded services state .e.g. you cannot see the services that shows in [ FAILED ] state and  which do show as [ OK ] to revert back the old behavior I'm used to for historical reasons and as it shows a lot of good Boot time debugging info, in previous Debian distributions this was possible  by setting the right configuration options in /etc/default/grub

which so far in my config was like so

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y zswap.enabled=1 text"


Note that zswap.enabled=1 passed option is because my notebook is pretty old machine from 2008 with 4GB of memory and zswap does accelerate performance when working with swap – especially helpful on Older PCs for more you can read more about zswap on ArchLinux wiki
After modifying this configuration to load the new config into grub the cmd is:
 

/usr/sbin/update-grub

 
As this was not working and tried number of reboots finally I found that annoying animated gif like picture shown up is caused by plymouth below is excerpts from Plymouth's manual page:


       "The plymouth sends commands to a running plymouthd. This is used during the boot process to control the display of the graphical boot splash."

Plymouth has a set of themes one can set:

 

# plymouth-set-default-theme -l
futureprototype
details
futureprototype
joy
lines
moonlight
softwaves
spacefun
text
tribar

 

I tried to change that theme to make the boot process as text boot as I'm used to historically with cmd:
 

update-alternatives –config text.plymouth

 
As after reboot I hoped the PC will start booting in text but this does not happened so the final fix to turn back to textmode service boot was to completely remove plymouth
 

apt-get remove –yes plymouth

How to start / Stop and Analyze system services and improve Linux system boot time performance

Friday, July 5th, 2019

systemd-components-systemd-utilities-targets-cores-libraries
This post is going to be a very short one and to walk through shortly to System V basic start / stop remove service old way and the new ways introduced over the last 10 years or so with the introduction of systemd on mass base across Linux distributions.
Finally I'll give you few hints on how to check (analyze) the boot time performance on a modern GNU / Linux system that is using systemd enabled services.
 

1. System V and the old days few classic used ways to stop / start / restart services (runlevels and common wrapper scripts)

 

The old fashioned days when Linux was using SystemV / e.g. no SystemD used way was to just go through all the running services with following the run script logic inside the runlevel the system was booting, e.g. to check runlevel and then potimize each and every run script via the respective location of the bash service init scripts:

 

root@noah:/home/hipo# /sbin/runlevel 
N 5

 

Or on some RPM based distros like Fedora / RHEL / SUSE Enterprise Linux to use chkconfig command, e.g. list services:

~]# chkconfig –list

etworkManager  0:off   1:off   2:on    3:on    4:on    5:on    6:off
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
anamon          0:off   1:off   2:off   3:off   4:off   5:off   6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off

And to start stop the service into (default runlevel) or respective runlevel:

 

~]#  chkconfig httpd on

~]# chkconfig –list httpd
httpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

 

 

~]# chkconfig service_name on –level runlevels

 


Debian / Ubuntu and other .deb based distributions with System V (which executes scripts without single order but one by one) are not having natively chkconfig but instead are famous for update-rc.d init script wrapper, here is few basic use  of it:

update-rc.d <service> defaults
update-rc.d <service> start 20 3 4 5
update-rc.d -f <service>  remove

Here defaults means default set boot runtime for system and numbers are just whether service is started or stopped for respective runlevels. To check what is your default one simply run /sbin/runlevel

Other useful tool to stop / start services and analyze what service is running and which not in real time (but without modifying boot time set for a service) – more universal nowadays is to use the service command.

root@noah:/home/hipo# service –status-all
 [ + ]  acpid
 [ – ]  alsa-utils
 [ – ]  anacron
 [ + ]  apache-htcacheclean
 [ – ]  apache2
 [ + ]  atd
 [ + ]  aumix

root@noah:/home/hipo# service cron restart/usr/sbin/service command is just a simple wrapper bash shell script that takes care about start / stop etc. operations of scripts found under /etc/init.d

For those who don't want to tamper with too much typing and manual configuration there is an all distribution system V compatible ncurses interface text itnerface sysv-rc-conf which could make your life easier on configuring services on non-systemd (old) Linux-es.

To install on Debian distros:

debian:~# apt-get install sysv-rc-conf

debian:~# sysv-rc-conf


SysV RC Conf desktop on GNU Linux using sysv-rc-conf systemV and systemd
 

2. SystemD basic use Start / stop check service and a little bit of information
for the novice

As most Linux kernel based distributions except some like Slackware and few others see the full list of Linux distributions without systemd (and aha yes slackw. users loves rc.local so much – we all do 🙂  migrated and are nowadays using actively SystemD, to start / stop analyze running system runnig services / processes

systemctl – Control the systemd system and service manager

To check whether a service is enabled

systemctl is-active application.service

To check whether a unit is in a failed state

systemctl is-failed application.service

To get a status of running application via systemctl messaging

# systemctl status sshd
● ssh.service – OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-07-06 20:01:02 EEST; 2h 3min ago Main PID: 1335 (sshd) Tasks: 1 (limit: 4915) CGroup: /system.slice/ssh.service └─1335 /usr/sbin/sshd -D юли 06 20:01:00 noah systemd[1]: Starting OpenBSD Secure Shell server… юли 06 20:01:02 noah sshd[1335]: Server listening on 0.0.0.0 port 22. юли 06 20:01:02 noah sshd[1335]: Server listening on :: port 22. юли 06 20:01:02 noah systemd[1]: Started OpenBSD Secure Shell server.

To enable / disable application with systemctl systemctl enable application.service

systemctl disable application.service

To stop / start given application systemcl stop sshd

systemctl stop tor

To reload running application

systemctl reload sshd

Some applications does not have the right functionality in systemd script to reload configuration without fully restarting the app if this is the case use systemctl reload-or-restart application.service

systemctl list-unit-files

Then to view the content of a single service unit file:

:~# systemctl cat apache2.service
# /lib/systemd/system/apache2.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
Environment=APACHE_STARTED_BY_SYSTEMD=true
ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl stop
ExecReload=/usr/sbin/apachectl graceful
PrivateTmp=true
Restart=on-abort

[Install]
WantedBy=multi-user.target


converting-traditional-init-scripts-to-systemd-graphical-diagram

systemd's advancement over normal SystemV services it is able to track and show dependencies
of a single run service for proper operation on other services

:~# systemctl list-dependencies sshd.service

 


● ├─system.slice
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─keyboard-setup.service
●   ├─kmod-static-nodes.service
●   ├─proc-sys-fs-binfmt_misc.automount
●   ├─sys-fs-fuse-connections.mount
●   ├─sys-kernel-config.mount
●   ├─sys-kernel-debug.mount
●   ├─systemd-ask-password-console.path
●   ├─systemd-binfmt.service
….

.

 

You can also mask / unmask service e.g. make it temporary unavailable via systemd with

sudo systemctl mask nginx.service

it will then appear as masked if you do list-unit-files

If you want to change something on a systemd unit file this is done with

systemctl edit –full nginx.service

In case if some modificatgion was done to systemd service files e.g. lets say to
/etc/systemd/system/apache2.service or even you've made a Linux system Upgrade recently
that added extra systemd service config files it will be necessery to reload all files
present in /etc/systemd/system/* with:

systemctl daemon-reload


Systemd has a target states which are pretty similar to the runlevel concept (e.g. runlevel 5 means graphical etc.), for example to check the default target for a system:

One very helpful feature is to restart systemd but it seems this is not well documented as of now and though this might work after some system package upgrade roll-outs it is always better to reboot the system, but you can give it a try if restart can't be done due to application criticallity.

To restart systemd and its spawned subprocesses do:
 

systemctl daemon-reexec

 

root@noah:/home/hipo# systemctl get-default
graphical.target


 to check all targets possible targets

root@noah:/home/hipo# systemctl list-unit-files –type=target
UNIT FILE                 STATE   
basic.target              static  
bluetooth.target          static  
busnames.target           static  
cryptsetup-pre.target     static  
cryptsetup.target         static  
ctrl-alt-del.target       disabled
default.target            static  
emergency.target          static  
exit.target               disabled
final.target              static  
getty.target              static  
graphical.target          static  

you can put the system in Single user mode if you like without running the good old well known command:

/sbin/init 1 

command with

systemctl rescue

You can even shutdown / poweroff / reboot system via systemctl (though I never did that and I don't recommend) 🙂
To do so use:

systemctl halt
systemctl poweroff
systemctl reboot


For the lazy ones that don't want to type all the time like crazy to configure and manage simple systemctl set services take a look at chkservice – an ncurses text based menu systemctl management interface

As chkservice is relatively new it is still not present in stable Stretch Debian repositories but it is in current testing Debian unstable Buster / Sid – Testing / Unstable distribution and has installable package for Ubuntu / Arch Linux and Fedora

chkservice-Linux-systemctl-ncurses-text-menu-service-management-interface-start-chkservice
Picture Source Tecmint.com

chkservice linux help screen


3. Analyzing and fix performance boot slowness issues due to a service taking long to boot


The first very useful thing is to know how long exactly all daemons / services got booted
on your GNU / Linux OS.

linux-server:~# systemd-analyze 
Startup finished in 4.135s (kernel) + 3min 47.863s (userspace) = 3min 51.998s

As you can see it reports both the kernel boot time and userspace (surrounding services
that had to boot for the system to be considered fully booted).


Once you have the system properly booted you have a console or / ssh access

root@pcfreak:/home/hipo# systemd-analyze blame
    2min 14.172s tor@default.service
    1min 40.455s docker.service
     1min 3.649s fail2ban.service
         58.806s nmbd.service
         53.992s rc-local.service
         51.458s systemd-tmpfiles-setup.service
         50.495s mariadb.service
         46.348s snort.service
         34.910s ModemManager.service
         33.748s squid.service
         32.226s ejabberd.service
         28.207s certbot.service
         28.104s networking.service
         23.639s munin-node.service
         20.917s smbd.service
         20.261s tinyproxy.service
         19.981s accounts-daemon.service
         18.501s loadcpufreq.service
         16.756s stunnel4.service
         15.575s oidentd.service
         15.376s dev-sda1.device
         15.368s courier-authdaemon.service
         15.301s sysstat.service
         15.154s gpm.service
         13.276s systemd-logind.service
         13.251s rsyslog.service
         13.240s lpd.service
         13.237s pppd-dns.service
         12.904s NetworkManager-wait-online.service
         12.540s lm-sensors.service
         12.525s watchdog.service
         12.515s inetd.service


As you can see you get a list of services time took to boot in secs and you can
further debug each of it to find out why it boots so slow (netwok / DNS / configuration isssue whatever).

On a servers it is useful to look up for some processes slowing it down like gdm.service etc.

 

Close up words rant on SystemD vs SysemV

init-and-systemd-comparison-commands-linux-booting-1

A lot could be ranted on what is better systemd or systemV. I personally hated systemd since day since I saw it being introduced first in Fedora / CentOS linuxes and a bit later in my beloved desktop used Debian Linux.
I still remember the bugs and headaches with systemd's intruduction as it is with all new the early adoption of technology makes a lot of pain in the ass.
Eventually systemd has become a standard and with my employment as a contractor through Itelligence GmBH for SAP AG I now am forced to work with systemd daily on SLES 12 based Linuces and I was forced to get used to it. 
But still there is my personal preference to SystemV even though the critics of slow boot etc.but for managing a multitude of Linux preinstalled servers like Virtual Machines and trying to standardize a Data Center with Tens of Thousands of Linuxes running on different Hypervisors VMWare / OpenXen + physical hosts etc. systemd brings a bit of more standardization that makes it a winner.