Archive for the ‘System Administration’ Category

Check the count and monitor of established / time_wait TCP, UDP connections on Linux and Windows with netstat command

Wednesday, February 6th, 2019


For me as a GNU / Linux sysadmin it is intuitive to check on a server the number of established connections / connections in time_wait state and so on .

I will not explain why this is necessery as every system administrator out there who had a performance or network issues due to server / applications connection overload or have been a target of Denial of Service (DoS)
or Distributed Denial of Service attacks (DDoS)  
is well aware that a number of connections in different states such as SYN_ACK /  TIME_WAIT or ESTABLISHED state could be very nasty thing and could cause a productive application or Infrastructure service to be downed for some time causing from thousands of Euros to even millions to some bussinesses as well as some amount of data loss …

To prevent this therefore sysadmins should always take a look periodically on the Connection states on the adminned server (and in this number I say not only sys admins but DevOps guys who are deploying micro-services for a customer in the Cloud – yes I believe Richard Stallman is right here they're clouding your minds :).

Even though cloud services could provide a very high amount of Hardware (CPU / Memory / Storage) resources, often for custom applications migrating the application in the Cloud does not solve it's design faults or even problems on a purely classical system administration level.


1. Get a statistic for FIN_WAIT1, FOREIGN, SYNC_RECV, LAST_ACK, TIME_WAIT, LISTEN and ESTABLISHED  Connections on GNU / Linux


On GNU / Linux and other Linux like UNIXes the way to do it is to grep out the TCP / UDP connection type you need via netstat a very useful cmd in that case is:


root@pcfreak:~# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
      1 established)
      1 FIN_WAIT1
      1 Foreign
      1 SYN_RECV
      3 LAST_ACK
      4 FIN_WAIT2
      8 TIME_WAIT
     45 LISTEN


2. Netstat 1 liner to Get only established and time_wait connections state 


Other ways to check only TCP ESTABLISHED connections on Linux I use frequently are:


root@pcfreak:~# netstat -etna|grep -i establi|wc -l



Or to get whole list of connections including the ones who are about to be esatablished in FIN_WAIT2, TIME_WAIT, SYN_RECV state:


root@pcfreak:~# netstat -tupen |wc -l


3. Other Linux useful one liner commands to track your connection types

netstat -n -p | grep SYN_REC | sort -u

List out the all IP addresses involved instead of just count.

netstat -n -p | grep SYN_REC | awk '{print $5}' | awk -F: '{print $1}'


List all the unique IP addresses of the node that are sending SYN_REC connection status.

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n


Use netstat command to calculate and count the number of connections each IP address makes to the server.

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n


List count of number of connections the IPs are connected to the server using TCP or UDP protocol.

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr


Check on ESTABLISHED connections instead of all connections, and displays the connections count for each IP.


netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1


Show and list IP address and its connection count that connect to port 80 on the server. Port 80 is used mainly by HTTP web page request.

Examples are taken from this nice blog post


4. Check the count of esatblished connections on M$ Windows


As I'm forced to optimize a couple of Microsoft Windows DNS servers which are really slow to resolve the
The logical question for me was how the Established and TIME_WAIT state connections then could be checked on Windows OS, after a quick investigation online I've come up with this:


C:\Users\admin> netstat -nao | find /i "estab" /c




C:\Users\admin> netsatt -nao | find /i "time_wait" /c



If you're used to Linux watch command, then to do same on Windows OS (e.g. check the output of netstat) command every second
and print output use:


netstat –an 1 | find “3334”


Below commands will show stats for services listening on TCP port 3334

To find out which process on system sends packets to remote destination:


netstat –ano 1 | find “Dest_IP_Addr”


The -o parameter outputs the process ID (PID) responsible for the connection.
then if you need further you can find the respective process name with tasklist< cmd.
Another handy Windows netstat option is -b which will show EXE file running as long as
the related used DLL Libraries which use TCP / UDP .

Other useful netsatat Win example is to grep for a port and show all established connections for it with:


netstat –an 1 | find “8080” | find “ESTABLISHED”


5. Closure

Hopefully this article will give you some idea on what is eating your bandwidth connections or overloading your GNU / Linux – Windows systems. And will point you to the next in line logical thing to do optimization / tuning
settings to be made on your system for example if Linux with sysctl – see my previous relater article here

I'll be intested to hear from sysadm colleagoes for other useful ways to track connections perhaps with something like ss tool (a utility to investigate sockets).
Also any optimization hints that would cause servers less downtime and improve network / performance thouroughput is mostly welcome.


Export / Import PuTTY Tunnels SSH Sessions from one to another Windows machine howto

Thursday, January 31st, 2019


As I've started on job position – Linux Architect in last November 2018 in Itelligence AG as a contractor (External Service) – a great German company who hires the best IT specialists out there and offers a flexible time schedules for emploees doing various very cool IT advanced operations and Strategic advancement of SAP's Cloud used Technology and Services improvements for SAP SE – SAP S4HANA and HEC (HANA Enterprise Cloud) and been given for work hardware a shiny Lenovo Thinkpad 500 Laptop with Windows 10 OS (SAP pre-installed), I needed to make some SSH Tunnels to machines to (Hop Station / Jump hosts) for that purpose, after some experimenting with MobaXterm Free (Personal Edition 11.0) and the presumable limitations of tunnels of the free client as well as my laziness to add the multiple ssh tunnels to different ssh / rdp / vnc etc. servers, finally I decided to just copy all the tunnels from a colleague who runs Putty and again use the good old Putty – old school Winblows SSH Terminal Client but just for creating the SSH tunnels and for rest use MobaXterm, just like in old times while still employe in Hewlett Packard. For that reason to copy the Tunnels from my dear German Colleague Henry Beck (A good herated collegue who works in field of Storage dealing with NetApps / filer Clusters QNap etc.).

Till that moment I had no idea how copying a saved SSH Tunnels definition is possible, I did a quick research just to find out this is done not with Putty Interface itself but, insetead through dumping Windows Putty Stored Registry records into a File, then transfer to the PC where Tunnels needs to be imported and then again (either double click the registry file) to load it, into registry or use Windows registry editor command line interface reg, here is how:

1. Export


Run cmd.exe (note below command) 

requires elevated Run as Administrator prompt:

Only sessions:

regedit /e "%USERPROFILE%\Desktop\putty-sessions.reg" HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions

All settings:

regedit /e "%USERPROFILE%\Desktop\putty.reg" HKEY_CURRENT_USER\Software\SimonTatham


If you have powershell installed on machine, to dump

Only sessions:


reg export HKCU\Software\SimonTatham\PuTTY\Sessions ([Environment]::GetFolderPath("Desktop") + "\putty-sessions.reg")

All settings:

reg export HKCU\Software\SimonTatham ([Environment]::GetFolderPath("Desktop") + "\putty.reg")

2. Import

Double-click on the 


 file and accept the import.


Alternative ways:



require elevated command prompt:

regedit /i putty-sessions.reg regedit /i putty.reg


reg import putty-sessions.reg reg import putty.reg

Below are some things to consider:

Note !do not replace 


 with your username.


Note !: It will create a 


 file on the Desktop of the current user (for a different location modify path)


Note !: It will not export your related (old system stored) SSH keys.

What to expect next?


The result is in Putty you will have the Tunnel sessions loadable when you launch (Portable or installed) Putty version.
Press Load button over the required saved Tunnels list and there you go under


Connection SSH -> Tunnels 


you will see all the copied tunnels.


Automatic network restart and reboot Linux server script if ping timeout to gateway is not responding as a way to reduce connectivity downtimes

Monday, December 10th, 2018


Inability of server to come back online server automaticallyafter electricity / network outage

These days my home server  is experiencing a lot of issues due to Electricity Power Outages, a construction dig operations to fix / change waterpipe tubes near my home are in action and perhaps the power cables got ruptered by the digger machine.
The effect of all this was that my server networking accessability was affected and as I didn't have network I couldn't access it remotely anymore at a certain point the electricity was restored (and the UPS charge could keep the server up), however the server accessibility did not due restore until I asked a relative to restart it or under a more complicated cases where Tech aquanted guy has to help – Alexander (Alex) a close friend from school years check his old site here – helps a restart the machine physically either run a quick restoration commands on root TTY terminal or generally do check whether default router is reachable.

This kind of downtime issues over the last month become too frequent (the machine was down about 5 times for 2 to 5 hours and this was too much (and weirdly enough it was not accessible from the internet even after electricity network was restored and the only solution to that was a physical server restart (from the Power Button).

To decrease the number of cases in which known relatives or friends has to  physically go to the server and restart it, each time after network or electricity outage I wrote a small script to check accessibility towards Default defined Network Gateway for my server with few ICMP packages sent with good old PING command
and trigger a network restart and system reboot
(in case if the network restart does fail) in a row.

1. Create reboot-if-nwork-is-downsh script under /usr/sbin or other dir

Here is the script itself:


# Script checks with ping 5 ICMP pings 10 times to DEF GW and if so
# triggers networking restart /etc/inid.d/networking restart
# Then does another 5 x 10 PINGS and if ping command returns errors,
# Reboots machine
# This script is useful if you run home router with Linux and you have
# electricity outages and machine doesn't go up if not rebooted in that case


run_ping () {
for i in $(seq 1 10); do
    ping -c 5 $GATEWAY_HOST


reboot_f () {
if [ $? -eq 0 ]; then
        echo "$(date "+%Y-%m-%d %H:%M:%S") Ping to $GATEWAY_HOST OK" >> /var/log/reboot.log
    /etc/init.d/networking restart
        echo "$(date "+%Y-%m-%d %H:%M:%S") Restarted Network Interfaces:" >> /tmp/rebooted.txt
    for i in $(seq 1 10); do ping -c 5 $GATEWAY_HOST; done
    if [ $? -eq 0 ] && [ $(cat /tmp/rebooted.txt) -lt ‘5’ ]; then
         echo "$(date "+%Y-%m-%d %H:%M:%S") Ping to $GATEWAY_HOST FAILED !!! REBOOTING." >> /var/log/reboot.log

    # increment 5 times until stop
    [[ -f /tmp/rebooted.txt ]] || echo 0 > /tmp/rebooted.txt
    n=$(< /tmp/rebooted.txt)
        echo $(( n + 1 )) > /tmp/rebooted.txt
    # if 5 times rebooted sleep 30 mins and reset counter
    if [ $(cat /tmprebooted.txt) -eq ‘5’ ]; then
    sleep 1800
        cat /dev/null > /tmp/rebooted.txt


You can download a copy of script here.

As you see in script successful runs  as well as its failures are logged on server in /var/log/reboot.log with respective timestamp.
Also a counter to 5 is kept in /tmp/rebooted.txt, incremented on each and every script run (rebooting) if, the 5 times increment is matched

a sleep is executed for 30 minutes and the counter is being restarted.
The counter check to 5 guarantees the server will not get restarted if access to Gateway is not continuing for a long time to prevent the system is not being restarted like crazy all time.

2. Create a cron job to run every 15 minutes or so 

I've set the script to re-run in a scheduled (root user) cron job every 15 minutes with following  job:

To add the script to the existing cron rules without rewriting my old cron jobs and without tempering to use cronta -u root -e (e.g. do the cron job add in a non-interactive mode with a single bash script one liner had to run following command:


{ crontab -l; echo "*/15 * * * * /usr/sbin/ 2>&1 >/dev/null; } | crontab –

I know restarting a server to restore accessibility is a stupid practice but for home-use or small client servers with unguaranteed networks with a cheap Uninterruptable Power Supply (UPS) devices it is useful.


Time will show how efficient such a  "self-healing script practice is.
Even though I'm pretty sure that even in a Corporate businesses and large Public / Private Hybrid Clouds where access to remote mounted NFS / XFS / ZFS filesystems are failing a modifications of the script could save you a lot of nerves and troubles and unhappy customers / managers screaming at you on the phone 🙂

I'll be interested to hear from others who have a better  ideas to restore ( resurrect ) access to inessible Linux server after an outage.?

Prevent rsync cronjob to run multiple times via cronjob on Linux

Wednesday, November 21st, 2018


Today I had a report of a server whose Load Avarage keeps at the high level of 86, the machine runs on a bare metal rock solid hardware and even with such high Loads of the kernel it runs fine, but due to the I/O overhead the SANs red from a remote NetApp storage device started to be sluggish and hence it needed to be reviewed, thus I jumped in via the hop station (jump host) into the server.

1. Short investation on root cause for high server load

After a short investigation, I've found an rsync job set by someone on a cron job to be routinely run every 30 minutes, thus the old scheduled rsync, which seemed to run multiple times on the server (about 50 processes) of same rsync (file system synchronization was running) and as expected the storage was saddled with mutiple Input / Output requests.

The root cron job was like that:

server:~# crontab -u root -l |grep -i rsync
/usr/bin/rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize

A process list showed the following high number of running mirrored rsyncs:


server:~# ps axuwwf | grep -i rsync | wc -l


2. The Fix – Set Rsync to only via cron only in case if it is not already running in background

In order to fix it, I had to kill all current running rsync (here luckily only same single instance of rsync was running, but generally I was cautious to check no other rsync jobs are running – otherwise I would have mistakenly killed some other rsync job ongoing …)

Then I set the following new cron job one liner quick shell script that does the job to assign a pid file that is created before rsync and deleted after rsync completion.

if [ ! -e /tmp/repo_dba_sync.lock ]; then touch /tmp/repo_dba_sync.lock; /usr/bin/rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize ; trap 'rm -f /tmp/repo_dba_sync.lock; fi' EXIT  >/dev/null 2>&1

The cron job looked like so:


*/30 * * * * if [ ! -e /tmp/repo_dba_sync.lock ]; then touch /tmp/repo_dba_sync.lock; /usr/bin/rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize ; trap 'rm -f /tmp/repo_dba_sync.lock; fi'  EXIT >/dev/null 2>&1

Just in case if you're wondering
a trap should be used to verify that the lock file is removed when the script is exited for any reason.
This way the lock file will be removed even if the script exits before the end of the script.

An alternative and more simple ways to do it is via:

pgrep rsync > /dev/null || rsync -ax /var/www/htdocs/directory_to_synchronize / /srv/www/synch_back/directory_to_synchrnize


Or if you don't want to use bash's:

if []; then; fi

condition but still use a file lock the flock command can be used like so:

flock -n lock_file -c "rsync …"

Ansible Quick Start Cheatsheet for Linux admins and DevOps engineers

Wednesday, October 24th, 2018


Ansible is widely used (Configuration management, deployment, and task execution system) nowadays for mass service depoyments on multiple servers and Clustered environments like, Kubernetes clusters (with multiple pods replicas) virtual swarms running XEN / IPKVM virtualization hosting multiple nodes etc. .

Ansible can be used to configure or deploy GNU / Linux tools and services such as Apache / Squid / Nginx / MySQL / PostgreSQL. etc. It is pretty much like Puppet (server / services lifecycle management) tool , except its less-complecated to start with makes it often a choose as a tool for mass deployment (devops) automation.

Ansible is used for multi-node deployments and remote-task execution on group of servers, the big pro of it it does all its stuff over simple SSH on the remote nodes (servers) and does not require extra services or listening daemons like with Puppet. It combined with Docker containerization is used very much for later deploying later on inside Cloud environments such as Amazon AWS / Google Cloud Platform / SAP HANA / OpenStack etc.


0. Instaling ansible on Debian / Ubuntu Linux

Ansible is a python script and because of that depends heavily on python so to make it running, you will need to have a working python installed on local and remote servers.

Ansible is as easy to install as running the apt cmd:


# apt-get install –yes ansible

The following additional packages will be installed:
  ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
Suggested packages:
  sshpass python-jinja2-doc ipython python-netaddr-docs python-gssapi
Recommended packages:
The following NEW packages will be installed:
  ansible ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,413 kB of archives.
After this operation, 22.8 MB of additional disk space will be used.

apt-get install –yes sshpass


Installing Ansible on Fedora Linux is done with:


# dnf install ansible –yes sshpass


On CentOS to install:

# yum install ansible –yes sshpass

sshpass needs to be installed only if you plan to use ssh password prompt authentication with ansible.

Ansible is also installable via python-pip tool, if you need to install a specific version of ansible you have to use it instead, the package is available as an installable package on most linux distros.

Ansible has a lot of pros and cons and there are multiple articles already written on people for and against it in favour of Chef or Puppet As I recently started learning Ansible. The most important thing to know about Ansible is though many of the things can be done directly using a simple command line, the tool is planned for remote installing of server services using a specially prepared .yaml format configuration files. The power of Ansible comes of the use of Ansible Playbooks which are yaml scripts that tells ansible how to do its activities step by step on remote server. In this article, I'm giving a quick cheat sheet to start quickly with it.

1. Remote commands execution with Ansible

First thing to do to start with it is to add the desired hostnames ansible will operate with it can be done either globally (if you have a number of remote nodes) to deploy stuff periodically by using /etc/ansible/hosts or use a custom host script for each and every ansible custom scripts developed.

a. Ansible main config files

A common ansible /etc/ansible/hosts definition looks something like that:


# cat /etc/ansible/hosts

Host to execute on can be also provided via a shell variable $ANSIBLE_HOSTS
b) is remote hosts reachable / execute commands on all remote host

To test whether hour hosts are properly configure from /etc/ansible/hosts you can ping all defined hosts with:


ansible all -m ping


This makes ansible try to remote to remote hosts (if you have properly configured SSH public key authorization) the command should return success statuses on every host.


ansible all -a "ifconfig -a"

If you don't have SSH keys configured you can also authenticate with an argument (assuming) all hosts are configured with same password with:


ansible all –ask-pass -a "ip all show" -u hipo –ask-pass


If you have configured group of hosts via hosts file you can also run certain commands on just a certain host group, like so:


ansible <host-group> -a <command>

It is a good idea to always check /etc/ansible/ansible.cfg which is the system global (main red ansible config file).

c) List defined host groups

ansible localhost -m debug -a 'var=groups.keys()'
ansible localhost -m debug -a 'var=groups'

d) Searching remote server variables


# Search remote server variables
ansible localhost -m setup -a 'filter=*ipv4*'



ansible localhost -m setup -a 'filter=ansible_domain'



ansible all -m setup -a 'filter=ansible_domain'



# uninstall package on RPM based distros
ansible centos -s -m yum -a "name=telnet state=absent"
# uninstall package on APT distro
ansible localhost -s -m apt -a "name=telnet state=absent"



2. Debugging – Listing information about remote hosts (facts) and state of a host


# All facts for one host
ansible -m setup
  # Only ansible fact for one host
-m setup -a 'filter=ansible_eth*'
# Only facter facts but for all hosts
ansible all -m setup -a 'filter=facter_*'

To Save outputted information per-host in separate files in lets say ~/ansible/host_facts


ansible all -m setup –tree ~/ansible/host_facts


3. Playing with Playbooks deployment scripts


a) Syntax Check of a playbook yaml


ansible-playbook –syntax-check

b) Run General Infos about a playbook such as get what a playbook would do on remote hosts (tasks to run) and list-hosts defined for a playbook (like above pinging).


ansible-playbook –list-hosts

To get the idea about what an yaml playbook looks like, here is example from official ansible docs, that deploys on remote defined hosts a simple Apache webserver.

– hosts: webservers
    http_port: 80
    max_clients: 200
  remote_user: root
  – name: ensure apache is at the latest version
      name: httpd
      state: latest
  – name: write the apache config file
      src: /srv/httpd.j2
      dest: /etc/httpd.conf
    – restart apache
  – name: ensure apache is running
      name: httpd
      state: started
    – name: restart apache
        name: httpd
        state: restarted

To give it a quick try save the file as webserver.yml and give it a run via ansible-playbook command

ansible-playbook -s playbooks/webserver.yml


The -s option instructs ansible to run play on remote server with super user (root) privileges.

The power of ansible is its modules, which are constantly growing over time a complete set of Ansible supported modules is in its official documenation.


There is a lot of things to say about playbooks, just to give the brief they have there own language like a  templates, tasks, handlers, a playbook could have one or multiple plays inside (for instance instructions for deployment of one or more services).

The downsides of playbooks are they're so hard to write from scratch and edit, because yaml syntaxing is much more stricter than a normal oldschool sysadmin configuration file.
I've stucked with problems with modifying and writting .yaml files and I should say the community in #ansible in was very helpful to help me debug the obscure errors.

yamllint (The YAML Linter tool) comes handy at times, when facing yaml syntax errors, to use it install via apt:

# apt-get install –yes yamllint

a) Running ansible in "dry mode" just show what ansible might do but not change anything

ansible-playbook playbooks/PLAYBOOK_NAME.yml –check

b) Running playbook with different users and separate SSH keys


ansible-playbook playbooks/your_playbook.yml –user ansible-user
ansible -m ping hosts –private-key=~/.ssh/keys/custom_id_rsa -u centos


c) Running ansible playbook only for certain hostnames part of a bigger host group


ansible-playbook playbooks/PLAYBOOK_NAME.yml –limit "host1,host2,host3"

d) Run Ansible on remote hosts in parallel

To run in raw of 10 hosts in parallel

# Run 10 hosts parallel
ansible-playbook <File.yaml> -f 10            

e) Passing variables to .yaml scripts using commandline

Ansible has ability to pre-define variables from .yml playbooks. This variables later can be passed from shell cli, here is an example:

# Example of variable substitution pass from command line the var in varsubsts.yaml if present is defined / replaced ansible-playbook playbooks/varsubst.yaml –extra-vars "myhosts=localhost gather=yes pkg=telnet"


4. Ansible Galaxy (A Docker Hub) like large repository with playbook (script) files


Ansible Galaxy has about 10000 active users which are contributing ansible automation playbooks in fields such as Development / Networking / Cloud / Monitoring / Database / Web / Security etc.

To install from ansible galaxy use ansible-galaxy

# install from galaxy the geerlingguy mysql playbook
ansible-galaxy install geerlingguy.mysql

The available packages you can use as a template for your purpose are not so much as with Puppet as Ansible is younger and not corporate supported like Puppet, anyhow they are a lot and does cover most basic sysadmin needs for mass deployments, besides there are plenty of other unofficial yaml ansible scripts in various github repos.

How to make Reverse SSH Tunnel to servers behind NAT

Thursday, October 11th, 2018


Those who remember the times of IRC chatting long nights and the need to be c00l guy and enter favorite IRC server through a really bizarre hostname, you should certainly remember the usefulness of Reverse SSH Tunnels to appear in IRC /whois like connecting from a remote host (mask yourself) from other IRC guys where are you physically.

The idea of Reverse SSH is to be able to SSH (or other protocols) connect to IPs that are situated behind a NAT server/s.
Creating SSH Reverse Tunnel is an easy task and up to 2 simple SSH commands

To better explain how SSH tunnel is achieved, here is a scenario:

A. Linux host behind NAT IP: (Destination host)
B. (Source Host) of Machine with External Public Internet IP through which SSH Tunnel will be established to

1. Create SSH Revere SSH from Destination to Source host (with Public IP)

Connect to the remote machine which has a real IP address and make port of the reverse SSH connection open (remove any firewall), lets say port 23000.

ssh -R 23000: -oPort=33

NB! On destination and source servers make sure you have enabled in /etc/ssh/sshd_config

AllowAgentForwarding yes
AllowTCPForwarding yes
PermitTunnel yes


2. Connect from Source IP to Destination through the established SSH tunnelling



Connecting to through ssh on 23000 will connect you to the back machine with the unreal IP address.

ssh local-username@ -p 23000

ssh -L 19999:localhost:19999 middleman@

If you want other server with hostname to access the Reverse SSH Tunneled server you can do it via external IP which in my case is

From just do:

 ssh username@


A text diagram of SSH Tunnel looks something like that:

Destination ( <- |NAT| <- Source ( <-


Above examples should work not only on Linux but on NetBSD / OpenBSD / FreeBSD or any other UNIX system with a modern SSH client installed.

Display Content of SSL certificate .pem file with openssl command

Thursday, October 11th, 2018


If you have generated a .pem formatted SSL certificate or you have multiple .pem SSL certificates and you're not sure which .pem file is generated for which domain / subdomain it is useful to Display content of SSL Certificate .PEM file with openssl command.

Viewing certificate's content is also very useful if you have hosted multiple websites hosted on a server and you want to check which of the SSLs assigned in the Virtualhosts has Expired (for example if you have domains that expire in short term period (365 days).

1. How to Display Content of SSL certificate .pem file?


root@pcfreak:~# openssl x509 -in cert.pem -text
        Version: 3 (0x2)
        Serial Number:
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
            Not Before: Oct 10 17:49:34 2018 GMT
            Not After : Jan  8 17:49:34 2019 GMT
        Subject: CN =

        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
            X509v3 Subject Key Identifier:
            X509v3 Authority Key Identifier:


            Authority Information Access:
                OCSP – URI:
                CA Issuers – URI:

            X509v3 Subject Alternative Name:
            X509v3 Certificate Policies:
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : E2:69:4B:AE:26:E8:E9:40:09:E8:86:1B:B6:3B:83:D4:
                    Timestamp : Oct 10 18:49:34.453 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                    Timestamp : Oct 10 18:49:34.451 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
    Signature Algorithm: sha256WithRSAEncryption


Same way a .der files content / encryption algorithm and domain name could be grasped.

root@pcfreak:~# openssl x509 -in cert.der -inform der -text
        Version: 3 (0x2)
        Serial Number:
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN =
            Not Before: Jun 22 04:00:37 2015 GMT
            Not After : Jul 22 04:00:37 2015 GMT

        Subject: CN =
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (512 bit)

                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
            X509v3 Authority Key Identifier:


            X509v3 Basic Constraints:
    Signature Algorithm: sha256WithRSAEncryption


2. How to display content and info about .CSR (Certificate Signing request)


root@pcfreak:~# openssl req -in cert.csr -noout -text

Certificate Request:
        Version: 1 (0x0)
        Subject: C = BG, ST = BG, L = Dobrich, O = Pc Freak, CN =, emailAddress =
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption

That's all folks 🙂


Preparing your Linux to work with the Cloud providers – Installing aws , gcloud, az, oc, cf CLI Cloud access command interfaces

Wednesday, October 10th, 2018

howto Install-Cloud-access-tools-for-google-aws-azure-openshift-cloud-foundryCloud_computing-explained-on-linux.svg

If you're a sysadmin / developer whose boss requires a migration of Stored Data, Database structures or Web Objects to Amazon Web Services / Google Clourd or you happen to be a DevOps Engineer you will certainly need to have installed as a minimumum amazon AWS and Google Clouds clients to do daily routines and script stuff in managing cloud resources without tampering to use the Web GUI interface.

Here is how to install the aws, gcloud, oc, az and cf next to your kubernetes client (kubectl) on your Linux Desktop.

1. Install Google Cloud  gcloud (to manage Google Cloud platform resources and developer workflow


Here is few cmds to run to install  gcloud, gcloud alpha, gcloud beta, gsutil, and bq commands to manage your Google Cloud from CLI

a.) On Debian / Ubuntu / Mint or any other deb based distro

# Create environment variable for correct distribution
export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)"


# Add the Cloud SDK distribution URI as a package source
# echo "deb $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list


# Import the Google Cloud Platform public key
$ sudo curl | sudo apt-key add –


# Update the package list and install the Cloud SDK
$ sudo apt-get update && sudo apt-get install google-cloud-sdk

b) On CentOS, RHEL, Fedora Linux and other rpm based ones

$ sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOM
name=Google Cloud SDK

# yum install google-cloud-sdk


That's all now the text client to talk to Google Cloud's API gcloud is installed under

Latest install instructions of Google Cloud SDK are here.

2. Install AWS Cloud command line interface tool for managing AWS (Amazon Web Services)


AWS client is dependent on Python PIP so before you proceed you will have to install python-pip deb package if on Debian / Ubuntu Linux use apt:


# apt-get install –yes python-pip


It is also possible to install newest version of PIP a tiny shell script provided by Amazon


# curl -O
# python –user


# pip install awscli –upgrade –user


3. Install Azure Cloud Console access CLI command interface


On Debian / Ubuntu or any other deb based distro:

# AZ_REPO=$(lsb_release -cs)
# echo "deb [arch=amd64] $AZ_REPO main" | \
$ sudo tee /etc/apt/sources.list.d/azure-cli.list

# curl -L | sudo apt-key add –
$ sudo apt-get update
$ sudo apt-get install apt-transport-https azure-cli


Finaly to check that Azure CLI is properly installed run simple login with:


$ az login


$ sudo rpm –import
$ sudo sh -c 'echo -e "[azure-cli]\nname=Azure CLI\nbaseurl=\nenabled=1\ngpgcheck=1\ngpgkey=" > /etc/yum.repos.d/azure-cli.repo'
$ sudo yum install azure-cli

$ az login

For Latest install instructions check Amazon's documentation here

4. Install OpenShift OC CLI tool to access OpenShift Open Source Cloud



Even thought OpenShift has its original Redhat produced package binaries, if you're not on RPM distro it is probably
best to install using official latest version from openshift github repo.

As of time of writting this article this is done with:


# wget
tar –xvf openshift-origin-client-tools-v1.5.1-7b451fc-linux-64bit.tar.gz


# # mv openshift-origin-client-tools-v1.5.1-7b451fc-linux-64bit oc-tool


# cd oc-tool
# echo'export PATH=$HOME/oc-tool:$PATH' >> ~/.bashrc


To test openshift, try to login to OpenShift cloud:


$ oc login
Server [https://localhost:8443]: https://128.XX.XX.XX:8443

Latest install instructions on OC here

5. Install Cloud Foundry cf CLI Cloud access tool


a) On Debian / Ubuntu Linux based distributions, do run:


$ wget -q -O – | sudo apt-key add –
$ echo "deb stable main" | sudo tee /etc/apt/sources.list.d/cloudfoundry-cli.list
$ sudo apt-get update
$ sudo apt-get install cf-cli


b) On RHEL Enterprise Linux / CentOS and Fedoras


$ sudo wget -O /etc/yum.repos.d/cloudfoundry-cli.repo
$ sudo yum install cf-cli

For latest install insructions on cf cli check Cloud Foundry's install site

There plenty of other Cloud providers with the number exponentially growing and most have their own custom cli tools to access but as there use is not so common as the 5 ones mentioned below, I've omited 'em. If you're interested to know the complete list of Cloud Providers providing Cloud Services check here.

6. Install Ruby GEMs RHC tools collection

If you have to work with Redhat Cloud Storage / OpenShift you will perhaps want to install also (RHC) Redhat Collection Tools.

Assuming that the Linux system is running an up2date version of ruby programming language do run:



root@jeremiah:~# gem install rhc
Fetching: net-ssh-5.0.2.gem (100%)
Successfully installed net-ssh-5.0.2
Fetching: net-ssh-gateway-2.0.0.gem (100%)
Successfully installed net-ssh-gateway-2.0.0
Fetching: net-ssh-multi-1.2.1.gem (100%)
Successfully installed net-ssh-multi-1.2.1
Fetching: minitar-0.7.gem (100%)
The `minitar` executable is no longer bundled with `minitar`. If you are
expecting this executable, make sure you also install `minitar-cli`.
Successfully installed minitar-0.7
Fetching: hashie-3.6.0.gem (100%)
Successfully installed hashie-3.6.0
Fetching: powerbar-1.0.18.gem (100%)
Successfully installed powerbar-1.0.18
Fetching: minitar-cli-0.7.gem (100%)
Successfully installed minitar-cli-0.7
Fetching: archive-tar-minitar-0.6.1.gem (100%)
'archive-tar-minitar' has been deprecated; just install 'minitar'.
Successfully installed archive-tar-minitar-0.6.1
Fetching: highline-1.6.21.gem (100%)
Successfully installed highline-1.6.21
Fetching: commander-4.2.1.gem (100%)
Successfully installed commander-4.2.1
Fetching: httpclient- (100%)
Successfully installed httpclient-
Fetching: open4-1.3.4.gem (100%)
Successfully installed open4-1.3.4
Fetching: rhc-1.38.7.gem (100%)


If this is your first time installing the RHC tools, please run 'rhc setup'

Successfully installed rhc-1.38.7
Parsing documentation for net-ssh-5.0.2
Installing ri documentation for net-ssh-5.0.2
Parsing documentation for net-ssh-gateway-2.0.0
Installing ri documentation for net-ssh-gateway-2.0.0
Parsing documentation for net-ssh-multi-1.2.1
Installing ri documentation for net-ssh-multi-1.2.1
Parsing documentation for minitar-0.7
Installing ri documentation for minitar-0.7
Parsing documentation for hashie-3.6.0
Installing ri documentation for hashie-3.6.0
Parsing documentation for powerbar-1.0.18
Installing ri documentation for powerbar-1.0.18
Parsing documentation for minitar-cli-0.7
Installing ri documentation for minitar-cli-0.7
Parsing documentation for archive-tar-minitar-0.6.1
Installing ri documentation for archive-tar-minitar-0.6.1
Parsing documentation for highline-1.6.21
Installing ri documentation for highline-1.6.21
Parsing documentation for commander-4.2.1
Installing ri documentation for commander-4.2.1
Parsing documentation for httpclient-
Installing ri documentation for httpclient-
Parsing documentation for open4-1.3.4
Installing ri documentation for open4-1.3.4
Parsing documentation for rhc-1.38.7
Installing ri documentation for rhc-1.38.7
Done installing documentation for net-ssh, net-ssh-gateway, net-ssh-multi, minitar, hashie, powerbar, minitar-cli, archive-tar-minitar, highline, commander, httpclient, open4, rhc after 10 seconds
13 gems installed

To start with rhc next do:

rhc setup
rhc app create my-app diy-0.1

and play with it to install software create services on the Redhat cloud.




This are just of the few of the numerous tools available and I definitely understand there is much more to be said on the topic.
If you can remember other tools tor interesting cloud starting up tips about stuff to do on a fresh installed Linux PC to make life easier with Cloud / PaaS / SaaS / DevOps engineer please drop a comment.

Install Java on Debian 9 Linux howto

Saturday, September 29th, 2018


As most system administrators and perhaps most people :), I dislike Java Virtual Machine. However because of its multi-platform support it is largely adopted and so many things are already written in java, even though I hate it I need it to run things every now and then on my personal desktop machine with Debian Linux 9 (Stretch).

From a programmer point of view Java applications are scalable and flexible and from a point of view a person who had to support computers and servers iwth Java it sucks.
To have a running Java Virtual Machine and run Java applications on your Linux PC you could use JRE (the Java Runtime Environment) and JDK (Java Development Kit) which is a set of Java tools and compilers to translate Java code to a .JAR .WAR and the rest of the Java Machine running formats.

OpenJDK (Open JDK) is free (open source) implementation of Oracle Sun Microsystems of Java SE 7 mostly licensed under GPLv2 (but with some linking to a Java class library that is not truly free. OpenJDK includes as components the backend Virtual Machine (HotSpot), the Java Class Library, javac (the java compiler) and IcedTea (which is Redhat's free implementation of Java Web Start plugin.

Install OpenJDK 8 JDK and JRE

OpenJDK is installable by default on Debian and most other distros, to install it on Debian:


linux:~# apt-get install –yes -qq default-jdk
linux:~# apt-get install –yes -qq default-jre


That would provide your system with support for  java-sdk, java2-sdk, java5-sdk, java6-sdk, java7-sdk, java8-sdk's free implementation.



linux:~#  java -version



linux:~# java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)



I have used openjdk but as there are issues with some Java programs because of Java compitability issues. Nowadays most of the timeI choose to usually install the Official Latest Oracle Java 8. The reason is I often have to install on servers application servers such as:

  •  Tomcat
  • JBoss
  • WildFly
  • Jetty
  • Glassfish
  • WebLogic
  • Cassandra
  • Jenkins

Install Latest Official Oracle Java 8


1. Download Oracle Java installable binary

To download latest official release check out Oracle's download page and copy the link to latest Java archive and select the appropriate architecture x64 / 32 bit / arm etc., as of time of writting this article latest Stable Java version is JDK-8U181.


linux:~# curl -L -b "oraclelicense=a" -O


Notice the -b "oraclelicense=a" which will pass on to the remote web server accept Oracle's license which is a prerequirement to continue to download.

2. Create directory for Java


linux:~# mkdir -p /usr/local/oracle-java-8


3. Extract the Java .tar.gz to /usr/local/oracle/java-8 or another directory
where java will live


linux:~# tar -zxf jdk-8u181-linux-x64.tar.gz -C /usr/local/oracle-java-8


If you have used Java OpenJDK beforehand and installed Oracle Java according to the instructions above you might end up with multiple Javas installed by default, however Debian Linux will have a symlinks to java javac (java compiler), javaws (Java web start).
Thus just executed java will return


linux:~# java -version


java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)


4. Set Oracle Java to be default for the Debian system


linux:~# update-alternatives –install "/usr/bin/java" "java" "/usr/local/oracle-java-8/jdk1.8.0_181/bin/java" 1500
linux:~# update-alternatives –install "/usr/bin/javac" "javac" "/usr/local/oracle-java-8/jdk1.8.0_181/bin/javac" 1500
linux:~# update-alternatives –install "/usr/bin/javaws" "javaws" "/usr/local/oracle-java-8/jdk1.8.0_181/bin/javaws" 1500


An alternative way to set Java to point to Oracle Java instead of OpenJDK is


linux:~# update-alternatives –config java


There are 2 choices for the alternative java (providing /usr/bin/java).


  Selection    Path                                            Priority   Status
* 0            /usr/local/oracle-java-8/jdk1.8.0_181/bin/java   1500      auto mode
  1            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode
  2            /usr/local/oracle-java-8/jdk1.8.0_181/bin/java   1500      manual mode


Press <enter> to keep the current choice[*], or type selection number:

Install Latest Official Oracle Java 10

To install Oracle Java the process is the same:

linux:~# wget –header "Cookie: oraclelicense=accept-securebackup-cookie" <paste the download link here> linux:~# mkdir /usr/local/oracle-java-10/ linux:~# tar xvzf jdk-10.0.1_linux-x64_bin.tar.gz -C/usr/local/oracle-java-10 linux:~# update-alternatives –install "/usr/bin/java" "java" "/usr/local/oracle-java-10/jdk-10.0.1/bin/java" 1500 linux:~# update-alternatives –install "/usr/bin/javac" "javac" "/usr/local/oracle-java-10/jdk-10.0.1/bin/javac" 1500 linux:~# update-alternatives –install "/usr/bin/javaws" "javaws" "/usr/local/oracle-java-10/jdk-10.0.1/bin/javaws" 1500


Uninstall OpenJDK previous installation

If you choose to use Oracle Java in favor of OpenJDK you might want to also remove the openjdk not to take space on your Hard disk, to so so:

linux:~# apt-get remove default-jre

Create user and password on Linux non interactive and add it to sudo a tiny Dev Ops script

Thursday, September 20th, 2018

A common task for SysAdmins who managed a multitude of servers remotely via Secure Shell was to add a user and assign password by using a script, this was sometimes necessery to set-up some system users and create access for university users on 10 / 20 testing Linux servers.

Nowadays this task of adding user to a list of remote servers and granting the new user superuser permissions through /etc/sudoers is practiced heavily by the so called Dev Ops (Just another Buziness Word for Senior System Admiistrators with good scripting skills and a little bit of development experience – same game different name.

The Dev Ops System Integration Engineers use this useful add non-interactive user via SSH in Cloud environments in order to prepare superuser (root permissioned through /etc/sudoers) user, that is later be used for lets say deployment on a few hundred of servers of lets say LAMP (Linux + Apache + MySQL + PHP) or LEMP (Linux NGINX MySQL PHP) or Software Load Balancer HAProxy  balacing for MySQL clusters / Nginx Application servers / JIRAs etc, through a Playbook script with some deployment automation tool such as Ansible.

Well enough talk here is the few lines of code which does create a user locally:

linux:~# apt-get install –yes sudo
linux:~# useradd devops –home /home/devops -s /bin/bash
linux:~# mkdir /home/devops
linux:~# chown -R devops:devops /home/devops
linux:~# echo 'username:testpass' | chpasswd

Though this lines could be invoked easily by passing it as arguments via ssh it is often unhandy to run them on remote host, because some of the remote hosts against executed, might have already the user existent with granted permissions for sudo

Thus a much better way to do things is use below script and first upload it to remote servers by running the scp command in a loop:

while read line; do
scp  root@$i:/root/
ssh ""
done < servers_list.txt

Where servers_list.txt contains a list of remote IPs:

# Create new user/group and add nopasswd login to sudoers
# Author: Georgi Georgiev
# has to be run sa root – sudo devops



check_install_sudo ()  {
if [ $(dpkg –get-selections | cut -f1|grep -E ‘^sudo’) ]; then
apt-get install –yes sudo
        printf "Nothing to do sudo installed";

check_install_user () {

if [ “$(sed -n “/$u_id/p” /etc/passwd|wc -l)” -eq 0 ]; then
apt-get install –yes sudo
apt-get install –yes sudo
useradd $u_id –home /home/$u_id
mkdir /home/$u_id
chown -R $u_id:$g_id /home/$u_id
echo "$u_id:$pass" | chpasswd
cp -rpf /etc/bash.bashrc /home/$u_id
if [ “$(sed -n “/$u_id/p” $sudoers_f|wc -l)” -eq “0” ]; then
echo "$u_id ALL=(ALL) NOPASSWD: ALL" >> $sudoers_f
        echo "$u_id existing. Exiting ..";
        exit 1;

        echo "Will do nothing because $u_id exists";



By the way this task was the simplest task given by a Company where I applied for a Dev Ops System Engineer, so I hope this will help someone else too.

P.S. If you prefer Shell scripts (even though much harder, time consuming etc.) as a mean of automation as an alternative to Ansible / Chef I suggest you check out and perhaps try to do the task with 🙂