Posts Tagged ‘Old’

Debugging routing and network issues on Linux common approaches. A step by step guide to find out why routing or network service fails

Thursday, November 30th, 2023

For system administrators having a Network issue is among the Hell-ish stuff that can happen every now and then. That is especially true in Heterogenous / Hybrid and complicated Network topologies (with missing well crafted documentation), that were build without an initial overview "on the fly".
Such a networking connectivity or routing issues are faced by every novice, mid or even expert system administrators as the Company's Network IT environments are becoming more and more complicated day by day.

When the "Disaster" of being unable to connect two servers or at times  home laptops / PCs to see each other even though on the Physical layer / Transport Layer (Hardware such as external Switches / Routers / Repeaters / Cabling etc.) is Present machines are connected and everything on the 1 Physical Layer from OSI layears is present happens, then it is time to Debug it with some software tools and methods.

To each operating system the tools and methods to test networking connection and routings is a bit different but generally speaking most concepts are pretty much the same across different types of operating systems (Linux ditros / OpenBSD / FreeBSD / Mac OS / Android / iOS / HP-UX / IBM AIX / DOS / Windows etc.).

Debugging network issues across separate operating systems has its variations but in this specific (ideas) are much close to this article. As the goal at that guide will be to point out how to debug network issues on Linux, in future if I have the time or need to debug other OS-es from Linux, I'll try to put an article on how to debug Network issues on Windows when have some time to do it.

Consider to look for the issue following the basic TCP / IP OSI Level model, every system administrator should have idea about it already, it is part of most basic networking courses such as Cisco's CCNA

TCPIP_OSI_model-networking-levels

1. Check what is the Link status of the Interface with ethtool
 

root@freak:~# ethtool eno1
Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 100Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    MDI-X: on (auto)
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

 

root@freak:~# ethtool eno2
Settings for eno2:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    MDI-X: on (auto)
    Supports Wake-on: pumbg
    Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
    Link detected: yes

 

For example lets check only if Cable of Network card is plugged in and detected to have a network connection to remote node or switch and show the connection speed on which the 'autoneg on' (autonegiation option) of the LAN card has detected the network exat maximum speed:

root@pcfreak:~# ethtool eth0|grep -i 'link detected'; ethtool eth0 |grep 'Speed: '
    Link detected: yes
    Speed: 100Mb/s


1. Check ip command network configuration output

root@freak:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 1000
    link/ether 70:e2:84:13:44:15 brd ff:ff:ff:ff:ff:ff
    altname enp7s0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 1000
    link/ether 70:e2:84:13:44:17 brd ff:ff:ff:ff:ff:ff
    altname enp8s0
4: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 70:e2:84:13:44:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.7/24 brd 192.168.1.255 scope global dynamic xenbr0
       valid_lft 7361188sec preferred_lft 7361188sec
5: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 70:e2:84:13:44:15 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.5/24 brd 192.168.0.255 scope global dynamic xenbr1
       valid_lft 536138sec preferred_lft 536138sec
10: vif2.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
11: vif2.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
12: vif3.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
13: vif3.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
14: vif4.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
15: vif4.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
16: vif5.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
17: vif5.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
18: vif6.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
19: vif6.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
30: vif17.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
31: vif17.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
34: vif21.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr0 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
35: vif21.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master xenbr1 state UP group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
48: vif25.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UNKNOWN group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
49: vif25.1-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr1 state UNKNOWN group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
50: vif25.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
51: vif25.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr1 state DOWN group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
118: vif47.0-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr0 state UNKNOWN group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
119: vif47.1-emu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master xenbr1 state UNKNOWN group default qlen 1000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
120: vif47.0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr0 state DOWN group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
121: vif47.1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master xenbr1 state DOWN group default qlen 2000
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
root@freak:~# 

ip a s (is a also a shortcut command alias) you can enjoy if you have to deal with ip command frequently.

2. Check the status of the interfaces

Old fashioned way is to just do:

/sbin/ifconfig

 

root@freak:~# ifconfig 
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 70:e2:84:13:44:15  txqueuelen 1000  (Ethernet)
        RX packets 52366502  bytes 10622469320 (9.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 242622195  bytes 274688121244 (255.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfb200000-fb27ffff  

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 70:e2:84:13:44:17  txqueuelen 1000  (Ethernet)
        RX packets 220995454  bytes 269698276095 (251.1 GiB)
        RX errors 0  dropped 7  overruns 0  frame 0
        TX packets 192319925  bytes 166233773782 (154.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfb100000-fb17ffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2553  bytes 147410 (143.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2553  bytes 147410 (143.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif17.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 14517375  bytes 133226551792 (124.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 139688950  bytes 145111993017 (135.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif17.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 86113294  bytes 156944058681 (146.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 181513904  bytes 267892940821 (249.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 1521875  bytes 88282472 (84.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 152691174  bytes 278372314505 (259.2 GiB)
        TX errors 0  dropped 3 overruns 0  carrier 0  collisions 0

vif2.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 454915  bytes 81069760 (77.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 266953989  bytes 425692364876 (396.4 GiB)
        TX errors 0  dropped 26 overruns 0  carrier 0  collisions 0

vif21.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 20043711  bytes 1283926794 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 141580485  bytes 277396881113 (258.3 GiB)
        TX errors 0  dropped 3 overruns 0  carrier 0  collisions 0

vif21.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 73004  bytes 3802174 (3.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 267151006  bytes 425621892663 (396.3 GiB)
        TX errors 0  dropped 14 overruns 0  carrier 0  collisions 0

vif25.0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif25.1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif25.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 2736348  bytes 295661367 (281.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 260385509  bytes 265751226663 (247.5 GiB)
        TX errors 0  dropped 200 overruns 0  carrier 0  collisions 0

vif25.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 145387  bytes 36011655 (34.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 370314760  bytes 394725961081 (367.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 55382861  bytes 130042280927 (121.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99040097  bytes 147929196318 (137.7 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

vif3.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 5132631  bytes 295493762 (281.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 262314199  bytes 425416945203 (396.2 GiB)
        TX errors 0  dropped 16 overruns 0  carrier 0  collisions 0

vif4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 4902015  bytes 615387539 (586.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 149342891  bytes 277802504143 (258.7 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

vif4.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 276927  bytes 30720101 (29.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 267132395  bytes 425745668273 (396.5 GiB)
        TX errors 0  dropped 14 overruns 0  carrier 0  collisions 0

vif47.0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif47.1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif47.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 208745  bytes 20096596 (19.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 110905731  bytes 110723486135 (103.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif47.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 140517  bytes 14596061 (13.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 150831959  bytes 162931572456 (151.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 2030528  bytes 363988589 (347.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 152264264  bytes 278131541781 (259.0 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

vif5.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 4169244  bytes 1045889687 (997.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 263561100  bytes 424894400987 (395.7 GiB)
        TX errors 0  dropped 7 overruns 0  carrier 0  collisions 0

vif6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 300242  bytes 16210963 (15.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 153909576  bytes 278461295620 (259.3 GiB)
        TX errors 0  dropped 2 overruns 0  carrier 0  collisions 0

vif6.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 43  bytes 1932 (1.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 154205631  bytes 278481298141 (259.3 GiB)
        TX errors 0  dropped 2 overruns 0  carrier 0  collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.8  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 70:e2:84:13:44:11  txqueuelen 1000  (Ethernet)
        RX packets 13689902  bytes 923464162 (880.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12072932  bytes 1307055530 (1.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.3  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 70:e2:84:13:44:12  txqueuelen 1000  (Ethernet)
        RX packets 626995  bytes 180026901 (171.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12815  bytes 942092 (920.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

root@freak:~# ifconfig        
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 70:e2:84:13:44:11  txqueuelen 1000  (Ethernet)
        RX packets 52373358  bytes 10623034427 (9.8 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 242660000  bytes 274734018669 (255.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfb200000-fb27ffff  

eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 70:e2:84:13:44:12  txqueuelen 1000  (Ethernet)
        RX packets 221197892  bytes 269978137472 (251.4 GiB)
        RX errors 0  dropped 7  overruns 0  frame 0
        TX packets 192573206  bytes 166491370299 (155.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfb100000-fb17ffff  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2553  bytes 147410 (143.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2553  bytes 147410 (143.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif17.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 14519247  bytes 133248290251 (124.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 139708738  bytes 145135168676 (135.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif17.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 86206104  bytes 157189755115 (146.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 181685983  bytes 268170806613 (249.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 1522072  bytes 88293701 (84.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 152712638  bytes 278417240910 (259.2 GiB)
        TX errors 0  dropped 3 overruns 0  carrier 0  collisions 0

vif2.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 454933  bytes 81071616 (77.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 267218860  bytes 426217224334 (396.9 GiB)
        TX errors 0  dropped 26 overruns 0  carrier 0  collisions 0

vif21.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 20045530  bytes 1284038375 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 141601066  bytes 277441739746 (258.3 GiB)
        TX errors 0  dropped 3 overruns 0  carrier 0  collisions 0

vif21.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 73010  bytes 3802474 (3.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 267415889  bytes 426146753845 (396.8 GiB)
        TX errors 0  dropped 14 overruns 0  carrier 0  collisions 0

vif25.0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif25.1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif25.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 2736576  bytes 295678097 (281.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 260429831  bytes 265797660906 (247.5 GiB)
        TX errors 0  dropped 200 overruns 0  carrier 0  collisions 0

vif25.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 145425  bytes 36018716 (34.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 370770440  bytes 395263409640 (368.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 55392503  bytes 130064444520 (121.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 99052116  bytes 147951838129 (137.7 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

vif3.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 5133054  bytes 295517366 (281.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 262578665  bytes 425941777243 (396.6 GiB)
        TX errors 0  dropped 16 overruns 0  carrier 0  collisions 0

vif4.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 4902949  bytes 615496460 (586.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 149363618  bytes 277847322538 (258.7 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

vif4.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 276943  bytes 30721141 (29.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 267397268  bytes 426270528575 (396.9 GiB)
        TX errors 0  dropped 14 overruns 0  carrier 0  collisions 0

vif47.0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif47.1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif47.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 208790  bytes 20100733 (19.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 110950236  bytes 110769932971 (103.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif47.1-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 1000  (Ethernet)
        RX packets 140551  bytes 14599509 (13.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 151287643  bytes 163469024604 (152.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif5.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 2030676  bytes 363997181 (347.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 152285777  bytes 278176471509 (259.0 GiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

vif5.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 4169387  bytes 1045898303 (997.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 263825846  bytes 425419251935 (396.2 GiB)
        TX errors 0  dropped 7 overruns 0  carrier 0  collisions 0

vif6.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 300266  bytes 16212271 (15.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 153931212  bytes 278506234302 (259.3 GiB)
        TX errors 0  dropped 2 overruns 0  carrier 0  collisions 0

vif6.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fe:ff:ff:ff:ff:ff  txqueuelen 2000  (Ethernet)
        RX packets 43  bytes 1932 (1.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 154227291  bytes 278526238467 (259.3 GiB)
        TX errors 0  dropped 2 overruns 0  carrier 0  collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.8  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 70:e2:84:13:44:11  txqueuelen 1000  (Ethernet)
        RX packets 13690768  bytes 923520126 (880.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12073667  bytes 1307127765 (1.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.3  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 70:e2:84:13:44:12  txqueuelen 1000  (Ethernet)
        RX packets 627010  bytes 180028847 (171.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12815  bytes 942092 (920.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

To see ethernet interfaces that seem up and then do a ifconfig -a to check whether some interfaces are down (e.g. not shown in the simple ifconfig list).
/sbin/ifconfig -a

! Please note that some virtual IP configurations might not appear and noly be visible in an (ip addr show) command.

 

3. Check iproute2 for special rt_tables (Routing Tables) rules
 

By default Linux distributions does not have any additional rules in /etc/iproute2/rt_tables however some Linux router machines, needs to have a multiple Gateways. Perhaps the most elegant way to do multiple routings with Linux is to use iproute2's routing tables rt_tables.

Here is example of an OpenXEN system that has 2 Internet providers attached and routes different traffic via

 

root@freak:~# cat /etc/iproute2/rt_tables
#
# reserved values
#
255    local
254    main
253    default

100    INET1
200     INET2
0    unspec
#
# local
#
#1    inr.ruhep

 

root@freak:~# ip rule list
0:    from all lookup local
32762:    from all to 192.168.1.8 lookup INET2
32763:    from 192.168.1.8 lookup INET2
32764:    from all to 192.168.0.3 lookup INET1
32765:    from 192.168.0.3 lookup INET1
32766:    from all lookup main
32767:    from all lookup default
root@freak:~# 
 

4. Using ip route get to find out traffic route (path)

root@freak:~# ip route get 192.168.0.1
192.168.0.1 via 192.168.0.1 dev xenbr1 src 192.168.0.3 uid 0 
    cache 

 

root@freak:~# /sbin/route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 xenbr0
192.168.0.0     192.168.0.1     255.255.255.0   UG    0      0        0 xenbr1
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 xenbr1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 xenbr0
root@freak:~# 

root@freak:~# ip route show
default via 192.168.1.1 dev xenbr0 
192.168.0.0/24 via 192.168.0.1 dev xenbr1 
192.168.0.0/24 dev xenbr1 proto kernel scope link src 192.168.0.3 
192.168.1.0/24 dev xenbr0 proto kernel scope link src 192.168.1.8 


If you find that gateway is missing you might want to add it with:

root@freak:~#  ip route add default via 192.168.5.1

If you need to add a speicic network IP range via separate gateways, you can use commands like:

To add routing for 192.168.0.1/24 / 192.168.1.1/24 via 192.168.0.1 and 192.168.1.1

# /sbin/route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.1.1 dev eth1
# /sbin/route add -net 192.168.0.0 netmask 255.255.255.0 gw 192.168.0.1 dev eth1

 

If you need to delete a configured wrong route with ip command

# ip route del 192.168.1.0/24 via 0.0.0.0 dev eth1
# ip route del 192.168.0.0/24 via 0.0.0.0 dev eth1

5. Use ping (ICMP protocol) the Destionation IP
 

root@freak:~# ping -c 3 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.219 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.295 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.270 ms

— 192.168.0.1 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2048ms
rtt min/avg/max/mdev = 0.219/0.261/0.295/0.031 ms
root@freak:~# ping -c 3 192.168.0.39
PING 192.168.0.39 (192.168.0.39) 56(84) bytes of data.
From 192.168.1.80: icmp_seq=2 Redirect Host(New nexthop: 192.168.0.39)
From 192.168.1.80: icmp_seq=3 Redirect Host(New nexthop: 192.168.0.39)
From 192.168.1.80 icmp_seq=1 Destination Host Unreachable


— 192.168.0.39 ping statistics —
3 packets transmitted, 0 received, +1 errors, 100% packet loss, time 2039ms
pipe 3

 

Note that sometimes you might get 100% traffic loss but still have connection to the destionation in case if the ICMP protocol is filtered for security.

However if you get something like Network is unreachable that is usually an indicator of some routing problem or wrongly configured network netmask.

root@freak:~# ping 192.168.0.5
ping: connect: Network is unreachable

Test network with different packet size. To send 8972 bytes of payload in a Ethernet frame without fragmentation, the following command can be used:

root@pcfreak:~# ping -s 8972 -M do -c 4 freak
PING xen (192.168.1.8) 8972(9000) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
^C
— xen ping statistics —
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2037ms

root@pcfreak:~# 


 -M pmtudisc_opt
           Select Path MTU Discovery strategy.  pmtudisc_option may be either do (prohibit fragmentation, even local one), want (do PMTU discovery, fragment locally when packet size is
           large), or dont (do not set DF flag).

 

root@pcfreak:~# ping -s 8972 -M want -c 4 freak
PING xen (192.168.1.8) 8972(9000) bytes of data.
8980 bytes from xen (192.168.1.5): icmp_seq=1 ttl=64 time=2.18 ms
8980 bytes from xen (192.168.1.5): icmp_seq=2 ttl=64 time=1.90 ms
8980 bytes from xen (192.168.1.5): icmp_seq=3 ttl=64 time=2.10 ms
^C
— xen ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.901/2.059/2.178/0.116 ms

root@pcfreak:~# 

  • -M do: prohibit fragmentation
  • -s 8972 8972 bytes of data
  • ICMP header: 8 bytes
  • IP header: 20 bytes (usually, it can be higher)
  • 8980 bytes of bytes is the IP payload
     

These commands can be used to capture for MTU (maximum transmition units) related issues between hosts that are preventing for hosts to properly send traffic between themselves.
A common issue for Linux hosts to be unable to see each other on the same network is caused by Jumbo Frames (MTU 9000) packets enabled on one of the sides and MTU of 1500 on the other side.
Thus it is always a good idea to thoroughully look up all configured MTUs for all LAN Devices on each server.

6. Check traceroute path to host

If there is no PING but ip route get shows routing is properly configured and the routes existing in the Linux machine routing tables, next step is to check the output of traceroute / tracepath / mtr

 

raceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets
 1  pcfreak (192.168.0.1)  0.263 ms  0.166 ms  0.119 ms
root@freak:~# tracepath 192.168.1.1
 1?: [LOCALHOST]                      pmtu 1500
 1:  vivacom-gigabit-router                                0.925ms reached
 1:  vivacom-gigabit-router                                0.835ms reached
     Resume: pmtu 1500 hops 1 back 1 

 

It might be useful to get a frequent output of the command (especially on Linux hosts) where mtr command is not installed with:

 

root@freak:~# watch -n 0.1 traceroute 192.168.0.1

 

root@freak:~# traceroute -4 google.com
traceroute to google.com (172.217.17.110), 30 hops max, 60 byte packets
 1  vivacom-gigabit-router (192.168.1.1)  0.657 ms  1.280 ms  1.647 ms
 2  213.91.190.130 (213.91.190.130)  7.983 ms  8.168 ms  8.097 ms
 3  * * *
 4  * * *
 5  212-39-66-222.ip.btc-net.bg (212.39.66.222)  16.613 ms  16.336 ms  17.151 ms
 6  * * *
 7  142.251.92.65 (142.251.92.65)  18.808 ms  13.246 ms 209.85.254.242 (209.85.254.242)  15.541 ms
 8  142.251.92.3 (142.251.92.3)  14.223 ms 142.251.227.251 (142.251.227.251)  14.507 ms 142.251.92.3 (142.251.92.3)  15.328 ms
 9  ams15s29-in-f14.1e100.net (172.217.17.110)  14.097 ms  14.909 ms 142.251.242.230 (142.251.242.230)  13.481 ms
root@freak:~# 

If you have MTR then you can get plenty of useful additional information such as the Network HOP name or the Country location of the HOP.

 

To get HOP name:

 

root@freak:~# mtr -z google.com

 

To get info on where (which Country) exactly network HOP is located physically:

root@freak:~# mtr -y 2 google.com

 

7. Check iptables INPUT / FORWARD / OUTPUT rules are messing with something
 

# iptables -L -n 

# iptables -t nat -L -n


Ideally you would not have any firewall

# iptables -L -n 

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
 


In case if something like firewalld is enabled as a default serviceto provide some modern Linux firewall as Ubuntu and Redhat / CentOS / Fedoras has it often turned on as a service stop and disable the service

# systemctl stop firewalld

# systemctl disable firewalld

 

8. Debug for any possible MAC address duplicates
 

root@pcfrxen:~# arp -an
? (192.168.1.33) at 00:16:3e:59:96:9e [ether] on eth0
? (192.168.1.1) at 18:45:93:c6:d8:00 [ether] on eth1
? (192.168.0.1) at 8c:89:a5:f2:e8:d9 [ether] on eth1
? (192.168.1.1) at 18:45:93:c6:d8:00 [ether] on eth0
? (192.168.1.11) at 7c:0a:3f:89:b6:fa [ether] on eth1
? (192.168.1.17) at <incomplete> on eth0
? (192.168.1.37) at 00:16:3e:ea:05:ce [ether] on eth0
? (192.168.1.80) at 8c:89:a5:f2:e7:d8 [ether] on eth0
? (192.168.1.11) at 7c:0a:3f:89:a5:fa [ether] on eth0
? (192.168.1.30) at 00:16:3e:bb:46:45 [ether] on eth1
? (192.168.0.210) at 00:16:3e:68:d9:55 [ether] on eth1
? (192.168.1.30) at 00:16:3e:bb:46:45 [ether] on eth0
? (192.168.1.18) at 00:16:3e:0d:40:05 [ether] on eth1
? (192.168.0.211) at 00:16:3e:4d:41:05 [ether] on eth1
? (192.168.1.35) at 00:16:3e:d1:8f:77 [ether] on eth0
? (192.168.1.18) at 00:16:3e:0d:43:05 [ether] on eth0
? (192.168.1.28) at 00:16:3e:04:12:1c [ether] on eth1
? (192.168.0.3) at 70:e2:84:13:43:12 [ether] on eth1
? (192.168.0.208) at 00:16:3e:51:de:9c [ether] on eth1
? (192.168.0.241) at 00:16:3e:0d:48:06 [ether] on eth1
? (192.168.1.28) at 00:16:3e:04:12:1c [ether] on eth0
? (192.168.1.33) at 00:16:3e:59:97:8e [ether] on eth1
? (192.168.0.241) at 00:16:3e:0d:45:06 [ether] on eth0
? (192.168.0.209) at 00:16:3e:5c:df:96 [ether] on eth1

root@pcfrxen:~# ip neigh show
192.168.1.33 dev eth0 lladdr 00:16:3e:59:96:9e REACHABLE
192.168.1.1 dev eth1 lladdr 18:45:93:c6:d8:00 STALE
192.168.0.1 dev eth1 lladdr 8c:89:a5:f2:e8:d9 REACHABLE
192.168.1.1 dev eth0 lladdr 18:45:93:c6:d9:01 REACHABLE
192.168.1.11 dev eth1 lladdr 7c:0a:3f:89:a6:fb STALE
192.168.1.17 dev eth0  FAILED
192.168.1.37 dev eth0 lladdr 00:16:3e:ea:06:ce STALE
192.168.1.80 dev eth0 lladdr 8c:89:a5:f2:e8:d9 REACHABLE
192.168.1.11 dev eth0 lladdr 7c:0a:3f:89:a7:fa STALE
192.168.1.30 dev eth1 lladdr 00:16:3e:bb:45:46 STALE
192.168.0.210 dev eth1 lladdr 00:16:3e:68:d8:56 REACHABLE
192.168.1.30 dev eth0 lladdr 00:16:3e:bb:45:46 STALE
192.168.1.18 dev eth1 lladdr 00:16:3e:0d:48:04 STALE
192.168.0.211 dev eth1 lladdr 00:16:3e:4d:40:04 STALE
192.168.1.35 dev eth0 lladdr 00:16:3e:d2:8f:76 STALE
192.168.1.18 dev eth0 lladdr 00:16:3e:0d:48:06 STALE
192.168.1.28 dev eth1 lladdr 00:16:3e:04:11:2c STALE
192.168.0.3 dev eth1 lladdr 70:e2:84:13:44:13 STALE
192.168.0.208 dev eth1 lladdr 00:16:3e:51:de:9c REACHABLE
192.168.0.241 dev eth1 lladdr 00:16:3e:0d:48:07 STALE
192.168.1.28 dev eth0 lladdr 00:16:3e:04:12:1c REACHABLE
192.168.1.33 dev eth1 lladdr 00:16:3e:59:96:9e STALE
192.168.0.241 dev eth0 lladdr 00:16:3e:0d:49:06 STALE
192.168.0.209 dev eth1 lladdr 00:16:3e:5c:dd:97 STALE
root@pcfrxen:~# 


9. Check out with netstat / ss for any irregularities such as high amount of error of faulty ICMP / TCP / UDP network packs

 

For example check out the netstat network stack output

# netstat -s

 

root@pcfrxen:~# netstat -s
Ip:
    Forwarding: 2
    440044929 total packets received
    1032 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    439988902 incoming packets delivered
    396161852 requests sent out
    3 outgoing packets dropped
    100 dropped because of missing route
Icmp:
    1025 ICMP messages received
    540 input ICMP message failed
    ICMP input histogram:
        destination unreachable: 1014
        timeout in transit: 11
    519 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 519
IcmpMsg:
        InType3: 1014
        InType11: 11
        OutType3: 519
Tcp:
    1077237 active connection openings
    1070510 passive connection openings
    1398236 failed connection attempts
    111345 connection resets received
    83 connections established
    438293250 segments received
    508143650 segments sent out
    42567 segments retransmitted
    546 bad segments received
    329039 resets sent
Udp:
    1661295 packets received
    278 packets to unknown port received
    0 packet receive errors
    1545720 packets sent
    0 receive buffer errors
    0 send buffer errors
    IgnoredMulti: 33046
UdpLite:
TcpExt:
    1 invalid SYN cookies received
    1398196 resets received for embryonic SYN_RECV sockets
    1737473 packets pruned from receive queue because of socket buffer overrun
    1118775 TCP sockets finished time wait in fast timer
    638 time wait sockets recycled by time stamp
    656 packetes rejected in established connections because of timestamp
    2218959 delayed acks sent
    2330 delayed acks further delayed because of locked socket
    Quick ack mode was activated 7172 times
    271799723 packet headers predicted
    14917420 acknowledgments not containing data payload received
    171078735 predicted acknowledgments
    52 times recovered from packet loss due to fast retransmit
    TCPSackRecovery: 337
    Detected reordering 1551 times using SACK
    Detected reordering 1501 times using reno fast retransmit
    Detected reordering 61 times using time stamp
    9 congestion windows fully recovered without slow start
    38 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 241
    104 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 11550
    1 timeouts after reno fast retransmit
    TCPSackFailures: 13
    3772 fast retransmits
    2 retransmits in slow start
    TCPTimeouts: 24104
    TCPLossProbes: 101748
    TCPLossProbeRecovery: 134
    TCPSackRecoveryFail: 3
    128989224 packets collapsed in receive queue due to low socket buffer
    TCPBacklogCoalesce: 715034
    TCPDSACKOldSent: 7168
    TCPDSACKOfoSent: 341
    TCPDSACKRecv: 16612
    150689 connections reset due to unexpected data
    27063 connections reset due to early user close
    17 connections aborted due to timeout
    TCPDSACKIgnoredOld: 158
    TCPDSACKIgnoredNoUndo: 13514
    TCPSpuriousRTOs: 9
    TCPSackMerged: 1191
    TCPSackShiftFallback: 1011
    TCPDeferAcceptDrop: 699473
    TCPRcvCoalesce: 3311764
    TCPOFOQueue: 14289375
    TCPOFOMerge: 356
    TCPChallengeACK: 621
    TCPSYNChallenge: 621
    TCPSpuriousRtxHostQueues: 4
    TCPAutoCorking: 1605205
    TCPFromZeroWindowAdv: 132380
    TCPToZeroWindowAdv: 132441
    TCPWantZeroWindowAdv: 1445495
    TCPSynRetrans: 23652
    TCPOrigDataSent: 388992604
    TCPHystartTrainDetect: 69089
    TCPHystartTrainCwnd: 3264904
    TCPHystartDelayDetect: 4
    TCPHystartDelayCwnd: 128
    TCPACKSkippedPAWS: 3
    TCPACKSkippedSeq: 2001
    TCPACKSkippedChallenge: 2
    TCPWinProbe: 123043
    TCPKeepAlive: 4389
    TCPDelivered: 389507445
    TCPAckCompressed: 7343781
    TcpTimeoutRehash: 23311
    TcpDuplicateDataRehash: 8
    TCPDSACKRecvSegs: 17335
IpExt:
    InMcastPkts: 145100
    OutMcastPkts: 9429
    InBcastPkts: 18226
    InOctets: 722933727848
    OutOctets: 759502627470
    InMcastOctets: 58227095
    OutMcastOctets: 3284379
    InBcastOctets: 1756918
    InNoECTPkts: 440286946
    InECT0Pkts: 936

 

  • List all listening established connections to host

# netstat -ltne

  • List all UDP / TCP connections

# netstat -ltua

or if you prefer to do it with the newer and more comprehensive tool ss:
 

  • List all listening TCP connections 

# ss -lt

  • List all listening UDP connections 

# ss -ua

  • Display statistics about recent connections

root@pcfrxen:~# ss -s
Total: 329
TCP:   896 (estab 70, closed 769, orphaned 0, timewait 767)

Transport Total     IP        IPv6
RAW      0         0         0        
UDP      40        36        4        
TCP      127       118       9        
INET      167       154       13       
FRAG      0         0         0 

  • If you need to debug some specific sport or dport filter out the connection you need by port number

# ss -at '( dport = :22 or sport = :22 )'

 

Debug for any possible issues with ICMP unreachable but ports reachable with NMAP / telnet / Netcat
 

# nc 192.168.0.1 -vz

root@pcfrxen:/ # nc 192.168.0.1 80 -vz
pcfreak [192.168.0.1] 80 (http) open


root@pcfrxen:/ # nc 192.168.0.1 5555 -vz
pcfreak [192.168.0.1] 5555 (?) : Connection refused

 

root@pcfrxen:/# telnet 192.168.0.1 3128
Trying 192.168.0.1…
Connected to 192.168.0.1.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

 

root@pcfrxen:/# nmap -sS -P0 192.168.0.1 -p 443 -O
Starting Nmap 7.80 ( https://nmap.org ) at 2023-11-27 19:51 EET
Nmap scan report for pcfreak (192.168.0.1)
Host is up (0.00036s latency).

PORT    STATE SERVICE
443/tcp open  https
MAC Address: 8C:89:A5:F2:E8:D8 (Micro-Star INT'L)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Aggressive OS guesses: Linux 3.11 (96%), Linux 3.1 (95%), Linux 3.2 (95%), AXIS 210A or 211 Network Camera (Linux 2.6.17) (94%), Linux 2.6.32 (94%), Linux 3.10 (94%), Linux 2.6.18 (93%), Linux 3.2 – 4.9 (93%), ASUS RT-N56U WAP (Linux 3.4) (93%), Linux 3.16 (93%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop

OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.24 seconds
root@pcfrxen:/# 

10. Add static MAC address to Ethernet Interface (if you find a MAC address being wrongly assigned to interface)

Sometimes problems with network unrechability between hosts is caused by wrongly defined MAC addresses on a Switch that did not correspond correctly to the ones assigned on the Linux host.
The easiest resolution here if you don't have access to Switch in work environment is to reassign the default MAC addresses of interfaces to proper MAC addresses, expected by remote router.

 

root@pcfrxen:/#  ​/sbin/ifconfig eth2 hw ether 8c:89:a5:f2:e8:d6

root@pcfrxen:/#  /sbin/ifconfig eth1 hw ether 8c:89:a5:f2:e8:d5

 

root@pcfrxen:/#  ifconfig eth0|grep -i ether
        ether 8c:89:a5:f2:e8:d6 txqueuelen 1000  (Ethernet)

 

11. Check for Network Address Translation (NAT) misconfigurations

If you do use some NAT-ing between Linux host and the remote Network Device you cannot reach, make sure IP Forwarding is enabled (i.e. /etc/sysctl.conf was not mistakenly overwritten by a script or admin for whatever reason).
 

root@server:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
root@server:~# sysctl net.ipv4.conf.all.forwarding
net.ipv4.conf.all.forwarding = 1

root@server:~# sysctl net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 0

12. Check for Resolving DNS irregularities with /etc/resolv.conf


If network connectivity is okay on TCP / IP , UDP Level but problems with DNS of course, check what you have configured inside /etc/resolv.conf

And if use newer Linux distributions and have resolving managed by systemd check status of resolvectl
 

root@server:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND — YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "resolvectl status" to see details about the actual nameservers.

nameserver 127.0.0.1
search pc-freak.net
domain pc-freak.net
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 109.104.195.2
nameserver 109.104.195.1
nameserver 208.67.222.222
nameserver 208.67.220.220
options timeout:2 rotate

root@pcfreak:~# 

 

root@server:~# resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (ens3)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.5.1
       DNS Servers: 192.168.5.1

 

  As seen see, the systemd-resolved service is used to provide domain names resolution and we can modify its configuration file /etc/systemd/resolved.conf to add the DNS server – the following line is set (two DNS servers’ addresses are added):

For example …

DNS=8.8.8.8 

13. Fix problems with wrongly configured Network Speed between hosts

It is not uncommon to have a Switch between two Linux hosts that is set to communicate on a certain maximum amount of Speed but a Linux host is set to communicate or lesser or more of Speed, this might create network issues so in such cases make sure either you use the Auto Negitionation network feature
or set both sides to be communicating on the same amount of network speed.

To turn on auto negotiation for ether interface 

# ethtool -s eth1 speed 1000 duplex full autoneg on


For example to set a Linux network interface to communicate on 1 Gigabit speed and switch off autonegotiation off.

# ethtool -s eth1 speed 1000 duplex full autoneg off

14. Check arp and icmp traffic with tcpdump

On both sides where the IPs can't see each other we can run a tcpdump to check the ARP and ICMP traffic flowing between the hosts.
 

# tcpdump -i eth1 arp or icmp

cpdump: verbose output suppressed, use -v[v]… for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:29:07.001841 IP freak-eth1 > pcfr_hware_local_ip: ICMP echo request, id 13348, seq 65, length 64
15:29:07.001887 IP pcfr_hware_local_ip > freak-eth1: ICMP echo reply, id 13348, seq 65, length 64
15:29:07.598413 ARP, Request who-has pcfr_hware_local_ip tell zabbix-server, length 46
15:29:07.598425 ARP, Reply pcfr_hware_local_ip is-at 8c:89:a5:f2:e8:d8 (oui Unknown), length 28
15:29:07.633055 ARP, Request who-has freak_vivacom_auto_assigned_dhcp_ip tell 192.168.1.1, length 46
15:29:08.025824 IP freak-eth1 > pcfr_hware_local_ip: ICMP echo request, id 13348, seq 66, length 64
15:29:08.025864 IP pcfr_hware_local_ip > freak-eth1: ICMP echo reply, id 13348, seq 66, length 64

 

# tcpdump -i eth1 -vvv

 

If you want to sniff for TCP protocol and specific port and look up for DATA transfered for SMTP you can use something like:

 

# tcpdump -nNxXi eth0 -s 0 proto TCP and port 25​

 

If you need a bit more thorough explanation on what it would do check out my previous article How to catch / track mail server traffic abusers with tcpdump
 

15. Debugging network bridge issues

Having bridge network interface is another brink where things could go totally wrong.
If you have network bridges configured, check out what is the status of the bridge.
 

root@freak:/etc/network# brctl show
bridge name    bridge id        STP enabled    interfaces
xenbr0        8000.70e284134411    yes        eno1
                            vif1.0
                            vif10.0
                            vif16.0
                            vif16.0-emu
                            vif2.0
                            vif3.0
                            vif4.0
                            vif5.0
                            vif6.0
                            vif9.0
                            vif9.0-emu
xenbr1        8000.70e284134412    yes        eno2
                            vif1.1
                            vif10.1
                            vif16.1
                            vif16.1-emu
                            vif2.1
                            vif3.1
                            vif4.1
                            vif5.1
                            vif6.1
                            vif9.1
                            vif9.1-emu


Check out any configurations such as /etc/sysconfig/network-scripts/ifcfg-* are not misconfigured if on Redhat / CentOS / Fedora.
Or if on Debian / Ubuntu and other deb based Linuxes look up for /etc/network/interfaces config problems that might be causing the bridge to misbehave.

For example one bridge network issue, I've experienced recently is related to bridge_ports variable configured as bridge_ports all.
This was causing the second bridge xenbr1 to be unable to see another local network that was directly connected with a cable to it.

The fix was bridge_ports none. Finding out this trivial issue caused by a restored network config from old backup took me days to debug.
As everything seemed on a network level to be perfect just like in Physical layer, same way and on Software level, routings were okay.

Checked everything multiple times and did not see anything irregular. ping was missing and hosts cannot see each other even though having the right netmask and
network configuration in place.

Below is my /etc/network/interfaces configuration with the correct bridge_ports none changed.

root@freak:/etc/network# cat /etc/network/interfaces
auto lo
iface lo inet loopback
 

auto eno1
allow-hotplug eno1
iface eno1 inet manual
dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220
auto eno2
allow-hotplug eno2
iface eno2 inet manual
dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4 207.67.222.222 208.67.220.220

auto xenbr0
allow-hotplug xenbr0
 # Bridge setup
# fetching dhcp ip from 192.168.1.20 (vivacom fiber optics router) routing traffic via 1Gigabit network
 iface xenbr0 inet dhcp
    hwaddress ether 70:e2:84:13:44:11
#    address 192.168.1.5/22
    address 192.168.1.5
    netmask 255.255.252.0
    # address 192.168.1.8 if dhcp takes from vivacom dhcpd
    bridge_ports eno1
    gateway 192.168.1.20
    bridge_stp on
    bridge_waitport 0
    bridge_fd 0
    bridge_ports none
    dns-nameservers 8.8.8.8 8.8.4.4

auto xenbr1
# fetching dhcp ip from pc-freak.net (192.168.0.1) bergon.net routing traffic through it
allow-hotplug xenbr1
 iface xenbr1 inet dhcp
    hwaddress ether 70:e2:84:13:44:11
##    address 192.168.0.3/22
    address 192.168.0.8
    netmask 255.255.252.0
   # address 192.168.0.8 if dhcp takes from vivacom dhcpd (currently mac deleted from vivacom router)
   # address 192.168.0.9 if dhcp takes from pc-freak.net hware host
#    hwaddress ether 70:e2:84:13:44:13
    gateway 192.168.0.1
    bridge_ports eno2
    bridge_stp on
    bridge_waitport 0
    bridge_fd 0
    bridge_ports none
    dns-nameservers 8.8.8.8 8.8.4.4
root@freak:/etc/network# 
 

 

root@freak:/etc/network# brctl showstp xenbr0
xenbr0
 bridge id        8000.70e284134411
 designated root    8000.70e284134411
 root port           0            path cost           0
 max age          20.00            bridge max age          20.00
 hello time           2.00            bridge hello time       2.00
 forward delay          15.00            bridge forward delay      15.00
 ageing time           0.00
 hello timer           1.31            tcn timer           0.00
 topology change timer       0.00            gc timer           0.00
 flags            


eno1 (1)
 port id        8001            state             forwarding
 designated root    8000.70e284134411    path cost          19
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8001            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif1.0 (2)
 port id        8002            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8002            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif10.0 (12)
 port id        800c            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    800c            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif16.0 (13)
 port id        800d            state               disabled
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    800d            forward delay timer       0.00
 designated cost       0            hold timer           0.00
 flags            

vif16.0-emu (14)
 port id        800e            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    800e            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif2.0 (4)
 port id        8004            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8004            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif3.0 (5)
 port id        8005            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8005            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif4.0 (3)
 port id        8003            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8003            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif5.0 (6)
 port id        8006            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8006            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif6.0 (7)
 port id        8007            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    8007            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

vif9.0 (10)
 port id        800a            state               disabled
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    800a            forward delay timer       0.00
 designated cost       0            hold timer           0.00
 flags            

vif9.0-emu (11)
 port id        800b            state             forwarding
 designated root    8000.70e284134411    path cost         100
 designated bridge    8000.70e284134411    message age timer       0.00
 designated port    800b            forward delay timer       0.00
 designated cost       0            hold timer           0.31
 flags            

root@freak:/etc/network# 


Sum it up

We have learned how to debug various routing issues, how to add and remote default gateways, check network reachability with ICMP protocol with ping, traceroute as well check for DNS issues and given some hints how to resolve DNS misconfigurations.
We also learned how to check the configured Network interfaces certain settings and resolve issues caused by Network sides max Speed misconfigurations as well how to track and resolve communication issues caused by wrongly configured MAC addresses.
Further more learned on how to do a basic port and protocol debugging of state of Network packets with netstat and nc and check problems related to iptables Firewall and IP Forwarding misconfigurations.
Finally we learned some basic usage of tcpdump on how to track arp and MAC traffic and look up for a specific TCP / UDP protocol  and its contained data.
There is certainly things this article is missing as the topic of debugging network connectivity issues on Linux is a whole ocean, especially as the complexity of Linux has grown dramatically these days.
I gues it is worthy to mention that unable to see remote network could be caused by wrong VLAN configurations on Linux or even buggy switches and router devices, due to hardware or software,
but I hope this article at least covers the very basics of network debugging and Linux. 

Enjojy 🙂

How to update expiring OpenSSL certificates without downtime on haproxy Pacemaker / Corosync PCS Cluster

Tuesday, July 19th, 2022

pcm-active-passive-scheme-corosync-pacemaker-openssl-renew-fix-certificate

Lets say you have a running PCS Haproxy cluster with 2 nodes and you have already a configuration in haproxy with a running VIP IP and this proxies
are tunneling traffic to a webserver such as Apache or directly to an Application and you end up in the situation where the configured certificates,
are about to expire soon. As you can guess having the cluster online makes replacing the old expiring SSL certificate with a new one relatively easy
task. But still there are a couple of steps to follow which seems easy but systemizing them and typing them down takes some time and effort.
In short you need to check the current certificates installed on the haproxy inside the Haproxy configuration files,
in my case the haproxy cluster was running 2 haproxy configs haproxyprod.cfg and haproxyqa.cfg and the certificates configured are places inside this
configuration.

Hence to do the certificate update, I had to follow few steps:

A. Find the old certificate key or generate a new one that will be used later together with the CSR (Certificate Request File) to generate the new Secure Socket Layer
certificate pair.
B. Either use the old .CSR (this is usually placed inside the old .CRT certificate file) or generate a new one
C. Copy those .CSR file to the Copy / Paste buffer and place it in the Website field on the step to fill in a CSR for the new certificate on the Domain registrer
such as NameCheap / GoDaddy / BlueHost / Entrust etc.
D. Registrar should then be able to generate files like the the new ServerCertificate.crt, Public Key Root Certificate Authority etc.
E. You should copy and store these files in some database for future perhaps inside some database such as .xdb
for example you can se the X – Certificate and Key management xca (google for xca download).
F. Copy this certificate and place it on the top of the old .crt file that is configured on the haproxies for each domain for which you have configured it on node2
G. standby node1 so the cluster sends the haproxy traffic to node2 (where you should already have the new configured certificate)
H. Prepare the .crt file used by haproxy by including the new ServerCertificate.crt content on top of the file on node1 as well
I. unstandby node1
J. Check in browser by accessing the URL the certificate is the new one based on the new expiry date that should be extended in future
K. Check the status of haproxy
L. If necessery check /var/log/haproxy.log on both clusters to check all works as expected

haserver_cluster_sample

Below are the overall commands to use to complete below jobs

Old extracted keys and crt files are located under /home/username/new-certs

1. Check certificate expiry start / end dates


[root@haproxy-serv01 certs]# openssl s_client -connect 10.40.18.88:443 2>/dev/null| openssl x509 -noout -enddate
notAfter=Aug 12 12:00:00 2022 GMT

2. Find Certificate location taken from /etc/haproxy/haproxyprod.cfg / /etc/haproxy/haproxyqa.cfg

# from Prod .cfg
   bind 10.40.18.88:443 ssl crt /etc/haproxy/certs/www.your-domain.com.crt ca-file /etc/haproxy/certs/ccnr-ca-prod.crt 
 

# from QA .cfg

    bind 10.50.18.87:443 ssl crt /etc/haproxy/certs/test.your-domain.com.crt ca-file /etc/haproxy/certs

3. Check  CRT cert expiry


# for haproxy-serv02 qa :443 listeners

[root@haproxy-serv01 certs]# openssl s_client -connect 10.50.18.87:443 2>/dev/null| openssl x509 -noout -enddate 
notAfter=Dec  9 13:24:00 2029 GMT

 

[root@haproxy-serv01 certs]# openssl x509 -enddate -noout -in /etc/haproxy/certs/www.your-domain.com.crt
notAfter=Aug 12 12:00:00 2022 GMT

[root@haproxy-serv01 certs]# openssl x509 -noout -dates -in /etc/haproxy/certs/www.your-domain.com.crt 
notBefore=May 13 00:00:00 2020 GMT
notAfter=Aug 12 12:00:00 2022 GMT


[root@haproxy-serv01 certs]# openssl x509 -noout -dates -in /etc/haproxy/certs/other-domain.your-domain.com.crt 
notBefore=Dec  6 13:52:00 2019 GMT
notAfter=Dec  9 13:52:00 2022 GMT

4. Check public website cert expiry in a Chrome / Firefox or Opera browser

In a Chrome browser go to updated URLs:

https://www.your-domain/login

https://test.your-domain/login

https://other-domain.your-domain/login

and check the certs

5. Login to one of haproxy nodes haproxy-serv02 or haproxy-serv01

Check what crm_mon (the cluster resource manager) reports of the consistancy of cluster and the belonging members
you should get some output similar to below:

[root@haproxy-serv01 certs]# crm_mon
Stack: corosync
Current DC: haproxy-serv01 (version 1.1.23-1.el7_9.1-9acf116022) – partition with quorum
Last updated: Fri Jul 15 16:39:17 2022
Last change: Thu Jul 14 17:36:17 2022 by root via cibadmin on haproxy-serv01

2 nodes configured
6 resource instances configured

Online: [ haproxy-serv01 haproxy-serv02 ]

Active resources:

 ccnrprodlbvip  (ocf::heartbeat:IPaddr2):       Started haproxy-serv01
 ccnrqalbvip    (ocf::heartbeat:IPaddr2):       Started haproxy-serv01
 Clone Set: haproxyqa-clone [haproxyqa]
     Started: [ haproxy-serv01 haproxy-serv02 ]
 Clone Set: haproxyprod-clone [haproxyprod]
     Started: [ haproxy-serv01 haproxy-serv02 ]


6. Create backup of existing certificates before proceeding to regenerate expiring
On both haproxy-serv01 / haproxy-serv02 run:

 

# cp -vrpf /etc/haproxy/certs/ /home/username/etc-haproxy-certs_bak_$(date +%d_%y_%m)/


7. Find the .key file etract it from latest version of file CCNR-Certificates-DB.xdb

Extract passes from XCA cert manager (if you're already using XCA if not take the certificate from keypass or wherever you have stored it.

+ For XCA cert manager ccnrlb pass
Find the location of the certificate inside the .xdb place etc.

+++++ www.your-domain.com.key file +++++

—–BEGIN PUBLIC KEY—–

—–END PUBLIC KEY—–


# Extracted from old file /etc/haproxy/certs/www.your-domain.com.crt
 

—–BEGIN RSA PRIVATE KEY—–

—–END RSA PRIVATE KEY—–


+++++

8. Renew Generate CSR out of RSA PRIV KEY and .CRT

[root@haproxy-serv01 certs]# openssl x509 -noout -fingerprint -sha256 -inform pem -in www.your-domain.com.crt
SHA256 Fingerprint=24:F2:04:F0:3D:00:17:84:BE:EC:BB:54:85:52:B7:AC:63:FD:E4:1E:17:6B:43:DF:19:EA:F4:99:L3:18:A6:CD

# for haproxy-serv01 prod :443 listeners

[root@haproxy-serv02 certs]# openssl x509 -x509toreq -in www.your-domain.com.crt -out www.your-domain.com.csr -signkey www.your-domain.com.key


9. Move (Standby) traffic from haproxy-serv01 to ccnrl0b2 to test cert works fine

[root@haproxy-serv01 certs]# pcs cluster standby haproxy-serv01


10. Proceed the same steps on haproxy-serv01 and if ok unstandby

[root@haproxy-serv01 certs]# pcs cluster unstandby haproxy-serv01


11. Check all is fine with openssl client with new certificate


Check Root-Chain certificates:

# openssl verify -verbose -x509_strict -CAfile /etc/haproxy/certs/ccnr-ca-prod.crt -CApath  /etc/haproxy/certs/other-domain.your-domain.com.crt{.pem?)
/etc/haproxy/certs/other-domain.your-domain.com.crt: OK

# openssl verify -verbose -x509_strict -CAfile /etc/haproxy/certs/thawte-ca.crt -CApath  /etc/haproxy/certs/www.your-domain.com.crt
/etc/haproxy/certs/www.your-domain.com.crt: OK

################# For other-domain.your-domain.com.crt ##############
Do the same

12. Check cert expiry on /etc/haproxy/certs/other-domain.your-domain.com.crt

# for haproxy-serv02 qa :15443 listeners
[root@haproxy-serv01 certs]# openssl s_client -connect 10.40.18.88:15443 2>/dev/null| openssl x509 -noout -enddate
notAfter=Dec  9 13:52:00 2022 GMT

[root@haproxy-serv01 certs]#  openssl x509 -enddate -noout -in /etc/haproxy/certs/other-domain.your-domain.com.crt 
notAfter=Dec  9 13:52:00 2022 GMT


Check also for 
+++++ other-domain.your-domain.com..key file +++++
 

—–BEGIN PUBLIC KEY—–

—–END PUBLIC KEY—–

 


# Extracted from /etc/haproxy/certs/other-domain.your-domain.com.crt
 

—–BEGIN RSA PRIVATE KEY—–

—–END RSA PRIVATE KEY—–


+++++

13. Standby haproxy-serv01 node 1

[root@haproxy-serv01 certs]# pcs cluster standby haproxy-serv01

14. Renew Generate CSR out of RSA PRIV KEY and .CRT for second domain other-domain.your-domain.com

# for haproxy-serv01 prod :443 renew listeners
[root@haproxy-serv02 certs]# openssl x509 -x509toreq -in other-domain.your-domain.com.crt  -out domain-certificate.com.csr -signkey domain-certificate.com.key


And repeat the same steps e.g. fill the CSR inside the domain registrer and get the certificate and move to the proxy, check the fingerprint if necessery
 

[root@haproxy-serv01 certs]# openssl x509 -noout -fingerprint -sha256 -inform pem -in other-domain.your-domain.com.crt
SHA256 Fingerprint=60:B5:F0:14:38:F0:1C:51:7D:FD:4D:C1:72:EA:ED:E7:74:CA:53:A9:00:C6:F1:EB:B9:5A:A6:86:73:0A:32:8D


15. Check private key's SHA256 checksum

# openssl pkey -in terminals-priv.KEY -pubout -outform pem | sha256sum
# openssl x509 -in other-domain.your-domain.com.crt -pubkey -noout -outform pem | sha256sum

# openssl pkey -in  www.your-domain.com.crt-priv-KEY -pubout -outform pem | sha256sum

# openssl x509 -in  www.your-domain.com.crt -pubkey -noout -outform pem | sha256sum


16. Check haproxy config is okay before reload cert


# haproxy -c -V -f /etc/haproxy/haproxyprod.cfg
Configuration file is valid


# haproxy -c -V -f /etc/haproxy/haproxyqa.cfg
Configuration file is valid

Good so next we can the output of status of certificate

17.Check old certificates are reachable via VIP IP address

Considering that the cluster VIP Address is lets say 10.40.18.88 and running one of the both nodes cluster to check it do something like:
 

# curl -vvI https://10.40.18.88:443|grep -Ei 'start date|expire date'


As output you should get the old certificate


18. Reload Haproxies for Prod and QA on node1 and node2

You can reload the haproxy clusters processes gracefully something similar to kill -HUP but without loosing most of the current established connections with below cmds:

Login on node1 (haproxy-serv01) do:

# /usr/sbin/haproxy -f /etc/haproxy/haproxyprod.cfg -D -p /var/run/haproxyprod.pid  -sf $(cat /var/run/haproxyprod.pid)
# /usr/sbin/haproxy -f /etc/haproxy/haproxyqa.cfg -D -p /var/run/haproxyqa.pid  -sf $(cat /var/run/haproxyqa.pid)

repeat the same commands on haproxy-serv02 host

19.Check new certificates online and the the haproxy logs

# curl -vvI https://10.50.18.88:443|grep -Ei 'start date|expire date'

*       start date: Jul 15 08:19:46 2022 GMT
*       expire date: Jul 15 08:19:46 2025 GMT


You should get the new certificates Issueing start date and expiry date.

On both nodes (if necessery) do:

# tail -f /var/log/haproxy.log

Why does Orthodox Christian priests and monks wears long beards and why Roman Catholics does not

Thursday, May 19th, 2011

A really Long bearded Orthodox Christian Priest

One might question why does Orthodox Christian priests wear beards? and why does the long beards of our Orthodox priests makes differences with the Roman Catholics?

Here are the few reasons:

1. Long beards wearing's tradition among Orthodox Christian priests and monks comes after Christ

Christ himself had a beard as it was normal and considered proper for a man to wear long beard.

The fact that our Lord Jesus Christ had a long beard himself can clearly be observed on all our Orthodox Christian icons:
The Lord Jesus Christ Sinai monastery ancient icon Pantocrator from the 6th century

The Lord's Pantecrator Icon (Pantocrator / Pantecrator ) from the 6th century

2. Long beards priest wearing comes as a natural tradition from the Old Testament's times and the tradition of early Church

If one reads thoroughfully the old testament, he will find out that even from Moses and Aaron and onwards the tradition is the same.
All the Godly man and the priests had their long beards unshaved as a mark for their belongship and dedication to God.
To generalize the long beards wearing is according to ancient old testamential ancient tradition.
The long beards tradition as an ancient Jewish religion (Old testamental) tradition can still be clearly observed in Jewish rabbis (nowdays the jewish priests), who still wears their beards long, like for example you can see in the picture below:

Jewish Rabbi weiss picture
A modern day Jewish Rabbi notice the beard 🙂

The long beards tradition later was adopted by Muslims when Islam emerged as a religion and more specificly by the muslim priests the Hodjas:

Sait Muslim Hodja Picture

One very interesting historical source of information which proofs that the ancient Church's priests had the tradition not to cut their beards is given by the historian Egezit who writes in his Chronicles that st. Apostle James, the head of the Church in Jerusalem, never cuts his hair.

A source of confirmation that the long hear and beards wearing was an established tradition that dates back to the old testament is found in the old testament in (Ezekiel 8:3)

Here is what exactly we read there:

He stretched out what looked like a hand and took me by the hair of my head.
The Spirit lifted me up between earth and heaven and in visions of God he took me to Jerusalem,
to the entrance to the north gate of the inner court, where the idol that provokes to jealousy stood.

3. Long hair and beards wearing by the Monks


Hieromonk_Sergij_Monk_Pomorie
An interesting fact is why does the Monks and novice neophyte lay brothers also stick to the ancient tradition.
It appears long hair and beards wearing traces back to the holy life of the ascetics of the deserts (e.g. the hermits).

The reason why ascetics did not shaved their hairs or bears as a way to avoid vanity and therefore this old hermitage practice has also had a spiritual reason.

4. The Nazarite old testament tradition

In the old testament in Numbers 6:1-21, we read about the term nazarite which means consecrated / separated

Each boy or man who was to become a Nazarine has been devoted to God for a certain period of time or in some cases for his all life, one of the many conditions for one to be a nazarite is not to shave his beard or hair.
One can read about this in the old testament in Leviticus 21:5

Leviticus 21:5
"They shall not make baldness upon their head,
neither shall they shave off the corner of their beard nor make any cuttings in their flesh."

There are some other prohibitions relating to Nazarite's one of the most notable ones is found in Numbers 6:4:

All the days of his Naziriteship shall he eat nothing that is made of the grape-vine,
from the pressed grapes even to the grapestone.

One example for people who gaves vow to become temporary Nazarites is found in 1 Maccabees 3:49 (this book is only available in the Orthodox Holy Bible).
One of the most important figure in Christianity that used to be Nazarite is Samson, his life can be read in the old testament in Judges 13 – 16

As we read in Judges , Samson's great God given power consituted in a prohibition to shave his hair and not to drink wine.

5. Reason why Roman Catholic Priests and monks abandoned the ancient tradition of wearing long hairs and beards

In the early Roman Empire it was a customfor a men to shave. The "enlightened" Romans believed that only the barbarians did not shaved themselves, and as you can imagine Jewish people and early Christians were of course considered to be barbarians, e.g. being unshaved was a sign for a cultural inferiorness in according to Romans comprehension.

The long hairs and beards tradition in the Western Church has started disappearing and consequentially completely lost with the Tyranny of Charlemagne at the end of the eight century.
With his massive 'barbarian' inferiority complex, it was his desire in all things to imitate pagan classical Rome.
It was therefore under him that Western clergy were ordered to shave regularly.
For example at the Council of Aachen (816), it was stipulated that priests and monks were to shave every two weeks.

By the beginning of the 11th century the tradition of wearing long beards was already completely torned apart and almost all the Roman Catholic clergy was regularly shaving.

In the sixteenth century beardlessness for Roman Catholic clergy was enforced by further canons,
which appear to have been dropped since the Second Vatican Council.

6. Why does protestants does not wear beards

As we all know protestant Church denominations has emerged as schismatics from Roman Catholic church and therefore mostly the influence they had was from Roman Catholics which already had the tradition within their clergy to regularly shave, thus pastors shaving was completely out of question and never come to an established reality among the Protestant Church pastors.

7. Is the Orthodox Christian layman obliged to wear beards

Absolutely not! The layman within the Orthodox Church can choose for themselves, if they want to wear their hair and beard and through that possess an image physically similar to Christ.
In my view it's more righteous for us the layman to wear our hairs and beards as I personally believe long hair and beards demonstrates mans dignity and God's dedication, but this is my own private opinion.
At many cases wearing beards or long hairs is an obstacle for a good integration in nowdays society, so if wearing a beard or hair as laymans does become an obstacle for our normal daily lifes then I believe cutting a long beard or hair is perfectly acceptable.
Moreover even the Orthodox Christian priests are not enforced to wear beards and in some cases where the priest's wife is against the beardness the Orthodox priest is allowed to shave himself, though as a matter of fact having a completely shaved priests in our Orthodox Churches is rare and less common today.

In conclusion wearing of beard and long hair by Orthodox Christna clergy, has come from the desire to physically resemble Christ.
This physical resemblance is a symbol of the spiritual resemblance of Christ's humility, which is the ultimate aim of our life.

The Church feast of St. Tsar (King) Boris Mihail (Michael) I – Baptizer of Bulgaria – 2nd of May a Triumph for Bulgarian Autocephalous Church and Church Slavonic

Monday, May 2nd, 2011

saint Tsar Boris the baptizer of all Bulgaria

Today on Second of May every year in the Bulgarian Orthodox Church calendar we commemorate the memory of Saint King Boris (Michael) who is a baptizer of Bulgarian nation and among the greatest saint of Bulgaria, perhaps second in saintship after St. John of Rila.

St. King Boris was the ruler of the First bulgarian Empire (852-889) the actual title as we know it from the chronicles is not king but archont which literally means ruler, but as the main chronicles who came to us are Byzantine we have to take in consideration that Byzantine Chronicles aimed to discredit the Bulgarian rulers because of feeling inferior every non-Byzantines who in Byzantine terms were considered barbarians.

He had a notable correspondence with Pope Nicolas II head of the Western Church before the Great Schism.
Saint Boris I is famous with with 115 Questions (which are fully preserved). He  beseech the pope to sent an apostoles of faith who will teach the new nation to faith in the same time he lead a correspondence with the Byzantine emperor and the Eastern Church.

The pope sent 2 missionaries cardinal Formiosa of Portuens and Bishop Paul of Papulon. King Boris requested for Cardinal Formiosa to become a future archibishop of Bulgaria, but the pope Nicolas II being afraid of loosing ground of a choice of future Bulgarian archibishop rejected king Boris request it is interesting fact that the same cardinal Formiosa become the next Pope of the Roman Western Church in period (891 – 896).

Taking in consideration the papal's desire to dominate over him and the more freedoms given by the Byzantine Church along with the fact that Saint Cyril and Methodius's pupils came to him with a ready Church Slavonic Glagolic version of the Holy Bible and a new Church language (and considering the fact that the seven pupils of St. Cyril and Methodius were being chased away from Great Moravia (Saint Clement of Ohrid (Kliment Ohridski) and Naum Ohrdiski with Sava, Angelarius and Gorazd) and seeing the fallacy of the filioque addition to the Creed of Faith finally King Boris received baptism by the hands of Patriarch Photius.

He received holy baptismal in year 864, receiving the Christian name Michael (Mihail) receiving his Christian name from his godfather, Emperor Michael III.

According to Church tradition the reason to Baptize in Christian faith was his amazement of an icon of the Judgement Day he saw in one of his visits to Constantinople.

The Church tradition also says saint Boris I's sister have lived for a long time in Constantinople, where she received baptism and once returning to the Bulgaria she bring the light of faith here too.

Besides that some of his family has already earlier converted to Christianity and some earlier Bulgarian Khans such as Trivelius (Tervel) who is considered to saved Europe from Muslim Invasion earlier were already Christians.

King_Boris I bapttish of Byzantine Emperor Michael III

During his holy reign he has established mass Christianization of Bulgaria, where the traditional ancient pagan traditions and belief in fake gods like Tangra were abolished completely.

St. Tsar Boris has secured the Bulgarian Church an autocephalousy, he also received the saints Cyril and Methodius, when they were banished from Great Moravia.

Our saint king has secured a refuge for st. Cyril and Methodius and provided them with assistance to develop the Slavonic alphabet and literature.

After he abdicated in 889, his eldest son and successor tried to restore the old pagan religion but was deposed by Boris I. During the Council of Preslav which followed that event, the Byzantine clergy was replaced with Bulgarian and the Greek language was replaced with Old Bulgarian as an official language of the Church and the state.

Bulgaria_under_rule_of_Boris_I-st-the-baptizer-king-of-Bulgaria

In 889 Boris abdicated the throne and became a monk. His son and successor Vladimir attempted a pagan reaction (to return the Bulgarians to the old belief in Tanra), which brought Boris out of his monastic life in 893 out of zeal for Christ and a fatherhood for the future of Bulgarian nation.

Vladimir was defeated and blinded by a miracle of God as Boris had less soldiers than his pagan son but in the name of the Lord Jesus Christ he showed victorious.

Once defeating his unbelieving son, Boris gathered the Council of Preslav in which the Old Greek language in Church was replaced with Church Slavonic and most imporantly on the council King Boris on the Church assembly placed his third son, Simeon I of Bulgaria on the throne, threatening him with the same fate if he too apostatized.

Simeon I was reluctant to become a secular ruler as according to some of the written sources he was preparing to become a great spiritual leader a monk and perhaps a next archibishop or even patriarch in the newly established autocephalous Bulgarian Church but following the monk rule of humility he decided and accepted the throne but throughout his life he kept his great zeal for monasticism and enlightenment. His ruling become a golden age for the Church Slavonic he financed seriously and worked hardly to prepare educated monks and to translate a major works of the Holy Fathers such as Shestodnev (The Six Days of Creation), Simeonov Sbornik (Simeon's Collection) and many of the key works of saint Athanasius the Great, Saint Basyl the Great, Saint John of Damascus etc.

According to some historians it was King Simeon who later give birth to the second by size Monastic Community in Byzantine Emperire the Stone Monasteries of (Meteora).

Boris returned to his monastery, emerging once again in c. 895 to help Simeon fight the Magyars, who had invaded Bulgaria in alliance with the Byzantines. After the passing of this crisis, Boris resumed monastic life and led a holy life until he pass out in year 907.

Here is the daily troparion assigned by the Bulgarian Orthodox Church sang at the churches today (the text is translation from Bulgarian):

St. Boris-Michael, prince of Bulgaria, Troparion

Full of the fear of God, and enlightened by holy baptism, thou becamest a habitation of the Holy Spirit,
O right-believing King Boris; and having established the Orthodox Faith in the land of Bulgaria,
and set aside the scepter of kingship,
thou madest thine abode in the wil derness,
didst flourish in ascetic struggles, and found grace before the Lord.
And now, standing before the throne of the Most High, pray thou, that He grant unto us who entreat thee salva tion for our souls.

Some of the most important Symbols for Orthodox Christians in The Eastern Orthodox Church – Symbols in the Eastern Orthodox Christian Faith (Eastern Orthodox Symbolism) and Christian Symbolism in the Roman Catholic Church (Symbolism in Western Catholicism)

Tuesday, April 13th, 2010

Some-of-the-most-important-symbols-for-orthodox-christiains-in-the-eastern-orthodox-church-symbols-in-eastern-orthodox-faith.

Yesterday, while browsing randomly I came across an interesting Roman Catholic webpage.
The website is created by Catholics with the idea to better explain the Catholic religion and Symbolism.
Though as an Orthodox Christian, my interest towards Roman Catholicism is only scientific, it's really interesting to see the common symbolism surrounding Roman Catholicism and compare with the Orthodox Christian symbolism. Many of the Roman Catholic Symbols are equal symbol with the one we nowadays used in the orthodox church.
I presume this common symbolism between Orthodox and Roman Catholic church,has stayed the same from the time before Roman Catholics split from the Only Holy Apostolic Church  to become the Church of the West Roman Empire, that's how the naming Roman Catholic came forward.

To find out more about Roman Catholic symbolism please see the following links I've mirrored the information from Fisheater's website which is btw is a great website targeting Roman Catholic layman. Everything on the website is explained in a simple everyday language without too much terminology which makes it a great resource for Roman Catholic Christians and people like me who who like to take a look in Roman Catholicism.

It's really a strange and intriguing fact let's call it a "co-incidence" that the inverted cross (upside-down) cross,also called "Peter's cross" on which saint Peter was crucified is also a symbol of Papacy .
It's a popular fact that nowadays Satanist use a similar inverted cross to the one said to be symbol of papacy for their "Black Masses" (Satanic Masses). Maybe some Roman Catholic priest or Cardinal has to explain, how comes that the Roman Catholics ended with such a significant symbol used nowdays in anti-christian satanic religion to be also a symbol of their beloved Pope??

I will skip forward to the heart of this article, which is to explain the Christian Symbolism which is important for us the Eastern Orthodox Christians. Many of the symbols might have in common, also with other Christian early Churches like the Coptic Oriental Orthodox Church, the Armenian Apostolic Orthodox Churches and other Chruches which somehow are closer to the One Holy and Apostolic Church – the Orthodox Church but officially are not in communion with us the Orthodox Christians.

Here I'll share only the most notable Christian Symbolism which is also used in the Eastern Orthodox Church.

Many of this symbolism was always bothering me while in Churches or Monasteries and was always pushing me to more and more questions without answers, thus I finally did some research on this symbols in get a better understanding on my Orthodox Christian faith.

Since I don't have a Theologian education and many of us the ordinary layman's in the church doesn't have such education I hope this orthodox Christian symbolism shared here and it's meanings will be of interest and will help you fortify your good faith in God and our Orthodox Christian faith.

Lamb of God Christian Symbol
Lamb

Lamb: symbol of Christ as the Paschal Lamb and also a symbol for Christians (as Christ is our Shepherd and Peter was told to feed His sheep).

This symbol is also presented in Bulgaria on the little yellow book they sell in our Bulgarian Orthodox Churches.
This tiny book contains the Divine Liturgy compiled by God's inspiration by st. John Chrysostom
If you're coming from an Catholic Background and you hold interest for Orthodox Christianity, as historically East Orthodox Christianity Symbol of Faith as well as basic doctrines were kept untouched, you might consider reading online here The Divine Liturgy by St. John Chrysostom
It's really important to say that the Divine Liturgy by St. John Chrysostom is the "backbone" of the church life, since it's the main and most served Liturgy in the eastern Orthodox Churches around the world.

Dove and Russian Patriarch
Dove: symbol of the The Holy Spirit and used especially in representations of our Lord's Baptism and the Pentecost. It is also used to recall Noe's dove, a harbinger of hope.

Chirchoao, Chi-Rho Sigla
"Chi-Rho" or "sigla": the letters "X" and "P," representing the first letters of the title "Christos," were eventually put together to form this symbol for Christ ("Chi" is pronounced "Kie"). It is this form of the Cross that the Emperor of Byzantia Constantine saw in his vision along with the Greek words, TOUTO NIKA, and which mean "in this sign thou shalt conquer.

Orthodox Tau Cross
"thau" or "tau": the T-shaped cross is mentioned in the Old Testament and is seen as a foreshadowing of the Cross of Christ.
Ezechiel 9:4:
"And the Lord said to him: Go through the midst of the city, through the midst of Jerusalem: and mark Thau upon the foreheads of the men that sigh, and
mourn for all the abominations that are committed in the midst thereof."
I've noticed that the tau_cross is often worn by Orthodox Monks as "a badge" on their clothes somewhere in the right of their chest

Greek Orthodox Cross
The Greek Orthodox Cross This symbol is one of the earliest Christian symbols which emerged right after Christ's resurrection.
The Greek Cross has all fours members the same shape and form (crux quadrata) and usually suggests the Christian church rather than a symbol of Christ's suffering.

Jerusalem Cross
Jerusalem Cross: also called the "Crusaders' Cross," it is made up of 5 Greek Crosses which are said to symbolize a) the 5 Wounds of Christ; and/or b) the 4 Gospels and the 4 corners of the earth (the 4 smaller crosses) and Christ Himself (the large Cross). This Cross was a common symbol used during the wars against Islamic aggression. (see less stylized version at right)

Baptismal Cross
Baptismal Cross: consisting of the Greek Cross with the Greek letter "X", the first initial of the title "Christ," this Cross is a symbol of regeneration, hence, its association with Baptism. Usually the Orthodox priest dress is decorated with a sign like this.

Red Orthodox Egg
The Scarlet red Egg:
Church tradition has it that St. Mary Magdalen went to Rome and met with the Emperor Tiberius to tell him about the Resurrection of Jesus. She held out an egg to him as a symbol of this, and he scoffed, saying that a man could no more rise from the dead than that egg that she held could turn scarlet. The egg turned deep red in her hands, and this is the origin of Easter eggs, and the reason why Mary Magdalen is often portrayed holding a scarlet egg.

Ichthus an early Christian Symbol
Ichtus (Ichthys) – The Fish:
Fish: the fish — ever-watchful with its unblinking eyes — was one of the most important symbols of Christ to the early Christians. In Greek, the phrase, "Jesus Christ, Son of God Savior," is "Iesous Christos Theou Yios Soter." The first letters of each of these Greek words, when put together, spell "ichthys," the Greek word for "fish" (ICQUS ). This symbol can be seen in the Sacraments Chapel of the Catacombs of St. Callistus. Because of the story of the miracle of the loaves and fishes, the fish symbolized, too, the Eucharist (see stylized fish symbol at right). Important note to make, here is that nowdays this sign's variations is not too often to be seen in Orthodox Churches. It's highly adopted by protestant Christians, seeing this sign on somebody's car or inside his home is a sure sign that probably he adhere's to Christian teachings different from orthodoxy.

Alpha Omega orthodox symbol
The Alpha-Omega symbol
Alpha, the first letter of the Greek alphabet, and Omega, the last letter of the Greek alphabet, became a symbol for Christ due to His being called "the First and the Last." The roots of symbolizing these attributes of God go back further, all the way to the Old Testament where, in Exodus 34:6, God is said to be "full of Goodness and Truth." The Hebrew spelling of the word "Truth" consists of the 3 letters "Aleph," "Mem," and "Thaw" — and because "Aleph" and "Thaw" are the first and last letters of the Hebrew alphabet, the ancients saw mystical relevance in God's being referred to as "Truth." At any rate, the Greek Alpha and Omega as a symbol for Christ has been found in the Catacombs, Christian signet rings, post-Constantine coins, and the frescoes and mosaics of ancient churches.

IC XC Nika Orthodox Symbol
The "IC XC Nika":
comes from Ancient Greek and was a widespread ancient Christian Symbol which is nowadays still present in the Eastern Orthodox Churches. IC XC Nika literally translated to english means "IC XC = Jesus Christ, NIKA = Glory to". In other words translated to modern english IC XC NIKA means Glory be to Jesus Christ!

Many Protestant Christians, nowdays falsely believe and claims this fish Christian symbol preceded the Crucifix as a symbol of veneration of Jesus and his Cross sufferings in the Church. This kind of belief is a falsely spread along many Protestant or "Evangelical" Christian denominations and Methodists. to be seen in many ancient Christian Church buildings is a Christian symbol. Today, some ancient Orthodox Churches still contain the "Christian fish" symbol. The reason why this symbol was used by early Christians is as a remembrance of the great miracle of Jesus to feed 5000 with 2 fishes and seven breads.

Holy Eucharist Cup, Bread and Wine

The Holy Eucharist vessels used by Orthodox Priests This is the cup of salvation as also called during the Divine Liturgy each time, the Wine and the Blood that the priest prepares in that Holy Cup is transformed by The Holy Spirit into a veracious flesh and blood of our Lord Jesus Christ.

Orthodox Byzantine Coat of Arms
The byzantine coat of arms
is an ancient Christian symbol used in the early Byzantine Church, nowadays it can be observed only in the Orthodox Churches.
It symbolizes the power of the Byzantian empire under the guidance of the the Holy Lord and the Gospel Truths.

Orthodox Bishop Crown
The Orthodox Bishop Crown is only worn by Bishops in the Orthodox Church. This crown indicates the Bishop's Church and spiritual (rank) and dignity.

Byzantine Orthodox Cross
Byzantine Orthodox or Russian Orthodox Cross
Is used most often by Eastern Catholics and Russian Orthodox, this Cross is the Byzantine Cross with the footrest at a diagonal. This slant is said to represent one of a few things:
– the footrest wrenched loose from the Christ's writhing in intense physical suffering; lower side representing "down," the fate of sinners, while the elevated side represents Heaven;
– the lower side represents the bad thief (known to us as Gestas through the apocryphal "Acts of Pilate" ("Gospel of Nicodemus") while the elevated side to Christ's right represents the thief who would be with Him in Paradise (St. Dismas);
– the "X" shape of the slanted "footrest" against the post symbolizes the cross on which St. Andrew was crucified.

Megaloschema a dress of a schimonk
The Megaloschema is a dress worn by schimonks. This monk rank is actually the highest possible rank an orthodox Christian monk can achieve. The symbolism on the dress is a brief form of:

  • IC XC (IECOYC XPICTOC) "Jesus Christ"
  • IC XC NIKA ("IECOYC XPICTOC NIKA") meaning: "Jesus Christ is Victorious"

The letters below IC XC Nika has a meaning – The Light of Christ shines on all.

  • XX. X.X letters. – means "Christ bestows grace on Christians"
  • The 4 Thitha (called) signs are a symbol for: Vision of God Divine wonder

Then the

  • T. K. P. G – Means "The Place of the Skull becomes Paradise"
  • The text placed in the lowest translated to English is "AdamThe First Man" and also is a symbol for the Place of the Skull (Golgotha).
  • In the Orthodox Church and the Church fathers teaches us that Golgotha or the Place of the Skull is the Place where the first man (Adam) was buried, and by God's divine providence coincides with the place where our Saviour Jesus Christ was crucified.

Orthodox Bishop Dress
Orthodox Priest dress / robe
This dress is only worn by Orthodox Christian Bishops.

Bulgarian Orthodox cross with 4 lights
The Cross with four lights emitating near the center of the cross This cross is actually used in more modern times as a Christian Orthodox symbol, The four lights coming out of the cross are added,
as the gospels speak that Christ is the Sun of righteousness
I've had quite a long time trying to figure out why exactly this cross is made with this 4 lights. It was a real joy when one time a priesttold me the meaning.
It's interesting fact that most of the Roman Catholic's crosses nowdays have the four lights radiating from Christ's Crucifix or the Cross symbolizing the Crucifix.

This is all I will say for symbolism for now. I hope this Christian symbolism will shed some light on the matters of Symbolism in both the Orthodox and the Catholoic eastern Church. I'll be glad if somebody out there more literate on the subject comment on my post and correct me if I'm wrong with smething.

The Celebration of the day of All Bulgarian Church Saints in the Bulgarian Orthodox Church

Sunday, June 6th, 2010

All Bulgarian Saints Icon

Today it’s the Sunday in which we the Bulgarian Orthodox Christians celebrate The Day of All the Bulgarian Saints
We have plenty of saints in our Bulgarian Christian history, many known and even more unknown ones. That feast is actually one of the greatest local church commemoration of the Christian saints who gave their life for the building, growing and preventing of the orthodox bulgarian church and the orthodox christian faith througout Bulgaria’s lands.
An almost complete list of the orthodox bulgarian saints can be seen here
However many are not mentioned since there are no historical documents about the countless victims (martyrs) of Ottoman muslics who were beheaded or violently killed because of his refusal to accept the Islam.
Here I will just mention few of the saints that are mostly venerated in the realm of the bulgarian orthodox christian faith:

1. st. Enravota (Prince Boyan)
is considered the first Bulgarian Saint martyr who gave his life to show hthe truthfulness of the Christian Orthodox faith.
His most notable speak is also a prophecy concerning the bulgarian orthodox church which was made before his martyrdom for Christ.
The prophecy saint Enravota has made about the Christianity establishment in Bulgaria is as follows:

This faith, which I now die for, will spread and increase across the whole Bulgarian land, although you may wish to oppress it with my death. In any case, the Sign of Christ will establish itself and churches of God will be built everywhere and pure priests will serve the pure God and will deliver “sacrifice of praise and confession” to the invigorating Trinity. Idols, and priests as well, and their ungodly temples, will crumble and will turn into nothing, as if they had not existed. Besides, you alone (to Malamir), after many years, will cast away your ungodly soul without receiving anything in reward for your cruelty.

2. St. Jonh of Rila (wonderworker) was a hermit in the Rila mountain who is a patron saint of the Bulgarian Orthodox Church. He is notable for his great spiritual purity achievement in his anchoretic life. Many miracles and healing has been received by people who has prayed and asked the saint to intercede in their behalf to God. He is known as the first bulgarian monk, and as an initiator of the first bulgarian orthodox christian monastery.
His uncorruptable body is found for pilgrimage until this very day in the monastery he became a founder of Rila Monastery .

2. St. Brothers Cyril and Methodius Who were the saint brothers who venerated for enlightening the slavonic nations with writting letters. They’re actually the inventors of the old Bulgarian also known as Old Church Slavonic. They and their followers were the first to translate the holy bible into the newly created language – the Old Bulgarian.

3. st. Knyaz (King) Boris-Mihail (Boris-Michael) . He is known for his baptism of the Bulgarian Nation and the initiator of the creation of the Bulgarian Orthodox Church as well as a brilliant statesman.
After baptising most of the bulgarian intelligentia and rooting up the Orthodox Christian faith in Bulgaria, he abdicted voluntery from his throne, leaving his son (Vladimir) to rule Bulgaria and started a voluntery monastic life in poverty.
When his governing son tried to reinforce back the old pagan bulgarian Tengriism belief (e.g. the belief in Tangra (Tengri) he left the monastery took back the power (defeating his son) and blinded Vladimir.
Then he returned back to the monastery and spend his remaining life as a monk.

4. st. Clement of Ohrid was one of the Seven Bulgarian Apostles
Born in 840, a Slav from southern Macedonia, he established a monastery at Okhrida and a bishopric at Velica not far away. He is regarded as the founder of this primatial see and the first Slav to become a bishop.
His extensive apostolate took the form of education of the clergy and of the laity, to whom he preached a series of sermons in Slavonic, suitable for neophytes and explaining the principal feasts of the liturgical year. Clement died at Okhrida on 27 July 916.
5. St. Nahum ucceeded St. Clement of Ohrid as bishop. Converted in Moravia by Cyril and Methodius, he journeyed with them to Rome and helped them with their translations into the vernacular: he is venerated in Russia as well as Bulgaria. Other companions of Clement called Sava and Angelar are also venerated with him. In different times and different places they all contributed to the fulfilment of the missionary plans of Cyril and Methodius.
6. Saint Zlata of Meglen has endured a martyrdom for Christ. She refused to accept muslim faith and has firmly confessed her Orhtodox Christian faith.
She was given many chances to accept muslim faith by some Turks and therefore save her life but she refused and was cast into a prison for 3 months flogging until her blood soacked the ground.
Finally, they suspended her upside down and lit a fire, to suffocate her with the smoke; but God was with Zlata, and gave her strength in suffering.
At last they hanged her from a tree and cut her into small pieces. Thus, this brave virgin gave her soul up to God, and went to dwell in Paradise on October 13, 1796. Pieces of her relics were taken by Christians to their homes for a blessing.

7. 26 Martyrs of the Zographou Monastery on Mt. Athos at the hands of the Crusaders they have endured martyrdom for Christ, from the Roman Catholic Crusaders. Though the were warned that the enemies of Christ are coming to the Zograph Monastery they refused to leave the place and was burned alive by the unpious Crusaders.
There is many others to be mentioned but I neither have the knowledge nor I’m worthy to speak about them.
All Bulgarian Saints new icon representation

Let by their Holy Prayers God have mercy on us the sinners.

How to Turn Off, Suppress PHP Notices and Warnings – PHP error handling levels via php.ini and PHP source code

Friday, April 25th, 2014

php-logo-disable-warnings-and-notices-in-php-through-htaccess-php-ini-and-php-code

PHP Notices are common to occur after PHP version upgrades or where an obsolete PHP code is moved from Old version PHP to new version. This is common error in web software using Frameworks which have been abandoned by developers.

Having PHP Notices to appear on a webpage is pretty ugly and give a lot of information which might be used by malicious crackers to try to break your site thus it is always a good idea to disable PHP Notices. There are plenty of ways to disable PHP Notices

The easiest way to disable it is globally in all Webserver PHP library via php.ini (/etc/php.ini) open it and make sure display_errors is disabled:

display_errors = 0

or

display_errors = Off

Note that that some claim in PHP 5.3 setting display_errors to Off will not work as expected. Anyways to make sure where your loaded PHP Version display_errors is ON or OFF use phpinfo();

It is also possible to disable PHP Notices and error reporting straight from PHP code you need code like:

 

<?php
// Turn off all error reporting
error_reporting(0);
?>

 

or through code:

 

ini_set('display_errors',0);


PHP has different levels of error reporting, here is complete list of possible error handling variables:

 

 

 

<?php
// Report simple running errors

error_reporting(E_ERROR | E_WARNING | E_PARSE);

// Reporting E_NOTICE can be good too (to report uninitialized
// variables or catch variable name misspellings …)

error_reporting(E_ERROR | E_WARNING | E_PARSE | E_NOTICE);

// Report all errors except E_NOTICE
// This is the default value set in php.ini

error_reporting(E_ALL ^ E_NOTICE);
// Report all PHP errors (see changelog)

error_reporting(E_ALL);
// Report all PHP errors error_reporting(-1);
// Same as error_reporting(E_ALL);

ini_set('error_reporting', E_ALL); ?>

The level of logging could be tuned on Debian Linux via /etc/php5/apache2/php.ini or if necessary to set PHP log level in PHP CLI through /etc/php5/cli/php.ini with:

error_reporting = E_ALL & ~E_NOTICE

 

If you need to remove to remove exact warning or notices from PHP without changing the way  PHPLib behaves is to set @ infront of variable or function that is causing NOTICES or WARNING:
For example:
 

@yourFunctionHere();
@var = …;


Its also possible to Disable PHP Notices and Warnings using .htaccess file (useful in shared hosting where you don't have access to global php.ini), here is how:

# PHP error handling for development servers
php_flag display_startup_errors off
php_flag display_errors off
php_flag html_errors off
php_flag log_errors on
php_flag ignore_repeated_errors off
php_flag ignore_repeated_source off
php_flag report_memleaks on
php_flag track_errors on
php_value docref_root 0
php_value docref_ext 0
php_value error_log /home/path/public_html/domain/php_errors.log
php_value error_reporting -1
php_value log_errors_max_len 0

This way though PHP Notices and Warnings will be suppressed errors will get logged into php_error.log

Sick again – Hoping God to solve it all – and no desire to study

Thursday, September 27th, 2007

Yesterday I was out for a coffee with Tony an old frined from the Old metal days, after that I went outwith Lily I drinked 200 gr of Vodka and a Beer. I was pretty desparate actually and I got very drunk I got home 04:10 and got to bed at 4:30. What can I say life is hard. Today Mitko called at dinner time and we drinked a coffee on the fountain. The night we went out in the Central park and drinked a beer per man. I’m starting to think about the meaningless of my life again, since my life seems without a real purpose and seems not going nowhere ( we humans are so weak ). Yesterday I have almost taken the decision to suicide. It almost a miracle after I send lily to her home and was going back home with the idea of my mind to suicide. Lily called and said she can’t unlock the door and everyone inside is sleeping and she didn’t want to disturb them. So we spend more than an hour outside. Today I feel terrible physically again. I really can’t take it anymore I need Jesus to bless me with a healing (Save me Holy Lord, Lord Jesus son of the Blessed God have mercy over me the sinner heal me Lord and Deliver me from all sickness and affliction and pain Heal my Soul, Body and Spirit, make me whole to be able to serve you again :[ Amen) My computer went down twice today I have a problem with the cooling. The cooler should be cleaned but I guess this will happen after our Room’s repair is completed. By the way Both Pc-Freak and Jericho are on the terrace. Because repair works are being done into the room. Day after day I ask my self the question when I will be healed again. I do smoke a lot of cigarettes for which I’m ashamed of myself (But it’s an act of desperation). By the way I don’t have any desire to study anymore I’d like to do things with FreeBSD and learn stuff. If God is merciful to me and heals me probably I’ll continue to live otherwise I guess I’ll end badly. Very often come to the point Nobody out there cares, them I feel the presence of God’s spirit I remember the Bible’s instruction towards doubtful mind and try return back to the Faith of the Eternal, It’s like a sort of schizofrenia (I believe, I don’t believe). I’m sure both cannot be true. Very often I think I deserve more but since God’s rightous probably that’s what I deserve. I tried do good for so many people and to care for as much people I’m very discouraged because as a reward I see it like receiving this infirmity and sickness … Something have to change or I’ll be gone …END—–

Curious: Old ladies from Plovdiv, Bulgaria are knitting mobile and computer equipment hand made cases

Monday, November 11th, 2013

new business idea manual hand crafted knitted mobile device cases

While reading one of my daily tech news websites today I stimbled on very unusual news and a business idea, worthy to re-blog 😉 The idea of the new Innovative business is just genious and simple as that – knitted digital equipment cases. There are plenty of old ladies in Bulgaria who are used to knitting sweaters, hats and all kind of hand made winter clothes. Knitting was among the must have skills for ladies in Bulgaria until some years time even my grandmother when a bit younger used to knit in most of her free time. Thus most of grandmothers in Bulgaria still when get old and have a lot of free time, kill their time with the craft they learned as young. For the young European people who probably never heard of it knitting is unique kind of manufacture for producing winter clothes from yarn.

yarn in different colors the base material for knitting

Knitting was a top ladies craft before the boom of industrial manufacture before 19th century industrial revolution, nowadays its almost a forgotten craft. Now some genious people in Bulgaria decided to give work to this old ladies and pay them for knitting computer and mobile equipment cases (cases for Tablets, Smartphones, Notebooks etc.) Knitting of mobile equipment cases opens new work places and introduces new way for old people to earn some extra income, such extra income is absolutely necessary in Bulgaria nowdays as most old ladies receive ridiculous pension of 120 to 150 EUR per month (money insufficient to pay expensive winter heating bills and so expensive food that is nowdays in Bulgaria).

lady knitting a sweater or some kind of clothe

The company who invented this new business and introduced it for the masses is Snug Essentials and is started by enthusiastic young people who had grandmothers practicing the old craft. The innovative product is distributed via Etsy Global Platform for distributing hand crafted goods as well as in   Facebook, Twitter and Ebay. As of time of writting the companyis extremely small 6 old grandmas work on producing the knitted cases for digital equipment but the tendency is this to rise soon, as there are plenty of people who love the hand made cases.
The price of a case is also not too high, it costs only around 10$ per knitted case. Here are few nice knitted cases produced by grandmas 🙂

notebook laptop in knitted hand made case

Your girlfriend will love it if you gift her one of these 🙂

knitted notebook case original business idea

 

WordPress blog MySQL data maintainance valuable plugin WP-OPTIMIZE

Tuesday, April 12th, 2011

The more my blog is growing the slower it becomes, this is a tendency I’ve noticed since a couple of weeks.

Very fortunately while reading some random articles online I’ve came across a super valuable wordpress plugin called WP-OPTIMIZE

I think it’s best if I present instead of taking the time to explain what the WP-optimize does for a wordpress powered blog:

WP-Optimize is a WordPress 2.9++ database cleanup and optimization tool. It doesn’t require PhpMyAdmin to optimize your database tables. It allows you to remove post revisions, comments in the spam queue, un-approved comments within few clicks.

Additionally you can rename any username to another username too.

For example: If you want to rename default ‘admin’ username to ‘someothername’; just put ‘admin’ (without quotes) to the Old username field and the new username to the New username field, then click “PROCESS”)

Now in short to rephrase above text, during MySQL database requests a lot of database starts needing the so called MySQL optimization , this operation becomes necessery especially for databases which are very actively used and is related to the basic design of the mysql server.

Very often many tables starts having a lot of garbage (repetitive) data and therefore read and writes from the SQL server gets slower and slower day by day.

Thus the MySQL server has it’s famous OPTIMIZE TABLE command which does wipe out all the garbage unnecessery data stored in a tables/s and hence seriously impacts the later read/write table operations.

Now to go back to wordpress the same optimization issues, very often are a cause for a performance bottleneck and some smart guy has came with the great idea to create a specific wordpress plugin to fix such an optimize table issues

The WP-Optimize plugin has 5 major options which somehow could have a positive impact on blog’s performance, these are:
 

  • Remove all Post revisions
  • Remove all auto draft posts
  • Clean marked Spam comments
  • lean Unapproved comments
  • Optimize database tables

Apart from the nice performance boost applicaiton the wp-optimize plugin has one super valuable function, it could change the default wordpress blog administrator user admin to some other desired username, for example rename the default blog administrator username “admin” user to “john”.

From a security perspective this is another must have feature in every wordpress blog as it can kill many of the possible brute force attacks with some web brute force softwares like Hydra

Installing and using wp-optimize is a real piece of cake, here is a very quick few command lines to install it:

host:~# cd /var/www/blog/wp-content/plugins/
host:/var/www/blog/wp-content/plugins:# wget https://www.pc-freak.net/files/wp-optimize.0.9.2.zip
host:/var/www/blog/wp-content/plugins:# unzip wp-optimize.0.9.2.zip
...

To launch WP-OPTIMIZE and fix many speed bottlenecks caused by unoptimized tables, or just hanging around in database old spam comments, go and login with your adminsitrator to wordpress.

In the left pane menu starting up with Dashboard somewhere near the end of the vertical menu you will notice a button reading WP-Optimize .
Click over the Wp-Optimize button and a screen like the one in below’s screenshot will appear in your browser screen:

wp optimize plugin database optimization options screen

I have personally decided to use just 2 of all the 5 available primary plugin options, I decided to select only:
 

  • Clean marked Spam comments
  • Optimize database tables

Scrolling below the PROCEED button I could also notice a number of tables which were missing optimization and hence required an optimize table to be executed on each one of them.
Further on I pressed the PROCESS button and after a couple of minutes (2, 3 minutes) of waiting the wp-optimize plugin was done it’s selected tasks:

In the screenshot below, you see all my blog tables which were optimized by the plugin:
WP-Optimize optimized blog tables screenshot

It’s interesting to say few words about the Remove All Posts revisions plugin option, as many might be wondering what this plugin option really means.

When writting posts, wordpress has an option to restore to a certain point of the write out and makes a sort of different versions in time of each of it’s written posts.

Therefore later restoration if something gots deleted by mistake is facilitated, this is actually the all meaning of Remove All Posts revisions

With time and the increase wp posts the amount of Posts Revisions is growing on and on and just taking space and possibly making the access to the database entries slower, thus it might be a good idea to also remove them as, it’s absolutely rare to use the wp post restoration function.
However I’ve personally decided to not wipe out the old posts revisions as someday i might need it and I’m a bit scared that it might cause me some database issues.

I’ll be glad if somebody has tried the Posts Revisions wp-Optimize funciton and is happy with the results.