Posts Tagged ‘cisco’

Zabbix script to track arp address cache loss (arp incomplete) from Linux server to gateway IP

Tuesday, January 30th, 2024

Zabbix_arp-network-incomplete-check-logo.svg

Some of the Linux servers recently, I'm responsible had a very annoying issue recently. The problem is ARP address to default configured server gateway is being lost, every now and then and it takes up time, fot the remote CISCO router to realize the problem and resolve it. We have debugged with the Network expert colleague, while he was checking the Cisco router and we were checking the arp table on the Linux server with arp command. And we came to conclusion this behavior is due to some network mess because of too many NAT address configurations on the network or due to a Cisco bug. The colleagues asked Cisco but cisco does not have any solution to the issue and the only close work around for the gateway loosing the mac is to set a network rule on the Cisco router to flush its arp record for the server it was loosing the MAC address for.
This does not really solve completely the problem but at least, once we run into the issue, it gets resolved as quick as 5 minutes time. }

As we run a cluster environment it is useful to Monitor and know immediately once we hit into the MAC gateway disappear issue and if the issue persists, exclude the Linux node from the Cluster so we don't loose connection traffic.
For the purpose of Monitoring MAC state from the Linux haproxy machine towards the Network router GW, I have developed a small userparameter script, that is periodically checking the state of the MAC address of the IP address of remote gateway host and log to a external file for any problems with incomplete MAC address of the Remote configured default router.

In case if you happen to need the same MAC address state monitoring for your servers, I though that might be of a help to anyone out there.
To monitor MAC address incomplete state with Zabbix, do the following:
 

1. Create  userparamater_arp_gw_check.conf Zabbix script
 

# cat userparameter_arp_gw_check.conf 
UserParameter=arp.check,/usr/local/bin/check_gw_arp.sh

 

2. Create the following shell script /usr/local/bin/check_gw_arp.sh

 

#!/bin/bash
# simple script to run on cron peridically or via zabbix userparameter
# to track arp loss issues to gateway IP
#gw_ip='192.168.0.55';
gw_ip=$(ip route show|grep -i default|awk '{ print $3 }');
log_f='/var/log/arp_incomplete.log';
grep_word='incomplete';
inactive_status=$(arp -n "$gw_ip" |grep -i $grep_word);
# if GW incomplete record empty all is ok
if [[ $inactive_status == ” ]]; then 
echo $gw_ip OK 1; 
else 
# log inactive MAC to gw_ip
echo "$(date '+%Y-%m-%d %H:%M:%S')" "ARP_ERROR $inactive_status 0" | tee -a $log_f 2>&1 >/dev/null;
# printout to zabbix
echo "1 ARP FAILED: $inactive_status"; 
fi

You can download the check_gw_arp.sh here.

The script is supposed to automatically grep for the Default Gateway router IP, however before setting it up. Run it and make sure this corresponds correctly to the default Gateway IP MAC you would like to monitor.
 

3. Create New Zabbix Template for ARP incomplete monitoring
 

arp-machine-to-default-gateway-failure-monitoring-template-screenshot

Create Application 

*Name
Default Gateway ARP state

4. Create Item and Dependent Item 
 

Create Zabbix Item and Dependent Item like this

arp-machine-to-default-gateway-failure-monitoring-item-screenshot

 

arp-machine-to-default-gateway-failure-monitoring-item1-screenshot

arp-machine-to-default-gateway-failure-monitoring-item2-screenshot


5. Create Trigger to trigger WARNING or whatever you like
 

arp-machine-to-default-gateway-failure-monitoring-trigger-screenshot


arp-machine-to-default-gateway-failure-monitoring-trigger1-screenshot

arp-machine-to-default-gateway-failure-monitoring-trigger2-screenshot


6. Create Zabbix Action to notify via Email etc.
 

arp-machine-to-default-gateway-failure-monitoring-action1-screenshot

 

arp-machine-to-default-gateway-failure-monitoring-action2-screenshot

That's all. Once you set up this few little things, you can enjoy having monitoring Alerts for your ARP state incomplete on your Linux / Unix servers.
Enjoy !

How to calculate connections from IP address with shell script and log to Zabbix graphic

Thursday, March 11th, 2021

We had to test the number of connections incoming IP sorted by its TCP / IP connection state.

For example:

TIME_WAIT, ESTABLISHED, LISTEN etc.


The reason behind is sometimes the IP address '192.168.0.1' does create more than 200 connections, a Cisco firewall gets triggered and the connection for that IP is filtered out. To be able to know in advance that this problem is upcoming. a Small userparameter script is set on the Linux servers, that does print out all connections from IP by its STATES sorted out.

 

The script is calc_total_ip_match_zabbix.sh is below:

#!/bin/bash
#  check ESTIMATED / FIN_WAIT etc. netstat output for IPs and calculate total
# UserParameter=count.connections,(/usr/local/bin/calc_total_ip_match_zabbix.sh)
CHECK_IP='192.168.0.1';
f=0; 

 

for i in $(netstat -nat | grep "$CHECK_IP" | awk '{print $6}' | sort | uniq -c | sort -n); do

echo -n "$i ";
f=$((f+i));
done;
echo
echo "Total: $f"

 

root@pcfreak:/bashscripts# ./calc_total_ip_match_zabbix.sh 
1 TIME_WAIT 2 ESTABLISHED 3 LISTEN 

Total: 6

 

root@pcfreak:/bashscripts# ./calc_total_ip_match_zabbix.sh 
2 ESTABLISHED 3 LISTEN 
Total: 5


images/zabbix-webgui-connection-check1

To make process with Zabbix it is necessery to have an Item created and a Depedent Item.

 

webguiconnection-check1

webguiconnection-check1
 

webgui-connection-check2-item

images/webguiconnection-check1

Finally create a trigger to trigger alarm if you have more than or eqaul to 100 Total overall connections.


images/zabbix-webgui-connection-check-trigger

The Zabbix userparameter script should be as this:

[root@host: ~]# cat /etc/zabbix/zabbix_agentd.d/userparameter_webgui_conn.conf
UserParameter=count.connections,(/usr/local/bin/webgui_conn_track.sh)

 

Some collleagues suggested more efficient shell script solution for suming the overall number of connections, below is less time consuming version of script, that can be used for the calculation.
 

#!/bin/bash -x
# show FIN_WAIT2 / ESTIMATED etc. and calcuate total
count=$(netstat -n | grep "192.168.0.1" | awk ' { print $6 } ' | sort -n | uniq -c | sort -nr)
total=$((${count// /+}))
echo "$count"
echo "Total:" "$total"

      2 ESTABLISHED
      1 TIME_WAIT
Total: 3

 


Below is the graph built with Zabbix showing all the fluctuations from connections from monitored IP. ebgui-check_ip_graph

 

Linux: Add routing from different class network A (192.168.1.x) to network B (192.168.10.x) with ip route command

Friday, July 12th, 2013

adding routing from one network to other linux with ip route

I had a Linux router which does NAT for a local network located behind a CISCO router receiving internet via its WAN interface routing traffic  to Linux with IP 192.168.1.235. The Linux router has few network interfaces and routes traffic for networks; 192.168.1.0/24 and 192.168.10.0/24. Another Linux with IP 192.168.1.8 had to talk to 192.168.10.0/24 (because it was necessary to be able access  ISCO's router web interface accessible via a local network interface with IP (192.168.10.1). Access to 192.168.10.1 wasn't possible from 192.168.1.8 because routing on NAT-ting Linux (192.168.1.235) to 192.168.10.0/24 network was missing. To make 192.168.1.8 Linux communicate with 192.168.10.1,  had to add following routing rules with ip command on both the Linux with IP 192.168.1.235 and Linux host behind NAT (192.168.1.8).

1. On Server (192.168.1.235) run in root shell and add to /etc/rc.local

# /sbin/ip r add 192.168.10.0/24 via 192.168.1.235
And then copy paste same line before exit 0 in /etc/rc.local

Its good idea always to check routing, after adding anything new, here is mine:
 

# ip r show

192.168.5.0/24 dev eth0  proto kernel  scope link  src 192.168.5.1
192.168.4.0/24 dev eth0  proto kernel  scope link  src 192.168.4.1
192.168.3.0/24 dev eth0  proto kernel  scope link  src 192.168.3.1
192.168.2.0/24 dev eth0  proto kernel  scope link  src 192.168.2.1
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.235
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1
192.168.10.0/24 dev eth1  proto kernel  scope link  src 192.168.10.2
default via 192.168.10.1 dev eth1 
 

2. And also on Second Linux host (192.168.1.8) 

# /sbin/ip r add 192.168.10.0/24 via 192.168.1.235
To make routing permanent again paste in /etc/rc.local before exit 0

After above rules, I can normally ping and access hosts on class C network 192.168.10.1-255  from 192.168.1.8.

Here I’m :)

Saturday, August 16th, 2008

Haven’t blogged for some time so I’m going to write down few lines. Today I passed succesfully the Cisco CCNA2 Final Exam and the CCNA2 Voucher exam thanks to God :). The Voucher exam grants everybody who passed succesfully a discount of the price of the CCNA Certification exam, whi ch is pretty cool. To be honest I cheated at the 2 tests otherwise I won’t have passed but I’m really busy this days and I’m a bit of tired so this was the only option. I also made my CCNA2 (9,10,11) tests for which I used the cisco answers page :). CCNA2 The Final Exam and the Voucher wasn’t included in the blogspot cisco answers so I googled around but thanksfully I found the answers on a website in the net. This days I had a lot of fun and saw a lot of old friends (some of them studying others working in Sofia). It’s the summers holiday period in which tons of friends who are in other cities are back in Dobrich so I was able to spend nice time with a lot of them. This are ppl like: hellpain, nomen, mariana (one of my ex-girlfriend). At Wednesday I and Nomen went to the beach at Balchik, we had great fun there. We met together in front of the place we call “The Young House” at 7:30, at 8:30 we were already travelling to Balchik and in 9:35 we were on the beach. We stayed there until 12:30. Vlado and Mitko tried to learn me to be able to relax on my on back while in the water I almost did it but still I need some more tries until I’m able to make it the right way. At 13:00 we were back in Dobrich and we went to an open restaurant “Seasons”. I ate vegetarenian pizza and Vlado ate a meal called “English Breakfast”. “English Breakfast”. Right after that at 14:00 I went to the regional Red Cross building, because I had to get a course on “First Aid”, it’s required to have that course passed until you’re able to apply for a driving license after you complete both the theoretical exam and the practice exam. My driving practice exam would be at Wednesday at 9:00 to be honest I’m not still driving well and my driving course is almost over. My only hope for passing it in the Lord, only if he support me and guide me during the exam I’ll be able to pass. The same night I went out with Shanar and we later met hellpain and Alex. We had nice discussions in the city park until somewhere around 11:30. When we were going back home I met Mariana and I proposed her to have a walk together she accepted and we negotiated to meet around 30 minutes later. We spend 4 hours with her talking about stuff and drinking beer. I should say she is one of the only girls I’m able to speak for hours and still feel good and confortable. The center of our talks were mainly God the Bible and in particular my belief in Christ Jesus as Lord and Saviour. At Friday morning I had driving courses early at the morning from 8:00 o’clock, later I went to Varna with my father because I had to pick up one bag I forgot the last time when I was in Varna, Thanks God we didn’t crashed. On our way to Varna there was a very dangerous situation in which the chances to crash was pretty big but thanksfully to God’s protection and kindness we didn’t. Later when we came back in Dobrich I went to the police office to check if international passport is ready. Thanksfully it was and now I have the “red passport” home and ready :). I called to Mariana to great her because it was the great christian feast, we at the Orthodox Church believe that at that day the Eartly Mother of our Lord Jesus has resurrected in the 3rd day and ascended to heaven just like our Lord! We call that celebrity “The Maryam’s day”. Everybody who is named after Mary’s name is also celebrating this great feast. We decided to meet at night time at 11:30 and have walk. Like I said earlier I really enjoy Mariana’s company. Unfortunately an hour later we met Bino so we wasn’t able to talk much about stuff with Mariana. Bino is pretty cool guy but sometimes his company is pretty annoying :). Later I went home and after a minute of prayer I went to bed. On August 22 I’m traveling to the Netherlands to continue my studies at the HRQM stream, sometimes I feel a sort of preliminary homesickness but I believe this decision is right and it’s God’s souvereign plan for my life. Well as Bugs Bunny (my favour cartoon character) says that’s all folks!END—–

Light, Camera, Action!

Wednesday, July 18th, 2007

It was a pretty hot day. In the morning we discussed a lot about the idea to move one of the colocated servers from Netinfo to Evolink. Also I have moved www.bcc.bg from one machine to another did various other Job, plamenko come home and we uploaded some of his other videos, Damqncho called and was my guest for some time ( This guy is going to become a good man in Faith I’m sure). In 6:00 we had to go to the ex-calculation center and to sign for the Cisco Academy with Mitko but it happened that he had urgent work in Balchik so I have to go alone. I used Plamenko’s bike to go to the calculation center which was located at the end of the city (Riding bike is great !). I signed for Cisco and signed also Niki (Mitko’s brother) and Niki. So we are going to share the same Cisco class! :]. After that I went to Mitko’s home to explain to his brother about the cisco way of studying. Also I forgot to mention that Doncho, has given us all the material for the semestar on a cd, we are going to have 4 semesters for the first degree, each of the semestars is going to be something like 2 months or so, the practice is going to be in Saturday. Later I have met Alex and we drink beer together. Later I saw Lily for a while and we had a walk I met Galio ( A homeless boy :[), and bought him some food, later I realised that I have lost my wallet with some money I walked again the walk where I walked after I bought him food but I didn’t found it I get a little distressed but at the same time I was feeling very calm (unusual for such a situations ). I have put my hope God would give it back. First I suspected that Galio has stolen the wallet from me but I was wrong this boy respects me. I found Galio and asked him did he stole the money, he denied to be guilty and I trusted him because he was honest. He was very kind to me he came with me looking again over all the way where I have walked at last I suggested to look in the central park where we have drinked beer together with Alex, under the bench where we have seated Me and Galio found the Wallet. And we got really happy about this. I checked my wallet there was 22 lv in there. When we was still searching the wallet I promised to God that If we found the wallet I will give 10 lv. to for his kindness and concern about my problem. After I have found the wallet I have completed my promise. Galio walked with me up to my home. The End. The only think that I can say after all this is PRAISE THE LORD, BLESS HIS HOLY NAME OH HEAVENS AND EARTH AND ALL LIVING, AND OH MY SOUL BLESS THE LORD GOD! BLESSED BE OH LORD!!!! :] END—–

Durankulak’s beach

Monday, August 27th, 2007

In Saturday we was on a Cisco course with Nomen, Niki (A friend programmer),and the Other Niki (Mitko’s brother). The course went smoothly up to some pointafter that Niki (the programmer) has received a call with an awful news…After the course I Nomen and Nomen’s brother went to the chineese restaurant and we ate rice with vegatables and spaghetti with vegetables and meat. An hour later we was at Nomen’s home we had to make the cisco’s Chapter One and Chapter two test. And Luckily I got 100% right answers on both of the two (I have to be honest that I used cheating and tricks on the tests so I probably deserve less.After that Nomen gave suggestion to go to Carapec or somewhere on the Beach on the Bulgarian Coast. At the end we ended beaching the next morning on Durankulak’s Camping beach after we setupped a fire for the night and we slept in tents. There was a lot of problems during the whole trip ( I won’t go into details) but Thanks to God Almighty all has ended well in the end. Talking about God, I’m smoking cigarettes again and I have to stop (I hope God would help again). Also something I have to note is I’m a sinner but God is faithful although I sin badly The Lord is gracious to me still. PRAISE THE LORD!!! HalleluJah! :]END—–

The day, Today

Tuesday, May 20th, 2008

The day started a bit normal. I did my morning excercise, then I prayed. I spoke with Dzemil (A macedonian colleague of mine) and we set up a meeting for 12:30, I ate. I received few calls from the office with requests to do few little things. At 12:30 I met Dzemil at the College restaurant. We spend some time talking with him and another turkish colleague. Then we went to speak with Bozhidar Bozhkov about the applications for Holland, what is the procedure of transfering from the college here to Arnhem Business School etc. Laters I went home and did some work on the servers and red and did my fourth cisco test. I went to my cousin and after that went to Javor, we went out with Ina and Javor for a coffee to Kukla. Afterwards I went home and played with Dynamips. For all that wonder what the hack Dynamips is. Well Dynamics is a Cisco emulator just like VMWare is an OS emulator with the exception that Dynamics is builded to run only Cisco’s IOS. I found that nice Video tutorial Cisco Router Emulation Software Dynamips Video Tutorial, check it out here Here . Since I needed a Cisco IOS image and I’m not a Cisco customer I used torrents to download a collection of Cisco ISO’s and used one of the isos to make it work on my Windows Vista. I have problems running it because of lack of permissions, caused by the famous UAC ( User Access Control ). The solution for me was to use a privileged command prompt and start, both the Dynamips sever and my custom configured simple1.net which connected to the server and loaded the cisco image. There is also a very nice and extended tutorial on the topic of Dynamips it’s located Here . Alto today tested the previously installed Wireshark. Wireshark is a very nice substitute for iptraf for windows it has a nice and easy to use graphical interface, supports capturing and has lot of traffic analysis possibilities I strongly recommend it to anyone coming from a Linux/BSD background like me and searching for a nice Windows substitute for iptraf. Check out wireshark on the following URL . Now I’m going to change the topic and say a few words for my spiritual state. Today it was a hard day. I was tempted by the devil to think bad thoughts and did sinned for which I search forgiveness. Life it so hard I realize it more and more day by day. Very often old spirits which tormented me for a long time are trying to come back. I haven’t smoked today also and again thanks for that should fly to God who delived me from this terrible vice. As a conclusion I should say that for everything I should thanks to God and pray for him to forgive my unfaithfulness. END—–