Posts Tagged ‘connections’

How to count number of ESTABLISHED state TCP connections to a Windows server

Wednesday, March 13th, 2024

count-netstat-established-connections-on-windows-server-howto-windows-logo-debug-network-issues-windows

Even if you have the background of a Linux system administrator, sooner or later you will have have to deal with some Windows hosts, thus i'll blog in this article shortly on how the established TCP if it happens you will have to administarte a Windows hosts or help a windows sysadmin noobie 🙂

In Linux it is pretty easy to check the number of established conenctions, because of the wonderful command wc (word count). with a simple command like:
 

$ netstat -etna |wc -l


Then you will get the number of active TCP connections to the machine and based on that you can get an idea on how busy the server is.

But what if you have to deal with lets say a Microsoft Windows 2012 /2019 / 2020 or 2022 Server, assuming you logged in as Administrator and you see the machine is quite loaded and runs multiple Native Windows Administrator common services such as IIS / Active directory Failover Clustering, Proxy server etc.
How can you identify the established number of connections via a simple command in cmd.exe?

1.Count ESTABLISHED TCP connections from Windows Command Line

Here is the answer, simply use netstat native windows command and combine it with find, like that and use the /i (ignores the case of characters when searching the string) /c (count lines containing the string) options

C:\Windows\system32>netstat -p TCP -n|  find /i "ESTABLISHED" /c
1268

Voila, here are number of established connections, only 1268 that is relatively low.
However if you manage Windows servers, and you get some kind of hang ups as part of the monitoring, it is a good idea to setup a script based on this simple command for at least Windows Task Scheduler (the equivallent of Linux's crond service) to log for Peaks in Established connections to see whether Server crashes are not related to High Rise in established connections.
Even better if company uses Zabbix / Nagios, OpenNMS or other  old legacy monitoring stuff like Joschyd even as of today 2024 used in some big of the TOP IT companies such as SAP (they were still using it about 4 years ago for their SAP HANA Cloud), you can set the script to run and do a Monitoring template or Alerting rules to draw you graphs and Trigger Alerts if your connections hits a peak, then you at least might know your Windows server is under a "Hackers" Denial of Service attack or there is something happening on the network, like Cisco Network Infrastructure Switch flappings or whatever.

Perhaps an example script you can use if you decide to implement the little nestat established connection checks Monitoring in Zabbix is the one i've writen about in the previous article "Calculate established connection from IP address with shell script and log to zabbix graphic".

2. Few Useful netstat options for the Windows system admin
 

C:\Windows\System32> netstat -bona


netstat-useful-arguments-for-the-windows-system-administrator

Cmd.exe will lists executable files, local and external IP addresses and ports, and the state in list form. You immediately see which programs have created connections or are listening so that you can find offenders quickly.

b – displays the executable involved in  creating the connection.
o – displays the owning process ID.
n – displays address and port numbers.
a – displays all connections and listening ports.

As you can see in the screenshot, by using netstat -bona you get which process has binded to which local address and the Process ID PID of it, that is pretty useful in debugging stuff.

3. Use a Third Party GUI tool to debug more interactively connection issues

If you need to keep an eye in interactive mode, sometimes if there are issues CurrPorts tool can be of a great help

currports-windows-network-connections-diagnosis-cports

CurrPorts Tool own Description

CurrPorts is network monitoring software that displays the list of all currently opened TCP/IP and UDP ports on your local computer. For each port in the list, information about the process that opened the port is also displayed, including the process name, full path of the process, version information of the process (product name, file description, and so on), the time that the process was created, and the user that created it.
In addition, CurrPorts allows you to close unwanted TCP connections, kill the process that opened the ports, and save the TCP/UDP ports information to HTML file , XML file, or to tab-delimited text file.
CurrPorts also automatically mark with pink color suspicious TCP/UDP ports owned by unidentified applications (Applications without version information and icons).

Sum it up

What we learned is how to calculate number of established TCP connections from command line, useful for scripting, how you can use netstat to display the process ID and Process name that relates to a used Local / Remote TCP connections, and how eventually you can use this to connect it to some monitoring tool to periodically report High Peaks with TCP established connections (usually an indicator of servere system issues).
 

How to calculate connections from IP address with shell script and log to Zabbix graphic

Thursday, March 11th, 2021

We had to test the number of connections incoming IP sorted by its TCP / IP connection state.

For example:

TIME_WAIT, ESTABLISHED, LISTEN etc.


The reason behind is sometimes the IP address '192.168.0.1' does create more than 200 connections, a Cisco firewall gets triggered and the connection for that IP is filtered out. To be able to know in advance that this problem is upcoming. a Small userparameter script is set on the Linux servers, that does print out all connections from IP by its STATES sorted out.

 

The script is calc_total_ip_match_zabbix.sh is below:

#!/bin/bash
#  check ESTIMATED / FIN_WAIT etc. netstat output for IPs and calculate total
# UserParameter=count.connections,(/usr/local/bin/calc_total_ip_match_zabbix.sh)
CHECK_IP='192.168.0.1';
f=0; 

 

for i in $(netstat -nat | grep "$CHECK_IP" | awk '{print $6}' | sort | uniq -c | sort -n); do

echo -n "$i ";
f=$((f+i));
done;
echo
echo "Total: $f"

 

root@pcfreak:/bashscripts# ./calc_total_ip_match_zabbix.sh 
1 TIME_WAIT 2 ESTABLISHED 3 LISTEN 

Total: 6

 

root@pcfreak:/bashscripts# ./calc_total_ip_match_zabbix.sh 
2 ESTABLISHED 3 LISTEN 
Total: 5


images/zabbix-webgui-connection-check1

To make process with Zabbix it is necessery to have an Item created and a Depedent Item.

 

webguiconnection-check1

webguiconnection-check1
 

webgui-connection-check2-item

images/webguiconnection-check1

Finally create a trigger to trigger alarm if you have more than or eqaul to 100 Total overall connections.


images/zabbix-webgui-connection-check-trigger

The Zabbix userparameter script should be as this:

[root@host: ~]# cat /etc/zabbix/zabbix_agentd.d/userparameter_webgui_conn.conf
UserParameter=count.connections,(/usr/local/bin/webgui_conn_track.sh)

 

Some collleagues suggested more efficient shell script solution for suming the overall number of connections, below is less time consuming version of script, that can be used for the calculation.
 

#!/bin/bash -x
# show FIN_WAIT2 / ESTIMATED etc. and calcuate total
count=$(netstat -n | grep "192.168.0.1" | awk ' { print $6 } ' | sort -n | uniq -c | sort -nr)
total=$((${count// /+}))
echo "$count"
echo "Total:" "$total"

      2 ESTABLISHED
      1 TIME_WAIT
Total: 3

 


Below is the graph built with Zabbix showing all the fluctuations from connections from monitored IP. ebgui-check_ip_graph

 

How to calculate connections from IP address with shell script and log to Zabbix graphic

Thursday, March 11th, 2021

We had to test the number of connections incoming IP sorted by its TCP / IP connection state.

For example:

TIME_WAIT, ESTABLISHED, LISTEN etc.


The reason behind is sometimes the IP address '192.168.0.1' does create more than 200 connections, a Cisco firewall gets triggered and the connection for that IP is filtered out. To be able to know in advance that this problem is upcoming. a Small userparameter script is set on the Linux servers, that does print out all connections from IP by its STATES sorted out.

 

The script is calc_total_ip_match_zabbix.sh is below:

#!/bin/bash
#  check ESTIMATED / FIN_WAIT etc. netstat output for IPs and calculate total
# UserParameter=count.connections,(/usr/local/bin/calc_total_ip_match_zabbix.sh)
CHECK_IP='192.168.0.1';
f=0; 

 

for i in $(netstat -nat | grep "$CHECK_IP" | awk '{print $6}' | sort | uniq -c | sort -n); do

echo -n "$i ";
f=$((f+i));
done;
echo
echo "Total: $f"

 

root@pcfreak:/bashscripts# ./calc_total_ip_match_zabbix.sh 
1 TIME_WAIT 2 ESTABLISHED 3 LISTEN 

Total: 6

 

root@pcfreak:/bashscripts# ./calc_total_ip_match_zabbix.sh 
2 ESTABLISHED 3 LISTEN 
Total: 5


To make process with Zabbix it is necessery to have an Item created and a Depedent Item.

images/zabbix-webgui-connection-check1

 

 

 

 

webguiconnection-check1

webguiconnection-check1
 

webgui-connection-check2-item

images/webguiconnection-check1

Finally create a trigger to trigger alarm if you have more than or eqaul to 100 Total overall connections.


images/zabbix-webgui-connection-check-trigger

The Zabbix userparameter script should be as this:
cat /etc/zabbix/zabbix_agentd.d/userparameter_webgui_conn.conf
UserParameter=count.connections,(/usr/local/bin/webgui_conn_track.sh)
 

Some collleagues suggested more efficient shell script solution for suming the overall number of connections, below is less time consuming version of script, that can be used for the calculation.
 

#!/bin/bash -x
# show FIN_WAIT2 / ESTIMATED etc. and calcuate total
count=$(netstat -n | grep "192.168.0.1" | awk ' { print $6 } ' | sort -n | uniq -c | sort -nr)
total=$((${count// /+}))
echo "$count"
echo "Total:" "$total"

 

      2 ESTABLISHED
      1 TIME_WAIT
Total: 3

 


Below is the graph built with Zabbix showing all the fluctuations from connections from monitored IP.
ebgui-check_ip_graph

Check the count and monitor of established / time_wait TCP, UDP connections on Linux and Windows with netstat command

Wednesday, February 6th, 2019

netstat-windows-linux-commands-to-better-understand-your-server-type-of-networrk-tcp-udp-connections

For me as a GNU / Linux sysadmin it is intuitive to check on a server the number of established connections / connections in time_wait state and so on .

I will not explain why this is necessery as every system administrator out there who had a performance or network issues due to server / applications connection overload or have been a target of Denial of Service (DoS)
or Distributed Denial of Service attacks (DDoS)  
is well aware that a number of connections in different states such as SYN_ACK /  TIME_WAIT or ESTABLISHED state could be very nasty thing and could cause a productive application or Infrastructure service to be downed for some time causing from thousands of Euros to even millions to some bussinesses as well as some amount of data loss …

To prevent this therefore sysadmins should always take a look periodically on the Connection states on the adminned server (and in this number I say not only sys admins but DevOps guys who are deploying micro-services for a customer in the Cloud – yes I believe Richard Stallman is right here they're clouding your minds :).

Even though cloud services could provide a very high amount of Hardware (CPU / Memory / Storage) resources, often for custom applications migrating the application in the Cloud does not solve it's design faults or even problems on a purely classical system administration level.

 

1. Get a statistic for FIN_WAIT1, FOREIGN, SYNC_RECV, LAST_ACK, TIME_WAIT, LISTEN and ESTABLISHED  Connections on GNU / Linux

 

On GNU / Linux and other Linux like UNIXes the way to do it is to grep out the TCP / UDP connection type you need via netstat a very useful cmd in that case is:

 

root@pcfreak:~# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
      1 established)
      1 FIN_WAIT1
      1 Foreign
      1 SYN_RECV
      3 LAST_ACK
      4 FIN_WAIT2
      8 TIME_WAIT
     45 LISTEN
    147 ESTABLISHED

 

2. Netstat 1 liner to Get only established and time_wait connections state 

 

Other ways to check only TCP ESTABLISHED connections on Linux I use frequently are:

 

root@pcfreak:~# netstat -etna|grep -i establi|wc -l
145

 

netstat-connection-types-statistics-linux-established-time-wait-check-count

Or to get whole list of connections including the ones who are about to be esatablished in FIN_WAIT2, TIME_WAIT, SYN_RECV state:

 

root@pcfreak:~# netstat -tupen |wc -l
164

 

3. Other Linux useful one liner commands to track your connection types
 

netstat -n -p | grep SYN_REC | sort -u

List out the all IP addresses involved instead of just count.

netstat -n -p | grep SYN_REC | awk '{print $5}' | awk -F: '{print $1}'

 

List all the unique IP addresses of the node that are sending SYN_REC connection status.

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

 

Use netstat command to calculate and count the number of connections each IP address makes to the server.

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

 

List count of number of connections the IPs are connected to the server using TCP or UDP protocol.

netstat -ntu | grep ESTAB | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr

 

Check on ESTABLISHED connections instead of all connections, and displays the connections count for each IP.

 

netstat -plan|grep :80|awk {'print $5'}|cut -d: -f 1|sort|uniq -c|sort -nk 1

 

Show and list IP address and its connection count that connect to port 80 on the server. Port 80 is used mainly by HTTP web page request.

Examples are taken from this nice blog post

 

4. Check the count of esatblished connections on M$ Windows

 

As I'm forced to optimize a couple of Microsoft Windows DNS servers which are really slow to resolve the
The logical question for me was how the Established and TIME_WAIT state connections then could be checked on Windows OS, after a quick investigation online I've come up with this:

 

C:\Users\admin> netstat -nao | find /i "estab" /c
78

 

netstat-check-number-of-established-ports-connections-windows
 

 

C:\Users\admin> netsatt -nao | find /i "time_wait" /c
333

 

 

If you're used to Linux watch command, then to do same on Windows OS (e.g. check the output of netstat) command every second
and print output use:

 

netstat –an 1 | find “3334”

 

Below commands will show stats for services listening on TCP port 3334

To find out which process on system sends packets to remote destination:

 

netstat –ano 1 | find “Dest_IP_Addr”

 

The -o parameter outputs the process ID (PID) responsible for the connection.
then if you need further you can find the respective process name with tasklist< cmd.
Another handy Windows netstat option is -b which will show EXE file running as long as
the related used DLL Libraries which use TCP / UDP .

Other useful netsatat Win example is to grep for a port and show all established connections for it with:

 

netstat –an 1 | find “8080” | find “ESTABLISHED”

 

5. Closure


Hopefully this article will give you some idea on what is eating your bandwidth connections or overloading your GNU / Linux – Windows systems. And will point you to the next in line logical thing to do optimization / tuning
settings to be made on your system for example if Linux with sysctl – see my previous relater article here

I'll be intested to hear from sysadm colleagoes for other useful ways to track connections perhaps with something like ss tool (a utility to investigate sockets).
Also any optimization hints that would cause servers less downtime and improve network / performance thouroughput is mostly welcome.