Posts Tagged ‘client’

Use haproxy to dynamically modify haproxy load balancer variables, view stastics, errors and much more via stats UNIX socket with socat via command line

Friday, December 15th, 2023

haproxy-modify-dynamic-through-haproxy-unix-sock-via-socat-netcat

Haproxy could be configured to use the listen stats interface to provide a tiny web interface with statistics on all configured haproxy frontends / backends state status (UP / DOWN), current connections to proxy, errors and other interesting bandwidth information.

That is mostly useful but not every haproxy has it configured and if you did not configure the HAproxy load balancer machines on your own it might be, the previous person who build the LB infrastructure did not create the haproxy listener. 

If that is the case and you still need to get various statistics on how haproxy performs and the status of active connections towards Frotnend i/ Backend interfaces this is still possible via configured stats socket (usually this is in Global or some of the other haproxy.cfg config sections..

It is possible to do many things with haproxy such as disable / enable frotnends / backends / servers

Lets say your Haproxy has a global section that looks like this:
 

global
        stats socket /var/run/haproxy/haproxy.sock mode 0600 level admin #Creates Unix-Like socket to fetch stats
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        maxconn 99999
        nbproc          1
        nbthread 2
        cpu-map         1 0
        cpu-map         2 1

1. Listing all available options that can be send via the haproxy.sock UNIX socket interface

root@pcfreak:/home/hipo/info# echo "show help" | socat stdio /var/run/haproxy/haproxy.sock
Unknown command. Please enter one of the following commands only :
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  show tls-keys [id|*]: show tls keys references or dump tls ticket keys when id specified
  set ssl tls-key [id|keyfile] <tlskey>: set the next TLS key for the <id> or <keyfile> listener to <tlskey>
  add ssl crt-list <filename> <certfile> [options] : add a line <certfile> to a crt-list <filename>
  del ssl crt-list <filename> <certfile[:line]> : delete a line <certfile> in a crt-list <filename>
  show ssl crt-list [-n] [
] : show the list of crt-lists or the content of a crt-list <filename>
  new ssl cert <certfile> : create a new certificate file to be used in a crt-list or a directory
  set ssl cert <certfile> <payload> : replace a certificate file
  commit ssl cert <certfile> : commit a certificate file
  abort ssl cert <certfile> : abort a transaction for a certificate file
  del ssl cert <certfile> : delete an unused certificate file
  show ssl cert [
] : display the SSL certificates used in memory, or the details of a <certfile>
  set maxconn global : change the per-process maxconn setting
  set rate-limit : change a rate limiting value
  set severity-output [none|number|string] : set presence of severity level in feedback information
  set timeout    : change a timeout setting
  show env [var] : dump environment variables known to the process
  show cli sockets : dump list of cli sockets
  show cli level   : display the level of the current CLI session
  show fd [num] : dump list of file descriptors in use
  show activity : show per-thread activity stats (for support/developers)
  operator       : lower the level of the current CLI session to operator
  user           : lower the level of the current CLI session to user
  clear counters : clear max statistics counters (add 'all' for all counters)
  show info      : report information about the running process [desc|json|typed]*
  show stat      : report counters for each proxy and server [desc|json|typed]*
  show schema json : report schema used for stats
  show sess [id] : report the list of current sessions or dump this session
  shutdown session : kill a specific session
  shutdown sessions server : kill sessions on a server
  disable agent  : disable agent checks (use 'set server' instead)
  disable health : disable health checks (use 'set server' instead)
  disable server : disable a server for maintenance (use 'set server' instead)
  enable agent   : enable agent checks (use 'set server' instead)
  enable health  : enable health checks (use 'set server' instead)
  enable server  : enable a disabled server (use 'set server' instead)
  set maxconn server : change a server's maxconn setting
  set server     : change a server's state, weight or address
  get weight     : report a server's current weight
  set weight     : change a server's weight (deprecated)
  show startup-logs : report logs emitted during HAProxy startup
  clear table    : remove an entry from a table
  set table [id] : update or create a table entry's data
  show table [id]: report table usage stats or dump this table's contents
  add acl        : add acl entry
  clear acl <id> : clear the content of this acl
  del acl        : delete acl entry
  get acl        : report the patterns matching a sample for an ACL
  show acl [id]  : report available acls or dump an acl's contents
  add map        : add map entry
  clear map <id> : clear the content of this map
  del map        : delete map entry
  get map        : report the keys and values matching a sample for a map
  set map        : modify map entry
  show map [id]  : report available maps or dump a map's contents
  show events [
] : show event sink state
  show threads   : show some threads debugging information
  show peers [peers section]: dump some information about all the peers or this peers section
  disable frontend : temporarily disable specific frontend
  enable frontend : re-enable specific frontend
  set maxconn frontend : change a frontend's maxconn setting
  show servers conn [id]: dump server connections status (for backend <id>)
  show servers state [id]: dump volatile server information (for backend <id>)
  show backend   : list backends in the current running config
  shutdown frontend : stop a specific frontend
  set dynamic-cookie-key backend : change a backend secret key for dynamic cookies
  enable dynamic-cookie backend : enable dynamic cookies on a specific backend
  disable dynamic-cookie backend : disable dynamic cookies on a specific backend
  show errors    : report last request and response errors for each proxy
  show resolvers [id]: dumps counters from all resolvers section and
                     associated name servers
  show pools     : report information about the memory pools usage
  show profiling : show CPU profiling options
  set  profiling : enable/disable CPU profiling
  show cache     : show cache status
  trace <module> [cmd [args…]] : manage live tracing
  show trace [
] : show live tracing state
 

2. View haproxy running threads

root@pcfreak:/home/hipo/info# echo "show threads" | socat stdio /var/run/haproxy/haproxy.sock
  Thread 1 : id=0x7f87b6e2c1c0 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
             stuck=0 prof=0 harmless=1 wantrdv=0
             cpu_ns: poll=3061065069437 now=3061065077880 diff=8443
             curr_task=0
* Thread 2 : id=0x7f87b6e20700 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
             stuck=0 prof=0 harmless=0 wantrdv=0
             cpu_ns: poll=2969050092523 now=2969050197848 diff=105325
             curr_task=0x7f87b006f740 (task) calls=1 last=0
               fct=0x560978846340(task_run_applet) ctx=0x7f87b0190720(<CLI>)
             strm=0x56097a763560 src=unix fe=GLOBAL be=GLOBAL dst=<CLI>
             rqf=c48200 rqa=0 rpf=80008000 rpa=0 sif=EST,200008 sib=EST,204018
             af=(nil),0 csf=0x56097a776ef0,8200
             ab=0x7f87b0190720,9 csb=(nil),0
             cof=0x56097a77fb00,1300:PASS(0x7f87b019a680)/RAW((nil))/unix_stream(22)
             cob=(nil),0:NONE((nil))/NONE((nil))/NONE(0)

3. Show haproxy server connections

root@pcfreak:/home/hipo/info# echo "show servers conn" | socat stdio /var/run/haproxy/haproxy.sock
# bkname/svname bkid/svid addr port – purge_delay used_cur used_max need_est unsafe_nb safe_nb idle_lim idle_cur idle_per_thr[2]
http-websrv/ha1server-1 3/1 192.168.0.209 80 – 5000 0 12 12 0 0 -1 0 0 0
http-websrv/ha1server-2 3/2 192.168.0.200 80 – 5000 1 142 142 0 0 -1 0 0 0
http-websrv/ha1server-3 3/3 192.168.1.30 80 – 5000 0 0 0 0 0 -1 0 0 0
http-websrv/ha1server-4 3/4 192.168.1.14 80 – 5000 0 0 0 0 0 -1 0 0 0
http-websrv/ha1server-5 3/5 192.168.0.1 80 – 5000 0 13 13 0 0 -1 0 0 0
https-websrv/ha1server-1 5/1 192.168.0.209 443 – 5000 0 59 59 0 0 -1 0 0 0
https-websrv/ha1server-2 5/2 192.168.0.200 443 – 5000 11 461 461 0 0 -1 0 0 0
https-websrv/ha1server-3 5/3 192.168.1.30 443 – 5000 0 0 0 0 0 -1 0 0 0
https-websrv/ha1server-4 5/4 192.168.1.14 443 – 5000 0 0 0 0 0 -1 0 0 0
https-websrv/ha1server-5 5/5 192.168.0.1 443 – 5000 1 152 152 0 0 -1 0 0 0
MASTER/cur-1 6/1 – 0 – 0 0 0 0 0 0 0 0

4. Show Load balancer servers state

root@pcfreak:/home/hipo/info# echo "show servers state" | socat stdio /var/run/haproxy/haproxy.sock
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
3 http-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 3929 6 3 4 6 0 0 0 – 80 –
3 http-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 3928 6 3 4 6 0 0 0 – 80 –
3 http-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 3927 6 3 4 6 0 0 0 – 80 –
3 http-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 3929 6 3 4 6 0 0 0 – 80 –
3 http-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1708087 6 3 4 6 0 0 0 – 80 –
5 https-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 3929 6 3 4 6 0 0 0 – 443 –
5 https-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 3928 6 3 4 6 0 0 0 – 443 –
5 https-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 3927 6 3 4 6 0 0 0 – 443 –
5 https-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 3929 6 3 4 6 0 0 0 – 443 –
5 https-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1708087 6 3 4 6 0 0 0 – 443 –
6 MASTER 1 cur-1 – 2 0 0 0 1708087 1 0 0 0 0 0 0 – 0 –

5. Get general haproxy info on variables that can be used for Load Balancer fine tuning

root@pcfreak:/home/hipo/info# echo "show info" | socat stdio /var/run/haproxy/haproxy.sock
Name: HAProxy
Version: 2.2.9-2+deb11u5
Release_date: 2023/04/10
Nbthread: 2
Nbproc: 1
Process_num: 1
Pid: 3103635
Uptime: 19d 18h11m49s
Uptime_sec: 1707109
Memmax_MB: 0
PoolAlloc_MB: 1
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 200059
Maxsock: 200059
Maxconn: 99999
Hard_maxconn: 99999
CurrConns: 8
CumConns: 19677218
CumReq: 2740072
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 1
ConnRateLimit: 0
MaxConnRate: 2161
SessRate: 1
SessRateLimit: 0
MaxSessRate: 2161
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 32
Run_queue: 1
Idle_pct: 100
node: pcfreak
Stopping: 0
Jobs: 13
Unstoppable Jobs: 0
Listeners: 4
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0
FailedResolutions: 0
TotalBytesOut: 744390344175
BytesOutRate: 30080
DebugCommandsIssued: 0
Build info: 2.2.9-2+deb11u5
 

root@pcfreak:/home/hipo/info# echo "show errors" | socat stdio /var/run/haproxy/haproxy.sock
Total events captured on [14/Dec/2023:17:29:17.930] : 0

6. View all opened sessions and, the session age (time since it has been opened) and session exp (expiry)

root@pcfreak:/home/hipo/info# echo "show sess" | socat stdio /var/run/haproxy/haproxy.sock
0x56097a763560: proto=tcpv4 src=113.120.74.123:54651 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=37s calls=3 rate=0 cpu=0 lat=0 rq[f=848000h,i=0,an=00h,rx=1m58s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=,wx=1m58s,ax=] s0=[8,200000h,fd=24,ex=] s1=[8,40018h,fd=25,ex=] exp=1m51s
0x56097a812830: proto=tcpv4 src=190.216.236.134:35526 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=17s calls=3 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m42s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m42s,wx=,ax=] s0=[8,200008h,fd=40,ex=] s1=[8,200018h,fd=41,ex=] exp=12s
0x56097a784ad0: proto=tcpv4 src=103.225.203.131:33835 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=17s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m44s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m44s,wx=,ax=] s0=[8,200008h,fd=20,ex=] s1=[8,200018h,fd=21,ex=] exp=13s
0x7f87b0082cc0: proto=tcpv4 src=190.216.236.134:35528 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=14s calls=3 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m46s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m46s,wx=,ax=] s0=[8,200008h,fd=34,ex=] s1=[8,200018h,fd=35,ex=] exp=15s
0x7f87b0089e10: proto=tcpv4 src=40.130.105.242:50669 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=11s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m49s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m49s,wx=,ax=] s0=[8,200008h,fd=15,ex=] s1=[8,200018h,fd=16,ex=] exp=18s
0x7f87b010b450: proto=tcpv4 src=64.62.202.82:37562 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=7s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m52s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m52s,wx=,ax=] s0=[8,200008h,fd=26,ex=] s1=[8,200018h,fd=27,ex=] exp=22s
0x56097a7b8bc0: proto=tcpv4 src=85.208.96.211:54226 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=0s calls=2 rate=2 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m59s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m59s,wx=,ax=] s0=[8,200008h,fd=22,ex=] s1=[8,200018h,fd=23,ex=] exp=29s
0x7f87b008ec00: proto=tcpv4 src=3.135.192.206:60258 fe=http-in be=http-websrv srv=ha1server-2 ts=00 age=0s calls=2 rate=2 cpu=0 lat=0 rq[f=848000h,i=0,an=00h,rx=1m59s,wx=1m59s,ax=] rp[f=80008000h,i=0,an=00h,rx=1m59s,wx=1m59s,ax=] s0=[8,200008h,fd=28,ex=] s1=[8,200018h,fd=29,ex=] exp=29s
0x56097a7b2490: proto=tcpv4 src=45.147.249.119:62283 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=0s calls=3 rate=3 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m59s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m59s,wx=,ax=] s0=[8,200008h,fd=17,ex=] s1=[8,200018h,fd=18,ex=] exp=29s
0x7f87b0114f90: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c48200h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp=

root@pcfreak:/home/hipo/info#

7. Disabling an haproxy frontend via UNIX socket

If you get some frontend that gets broken and this is monitored in Zabbix or other monitoring tool used to monitor you can use the haproxy stats interface to disable frontend

root@pcfreak:/home/hipo/info# echo "disable frontend https-websrv" | socat stdio /var/run/haproxy/haproxy.sock

8. Show general haproxy statistics (could tell you much about customer connections health state) and state of connection to backend

Lets check uptime details for frontends / backends, that is done with show stat command.

root@pcfreak:/home/hipo/info# echo "show stat" | socat stdio /var/run/haproxy/haproxy.sock
#

 

pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp

,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,
pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status

,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx

,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt

,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk

,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,

agent_duration,check_desc,agent_desc,check_rise,

check_fall,check_health,agent_rise,

agent_fall,agent_health,addr,cookie,mode,

algo,conn_rate,conn_rate_max,conn_tot,intercepted

,dcon,dses,wrew,connect,reuse,cache_lookups,

cache_hits,srv_icur,src_ilim,qtime_max,ctime_max,

rtime_max,ttime_max,eint,idle_conn_cur,

safe_conn_cur,used_conn_cur,need_conn_est,

    http-in,FRONTEND,,,0,142,99999,371655,166897324,

1462777381,0,0,62,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,

1080,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,0,1080,

371655,,0,0,0,,,,,,,,,,,0,,,,,

    http-websrv,ha1server-1,0,0,0,12,,9635,3893561

,64880833,,0,,0,3,15,0,UP

,254,0,1,41,9,4686,34728,,1,3,1,,4924,,2,0,,56,L4OK

,,0,,,,,,,,,,,900,168,,,,,1292679,,,0,0,0,2843,,,,

Layer4 check passed,,2,3,4,,,,192.168.0.209:80,,tcp,,,,,,,,

0,9635,0,,,0,,0,15024,0,672888,0,0,0,0,12,

    http-websrv,ha1server-2,0,0,0,142,,321867,

149300590,1350577153,,0,,

1,4,30,0,UP,255,1,0,37,10,4685,89418,,1,3,2,,111864,,2

,0,,1080,L4OK,,0,,,,,,,,,,,37161,4822,,,,,6,,,0,12,0,

2120,,,,Layer4 check passed,,2,3,4,,,,192.168.0.200:80,,tcp,,,,,,,,0,321867,

0,,,0,,0,30223,0,1783442,0,0,0,0,142,

List continues here
….

..
.

 

 

 

 

9. Using netcat to view UNIX socket instead of socat

If you don't have the socat command on the server but you have netcat installed, you can also send the commands to the running haproxy daemon via nc's capability to send via UNIX socket via nc -U option.

   -U      Use UNIX-domain sockets.  Cannot be used together with -F or -x.

 

root@pcfreak:/home/hipo/info# echo "set server"|nc -U /var/run/haproxy/haproxy.sock
Require 'backend/server'.

10. Get only statistics about running LB Backends and Frontends

To get only haproxy statistics about running Load Balancer BACKENDs and FRONTENDs

root@pcfreak:/home/hipo/info# echo "show stat" | sudo socat unix-connect:/var/run/haproxy/haproxy.sock stdio | awk -F '.' '/BACKEND/ {print $1, $6}'
http-websrv,BACKEND,0,0,2,142,10000,371880,167022255,1462985601,0,0,,1,7,46,0,UP

,255,1,4,,0,1709835,0,,1,3,0,,118878,,1,0,,1080,,,,,,,,,,,,,,38782,5001,0,0,0,0,5,,,0,8,0,2034

,,,,,,,,,,,,,,tcp,source,,,,,,,0,371864,0,,,,,0,30223,0,1783442,0,,,,,
https-websrv,BACKEND,0,0,5,461,10000,2374328,3083873321,740021649129,0,0,,28,42,626,0,UP
,255,1,4,,0,1709835,0,,1,5,0,,474550,,1,1,,1081,,,,,,,,,,,,,,451783,72307,0,0,0,0,0,,,0,0,0,6651

,,,,,,,,,,,,,,tcp,source,,,,,,,0,2374837,0,,,,,0,32794,0,46414141,0,,,,,

As you can see there are two configured BACKENDs that are in UP state, the other possibility is that they're DOWN if haproxy can't reach the backend.

root@pcfreak:/home/hipo/info# echo "show stat" | sudo socat unix-connect:/var/run/haproxy/haproxy.sock stdio | awk -F '.' '/FRONTEND/ {print $1, $6}'
http-in,FRONTEND,,,2,142,99999,371887,167024040,1462990718,0,0,62,,,,,OPEN

,,,,,,,,,1,2,0,,,,0,1,0,1080,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,1,1080,371887,,0,0,0,,,,,,,,,,,0,,,,,
https-in,FRONTEND,,,4,461,99999,2374337,3083881912,740021909870,0,0,112,,,,,OPEN

,,,,,,,,,1,4,0,,,,0,1,0,1081,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,tcp,,1,1081,2374337,,0,0,0,,,,,,,,,,,0,,,,,
root@pcfreak:/home/hipo/info#

As you can see from the list of show help you can change maxconns supported, change the proxy rate-limit and even in real time change a haproxy.cfg configured section timeouts or even modify ACLs dynamicly for Backends and Frontends.

If you use those to make a modifications to the haproxy, that modifications should been written also to Haproxy's configured instance haproxy.cfg file.
If you want to check it reload the haproxy instance with the new written haproxy.cfg, through the Unix socket.

11. Shutting down specific opened sessions

Shutting down specific session that has been opened for too long is particularly useful to do, especially if you have some kind of VPN encryption device before the Haproxy server and an Application Backend server that is buggy and fails to properly close sessions at time, to cut off a specific sessions that has been hanging for days after reviewing it with "show sess".

root@pcfreak:/home/hipo/info# echo "shutdown session 0x56097a7707d0" | socat stdio /var/run/haproxy/haproxy.sock

12. Sending shutdown to backend on a certain configured LB service


To bring down a configured backend on a certain server after listing it:
 

root@pcfreak:/home/hipo/info# echo "disable server bk_mybackend/srv_myserver" | socat /var/run/haproxy.sock stdio


12. Sending multiple commands to haproxy socket

# echo "show info;show stat" | socat /var/run/haproxy/haproxy.sock stdio

 

13. Report table usage information or dump table data content


It is possible to view exact queued connections inside the sticky table. To get a list of available, available configured tables on the haproxy

root@pcfreak:/home/hipo/info# echo "show table" | socat /var/run/haproxy/haproxy.sock stdio
# table: https-websrv, type: ip, size:204800, used:498
# table: http-websrv, type: ip, size:204800, used:74


To get the exact record of queued IPs inside https-websrv.

root@pcfreak:/home/hipo/info# echo "show table https-websrv" | socat /var/run/haproxy/haproxy.sock stdio|head -10
# table: https-websrv, type: ip, size:204800, used:502
0x56097a7444e0: key=2.147.73.42 use=0 exp=1090876 server_id=2 server_name=ha1server-2
0x56097a792ac0: key=3.14.130.119 use=0 exp=1038004 server_id=2 server_name=ha1server-2
0x7f87b006a4e0: key=3.15.203.28 use=0 exp=1536721 server_id=2 server_name=ha1server-2
0x56097a7467f0: key=3.16.54.132 use=0 exp=387191 server_id=2 server_name=ha1server-2
0x7f87b0075f90: key=3.17.180.28 use=0 exp=353211 server_id=2 server_name=ha1server-2
0x56097a821b10: key=3.23.114.130 use=0 exp=1521100 server_id=2 server_name=ha1server-2
0x56097a7475b0: key=3.129.250.144 use=0 exp=121043 server_id=2 server_name=ha1server-2
0x7f87b004d240: key=3.134.112.27 use=0 exp=1182169 server_id=2 server_name=ha1server-2
0x56097a754c90: key=3.135.192.206 use=0 exp=1383882 server_id=2 server_name=ha1server-2

14. Show information about Haproxy startup

Sometimes, where logrotation is integrated on the server and haproxy's logs are log rotated to a central logging server, it might be hard to get information about Haproxy startup messages (warnings, errors etc.).
As digging through old haproxy logs might be tedious, you can simply get it via the stats interface.

root@pcfreak:/home/hipo/info# echo "show startup-logs" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio              

[WARNING] 327/231534 (3103633) : parsing [/etc/haproxy/haproxy.cfg:62] : 'fullconn' ignored because frontend 'http-in' has no backend capability. Maybe you want 'maxconn' instead ?
[WARNING] 327/231534 (3103633) : parsing [/etc/haproxy/haproxy.cfg:69] : 'maxconn' ignored because backend 'http-websrv' has no frontend capability. Maybe you want 'fullconn' instead ?
[WARNING] 327/231534 (3103633) : parsing [/etc/haproxy/haproxy.cfg:114] : 'maxconn' ignored because backend 'https-websrv' has no frontend capability. Maybe you want 'fullconn' instead ?
[WARNING] 327/231534 (3103633) : config : missing timeouts for frontend 'http-in'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for frontend 'http-in' as it requires HTTP mode.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for backend 'http-websrv' as it requires HTTP mode.
[WARNING] 327/231534 (3103633) : config : missing timeouts for frontend 'https-in'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for frontend 'https-in' as it requires HTTP mode.
[WARNING] 327/231534 (3103633) : config : 'option forwardfor' ignored for backend 'https-websrv' as it requires HTTP mode.

15. Disable / Enable health check for haproxy configured backend

 Disable health checks is useful, especially on non production server environments, during integration phase of application with Haproxy load balancer.

The general syntax is like this:

> disable health backend/server1

 

root@pcfreak:/home/hipo/info# echo "show servers state" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio             1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
3 http-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 13709 6 3 4 6 0 0 0 – 80 –
3 http-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 13708 6 3 4 6 0 0 0 – 80 –
3 http-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 13707 6 3 4 6 0 0 0 – 80 –
3 http-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 13709 6 3 4 6 0 0 0 – 80 –
3 http-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1717867 6 3 4 6 0 0 0 – 80 –
5 https-websrv 1 ha1server-1 192.168.0.209 2 0 254 254 13709 6 3 4 6 0 0 0 – 443 –
5 https-websrv 2 ha1server-2 192.168.0.200 2 0 255 255 13708 6 3 4 6 0 0 0 – 443 –
5 https-websrv 3 ha1server-3 192.168.1.30 2 0 252 252 13707 6 3 4 6 0 0 0 – 443 –
5 https-websrv 4 ha1server-4 192.168.1.14 2 0 253 253 13709 6 3 4 6 0 0 0 – 443 –
5 https-websrv 5 ha1server-5 192.168.0.1 2 0 251 251 1717867 6 3 4 6 0 0 0 – 443 –
6 MASTER 1 cur-1 – 2 0 0 0 1717867 1 0 0 0 0 0 0 – 0 –

 

Lets disable health checks for ha1server-1 server and http-websrv backend.

root@pcfreak:/home/hipo/info# echo "disable health http-websrv/ha1server-1" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio

 

To enable back health checks 

root@pcfreak:/home/hipo/info# echo "enable health http-websrv/ha1server-1" | socat unix-connect:/var/run/haproxy/haproxy.sock stdio

16. Change weight for server

if you have a round-robin Load balancing configured and already have a predefined configuration on how many percentage of the server to be sent to which application server (e.g. have a configured weight to dynamically change it via UNIX sock iface).

# Change weight by percentage of its original value

# socat unix-connect:/var/run/haproxy/haproxy.sock stdio


> set server be_app/webserv1 weight 50%
 
# Change weight in proportion to other servers
> set server be_app/webserv1 weight 100

 

root@pcfreak:/home/hipo/info#  socat unix-connect:/var/run/haproxy/haproxy.sock stdio                                        
set server http-websrv/ha1server-1 weight 50%
Backend is using a static LB algorithm and only accepts weights '0%' and '100%'.

17. Draining traffic from server / backend App in case of Maintenance

You can gradually drain traffic away from a particular server if those backend Application server should be put in maintenance mode for update or whatever. The drain option is very interesting and combined with scripting does open a lot of possibilities for the Load balancer system administrator to put an extra automation.

To drain, set server command with the state argument set to drain:
 

# Drain traffic
> set server backend_app/server1 state drain

# Allow server to accept traffic again
> set server backend_app/server1 state ready

 


root@pcfreak:/home/hipo/info#  socat unix-connect:/var/run/haproxy/haproxy.sock stdio
set server http-websrv/ha1server-1 state drain

 

root@pcfreak:/home/hipo/info#  socat unix-connect:/var/run/haproxy/haproxy.sock stdio
set server http-websrv/ha1server-1 state ready

18. Run Interactive Mode connection to haproxy UNIX stats socket

For a haproxies that has multiple configured proxied rules backends / frontends, it is nice to use the interactive mode.
Instead of processing a single line of semicolon separate commands, HAProxy takes one command at a time and waits for the user.
In interactive mode, HAProxy sends a “>” character and waits for input command. After command is submitted, HAProxy sends back the result and waits for a new command.
The interactive mode is especially useful during phase of integrating a new haproxy towards an application, where multiple things has to be tuned on the fly without, reloading the haproxy again and again.

On RPM based distros socat is compiled to have the readline interactive capability. Thus to use the haproxy haproxy stats connect interactive mode on RHEL / CentOS / Fedora and other RPM based distros simply use:

# socat /var/run/haproxy.sock readline
> show info
Name: HAProxy
Version: 2.2.9-2+deb11u5
Release_date: 2023/04/10
Nbthread: 2
Nbproc: 1
Process_num: 1
Pid: 3103635
Uptime: 19d 20h48m50s
Uptime_sec: 1716530
Memmax_MB: 0
PoolAlloc_MB: 1
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 200059
Maxsock: 200059
Maxconn: 99999
Hard_maxconn: 99999
CurrConns: 9
CumConns: 19789176
CumReq: 2757976
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 2161
SessRate: 0
SessRateLimit: 0
MaxSessRate: 2161
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 35
Run_queue: 1
Idle_pct: 100
node: pcfreak
Stopping: 0
Jobs: 14
Unstoppable Jobs: 0
Listeners: 4
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0
FailedResolutions: 0
TotalBytesOut: 744964070459
BytesOutRate: 0
DebugCommandsIssued: 0
Build info: 2.2.9-2+deb11u5

On Deb (Debian) based distributions such as Debian, Ubuntu Mint Linux, unfortunately the readline inractive mode is disabled due to licensing issues that makes readline not GPL license compliant.

root@pcfreak:/home/hipo/info# socat -V|awk 'NR < 5 || tolower($0) ~ /readline/'
socat by Gerhard Rieger and contributors – see www.dest-unreach.org
socat version 1.7.4.1 on Feb  3 2021 12:58:17
   running on Linux version #1 SMP Debian 5.10.179-3 (2023-07-27), release 5.10.0-23-amd64, machine x86_64
features:
  #undef WITH_READLINE

There is a workaround to emulate the Intearactive mode on Debians however like this:

root@pcfreak:/home/hipo/info# while [ 1 ]; do socat – /var/run/haproxy/haproxy.sock ; done

show table
# table: https-websrv, type: ip, size:204800, used:511
# table: http-websrv, type: ip, size:204800, used:67

show sess
0x56097a784ad0: proto=tcpv4 src=45.61.161.66:51416 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=1m13s calls=3 rate=0 cpu=0 lat=0 rq[f=848000h,i=0,an=00h,rx=47s,wx=,ax=] rp[f=80048000h,i=0,an=00h,rx=47s,wx=,ax=] s0=[8,200008h,fd=17,ex=] s1=[8,200018h,fd=23,ex=] exp=47s
0x56097a7707d0: proto=tcpv4 src=47.128.41.242:39372 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=16s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m45s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m45s,wx=,ax=] s0=[8,200008h,fd=35,ex=] s1=[8,200018h,fd=36,ex=] exp=14s
0x56097a781300: proto=tcpv4 src=54.36.148.40:17439 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=13s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m47s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m47s,wx=,ax=] s0=[8,200008h,fd=26,ex=] s1=[8,200018h,fd=28,ex=] exp=17s
0x56097a7fca80: proto=tcpv4 src=18.217.94.243:4940 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=7s calls=2 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m53s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m53s,wx=,ax=] s0=[8,200008h,fd=21,ex=] s1=[8,200018h,fd=22,ex=] exp=23s
0x7f87b00778c0: proto=tcpv4 src=85.208.96.206:51708 fe=https-in be=https-websrv srv=ha1server-2 ts=00 age=4s calls=3 rate=0 cpu=0 lat=0 rq[f=848202h,i=0,an=00h,rx=1m56s,wx=,ax=] rp[f=80048202h,i=0,an=00h,rx=1m56s,wx=,ax=] s0=[8,200008h,fd=20,ex=] s1=[8,200018h,fd=24,ex=] exp=26s
0x56097a80c1e0: proto=unix_stream src=unix:1 fe=GLOBAL be=<NONE> srv=<none> ts=00 age=3s calls=1 rate=0 cpu=0 lat=0 rq[f=c48202h,i=0,an=00h,rx=10s,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,200008h,fd=15,ex=] s1=[8,204018h,fd=-1,ex=] exp=7s


To end the eternal loop press CTRL + z and kill first detached job %1 run:

# kiill %1


Sum it up what learned

What we learned in this article is how to use socat and netcat to connect and manage dynamically haproxy via its haproxy stats interface, without reloading the proxqy itself. We learned how to view various statistics and information on the proxy, its existing tables, caches, session information (such as age, and expiry). Also you've seen how to disable / enable configured backends as well as get available backends and frontends and their state.
You've seen how the drained option could be used to slowly drain connections towards configured backend, in case if you need to a maintenance on a backend node.
Also was pointed how to shutdown a specific long lived sessions that has been hanging and creating troubles towards app backends.

Finally, you've seen how to open an interactive connection towards the haproxy socket and send commands in a raw with socat (on distros where compiled with readline support) as well shown how to emulate the interactive mode of rest of distros whose socat is missing the readline support. 

Monitor service log is continously growing with Zabbix on Windows with batch userparameter script and trigger Alert if log is unchanged

Thursday, March 17th, 2022

monitor-if-log-file-is-growing-with-zabbix-zabbix-userparameter-script-howto

Recently we had an inteteresting Monitoring work task to achieve. We have an Application that is constantly simulating encrypted connections traffic to a remote side machine and sending specific data on TCP/IP ports.
Communiucation between App Server A -> App Server B should be continous and if all is working as expected App Server A messages output are logged in the Application log file on the machine which by the way Runs
Windows Server 2020.

Sometimes due to Network issues this constant reconnections from the Application S. A to the remote checked machine TCP/IP ports gets interrupted due to LAN issues or a burned Network Switch equipment, misconfiguration on the network due to some Network admin making stoopid stuff etc..

Thus it was important to Monitor somehow whether the log is growing or not and feed the output of whether Application log file is growing or it stuck to a Central Zabbix Server. 
To be able to better understand the task, lets divide the desired outcome in few parts on required:

1. Find The latest file inside a folder C:\Path-to-Service\Monitoring\Log\
2. Open the and check it is current logged records and log the time
3. Re-open file in a short while and check whether in few seconds new records are written
4. Report the status out to Zabbix
5. Make Zabbix Item / Trigger / Action in case if monitored file is not growing

In below article I'll briefly explain how Monitoring a Log on a Machine for growing was implemented using a pure good old WIN .BAT (.batch) script and Zabbix Userparameter key

 

1. Enable userparameter script for Local Zabbix-Agent on the Windows 10 Server Host


Edit Zabbix config file usually on Windows Zabbix installs file is named:

zabbix_agentd.win ]


Uncomment the following lines to enable userparameter support for zabbix-agentd:

 

# Include=c:\zabbix\zabbix_agentd.userparams.conf

Include=c:\zabbix\zabbix_agentd.conf.d\

# Include=c:\zabbix\zabbix_agentd.conf.d\*.conf


2. Create folders for userparameter script and for the userparameter.conf

Before creating userparameter you can to create the folder and grant permissions

Folder name under C:\Zabbix -> zabbix_agentd.conf.d

If you don't want to use Windows Explorer) but do it via cmd line:

C:\Users\LOGUser> mkdir \Zabbix\zabbix_agentd.conf\
C:\User\LOGUser> mkdir \Zabbix\zabbix_scripts\


3. Create Userparameter with some name file ( Userparameter-Monitor-Grow.conf )

In the directory C:\Zabbix\zabbix_agentd.conf.d you should create a config file like:
Userparameter-Monitor-Grow.conf and in it you should have a standard userparameter key and script so file content is:

UserParameter=service.check,C:\Zabbix\zabbix_scripts\GROW_LOG_MONITOR-USERPARAMETER.BAT


4. Create the Batch script that will read the latest file in the service log folder and will periodically check and report to zabbix that file is changing

notepad C:\Zabbix\zabbix_scripts\GROW_LOG_MONITOR-USERPARAMETER.BAT

REM "SCRIPT MONITOR IF FILE IS GROWING OR NOT"
@echo off

set work_dir=C:\Path-to-Service\Monitoring\Log\

set client=client Name

set YYYYMMDD=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%

set name=csv%YYYYMMDD%.csv

set mytime=%TIME:~0,8%

for %%I in (..) do set CurrDirName=%%~nxI

 

setlocal EnableDelayedExpansion

set "line1=findstr /R /N "^^" %work_dir%\output.csv | find /C ":""


for /f %%a in ('!line1!') do set number1=%%a

set "line2=findstr /R /N "^^" %work_dir%\%name% | find /C ":""


for /f %%a in ('!line2!') do set number2=%%a

 

IF  %number1% == %number2% (

echo %YYYYMMDD% %mytime% MAJOR the log is not incrementing for %client%

echo %YYYYMMDD% %mytime% MAJOR the log is not incrementing for %client% >> monitor-grow_err.log

) ELSE (

echo %YYYYMMDD% %mytime% NORMAL the log is incrementing for %client%

SETLOCAL DisableDelayedExpansion

del %work_dir%\output.csv

FOR /F "usebackq delims=" %%a in (`"findstr /n ^^ %work_dir%\%name%"`) do (

    set "var=%%a"

    SETLOCAL EnableDelayedExpansion

    set "var=!var:*:=!"

    echo(!var! >> %work_dir%\output.csv

    ENDLOCAL

)

)
 

 

To download GROW_LOG_MONITOR-USERPARAMETER.BAT click here.
The script needs to have configured the path to directory containing multiple logs produced by the Monitored Application.
As prior said it will, list the latest created file based on DATE timestamp in the folder will output a simple messages:

If the log file is being fed with data the script will output to output.csv messages continuously, either:

%%mytime%% NORMAL the log is incrementing for %%client%%

Or if the Monitored application log is not writting anything for a period it will start writting messages

%%mytime%%mytime MAJOR the log is not incrementing for %client%

The messages will also be sent in Zabbix.

Before saving the script make sure you modify the Full Path location to the Monitored file for growing, i.e.:

set work_dir=C:\Path-to-Service\Monitoring\Log\


5. Create The Zabbix Item

Set whatever service.check name you would like and a check interval to fetch the info from the userparameter (if you're dealing with very large log files produced by Monitored log of application, then 10 minutes might be too frequent, in most cases 10 minutes should be fine)
monitor-if-log-grows-windows-zabbix-item-service-check-screenshot
 

6. Create Zabbix Trigger


You will need a Trigger something similar to below:

Now considering that zabbix server receives correctly data from the client and the monitored log is growing you should in Zabbix:

%%mytime%% NORMAL the log is incrementing for %%client%%


7. Lastly create an Action to send Email Alert if log is not growing

Fix weird double logging in haproxy.log file due to haproxy.cfg misconfiguration

Tuesday, March 8th, 2022

haproxy-logging-front-network-back-network-diagram

While we were building a new machine that will serve as a Haproxy server proxy frontend to tunnel some traffic to a number of backends, 
came across weird oddity. The call requests sent to the haproxy and redirected to backend servers, were being written in the log twice.

Since we have two backend Application servers hat are serving the request, my first guess was this is caused by the fact haproxy
tries to connect to both nodes on each sent request, or that the double logging is caused by the rsyslogd doing something strange on each
received query. The rsyslog configuration configured to send via local6 facility to rsyslog is like that:

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514
#2022/02/02: HAProxy logs to local6, save the messages
local6.*                                                /var/log/haproxy.log


The haproxy basic global and defaults and frontend section config is like that:
 

global
    log          127.0.0.1 local6 debug
    chroot       /var/lib/haproxy
    pidfile      /run/haproxy.pid
    stats socket /var/lib/haproxy/haproxy.sock mode 0600 level admin
    maxconn      4000
    user         haproxy
    group        haproxy
    daemon

 

defaults
    mode        tcp
    log         global
#    option      dontlognull
#    option      httpclose
#    option      httplog
#    option      forwardfor
    option      redispatch
    option      log-health-checks
    timeout connect 10000 # default 10 second time out if a backend is not found
    timeout client 300000
    timeout server 300000
    maxconn     60000
    retries     3
 

listen FRONTEND1
        bind 10.71.80.5:63750
        mode tcp
        option tcplog
        log global
        log-format [%t]\ %ci:%cp\ %bi:%bp\ %b/%s:%sp\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
        balance roundrobin
        timeout client 350000
        timeout server 350000
        timeout connect 35000
        server backend-host1 10.80.50.1:13750 weight 1 check port 15000
        server backend-host2 10.80.50.2:13750 weight 2 check port 15000

After quick research online on why this odd double logging of request ends in /var/log/haproxy.log it turns out this is caused by the 

log global  defined double under the defaults section as well as in the frontend itself, hence to resolve simply had to comment out the log global in Frontend, so it looks like so:

listen FRONTEND1
        bind 10.71.80.5:63750
        mode tcp
        option tcplog
  #      log global
        log-format [%t]\ %ci:%cp\ %bi:%bp\ %b/%s:%sp\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq
        balance roundrobin
        timeout client 350000
        timeout server 350000
        timeout connect 35000
        server backend-host1 10.80.50.1:13750 weight 1 check port 15000
        server backend-host2 10.80.50.2:13750 weight 2 check port 15000

 


Next just reloaded haproxy and one request started leaving only one trail inside haproxy.log as expected 🙂
 

Configure rsyslog buffering on Linux to avoid message lost to Central Logging server

Wednesday, January 13th, 2021

rsyslog-Centralized-Logging-System-using-Rsyslog_logo

1. Rsyslog Buffering

One of the best practice about logs management is to send syslog to a central server. However, a logging system should be capable of avoiding message loss in situations where the server is not reachable. To do so, unsent data needs to be buffered at the client when central server is not available. You might have recently noticed that many servers forwarding logs messages to a central server do not have buffering functionalities activated. Thus I strongly advise you to have look to this documentation to know how to check your configuration: http://www.rsyslog.com/doc/rsyslog_reliable_forwarding.html

Rsyslog buffering with TCP/UDP configured

In rsyslog, every action runs on its own queue and each queue can be set to buffer data if the action is not ready. Of course, you must be able to detect that "the action is not ready", which means the remote server is offline. This can be detected with plain TCP syslog and RELP, but not with UDP. So you need to use either of the two. In this howto, we use plain TCP syslog.

– Version requirement

Please note that we are using rsyslog-specific features. The are required on the client, but not on the server. So the client system must run rsyslog (at least version 3.12.0), while on the server another syslogd may be running, as long as it supports plain tcp syslog.

How To Setup rsyslog buffering on Linux

First, you need to create a working directory for rsyslog. This is where it stores its queue files (should need arise). You may use any location on your local system. Next, you need to do is instruct rsyslog to use a disk queue and then configure your action. There is nothing else to do. With the following simple config file, you forward anything you receive to a remote server and have buffering applied automatically when it goes down. This must be done on the client machine.

# Example:
# $ModLoad imuxsock             # local message reception
# $WorkDirectory /rsyslog/work  # default location for work (spool) files
# $ActionQueueType LinkedList   # use asynchronous processing
# $ActionQueueFileName srvrfwd  # set file name, also enables disk mode
# $ActionResumeRetryCount -1    # infinite retries on insert failure
# $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
# *.*       @@server:port

Fix FTP active connection issues “Cannot create a data connection: No route to host” on ProFTPD Linux dedicated server

Tuesday, October 1st, 2019

proftpd-linux-logo

Earlier I've blogged about an encounter problem that prevented Active mode FTP connections on CentOS
As I'm working for a client building a brand new dedicated server purchased from Contabo Dedi Host provider on a freshly installed Debian 10 GNU / Linux, I've had to configure a new FTP server, since some time I prefer to use Proftpd instead of VSFTPD because in my opinion it is more lightweight and hence better choice for a small UNIX server setups. During this once again I've encounted the same ACTIVE FTP not working from FTP server to FTP client host machine. But before shortly explaining, the fix I find worthy to explain briefly what is ACTIVE / PASSIVE FTP connection.

 

1. What is ACTIVE / PASSIVE FTP connection?
 

Whether in active mode, the client specifies which client-side port the data channel has been opened and the server starts the connection. Or in other words the default FTP client communication for historical reasons is in ACTIVE MODE. E.g.
Client once connected to Server tells the server to open extra port or ports locally via which the overall FTP data transfer will be occuring. In the early days of networking when FTP protocol was developed security was not of such a big concern and usually Networks did not have firewalls at all and the FTP DATA transfer host machine was running just a single FTP-server and nothing more in this, early days when FTP was not even used over the Internet and FTP DATA transfers happened on local networks, this was not a problem at all.

In passive mode, the server decides which server-side port the client should connect to. Then the client starts the connection to the specified port.

But with the ever increasing complexity of Internet / Networks and the ever tightening firewalls due to viruses and worms that are trying to own and exploit networks creating unnecessery bulk loads this has changed …

active-passive-ftp-explained-diagram
 

2. Installing and configure ProFTPD server Public ServerName

I've installed the server with the common cmd:

 

apt –yes install proftpd

 

And the only configuration changed in default configuration file /etc/proftpd/proftpd.conf  was
ServerName          "Debian"

I do this in new FTP setups for the logical reason to prevent the multiple FTP Vulnerability Scan script kiddie Crawlers to know the exact OS version of the server, so this was changed to:

 

ServerName "MyServerHostname"

 

Though this is the bad security through obscurity practice doing so is a good practice.
 

3. Create iptable firewall rules to allow ACTIVE FTP mode


But anyways, next step was to configure the firewall to be allowed to communicate on TCP PORT 21 and 20 to incoming source ports range 1024:65535 (to enable ACTIVE FTP) on firewal level with iptables on INPUT and OUTPUT chain rules, like this:

 

iptables -A INPUT -p tcp –sport 1024:65535 -d 0/0 –dport 21 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp -s 0/0 –sport 1024:65535 -d 0/0 –dport 20 -m state –state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 0/0 –sport 21 -d 0/0 –dport 1024:65535 -m state –state ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -s 0/0 –sport 20 -d 0/0 –dport 1024:65535 -m state –state ESTABLISHED,RELATED -j ACCEPT


Talking about Active and Passive FTP connections perhaps for novice Linux users it might be worthy to say few words on Active and Passive FTP connections

Once firewall has enabled FTP Active / Passive connections is on and FTP server is listening, to test all is properly configured check iptable rules and FTP listener:
 

/sbin/iptables -L INPUT |grep ftp
ACCEPT     tcp  —  anywhere             anywhere             tcp spts:1024:65535 dpt:ftp state NEW,ESTABLISHED
ACCEPT     tcp  —  anywhere             anywhere             tcp spts:1024:65535 dpt:ftp-data state NEW,ESTABLISHED
ACCEPT     tcp  —  anywhere             anywhere             tcp dpt:ftp
ACCEPT     tcp  —  anywhere             anywhere             tcp dpt:ftp-data

netstat -l | grep "ftp"
tcp6       0      0 [::]:ftp                [::]:*                  LISTEN    

 

4. Loading nf_nat_ftp module and net.netfilter.nf_conntrack_helper (for backward compitability)


Next step of course was to add the necessery modules nf_nat_ftp nf_conntrack_sane that makes FTP to properly forward ports with respective Firewall states on any of above source ports which are usually allowed by firewalls, note that the range of ports given 1024:65535 might be too much liberal for paranoid sysadmins and in many cases if ports are not filtered, if you are a security freak you can use some smaller range such as 60000-65535.

 

Here is time to say for sysadmins who haven't recently had a task to configure a new (unecrypted) File Transfer Server as today Secure FTP is almost alltime used for file transfers for the sake of security might be puzzled to find out the old Linux kernel ip_conntrack_ftp which was the standard module used to make FTP Active connections work is substituted nowadays with  nf_nat_ftp and nf_conntrack_sane.

To make the 2 modules permanently loaded on next boot on Debian Linux they have to be added to /etc/modules

Here is how sample /etc/modules that loads the modules on next system boot looks like

cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
softdog
nf_nat_ftp
nf_conntrack_sane


Next to say is that in newer Linux kernels 3.x / 4.x / 5.x the nf_nat_ftp and nf_conntrack-sane behaviour changed so  simply loading the modules would not work and if you do the stupidity to test it with some FTP client (I used gFTP / ncftp from my Linux desktop ) you are about to get FTP No route to host errors like:

 

Cannot create a data connection: No route to host

 

cannot-create-a-data-connection-no-route-to-host-linux-error-howto-fix


Sometimes, instead of No route to host error the error FTP client might return is:

 

227 entering passive mode FTP connect connection timed out error


To make the nf_nat_ftp module on newer Linux kernels hence you have to enable backwards compatibility Kernel variable

 

 

/proc/sys/net/netfilter/nf_conntrack_helper

 

echo 1 > /proc/sys/net/netfilter/nf_conntrack_helper

 

To make it permanent if you have enabled /etc/rc.local legacy one single file boot place as I do on servers – for how to enable rc.local on newer Linuxes check here

or alternatively add it to load via sysctl

sysctl -w net.netfilter.nf_conntrack_helper=1

And to make change permanent (e.g. be loaded on next boot)

echo 'net.netfilter.nf_conntrack_helper=1' >> /etc/sysctl.conf

 

5. Enable PassivePorts in ProFTPD or PassivePortRange in PureFTPD


Last but not least open /etc/proftpd/proftpd.conf find PassivePorts config value (commented by default) and besides it add the following line:

 

PassivePorts 60000 65534

 

Just for information if instead of ProFTPd you experience the error on PureFTPD the configuration value to set in /etc/pure-ftpd.conf is:
 

PassivePortRange 30000 35000


That's all folks, give the ncftp / lftp / filezilla or whatever FTP client you prefer and test it the FTP client should be able to talk as expected to remote server in ACTIVE FTP mode (and the auto passive mode) will be not triggered anymore, nor you will get a strange errors and failure to connect in FTP clients as gftp.

Cheers 🙂

Export / Import PuTTY Tunnels SSH Sessions from one to another Windows machine howto

Thursday, January 31st, 2019

Putty-copy-ssh-tunnels-howto-from-one-to-another-windows-machine-3

As I've started on job position – Linux Architect in last November 2018 in Itelligence AG as a contractor (External Service) – a great German company who hires the best IT specialists out there and offers a flexible time schedules for emploees doing various very cool IT advanced operations and Strategic advancement of SAP's Cloud used Technology and Services improvements for SAP SE – SAP S4HANA and HEC (HANA Enterprise Cloud) and been given for work hardware a shiny Lenovo Thinkpad 500 Laptop with Windows 10 OS (SAP pre-installed), I needed to make some SSH Tunnels to machines to (Hop Station / Jump hosts) for that purpose, after some experimenting with MobaXterm Free (Personal Edition 11.0) and the presumable limitations of tunnels of the free client as well as my laziness to add the multiple ssh tunnels to different ssh / rdp / vnc etc. servers, finally I decided to just copy all the tunnels from a colleague who runs Putty and again use the good old Putty – old school Winblows SSH Terminal Client but just for creating the SSH tunnels and for rest use MobaXterm, just like in old times while still employe in Hewlett Packard. For that reason to copy the Tunnels from my dear German Colleague Henry Beck (A good herated collegue who works in field of Storage dealing with NetApps / filer Clusters QNap etc.).

Till that moment I had no idea how copying a saved SSH Tunnels definition is possible, I did a quick research just to find out this is done not with Putty Interface itself but, insetead through dumping Windows Putty Stored Registry records into a File, then transfer to the PC where Tunnels needs to be imported and then again (either double click the registry file) to load it, into registry or use Windows registry editor command line interface reg, here is how:
 

1. Export

 

Run cmd.exe (note below command) 

requires elevated Run as Administrator prompt:

Only sessions:

regedit /e "%USERPROFILE%\Desktop\putty-sessions.reg" HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions

All settings:

regedit /e "%USERPROFILE%\Desktop\putty.reg" HKEY_CURRENT_USER\Software\SimonTatham

Powershell:

If you have powershell installed on machine, to dump

Only sessions:

 

reg export HKCU\Software\SimonTatham\PuTTY\Sessions ([Environment]::GetFolderPath("Desktop") + "\putty-sessions.reg")

All settings:

reg export HKCU\Software\SimonTatham ([Environment]::GetFolderPath("Desktop") + "\putty.reg")


2. Import

Double-click on the 

*.reg

 file and accept the import.

 

Alternative ways:

 

cmd.exe

require elevated command prompt:

regedit /i putty-sessions.reg regedit /i putty.reg

PowerShell:

reg import putty-sessions.reg reg import putty.reg



Below are some things to consider:

Note !do not replace 

SimonTatham

 with your username.

 

Note !: It will create a 

reg

 file on the Desktop of the current user (for a different location modify path)

 

Note !: It will not export your related (old system stored) SSH keys.

What to expect next?

Putty-Tunnels-SSH-Sessions-screenshot-Windows

The result is in Putty you will have the Tunnel sessions loadable when you launch (Portable or installed) Putty version.
Press Load button over the required saved Tunnels list and there you go under

 

Connection SSH -> Tunnels 

 

you will see all the copied tunnels.

Enjoy!

Installing the phpbb forum on Debian (Squeeze/Sid) Linux

Saturday, September 11th, 2010

howto-easily-install-phpbb-on-debian-gnu-linux

I've just installed the phpbb forum on a Debian Linux because we needed a goodquick to install communication media in order to improve our internal communication in a student project in Strategic HR we're developing right now in Arnhem Business School.

Here are the exact steps I followed to have a properly it properly instlled:

1. Install the phpbb3 debian package
This was pretty straight forward:

debian:~# apt-get install phpbb3

At this point of installation I've faced a dpkg-reconfigure phpbb deb package configuration issue:
I was prompted to pass in the credentials for my MySQL password right after I've selected the MySQL as my preferred database back engine.
I've feeded my MySQL root password as well as my preferred forum database name, however the database installation failed because, somehow the configuration procedure tried to connect to my MySQL database with the htcheck user.
I guess this has to be a bug in the package itself or something from my previous installation misconfigured the way the debian database backend configuration was operating.
My assumption is that my previously installed htcheck package or something beforehand I've done right after the htcheck and htcheck-php packages installation.

after the package configuration failed still the package had a status of properly installed when I reviewed it with dpkg
I've thought about trying to manually reconfigure it using the dpkg-reconfigure debian command and I gave it a try like that:

debian:~# dpkg-reconfigure phpbb3

This time along with the other fields I've to fill in the ncurses interface I was prompted for a username before the password prompted appeared.
Logically I tried to fill in the root as it's my global privileges MySQL allowed user.
However that didn't helped at all and again the configuration tried to send the credentials with user htcheck to my MySQL database server.
To deal with the situation I had to approach it in the good old manual way.

2. Manually prepare / create the required phpbb forum database

To completet that connected to the MySQL server with the mysql client and created the proper database like so:

debian:~# mysql -u root -p
mysql>
CREATE database phpbb3forum;

3. Use phpmyadmin or the mysql client command line to create a new user for the phpbb forum

Here since adding up the user using the phpmyadmin was a way easier to do I decided to go that route, anyways using the mysql cli is also an option.

From phpmyadmin It's pretty easy to add a new user and grant privileges to a certain database, to do so navigate to the following database:

Privileges -> -> Add a new user ->

Now type your User name: , Host , Password , Re-type password , also for a Host: you have to choose Local from the drop down menu.

Leave the Database for user field empty as we have already previously created our desired database in step 2 of this article

Now press the "Go" button and the user will get created.

Further after choose the Privileges menu right on the bottom of the page once again, select through the checkbox the username you have just created let's say the previously created user is phpbb3

Go to Action (There is a picture with a man and a pencil on the right side of this button

Scroll down to the page part saying Database-specific privileges and in the field Add privileges on the following database: fill in your previosly created database name in our case it's phpbb3forum

and then press the "Go" button once again.
A page will appear where you will have to select the exact privileges you would like to grant on the specific selected database.
For some simplicity just check all the checkbox to grant as many privilegs to your database as you could.
Then again you will have to press the "Go" button and there you go you should have already configured an username and database ready to go with your new phpbb forum.

4. Create a virtualhost if you would like to have the forum as a subdomain or into a separate domain

If you decide to have the forum on a separate sub-domain or domain as I did you will have to add some kind of Virtualhost into either your Apache configuration /etc/apache2/apache2.conf or into where officially the virutualhosts are laid in Debian Linux in /etc/apache2/sites-available
I've personally created a new file like for instance /etc/apache2/sites-available/mysubdomain.mydomain.com

Here is an example content of the new Virtualhost:

<VirtualHost *>
ServerAdmin admin-email@domain.com
ServerName mysubdomain.domain.com

# Indexes + Directory Root.
DirectoryIndex index.php index.php5 index.htm index.html index.pl index.cgi index.phtml index.jsp index.py index.asp

DocumentRoot /usr/share/phpbb3/www/

# Logfiles
ErrorLog /var/log/apache2/yourdomain/error.log
CustomLog /var/log/apache2/yourdomain/access.log combined
# CustomLog /dev/null combined
<Directory /usr/share/phpbb3/www/>
Options FollowSymLinks MultiViews -Includes ExecCGI
AllowOverride All
Order allow,deny
allow from all </Directory>
</VirtualHost>

In above Virtualhost just change the values for ServerAdmin , ServerName , DocumentRoot , ErrorLog , CustomLog and Directory declaration to adjust it to your situation.

5. Restart the Apache webserver for the new Virtualhost to take affect

debian:~# /etc/init.d/apache2 restart

Now accessing your http://mysubdomain.domain.com should display the installed phpbb3 forum
The default username and password for your forum you can use straight are:

username: admin
password: admin

So far so good you by now have the PHPBB3 forum properly installed and running, however if you try to Register a new user in the forum you will notice that it's impossible because of a terrible ugly message reading:

Sorry but this board is currently unavailable.

I've spend few minutes online to scrape through the forums before I can understand what I have to stop that annoying message from appearing and allow new users to register in the phpbb forum

The solution came natural and was a setting that had to be changed with the forum admin account, thus login as admin and look at the bottom of the page, below the text reading Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group you will notice a link with Administration Control Panel
just press there a whole bunch of menus will appear on the screen allowing you to do numerous things, however what you will have to do is go to
Board Settings -> Disable Board

and change the radio button there to say No

That's all now your forum will be ready to go and your users can freely register and if the server where the forum is installed has an already running mail server, they will receive an emails with a registration data concerning their new registrations in your new phpbb forum.
Cheers and Enjoy your new shiny phpbb Forum 🙂

Block Web server over loading Bad Crawler Bots and Search Engine Spiders with .htaccess rules

Monday, September 18th, 2017

howto-block-webserver-overloading-bad-crawler-bots-spiders-with-htaccess-modrewrite-rules-file

In last post, I've talked about the problem of Search Index Crawler Robots aggressively crawling websites and how to stop them (the article is here) explaning how to raise delays between Bot URL requests to website and how to completely probhit some bots from crawling with robots.txt.

As explained in article the consequence of too many badly written or agressive behaviour Spider is the "server stoning" and therefore degraded Web Server performance as a cause or even a short time Denial of Service Attack, depending on how well was the initial Server Scaling done.

The bots we want to filter are not to be confused with the legitimate bots, that drives real traffic to your website, just for information

 The 10 Most Popular WebCrawlers Bots as of time of writting are:
 

1. GoogleBot (The Google Crawler bots, funnily bots become less active on Saturday and Sundays :))

2. BingBot (Bing.com Crawler bots)

3. SlurpBot (also famous as Yahoo! Slurp)

4. DuckDuckBot (The dutch search engine duckduckgo.com crawler bots)

5. Baiduspider (The Chineese most famous search engine used as a substitute of Google in China)

6. YandexBot (Russian Yandex Search engine crawler bots used in Russia as a substitute for Google )

7. Sogou Spider (leading Chineese Search Engine launched in 2004)

8. Exabot (A French Search Engine, launched in 2000, crawler for ExaLead Search Engine)

9. FaceBot (Facebook External hit, this crawler is crawling a certain webpage only once the user shares or paste link with video, music, blog whatever  in chat to another user)

10. Alexa Crawler (la_archiver is a web crawler for Amazon's Alexa Internet Rankings, Alexa is a great site to evaluate the approximate page popularity on the internet, Alexa SiteInfo page has historically been the Swift Army knife for anyone wanting to quickly evaluate a webpage approx. ranking while compared to other pages)

Above legitimate bots are known to follow most if not all of W3C – World Wide Web Consorium (W3.Org) standards and therefore, they respect the content commands for allowance or restrictions on a single site as given from robots.txt but unfortunately many of the so called Bad-Bots or Mirroring scripts that are burning your Web Server CPU and Memory mentioned in previous article are either not following /robots.txt prescriptions completely or partially.

Hence with the robots.txt unrespective bots, the case the only way to get rid of most of the webspiders that are just loading your bandwidth and server hardware is to filter / block them is by using Apache's mod_rewrite through

 

.htaccess


file

Create if not existing in the DocumentRoot of your website .htaccess file with whatever text editor, or create it your windows / mac os desktop and transfer via FTP / SecureFTP to server.

I prefer to do it directly on server with vim (text editor)

 

 

vim /var/www/sites/your-domain.com/.htaccess

 

RewriteEngine On

IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

SetEnvIfNoCase User-Agent "^Black Hole” bad_bot
SetEnvIfNoCase User-Agent "^Titan bad_bot
SetEnvIfNoCase User-Agent "^WebStripper" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck" bad_bot
SetEnvIfNoCase User-Agent "^Crescent" bad_bot
SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot" bad_bot
SetEnvIfNoCase User-Agent "^Teleport" bad_bot
SetEnvIfNoCase User-Agent "^TeleportPro" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft" bad_bot
SetEnvIfNoCase User-Agent "^Website Quester" bad_bot
SetEnvIfNoCase User-Agent "^WebZip" bad_bot
SetEnvIfNoCase User-Agent "^moget/2.1" bad_bot
SetEnvIfNoCase User-Agent "^WebZip/4.0" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E" bad_bot
SetEnvIfNoCase User-Agent "^RMA" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^httplib" bad_bot
SetEnvIfNoCase User-Agent "^turingos" bad_bot
SetEnvIfNoCase User-Agent "^spanner" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot" bad_bot
SetEnvIfNoCase User-Agent "^Harvest/1.5" bad_bot
SetEnvIfNoCase User-Agent "Bullseye/1.0" bad_bot
SetEnvIfNoCase User-Agent "^Mozilla/4.0 (compatible; BullsEye; Windows 95)" bad_bot
SetEnvIfNoCase User-Agent "^Crescent Internet ToolPak HTTP OLE Control v.1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerSE/1.0" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker /1.0" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit/3.50" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control – 5.01.4511" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder" bad_bot
SetEnvIfNoCase User-Agent "^Foobot" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot" bad_bot
SetEnvIfNoCase User-Agent "^BotALot" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial/1.34" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers" bad_bot
SetEnvIfNoCase User-Agent "^Microsoft URL Control – 6.00.8169" bad_bot
SetEnvIfNoCase User-Agent "^URLy Warning" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker" bad_bot
SetEnvIfNoCase User-Agent "^cosmos" bad_bot
SetEnvIfNoCase User-Agent "^moget" bad_bot
SetEnvIfNoCase User-Agent "^hloader" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer" bad_bot
SetEnvIfNoCase User-Agent "^Mata Hari" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot" bad_bot
SetEnvIfNoCase User-Agent "^Web Image Collector" bad_bot
SetEnvIfNoCase User-Agent "^The Intraformant" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish/1.0" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc/4.2" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot/1.0" bad_bot
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot" bad_bot
SetEnvIfNoCase User-Agent "^suzuran" bad_bot
SetEnvIfNoCase User-Agent "^VCI WebViewer VCI WebViewer Win32" bad_bot
SetEnvIfNoCase User-Agent "^VCI" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4" bad_bot
SetEnvIfNoCase User-Agent "^QueryN Metasearch" bad_bot
SetEnvIfNoCase User-Agent "^Openfind data gathere" bad_bot
SetEnvIfNoCase User-Agent "^Openfind" bad_bot
SetEnvIfNoCase User-Agent "^Xenu’s Link Sleuth 1.1c" bad_bot
SetEnvIfNoCase User-Agent "^Xenu’s" bad_bot
SetEnvIfNoCase User-Agent "^Zeus" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey Bait & Tackle/v1.01" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 32297 Webster Pro V2.9 Win32" bad_bot
SetEnvIfNoCase User-Agent "^Webster Pro" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a Unix" bad_bot
SetEnvIfNoCase User-Agent "^Keyword Density/0.9" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin Spider" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh" bad_bot

 

<Limit GET POST>
order allow,deny
allow from all
Deny from env=bad_bot
</Limit>

 


Above rules are Bad bots prohibition rules have RewriteEngine On directive included however for many websites this directive is enabled directly into VirtualHost section for domain/s, if that is your case you might also remove RewriteEngine on from .htaccess and still the prohibition rules of bad bots should continue to work
Above rules are also perfectly suitable wordpress based websites / blogs in case you need to filter out obstructive spiders even though the rules would work on any website domain with mod_rewrite enabled.

Once you have implemented above rules, you will not need to restart Apache, as .htaccess will be read dynamically by each client request to Webserver

2. Testing .htaccess Bad Bots Filtering Works as Expected


In order to test the new Bad Bot filtering configuration is working properly, you have a manual and more complicated way with lynx (text browser), assuming you have shell access to a Linux / BSD / *Nix computer, or you have your own *NIX server / desktop computer running
 

Here is how:
 

 

lynx -useragent="Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)" -head -dump http://www.your-website-filtering-bad-bots.com/

 

 

Note that lynx will provide a warning such as:

Warning: User-Agent string does not contain "Lynx" or "L_y_n_x"!

Just ignore it and press enter to continue.

Two other use cases with lynx, that I historically used heavily is to pretent with Lynx, you're GoogleBot in order to see how does Google actually see your website?
 

  • Pretend with Lynx You're GoogleBot

 

lynx -useragent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -head -dump http://www.your-domain.com/

 

 

  • How to Pretend with Lynx Browser You are GoogleBot-Mobile

 

lynx -useragent="Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" -head -dump http://www.your-domain.com/

 


Or for the lazy ones that doesn't have Linux / *Nix at disposal you can use WannaBrowser website

Wannabrowseris a web based browser emulator which gives you the ability to change the User-Agent on each website req1uest, so just set your UserAgent to any bot browser that we just filtered for example set User-Agent to CheeseBot

The .htaccess rule earier added once detecting your browser client is coming in with the prohibit browser agent will immediately filter out and you'll be unable to access the website with a message like:
 

HTTP/1.1 403 Forbidden

 

Just as I've talked a lot about Index Bots, I think it is worthy to also mention three great websites that can give you a lot of Up to Date information on exact Spiders returned user-agent, common known Bot traits as well as a a current updated list with the Bad Bots etc.

Bot and Browser Resources information user-agents, bad-bots and odd Crawlers and Bots specifics

1. botreports.com
2. user-agents.org
3. useragentapi.com

 

An updated list with robots user-agents (crawler-user-agents) is also available in github here regularly updated by Caia Almeido

There are also a third party plugin (modules) available for Website Platforms like WordPress / Joomla / Typo3 etc.

Besides the listed on these websites as well as the known Bad and Good Bots, there are perhaps a hundred of others that might end up crawling your webdsite that might or might not need  to be filtered, therefore before proceeding with any filtering steps, it is generally a good idea to monitor your  HTTPD access.log / error.log, as if you happen to somehow mistakenly filter the wrong bot this might be a reason for Website Indexing Problems.

Hope this article give you some valueable information. Enjoy ! 🙂

 

How to use wget and curl via HTTP Proxy server / How to set a HTTPS proxy server on a bash shell on Linux

Wednesday, January 27th, 2016

linux-ssl-proxy-configuration-from-command-line-with-wget-and-curl-howto

I've been working a bit on a client's automation, the task is to automate process of installations of Apaches / Tomcats / JBoss and Java servers, so me and colleagues don't waste too
much time in trivial things. To complete that I've created a small repository on a Apache with a WebDav server with major versions of each general branch of Application servers and Javas.
In order to access the remote URL where the .tar.gz binaries archives reside, I had to use a proxy serve as the client runs all his network in a DMZ and all Web Port 80 and 443 HTTPS traffic inside the client network
has to pass by the network proxy.

Thus to make the downloads possible via the shell script, writting I needed to set the script to use the HTTPS proxy server. I've been using proxy earlier and I was pretty aware of the http_proxy bash shell
variable thus I tried to use this one for the Secured HTTPS proxy, however the connection was failing and thanks to colleague Anatoliy I realized the whole problem is I'm trying to use http_proxy shell variable
which has to only be used for unencrypted Proxy servers and in this case the proxy server is over SSL encrypted HTTPS protocol so instead the right variable to use is:
 

https_proxy


The https_proxy var syntax, goes like this:

proxy_url='http-proxy-url.net:8080';
export https_proxy="$proxy_url"

how-to-set-https_proxy_url-on-linux-freebsd-openbsd-bsd-and-unix-from-terminal-console

Once the https_proxy variable is set  UNIX's wget non interactive download tool starts using the proxy_url variable set proxy and the downloads in my script works.

Hence to make the different version application archives download work out, I've used wget like so:
 

 wget –no-check-certificate –timeout=5 https://full-path-to-url.net/file.rar


For other BSD / HP-UX / SunOS UNIX Servers where  shells are different from Bourne Again (Bash) Shell, the http_proxy and  https_proxy variable might not be working.
In such cases if you have curl (command line tool) is available instead of wget to script downloads you can use something like:
 

 curl -O -1 -k –proxy http-proxy-url.net:8080 https://full-path-to-url.net/file.rar

The http_proxy and https_proxy variables works perfect also on Mac OS X, default bash shell, so Mac users enjoy.
For some bash users in some kind of firewall hardened environments like in my case, its handy to permanently set a proxy to all shell activities via auto login Linux / *unix scripts .bashrc or .bash_profile that saves the inconvenience to always
set the proxy so lynx and links, elinks text console browsers does work also anytime you login to shell.

Well that's it, my script enjoys proxying traffic 🙂