Archive for the ‘Networking’ Category

Scanning ports with netcat “nc” command on Linux and UNIX / Checking for firewall filtering between source and destination with nc

Friday, September 6th, 2019

scanning-ports-with-netcat-nc-command-on-Linux-and-UNIX-checking-for-firewall-filtering-between-source-destination-host-with-netcat

Netcat ( nc ) is one of that tools, that is well known in the hacker (script kiddie) communities, but little underestimated in the sysadmin world, due to the fact nmap (network mapper) – the network exploratoin and security auditing tool has become like the standard penetration testing TCP / UDP port tool
 

nc is feature-rich network debugging and investigation tool with tons of built-in capabilities for reading from and writing to network connections using TCP or UDP.

Its Plethora of features includes port listening, port scanning & Transferring files due to which it is often used by Hackers and PenTesters as Backdoor. Netcat was written by a guy we know as the Hobbit <hobbit@avian.org>.

For a start-up and middle sized companies if nmap is missing on server usually it is okay to install it without risking to open a huge security hole, however in Corporate world, due to security policies often nmap is not found on the servers but netcat (nc) is present on the servers so you have to learn, if you haven't so to use netcat for the usual IP range port scans, if you're so used to nmap.

There are different implementations of Netcat, whether historically netcat was UNIX (BSD) program with a latest release of March 1996. The Linux version of NC is GNU Netcat (official source here) and is POSIX compatible. The other netcat in Free Software OS-es is OpenBSD's netcat whose ported version is also used in FreeBSD. Mac OS X also comes with default prebundled netcat on its Mac OS X from OS X version (10.13) onwards, on older OS X-es it is installable via MacPorts package repo, even FreeDOS has a port of it called NTOOL.

The (Swiss Army Knife of Embedded Linux) busybox includes a default leightweight version of netcat and Solaris has the OpenBSD netcat version bundled.

A cryptography enabled version fork exists that supports that supports integrated transport encryption capabilities called Cryptcat.

The Nmap suite also has included rewritten version of GNU Netcat named Ncat, featuring new possibilities such as "Connection Brokering", TCP/UDP Redirection, SOCKS4 client and server support, ability to "Chain" Ncat processes, HTTP CONNECT proxying (and proxy chaining), SSL connect/listen support and IP address/connection filtering. Just like Nmap, Ncat is cross-platform.

In this small article I'll very briefly explain on basic netcat – known as the TCP Army knife tool port scanning for an IP range of UDP / TCP ports.

 

1. Scanning for TCP opened / filtered ports remote Linux / Windows server

 

Everyone knows scanning of a port is possible with a simple telnet request towards the host, e.g.:

telnet SMTP.EMAIL-HOST.COM 25

 

The most basic netcat use that does the same is achiavable with:

 

$ nc SMTP.EMAIL-HOST.COM 25
220 jeremiah ESMTP Exim 4.92 Thu, 05 Sep 2019 20:39:41 +0300


Beside scanning the remote port, using netcat interactively as pointing in above example, if connecting to HTTP Web services, you can request remote side to return a webpage by sending a false referer, source host and headers, this is also easy doable with curl / wget and lynx but doing it with netcat just like with telnet could be fun, here is for example how to request an INDEX page with spoofed HTTP headers.
 

nc Web-Host.COM 25
GET / HTTP/1.1
Host: spoofedhost.com
Referrer: mypage.com
User-Agent: my-spoofed-browser

 

2. Performing a standard HTTP request with netcat

 

To do so just pype the content with a standard bash integrated printf function with the included end of line (the unix one is \n but to be OS independent it is better to use r\n  – the end of line complition character for Windows.

 

printf "GET /index.html HTTP/1.0\r\nHost: www.pc-freak.net\r\n\r\n" | nc www.pc-freak.net 80

 

3. Scanning a range of opened / filtered UDP ports

 

To scan for lets say opened remote system services on the very common important ports opened from UDP port 25 till, 1195 – more specifically for:

  • UDP Bind Port 53
  • Time protocol Port (37)
  • TFTP (69)
  • Kerberos (88)
  • NTP 123
  • Netbios (137,138,139)
  • SNMP (161)
  • LDAP 389
  • Microsoft-DS (Samba 445)
  • Route BGP (52)
  • LDAPS (639)
  • openvpn (1194)

 

nc -vzu 192.168.0.1 25 1195

 

UDP tests will show opened, if no some kind of firewall blocking, the -z flag is given to scan only for remote listening daemons without sending any data to them.

 

4. Port Scanning TCP listening ports with Netcat

 

As prior said using netcat to scan for remote opened HTTP Web Server on port 80 an FTP on Port 23 or a Socks Proxy or MySQL Database on 3306 / PostgreSQL DB on TCP 5432 is very rare case scenario.

Below is example to scan a Local network situated IP for TCP open ports from port 1 till 7000.

 

# nc -v -n -z -w 5 192.168.1.2 1-7000

           nc: connect to host.example.com 80 (tcp) failed: Connection refused
           nc: connect to host.example.com 20 (tcp) failed: Connection refused
           Connection to host.example.com port [tcp/ssh] succeeded!
           nc: connect to host.example.com 23 (tcp) failed: Connection refused

 

Be informed that scanning with netcat is much more slower, than nmap, so specifying smaller range of ports is always a good idea to reduce annoying waiting …


The -w flag is used to set a timeout to remote connection, usually on a local network situated machines the timeout could be low -w 1 but for machines across different Data Centers (let say one in Berlin and one in Seattle), use as a minimum -w 5.

If you expect remote service to be responsive (as it should always be), it is a nice idea to use netcat with a low timeout (-w) value of 1 below is example:
 

netcat -v -z -n -w 1 scanned-hosts 1-1023

 

5. Port scanning range of IP addresses with netcat


If you have used Nmap you know scanning for a network range is as simple as running something like nmap -sP -P0 192.168.0.* (to scan from IP range 1-255 map -sP -P0 192.168.0.1-150 (to scan from local IPs ending in 1-150) or giving the network mask of the scanned network, e.g. nmap -sF 192.168.0.1/24 – for more examples please check my previous article Checking port security on Linux with nmap (examples).

But what if nmap is not there and want to check a bunch 10 Splunk servers (software for searching, monitoring, and analyzing machine-generated big data, via a Web-style interface.), with netcat to find, whether the default Splunk connection port 9997 is opened or not:

 

for i in `seq 1 10`; do nc -z -w 5 -vv splunk0$i.server-domain.com 9997; done

 

6. Checking whether UDP port traffic is allowed to destination server

 

Assuring you have access on Source traffic (service) Host A  and Host B (remote destination server where a daemon will be set-upped to listen on UDP port and no firewall in the middle Network router or no traffic control and filtering software HUB is preventing the sent UDP proto traffic, lets say an ntpd will be running on its standard 123 port there is done so:

– On host B (the remote machine which will be running ntpd and should be listening on port 123), run netcat to listen for connections

 

# nc -l -u -p 123
Listening on [0.0.0.0] (family 2, port 123)


Make sure there is no ntpd service actively running on the server, if so stop it with /etc/init.d/ntpd stop
and run above command. The command should run as superuser as UDP port 123 is from the so called low ports from 1-1024 and binding services on such requires root privileges.

– On Host A (UDP traffic send host

 

nc -uv remote-server-host 123

 

netcat-linux-udp-connection-succeeded

If the remote port is not reachable due to some kind of network filtering, you will get "connection refused".
An important note to make is on some newer Linux distributions netcat might be silently trying to connect by default using IPV6, bringing false positives of filtered ports due to that. Thus it is generally a good idea, to make sure you're connecting to IPV6

 

$ nc -uv -4 remote-server-host 123

 

Another note to make here is netcat's UDP connection takes 2-3 seconds, so make sure you wait at least 4-8 seconds for a very distant located hosts that are accessed over a multitude of routers.
 

7. Checking whether TCP port traffic allowed to DST remote server


To listen for TCP connections on a specified location (external Internet IP or hostname), it is analogous to listening for UDP connections.

Here is for example how to bind and listen for TCP connections on all available Interface IPs (localhost, eth0, eth1, eth2 etc.)
 

nc -lv 0.0.0.0 12345

 

Then on client host test the connection with

 

nc -vv 192.168.0.103 12345
Connection to 192.168.0.103 12345 port [tcp/*] succeeded!

 

8. Proxying traffic with netcat


Another famous hackers use of Netcat is its proxying possibility, to proxy anything towards a third party application with UNIX so any content returned be printed out on the listening nc spawned daemon like process.
For example one application is traffic SMTP (Mail traffic) with netcat, below is example of how to proxy traffic from Host B -> Host C (in that case the yandex current mail server mx.yandex.ru)

linux-srv:~# nc -l 12543 | nc mx.yandex.ru 25


Now go to Host A or any host that has TCP/IP protocol access to port 12543 on proxy-host Host B (linux-srv) and connect to it on 12543 with another netcat or telnet.

to make netcat keep connecting to yandex.ru MX (Mail Exchange) server you can run it in a small never ending bash shell while loop, like so:

 

linux-srv:~# while :; do nc -l 12543 | nc mx.yandex.ru 25; done


 Below are screenshots of a connection handshake between Host B (linux-srv) proxy host and Host A (the end client connecting) and Host C (mx.yandex.ru).

host-B-running-as-a-proxy-daemon-towards-Host-C-yandex-mail-exchange-server

 

Host B netcat as a (Proxy)

Host-A-Linux-client-connection-handshake-to-proxy-server-with-netcat
that is possible in combination of UNIX and named pipes (for more on Named pipes check my previous article simple linux logging with named pipes), here is how to run a single netcat version to proxy any traffic in a similar way as the good old tinyproxy.

On Proxy host create the pipe and pass the incoming traffic towards google.com and write back any output received back in the named pipe.
 

# mkfifo backpipe
# nc -l 8080 0<backpipe | nc www.google.com 80 1>backpipe

Other useful netcat proxy set-up is to simulate a network connectivity failures.

For instance, if server:port on TCP 1080 is the normal host application would connect to, you can to set up a forward proxy from port 2080 with

    nc -L server:1080 2080

then set-up and run the application to connect to localhost:2080 (nc proxy port)

    /path/to/application_bin –server=localhost –port=2080

Now application is connected to localhost:2080, which is forwarded to server:1080 through netcat. To simulate a network connectivity failure, just kill the netcat proxy and check the logs of application_bin.

Using netcat as a bind shell (make any local program / process listen and deliver via nc)

 

netcat can be used to make any local program that can receive input and send output to a server, this use is perhaps little known by the junior sysadmin, but a favourite use of l337 h4x0rs who use it to spawn shells on remote servers or to make connect back shell. The option to do so is -e

-e – option spawns the executable with its input and output redirected via network socket.

One of the most famous use of binding a local OS program to listen and receive / send content is by
making netcat as a bind server for local /bin/bash shell.

Here is how

nc -l -p 4321 -e /bin/sh


If necessery specify the bind hostname after -l. Then from any client connect to 4321 (and if it is opened) you will gain a shell with the user with which above netcat command was run. Note that many modern distribution versions such as Debian / Fedora / SuSE Linux's netcat binary is compiled without the -e option (this works only when compiled with -DGAPING_SECURITY_HOLE), removal in this distros is because option is potentially opening a security hole on the system.

If you're interested further on few of the methods how modern hackers bind new backdoor shell or connect back shell, check out Spawning real tty shells article.

 

For more complex things you might want to check also socat (SOcket CAT) – multipurpose relay for bidirectional data transfer under Linux.
socat is a great Linux Linux / UNIX TCP port forwarder tool similar holding the same spirit and functionality of netcat plus many, many more.
 

On some of the many other UNIX operating systems that are lacking netcat or nc / netcat commands can't be invoked a similar utilitiesthat should be checked for and used instead are:

ncat, pnetcat, socat, sock, socket, sbd

To use nmap's ncat to spawn a shell for example that allows up to 3 connections and listens for connects only from 192.168.0.0/24 network on port 8081:

ncat –exec "/bin/bash" –max-conns 3 –allow 192.168.0.0/24 -l 8081 –keep-open

 

9. Copying files over network with netcat


Another good hack often used by hackers to copy files between 2 servers Server1 and Server2 who doesn't have any kind of FTP / SCP / SFTP / SSH / SVN / GIT or any kind of Web copy support service – i.e. servers only used as a Database systems that are behind a paranoid sysadmin firewall is copying files between two servers with netcat.

On Server2 (the Machine on which you want to store the file)
 

nc -lp 2323 > files-archive-to-copy.tar.gz


On server1 (the Machine from where file is copied) run:
 

nc -w 5 server2.example.com 2323 < files-archive-to-copy.tar.gz

 

Note that the downside of such transfers with netcat is data transferred is unencrypted so any one with even a simple network sniffer or packet analyzier such as iptraf or tcpdump could capture the file, so make sure the file doesn't contain sensitive data such as passwords.

Copying partition images like that is perhaps best way to get disk images from a big server onto a NAS (when you can't plug the NAS into the server).
 

10. Copying piped archived directory files with netcat

 

On computer A:

export ARIBTRARY_PORT=3232
nc -l $ARBITRARY_PORT | tar vzxf –

On Computer B:

tar vzcf – files_or_directories | nc computer_a $ARBITRARY_PORT

 

11. Creating a one page webserver with netcat and ncat


As netcat could listen to port and print content of a file, it can be set-up with a bit of bash shell scripting to serve
as a one page webserver, or even combined with some perl scripting and bash to create a multi-serve page webserver if needed.

To make netact serve a page to any connected client run in a screen / tmux session following code:

 

while true; do nc -l -p 80 -q 1 < somepage.html; done

 

Another interesting fun example if you have installed ncat (is a small web server that connects current time on server on connect).
 

ncat -lkp 8080 –sh-exec 'echo -ne "HTTP/1.0 200 OK\r\n\r\nThe date is "; date;'

 

12. Cloning Hard disk partitions with netcat


rsync is a common tool used to clone hard disk partitions over network. However if rsync is not installed on a server and netcat is there you can use it instead, lets say we want to clone /dev/sdb
from Server1 to Server2 assuming (Server1 has a configured working Local or Internet connection).

 

On Server2 run:
 

nc -l -p 4321 | dd of=/dev/sdb

 

Following on Server2 to start the Partition / HDD cloning process run

 

dd if=/dev/sdb | nc 192.168.0.88 4321

 


Where 192.168.0.88 is the IP address listen configured on Server2 (in case you don't know it, check the listening IP to access with /sbin/ifconfig).

Next you have to wait for some short or long time depending on the partiiton or Hard drive, number of files / directories and allocated disk / partition size.

To clone /dev/sda (a main partiiton) from Server1 to Server2 first requirement is that it is not mounted, thus to have it unmounted on a system assuming you have physical access to the host, you can boot some LiveCD Linux distribution such as Knoppix Live CD on Server1, manually set-up networking with ifconfig or grab an IP via DHCP from the central DHCP server and repeat above example.


Happy netcating 🙂

Fix staled NFS on server with dmesg error log nfs: server nfs-server not responding, still trying

Saturday, March 16th, 2019

NFS_Filesystem-fix-staled-NFS-System-dmesg-error-nfs-server-not-responding-still-trying

On a server today I've found to have found a number of NFS mounts mounted through /etc/fstab file definitions that were hanging;
 

nfs-server:~# df -hT


 command kept hanging as well as any attempt to access the mounted NFS directory was not possible.
The server with the hanged Network File System is running SLES (SuSE Enterprise Linux 12 SP3) a short investigation in the kernel logs (dmesg) as well as /var/log/messages reveales following errors:

 

nfs-server:~# dmesg
[3117414.856995] nfs: server nfs-server OK
[3117595.104058] nfs: server nfs-server not responding, still trying
[3117625.032864] nfs: server nfs-server OK
[3117805.280036] nfs: server nfs-server not responding, still trying
[3117835.209110] nfs: server nfs-server OK
[3118015.456045] nfs: server nfs-server not responding, still trying
[3118045.384930] nfs: server nfs-server OK
[3118225.568029] nfs: server nfs-server not responding, still trying
[3118255.560536] nfs: server nfs-server OK
[3118435.808035] nfs: server nfs-server not responding, still trying
[3118465.736463] nfs: server nfs-server OK
[3118645.984057] nfs: server nfs-server not responding, still trying
[3118675.912595] nfs: server nfs-server OK
[3118886.098614] nfs: server nfs-server OK
[3119066.336035] nfs: server nfs-server not responding, still trying
[3119096.274493] nfs: server nfs-server OK
[3119276.512033] nfs: server nfs-server not responding, still trying
[3119306.440455] nfs: server nfs-server OK
[3119486.688029] nfs: server nfs-server not responding, still trying
[3119516.616622] nfs: server nfs-server OK
[3119696.864032] nfs: server nfs-server not responding, still trying
[3119726.792650] nfs: server nfs-server OK
[3119907.040037] nfs: server nfs-server not responding, still trying
[3119936.968691] nfs: server nfs-server OK
[3120117.216053] nfs: server nfs-server not responding, still trying
[3120147.144476] nfs: server nfs-server OK
[3120328.352037] nfs: server nfs-server not responding, still trying
[3120567.496808] nfs: server nfs-server OK
[3121370.592040] nfs: server nfs-server not responding, still trying
[3121400.520779] nfs: server nfs-server OK
[3121400.520866] nfs: server nfs-server OK


It took me a short while to investigate and check the NetApp remote NFS storage filesystem and investigate the Virtual Machine that is running on top of OpenXen Hypervisor system.
The NFS storage permissions of the exported file permissions were checked and they were in a good shape, also a reexport of the NFS mount share was re-exported and on the Linux
mount host the following commands ran to remount the hanged Filesystems:

 

nfs-server:~# umount -f /mnt/nfs_share
nfs-server:~# umount -l /mnt/nfs_share
nfs-server:~# umount -lf /mnt/nfs_share1
nfs-server:~# umount -lf /mnt/nfs_share2
nfs-server:~# mount -t nfs -o remount /mnt/nfs_share


that fixed one of the hanged mount, but as I didn't wanted to manually remount each of the NFS FS-es, I've remounted them all with:

nfs-server:~# mount -a -t nfs


This solved it but, the fix seemed unpermanent as in a time while the issue started reoccuring and I've spend some time
in further investigation on the weird NFS hanging problem has led me to the following blog post where the same problem was described and it was pointed the root cause of it lays
in parameter for MTU which seems to be quite high MTU 9000 and this over the years has prooven to cause problems with NFS especially due to network router (switches) configurations
which seem to have a filters for MTU and are passing only packets with low MTU levels and using rsize / wzise custom mount NFS values in /etc/fstab could lead to this strange NFS hangs.

Below is a list of Maximum Transmission  Unit (MTU) for Media Transport excerpt taken from wikipedia as of time of writting this article.

http://pc-freak.net/images/Maximum-Transmission-Unit-for-Media-Transport-diagram-3.png

In my further research on the issue I've come across this very interesting article which explains a lot on "Large Internet" and Internet Performance

I've used tracepath command which is doing basicly the same as traceroute but could be run without root user and discovers hops (network routers) and shows MTU between path -> destionation.

Below is a sample example

nfs-server:~# tracepath bergon.net
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.6.1                                           0.909ms
 1:  192.168.6.1                                           0.966ms
 2:  192.168.222.1                                         0.859ms
 3:  6.192.104.109.bergon.net                              1.138ms reached
     Resume: pmtu 1500 hops 3 back 3

 

Optiomal pmtu for this connection is to be 1500 .traceroute in some cases might return hops with 'no reply' if there is a router UDP  packet filtering implemented on it.

The high MTU value for the Storage network connection interface on eth1 was evident with a simple:

 

 nfs-server:~# /sbin/ifconfig |grep -i eth -A 2
eth0      Link encap:Ethernet  HWaddr 00:16:3E:5C:65:74
          inet addr:100.127.108.56  Bcast:100.127.109.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth1      Link encap:Ethernet  HWaddr 00:16:3E:5C:65:76
          inet addr:100.96.80.94  Bcast:100.96.83.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1


The fix was as simple to lower MTU value for eth1 Ethernet interface to 1500 which is the value which most network routers are configured too.

To apply the new MTU to the eth1 interface without restarting the SuSE SLES networking , I first used ifconfig one time with:

 

 nfs-server:~# /sbin/ifconfig eth1 mtu 1500
 nfs-server:~# ip addr show
 …


To make the setting permanent on next  SuSE boot:

I had to set the MTU=1500 value in

 

nfs-server:~#/etc/sysconfig/network/ifcfg-eth1
nfs-server:~#  ip address show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 8c:89:a5:f2:e8:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global eth1
       valid_lft forever preferred_lft forever

 


Then to remount the NFS mounted hanged filesystems once again ran:
 

nfs-server:~# mount -a -t nfs


Many network routers keeps the MTU to low as 1500 also because a higher values causes IP packet fragmentation when using NFS over UDP where IP packet fragmentation and packet
reassembly requires significant amount of CPU at both ends of the network connection.
Packet fragmentation also exposes network traffic to greater unreliability, since a complete RPC request must be retransmitted if a UDP packet fragment is dropped for any reason.
Any increase of RPC retransmissions, along with the possibility of increased timeouts, are the single worst impediment to performance for NFS over UDP.
This and many more is very well explained in Optimizing NFS Performance page (which is a must reading) for any sys admin that plans to use NFS frequently.

Even though lowering MTU (Maximum Transmission Union) value does solved my problem at some cases especially in a modern local LANs with Jumbo Frames, allowing and increasing the MTU to 9000 bytes
might be a good idea as this will increase the amount of packet size.and will raise network performance, however as always on distant networks with many router hops keeping MTU value as low as 1492 / 5000 is always a good idea.

 

Automatic network restart and reboot Linux server script if ping timeout to gateway is not responding as a way to reduce connectivity downtimes

Monday, December 10th, 2018

automatic-server-network-restart-and-reboot-script-if-connection-to-server-gateway-inavailable-tux-penguing-ascii-art-bin-bash

Inability of server to come back online server automaticallyafter electricity / network outage

These days my home server  is experiencing a lot of issues due to Electricity Power Outages, a construction dig operations to fix / change waterpipe tubes near my home are in action and perhaps the power cables got ruptered by the digger machine.
The effect of all this was that my server networking accessability was affected and as I didn't have network I couldn't access it remotely anymore at a certain point the electricity was restored (and the UPS charge could keep the server up), however the server accessibility did not due restore until I asked a relative to restart it or under a more complicated cases where Tech aquanted guy has to help – Alexander (Alex) a close friend from school years check his old site here – alex.pc-freak.net helps a lot.to restart the machine physically either run a quick restoration commands on root TTY terminal or generally do check whether default router is reachable.

This kind of Pc-Freak.net downtime issues over the last month become too frequent (the machine was down about 5 times for 2 to 5 hours and this was too much (and weirdly enough it was not accessible from the internet even after electricity network was restored and the only solution to that was a physical server restart (from the Power Button).

To decrease the number of cases in which known relatives or friends has to  physically go to the server and restart it, each time after network or electricity outage I wrote a small script to check accessibility towards Default defined Network Gateway for my server with few ICMP packages sent with good old PING command
and trigger a network restart and system reboot
(in case if the network restart does fail) in a row.

1. Create reboot-if-nwork-is-downsh script under /usr/sbin or other dir

Here is the script itself:

 

#!/bin/sh
# Script checks with ping 5 ICMP pings 10 times to DEF GW and if so
# triggers networking restart /etc/inid.d/networking restart
# Then does another 5 x 10 PINGS and if ping command returns errors,
# Reboots machine
# This script is useful if you run home router with Linux and you have
# electricity outages and machine doesn't go up if not rebooted in that case

GATEWAY_HOST='192.168.0.1';

run_ping () {
for i in $(seq 1 10); do
    ping -c 5 $GATEWAY_HOST
done

}

reboot_f () {
if [ $? -eq 0 ]; then
        echo "$(date "+%Y-%m-%d %H:%M:%S") Ping to $GATEWAY_HOST OK" >> /var/log/reboot.log
    else
    /etc/init.d/networking restart
        echo "$(date "+%Y-%m-%d %H:%M:%S") Restarted Network Interfaces:" >> /tmp/rebooted.txt
    for i in $(seq 1 10); do ping -c 5 $GATEWAY_HOST; done
    if [ $? -eq 0 ] && [ $(cat /tmp/rebooted.txt) -lt ‘5’ ]; then
         echo "$(date "+%Y-%m-%d %H:%M:%S") Ping to $GATEWAY_HOST FAILED !!! REBOOTING." >> /var/log/reboot.log
        /sbin/reboot

    # increment 5 times until stop
    [[ -f /tmp/rebooted.txt ]] || echo 0 > /tmp/rebooted.txt
    n=$(< /tmp/rebooted.txt)
        echo $(( n + 1 )) > /tmp/rebooted.txt
    fi
    # if 5 times rebooted sleep 30 mins and reset counter
    if [ $(cat /tmprebooted.txt) -eq ‘5’ ]; then
    sleep 1800
        cat /dev/null > /tmp/rebooted.txt
    fi
fi

}
run_ping;
reboot_f;

You can download a copy of reboot-if-nwork-is-down.sh script here.

As you see in script successful runs  as well as its failures are logged on server in /var/log/reboot.log with respective timestamp.
Also a counter to 5 is kept in /tmp/rebooted.txt, incremented on each and every script run (rebooting) if, the 5 times increment is matched

a sleep is executed for 30 minutes and the counter is being restarted.
The counter check to 5 guarantees the server will not get restarted if access to Gateway is not continuing for a long time to prevent the system is not being restarted like crazy all time.
 

2. Create a cron job to run reboot-if-nwork-is-down.sh every 15 minutes or so 

I've set the script to re-run in a scheduled (root user) cron job every 15 minutes with following  job:

To add the script to the existing cron rules without rewriting my old cron jobs and without tempering to use cronta -u root -e (e.g. do the cron job add in a non-interactive mode with a single bash script one liner had to run following command:

 

{ crontab -l; echo "*/15 * * * * /usr/sbin/reboot-if-nwork-is-down.sh 2>&1 >/dev/null; } | crontab –


I know restarting a server to restore accessibility is a stupid practice but for home-use or small client servers with unguaranteed networks with a cheap Uninterruptable Power Supply (UPS) devices it is useful.

Summary

Time will show how efficient such a  "self-healing script practice is.
Even though I'm pretty sure that even in a Corporate businesses and large Public / Private Hybrid Clouds where access to remote mounted NFS / XFS / ZFS filesystems are failing a modifications of the script could save you a lot of nerves and troubles and unhappy customers / managers screaming at you on the phone 🙂


I'll be interested to hear from others who have a better  ideas to restore ( resurrect ) access to inessible Linux server after an outage.?
 

How to make Reverse SSH Tunnel to servers behind NAT

Thursday, October 11th, 2018

create-reverse-ssh-tunnel-reverse_ssh_diagram-connection

Those who remember the times of IRC chatting long nights and the need to be c00l guy and enter favorite IRC server through a really bizarre hostname, you should certainly remember the usefulness of Reverse SSH Tunnels to appear in IRC /whois like connecting from a remote host (mask yourself) from other IRC guys where are you physically.

The idea of Reverse SSH is to be able to SSH (or other protocols) connect to IPs that are situated behind a NAT server/s.
Creating SSH Reverse Tunnel is an easy task and up to 2 simple SSH commands
,

To better explain how SSH tunnel is achieved, here is a scenario:

A. Linux host behind NAT IP: 192.168.10.70 (Destination host)
B. (Source Host) of Machine with External Public Internet IP 83.228.93.76 through which SSH Tunnel will be established to 192.168.10.70.

1. Create SSH Revere SSH from Destination to Source host (with Public IP)

Connect to the remote machine which has a real IP address and make port of the reverse SSH connection open (remove any firewall), lets say port 23000.

ssh -R 23000:127.0.0.1:22 username@DOMAIN.com -oPort=33

NB! On destination and source servers make sure you have enabled in /etc/ssh/sshd_config
 

AllowAgentForwarding yes
AllowTCPForwarding yes
PermitTunnel yes

 


2. Connect from Source IP to Destination through the established SSH tunnelling

 

 

Connecting to DOMAIN.com through ssh on 23000 will connect you to the back machine with the unreal IP address.
 

ssh local-username@127.0.0.1 -p 23000


ssh -L 19999:localhost:19999 middleman@178.78.78.78

If you want other server with hostname whatever-host.com to access the Reverse SSH Tunneled server you can do it via external IP which in my case is 83.228.93.76

From whatever-host.com just do:

 ssh username@82.228.93.76

 

reverse_tunnel-linux-diagram-explained
A text diagram of SSH Tunnel looks something like that:

Destination (192.168.10.70) <- |NAT| <- Source (83.228.93.76) <- whatever-host.com

 

Above examples should work not only on Linux but on NetBSD / OpenBSD / FreeBSD or any other UNIX system with a modern SSH client installed.

Change Linux Wireless Access Point connection from text terminal with iwconfig

Monday, October 8th, 2018

wireless-change-wireless-network-to-connect-to-using-console

If you have configured a couple of Wireless connections at home or work on your Laptop  and each of the remote Wi-FI access points are at different distance (some APs are situated at closer range than others) and your Linux OS keeps connecting sometimes to the wrong AP by default you'll perhaps want to change that behavior, so you keep connected to the Wi-Fi AP that has the best Link Quality (is situatated physically at closest location to your laptop integrated wifi card).
Using a Graphical tool such as Gnome Network Manager / Wicd Network Manager or KDE's Network Manager is great and easy way to do it but sometimes if you do upgrade of your GNU / Linux and the upgrade fails and your Graphical Environment GNOME / KDE / OpenBox / Window Maker or whatever Window Manager you use fails to start it is super handy to use text console (terminal) to connect to the right wiki in order to do a deb / rpm package rollback to revert your GUI environment or Xorg to the older working release.

Connection to WPA or WEP protected APs on GNU / Linux on a low level is done by /sbin/iwlist , /sbin/iwconfig and wpa_supplicant

wpasupplicant and network-manager (if you're running Xorg server).

 

/sbin/iwlist scan
 

 

wlp3s0    Scan completed :
          Cell 01 – Address: 10:FE:ED:43:CB:0E
                    Channel:6
                    Frequency:2.437 GHz (Channel 6)
                    Quality=64/70  Signal level=-46 dBm  
                    Encryption key:on
                    ESSID:"Magdanoz"
                    Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 11 Mb/s; 6 Mb/s
                              9 Mb/s; 12 Mb/s; 18 Mb/s
                    Bit Rates:24 Mb/s; 36 Mb/s; 48 Mb/s; 54 Mb/s
                    Mode:Master
                    Extra:tsf=00000032cff7c214
                    Extra: Last beacon: 144ms ago
                    IE: Unknown: 00084D616764616E6F7A
                    IE: Unknown: 010882848B960C121824
                    IE: Unknown: 030106
                    IE: Unknown: 0706555320010B1B
                    IE: Unknown: 2A0100
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : CCMP TKIP
                        Authentication Suites (1) : PSK
                    IE: Unknown: 32043048606C

 

iwlist command is used to get more detailed wireless info from a wireless interface (in terminal this command shows you the wifi networks available to connect to and various info such as the type of Wifi network the Wifi Name / network quality Frequency (is it it spreading the wifi signal at 2.4 Ghz or 5 Ghz frequency) etc.

 

# ifconfig interafce_name down

 

For example on my Thinkpad the wifi interface is wlp3s0 to check what is yours do ifconfig -a e.g.

 

root@jeremiah:~# /sbin/ifconfig -a
enp0s25: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:21:cc:cc:b2:27  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf3900000-f3920000  

 

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 350  bytes 28408 (27.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 350  bytes 28408 (27.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.103  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::6267:20ff:fe3c:20ec  prefixlen 64  scopeid 0x20<link>
        ether 60:67:20:3c:20:ec  txqueuelen 1000  (Ethernet)
        RX packets 299735  bytes 362561115 (345.7 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 278518  bytes 96996135 (92.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

Next use iwconfig on Debian / Ubuntu Linux it is part of wireless-tools deb package.

 

root@jeremiah:~# /sbin/iwconfig interface essid "Your-Acess-Point-name"

 

To check whether you're connected to a wireless network you can do:

http://pc-freak.net/images/check-wireless-frequency-access-point-mac-and-wireless-name-iwconfig-linux

root@jeremiah:~# iwconfig
enp0s25   no wireless extensions.

 

lo        no wireless extensions.

wlp3s0    IEEE 802.11  ESSID:"Magdanoz"  
          Mode:Managed  Frequency:2.437 GHz  Access Point: 10:FE:ED:43:CB:0E   
          Bit Rate=150 Mb/s   Tx-Power=15 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off
          Link Quality=61/70  Signal level=-49 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:5  Invalid misc:1803   Missed beacon:0


N.B. ! To get a list of all your PC network interfaces you can use cmd:

 

root@jeremiah:/home/hipo# ls -al /sys/class/net/
total 0
drwxr-xr-x  2 root root 0 Oct  8 22:53 .
drwxr-xr-x 52 root root 0 Oct  8 22:53 ..
lrwxrwxrwx  1 root root 0 Oct  8 22:53 enp0s25 -> ../../devices/pci0000:00/0000:00:19.0/net/enp0s25
lrwxrwxrwx  1 root root 0 Oct  8 22:53 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx  1 root root 0 Oct  8 22:53 wlp3s0 -> ../../devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlp3s0

show-all-network-interfaces-with-netstat-linux

or use netstat like so:

root@jeremiah:/home/hipo# netstat -i | column -t
Kernel   Interface  table
Iface    MTU        RX-OK   RX-ERR  RX-DRP  RX-OVR  TX-OK   TX-ERR  TX-DRP  TX-OVR  Flg
enp0s25  1500       0       0       0       0       0       0       0       0       BMU
lo       65536      590     0       0       0       590     0       0       0       LRU
wlp3s0   1500       428112  0       1       0       423538  0       0       0       BMRU

 


To get only the Wireless network card interface on Linux (e.g. find out which of the listed above interfaces is your wireless adapter's name), use iw command (that shows devices and their configuration):

 

root@jeremiah:/home/hipo# iw dev
phy#0
    Interface wlp3s0
        ifindex 3
        wdev 0x1
        addr 60:67:20:3c:20:ec
        type managed
        channel 6 (2437 MHz), width: 40 MHz, center1: 2427 MHz
        txpower 15.00 dBm

 

linux-wireless-terminal-console-check-wireless-interfaces-command

  • If you need to get only the active Wireless adapter device assigned by Linux kernel

 

root@jeremiah:~# iw dev | awk '$1=="Interface"{print $2}'

 

To check the IP / Netmask and Broadcase address assigned by connected Access Point use ifconfig
with your Laptop Wireless Interface Name.

show-extra-information-ip-netmask-broadcast-about-wireless-interface-linux

root@jeremiah:~# /sbin/ifconfig wlp3s0
wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.103  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::6267:20ff:fe3c:20ec  prefixlen 64  scopeid 0x20<link>
        ether 60:67:20:3c:20:ec  txqueuelen 1000  (Ethernet)
        RX packets 319534  bytes 365527097 (348.5 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 285464  bytes 99082701 (94.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


As you can see in above 3 examples iwconfig could configure various settings regarding the wireless network interface.

It is really annoying because sometimes if you have configured your Linux to connect to multiple access points, the wifi adapter might keep connecting to an access point that is more distanced from you and because of that the Bandwidth might be slower and that could impact your Internet connectivity, to fix that and get rid of any networks that are automatically set to connect to that you don't want to, just delete the correspodning files (the Wifi file name coincides with the Wireless AP network name).
All stored Wi-FI access points that your Linux is configured to connect to are stored inside /etc/NetworkManager/system-connections/

For example to delete an auto connection to wireless router with a name NetGear do:

 

root@jeremiah:~# rm -f /etc/NetworkManager/system-connections/NetGear

 

For a complete list of stored Wifi Networks that your PC might connect (and authorize to if configured so) do:

 

root@jeremiah:~# ls -a /etc/NetworkManager/system-connections/
Magdanoz
NetGear

LinkSys
Cobra
NetIs
WirelessNet

 

After deleting the required Networks you want your computer to not automatically connect to to make NetworkManager aware of that restart it with:
 

hipo@jeremiah:~# systemctl restart NetworkManager.service


or if you hate systemd like I do just use the good old init script to restart:

 

hipo@jeremiah:~# /etc/init.d/network-manager restart


To get some more informatoin on the exact network you're connected, you can run:

show-information-about-wireless-connection-on-gnu-linux

 

hipo@jeremiah:~# systemctl status NetworkManager.service
● NetworkManager.service – Network Manager
   Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-10-08 22:35:09 EEST; 15s ago
     Docs: man:NetworkManager(8)
 Main PID: 13721 (NetworkManager)
    Tasks: 5 (limit: 4915)
   CGroup: /system.slice/NetworkManager.service
           ├─13721 /usr/sbin/NetworkManager –no-daemon
           └─13742 /sbin/dhclient -d -q -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /var/run/dhclient-wlp3s0.pid -lf /var/lib/NetworkManager/dhclie

 

Oct 08 22:35:15 jeremiah NetworkManager[13721]:   [1539027315.6657] dhcp4 (wlp3s0): state changed unknown -> bound
Oct 08 22:35:15 jeremiah dhclient[13742]: bound to 192.168.0.103 — renewal in 2951 seconds.
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6735] device (wlp3s0): state change: ip-config -> ip-check (reason 'none') [70 80
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6744] device (wlp3s0): state change: ip-check -> secondaries (reason 'none') [80 9
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6747] device (wlp3s0): state change: secondaries -> activated (reason 'none') [90
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6749] manager: NetworkManager state is now CONNECTED_LOCAL
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6812] manager: NetworkManager state is now CONNECTED_GLOBAL
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6813] policy: set 'Magdanoz' (wlp3s0) as default for IPv4 routing and DNS
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6816] device (wlp3s0): Activation: successful, device activated.
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6823] manager: startup complete

 

Why and How to Increase and decrease txqueuelen (Transmit Queue Lenght) in GNU / Linux

Friday, September 22nd, 2017

linuxnetworkingchangetxquelenincreasedecreasefornetworklatencyimprove

In GNU / Linux network routers and sometimes even home PCs it is mostly helpful to play with a TCP / IP stack Network Interface Card (NIC) value called (txqueuelen) in order to make the Network interface (Ethernet device) to work better with the type of network that is connected to it. On a slower LAN or Internet connections txqueuelen is better to be decreased whether on a high speed connections raising it will increase the performance of network traffic to perform large homogenous network transfers.

Therefore on a small private networks in many occasions (though it depends on the served services and exact type of network) it is more useful to decrease the txqueuelen Linux value whether on a High latency networks increasing it will reap a great benefits for your ISP or Hosting routers.

So what is txqueuelen value instructs the kernel on the largeness of transmit queue of the network interface device.
 

1. Decrease the number of txqueuelen in GNU / Linux on a slower networks (ADSL routers), Mobile networks etc to improve network latency


For a slower router devices be it WI-FI routers such as D-Link or any cheap chineese Wifi or LAN or Telephone or ISP compnay networks that provide internet via ADSL routers that run Linux or has access to Linux console with available access to ifconfig command or have an iproute2 Linux package installed (most of them have) it is very helpful to reduce its size for a smaller values in order to guarantee a network high latency.

Therefore reducing the txquelen value to a number like 200 for ADSL provided internet can benefit you.
Assuming that you know the NIC name in Linux the first one is usually eth0 in order to reduce this value to 200 issue:

 

ifconfig eth0 txqueuelen length
ifconfig eth0 txqueuelen 200

 

To do the same with iproute2 (ip) command in case if the router is missing ifconfig or you just prefer to use iproute2 (advanced and newer bunch of commands that improved the Linux networking functionalities thanks to Russian Alexei Kuznecov 15 years ago or so) run:

 

ip link set eth0 txqueuelen length
ip link set eth0 txqueuelen 200

 

To make it permanent you can either create a brand new rc. script lets say /etc/init.d/rc.queuelenght and add the commands or use the good old /etc/rc.local to make the commands get loaded on every GNU / linux or router boot time.

Above reducement will make your Network TCP stack more responsive.

2. Increase the number of txquelen in GNU / Linux to improve the network latency on a company NAT or other ISP routers

Depending on the scale of Internet bandwidth that is coming inside your router you should decide how much you would like to increase the txqueuelen value.
Generally speaking txqueuelen value could be played with on a Large IN / OUT Company Routers with values between 1000 and 20000

Most GNU / Linux distributions comes preconfigured as a default a value of 1000, so lets say you would like to raise the value to 5000 to improve network transfer latency, here is how to do it via ifconfig:

 

ifconfig eth0 txqueuelen length
ifconfig eth0 txqueuelen 5000

 

With IPRoute2 analogously run:

 

ip link set eth0 txqueuelen length
ip link set eth0 txqueuelen 5000

 

Of course the best way to figure out what will be the best value for your case is to experiment a little bit with the value and use some kind of speedtest (test bandwitdh service – this one is mine) from the many available online.
You have to consider the type of hardware of Server and most importantly the Network Card hardware (which vendor and what are the recommended ones from the vendor).

If the router is some Linux distribution lets say a Debian / Ubuntu Linux etc other deb based one, it is better to make the necessery permanent changes to make them be loaded on system boot not via /etc/rc.local but by adding to file /etc/network/interfaces:

 

vim /etc/network/interfaces

 

up ifconfig $IFACE txqueuelen 8000

 

Note that as network traffic processing depends on CPU power, decreasing the txqueuelen will add a little bit of extra load to the Central Processing Unit.


In RHEL / CentOS Linux and SuSE server a good way to increase txqueuelen network latency is by running:

 

# cat <> /etc/rules.d/71-net-txqueuelen.rules
SUBSYSTEM==”net”, ACTION==”add”, KERNEL==”eth*”, ATTR{tx_queue_len}=”10000″
EOF

 

 

udevadm trigger

 

I've red on the internet that a lot of people are reporting a very good results with a txqueuelen setting of 2000 as this value is said to give good results and not hammering the CPU too hardly, so it might be a good idea if you're experimenting with values to find out which one suits you better is to start with this one, e.g. run:
 

ifconfig eth0 txqueuelen 2000
echo 2000 > /proc/sys/net/core/netdev_max_backlog


Well of course keep in mind that sometimes increasing the txqueuelen to too large values will make you an easier target for Denial of Service Attack attempts, though most modern computer routers should be able to behave good even with the highest value like 10000.
 

 

 

Mount remote Linux SSHFS Filesystem harddisk on Windows Explorer SWISH SSHFS file mounter and a short evaluation on what is available to copy files to SSHFS from Windows PC

Monday, February 22nd, 2016

swish-mount-and-copy-files-from-windows-to-linux-via-sshfs-mount

I'm forced to use Windows on my workbook and I found it really irritating, that I can't easily share files in a DropBox, Google Drive, MS OneDrive, Amazon Storage or other cloud-storage free remote service. etc.
I don't want to use DropBox like non self-hosted Data storage because I want to keep my data private and therefore the only and best option for me was to make possible share my Linux harddisk storage
dir remotely to the Windows notebook.

I didn't wanted to setup some complex environment such as Samba Share Server (which used to be often a common option to share file from Linux server to Windows), neither wanted to bother with  installing FTP service and bother with FTP clients, or configuring some other complex stuff such as WebDav – which BTW is an accepted and heavily used solution across corporate clients to access read / write files on a remote Linux servers.
Hence, I made a quick research what else besides could be used to easily share files and data from Windows PC / notebook to a home brew or professional hosting Linux server.

It turned out, there are few of softwares that gives a similar possibility for a home lan small network Linux / Windows hybrid network users such, here is few of the many:

  • SyncThingSyncthing is an open-source file synchronization client/server application, written in Go, implementing its own, equally free Block Exchange Protocol. The source code's content-management repository is hosted on GitHub

     

     

     

     

     

    syncthing-logo

  • OwnCloud – ownCloud provides universal access to your files via the web, your computer or your mobile devices

     

     

     

     

     

    owncloud-logo

  • Seafile – Seafile is a file hosting software system. Files are stored on a central server and can be synchronized with personal computers and mobile devices via the Seafile client. Files can also be accessed via the server's web interface


seafile-client-in-browser

I've checked all of them and give a quick try of Syncthing which is really easy to start, just download the binary launch it and configure it under https://Localhost:8385 URL from a browser on the Linux server.
Syncthing seemed to be nice and easy to configure solution to be able to Sync files between Server A (Windows) and Server B(Linux) and guess many would enjoy it, if you want to give it a try you can follow this short install syncthing article.
However what I didsliked in both SyncThing and OwnCloud and Seafile and all of the other Sync file solutions was, they only supported synchronization via web and didn't seemed to have a Windows Explorer integration and did required
the server to run more services, posing another security hole in the system as such third party softwares are not easily to update and maintain.

Because of that finally after rethinking about some other ways to copy files to a locally mounted Sync directory from the Linux server, I've decided to give SSHFS a try. Mounting SSHFS between two Linux / UNIX hosts is
quite easy task with SSHFS tool

In Windows however the only way I know to transfer files to Linux via SSHFS was with WinSCP client and other SCP clients as well as the experimental:

As well as few others such as ExpandDrive, Netdrive, Dokan SSHFS (mirrored for download here)
I should say that I first decided to try copying few dozen of Gigabyte movies, text, books etc. using WinSCP direct connection, but after getting a couple of timeouts I was tired of WinSCP and decided to look for better way to copy to remote Linux SSHFS.
However the best solution I found after a bit of extensive turned to be:

SWISH – Easy SFTP for Windows

Swish is very straight forward to configure compared to all of them you download the .exe which as of time of writting is at version 0.8.0 install on the PC and right in My Computer you will get a New Device called Swish next to your local and remote drives C:/ D:/ , USBs etc.

swish-new-device-to-appear-in-my-computer-to-mount-sshfs

As you see in below screenshot two new non-standard buttons will Appear in Windows Explorer that lets you configure SWISH

windows-mount-sshfs-swish-add-sftp-connection-button-screenshot

Next and final step before you have the SSHFS remote Linux filesystem visible on Windows Xp / 7 / 8 / 10 is to fill in remote Linux hostname address (or even better fill in IP to get rid of possible DNS issues), UserName (UserID) and Direcory to mount.

swish-new-fill-in-dialog-to-make-new-linux-sshfs-mount-directory-possible-on-windows

Then you will see the SSHFS moutned:

swish-sshfs-mounted-on-windows

You will be asked to accept the SSH host-key as it used to be unknown so far

swish-mount-sshfs-partition-on-windows-from-remote-linux-accept-key

That's it now you will see straight into Windows Explorer the remote Linux SSHFS mounted:

remote-sshfs-linux-filesystem-mounted-in-windows-explorer-with-swish

Once setupped a Swish connection to copy files directly to it you can use the Send to Embedded Windows dialog, as in below screenshot

swish-send-to-files-windows-screenshot

The only 3 problem with SWISH are:

1. It doesn't support Save password, so on every Windows PC reboot when you want to connect to remote Linux SSHFS, you will have to retype remote login user pass.
Fron security stand point this is not such a bad thing, but it is a bit irritating to everytime type the password without an option to save permanently.
The good thing here is you can use Launch Key Agent
as visible in above screenshot and set in Putty Key Agent your remote host SSH key so the passwordless login will work without any authentication at
all however, this might open a security hole if your Win PC gets infected by virus, which might delete something on remote mounted SSHFS filesystem so I personally prefer to retype password on every boot.

2. it is a bit slow so if you're planning to Transfer large amounts of Data as hundreds of megabytes, expect a very slow transfer rate, even in a Local  10Mbit Network to transfer 20 – 30 GB of data, it took me about 2-3 hours or so.
SWISH is not actively supported and it doesn't have new release since 20th of June 2013, but for the general work I need it is just perfect, as I don't tent to be Synchronizing Dozens of Gigabytes all the time between my notebook PC and the Linux server.

3. If you don't use the established mounted connection for a while or your computer goes to sleep mode after recovering your connection to remote Linux HDD if opened in Windows File Explorer will probably be dead and you will have to re-enable it.

For Mac OS X users who want to mount / attach remote directory from a Linux partitions should look in fuguA Mac OS X SFTP, SCP and SSH Frontend

I'll be glad to hear from people on other good ways to achieve same results as with SWISH but have a better copy speed while using SSHFS.

Apache Webserver disable hostnamelookups “HostnameLookups off” for minor performance increase

Friday, February 12th, 2016

apache-disable-dns-lookups-for-speed-hostnamelookups-off-directive-building-scalable-php-applications

If you don't much care about logging in logs from which domain / hostnames requests to webserver originate and you want to boost up the Apache Webserver performance a bit especially on a heavy loaded Websites, where no need for stuff like Webalizer, Awstats etc. , e.g. you're using GoogleAnalytics to already track requests (beware as sometimes GoogleAnalytics could be missing requests to your webserver, so having some kind of LogAnalyzer software on server is always a plus). But anyways accepting that many of us already trust GoogleAnalytitcs.


Then a great tuning option to use in default domain configuration or in multiple VirtualHosts config is:

HostnameLookups off

If you want to make the HostnameLookups off as a default behaviour to all your virtualhosts on  Debian / Ubuntu / CentOS / SuSE / RHEL distro virtualhosts add either to default config /etc/apache2/sites-enabled/000-default (on Deb based Linuxes) or (on RPM based ones), add directive to /etc/httpd/httpd.conf

For self-hosted websites (if run your own small hosting) or for a home situated webservers with up to 20-50 websites it is also a useful optimization tip to include in /etc/hosts file all the IPs of sites with respective domain names following the normal syntax of /etc/hosts, e.g. in my own /etc/hosts, I have stuff like:
 

pcfreak:~$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain
127.0.1.1 pcfreak.pc-freak.net pcfreak mail.pc-freak.net
192.168.0.14 new-pcfreak
219.22.88.70 fw
212.36.0.70 ftp.bg.debian.org
212.211.132.32 security.debian.org
83.228.93.76 pcfreak.biz pc-freak.net www.pc-freak.net
# for wordpress plugins
216.58.209.3 gstatic.com
91.225.248.129 www.linkedin.com
74.50.119.198 www.blogtopsites.com
94.31.29.40 static.addtoany.com
216.58.209.202 fonts.googleapis.com
216.58.209.14 www.google-analytics.com
216.58.209.14 feeds.feedburner.com
93.184.220.241 wprp.zemanta.com
199.30.80.32 stumbleupon.com
156.154.168.17 stumbleupon.com
2.18.89.251 platform.linkedin.com
# The following lines are desirable for IPv6 capable hosts

# … etc. put IPs and hostnames following above syntax


As you see from above commented section for wordpress plugins, I've included some common websites used by WordPress enabled plugins to prevent my own hosting server to query DNS server every time. The normal way the Linux / Unix works is it first checks in /etc/hosts and only if the hostname is not defined there then it queries the DNS caching server in my case this is a local DJBDNS cache server, however defining the hosts in /etc/hosts saves a lot of milisecons on every request and often if multiple hosts are defined could save (decrease site opening for end users) with seconds.


Well now use some website speed testing plugin like Yslow, Firebug Fiddler or HTTPWatch

 

How to mount NFS network filesystem to remote server via /etc/fstab on Linux

Friday, January 29th, 2016

mount-nfs-in-linux-via--etc-fstab-howto-mount-remote-partitions-from-application-server-to-storage-server
If you have a server topology part of a project where 3 (A, B, C) servers need to be used to deliver a service (one with application server such as Jboss / Tomcat / Apache, second just as a Storage Server holding a dozens of LVM-ed SSD hard drives and an Oracle database backend to provide data about the project) and you need to access server A (application server) to server B (the Storage "monster") one common solution is to use NFS (Network FileSystem) Mount. 
NFS mount is considered already a bit of obsoleted technology as it is generally considered unsecre, however if SSHFS mount is not required due to initial design decision or because both servers A and B are staying in a serious firewalled (DMZ) dedicated networ then NTS should be a good choice.
Of course to use NFS mount should always be a carefully selected Environment Architect decision so remote NFS mount, imply  that both servers are connected via a high-speed gigabyte network, e.g. network performance is calculated to be enough for application A <-> to network storage B two sides communication not to cause delays for systems end Users.

To test whether the NFS server B mount is possible on the application server A, type something like:

 

mount -t nfs -o soft,timeo=900,retrans=3,vers=3, proto=tcp remotenfsserver-host:/home/nfs-mount-data /mnt/nfs-mount-point


If the mount is fine to make the mount permanent on application server host A (in case of server reboot), add to /etc/fstab end of file, following:

1.2.3.4:/application/local-application-dir-to-mount /application/remote-application-dir-to-mount nfs   rw,bg,nolock,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr 1 2


If the NTFS server has a hostname you can also type hostname instead of above example sample IP 1.2.3.4, this is however not recommended as this might cause in case of DNS or Domain problems.
If you want to mount with hostname (in case if storage server IP is being commonly changed due to auto-selection from a DHCP server):

server-hostA:/application/local-application-dir-to-mount /application/remote-application-dir-to-mount nfs   rw,bg,nolock,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr 1 2

In above example you need to have the /application/local-application-dir-to-mount (dir where remote NFS folder will be mounted on server A) as well as the /application/remote-application-dir-to-mount
Also on server Storage B server, you have to have running NFS server with firewall accessibility from server A working.

The timeou=600 (is defined in) order to make the timeout for remote NFS accessibility 1 hour in order to escape mount failures if there is some minutes network failure between server A and server B, the rsize and wsize
should be fine tuned according to the files that are being red from remote NFS server and the network speed between the two in the example are due to environment architecture (e.g. to reflect the type of files that are being transferred by the 2)
and the remote NFS server running version and the Linux kernel versions, these settings are for Linux kernel branch 2.6.18.x which as of time of writting this article is obsolete, so if you want to use the settings check for your kernel version and
NTFS and google and experiment.

Anyways, if you're not sure about wsize and and rise, its perfectly safe to omit these 2 values if you're not familiar to it.

To finally check the NFS mount is fine,  grep it:

 

# mount|grep -i nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
server-hostA:/application/remote-application-dir-to-mount on /application/remote-application-dir-to-mount type nfs (rw,bg,nolock,nfsvers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr,addr=1.2.3.4)


That's all enjoy 🙂