Archive for the ‘Networking’ Category

How to make Reverse SSH Tunnel to servers behind NAT

Thursday, October 11th, 2018

create-reverse-ssh-tunnel-reverse_ssh_diagram-connection

Those who remember the times of IRC chatting long nights and the need to be c00l guy and enter favorite IRC server through a really bizarre hostname, you should certainly remember the usefulness of Reverse SSH Tunnels to appear in IRC /whois like connecting from a remote host (mask yourself) from other IRC guys where are you physically.

The idea of Reverse SSH is to be able to SSH (or other protocols) connect to IPs that are situated behind a NAT server/s.
Creating SSH Reverse Tunnel is an easy task and up to 2 simple SSH commands
,

To better explain how SSH tunnel is achieved, here is a scenario:

A. Linux host behind NAT IP: 192.168.10.70 (Destination host)
B. (Source Host) of Machine with External Public Internet IP 83.228.93.76 through which SSH Tunnel will be established to 192.168.10.70.

1. Create SSH Revere SSH from Destination to Source host (with Public IP)

Connect to the remote machine which has a real IP address and make port of the reverse SSH connection open (remove any firewall), lets say port 23000.

ssh -R 23000:127.0.0.1:22 username@DOMAIN.com -oPort=33

NB! On destination and source servers make sure you have enabled in /etc/ssh/sshd_config
 

AllowAgentForwarding yes
AllowTCPForwarding yes
PermitTunnel yes

 


2. Connect from Source IP to Destination through the established SSH tunnelling

 

 

Connecting to DOMAIN.com through ssh on 23000 will connect you to the back machine with the unreal IP address.
 

ssh local-username@127.0.0.1 -p 23000


ssh -L 19999:localhost:19999 middleman@178.78.78.78

If you want other server with hostname whatever-host.com to access the Reverse SSH Tunneled server you can do it via external IP which in my case is 83.228.93.76

From whatever-host.com just do:

 ssh username@82.228.93.76

 

reverse_tunnel-linux-diagram-explained
A text diagram of SSH Tunnel looks something like that:

Destination (192.168.10.70) <- |NAT| <- Source (83.228.93.76) <- whatever-host.com

 

Above examples should work not only on Linux but on NetBSD / OpenBSD / FreeBSD or any other UNIX system with a modern SSH client installed.

Change Linux Wireless Access Point connection from text terminal with iwconfig

Monday, October 8th, 2018

wireless-change-wireless-network-to-connect-to-using-console

If you have configured a couple of Wireless connections at home or work on your Laptop  and each of the remote Wi-FI access points are at different distance (some APs are situated at closer range than others) and your Linux OS keeps connecting sometimes to the wrong AP by default you'll perhaps want to change that behavior, so you keep connected to the Wi-Fi AP that has the best Link Quality (is situatated physically at closest location to your laptop integrated wifi card).
Using a Graphical tool such as Gnome Network Manager / Wicd Network Manager or KDE's Network Manager is great and easy way to do it but sometimes if you do upgrade of your GNU / Linux and the upgrade fails and your Graphical Environment GNOME / KDE / OpenBox / Window Maker or whatever Window Manager you use fails to start it is super handy to use text console (terminal) to connect to the right wiki in order to do a deb / rpm package rollback to revert your GUI environment or Xorg to the older working release.

Connection to WPA or WEP protected APs on GNU / Linux on a low level is done by /sbin/iwlist , /sbin/iwconfig and wpa_supplicant

wpasupplicant and network-manager (if you're running Xorg server).

 

/sbin/iwlist scan
 

 

wlp3s0    Scan completed :
          Cell 01 – Address: 10:FE:ED:43:CB:0E
                    Channel:6
                    Frequency:2.437 GHz (Channel 6)
                    Quality=64/70  Signal level=-46 dBm  
                    Encryption key:on
                    ESSID:"Magdanoz"
                    Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 11 Mb/s; 6 Mb/s
                              9 Mb/s; 12 Mb/s; 18 Mb/s
                    Bit Rates:24 Mb/s; 36 Mb/s; 48 Mb/s; 54 Mb/s
                    Mode:Master
                    Extra:tsf=00000032cff7c214
                    Extra: Last beacon: 144ms ago
                    IE: Unknown: 00084D616764616E6F7A
                    IE: Unknown: 010882848B960C121824
                    IE: Unknown: 030106
                    IE: Unknown: 0706555320010B1B
                    IE: Unknown: 2A0100
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : CCMP TKIP
                        Authentication Suites (1) : PSK
                    IE: Unknown: 32043048606C

 

iwlist command is used to get more detailed wireless info from a wireless interface (in terminal this command shows you the wifi networks available to connect to and various info such as the type of Wifi network the Wifi Name / network quality Frequency (is it it spreading the wifi signal at 2.4 Ghz or 5 Ghz frequency) etc.

 

# ifconfig interafce_name down

 

For example on my Thinkpad the wifi interface is wlp3s0 to check what is yours do ifconfig -a e.g.

 

root@jeremiah:~# /sbin/ifconfig -a
enp0s25: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:21:cc:cc:b2:27  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf3900000-f3920000  

 

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 350  bytes 28408 (27.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 350  bytes 28408 (27.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.103  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::6267:20ff:fe3c:20ec  prefixlen 64  scopeid 0x20<link>
        ether 60:67:20:3c:20:ec  txqueuelen 1000  (Ethernet)
        RX packets 299735  bytes 362561115 (345.7 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 278518  bytes 96996135 (92.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

Next use iwconfig on Debian / Ubuntu Linux it is part of wireless-tools deb package.

 

root@jeremiah:~# /sbin/iwconfig interface essid "Your-Acess-Point-name"

 

To check whether you're connected to a wireless network you can do:

http://pc-freak.net/images/check-wireless-frequency-access-point-mac-and-wireless-name-iwconfig-linux

root@jeremiah:~# iwconfig
enp0s25   no wireless extensions.

 

lo        no wireless extensions.

wlp3s0    IEEE 802.11  ESSID:"Magdanoz"  
          Mode:Managed  Frequency:2.437 GHz  Access Point: 10:FE:ED:43:CB:0E   
          Bit Rate=150 Mb/s   Tx-Power=15 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off
          Link Quality=61/70  Signal level=-49 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:5  Invalid misc:1803   Missed beacon:0


N.B. ! To get a list of all your PC network interfaces you can use cmd:

 

root@jeremiah:/home/hipo# ls -al /sys/class/net/
total 0
drwxr-xr-x  2 root root 0 Oct  8 22:53 .
drwxr-xr-x 52 root root 0 Oct  8 22:53 ..
lrwxrwxrwx  1 root root 0 Oct  8 22:53 enp0s25 -> ../../devices/pci0000:00/0000:00:19.0/net/enp0s25
lrwxrwxrwx  1 root root 0 Oct  8 22:53 lo -> ../../devices/virtual/net/lo
lrwxrwxrwx  1 root root 0 Oct  8 22:53 wlp3s0 -> ../../devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlp3s0

show-all-network-interfaces-with-netstat-linux

or use netstat like so:

root@jeremiah:/home/hipo# netstat -i | column -t
Kernel   Interface  table
Iface    MTU        RX-OK   RX-ERR  RX-DRP  RX-OVR  TX-OK   TX-ERR  TX-DRP  TX-OVR  Flg
enp0s25  1500       0       0       0       0       0       0       0       0       BMU
lo       65536      590     0       0       0       590     0       0       0       LRU
wlp3s0   1500       428112  0       1       0       423538  0       0       0       BMRU

 


To get only the Wireless network card interface on Linux (e.g. find out which of the listed above interfaces is your wireless adapter's name), use iw command (that shows devices and their configuration):

 

root@jeremiah:/home/hipo# iw dev
phy#0
    Interface wlp3s0
        ifindex 3
        wdev 0x1
        addr 60:67:20:3c:20:ec
        type managed
        channel 6 (2437 MHz), width: 40 MHz, center1: 2427 MHz
        txpower 15.00 dBm

 

linux-wireless-terminal-console-check-wireless-interfaces-command

  • If you need to get only the active Wireless adapter device assigned by Linux kernel

 

root@jeremiah:~# iw dev | awk '$1=="Interface"{print $2}'

 

To check the IP / Netmask and Broadcase address assigned by connected Access Point use ifconfig
with your Laptop Wireless Interface Name.

show-extra-information-ip-netmask-broadcast-about-wireless-interface-linux

root@jeremiah:~# /sbin/ifconfig wlp3s0
wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.103  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::6267:20ff:fe3c:20ec  prefixlen 64  scopeid 0x20<link>
        ether 60:67:20:3c:20:ec  txqueuelen 1000  (Ethernet)
        RX packets 319534  bytes 365527097 (348.5 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 285464  bytes 99082701 (94.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


As you can see in above 3 examples iwconfig could configure various settings regarding the wireless network interface.

It is really annoying because sometimes if you have configured your Linux to connect to multiple access points, the wifi adapter might keep connecting to an access point that is more distanced from you and because of that the Bandwidth might be slower and that could impact your Internet connectivity, to fix that and get rid of any networks that are automatically set to connect to that you don't want to, just delete the correspodning files (the Wifi file name coincides with the Wireless AP network name).
All stored Wi-FI access points that your Linux is configured to connect to are stored inside /etc/NetworkManager/system-connections/

For example to delete an auto connection to wireless router with a name NetGear do:

 

root@jeremiah:~# rm -f /etc/NetworkManager/system-connections/NetGear

 

For a complete list of stored Wifi Networks that your PC might connect (and authorize to if configured so) do:

 

root@jeremiah:~# ls -a /etc/NetworkManager/system-connections/
Magdanoz
NetGear

LinkSys
Cobra
NetIs
WirelessNet

 

After deleting the required Networks you want your computer to not automatically connect to to make NetworkManager aware of that restart it with:
 

hipo@jeremiah:~# systemctl restart NetworkManager.service


or if you hate systemd like I do just use the good old init script to restart:

 

hipo@jeremiah:~# /etc/init.d/network-manager restart


To get some more informatoin on the exact network you're connected, you can run:

show-information-about-wireless-connection-on-gnu-linux

 

hipo@jeremiah:~# systemctl status NetworkManager.service
● NetworkManager.service – Network Manager
   Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-10-08 22:35:09 EEST; 15s ago
     Docs: man:NetworkManager(8)
 Main PID: 13721 (NetworkManager)
    Tasks: 5 (limit: 4915)
   CGroup: /system.slice/NetworkManager.service
           ├─13721 /usr/sbin/NetworkManager –no-daemon
           └─13742 /sbin/dhclient -d -q -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /var/run/dhclient-wlp3s0.pid -lf /var/lib/NetworkManager/dhclie

 

Oct 08 22:35:15 jeremiah NetworkManager[13721]:   [1539027315.6657] dhcp4 (wlp3s0): state changed unknown -> bound
Oct 08 22:35:15 jeremiah dhclient[13742]: bound to 192.168.0.103 — renewal in 2951 seconds.
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6735] device (wlp3s0): state change: ip-config -> ip-check (reason 'none') [70 80
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6744] device (wlp3s0): state change: ip-check -> secondaries (reason 'none') [80 9
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6747] device (wlp3s0): state change: secondaries -> activated (reason 'none') [90
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6749] manager: NetworkManager state is now CONNECTED_LOCAL
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6812] manager: NetworkManager state is now CONNECTED_GLOBAL
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6813] policy: set 'Magdanoz' (wlp3s0) as default for IPv4 routing and DNS
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6816] device (wlp3s0): Activation: successful, device activated.
Oct 08 22:35:15 jeremiah NetworkManager[13721]:
  [1539027315.6823] manager: startup complete

 

Why and How to Increase and decrease txqueuelen (Transmit Queue Lenght) in GNU / Linux

Friday, September 22nd, 2017

linuxnetworkingchangetxquelenincreasedecreasefornetworklatencyimprove

In GNU / Linux network routers and sometimes even home PCs it is mostly helpful to play with a TCP / IP stack Network Interface Card (NIC) value called (txqueuelen) in order to make the Network interface (Ethernet device) to work better with the type of network that is connected to it. On a slower LAN or Internet connections txqueuelen is better to be decreased whether on a high speed connections raising it will increase the performance of network traffic to perform large homogenous network transfers.

Therefore on a small private networks in many occasions (though it depends on the served services and exact type of network) it is more useful to decrease the txqueuelen Linux value whether on a High latency networks increasing it will reap a great benefits for your ISP or Hosting routers.

So what is txqueuelen value instructs the kernel on the largeness of transmit queue of the network interface device.
 

1. Decrease the number of txqueuelen in GNU / Linux on a slower networks (ADSL routers), Mobile networks etc to improve network latency


For a slower router devices be it WI-FI routers such as D-Link or any cheap chineese Wifi or LAN or Telephone or ISP compnay networks that provide internet via ADSL routers that run Linux or has access to Linux console with available access to ifconfig command or have an iproute2 Linux package installed (most of them have) it is very helpful to reduce its size for a smaller values in order to guarantee a network high latency.

Therefore reducing the txquelen value to a number like 200 for ADSL provided internet can benefit you.
Assuming that you know the NIC name in Linux the first one is usually eth0 in order to reduce this value to 200 issue:

 

ifconfig eth0 txqueuelen length
ifconfig eth0 txqueuelen 200

 

To do the same with iproute2 (ip) command in case if the router is missing ifconfig or you just prefer to use iproute2 (advanced and newer bunch of commands that improved the Linux networking functionalities thanks to Russian Alexei Kuznecov 15 years ago or so) run:

 

ip link set eth0 txqueuelen length
ip link set eth0 txqueuelen 200

 

To make it permanent you can either create a brand new rc. script lets say /etc/init.d/rc.queuelenght and add the commands or use the good old /etc/rc.local to make the commands get loaded on every GNU / linux or router boot time.

Above reducement will make your Network TCP stack more responsive.

2. Increase the number of txquelen in GNU / Linux to improve the network latency on a company NAT or other ISP routers

Depending on the scale of Internet bandwidth that is coming inside your router you should decide how much you would like to increase the txqueuelen value.
Generally speaking txqueuelen value could be played with on a Large IN / OUT Company Routers with values between 1000 and 20000

Most GNU / Linux distributions comes preconfigured as a default a value of 1000, so lets say you would like to raise the value to 5000 to improve network transfer latency, here is how to do it via ifconfig:

 

ifconfig eth0 txqueuelen length
ifconfig eth0 txqueuelen 5000

 

With IPRoute2 analogously run:

 

ip link set eth0 txqueuelen length
ip link set eth0 txqueuelen 5000

 

Of course the best way to figure out what will be the best value for your case is to experiment a little bit with the value and use some kind of speedtest (test bandwitdh service – this one is mine) from the many available online.
You have to consider the type of hardware of Server and most importantly the Network Card hardware (which vendor and what are the recommended ones from the vendor).

If the router is some Linux distribution lets say a Debian / Ubuntu Linux etc other deb based one, it is better to make the necessery permanent changes to make them be loaded on system boot not via /etc/rc.local but by adding to file /etc/network/interfaces:

 

vim /etc/network/interfaces

 

up ifconfig $IFACE txqueuelen 8000

 

Note that as network traffic processing depends on CPU power, decreasing the txqueuelen will add a little bit of extra load to the Central Processing Unit.


In RHEL / CentOS Linux and SuSE server a good way to increase txqueuelen network latency is by running:

 

# cat <> /etc/rules.d/71-net-txqueuelen.rules
SUBSYSTEM==”net”, ACTION==”add”, KERNEL==”eth*”, ATTR{tx_queue_len}=”10000″
EOF

 

 

udevadm trigger

 

I've red on the internet that a lot of people are reporting a very good results with a txqueuelen setting of 2000 as this value is said to give good results and not hammering the CPU too hardly, so it might be a good idea if you're experimenting with values to find out which one suits you better is to start with this one, e.g. run:
 

ifconfig eth0 txqueuelen 2000
echo 2000 > /proc/sys/net/core/netdev_max_backlog


Well of course keep in mind that sometimes increasing the txqueuelen to too large values will make you an easier target for Denial of Service Attack attempts, though most modern computer routers should be able to behave good even with the highest value like 10000.
 

 

 

Mount remote Linux SSHFS Filesystem harddisk on Windows Explorer SWISH SSHFS file mounter and a short evaluation on what is available to copy files to SSHFS from Windows PC

Monday, February 22nd, 2016

swish-mount-and-copy-files-from-windows-to-linux-via-sshfs-mount

I'm forced to use Windows on my workbook and I found it really irritating, that I can't easily share files in a DropBox, Google Drive, MS OneDrive, Amazon Storage or other cloud-storage free remote service. etc.
I don't want to use DropBox like non self-hosted Data storage because I want to keep my data private and therefore the only and best option for me was to make possible share my Linux harddisk storage
dir remotely to the Windows notebook.

I didn't wanted to setup some complex environment such as Samba Share Server (which used to be often a common option to share file from Linux server to Windows), neither wanted to bother with  installing FTP service and bother with FTP clients, or configuring some other complex stuff such as WebDav – which BTW is an accepted and heavily used solution across corporate clients to access read / write files on a remote Linux servers.
Hence, I made a quick research what else besides could be used to easily share files and data from Windows PC / notebook to a home brew or professional hosting Linux server.

It turned out, there are few of softwares that gives a similar possibility for a home lan small network Linux / Windows hybrid network users such, here is few of the many:

  • SyncThingSyncthing is an open-source file synchronization client/server application, written in Go, implementing its own, equally free Block Exchange Protocol. The source code's content-management repository is hosted on GitHub

     

     

     

     

     

    syncthing-logo

  • OwnCloud – ownCloud provides universal access to your files via the web, your computer or your mobile devices

     

     

     

     

     

    owncloud-logo

  • Seafile – Seafile is a file hosting software system. Files are stored on a central server and can be synchronized with personal computers and mobile devices via the Seafile client. Files can also be accessed via the server's web interface


seafile-client-in-browser

I've checked all of them and give a quick try of Syncthing which is really easy to start, just download the binary launch it and configure it under https://Localhost:8385 URL from a browser on the Linux server.
Syncthing seemed to be nice and easy to configure solution to be able to Sync files between Server A (Windows) and Server B(Linux) and guess many would enjoy it, if you want to give it a try you can follow this short install syncthing article.
However what I didsliked in both SyncThing and OwnCloud and Seafile and all of the other Sync file solutions was, they only supported synchronization via web and didn't seemed to have a Windows Explorer integration and did required
the server to run more services, posing another security hole in the system as such third party softwares are not easily to update and maintain.

Because of that finally after rethinking about some other ways to copy files to a locally mounted Sync directory from the Linux server, I've decided to give SSHFS a try. Mounting SSHFS between two Linux / UNIX hosts is
quite easy task with SSHFS tool

In Windows however the only way I know to transfer files to Linux via SSHFS was with WinSCP client and other SCP clients as well as the experimental:

As well as few others such as ExpandDrive, Netdrive, Dokan SSHFS (mirrored for download here)
I should say that I first decided to try copying few dozen of Gigabyte movies, text, books etc. using WinSCP direct connection, but after getting a couple of timeouts I was tired of WinSCP and decided to look for better way to copy to remote Linux SSHFS.
However the best solution I found after a bit of extensive turned to be:

SWISH – Easy SFTP for Windows

Swish is very straight forward to configure compared to all of them you download the .exe which as of time of writting is at version 0.8.0 install on the PC and right in My Computer you will get a New Device called Swish next to your local and remote drives C:/ D:/ , USBs etc.

swish-new-device-to-appear-in-my-computer-to-mount-sshfs

As you see in below screenshot two new non-standard buttons will Appear in Windows Explorer that lets you configure SWISH

windows-mount-sshfs-swish-add-sftp-connection-button-screenshot

Next and final step before you have the SSHFS remote Linux filesystem visible on Windows Xp / 7 / 8 / 10 is to fill in remote Linux hostname address (or even better fill in IP to get rid of possible DNS issues), UserName (UserID) and Direcory to mount.

swish-new-fill-in-dialog-to-make-new-linux-sshfs-mount-directory-possible-on-windows

Then you will see the SSHFS moutned:

swish-sshfs-mounted-on-windows

You will be asked to accept the SSH host-key as it used to be unknown so far

swish-mount-sshfs-partition-on-windows-from-remote-linux-accept-key

That's it now you will see straight into Windows Explorer the remote Linux SSHFS mounted:

remote-sshfs-linux-filesystem-mounted-in-windows-explorer-with-swish

Once setupped a Swish connection to copy files directly to it you can use the Send to Embedded Windows dialog, as in below screenshot

swish-send-to-files-windows-screenshot

The only 3 problem with SWISH are:

1. It doesn't support Save password, so on every Windows PC reboot when you want to connect to remote Linux SSHFS, you will have to retype remote login user pass.
Fron security stand point this is not such a bad thing, but it is a bit irritating to everytime type the password without an option to save permanently.
The good thing here is you can use Launch Key Agent
as visible in above screenshot and set in Putty Key Agent your remote host SSH key so the passwordless login will work without any authentication at
all however, this might open a security hole if your Win PC gets infected by virus, which might delete something on remote mounted SSHFS filesystem so I personally prefer to retype password on every boot.

2. it is a bit slow so if you're planning to Transfer large amounts of Data as hundreds of megabytes, expect a very slow transfer rate, even in a Local  10Mbit Network to transfer 20 – 30 GB of data, it took me about 2-3 hours or so.
SWISH is not actively supported and it doesn't have new release since 20th of June 2013, but for the general work I need it is just perfect, as I don't tent to be Synchronizing Dozens of Gigabytes all the time between my notebook PC and the Linux server.

3. If you don't use the established mounted connection for a while or your computer goes to sleep mode after recovering your connection to remote Linux HDD if opened in Windows File Explorer will probably be dead and you will have to re-enable it.

For Mac OS X users who want to mount / attach remote directory from a Linux partitions should look in fuguA Mac OS X SFTP, SCP and SSH Frontend

I'll be glad to hear from people on other good ways to achieve same results as with SWISH but have a better copy speed while using SSHFS.

Apache Webserver disable hostnamelookups “HostnameLookups off” for minor performance increase

Friday, February 12th, 2016

apache-disable-dns-lookups-for-speed-hostnamelookups-off-directive-building-scalable-php-applications

If you don't much care about logging in logs from which domain / hostnames requests to webserver originate and you want to boost up the Apache Webserver performance a bit especially on a heavy loaded Websites, where no need for stuff like Webalizer, Awstats etc. , e.g. you're using GoogleAnalytics to already track requests (beware as sometimes GoogleAnalytics could be missing requests to your webserver, so having some kind of LogAnalyzer software on server is always a plus). But anyways accepting that many of us already trust GoogleAnalytitcs.


Then a great tuning option to use in default domain configuration or in multiple VirtualHosts config is:

HostnameLookups off

If you want to make the HostnameLookups off as a default behaviour to all your virtualhosts on  Debian / Ubuntu / CentOS / SuSE / RHEL distro virtualhosts add either to default config /etc/apache2/sites-enabled/000-default (on Deb based Linuxes) or (on RPM based ones), add directive to /etc/httpd/httpd.conf

For self-hosted websites (if run your own small hosting) or for a home situated webservers with up to 20-50 websites it is also a useful optimization tip to include in /etc/hosts file all the IPs of sites with respective domain names following the normal syntax of /etc/hosts, e.g. in my own /etc/hosts, I have stuff like:
 

pcfreak:~$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain
127.0.1.1 pcfreak.pc-freak.net pcfreak mail.pc-freak.net
192.168.0.14 new-pcfreak
219.22.88.70 fw
212.36.0.70 ftp.bg.debian.org
212.211.132.32 security.debian.org
83.228.93.76 pcfreak.biz pc-freak.net www.pc-freak.net
# for wordpress plugins
216.58.209.3 gstatic.com
91.225.248.129 www.linkedin.com
74.50.119.198 www.blogtopsites.com
94.31.29.40 static.addtoany.com
216.58.209.202 fonts.googleapis.com
216.58.209.14 www.google-analytics.com
216.58.209.14 feeds.feedburner.com
93.184.220.241 wprp.zemanta.com
199.30.80.32 stumbleupon.com
156.154.168.17 stumbleupon.com
2.18.89.251 platform.linkedin.com
# The following lines are desirable for IPv6 capable hosts

# … etc. put IPs and hostnames following above syntax


As you see from above commented section for wordpress plugins, I've included some common websites used by WordPress enabled plugins to prevent my own hosting server to query DNS server every time. The normal way the Linux / Unix works is it first checks in /etc/hosts and only if the hostname is not defined there then it queries the DNS caching server in my case this is a local DJBDNS cache server, however defining the hosts in /etc/hosts saves a lot of milisecons on every request and often if multiple hosts are defined could save (decrease site opening for end users) with seconds.


Well now use some website speed testing plugin like Yslow, Firebug Fiddler or HTTPWatch

 

How to mount NFS network filesystem to remote server via /etc/fstab on Linux

Friday, January 29th, 2016

mount-nfs-in-linux-via--etc-fstab-howto-mount-remote-partitions-from-application-server-to-storage-server
If you have a server topology part of a project where 3 (A, B, C) servers need to be used to deliver a service (one with application server such as Jboss / Tomcat / Apache, second just as a Storage Server holding a dozens of LVM-ed SSD hard drives and an Oracle database backend to provide data about the project) and you need to access server A (application server) to server B (the Storage "monster") one common solution is to use NFS (Network FileSystem) Mount. 
NFS mount is considered already a bit of obsoleted technology as it is generally considered unsecre, however if SSHFS mount is not required due to initial design decision or because both servers A and B are staying in a serious firewalled (DMZ) dedicated networ then NTS should be a good choice.
Of course to use NFS mount should always be a carefully selected Environment Architect decision so remote NFS mount, imply  that both servers are connected via a high-speed gigabyte network, e.g. network performance is calculated to be enough for application A <-> to network storage B two sides communication not to cause delays for systems end Users.

To test whether the NFS server B mount is possible on the application server A, type something like:

 

mount -t nfs -o soft,timeo=900,retrans=3,vers=3, proto=tcp remotenfsserver-host:/home/nfs-mount-data /mnt/nfs-mount-point


If the mount is fine to make the mount permanent on application server host A (in case of server reboot), add to /etc/fstab end of file, following:

1.2.3.4:/application/local-application-dir-to-mount /application/remote-application-dir-to-mount nfs   rw,bg,nolock,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr 1 2


If the NTFS server has a hostname you can also type hostname instead of above example sample IP 1.2.3.4, this is however not recommended as this might cause in case of DNS or Domain problems.
If you want to mount with hostname (in case if storage server IP is being commonly changed due to auto-selection from a DHCP server):

server-hostA:/application/local-application-dir-to-mount /application/remote-application-dir-to-mount nfs   rw,bg,nolock,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr 1 2

In above example you need to have the /application/local-application-dir-to-mount (dir where remote NFS folder will be mounted on server A) as well as the /application/remote-application-dir-to-mount
Also on server Storage B server, you have to have running NFS server with firewall accessibility from server A working.

The timeou=600 (is defined in) order to make the timeout for remote NFS accessibility 1 hour in order to escape mount failures if there is some minutes network failure between server A and server B, the rsize and wsize
should be fine tuned according to the files that are being red from remote NFS server and the network speed between the two in the example are due to environment architecture (e.g. to reflect the type of files that are being transferred by the 2)
and the remote NFS server running version and the Linux kernel versions, these settings are for Linux kernel branch 2.6.18.x which as of time of writting this article is obsolete, so if you want to use the settings check for your kernel version and
NTFS and google and experiment.

Anyways, if you're not sure about wsize and and rise, its perfectly safe to omit these 2 values if you're not familiar to it.

To finally check the NFS mount is fine,  grep it:

 

# mount|grep -i nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
server-hostA:/application/remote-application-dir-to-mount on /application/remote-application-dir-to-mount type nfs (rw,bg,nolock,nfsvers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr,addr=1.2.3.4)


That's all enjoy 🙂

 

 

Secure your work PC internet traffic using SSH Dynamic Tunnel as Proxy to get around Corporate Spy Proxy and Site Filtering

Friday, March 20th, 2015

use-ssh-dynamic-tunnel-as-socks5-proxy-to-get-around-corporate-website-filtering-restrictions

If you work for some huge corporations such as IBM / Sony / Toshiba / Concentrix / HP etc. and you're using a Windows Work Computer (notebook), pre-installed with a custom Company software which is by default configured to use a Proxy Server for all your Browsing activities and at a certain point you start being filtered some of the websites you love to visit so much because of some Corporate policies (limitations) at some filtered sites you will start getting empty pages or some   nasty filtering messages.

Even if you don't get a filtering message but you know all your Company Internal Network traffic is proxified for the sake of keeping your personal (privacy) high stop browsing using company's default proxy, because all your access requests (passwords) and queries to the internet are probably logged for later (review) in case if you enter the company's paragraph of "non-compliant employee".
If you fail on time to get around the default set "Corporate Proxy", sooner or later you will start getting filtering messages to some of the regular websites you use daily, as I did today while trying to open my personal blog (to check if there are new user comments):

Your request was denied because of its content categorization: "Hacking;Malicious Sources/Malnets;Religion"
For assistance, contact your network support team.

Screenshot of above message from today here

You see this guys or automated Proxy filter became so prudent that my site was filtered because it contains some Proof of Concept (PoC) security tools and content related to Christian (Faith) Religion. I guess its the time to think seriously is there a censorship in large corporations and how far could censorship go and if such censorship so easily adopted in large companies wouldn't same happen also on a backbone ISP level in short future??
If today my site is being filtered out to be unable to open from a corporation network because it contains "Religious" contain I would not be surprised if tomorrow, I've been prohibited to confess publicly my faith in salvation power of the Cross of our Lord Jesus Christ or even already in a blacklist because I'm trying to be a dedicated Orthodox Christian …
The fact that Religion is already perceived in same light as Hacking and Malicious Source or Malnet bots is also very eloquent and shows how very big part of people nowdays (including the person that added my site to this proxy filtering rules) think of religion and in what bad state our society and understanding of freedom and respect for others went.

Obviously it is time to react to this censorship and stop the evil corporation from spying on your traffic and logging all that matches there "kilometer long" prohibited sites filter lists. There are few ways to do that and the most straight forward is to set-up and use a Own Proxy server such as Privoxy / Polipo or Squid Proxy, however the proxy method requires that your company local network doesn't have too strick (restrictive) firewall rules (e.g. you need some port opened to the Internet such as 8080, 3128, 8118, 1080 standard port for (socks) etc.

As many companies are too restrictive in their outbound firewall rules and you might be in situation like with me where Browsers such as Internet Explorer / Opera / Firefox and Chrome are configured to use by default company proxy host (autocache.proxy-ur-company.hp.com:80) (with a custom Proxy PAC file filtering out a whole ranges of useful domains and IPs) and only allowed firewall access outside of local corporate network in on port 22 (for outside ssh session purposes) only.

Then your best way to get across such restrictive network configuration is to run your own home Linux / BSD / Windows server with opensshd installed and use OpenSSH protocol Dynamic Tunneling (Proxy socks5 like) capabilities to tunnel all your favourite Web Browser Traffic (lets say Firefox's) through your remote-home-host.com:22.

 


In short once you have installed plink.exe on your PC run manually from command line (cmd.exe)

 

plink.exe -ssh UserName@remote-home-host.com -P 22 -pw Secret_Password -D 127.0.0.1:8080 -N


For people who use MobaXTerm it is even easier as there is an integrated SSH tunneling input interface which can be used to create the SSH tunnel.

To have a quick way to Enable SSH Dynamic Tunnel button on your Desktop make a SymLink to Plink with Target below command line:

web-tunnel-maker-with-plink-win-ssh-connection-tool-screenshot-on-ms-windows-7

  • If from Linux / *BSD / Mac OS host to create Dynamic SSH Tunnel to your remote home SSH server host run in a Terminal
     

ssh -D 8080 Username@remote-home-host.com


To start tunneling all your Web traffic via just created Dynamic SSH Tunnel to host remote-home-host.com, just set in browser's proxy options to use as proxy socks5 – localhost:8080

Secure-your-work-PC-notebook-internet-traffic-using-SSH-Dynamic-Tunnel-as-Proxy

To test whether your traffic is going to the Internet from remote-home-host.com open in just set proxy browser www.myip.ru .
You should see your home SSH server IP as IP which made the request to www.myip.ru.

Linux: How to see / change supported network bandwidth of NIC interface and get various eth network statistics with ethtool

Monday, January 19th, 2015

linux-how-to-see-change-supported-network-bandwidth-of-NIC-interface-and-view-network-statistics
If you're a novice Linux sysadmin and inherited some dedicated servers without any documentation and hence on of the first things you have to do to start a new server documentation is to check the supported TCP/IP network speed of servers Network (ethernet) Interfaces. On Linux this is very easy task to verify the speed of LAN card supported Local / Internet traffic install ethtool (if not already preseont on the servers) – assuming you're dealing with Debian / Ubuntu Linux servers.

1. Install ethtool on Deb and RPM based distros

dedi-server1:~# apt-cache show ethtool|grep -i desc -A 3
Description: display or change Ethernet device settings
 ethtool can be used to query and change settings such as speed, auto-
 negotiation and checksum offload on many network devices, especially
 Ethernet devices.

dedi-server1:~# apt-get install –yes ethtool
..

ethtool should be installed by default on CentOS / Fedora / RHEL and  syntax is same like on Debs. If you happen to miss ethtool on any (SuSE) / RedHat / RPM based distro install it with yum

[root@centos:~] # yum -y install ethtool


2. Get ethernet configurations

To check the current eth0 / eth1 / ethX network (Speed / Duplex) and other network related configuration configuration:
 

dedi-server5:~# ethtool eth0

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full

        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off
        Supports Wake-on: pumbag
        Wake-on: g
        Current message level: 0x00000001 (1)
        Link detected: yes

Having a NIC configured to act as Duplex is very important as Duplex communication enables LAN card to communicate both sides (Sent / Receive) packets simultaneously.

full-duplex-half-duplex-explained-picture

Probably most interesting parameters for most admins are the ones that are telling whether the NIC UpLink is 10megabyte / 100 megabyte or 1Gigabyte as well as supported Receive / Send ( Transfer ) speeds of LAN, a common useful ethtool admin use to just show current LAN ethernet interface speed:

server-admin1:~# ethtool eth0 |grep -i speed
        Speed: 1000Mb/s

 

To get info about NIC (kernel module / driver) used with ethtool:

dedi-server3:~# ethtool -i eth0 driver: e1000e
version: 1.2.20-k2
firmware-version: 1.8-0
bus-info: 0000:06:00.0

3. Make LAN Card blink to recognize eth is mapped to which Physical LAN

Besides that ethtool has many other useful use cases, for example if you have a server with 5 lan or more LAN cards, but you're not sure to which of all different EthX interfaces correspond, a very useful thing is to make eth0, eth1, eth2, eth3, etc. blink for 5 seconds in order to identify which static IP is binded physically to which NIC , here is how:

ethtool -p eth0 5


Then you can follow the procedure for any interface on the server and map them with a sticker 🙂

Ethtool is also useful for getting "deep" (thorough) statistics on Server LAN cards, this could be useful to identify sometimes hard to determine broadcast flood attacks:
 

4. Get network statistics with ethtool for interfaces
 

dedi-server5:~# ethtool -S eth0|grep -vw 0
NIC statistics:
     rx_packets: 6196644448
     tx_packets: 7197385158
     rx_bytes: 2038559235701
     tx_bytes: 8281206569250
     rx_broadcast: 357508947
     tx_broadcast: 172
     rx_multicast: 34731963
     tx_multicast: 20
     rx_errors: 115
     multicast: 34731963
     rx_length_errors: 115
     rx_no_buffer_count: 26391
     rx_missed_errors: 10059
     tx_timeout_count: 3
     tx_restart_queue: 2590
     rx_short_length_errors: 115
     tx_tcp_seg_good: 964136993
     rx_long_byte_count: 2038559235701
     rx_csum_offload_good: 5824813965
     rx_csum_offload_errors: 42186
     rx_smbus: 383640020

5. Turn on Auto Negotiation and change NIC set speed to 10 / 100 / 1000 Mb/s

Auto-negotiation is important as an ethernet procedure by which two communication devices (2 network cards) choose common transmission parameters such as speed, duplex mode, and flow control in order to achieve maximum transmission speed over the network. On 1000BASE-T basednetworks the standard is a mandatory. There is also backward compatability for older 10BASE-T Networks.

a) To raise up NIC to use 1000 Mb/s in case if the bandwidth was raised to 1Gb/s but NIC settings were not changed:

dedi-server1:~# ethtool -s eth0 speed 1000 duplex half autoneg off


b) In case if LAN speed has to be reduced for some weird reason to 10 / 100Mb/s

 

dedi-server1:~# ethtool -s eth0 speed 10 duplex half autoneg off

dedi-server1:~# ethtool -s eth0 speed 100 duplex half autoneg off

c) To enable disable NIC Autonegotiation:

dedi-server1:~# ethtool -s eth0 autoneg on


6. Change Speed / Duplex settings to load on boot

a) Set Network to Duplex on Fedora / CentOS etc.

Quickest way to do it is of course to use /etc/rc.local. If you want to do it following distribution logic on CentOS / RHEL Linux:

Add to /etc/sysconfig/network-scripts/ifcfg-eth0

vim /etc/sysconfig/network/-scripts/ifcfg-eth0

ETHTOOL_OPTS="speed 1000 duplex full autoneg off"

To load the new settings restart networking (be careful to have physical access to server if something goes wrong 🙂 )

service network restart

b) Change network speed / duplex setting on Debian / Ubuntu Linux

Add at the end of /etc/network/interfaces

vim /etc/network/interfaces

post-up ethtool -s eth0 speed 100 duplex full autoneg off

7. Tune NIC ring buffers

dedi-server1:~# ethtool -g eth0

Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             256

As you can see the default setting of RX (receive) buffer size is low 256 and on busy servers with high traffic loads, depending on the hardware NIC vendor this RX buffer size varies.
Through increasing the Rx/Tx ring buffer size , you can decrease the probability of discarding packets in the NIC during a scheduling delay.
A change in rx buffer ring requires NIC restart so (be careful not to loose connection to remote server), be sure to have iLO access to it.

Here is how to raise Rx ring buffer size 4 times from default value:

ethtool -G eth0 rx 4096 tx 4069