bandwidth Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘bandwidth’

Check server Internet connectivity Speedtest from Linux terminal CLI

Friday, August 7th, 2020

check-server-console-speedtest

If you are a system administrator of a dedicated server and you have no access to Xserver Graphical GNOME / KDE etc. environment and you wonder how you can track the bandwidth connectivity speed of remote system to the internet and you happen to have a modern Linux distribution, here is few ways to do a speedtest.

1. Use speedtest-cli command line tool to test connectivity

speedtest-cli is a tiny tool written in python, to use it hence you need to have python installed on the server.
It is available both for Redhat Linux distros and Debians / Ubuntus etc. in the list of standard installable packages.

a) Install speedtest-cli on Fedora / CentOS / RHEL

On CentOS / RHEL / Scientific Linux lower than ver 8:

$ sudo yum install python

On CentOS 8 / RHEL 8 user type the following command to install Python 3 or 2:

$sudo yum install python3
$ sudo yum install python2

On Fedora Linux version 22+

$ sudo dnf install python
$ sudo dnf install pytho3

Once python is at place download speedtest.py or in case if link is not reachable download mirrored version of speedtest.py on www.pc-freak.net here

$ wget -O speedtest-cli https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py
$ chmod +x speedtest-cli

Then it is time to run script speedtest-screenshot-linux-terminal-console-cli-cmd
To test enabled Bandwidth on the server

$ python speedtest-cli

b) Install speedtest-cli on Debian

On Latest Debian 10 Buster speedtest is available out of the box in regular .deb repositories, so fetch it with apt

# apt install –yes speedtest-cli
…

You can give now speedtest-cli a try with –bytes arguments to get speed values in bytes instead of bits or if you want to generate an image with test results in picture just like it will appear if you use speedtest.net inside a gui browser, use the –share option

speedtest-screenshot-linux-terminal-console-cli-cmd-options

2. Getting connectivity results of all defined speedtest test City Locations

Speedtest has a list of servers through which a Upload and Download speed is tested, to run speedtest-cli to test with each and every server and get a better picture on what kind of connectivity to expect from your server towards the closest region capital cities, fetch speedtest-servers.php list and use a small shell loop below is how:

root@pcfreak:~# wget http://www.speedtest.net/speedtest-servers.php
–2020-08-07 16:31:34– http://www.speedtest.net/speedtest-servers.php
Преобразувам www.speedtest.net (www.speedtest.net)… 151.101.2.219, 151.101.66.219, 151.101.130.219, …
Connecting to www.speedtest.net (www.speedtest.net)|151.101.2.219|:80… успешно свързване.
HTTP изпратено искане, чакам отговор… 301 Moved Permanently
Адрес: https://www.speedtest.net/speedtest-servers.php [следва]
–2020-08-07 16:31:34– https://www.speedtest.net/speedtest-servers.php
Connecting to www.speedtest.net (www.speedtest.net)|151.101.2.219|:443… успешно свързване.
HTTP изпратено искане, чакам отговор… 307 Temporary Redirect
Адрес: https://c.speedtest.net/speedtest-servers-static.php [следва]
–2020-08-07 16:31:35– https://c.speedtest.net/speedtest-servers-static.php
Преобразувам c.speedtest.net (c.speedtest.net)… 151.101.242.219
Connecting to c.speedtest.net (c.speedtest.net)|151.101.242.219|:443… успешно свързване.
HTTP изпратено искане, чакам отговор… 200 OK
Дължина: 211695 (207K) [text/xml]
Saving to: ‘speedtest-servers.php’ speedtest-servers.php 100%[==========================================================================>] 206,73K –.-KB/s in 0,1s
2020-08-07 16:31:35 (1,75 MB/s) – ‘speedtest-servers.php’ saved [211695/211695]

Once file is there with below loop we extract all file defined servers id="" 's

root@pcfreak:~# for i in $(cat speedtest-servers.php | egrep -Eo 'id="[0-9]{4}"' |sed -e 's#id="##' -e 's#"##g'); do speedtest-cli –server $i; done
Retrieving speedtest.net configuration…
Testing from Vivacom (83.228.93.76)…
Retrieving speedtest.net server list…
Retrieving information for the selected server…
Hosted by Telecoms Ltd. (Varna) [38.88 km]: 25.947 ms
Testing download speed……………………………………………………………………..
Download: 57.71 Mbit/s
Testing upload speed…………………………………………………………………………………………
Upload: 93.85 Mbit/s
Retrieving speedtest.net configuration…
Testing from Vivacom (83.228.93.76)…
Retrieving speedtest.net server list…
Retrieving information for the selected server…
Hosted by GMB Computers (Constanta) [94.03 km]: 80.247 ms
Testing download speed……………………………………………………………………..
Download: 35.86 Mbit/s
Testing upload speed…………………………………………………………………………………………
Upload: 80.15 Mbit/s
Retrieving speedtest.net configuration…
Testing from Vivacom (83.228.93.76)…
…..
…

etc.

For better readability you might want to add the ouput to a file or even put it to run periodically on a cron if you have some suspcion that your server Internet dedicated lines dies out to some general locations sometimes.

3. Testing UPlink speed with Download some big file from source location

In the past a classical way to test the bandwidth connectivity of your Internet Service Provider was to fetch some big file, Linux guys should remember it was almost a standard to roll a download of Linux kernel source .tar file with some test browser as elinks / lynx / w3c.

or if those are not at hand test connectivity on remote free shell servers whatever file downloader as wget or curl was used.
Analogical method is still possible, for example to use wget to get an idea about bandwidtch connectivity, let it roll below 500 mb from speedtest.wdc01.softlayer.com to /dev/null few times:

$ wget –output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip

$ wget –output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip

$ wget –output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip

# wget -O /dev/null –progress=dot:mega http://cachefly.cachefly.net/10mb.test ; date
–2020-08-07 13:56:49– http://cachefly.cachefly.net/10mb.test
Resolving cachefly.cachefly.net (cachefly.cachefly.net)… 205.234.175.175
Connecting to cachefly.cachefly.net (cachefly.cachefly.net)|205.234.175.175|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 10485760 (10M) [application/octet-stream]
Saving to: ‘/dev/null’
0K …….. …….. …….. …….. …….. …….. 30% 142M 0s
3072K …….. …….. …….. …….. …….. …….. 60% 179M 0s
6144K …….. …….. …….. …….. …….. …….. 90% 204M 0s
9216K …….. …….. 100% 197M=0.06s
2020-08-07 13:56:50 (173 MB/s) – ‘/dev/null’ saved [10485760/10485760]

Fri 07 Aug 2020 01:56:50 PM UTC

To be sure you have a real picture on remote machine Internet speed it is always a good idea to run download of random big files on a certain locations that are well known to have a very stable Internet bandwidth to the Internet backbone routers.

4. Using Simple shell script to test Internet speed

Fetch and use speedtest.sh

wget https://raw.github.com/blackdotsh/curl-speedtest/master/speedtest.sh && chmod u+x speedtest.sh && bash speedtest.sh

5. Using iperf to test connectivity between two servers

iperf is another good tool worthy to mention that can be used to test the speed between client and server.

To use iperf install it with apt and do on the server machine to which bandwidth will be tested:

# iperf -s

On the client machine do:

# iperf -c 192.168.1.1

where 192.168.1.1 is the IP of the server where iperf was spawned to listen.

6. Using Netflix fast to determine Internet connection speed on host

Fast

fast is a service provided by Netflix. Its web interface is located at Fast.com and it has a command-line interface available through npm (npm is a package manager for nodejs) so if you don't have it you will have to install it first with:

# apt install –yes npm

Note that if you run on Debian this will install you some 249 new nodejs packages which you might not want to have on the system, so this is useful only for machines that has already use of nodejs.

$ fast

82 Mbps ↓

The command returns your Internet download speed. To get your upload speed, use the -u flag:

$ fast -u

⠧ 80 Mbps ↓ / 8.2 Mbps ↑

7. Use speedometer / iftop to measure incoming and outgoing traffic on interface

If you're measuring connectivity on a live production server system, then you might consider that the measurement output might not be exactly correct especially if you're measuring the Uplink / Downlink on a Heavy loaded webserver / Mail Server / Samba or DNS server.
If this is the case a very useful tools to consider to extract the already taken traffic used on your Incoming and Outgoing ( TX / RX ) Network interfaces are speedometer and iftop, they're present and installable depending on the OS via yum / apt or the respective package manager.

To install on Debian server:

# apt install –yes iftop speedometer

The most basic use to check the live received traffic in a nice Ncurses like text graphic is with:

# speedometer -r

speedometer-check-received-transmitted-network-traffic-on-linux1

To generate real time ASCII art graph on RX / TX traffic do:

# speedometer -r eth0 -t eth0

speedometer-check-received-transmitted-network-traffic-on-linux

# iftop -P -i eth0

Tags: about, access, bandwidth, curl, download, elinks, howto, iftop, linux?, Lynx, python, script, server, speedometer, speedtest-cli, test connection, traffic, upload, w3m, wget
Posted in Educational, Linux, Monitoring, Networking, System Administration | No Comments »

TinyProxy Quick and Easy way to run a Proxy Caching server to protect and speed up web traffic on Linux / BSD and Mac

Tuesday, March 24th, 2015

tinyproxy-quick-and-easy-way-to-run-proxy-caching-server-on-linux-bsd-unix-and-mac
If you happen to have SSH account on a Dedicated server or a VPS that runs on some kind of UNIX like OS such as Linux / *BSD or just a friends Mac OS notebook and you don't want your HTTP traffic to be sniffed (spyed) by your local ISP, e.g. you want to get some kind of simple Web browsing anonimity on the Internet, easier alternative to using SSH Dynamic Tunnel to Proxify encrypted traffic is to just run a Proxy Cache server on remote *nix host to which you have access and configure your browser to use the host as proxy. Besides enhanced traffic privacy other advantage of running a Proxy server is of course the fact that using proxy server cache significantly improve page speed opening times, optimize web-sites content delivery saving you a lot of bandwidth and off-loads remote sites because already cached content at the proxy is served directly from the proxy server instead of each time requested from the server. Those admins who remember "the dawn of mass internet use" should remember that a lot of the small and middle sized internet providers were into the habit to use Squid transparent proxy with huge Disk Cache in order to speed-up their customers internet page opening times and thus offer a superior service and save money for themselves since the ISPs were also paying not for a bandwidth to end providers but for used overall Traffic.

Historically I've personally used TinyProxy a lot to "obfuscate" my traffic (hide my originating host IP) and to save-bandwidth cause in late 1990's I paid my internet bills based on used traffic and starting a TinyProxy saved me network traffic and hence cut my Internet bill but even today it is great to optimize web traffic.

Usually for a single or few persons proxy it is not worthy to use "Full Featured" Complex Caching servers such as Squid Cache because installing and configuring one has plenty of dependency packages and a lot of time to spend in configuring (dealing with squid's cryptic squid.conf) it is much better and easier to use a light-weight HTTP/HTTPS proxy service (daemon) such as TinyProxy if using Tor Project (Anonymity Online) Network to protect against network surveillance / traffic analysis Privoxy.

Below is TinyProxy package description from Debian

apt-cache show tinyproxy|grep -i desc -A 5
Description: A lightweight, non-caching, optionally anonymizing http proxy
An anonymizing http proxy which is very light on system resources,
ideal for smaller networks and similar situations where other proxies
(such as Squid) may be overkill and/or a security risk. Tinyproxy can
also be configured to anonymize http requests (allowing for exceptions
on a per-header basis).

To install TinyProxy on Debian Linux:

root@debian:~# apt-get update && apt-get install –yes tinyproxy

To TinyProxy on CentOS 6.5 / 7 Linux:

Install first EPEL (Packages for Enterprise Linux) RPM repository (included by default in CentOS 6 / 7) – also possible to install easily on RHELs (but on RHEL you need to add epel rpm package manually)

[root@centos ~]# yum install -y epel-release
[root@centos ~]# yum -y install tinyproxy

Configuring TinyProxy Linux:

On Debian Linux config is in:

vim /etc/tinyproxy.conf

On CentOS / Fedora / RHEL RPM based ones:

vim /etc/tinyproxy/tinyproxy.conf

You might want to change some of below default values:

# User and Group with which TinyProxy will be running
User nobody
Group nogroup
# Default proxy listen port
Port 8888
# Specifies IP (interface) to be used for outgoing proxy host connections
Bind 192.168.0.1
# To which IP address (interface) tinyproxy will listen for connections. If uncommented it will listen to all available network interfaces
Listen 192.168.0.1
# Seconds after which connection will be terminated
Timeout 600
# Where proxy queries will be logged (very useful to keep a history for yourself on pages you have visted), sometimes useful if you wipe out browser cache
Logfile "/var/log/tinyproxy/tinyproxy.log"
# How many clients (connections) can be made to tinyproxy for one client (personal use) 100 is a good value
MaxClients 100
# This two values are settings on how many (minumum and maximum) tinyproxy instances will listen for connection from remote hosts – this should be familiar to people who configured Apache webserver, note that tinyproxy uses process fork and not threads
MinSpareServers 5
MaxSpareServers 20
# This is how many sites connections the proxy will listen to
# if you open 20 sites in 20 tabs which will refresh make here 20
# you might need to raise this if you have 10 clients
StartServers 10
# hosts to allow connection to proxy server from (Be careful to allow only access from your network otherwise you might end up being an open proxy), allowing some hax0r to do something terrible proxying through you
Allow 127.0.0.1
#The Log Level – "Error" is generally Okay, who may like more information can take Notice – Connect and Info
LogLevel Error
# pidfile location
PidFile "/var/run/tinyproxy/tinyproxy.pid"
# The "Via" header is required by the HTTP RFC, but using the real host name
# is a security concern. If the following directive is enabled, the string
# supplied will be used as the host name in the Via header; otherwise, the
# server's host name will be used.
#
ViaProxyName "tinyproxy

To install TinyProxy on FreeBSD

freebsd# cd /usr/ports/www/tinyproxy
freebsd# make install distclean

If you need to change the default TinyProxy port from port 8888 to something else for security reasons edit:

vim /usr/local/etc/tinyproxy.conf

Find port and change is to lets say 7000 or whatever network port is good for you

To make TinyProxy auto start on BSD boot time:

freebsd# vim /etc/rc.conf
tinyproxy_enable="YES"

To start / stop tinyproxy any time use:

freebsd# /usr/local/etc/rc.d/tinyproxy start

For Firefox users I also recommend using TinyProxy is to install FoxyProxy FF extension

add-multiple-proxies-and-choose-between-them-on-firefox-foxyproxy-screenshot

Securing TinyProxy traffic with SSH Tunnel

If you're concerned about transferred data security between your client host (Desktop) machine and remote installed Proxy server that your ISP or some malicious guy could sniff your website login credentials using Man in the Middle attack, its also a good idea to use TinyProxy together with SSH Tunnel, that's pretty easy if you have an SSH client on your machine (if you're on a Mac OS X).

ssh User@server.remote-tinyproxy-server-host.com -L 1234:localhost:8888 -N

To make SSH Tunnel Windows users could use Plink or MobaXterm embedded Tunneling interface

Then setup your Firefox / Chrome / Opera browser to use as a Proxy localhost:1234

Tags: bandwidth, Below, data security, debian linux, Easy, easy proxy, freebsd, host name, hosts, internet providers, login credentials, lot, opening, page, ports, privoxy, Proxy Caching, proxy server, running, save bandwidth proxy, squid cache, ssh, tinyproxy, tunnel proxy traffic, use, vim, website login
Posted in Everyday Life, Firefox, Linux, Linux and FreeBSD Desktop, Networking, System Administration, Various | 3 Comments »

How to make a mirror of website on GNU / Linux with wget / Few tips on wget site mirroring

Wednesday, February 22nd, 2012

how-to-make-mirror-of-website-on-linux-wget

Everyone who used Linux is probably familiar with wget or has used this handy download console tools at least thousand of times. Not so many Desktop GNU / Linux users like Ubuntu and Fedora Linux users had tried using wget to do something more than single files download.
Actually wget is not so popular as it used to be in earlier linux days. I've noticed the tendency for newer Linux users to prefer using curl (I don't know why).

With all said I'm sure there is plenty of Linux users curious on how a website mirror can be made through wget.
This article will briefly suggest few ways to do website mirroring on linux / bsd as wget is both available on those two free operating systems.

1. Most Simple exact mirror copy of website

The most basic use of wget's mirror capabilities is by using wget's -mirror argument:

# wget -m http://website-to-mirror.com/sub-directory/

Creating a mirror like this is not a very good practice, as the links of the mirrored pages will still link to external URLs. In other words link URL will not pointing to your local copy and therefore if you're not connected to the internet and try to browse random links of the webpage you will end up with many links which are not opening because you don't have internet connection.

2. Mirroring with rewritting links to point to localhost and in between download page delay

Making mirror with wget can put an heavy load on the remote server as it fetches the files as quick as the bandwidth allows it. On heavy servers rapid downloads with wget can significantly reduce the download server responce time. Even on a some high-loaded servers it can cause the server to hang completely.
Hence mirroring pages with wget without explicity setting delay in between each page download, could be considered by remote server as a kind of DoS – (denial of service) attack. Even some site administrators have already set firewall rules or web server modules configured like Apache mod_security which filter requests to IPs which are doing too frequent HTTP GET /POST requests to the web server.
To make wget delay with a 10 seconds download between mirrored pages use:

# wget -mk -w 10 -np --random-wait http://website-to-mirror.com/sub-directory/

The -mk stands for -m/-mirror and -k / shortcut argument for –convert-links (make links point locally), –random-wait tells wget to make random waits between o and 10 seconds between each page download request.

3. Mirror / retrieve website sub directory ignoring robots.txt "mirror restrictions"

Some websites has a robots.txt which restricts content download with clients like wget, curl or even prohibits, crawlers to download their website pages completely.

/robots.txt restrictions are not a problem as wget has an option to disable robots.txt checking when downloading.
Getting around the robots.txt restrictions with wget is possible through -e robots=off option.
For instance if you want to make a local mirror copy of the whole sub-directory with all links and do it with a delay of 10 seconds between each consequential page request without reading at all the robots.txt allow/forbid rules:

# wget -mk -w 10 -np -e robots=off --random-wait http://website-to-mirror.com/sub-directory/

4. Mirror website which is prohibiting Download managers like flashget, getright, go!zilla etc.

Sometimes when try to use wget to make a mirror copy of an entire site domain subdirectory or the root site domain, you get an error similar to:

Sorry, but the download manager you are using to view this site is not supported.
We do not support use of such download managers as flashget, go!zilla, or getright

This message is produced by the site dynamic generation language PHP / ASP / JSP etc. used, as the website code is written to check on the browser UserAgent sent.
wget's default sent UserAgent to the remote webserver is:
Wget/1.11.4

As this is not a common desktop browser useragent many webmasters configure their websites to only accept well known established desktop browser useragents sent by client browsers.
Here are few typical user agents which identify a desktop browser:

Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0
Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0
Mozilla/6.0 (Macintosh; I; Intel Mac OS X 11_7_9; de-LI; rv:1.9b4) Gecko/2012010317 Firefox/10.0a4
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.2a1pre) Gecko/20110324 Firefox/4.2a1pre

etc. etc.

If you're trying to mirror a website which has implied some kind of useragent restriction based on some "valid" useragent, wget has the -U option enabling you to fake the useragent.

If you get the Sorry but the download manager you are using to view this site is not supported , fake / change wget's UserAgent with cmd:

# wget -mk -w 10 -np -e robots=off \ --random-wait --referer="http://www.google.com" \--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \--header="Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \--header="Accept-Language: en-us,en;q=0.5" \--header="Accept-Encoding: gzip,deflate" \--header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \--header="Keep-Alive: 300"

For the sake of some wget anonimity – to make wget permanently hide its user agent and pretend like a Mozilla Firefox running on MS Windows XP use .wgetrc like this in home directory.

5. Make a complete mirror of a website under a domain name

To retrieve complete working copy of a site with wget a good way is like so:

# wget -rkpNl5 -w 10 --random-wait www.website-to-mirror.com

Where the arguments meaning is:
-r – Retrieve recursively
-k – Convert the links in documents to make them suitable for local viewing
-p – Download everything (inline images, sounds and referenced stylesheets etc.)
-N – Turn on time-stamping
-l5 – Specify recursion maximum depth level of 5

6. Make a dynamic pages static site mirror, by converting CGI, ASP, PHP etc. to HTML for offline browsing

It is often websites pages are ending in a .php / .asp / .cgi … extensions. An example of what I mean is for instance the URL http://php.net/manual/en/tutorial.php. You see the url page is tutorial.php once mirrored with wget the local copy will also end up in .php and therefore will not be suitable for local browsing as .php extension is not understood how to interpret by the local browser.
Therefore to copy website with a non-html extension and make it offline browsable in HTML there is the –html-extension option e.g.:

# wget -mk -w 10 -np -e robots=off \ --random-wait \ --convert-links http://www.website-to-mirror.com

A good practice in mirror making is to set a download limit rate. Setting such rate is both good for UP and DOWN side (the local host where downloading and remote server). download-limit is also useful when mirroring websites consisting of many enormous files (documental movies, some music etc.).
To set a download limit to add –limit-rate= option. Passing by to wget –limit-rate=200K would limit download speed to 200KB.

Other useful thing to assure wget has made an accurate mirror is wget logging. To use it pass -o ./my_mirror.log to wget.

Tags: adm, argument, Auto, bandwidth, briefly, browser, BSD, capabilities, cgi, common, connection, copy, denial of service, denial of service attack, Desktop, download, Draft, exact mirror, external urls, fedora linux, flashget, free operating systems, gnu linux, heavy load, home directory, How to, internet connection, Linux, linux users, localhost, make, manager, mirror copy, Mozilla, option, page, php extension, quot, random links, remote server, request, servers, site mirroring, something, tendency, text, text html, time, Ubuntu, URLs, UserAgent, website mirror, wget
Posted in Linux, System Administration, Web and CMS | 2 Comments »

How to enable output compression (gzipfile content compression) in nginx webserver

Friday, April 8th, 2011

I have recently installed and configured a Debian Linux server with nginx
. Since then I’ve been testing around different ways to optimize the nginx performance.

In my nginx quest, one of the most crucial settings which dramatically improved the end client performance was enabling the so called output compression which in Apache based servers is also known as content gzip compression .
In Apache webservers the content gzip compression is provided by a server module called mod_deflate .

The output compression nginx settings saves a lot of bandwidth and though it adds up a bit more load to the server, the plain text files like html, xml, js and css’s download time reduces drasticly as they’re streamed to the browser in gzip compressed format.
This little improvement in download speed also does impact the overall end user browser experience and therefore improves the browsing speed experience with websites.

If you have already had experience nginx you already know it is a bit fastidious and you have to be very careful with it’s configuration, however thanksfully enabling the gzip compression was actually rather easier than I thought.

Here is what I added in my nginx config to enable output compression:

## Compression gzip on; gzip_buffers 16 8k; gzip_comp_level 9; gzip_http_version 1.1; gzip_min_length 0; gzip_vary on;

Important note here is that need to add this code in the nginx configuration block starting with:

http { .... ## Compression gzip on; gzip_buffers 16 8k; gzip_comp_level 9; gzip_http_version 1.1; gzip_min_length 0; gzip_vary on;

In order to load the gzip output compression as a next step you need to restart the nginx server, either by it’s init script if you use one or by killing the old nginx server instances and starting up the nginx server binary again:
I personally use an init script, so restarting nginx for me is done via the cmd:

debian:~# /etc/init.d/nginx restart Restarting nginx: nginx.

Now to test if the output gzip compression is enabled for nginx, you can simply use telnet

hipo@linux:~$ telnet your-nginx-webserver-domain.com 80 Escape character is '^]'.

After the Escape character is set ‘^]’ appears on your screen type in the blank space:

HEAD / HTTP/1.0

and press enter twice.
The output which should follow should look like:

HTTP/1.1 200 OK Server: nginx Date: Fri, 08 Apr 2011 12:04:43 GMT Content-Type: text/html Content-Length: 13 Last-Modified: Tue, 22 Mar 2011 15:04:26 GMT Connection: close Vary: Accept-Encoding Expires: Fri, 15 Apr 2011 12:04:43 GMT Cache-Control: max-age=604800 Accept-Ranges: bytes

The whole transaction with telnet command issued and the nginx webserver output should look like so:

hipo@linux:~$ telnet your-nginx-webserver-domain.com 80 Trying xxx.xxx.xxx.xxx... Connected to your-nginx-webserver-domain.com .Escape character is '^]'. HEAD / HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 08 Apr 2011 12:04:43 GMT
Content-Type: text/html
Content-Length: 13
Last-Modified: Tue, 22 Mar 2011 15:04:26 GMT
Connection: close
Vary: Accept-Encoding
Expires: Fri, 15 Apr 2011 12:04:43 GMT
Cache-Control: max-age=604800
Accept-Ranges: bytes

The important message in the returned output which confirms your nginx output compression is properly configured is:

Vary: Accept-Encoding

If this message is returned by your nginx server, this means your nginx now will distribute it’s content to it’s clients in compressed format and apart from the browsing boost a lot of server and client bandwitdth will be saved.

Tags: apache webservers, bandwidth, client, client performance, com, compression, compression gzip, config, configuration block, content, different ways, download, EncodingExpires, experience, Fri, GMTConnection, gzip, gzip compression, gzipfile, HEAD, How to, html xml, HTTP, init, init script, js, level, Linux, linux server, Mar, mod, Modified, nginx, OKServer, output compression, plain text files, server instances, servers, text, time, use, xml
Posted in Linux, System Administration, Web and CMS | 3 Comments »

Using rsync to copy / synchronize files or backups between Linux / BSD / Unix servers

Monday, November 21st, 2011

Rsync and Rsync over ssh logo picture

Many of us have already taken advantage of the powerful Rsync proggie, however I'm quite sure there are still people who never used rsync to transfer files between servers.. That's why I came with this small post to possibly introduce rsync to my blog readers.
Why Rsync and not Scp or SFTP? Well Rsync is designed from the start for large files transfer and optimized to do the file copying job really efficient. Some tests with scp against rsync will clearly show rsync's superiority.
Rsync is also handy to contiue copying of half copied files or backups and thus in many cases saves bandwidth and machine hdd i/o operations.

The most simple way to use rsync is:

server:~# rsync -avz -e ssh remoteuser@remotehost:/remote/directory /local/directory/

Where remoteuser@remotehost — is the the username and hostname of remote server to copy files to.
/remote/directory — is the directory where the rsync copied files should be stored
/local/directory — is the local directory from which files will be copied to remote directory

If not a preliminary passwordless ssh key (RSA / DSA) authentication is configured on remote server, the above command will prompt for a password otherwise the rsync will start doing the transfer.

If one needs to have a RSA or DSA (public / private key) passwordless SSH key authentication , a RSA key first should be generated and copied over to the remote server, like so:

server:~# ssh-keygen -t dsa ... server:~# ssh-copy-id -i ~/.ssh/id_dsa.pub root@remotehost ...

That's all folks, enjoy rsyncing 😉

Tags: authentication, backups, bandwidth, blog, bsd unix, command, copying, directoryIf, DSA, file, half, hostname, job, Linux, local directory, password, passwordless ssh, private key, proggie, remote server, remotehost, remoteuser, root, rsa key, rsync, scp, sftp, ssh, ssh copy, superiority, transfer, unix, unix servers, username, way
Posted in Everyday Life, Linux, System Administration, Various, Web and CMS | No Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘bandwidth’

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites