Posts Tagged ‘nginx’

How to Install and Use Grafana Loki on Linux for mupltiple server Log Metrics Monitoring

Tuesday, March 31st, 2026

how-to-install-and-use-grafana-loki-on-linux-for-log-metrics-monitoring-for-multiple-server-observability-logo
Grafana Loki has become a popular choice for log management on Linux systems, nowadays, because free software like under AGPLv3 licence, it’s lightweight, cost-efficient, and integrates seamlessly with modern observability stacks. Unlike traditional log systems, Loki focuses on indexing metadata (labels) instead of full log content, which makes it especially attractive for Linux environments where logs can grow quickly.

Grafana Loki can be used to create fully featured logging stack. It has a small index and highly compressed chunks which simplifies the operation and significantly lowers the Storage expense of it.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs labels (just like Prometheus labels).
Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.

In this article will give you some real-world, practical usage of Loki on Linux, from its setup from zero to day-to-day use workflows.

Reasons why to use Loki on Linux ?

Linux systems generate logs mainly in /var/log but often used extra installed Apps tend to log in different locations for easier log distinguishment, e.g.
logs location might lack a good structure (be everywhere) :

Some common example locations, where logs are stored

  • /var/log/syslog
  • /var/log/auth.log
  • Application logs (/opt/app/logs/*.log)
  • Container logs, are kept within respective container ( Docker /  PodMan Kubernetes )

Sonner or later if you have to manage a large infrastructure of servers you end up, it is pretty easy to end up in a log mess.

This is exaclty where Loki helps you solve:

  • Centralize logs from multiple machines (within Grafana)
  • Search logs efficiently using log craeted labels
  • Correlate logs with metrics in Grafana

Loki Architecture Overview


loki-use-stack-chain-diagram-from-cloud-to-grafana

A typical Loki setup on Linux has 3 components:

  1. Loki server -> stores and queries logs
  2. Promtail -> collects logs from the around the system
  3. Grafana -> Use it to visualizes and queries logs

Promtail acts like a lightweight agent that tails log files and sends them to Loki.

I. Installing Loki on Linux

1. Download Loki

$ cd /usr/local/src
$ wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64
$ chmod +x loki-linux-amd64
# mv loki-linux-amd64 /usr/local/bin/loki

2. Create a simple config like

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
  chunk_idle_period: 5m

schema_config:
  configs:
    – from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /var/lib/loki/chunks

3. Run Loki

# loki -config.file=loki.yaml


Hopefully if all is okay with loki.yaml config the service will start.

a. Installing Promtail (Log Collection)

Example  config (to modify to your preferences):

scrape_configs:
  – job_name: linux-logs
    static_configs:
      – targets:
          – localhost
        labels:
          job: syslog
          host: my-linux-server
          __path__: /var/log/*.log

This collects all logs in /var/log/ and labels them.

b. Run Promtail

# promtail -config.file=promtail.yaml

! Note that loki and promtail it is run as root (to have permissions to files which will be processed). This is not the best practice, so for security reasons,
if you have the necessery storage move out the files to a central log aggregator directory with a script set a unprevileged non-root user for it and run the services with those user.

c. Run loki / promtail as non-root user:

Once tested it runs, it is good idea to run two tools with non-root user, i.e.:
Run promtail as a dedicated user (e.g., promtail).

Add that user to groups like:

adm (for /var/log)

systemd-journal (for journal logs)
Adjust file permissions if needed

# useradd –system –no-create-home promtail
# usermod -aG adm promtail

$ loki -config.file=loki.yaml
$ promtail -config.file=promtail.yaml

II. Practical Use Cases of Loki on Linux

1. System Troubleshooting

One good use of Loki is to Search for errors in syslog:

{job="syslog"} |= "error"

By this you can Quickly diagnose:

  • Boot issues
  • Service failures
  • Kernel errors

2. SSH Login Monitoring

Track login attempts from /var/log/auth.log for many VM hosts:

{job="syslog"} |= "sshd"

You can detect:

  • Failed login attempts
  • Brute-force attacks
  • Unauthorized access

3. Application Debugging (look for exceptions)

If your app logs to /var/log/app.log and you App running it, to get a view on java thrown exceptions:

{job="app"} |= "exception"

This use case can Help developers to:

  • Trace bugs
  • Monitor runtime issues
  • Correlate logs with deployments

4. Multi-Server Log Aggregation

Once you run Promtail on multiple Linux servers:

labels:
  host: server1

Then you can do query to extract collected data for each one if it:

{job="syslog", host=~"server1|server2"}

This makes multiple machines behave like one unified log source.

5. Log-Based Metrics

You can extract metrics from logs:

count_over_time({job="syslog"} |= "error" [5m])

Use this for:

  • Alerting
  • Error rate tracking
  • Incident detection

III. Using Grafana for Visualization

In Grafana, you can:

  • View logs in real time
  • Build dashboards
  • Create alerts based on log patterns

Example use would be:

Create Grafana Panel showing error rate per host and Alert when errors exceed a threshold.

loki-log-drill-down-sample-in-grafana

Good Practices on Loki use

1. Always Use Meaningful Labels

Example for Good label should contain as many descriptory parameters as possible:

labels:
  app: nginx
  env: prod
  virtualization: vmware
  type: Middleware
  service:: proxy
  Customer: customerA

Bad obscure label:

labels:
  request_id: 123456  


2. Avoid Too many Unique labels

Keep in mind Too many unique labels leads to poor performance !.

3. Rotate Logs Properly and optimize with Secure Loki Endpoint

Loki won't manage your internal logs, as it can well complement ( but not replaces ), on Server / VM traditional tools like journalctl / grep / logrotate. but just give you a better overview of what is inside of service spit logs based on easy to give criterias from Grafana.
You will still need usually at best scenario to  setup of a Central Logging Server (to store all Infrastucture logs).
Consider also that sending data from your logs with Loki, like with a zabbix client it is always a idea to have reverse proxy like NGINX or Haproxy to reduce Network bandwith and for better management centralization of the infra.

4. Secure Loki Endpoint

  • Use reverse proxy (NGINX)
  • Enable authentication in production

Closure Summary

On Linux, Grafana Loki can help when:

  • You have multiple servers
  • Logs are growing fast
  • You need centralized  and relatively easy observability

Loki has its downtimes too as processing the logs to really extract data hits a high CPU use. Running it on a multiple machines is useful,
especially if your machines has high unutilized CPU IDLE time and you want to make the log data collection per server based being so to say partially duplicated and indepdendent from centralized logging. .
For high scale infrastructure, however sysadmins prefer to use an ELK OpenSearch Stack or log databases such as:
VictoriaLogs. With having infrastrcture of 100 servers or so perhaps setting up with some Ansible automation Loki makes sense.
Loki is not meant to replace databases or full-text search engines, but great often for simple  log aggregation and analysis and of the simplistic tools available today.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Cloud services, Linux, Monitoring, System Administration | No Comments »

Unlocking the Power of lnav: Logfile Navigator – ncurses text based tool guide to mutiple Logs on multiple servers easy analysis on Linux

Saturday, September 13th, 2025

lnav-syslog-screenshot-linux-virtual-machine

If you've ever found yourself buried under a mountain of log files, tailing multiple outputs, or grepping through endless lines trying to spot an error, it's time to meet your new best friend: lnav, the Logfile Navigator.

Lightweight, terminal-based, and surprisingly powerful, lnav is one of the most underrated tools for developers, sysadmins, and anyone who regularly digs into logs. It turns your chaotic logs into something that’s not only readable—but genuinely useful.

What is lnav and why use it ?

lnav (Logfile Navigator) is a command-line tool for viewing and analyzing log files. It goes beyond tail, less, or grep by:

  • Automatically detecting and merging log formats.
  • Highlighting timestamps, log levels, and errors.
  • Providing SQL-like queries over your logs.
  • Offering interactive navigation with a UI inside the terminal.

And yes, all of that without needing to set up a database or a server.

1. Installing lnav on Linux

Installation is straightforward. On most systems, you can install it via package managers:

On Ubuntu/Debian:

# apt install lnav

On Fedora:

# dnf install lnav

On Arch Linux:

# pacman -S lnav

Or build from source via GitHub if you want the latest version.

2. Use lnav Instead of Tail / Grep why?

Traditional tools are powerful, but they require manual work to chain together functionality. lnav gives you:

  • Automatic multi-log parsing: Drop multiple logs in, and it merges them chronologically.
  • Syntax highlighting: Errors and warnings stand out.
  • SQL querying: Run queries like SELECT * FROM syslog_log WHERE log_level = 'error';
  • Filtering and searching: Use intuitive filters and bookmarks to highlight specific entries.

3. Basic tool Usage is simple

Let’s say you want to inspect a system log:

# lnav /var/log/syslog

You'll immediately get:

  • Color-coded output (timestamps, levels, messages).
  • Scrollable view (arrow keys, PgUp, PgDn).
  • Real-time updates (like tail -f).
  • Search with /, filter with :filter-in, and even SQL queries.

Lets say you need to analyze Apache webserver logs recursively including the logs already rotated and gunzipped with *.gz extension on CentOS / Fedora / RHEL, you can do it with:

# lnav -r /var/log/httpd

You can parse the log file and get additional information about requests as well as you can print overall summary of log file.

Choose the line you want to parse. The selected line is always the one at the top of the window. Then press 'p' and you should see the following result:

https://pc-freak.net/images/lnav-get-extra-information-about-apache-query-with-P-press-key-screenshot-linux

Now, if you want to see a summary view of the logs by date and time, simply press 'i'.

lnav-linux-apache-log-review-summary-of-errors-warnings-normal-screenshot

To quit a screen you have chosen press 'q'.

4. LNAV helpful options and hotkeys

Once you've opened a log file/s for analyze you can use few hotkeys that will allow us to move through the output of lnav and the available views more easily:

e or E to jump to the next / previous error message.
w or W to jump to the next / previous warning message.
b or Backspace to move to the previous page.
Space to move to the next page.
g or G to move to the top / bottom of the current view.

To take a closer look at the way lnav operates, use -d option, the debug information is to be spit inside a .txt file:

# lnav /var/log/httpd -d lnav.txt

In this example, the debug information that is generated when lnav starts will be written to a file named lnav.txt inside the current working directory.

5. Real-World Use Cases

a. Troubleshooting application or system process Crashes

Open all relevant logs in one go:

# lnav /var/log/*.log

Errors are highlighted, and you can jump between them with n / N kbd keys.

b. Combining Multiple Logs

Working with an app that logs to different files and you need to combine:

# lnav /var/log/nginx/access.log /var/log/nginx/error.log


Or lets say you want to combine Apache Webserver with Haproxy log and get log summaries or filter out stuff:

# lnav /var/log/apache2/access.log /var/log/haproxy.log


Now you will get a single, chronological timeline of events.

 

If you want to Search for a concrete occurance of Error / Warning or IP address inside a bunch of loaded combined logs you can do it with the same command like in simple vim by pressing / (slash) from kbd and type out what you want to filter out to get shown.

c. Analyze SQL Queries Logs

Yes, you can actually do this by passing it query in its command prompt :

:.schema
:SELECT log_time, log_level, log_message FROM syslog_log WHERE log_level = 'error';

You get a table of filtered logs, sortable by columns.
 

6. lnav more usage command tips

  • :help — Opens the help menu.
  • :filter-in <string> — Show only lines matching <string>.
  • :filter-out <string> — Hide lines matching <string>.
  • :export-to <filename> — Export current view to a file.
  • :tag <tagname> — Tag lines for later reference.
  • q — Quit (but why would you want to?).

 

7. Using lnav as a pager for systemd-journald

# journalctl | lnav
# journalctl -f | lnav
# journalctl -u ssh.service | lnav

https://pc-freak.net/images/lnav_sshservice-log-view-screenshot-linux
 

8. Use lnav to review remote ssh logs

Newer versions after 0.10 supports ssh protocol as well and theoretically should work:

# lnav user@server-name-here:/var/log/file.log


To read all logs inside /var/log

# lnav root@server-name-here:/var/log/
# lnav root@server-name-here:/var/log/*.err

9. Using lnav to view docker container logs

# docker logs 811ab84aa95l | lnav
# docker logs -f application | lnav

The latest version of lnav supports even the following  simplified docker:// URL syntax:

# lnav docker://{container_id_or_name}/dir_path/to/log/file
# lnav docker://{container_id_or_name}/var/dir_path/log
# lnav docker://application/var/log/
# lnav docker://applcation/var/log/nginx/nginx.app.log

10. Monitoring compilation and command output useful for developers
 

Compilation from archived tar balls with ./configure && make etc. generate lot of outputs and logs while working. 
Here is where the tool can come handy. 
For example, here is how to watch the output of make command when compiling something:

# lnav -e './configure && make'

 11. Learning lnav tool through online ssh service availability via lnav.org

f you're lazy to install it and want to test it anyways:
 

# Start The Basic Tutorial:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password tutorial1@demo.lnav.org


# Playground:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password playground@demo.lnav.org


Closure

While tools like Kibana, Grafana, and ELK stacks are powerful, they can be overkill for many use cases—especially when you're SSHed into a box and just need to get answers fast. That’s where lnav shines as it is fast, lightweight, visual and can be used offline.

If you’re a developer, sysadmin, SRE (Site Reliability Engineer), or just someone who cares about logs, give lnav a spin. It might just become among your favorite sysadm tools on Linux and safe you pretty much of time if you have to do log reading and analyzing on daily basis (for example if you're admining 20+ or more Linux servers.

 

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Monitoring, System Administration, Various | No Comments »

How to enable output compression (gzipfile content compression) in nginx webserver

Friday, April 8th, 2011

I have recently installed and configured a Debian Linux server with nginx
. Since then I’ve been testing around different ways to optimize the nginx performance.

In my nginx quest, one of the most crucial settings which dramatically improved the end client performance was enabling the so called output compression which in Apache based servers is also known as content gzip compression .
In Apache webservers the content gzip compression is provided by a server module called mod_deflate .

The output compression nginx settings saves a lot of bandwidth and though it adds up a bit more load to the server, the plain text files like html, xml, js and css’s download time reduces drasticly as they’re streamed to the browser in gzip compressed format.
This little improvement in download speed also does impact the overall end user browser experience and therefore improves the browsing speed experience with websites.

If you have already had experience nginx you already know it is a bit fastidious and you have to be very careful with it’s configuration, however thanksfully enabling the gzip compression was actually rather easier than I thought.

Here is what I added in my nginx config to enable output compression:

## Compression
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 9;
gzip_http_version 1.1;
gzip_min_length 0;
gzip_vary on;

Important note here is that need to add this code in the nginx configuration block starting with:

http {
....
## Compression
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 9;
gzip_http_version 1.1;
gzip_min_length 0;
gzip_vary on;

In order to load the gzip output compression as a next step you need to restart the nginx server, either by it’s init script if you use one or by killing the old nginx server instances and starting up the nginx server binary again:
I personally use an init script, so restarting nginx for me is done via the cmd:

debian:~# /etc/init.d/nginx restart
Restarting nginx: nginx.

Now to test if the output gzip compression is enabled for nginx, you can simply use telnet

hipo@linux:~$ telnet your-nginx-webserver-domain.com 80
Escape character is '^]'.

After the Escape character is set ‘^]’ appears on your screen type in the blank space:

HEAD / HTTP/1.0

and press enter twice.
The output which should follow should look like:


HTTP/1.1 200 OK
Server: nginx
Date: Fri, 08 Apr 2011 12:04:43 GMT
Content-Type: text/html
Content-Length: 13
Last-Modified: Tue, 22 Mar 2011 15:04:26 GMT
Connection: close
Vary: Accept-Encoding
Expires: Fri, 15 Apr 2011 12:04:43 GMT
Cache-Control: max-age=604800
Accept-Ranges: bytes

The whole transaction with telnet command issued and the nginx webserver output should look like so:

hipo@linux:~$ telnet your-nginx-webserver-domain.com 80
Trying xxx.xxx.xxx.xxx...
Connected to your-nginx-webserver-domain.com
.Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 08 Apr 2011 12:04:43 GMT
Content-Type: text/html
Content-Length: 13
Last-Modified: Tue, 22 Mar 2011 15:04:26 GMT
Connection: close
Vary: Accept-Encoding
Expires: Fri, 15 Apr 2011 12:04:43 GMT
Cache-Control: max-age=604800
Accept-Ranges: bytes

The important message in the returned output which confirms your nginx output compression is properly configured is:

Vary: Accept-Encoding

If this message is returned by your nginx server, this means your nginx now will distribute it’s content to it’s clients in compressed format and apart from the browsing boost a lot of server and client bandwitdth will be saved.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration, Web and CMS | 3 Comments »

Fix to “413 Request Entity Too Large” error in Nginx webserver and what causes it

Friday, November 14th, 2014

nginx_413_request_entity_too_large-fix

If you administer NGINX caching server serving static files content and redirecting some requests to Apache and you end up with errors when uploading big files (using HTTP PUT method), even though in Apache's PHP  upload_max_filesize is set to relatively high number upload_max_filesize = 60M.

Here is what happens during hand shake of web-browser -> server interaction 'till status is returned:
 

Web browser or Webcrawler robot goes through the following phases while talking to Web server:

 

1. Obtain an IP address from the IP name of the site (base on site URL without the leading 'http://'). 
This is provided by domain name servers (DNSs) configured for PC.
2. Open an IP socket connection to that IP address.
3. Write an HTTP data stream through that socket
(4) Receive an HTTP data stream back from the Web server in response. 
This data stream contains status codes whose values are determined by the HTTP protocol
whether successful. 

 

In the case the is recognized and reported to client 'web browser', causing the error.

The fix is to also increase max file upload limit in NGINX this is done via:
 
client_max_body_size variable in /usr/local/nginx/nginx.conf (or /etc/nginx/nginx.conf whether Nginx is installed from package).
Here is extract from nginx.conf

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

 

    server {
        client_max_body_size 60M;
        listen       80;
        server_name  localhost;

        # Main location
        location / {
            proxy_pass         http://127.0.0.1:8000/;
        }
    }
}


To make new configuration active Restart Nginx:

/etc/init.d/nginx restart

Tags: , , , , , , , , , , ,
Posted in Nginx, System Administration, Various, Web and CMS | No Comments »

Tracking multiple log files in real time in Linux console / terminal (MultiTail)

Monday, July 29th, 2013

Multitail multiple tail Debian GNU Linux viewing Apache access and error log in shared screen
Whether you have to administer Apache, Nginx or Lighttpd, or whatever other kind of daemon which interactively logs user requests or errors you probably already know well of tail command (tail -f /var/log/apache2/access.log) is something Webserver Linux admin can't live without. Sometimes however you have number of Virtualhost (domains) each configured to log site activity in separate log file. One solution to the problem is to use GNU Screen (screen – terminal emulator) to launch multiple screen session and launch separate tail -f /var/log/apache2/domain1/access.log , tail -f /var/log/apache2/domain2/access.log etc. This however is a bit of hack and except configuring screen to show multiple windows on one Virtual Terminal (tty or vty in gnome), you can't really see output simultaneously in one separated window.

Here is where multitail comes handy. MultiTail is tool to visualize in real time log records output of multiple logs (tails) in one shared terminal Window. MultiTail is written to use ncurses library used by a bunch of other useful tools like Midnight Command so output is colorful and very nice looking.

Here is MultiTail package description on Debian Linux:

linux:~# apt-cache show multitail|grep -i description -A 1
Description-en: view multiple logfiles windowed on console
 multitail lets you view one or multiple files like the original tail
—
Description-md5: 5e2f688efb214b063bdc418a705860a1
Tag: interface::text-mode, role::program, scope::utility, uitoolkit::ncurses,
root@noah:/home/hipo# apt-cache show multitail|grep -i description -A 1
Description-en: view multiple logfiles windowed on console
 multitail lets you view one or multiple files like the original tail
—
Description-md5: 5e2f688efb214b063bdc418a705860a1
Tag: interface::text-mode, role::program, scope::utility, uitoolkit::ncurses,
 

Multiple Tail is available across most Linux distributions to install on Debian / Ubuntu / Mint etc. Linux:

debian:~# apt-get install --yes multitail
...

On recent Fedora / RHEL / CentOS etc. RPM based Linuces to install:

[root@centos ~]# yum -y install multitail
...

On FreeBSD multitail is available to install from ports:

freebsd# cd /usr/ports/sysutils/multitail
freebsd# make install clean
...

Once installed to display records in multiple files lets say Apache domain name access.log and error.log

debian:~# multitail -f /var/log/apache2/access.log /var/log/apache2/error.log
…

It has very extensive help invoked by simply pressing h while running

multtail-viewing-in-gnome-shared-screen-debian-2-log-files-screenshot

Even better multitail is written to already have integrated color schemes for most popular Linux services log files

multitail multiple tail debian gnu linux logformat different color schemes screenshot
List of supported MulLog Color schemes as of time of writting article is:

acctail, acpitail, apache, apache_error, argus, asterisk, audit, bind, boinc, boinctail ,checkpoint, clamav, cscriptexample, dhcpd, errrpt, exim, httping, ii, inn, kerberos, lambamoo, liniptfw, log4j, mailscanner, motion, mpstat, mysql, nagtail, netscapeldap, netstat, nttpcache, ntpd, oracle, p0f, portsentry, postfix, pptpd, procmail, qmt-clamd, qmt-send, qmt-smtpd, qmt-sophie, qmt-spamassassin, rsstail, samba, sendmail, smartd, snort spamassassin, squid, ssh, strace, syslog, tcpdump, vmstat, vnetbr, websphere, wtmptail

To tell it what kind of log Color scheme to use from cmd line use:

debian:~# multitail -Csapache /var/log/apache2/access.log /var/log/apache2/error.log

multiple tail with Apache highlight on Debian Linux screenshot

Useful feature is to run command display in separate Windows while still following log output, i.e.:

[root@centos:~]# multitail /var/log/httpd.log -l "netstat -nat"
...

Multitail can also merge output from files in one Window, while in second window some other log or command output is displayed. To merge output from Apache access.log and error.log:

debian:~# multitail /var/log/apache2/access.log -I /var/log/apache2/error.log

When merging two log files output to show in one Window it is useful to display each file output in different color for the sake of readability

For example:
 

debian:~# multitail -ci green /var/log/apache/access.log -ci red -I /var/log/apache/error.log
multitail merged Apache access and error log on Debian Linux

To display output from 3 log files in 3 separate shared Windows in console use:

linux:~# multitail -s 2 /var/log/syslog /var/log/apache2/access.log /var/log/apache2/error.log

For some more useful examples, check out MultiTail's official page examples
There is plenty of other useful things to do with multitail, for more RTFM 🙂

Tags: , , , , , , , , , , , , , , ,
Posted in Everyday Life, FreeBSD, Linux, System Administration | No Comments »

How to disable nginx static requests access.log logging

Monday, March 5th, 2012

NGINX logo Static Content Serving Stop logging

One of the companies, where I'm employed runs nginx as a CDN (Content Delivery Network) server.
Actually nginx, today has become like a standard for delivering tremendous amounts of static content to clients.
The nginx, server load has recently increased with the number of requests, we have much more site visitors now.
Just recently I've noticed the log files are growing to enormous sizes and in reality this log files are not used at all.
As I've used disabling of web server logging as a way to improve Apache server performance in past time, I thought of implying the same little "trick" to improve the hardware utilization on the nginx server as well.

To disable logging, I proceeded and edit the /usr/local/nginx/conf/nginx.conf file, commenting inside every occurance of:

access_log /usr/local/nginx/logs/access.log main;

to

#access_log /usr/local/nginx/logs/access.log main;

Next, to load the new nginx.conf settings I did a restart:

nginx:~# killall -9 nginx; sleep 1; /etc/init.d/nginx start

I expected, this should be enough to disable completely access.log, browser request logins. Unfortunately /usr/local/nginx/logs/access.log was still displaying growing with:

nginx:~# tail -f /usr/local/nginx/logs/access.log

After a bit thorough reading of nginx.conf config rules, I've noticed there is a config directive:

access_log off;

Therefore to succesfully disable logging I had to edit config occurance of:

access_log /usr/local/nginx/logs/access.log main

to

After a bit thorough reading of nginx.conf config rules, I've noticed there is a config directive:

access_log off;

Therefore to succesfully disable logging I had to edit config occurance of:

access_log /usr/local/nginx/logs/access.log main

to

access_log /usr/local/nginx/logs/access.log main
access_log off;

Finally to load the new settings, which thanksfully this time worked, I did nginx restart:

nginx:~# killall -9 nginx; sleep 1; /etc/init.d/nginx start

And hooray! Thanks God, now nginx logging is disabled!

As a result, as expected the load avarage on the server reduced a bit 🙂

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration, Various | 3 Comments »

How to Prevent Server inaccessibility by using a secondary SSH Server access port

Monday, December 12th, 2011

One of the Debian servers’s SSH daemon suddenly become inaccessible today. While trying to ssh I experienced the following error:

$ ssh root@my-server.net -v
OpenSSH_5.8p1 Debian-2, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to mx.soccerfame.com [83.170.104.169] port 22.
debug1: Connection established.
debug1: identity file /home/hipo/.ssh/id_rsa type -1
debug1: identity file /home/hipo/.ssh/id_rsa-cert type -1
debug1: identity file /home/hipo/.ssh/id_dsa type -1
debug1: identity file /home/hipo/.ssh/id_dsa-cert type -1
...
Connection closed by remote host

Interestingly only the SSH server and sometimes the mail server was failing to respond and therefore any mean to access the server was lost. Anyways some of the services on the server for example Nginx continued working just fine.
Some time ago while still working for design.bg – web development company, I’ve experienced some similar errors with SSH servers, so I already had a clue, on a way to work around the issue and to secure myself against the situation to loose access to remote server because the secure shell daemon has broken up.

My work around is actually very simple, I run a secondary sshd (different sshd instance) listening on a different port number.

To do so I invoke the sshd daemon on port 2207 like so:

debian:~# /usr/sbin/sshd -p 2207
debian:~#

Besides that to ensure my sshd -p 2207 will be running on next boot I add:

/usr/sbin/sshd -p 2207

to /etc/rc.local (before the script end line exit 0 ). I do set the sshd -p 2207 to run via /etc/rc.local on purpose instead of directly adding a Port 2207 line in /etc/ssh/sshd_config. The reason, why I’m not using /etc/ssh/sshd_config is that I’m not sure if using the sshd config to set a secondary port does run the port under a different sshd parent. If using the config doesn’t run the separate ssh port under a different server parent this will mean that once the main parent hangs, the secondary port will become inaccessible as well.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in FreeBSD, Linux, System Administration | No Comments »

How to find out all programs bandwidth use with (nethogs) top like utility on Linux

Friday, September 30th, 2011

Just run across across a super nice top like, program for system administrators, its called nethogs and is definitely entering my “l337” admin outfit next to tools like iftop, nettop, ettercap, darkstat htop, iotop etc.

nethogs is ultra easy to use, to get immediately in console statistics about running processes UPLOAD and DOWNLOAD bandwidth consumption just run it:

linux:~# nethogs

Nethogs screenshot on Linux Server with Nginx
Nethogs running on Debian GNU/Linux serving static web content with Nginx

If you need to check what program is using what amount of network bandwidth, you will definitely love this tool. Having information of bandwidth consumption is also viewable partially with iftop, however iftop is unable to track the bandwidth consumption to each process using the network thus it seems nethogs is unique at what it does.

Nethogs supports IPv4 and IPv6 as well as supports network traffic over ppp. The tool is available via package repositories for Debian GNU/Lenny 5 and Debian Squeeze 6.

To install Nethogs on CentOS and Fedora distributions, you will have to install it from source. On CentOS 5.7, latest nethogs which as of time of writting this article is 0.8.0 compiles and installs fine with make && make install commands.

In the manner of thoughts of network bandwidth monitoring, another very handy tool to add extra understanding on what kind of traffic is crossing over a Linux server is jnettop
jnettop shows which hosts/ports is taking up the most network traffic.
It is available for install via apt in Debian 5/6).

Here is a screenshot on jnettop in action:

Jnettop check network traffic in console

To install jnettop on latest Fedoras / CentOS / Slackware Linux it has to be download and compiled from source via jnettop’s official wiki page
I’ve tested jnettop install from source on CentOS release 5.7 and it seems to compile just fine using the usual compile commands:

[root@prizebg jnettop-0.13.0]# ./configure
...
[root@prizebg jnettop-0.13.0]# make
...
[root@prizebg jnettop-0.13.0]# make install

If you need to have an idea on the network traffic passing by your Linux server distringuished by tcp/udp/icmp network protocols and services like ssh / ftp / apache, then you will definitely want to take a look at nettop (if of course not familiar with it yet).
Nettop is not provided as a deb package in Debian and Ubuntu, where it is included as rpm for CentOS and presumably Fedora?
Here is a screenshot on nettop network utility in action:

Nettop server traffic division by protocol screenshot
FreeBSD users should be happy to find out that jnettop and nettop are part of the ports tree and the two can be installed straight, however nethogs would not work on FreeBSD, I searched for a utility capable of what Nethogs can, but couldn’t find such.
It seems the only way on FreeBSD to track bandwidth back and from originating process is using a combination of iftop and sockstat utilities. Probably there are other tools which people use to track network traffic to the processes running on a hos and do general network monitoringt, if anyone knows some good tools, please share with me.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in FreeBSD, Linux, System Administration | No Comments »

Few nginx.conf configuration options for Nginx to improve webserver performance

Tuesday, April 12th, 2011

Nginx server main logo with russian star
From my previous two articles How to install nginx webserver from source on Debian Linux / Install Latest Nginx on Debian and How to enable output compression (gzipfile content compression) in nginx webserver , I have explained how the Nginx server can be installed and configured easily.

As I’m continuing my nginx adventures this days, by trying to take the best out of the installed nginx server, I’ve found few configuration options, which does improve nginx’s server performance and thought it might be nice to share it here in hope that some other nginx novice might benefit out if them.
To setup and start using the options you will have of course to place the conf directives in /usr/local/nginx/conf/nginx.conf or wherever your nginx.conf is located.

The configuration options should be placed in nginx’s conf section which starts up with:

http {

Here are the configuration options useful in hastening my nginx’s performance:

1. General options nginx settings

## General Options
ignore_invalid_headers on;
keepalive_requests 2000;
recursive_error_pages on;
server_name_in_redirect off;
server_tokens off;

2. Connection timeout nginx settings

## Timeouts
client_body_timeout 60;
client_header_timeout 60;
keepalive_timeout 60 60;
send_timeout 60;
expires 24h;

3. server options for better nginx tcp/ip performance

## TCP options
tcp_nodelay on;
tcp_nopush on;

4. Increase the number of nginx worker processes

Somewhere near the beginning of nginx.conf file you should have the directive option:

worker_processes 1;

Make sure you change this option to:

worker_processes 4;

This will increase the number of spawned nginx worker processes in a way that more spawned threaded servers will await for client connections:

Being done with all the above settings, as a next step you have to restart the nginx server, in my case via the init script:

debian:~# /etc/init.d/nginx restart
Restarting nginx: nginx.

Now to check everything is fine with nginx and more specific that the worker_processes 4 options has taken place issue the command:

debian:~# ps axu |grep -i nginx|grep -v grep
root 20456 0.0 0.0 25280 816 ? Ss 10:35 0:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody 20457 0.0 0.0 25844 1820 ? S 10:35 0:00 nginx: worker process
nobody 20458 0.0 0.0 25624 1376 ? S 10:35 0:00 nginx: worker process
nobody 20459 0.0 0.0 25624 1376 ? S 10:35 0:00 nginx: worker process
nobody 20460 0.0 0.0 25624 1368 ? S 10:35 0:00 nginx: worker process

Above you notice the 4 nginx processes running with user nobody, they’re the same configured worker_processes I just pointed out above.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, System Administration | 2 Comments »