Grafana Loki has become a popular choice for log management on Linux systems, nowadays, because free software like under AGPLv3 licence, it’s lightweight, cost-efficient, and integrates seamlessly with modern observability stacks. Unlike traditional log systems, Loki focuses on indexing metadata (labels) instead of full log content, which makes it especially attractive for Linux environments where logs can grow quickly.
Grafana Loki can be used to create fully featured logging stack. It has a small index and highly compressed chunks which simplifies the operation and significantly lowers the Storage expense of it.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs labels (just like Prometheus labels).
Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.
In this article will give you some real-world, practical usage of Loki on Linux, from its setup from zero to day-to-day use workflows.
Reasons why to use Loki on Linux ?
Linux systems generate logs mainly in /var/log but often used extra installed Apps tend to log in different locations for easier log distinguishment, e.g.
logs location might lack a good structure (be everywhere) :
Some common example locations, where logs are stored
- /var/log/syslog
- /var/log/auth.log
- Application logs (/opt/app/logs/*.log)
- Container logs, are kept within respective container ( Docker / PodMan Kubernetes )
Sonner or later if you have to manage a large infrastructure of servers you end up, it is pretty easy to end up in a log mess.
This is exaclty where Loki helps you solve:
- Centralize logs from multiple machines (within Grafana)
- Search logs efficiently using log craeted labels
- Correlate logs with metrics in Grafana
Loki Architecture Overview
A typical Loki setup on Linux has 3 components:
- Loki server -> stores and queries logs
- Promtail -> collects logs from the around the system
- Grafana -> Use it to visualizes and queries logs
Promtail acts like a lightweight agent that tails log files and sends them to Loki.
I. Installing Loki on Linux
1. Download Loki
$ cd /usr/local/src
$ wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64
$ chmod +x loki-linux-amd64
# mv loki-linux-amd64 /usr/local/bin/loki
2. Create a simple config like
auth_enabled: false
server:
http_listen_port: 3100ingester:
lifecycler:
address: 127.0.0.1
chunk_idle_period: 5mschema_config:
configs:
– from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24hstorage_config:
filesystem:
directory: /var/lib/loki/chunks
3. Run Loki
# loki -config.file=loki.yaml
Hopefully if all is okay with loki.yaml config the service will start.
a. Installing Promtail (Log Collection)
Example config (to modify to your preferences):
scrape_configs:
– job_name: linux-logs
static_configs:
– targets:
– localhost
labels:
job: syslog
host: my-linux-server
__path__: /var/log/*.log
This collects all logs in /var/log/ and labels them.
b. Run Promtail
# promtail -config.file=promtail.yaml
! Note that loki and promtail it is run as root (to have permissions to files which will be processed). This is not the best practice, so for security reasons,
if you have the necessery storage move out the files to a central log aggregator directory with a script set a unprevileged non-root user for it and run the services with those user.
c. Run loki / promtail as non-root user:
Once tested it runs, it is good idea to run two tools with non-root user, i.e.:
Run promtail as a dedicated user (e.g., promtail).
Add that user to groups like:
adm (for /var/log)
systemd-journal (for journal logs)
Adjust file permissions if needed
# useradd –system –no-create-home promtail
# usermod -aG adm promtail
$ loki -config.file=loki.yaml
$ promtail -config.file=promtail.yaml
II. Practical Use Cases of Loki on Linux
1. System Troubleshooting
One good use of Loki is to Search for errors in syslog:
{job="syslog"} |= "error"
By this you can Quickly diagnose:
- Boot issues
- Service failures
- Kernel errors
2. SSH Login Monitoring
Track login attempts from /var/log/auth.log for many VM hosts:
{job="syslog"} |= "sshd"
You can detect:
- Failed login attempts
- Brute-force attacks
- Unauthorized access
3. Application Debugging (look for exceptions)
If your app logs to /var/log/app.log and you App running it, to get a view on java thrown exceptions:
{job="app"} |= "exception"
This use case can Help developers to:
- Trace bugs
- Monitor runtime issues
- Correlate logs with deployments
4. Multi-Server Log Aggregation
Once you run Promtail on multiple Linux servers:
labels:
host: server1
Then you can do query to extract collected data for each one if it:
{job="syslog", host=~"server1|server2"}
This makes multiple machines behave like one unified log source.
5. Log-Based Metrics
You can extract metrics from logs:
count_over_time({job="syslog"} |= "error" [5m])
Use this for:
- Alerting
- Error rate tracking
- Incident detection
III. Using Grafana for Visualization
In Grafana, you can:
- View logs in real time
- Build dashboards
- Create alerts based on log patterns
Example use would be:
Create Grafana Panel showing error rate per host and Alert when errors exceed a threshold.
Good Practices on Loki use
1. Always Use Meaningful Labels
Example for Good label should contain as many descriptory parameters as possible:
labels:
app: nginx
env: prod
virtualization: vmware
type: Middleware
service:: proxy
Customer: customerA
Bad obscure label:
labels:
request_id: 123456
2. Avoid Too many Unique labels
Keep in mind Too many unique labels leads to poor performance !.
3. Rotate Logs Properly and optimize with Secure Loki Endpoint
Loki won't manage your internal logs, as it can well complement ( but not replaces ), on Server / VM traditional tools like journalctl / grep / logrotate. but just give you a better overview of what is inside of service spit logs based on easy to give criterias from Grafana.
You will still need usually at best scenario to setup of a Central Logging Server (to store all Infrastucture logs).
Consider also that sending data from your logs with Loki, like with a zabbix client it is always a idea to have reverse proxy like NGINX or Haproxy to reduce Network bandwith and for better management centralization of the infra.
4. Secure Loki Endpoint
- Use reverse proxy (NGINX)
- Enable authentication in production
Closure Summary
On Linux, Grafana Loki can help when:
- You have multiple servers
- Logs are growing fast
- You need centralized and relatively easy observability
Loki has its downtimes too as processing the logs to really extract data hits a high CPU use. Running it on a multiple machines is useful,
especially if your machines has high unutilized CPU IDLE time and you want to make the log data collection per server based being so to say partially duplicated and indepdendent from centralized logging. .
For high scale infrastructure, however sysadmins prefer to use an ELK OpenSearch Stack or log databases such as:
VictoriaLogs. With having infrastrcture of 100 servers or so perhaps setting up with some Ansible automation Loki makes sense.
Loki is not meant to replace databases or full-text search engines, but great often for simple log aggregation and analysis and of the simplistic tools available today.
More helpful Articles
Tags: Application Debugging, Avoid High Cardinality, Download Loki, Failed, Final Thoughts, Hopefully, host, How to, Install, issues, job, linux systems, log files, multiple, nginx, poor performance, queries, real time, Rotate Logs Properly, rsyslog, Run Loki, Run Promtail, Secure Loki Endpoint, sysadmins, syslog, System Troubleshooting, Traditional Linux Logging, Use Meaningful Labels, var
