Posts Tagged ‘linux systems’

Why modern Linux systems feel Slow and how to Speed it up. Common RAM, CPU, and performance Linux problems in 2026 explained

Monday, May 18th, 2026

For years Linux we the Linux users proudly mocked Windows for bloated resource usage and that was a reason for many enthusiasts like me to start in the Linux realm.
There used to be the good old times where, lightweight distributions running comfortably inside 128MB of RAM were once common, and old computers and the hackers good old ThinkPads series were perfect for becoming a computer professional.

Fast-forward trip to 2026 and many modern GNU / Linux desktop's resource hunger has topped UP and a typical GUI environment such as Gnome is consuming as minimum 2 GB of RAM and often  4GB of RAM immediately after enters through the Login manager and machine. So many of the old computers if even running for 7-8 years and served well once updated or reused with Linux on a fresh install  prformance feels really bad. There of course work arounds to that as there are distributions such PuppyLinux / Tiny Core Linux / Linux Lite / Lubutuntu and even multiple articles online suggesting on how to place an ordinary Debian on Ubuntu and optimize it to work better on older hardware but still this article might be of help not only for old school Linux fans who install on old harware but also for sysadmins who has to deploy and install brand new Linux distributions and want to squeeze best of performance from the machine and make it as minimimalistic as possible in order to reduce the number of problems that might occur for system management.

So What happened, to make Linux performance degrade so dramatically over last 15 years ?

Old Hardware feels Slower Even With Linux

People often install Linux expecting miracles on ancient hardware.

Modern workloads assume:

  • SSD storage
  • multiple CPU cores
  • AVX instructions
  • GPU acceleration
  • large memory pools

Even lightweight Linux distributions struggle when rendering modern web applications on decade-old CPUs.

A 2007 machine browsing modern JavaScript-heavy websites experiences a fundamentally different workload than it did originally.

Web site use became computationally expensive.

 

Modern Linux Is Carrying the Weight of the development of Tech and Internet industry

A contemporary Linux desktop is no longer just:

  • X11
  • a window manager
  • a browser
  • a terminal emulator

Modern systems now run dozens of background services (as people run into complexity more and more instead of minimalism). Even a basic Linux install often runs by default things such as:
telemetry collectors, hardware abstraction layers, sandboxing frameworks, package management daemons, web server management platforms, indexing systems, GPU compositors, browser engines that resemble miniature operating systems and even with some specific distros embedded containers.

A typical desktop session environment on Linux today often includes as a base a bunch of software that is not always necessery such as:

  • systemd
  • dbus-daemon
  • pipewire
  • wireplumber
  • NetworkManager
  • xdg-desktop-portal
  • gvfsd
  • tracker-miner
  • udisksd
  • polkitd
  • bluetoothd
  • ModemManager
  • cupsd
  • flatpak-session-helper

Many younger users won't  never notice the burden of having those services running all time on the hardware as hardware today is mostly powerful and modern PC and notebooks often ship with 16GB or even some gaiming machines have 32 GB of memory.

As the default amount of memory on a laptop PC has become so abundant as 16GB RAM has become  "normal / standard ",  so software developers stopped aggressively optimizing memory consumption, plus the inclusing of AI vibe coding today and the abundant resource makes things with program optimization even more bloated.

The result of all this is more and more software entropy (the tendency of software systems to become more disorganized, complex, and harder to maintain over time).

The older UNIX philosophy no longer remembered by newer developers is completely forgotten. The old unix thinking was "Do one thing well.",
the new is "use everything no matter the efficiency if that would save you time"

As a result modern desktop applications instead ship entire browser engines for  simple things as displaying buttons.
This is exactly where Linux desktop gets heavily loaded and cause for whole system to work sluggish even on newer hardware. 
Very large part of those ineffiicient developed is caused by Electron:

Electron Framework for building Desktop apps worsened Linux performance

One of the largest reasons modern Desktop Linux / Windows systems is Electron (a framework for building desktop applications using JavaScript, HTML, and CSS).

Electron bundles essentially with:

  • Chromium
  • Node.js
  • V8 JavaScript engine
  • application runtime
  • UI rendering engine

and this is used in …inside many of the third party applications, which unfurtunately has to be used also on Linux, few examples that has heavy electron use are:

  • Discord client
  • Slack client
  • VS Code
  • Element 
  • Spotify
  • Visual Studio Code
  • Discord
  • Signal Desktop
  • Postman
  • Countless App launchers part of extra packages that one needs to use on Linux Desktop

…are frequently separate Chromium instances or use large part of chromium libs pretending to be native applications.

1. Finding top resource hungry Apps on Linux

To get a list of most memory heavy Apps on a Linux system:

# ps aux –sort=-%mem | head

 

You may discover that “lightweight desktop apps” and background services are consuming much more RAM than imagined.

Measuring Real Resource Usage Properly

Many users misunderstand Linux memory reporting.

Linux aggressively uses RAM for:

  • filesystem cache
  • buffers
  • inode caching

Note! Unused RAM is wasted RAM.

# free -h

Focus on:

  • available memory
  • swap activity
  • sustained pressure

Better command tools to optimize OS include:

htop
btop
smem
iotop
vmstat

Systemd  Useful but running default unused services

Mentioning systemd still starts wars on Linux forums.

Reality is nuanced.

Systemd solved real problems:

  • dependency management
  • predictable service startup
  • cgroup integration
  • journal logging
  • parallel boot
  • service supervision

However, it also dramatically expanded the scope of PID 1 responsibilities.

Leading to many Linux-es to now launch numerous services laying around, not known by the users and never (intially needed).

If you want to check and optimize systemd ecosystem to improve performance

2. Check systemd OS boot chain and disable unnecessery services

# systemd-analyze blame

And inspect active systemd units:

# systemctl list-units –type=service

Many Linux distributions has by default setup of unused:

  • printer services on systems without printers
  • modem services on desktops without modems
  • Bluetooth stacks on machines without Bluetooth devices
  • indexing daemons nobody uses

Disable unnecessary services carefully:

sudo systemctl disable –now ModemManager
sudo systemctl disable –now bluetooth
sudo systemctl disable –now cups


This alone will reduce memory usage and boot time.
A common set of unused Apps on Desktop and servers goes like this:
 

# Printing system (disable if you never use printers)

# systemctl disable –now cups.service cups-browsed.service

# Bluetooth support (disable if you don’t use Bluetooth devices)

# systemctl disable –now bluetooth.service

# Mobile broadband / modem support (disable if no 4G / 5G dongles)

# systemctl disable –now ModemManager.service

# Network discovery (AirPrint, LAN service discovery; disable if not needed)

# systemctl disable –now avahi-daemon.service

# Location services for apps/browser geolocation (disable if not used)

# systemctl disable –now geoclue.service

# Ubuntu crash reporting services (safe to disable for privacy/no reporting)

# systemctl disable –now apport.service whoopsie.service

# Desktop search indexing (GNOME file search; disable if you don’t use fast search)

# systemctl disable –now tracker-miner-fs.service tracker-extract.service tracker-store.service

 

For deeper analysis check out systemd cg groups use:

# systemd-cgtop

Or inspect slab allocator usage:

slabtop


3. Avoid using Flatpak and Snap for extra Apps

Flatpak and Snap Increase Isolation, provides many modern Apps that are not default shipped by Debian / Ubuntu / Fedora OS  etc (Deb / RPM) repos and keeps packages easily at latest but also puts great worthless overhead on system.
 

a) Modern packaging systems like Flatpak and Snap (Pros) prioritize:

  • sandboxing
  • dependency isolation
  • reproducibility
  • cross-distribution compatibility

This is good for security, however it comes at a cost.

b) Use of Flatpak and Snap pack. managers downsides

Flatpak applications frequently duplicate:

  • runtimes
  • libraries
  • graphics stacks
  • helper services

Snap packages compress applications into loop-mounted filesystem images which increase startup overhead and general memory fragmentation.

Inspect mounted Snap filesystems

# mount | grep snap


Inspect Flatpak runtimes:

# flatpak list


Considering that, traditional native packages remain significantly leaner in many cases.

4. Use Minimalistic GUI Desktop environment to reduce resource and use of complexity on Linux

Being mimimalist nowadays in a world of abundancy is considered wrong. However minimalism has its well known provent benefits. 

Wayland Is Efficient,  but X11 env with Minimalist GUI is better

Wayland itself is not inherently bloated.

However, modern compositors increasingly rely on:

  • GPU acceleration
  • animation pipelines
  • texture buffering
  • fractional scaling
  • HDR rendering
  • Vulkan / OpenGL abstractions

This improves:

  • smoothness
  • latency
  • security
  • multi-monitor support

…but increases baseline GPU and memory usage and still for performance cautious desktop users it is most likely not the best option.

For example, try to compare CPU / Mem / Disk use of:

  • Openbox on X11
  • KDE Plasma on Wayland with effects enabled

The performance difference is dramatic.


If you want to be a Linux Minimalist (benefit) and get astonishingly low resource usage try:

  • dwm
  • i3
  • bspwm
  • Openbox
  • Wmaker
  • XFCE
  • IceWM

Switching to one of those Linux ecosystem instead of the default heavy GNOME or KDE permits even further optimizations on Graphical environment level,  if users intentionally choose it. The downsides of that is twitching it will take you usually longer but if you setup one and the same desktop with the basic minimalist environment and you keep using it for daily work / development for years, invested time is worth.

5. Use browser extensions, habits or a lightweight  browser. 

Web browser a common source of slowness 

Web Browsers, became nowadays a fully featured Operating Systems. On many machines they are the largest consumers of RAM on Linux systems and on old computers main source of slowness. On older PCs try to use other small browser alternatives

A single browser tab may include:

  • isolated sandbox process
  • JavaScript runtime
  • GPU process
  • extension subsystem
  • video decoder
  • site isolation sandbox
  • service workers

a) Inspect Chromium process trees

# ps -ef | grep chromium

b) Inspect Firefox process trees

about:processes

 

A few “simple” tabs can easily consume several gigabytes.

The modern web itself is bloated:

  • gigantic JavaScript frameworks
  • endless analytics
  • autoplay video
  • AI scripts
  • tracking engines
  • real-time rendering

Shamefully, many websites today consume more RAM than entire operating systems from the early 2000s.

If you have to work on a PC with 4 or 8 GB with Linux maybe you can try to use a GUI browser only when necessery and for general reading and stuff use a minimalist version of browsers such as using a text / console web  browser and ones that are capable to support ncurses and javascript partially, a good candidate for a real console maniac or an old school hacker will be some of below:

  • Lynx (lightest, pure text)
  • w3m (text browser but supports javascripts partially)
  • Links / Links2 (fast, ultra-lightweight web browser works in both text and gui modes)
    NetSurf (graphical web browser built from scratch with its own independent layout and rendering engine, performs poor with javascript)
  • Browsh (can be often used instead of fully functional browser but buggy)

xlinks2-graphical-mode-lightweight-browser-linux

c) Use Lightweight Browsing Habits

Extensions matter enormously.

Block:

  • ads
  • trackers
  • autoplay
  • unnecessary scripts

uBlock Origin (free and open source browser block extension) alone can dramatically reduce CPU and RAM consumption.

Final words; the modern computing efficiency degredal

What a paradox, Modern hardware is unbelievably powerful, yet modern software consumes resources at almost the same rate hardware improves.

This phenomenon is partially explained by:abstraction layers, developer convenience, use of cross-platform frameworks, increased security isolation, the web technologies heaviness and reduced optimization pressure.

Even though the degredal in perforamance on old hardware, Linux itself remains extremely efficient at the kernel level.

The bloat largely exists and widens though in:

  • userland
  • desktop ecosystems
  • browser-centric software culture
     

The computing as we know it changed.

What once was: terminal-centric, native, lightweight,locally optimized, inter-dependent

turned over  last 10 years: browser-centricm, all time cloud-connected, sandboxed, abstraction-heavy, outer dependent

The good news is that GNU / Linux still gives users freedom, even though the freedom has reduced.

Even though the performance reduced,  Linux still remains one of the few environments where users retain meaningful control over their data and system complexity in the AI, Clouds era

 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Curious Facts, Educational, Hacks, Linux, Linux and FreeBSD Desktop, Performance Tuning, System Administration, System Optimization | No Comments »

How to Install and Use Grafana Loki on Linux for mupltiple server Log Metrics Monitoring

Tuesday, March 31st, 2026

how-to-install-and-use-grafana-loki-on-linux-for-log-metrics-monitoring-for-multiple-server-observability-logo
Grafana Loki has become a popular choice for log management on Linux systems, nowadays, because free software like under AGPLv3 licence, it’s lightweight, cost-efficient, and integrates seamlessly with modern observability stacks. Unlike traditional log systems, Loki focuses on indexing metadata (labels) instead of full log content, which makes it especially attractive for Linux environments where logs can grow quickly.

Grafana Loki can be used to create fully featured logging stack. It has a small index and highly compressed chunks which simplifies the operation and significantly lowers the Storage expense of it.
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs labels (just like Prometheus labels).
Log data itself is then compressed and stored in chunks in object stores such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even locally on the filesystem.

In this article will give you some real-world, practical usage of Loki on Linux, from its setup from zero to day-to-day use workflows.

Reasons why to use Loki on Linux ?

Linux systems generate logs mainly in /var/log but often used extra installed Apps tend to log in different locations for easier log distinguishment, e.g.
logs location might lack a good structure (be everywhere) :

Some common example locations, where logs are stored

  • /var/log/syslog
  • /var/log/auth.log
  • Application logs (/opt/app/logs/*.log)
  • Container logs, are kept within respective container ( Docker /  PodMan Kubernetes )

Sonner or later if you have to manage a large infrastructure of servers you end up, it is pretty easy to end up in a log mess.

This is exaclty where Loki helps you solve:

  • Centralize logs from multiple machines (within Grafana)
  • Search logs efficiently using log craeted labels
  • Correlate logs with metrics in Grafana

Loki Architecture Overview


loki-use-stack-chain-diagram-from-cloud-to-grafana

A typical Loki setup on Linux has 3 components:

  1. Loki server -> stores and queries logs
  2. Promtail -> collects logs from the around the system
  3. Grafana -> Use it to visualizes and queries logs

Promtail acts like a lightweight agent that tails log files and sends them to Loki.

I. Installing Loki on Linux

1. Download Loki

$ cd /usr/local/src
$ wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64
$ chmod +x loki-linux-amd64
# mv loki-linux-amd64 /usr/local/bin/loki

2. Create a simple config like

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
  chunk_idle_period: 5m

schema_config:
  configs:
    – from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /var/lib/loki/chunks

3. Run Loki

# loki -config.file=loki.yaml


Hopefully if all is okay with loki.yaml config the service will start.

a. Installing Promtail (Log Collection)

Example  config (to modify to your preferences):

scrape_configs:
  – job_name: linux-logs
    static_configs:
      – targets:
          – localhost
        labels:
          job: syslog
          host: my-linux-server
          __path__: /var/log/*.log

This collects all logs in /var/log/ and labels them.

b. Run Promtail

# promtail -config.file=promtail.yaml

! Note that loki and promtail it is run as root (to have permissions to files which will be processed). This is not the best practice, so for security reasons,
if you have the necessery storage move out the files to a central log aggregator directory with a script set a unprevileged non-root user for it and run the services with those user.

c. Run loki / promtail as non-root user:

Once tested it runs, it is good idea to run two tools with non-root user, i.e.:
Run promtail as a dedicated user (e.g., promtail).

Add that user to groups like:

adm (for /var/log)

systemd-journal (for journal logs)
Adjust file permissions if needed

# useradd –system –no-create-home promtail
# usermod -aG adm promtail

$ loki -config.file=loki.yaml
$ promtail -config.file=promtail.yaml

II. Practical Use Cases of Loki on Linux

1. System Troubleshooting

One good use of Loki is to Search for errors in syslog:

{job="syslog"} |= "error"

By this you can Quickly diagnose:

  • Boot issues
  • Service failures
  • Kernel errors

2. SSH Login Monitoring

Track login attempts from /var/log/auth.log for many VM hosts:

{job="syslog"} |= "sshd"

You can detect:

  • Failed login attempts
  • Brute-force attacks
  • Unauthorized access

3. Application Debugging (look for exceptions)

If your app logs to /var/log/app.log and you App running it, to get a view on java thrown exceptions:

{job="app"} |= "exception"

This use case can Help developers to:

  • Trace bugs
  • Monitor runtime issues
  • Correlate logs with deployments

4. Multi-Server Log Aggregation

Once you run Promtail on multiple Linux servers:

labels:
  host: server1

Then you can do query to extract collected data for each one if it:

{job="syslog", host=~"server1|server2"}

This makes multiple machines behave like one unified log source.

5. Log-Based Metrics

You can extract metrics from logs:

count_over_time({job="syslog"} |= "error" [5m])

Use this for:

  • Alerting
  • Error rate tracking
  • Incident detection

III. Using Grafana for Visualization

In Grafana, you can:

  • View logs in real time
  • Build dashboards
  • Create alerts based on log patterns

Example use would be:

Create Grafana Panel showing error rate per host and Alert when errors exceed a threshold.

loki-log-drill-down-sample-in-grafana

Good Practices on Loki use

1. Always Use Meaningful Labels

Example for Good label should contain as many descriptory parameters as possible:

labels:
  app: nginx
  env: prod
  virtualization: vmware
  type: Middleware
  service:: proxy
  Customer: customerA

Bad obscure label:

labels:
  request_id: 123456  


2. Avoid Too many Unique labels

Keep in mind Too many unique labels leads to poor performance !.

3. Rotate Logs Properly and optimize with Secure Loki Endpoint

Loki won't manage your internal logs, as it can well complement ( but not replaces ), on Server / VM traditional tools like journalctl / grep / logrotate. but just give you a better overview of what is inside of service spit logs based on easy to give criterias from Grafana.
You will still need usually at best scenario to  setup of a Central Logging Server (to store all Infrastucture logs).
Consider also that sending data from your logs with Loki, like with a zabbix client it is always a idea to have reverse proxy like NGINX or Haproxy to reduce Network bandwith and for better management centralization of the infra.

4. Secure Loki Endpoint

  • Use reverse proxy (NGINX)
  • Enable authentication in production

Closure Summary

On Linux, Grafana Loki can help when:

  • You have multiple servers
  • Logs are growing fast
  • You need centralized  and relatively easy observability

Loki has its downtimes too as processing the logs to really extract data hits a high CPU use. Running it on a multiple machines is useful,
especially if your machines has high unutilized CPU IDLE time and you want to make the log data collection per server based being so to say partially duplicated and indepdendent from centralized logging. .
For high scale infrastructure, however sysadmins prefer to use an ELK OpenSearch Stack or log databases such as:
VictoriaLogs. With having infrastrcture of 100 servers or so perhaps setting up with some Ansible automation Loki makes sense.
Loki is not meant to replace databases or full-text search engines, but great often for simple  log aggregation and analysis and of the simplistic tools available today.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Cloud services, Linux, Monitoring, System Administration | No Comments »

How to improve Linux kernel security with GrSecurity / Maximum Linux kernel security with GrSecurity

Tuesday, May 3rd, 2011

In short I’ll explain here what is Grsecurity http://www.grsecurity.net/ for all those who have not used it yet and what kind of capabilities concerning enhanced kernel security it has.

Grsecurity is a combination of patches for the Linux kernel accenting at the improving kernel security.

The typical application of GrSecurity is in the field of Linux systems which are administered through SSH/Shell, e.g. (remote hosts), though you can also configure grsecurity on a normal Linux desktop system if you want a super secured Linux desktop ;).

GrSecurity is used heavily to protect server system which require a multiple users to have access to the shell.

On systems where multiple user access is required it’s a well known fact that (malicious users, crackers or dumb script kiddies) get administrator (root) privileges with a some just poped in 0 day root kernel exploit.
If you’re an administrator of a system (let’s say a web hosting) server with multiple users having access to the shell it’s also common that exploits aiming at hanging in certain daemon service is executed by some of the users.
In other occasions you have users which are trying to DoS the server with some 0 day Denial of Service exploit.
In all this cases GrSecurity having a kernel with grsecurity is priceless.

Installing grsecurity patched kernel is an easy task for Debian and Ubuntu and is explained in one of my previous articles.
This article aims to explain in short some configuration options for a GrSecurity tightened kernel, when one have to compile a new kernel from source.

I would skip the details on how to compile the kernel and simply show you some picture screens with GrSecurity configuration options which are working well and needs to be set-up before a make command is issued to compile the new kernel.

After preparing the kernel source for compilation and issuing:

linux:/usr/src/kernel-source$ make menuconfig

You will have to select options like the ones you see in the pictures below:

[nggallery id=”8″]

After completing and saving your kernel config file, continue as usual with an ordinary kernel compilation, e.g.:

linux:/usr/src/kernel-source$ make
linux:/usr/src/kernel-source$ make modules
linux:/usr/src/kernel-source$ su root
linux:/usr/src/kernel-source# make modules_install
linux:/usr/src/kernel-source# make install
linux:/usr/src/kernel-source# mkinitrd -o initrd.img-2.6.xx 2.6.xx

Also make sure the grub is properly configured to load the newly compiled and installed kernel.

After a system reboot, if all is fine you should be able to boot up the grsecurity tightened newly compiled kernel, but be careful and make sure you have a backup solution before you reboot, don’t blame me if your new grsecurity patched kernel fails to boot! You’re on your own boy 😉
This article is written thanks to based originally on his article in Bulgarian. If you’re a Bulgarian you might also checkout static’s blog

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, System Administration | No Comments »

Tracking I/O hard disk server bottlenecks with iostat on GNU / Linux and FreeBSD

Tuesday, March 27th, 2012

Hard disk overhead tracking on Linux and FreeBSD with iostat

I've earlier wrote an article How to find which processes are causing hard disk i/o overhead on Linux there I explained very rawly few tools which can be used to benchmark hard disk read / write operations. My prior article accent was on iotop and dstat and it just mentioned of iostat. Therefore I've wrote this short article in attempt to explain a bit more thoroughfully on how iostat can be used to track problems with excessive server I/O read/writes.

Here is the command man page description;
iostat – Report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems

I will further proceed with few words on how iostat can be installed on various Linux distros, then point at few most common scenarious of use and a short explanation on the meaning of each of the command outputs.

1. Installing iostat on Linux

iostat is a swiss army knife of finding a server hard disk bottlenecks. Though it is a must have tool in the admin outfut, most of Linux distributions will not have iostat installed by default.
To have it on your server, you will need to install sysstat package:

a) On Debian / Ubuntu and other Debian GNU / Linux derivatives to install sysstat:

debian:~# apt-get --yes install sysstat

b) On Fedora, CentOS, RHEL etc. install is with yum:

[root@centos ~]# yum -y install sysstat
c) On Slackware Linux sysstat package which contains iostat is installed by default. 

d) In FreeBSD, there is no need for installation of any external package as iostat is part of the BSD world (bundle commands).
I should mention bsd iostat and Linux's iostat commands are not the same and hence there use to track down hard disk bottlenecks differs a bit, however the general logic of use is very similar as with most tools in BSD and Linux.

2. Checking a server hard disk for i/o disk bottlenecks on G* / Linux

Once having the sysstat installed on G* / Linux systems, the iostat command will be added in /usr/bin/iostat
a) To check what is the hard disk read writes per second (in megabytes) use:

debian:~# /usr/bin/iostat -m
Linux 2.6.32-5-amd64 (debian) 03/27/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
15.34 0.36 2.76 2.66 0.00 78.88
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 63.89 0.48 8.20 6730223 115541235
sdb 64.12 0.44 8.23 6244683 116039483
md0 2118.70 0.22 8.19 3041643 115528074

In the above output the server, where I issue the command is using sda and sdb configured in software RAID 1 array visible in the output as (md0)

The output of iostat should already be easily to read, for anyone who didn't used the tool here is a few lines explanation of the columns:

The %user 15.34 meaning is that 15.34 out of 100% possible i/o load is generad by system level read/write operations.
%nice – >Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%iowait – just like the top command idle it shows the idle time when the system didn't have an outstanding disk I/O requests.
%steal – show percentage in time spent in time wait of CPU or virtual CPUs to service another virtual processor (high numbers of disk is sure sign for i/o problem).
%idle – almost the same as meaning to %iowait
tps – HDD transactions per second
MB_read/s (column) – shows the actual Disk reads in Mbytes at the time of issuing iostat
MB_wrtn/s – displays the writes p/s at the time of iostat invocation
MB_read – shows the hard disk read operations in megabytes, since the server boot 'till moment of invocation of iostat
MB_wrtn – gives the number of Megabytes written on HDD since the last server boot filesystem mount

The reason why the Read / Write values for sda and sdb are similar in this example output is because my disks are configured in software RAID1 (mirror)

The above iostat output reveals in my specific case the server is experiencing mostly Disk writes (observable in the high MB_wrtn/s 8.19 md0 in the above sample output).

It also reveals, the I/O reads experienced on that server hard disk are mostly generated as a system (user level load) – see (%user 15.34 and md0 2118.70).

For all those not familiar with system also called user / level load, this is all kind of load which is generated by running programs on the server – (any kind of load not generated by the Linux kernel or loaded kernel modules).

b) To periodically keep an eye on HDD i/o operations with iostat, there are two ways:

– Use watch in conjunction with iostat;

[root@centos ~]# watch "/usr/bin/iostat -m"
Every 2.0s: iostat -m Tue Mar 27 11:00:30 2012
Linux 2.6.32-5-amd64 (centos) 03/27/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
15.34 0.36 2.76 2.66 0.00 78.88
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 63.89 0.48 8.20 6730255 115574152
sdb 64.12 0.44 8.23 6244718 116072400
md0 2118.94 0.22 8.20 3041710 115560990
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 55.00 0.01 25.75 0 51
sdb 52.50 0.00 24.75 0 49
md0 34661.00 0.01 135.38 0 270

Even though watch use and -d might appear like identical, they're not watch does refresh the screen, executing instruction similar to the clear command which clears screen on every 2 seconds, so the output looks like the top command refresh, while passing the -d 2 will output the iostat command output on every 2 secs in a row so all the data is visualized on the screen. Hence -d 2 in cases, where more thorough debug is necessery is better. However for a quick routine view watch + iostat is great too.

c) Outputting extra information for HDD input/output operations;

root@debian:~# iostat -x
Linux 2.6.32-5-amd64 (debian) 03/27/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
15.34 0.36 2.76 2.66 0.00 78.88
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 4.22 2047.33 12.01 51.88 977.44 16785.96 278.03 0.28 4.35 3.87 24.72
sdb 3.80 2047.61 11.97 52.15 906.93 16858.32 277.05 0.03 5.25 3.87 24.84
md0 0.00 0.00 20.72 2098.28 441.75 16784.05 8.13 0.00 0.00 0.00 0.00

This command will output extended useful Hard Disk info like;
r/s – number of read requests issued per second
w/s – number of write requests issued per second
rsec/s – numbers of sector reads per second
b>wsec/s – number of sectors wrote per second
etc. etc.

Most of ppl will never need to use this, but it is good to know it exists.

3. Tracking read / write (i/o) hard disk bottlenecks on FreeBSD

BSD's iostat is a bit different in terms of output and arguments.

a) Here is most basic use:

freebsd# /usr/sbin/iostat
tty ad0 cpu
tin tout KB/t tps MB/s us ni sy in id
1 561 45.18 44 1.95 14 0 5 0 82

b) Periodic watch of hdd i/o operations;

freebsd# iostat -c 10
tty ad0 cpu
tin tout KB/t tps MB/s us ni sy in id
1 562 45.19 44 1.95 14 0 5 0 82
0 307 51.96 113 5.73 44 0 24 0 32
0 234 58.12 98 5.56 16 0 7 0 77
0 43 0.00 0 0.00 1 0 0 0 99
0 485 0.00 0 0.00 2 0 0 0 98
0 43 0.00 0 0.00 0 0 1 0 99
0 43 0.00 0 0.00 0 0 0 0 100
...

As you see in the output, there is information like in the columns tty, tin, tout which is a bit hard to comprehend.
Thanksfully the tool has an option to print out only more essential i/o information:

freebsd# iostat -d -c 10
ad0
KB/t tps MB/s
45.19 44 1.95
58.12 97 5.52
54.81 108 5.78
0.00 0 0.00
0.00 0 0.00
0.00 0 0.00
20.48 25 0.50

The output info is quite self-explanatory.

Displaying a number of iostat values for hard disk reads can be also achieved by omitting -c option with:

freebsd# iostat -d 1 10
...
Tracking a specific hard disk partiotion with iostat is done with:

freebsd# iostat -n /dev/ad0s1a
tty cpu
tin tout us ni sy in id
1 577 14 0 5 0 81
 c) Getting Hard disk read/write information with gstat

gstat is a FreeBSD tool to print statistics for GEOM disks. Its default behaviour is to refresh the screen in a similar fashion like top command, so its great for people who would like to periodically check all attached system hard disk and storage devices:

freebsd# gstat
dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 10 0 0 0.0 10 260 2.6 15.6| ad0
0 10 0 0 0.0 10 260 2.6 11.4| ad0s1
0 10 0 0 0.0 10 260 2.8 12.5| ad0s1a
0 0 0 0 0.0 0 0 0.0 20.0| ad0s1b
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1c
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1d
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1e
0 0 0 0 0.0 0 0 0.0 0.0| acd0

It even has colors if your tty supports colors 🙂

Another useful tool in debugging the culprit of excessive hdd I/O operations is procstat command:

Here is a sample procstat run to track (httpd) one of my processes imposing i/o hdd load:

freebsd# procstat -f 50404
PID COMM FD T V FLAGS REF OFFSET PRO NAME
50404 httpd cwd v d -------- - - - /
50404 httpd root v d -------- - - - /
50404 httpd 0 v c r------- 56 0 - -
50404 httpd 1 v c -w------ 56 0 - -
50404 httpd 2 v r -wa----- 56 75581 - /var/log/httpd-error.log
50404 httpd 3 s - rw------ 105 0 TCP ::.80 ::.0
50404 httpd 4 p - rw---n-- 56 0 - -
50404 httpd 5 p - rw------ 56 0 - -
50404 httpd 6 v r -wa----- 56 25161132 - /var/log/httpd-access.log
50404 httpd 7 v r rw------ 56 0 - /tmp/apr8QUOUW
50404 httpd 8 v r -w------ 56 0 - /var/run/accept.lock.49588
50404 httpd 9 v r -w------ 1 0 - /var/run/accept.lock.49588
50404 httpd 10 v r -w------ 1 0 - /tmp/apr8QUOUW
50404 httpd 11 ? - -------- 2 0 - -

Btw fstat is sometimes helpful in identifying the number of open files and trying to estimate which ones are putting the hdd load.
Hope this info helps someone. If you know better ways to track hdd excessive loads on Linux / BSD pls share 'em pls.
 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in FreeBSD, Linux, System Administration | 1 Comment »

Easy way to look for irregularities and problems in log files / Facilitate reading log files on GNU / Linux and FreeBSD

Thursday, November 24th, 2011

LogWatch logo picture check Logcheck Linux BSD look for irregularities in log files

As a System Administrator I need to check daily the log files produced on various GNU / Linux distributions or FreeBSD. This can sometimes take too much time if the old fashioned way using the normal system tools cat, less and tail etc. is used.

Reading logs one by one eats too much of my time and often as logs are reviewed in a hurry some crucial system irregularities, failed ssh or POP3 / Imap logins, filling disk spaces etc. are missed.

Therefore I decided to implement automated log parsing programs which will summary and give me an overview (helicopter view) on what were the system activities from the previous day (24h) until the moment I logged the system and issued the log analyzer program.
There are plenty of programs available out there that does “wide scale” log analysis, however there are two applications which on most GNU / Linux and BSD systems had become a de-facto standard programs to scan system log files for interesting lines.

These are:
 

  • 1. logwatch – system log analyzer and reporter
  • 2. logcheck – program to scan system log files for interesting lines

1. logwatch is by default installed on most of the Redhat based Linux systems (Fedora, RHEL, CentOS etc.). On Debian distributions and as far as I know (Ubuntu) and the other deb based distros logwatch is not installed by default. Most of the servers I manage these days are running Debian GNU / Linux so, to use logwatch I needed to install it from the available repository package, e.g.:

debian:~# apt-get install logwatch
...

logwatch is written in perl and with some big files to analyze, parsing them might take hell a lot of time. It does use a bunch of configuration scripts which defines how logwatch should read and parse the various services logwatch support by default. These conf scripts are also easily extensible, so if one has to analyze some undefined service in the conf files he can easily come up with a new conf script that will support the service/daemon of choice.Using logwatch is very easy, to get an overview about server system activity invoke the logwatch command:

debian:~# logwatch
################### Logwatch 7.3.6+cvs20080702-debian (07/02/08) ####################
Processing Initiated: Thu Nov 24 05:22:07 2011
Date Range Processed: yesterday
( 2011-Nov-23 )
Period is day.
Detail Level of Output: 0
Type of Output/Format: stdout / text
Logfiles for Host: debian
 ################################################# 

——————— dpkg status changes Begin ————- 

Upgraded:
libfreetype6 2.3.7-2+lenny7 => 2.3.7-2+lenny8
libfreetype6-dev 2.3.7-2+lenny7 => 2.3.7-2+lenny8

———————- dpkg status changes End ————————-

——————— httpd Begin ————————

Requests with error response codes
400 Bad Request
HTTP/1.1: 2 Time(s)
admin/scripts/setup.php: 2 Time(s)
401 Unauthorized
…
…
———————- vpopmail End ————————-

——————— Disk Space Begin ————————

Filesystem Size Used Avail Use% Mounted on
/dev/md0 222G 58G 154G 28% /

———————- Disk Space End ————————-

###################### Logwatch End #########################

The execution might take up from 10 to 20 seconds up to 10 or 20 minutes depending on the log files size and the CPU / RAM hardware on the machine where /var/log/… logs will be analyzed.

logwatch output can be easily mailed to a custom mail address using a crontab if the server runs a properly configured SMTP server. Using a cron like:

00 5 * * * /usr/sbin/logwatch | mail -s "$(hostname) log files for $(date)"

Here is time to make a note that logwatch is ported also to FreeBSD and is available from BSD’s port tree, from a port with path:

/usr/ports/security/logcheck

2. logcheck is another handy program, which does very similar job to logwatch . The “interesting” information it returns is a bit less than compared to logwatch

The good thing about logcheck is that by default it is made to mail every 1 hour a brief data summary which might be of an interest to the sys admin.
Logcheck is available for install on RedHat distros via yum and has existing package for Debian as well as a port for FreeBSD under the port location /usr/ports/security/logcheck

To install on logcheck on Debian:

debian:~# apt-get install logcheck
...

After installation I found it wise to change the default mailing time from each and every hour to just once per day to prevent my email from overfilling with “useless” mails.

This is done by editting the default cron tab installed by the package located in /etc/cron.d/logcheck

The default file looks like so:

# /etc/cron.d/logcheck: crontab entries for the logcheck package
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
@reboot logcheck if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R; fi
2 * * * * logcheck if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi
# EOF

To change it run only once per day its content should looks something like:

# /etc/cron.d/logcheck: crontab entries for the logcheck package
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
@reboot logcheck if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R; fi
2 5 * * * logcheck if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck; fi
# EOF

Altering it that way the log summary interesting info analysis will be sent on mail every day in 05:02 a.m.
Changing the default email logcheck will ship its log analyzer report emails on deb based distros is done via editting the file:

/etc/logcheck/logcheck.conf

And changing the SENDMAILTO=”” variable to point to the appropriate admin email email addr.
 

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in FreeBSD, Linux, System Administration, Various | 1 Comment »

How to make Video from your Linux Desktop with xvidcap / Capture desktop output in a video on Linux

Wednesday, April 6th, 2011

If you have wondered on how to create videos aiming at manuals on how you do certain stuff on Linux, let’s say related to programming or system administration.
Then you should definitely check out

xvidcap

Below is the package description as taken from apt-cache show xvidcap

A screen capture enabling you to capture videos off your X-Window desktop
for illustration or documentation purposes. It is intended to be a
standards-based alternative to tools like Lotus ScreenCam.

On Debian based Linux systems (e.g. Debian Ubuntu) xvidcap is available straight from the package repositories. To install and test it you can straight issue:

linux:~# apt-get install xvidcap
...

To start using xvidcap, either by starting it with alt+f2 in gnome or straight launch it from the applications menu via:

Applications -> Sound & Video -> xvidcap

Here is how the xvidcap program looks like right after you start it;
xvidcap screenshot main menu

As you see in the screenshot xvidcap’s menu interface is extraordinary simple.

As you see it only has a stop, pause, rec, back and forward buttons, a capture selector and movie editor.
Pitily xvidcap does not support music capturing, but at least for me that’s not such an issue.

If you click over the field test-0000.mpeg[0000] with your last mouse button, you will notice a drop down menu with an option for preferences of xvidcap.

Take the time to play with the preferences, since there are quite a few of them.

The most important preference that you might like to straightly adjust in my view is in the:

Preferences -> Multi-Frame tab -> File Name:

The default file that xvidcap uses to store it’s content files as you will see in the preferences is utest-%04d.mpeg

If you want to change the type of the output file format to let’s say flv change the File Name: value to utest-%04d.flv
Next time you record with xvidcap, you will have the file stored in flv format.

The red lines which you see in the above screenshot is the capture area, you will have to also tune the screen capture area before you can proceed with recording a video from your desktop.

The way to capture your Desktop in fullscreen is a bit unusual, you first need to mark up all your visible Desktop and before that you will have to select from xvidcap’s preferences from:

Preferences -> General -> Minimize to System Tray

By selecting this option each time you press the xvidcap’s record button the xvidcap’s controller interface will be minimized to tray and capturing the video of the region previously selected with the capture selector will start up.

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Linux, Linux and FreeBSD Desktop, Linux Audio & Video | 7 Comments »