Posts Tagged ‘hdd’

Best Windows tools to Test (Benchmark) Hard Drives, SSD Drives and RAID Storage Controllers

Wednesday, April 23rd, 2014

atto-windows-hard-disk-benchmark-freeware-tool-screenshot-check-hard-disk-speed-windows
Disk Benchmarking is very useful for people involved in Graphic Design, 3D modelling, system admins  and anyone willing to squeeze maximum of his PC hardware.

If you want to do some benchmarking on newly built Windows server targetting Hard Disk performance, just bought a new hard SSD (Solid State Drives) and you want to test how well Hard Drive I/O operations behave or you want to see a regular HDD benchmarking of group of MS Windows PCs and plan hardware optiomization, check out ATTO Disk Benchmark.

So why exactly ATTO Benchmark? – Cause it is one of the best Windows Free Benchmark tools on the internet.

ATTO is a widely-accepted Disk Benchmark freeware utility to help measure storage system performance. ATTO though being freeware is among top tools utilized in industry. It is very useful in comparing different Hard Disk vendors speed, measure Windows storage systems performance with various transfer sizes and test lengths for reads and writes.

ATTO Disk Benchmark is used by manufacturers of Hardware RAID controllers, its precious tool to test Windows storage controllers, host bus adapters (HBAs).

Here is ATTO Benchmark tool specifications (quote from their webstie):
 

  • Transfer sizes from 512KB to 8MB
  • Transfer lengths from 64KB to 2GB
  • Support for overlapped I/O
  • Supports a variety of queue depths
  • I/O comparisons with various test patterns
  • Timed mode allows continuous testing
  • Non-destructive performance measurement on formatted drives
  • Transfer sizes from 512KB to 8MB
  • Transfer lengths from 64KB to 2GB
  • Support for overlapped I/O
  • Supports a variety of queue depths
  • I/O comparisons with various test patterns
  • Timed mode allows continuous testing
  • Non-destructive performance measurement on formatted drives
  • – See more at: http://www.attotech.com/disk-benchmark/#sthash.rRlgSTOE.dpuf

Here is mirrored latest version of ATTO Disk for Download. Once you get your HDD statistics you will probably want to compare to other people results. On  TomsHardware's world famous Hardware geek site there are plenty of Hard Drives performance Charts

Of course there are other GUI alternatives to ATTO Benchmark one historically famous is NBench

NBench

nbench_benchmark_windows_hard-drive-cpu-and-memory

Nbench is nice little benchmarking program for Windows NT. Nbench reports the following components of performance:

CPU speed: integer and floating operations/sec
L1 and L2 cache speeds: MB/sec
main memory speed: MB/sec
disk read and write speeds: MB/sec

SMP systems and multi-tasking OS efficiency can be tested using up to 20 separate threads of execution.

For Console Geeks or Windows server admins there are also some ports of famous *NIX Hard Disk Benchmarking tools:

NTiogen

NTiogen benchmark was written by Symbios Logic, It's Windows NT port of their popular UNIX benchmark IOGEN. NTIOGEN is the parent processes that spawns the specified number of IOGEN processes that actually do the I/O.
The program will display as output the number of processes, the average response time, the number of I/O operations per second, and the number of KBytes per second. You can download mirror copy of Ntiogen here


There are plenty of other GUI and Console HDD Benchmarking Win Tools, i.e.:

IOMeter (ex-developed by Intel and now abandoned available as open source available on SourceForge)

iometer-benchmark-disk-storage-speed-windows
 

Bench32 – Comprehensive benchmark that measures overall system performance under Windows NT or Windows 95, now obsolete not developed anymore abandoned by producer company.

ThreadMark32 – capable of bench (ex developed and supported by ADAPTEC) but also already unsupported

IOZone – filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems.
 

N! B! Important note to make here is above suggested tools will provide you more realistic results than the proprietary vendor tools shipped by your hardware vendor. Using proprietary software produced by a single vendor makes it impossible to analyze and compare different hardwares, above HDD benchmarking tools are for "open systems", e.g. nomatter what the hardware producer is produced results can be checked against each other.
Another thing to consider is even though if you use any of above tools to test and compare two storage devices still results will be partially imaginary, its always best to conduct tests in Real Working Application Environments. If you're planning to launch a new services structure always test it first and don't rely on preliminary returned soft benchmarks.

if you know some other useful benchmarking software i'm missing please share.

Share this on

Checking I/O Hard Disk (Overhead) Read / Write operations on Microsoft Windows 7 – Resource Monitor

Friday, July 26th, 2013

I mainly have to deal with Linux servers. Today however I had to check for problems Microsoft Windows 7 server. The machine looked Okay but was reading from hard drive all the time. Hence I needed to check what is the Hard disk approximate read / write speed per second. I know on lInux tracking i/o hard disk server bottlenecks is done with iostat or dstat.

However I never did that on Windows, so I had to learn it by experience. Its actually pretty easy and you don't even need to install external program to see read / write hdd speed operations. Windows 7 is bundled with a Program called Resource Manager. Running Resource Manager's easiest method is from Windows Task Manager, i.e.:

Windows 7 Task Manager processes performance tab - how to check hard drive bottleneck windows server
Press Ctrl + Alt + Del (Choose Start Task Manager) and from Task Manager click on Resource Monitor Button.
Immediately resource Monitor pops up and selecting the Disk tab priovides information on HDD Read / Write speed per sec. Using Resource Monitor, you can quickly also see which process is creating the most HDD overhead for server.

microsoft windows 7 resource monitor screenshot

windows resource monitor disk tab hard disk show read write speed win7

Though I'm not Microsoft fan, I should admit Resource Manager does a great job.

 

Share this on

How to mount NTFS Windows XP filesystem on FreeBSD, NetBSD, OpenBSD

Friday, May 11th, 2012

Mounting NTFS hdd partitions on FreeBSD logo picture

A friend of mine bring home a Seagate External Hard Disk Drive using USB 3 as a communication media. I needed to attach the hard disk to my FreeBSD router to transfer him some data, the External HDD is formatted to use NTFS as a main file partition and hence to make the file transfers I had to somehow mount the NTFS partition on the HDD.

FreeBSD and other BSDs, just like Linux does not have embedded NTFS file system mount support.
In order to add an external write support for the plugged hdd NTFS I looked in the ports tree:

freebsd# cd /usr/ports
freebsd# make search name='ntfs'
Port: fusefs-ntfs-2010.10.2
Path: /usr/ports/sysutils/fusefs-ntfs
Info: Mount NTFS partitions (read/write) and disk images
Maint: ports@FreeBSD.org
B-deps: fusefs-libs-2.7.4 libiconv-1.13.1_1 libtool-2.4 libublio-20070103 pkg-config-0.25_1
R-deps: fusefs-kmod-0.3.9.p1.20080208_7 fusefs-libs-2.7.4 libiconv-1.13.1_1 libublio-20070103 pkg-config-0.25_1
WWW: http://www.tuxera.com/community/

Port: ntfsprogs-2.0.0_1
Path: /usr/ports/sysutils/ntfsprogs
Info: Utilities and library to manipulate NTFS partitions
Maint: ports@FreeBSD.org
B-deps: fusefs-libs-2.7.4 libiconv-1.13.1_1 libublio-20070103 pkg-config-0.25_1
R-deps: libublio-20070103 pkg-config-0.25_1
WWW: http://www.linux-ntfs.org/
freebs# cd sysutils/fusefs-ntfs/
freebsd# ls
Makefile distinfo files/ pkg-descr pkg-plist
freebsd# cat pkg-descr
The ntfs-3g driver is an open source, freely available read/write NTFS
driver, which provides safe and fast handling of the Windows XP, Windows
Server 2003 and Windows 2000 filesystems. Almost the full POSIX filesystem
functionality is supported, the major exceptions are changing the file
ownerships and the access rights.
WWW: http://www.tuxera.com/community/

Using ntfs-3g I managed to succeed mounting the NTFS on my old PC running FreeBSD ver. 7_2

1. Installing fuserfs-ntfs support on BSD

Before I can use ntfs-3g, to mount the paritition, I had to install fuserfs-ntfs bsd port, with:

freebsd# cd /usr/ports/sysutils/fusefs-ntfs
freebsd# make install clean
.....

I was curious if ntfsprogs provides other utilities to do the ntfs mount but whilst trying to install it I realized it is already installed as a dependency package to fusefs-ntfs.

fusefs-ntfs package provides a number of utilities for creating, mounting, fixing and doing various manipulations with Microsoft NTFS filesystems.

Here is a list of all the executable utilities helpful in NTFS fs management:

freebsd# pkg_info -L fusefs-ntfs\* | grep -E "/bin/|/sbin|README"
/usr/local/bin/lowntfs-3g
/usr/local/bin/ntfs-3g
/usr/local/bin/ntfs-3g.probe
/usr/local/bin/ntfs-3g.secaudit
/usr/local/bin/ntfs-3g.usermap
/usr/local/bin/ntfscat
/usr/local/bin/ntfscluster
/usr/local/bin/ntfscmp
/usr/local/bin/ntfsfix
/usr/local/bin/ntfsinfo
/usr/local/bin/ntfsls
/usr/local/sbin/mkntfs
/usr/local/sbin/ntfsclone
/usr/local/sbin/ntfscp
/usr/local/sbin/ntfslabel
/usr/local/sbin/ntfsresize
/usr/local/sbin/ntfsundelete
/usr/local/share/doc/ntfs-3g/README
/usr/local/share/doc/ntfs-3g/README.FreeBSD

The README and README.FreeBSD are wonderful, reading for those who want to get more in depth knowledge on using the up-listed utilities.

One utility, worthy to mention, I have used in the past is ntfsfix. ntfsfix resolve issues with NTFS partitions which were not unmounted on system shutdown (electricity outage), system hang up etc.

2. Start fusefs (ntfs) and configure it to auto load on system boot

Once fuserfs-ntfs is installed, if its necessery ntfs fs mounts to be permanently supported on the BSD system add fusefs_enable="YES" to /etc/rc.conf(the FreeBSD services auto load conf).

freebsd# echo 'fusefs_enable="YES"' >> /etc/rc.conf

One note to make here is that you need to have also dbus_enable="YES" and hald_enable="YES" in /etc/rc.conf, not having this two in rc.conf will prevent fusefs to start properly. Do a quick grep to make sure this two variables are enabled:

Afterwards fsusefs load up script should be run:

freebsd# /usr/local/etc/rc.d/fusefs start
Starting fusefs.

Another alternative way to load ntfs support on the BSD host is to directly load fuse.ko kernel module:

freebsd# /sbin/kldload fuse.ko

3. Mounting the NTFS partition

In my case, the Seagate hard drive was detected as da0, where the NTFS partition was detected as s1 (da0s1):

freebsd# dmesg|grep -i da0
da0 at umass-sim0 bus 0 target 0 lun 0
da0: Fixed Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 953869MB (1953525164 512 byte sectors: 255H 63S/T 121601C)br />GEOM_LABEL: Label for provider da0s1 is ntfs/Expansion Drive.
GEOM_LABEL: Label for provider da0s1 is ntfs/Expansion Drive.

Therefore further to mount it one can use mount_ntfs (to quickly mount in read only mode) or ntfs-3g for (read / write mode):

If you need to just quickly mount a disk drive to copy some data and umount it with no need for writting to the NTFS partition do;

freebsd# /sbin/mount_ntfs /dev/ad0s1 /mnt/disk

Note that mount_ntfs command is a native BSD command and have nothing to do with ntfs-3g. Therefore using it to mount NTFS is not the same as mounting it via ntfs-3g cmd

freebsd# /usr/local/bin/ntfs-3g -o rw /dev/da0s1 /mnt/disk/

Something, I've noticed while using ntfs-3g is, it fails to properly exit even when the ntfs-3g shell execution is over:

freebsd# ps ax |grep -i ntfs|grep -v grep
18892 ?? Is 0:00.00 /usr/local/bin/ntfs-3g -o rw /dev/da0s1 /mnt/disk/

I dunno if this is some kind of ntfs-3g bug or feature specific to all versions of FreeBSD or it is something local to FBSD 7.2

Thought ntfs-3g, keeps appearing in process list, praise God as of time of writting NTFS support on FreeBSD prooved to be stable.
Read / Write disk operations to the NTFS I tested it with works great. Just about 5 years ago I still remember write mode was still experimental. Now it seems NTFS mounts can be used with no hassle even on production machines.

4. Auto mounting NTFS partition on FreeBSD system boot

There are two approaches towards 'the problem' I can think of.
The better way to auto mount on boot (in my view) is through /etc/fstab use

If /etc/fstab + ntfs-3g is to be used, you will however change the default /sbin/mount_ntfs command to point to /usr/local/bin/ntfs-3g, i.e.:

freebsd# mv /sbin/mount_ntfs /sbin/mount_ntfs.orig
freebsd# ln -s /usr/local/bin/ntfs-3g /sbin/mount_ntfs

Then to mount /dev/da0s1 via /etc/fstab add line:

/dev/ad0s1 /mnt/disk ntfs rw,late 0 0

To not bother with text editor run:

freebsd# echo '/dev/ad0s1 /mnt/disk ntfs rw,late 0 0' >> /etc/fstab

I've red in posts in freebsd forums, there is also a way to use ntfs-3g for mounting partitions, without substituting the original bsd /sbin/mount_ntfs, the exact commands suggested to be used with no need to prior mv /sbin/mount_ntfs to /sbin/mount_ntfs.orig and link it to ntfs is:

/dev/ad0s1 /disk ntfs rw,mountprog=/usr/local/bin/ntfs-3g,late 0 0

For any other NTFS partitions, for instance /dev/ad0s2, /dev/ad2s1 etc. simply change the parititon name and mount points.

The second alternative to adding the NTFS to auto mount is through /etc/rc.local. /etc/rc.local content will be executed very late in system boot. :

echo '/usr/local/bin/ntfs-3g -o rw /dev/da0s1' >> /etc/rc.local

One disadvanage of using /etc/rc.local for mounting the partition is the hanging ntfs-3g in proc list:

freebsd# ps ax |grep -i ntfs|grep -v grep
18892 ?? Is 0:00.00 /usr/local/bin/ntfs-3g -o rw /dev/da0s1 /mnt/disk/

Though, I haven't tested it yet. Using the same methodology should be perfectly working on PC-BSD, DragonFlyBSD, NetBSD and OpenBSD.
I will be glad if someone who runs any of the other BSDs can confirm, following this instructions works fine on these BSDs too.

Share this on

Text mode (console) browsing with tabs with Elinks / Text browsers – (lynx, elinks, links and w3m) useful HTTP debugging tools for Linux and FreeBSD servers

Friday, April 27th, 2012

The last days, I'm starting to think the GUI use is making me brainless so I'm getting back to my old habits of using console.
I still remember with a grain of nostalgy how much more efficient I used to be when the way to interact with my computer was primary in text mode console.
Actually, I'm starting to get this idea the more new a software is the more inefficient it makes your use of computer, not to mention the hardware resources required by newer software is constantly increasing.

With this said, I started occasionally browsing again like in the old days by using links text browser.
In the old days I mostly used lynx and its more advanced "brother" text browser links.
The main difference between lynx and links is that lynx does not have any support for the terrible "javascript", whether links supports most of the Javascript ver 2.
Also links and has a midnight commander like pull down menus on the screen top, – handy for people who prefer some more interactivity.

In the past I remember I used also to browse graphically in normal consoles (ttys) with a hacked version of links calledTThere is also a variation of linksxlinks suitable for people who would like to have graphical browser in console (ttys).

I used xlinks quite heavily in the past, when I have slower computer P166Mhz with 64MB of memory 2.5 GB HDD (What a times boy what a times) .
Maybe when I have time I will install it on my PC and start using it again like in the old days to boost my computer use efficiency…
I remember the only major xlinks downside was it doesn't included support for Adobe flash (though this is due to the bad non-free software nature of Adobe lack of proper support for free software and not a failure of xlinks developers. Anyways for me this wasn't a big trouble since, ex Macromedia (Adobe) Flash support is not something essential for most of my work…

links2 is actually the naming of links version 2. elinks emerged later (if I remember correctly, as fork project of links).
elinks difference with links constitutes in this it supports tabbed browsing as well as colors (links browser displays results monochrome).

Having a tabbed browsing support in tty console is a great thing…
I personally belive text browsing if properly used can in many ways outbeat, graphic browsing in terms of performance and time spend to obtain data. I'm convinced text browsing is superior for two reasons:
1. with text there is way less elements to obstruct your attention.
– No graphical annoying flash banners, no annoying taking the attention pictures

2. Navigating in web pages using the keyboard is more efficient than mouse
– Using keyboard shorcuts is always quicker than mouse, generally keboard has always been a quicker way to access computer commands.

Another reason to use text browsing is, it is mostly the text part of a page that matters, most of the pages that provide images to better explain a topic are bloated (this is my personal view though, i'm sure designer guys will argue me :D).
Here is a screenshot of a my links text browser in action, I'm sorry the image is a bit unreadable, but after taking a screenshot of the console and resizing it with GIMP this is what I got …

Links text console browser screenshot with 2 tabs opened Debian GNU / Linux

For all those new to Linux who didn't tried text browsing yet and for those interested in computer history, I suggest you install and give a try to following text browsers:
 

  • lynx
  • (Supports colorful text console text browsing)
    lynx text console browser Debian Squeeze GNU / Linux Screenshot

  • links
  • Links www text console browser screenshot on Debian Linux

  • elinks
  • (Supports colors filled text browsing and tabs)
    elinks opened duckduckgo.com google alternative search engine in mlterm terminal Debian Linux

  • w3m
  • w3m one of the oldest text console browsers screenshot Debian Linux Squeeze 6.2

By the way having the 4 text browsers is very useful for debugging purposes for system administrators too, so in any case I think this 4 web browsers are absoutely required software for newly installed GNU / Linux or BSD* based servers.

For Debian and the derivatives Linux distributions, the 4 browsers are available as deb packages, so install them with following apt 1 liner:
 

debian:~# apt-get –yes install w3m elinks links lynx
….

FreeBSD users can install the browsers using, cmd:
 

freebsd# cd /usr/ports/www/w3mfreebsd# make install clean
….
freebsd# cd /usr/ports/www/elinksfreebsd# make install clean
….
freebsd# cd /usr/ports/www/linksfreebsd# make install clean
….
freebsd# cd /usr/ports/www/lynxfreebsd# make install clean
….

In links using the tabs functionality appeared, somewhere near the 2001 or 2000 (at least that was the first time I saw links with tabbed browsing enabled). My first time to saw links support opening multiple pages within the same screen under tabs was on Redhat Linux 9

Opening multiple pages in tabs in the text browser is done by pressing the t key and typing in the desired URL to open isnide.
For more than 2 tabs, again t has to be pressed and same procedure goes on and on.
It was pretty hard for me to figure out how I can do a text browsing with tabs, though I found a way to open new tabs it took me some 10 minutes in pondering how to switch between the new opened links browser tabs.

Hence, I thought it would be helpful to mention here how tabs can be switched in links text browser. Actually it turned it is pretty easy to Switch tabs tabs back and foward.

1 tab to move backwards is done with < (key), wheter switching one tab forward is done with the > key.

On UK and US qwerty keyboards alignment the movement a tab backward and forward is done with holding shift and pressing < onwards holding both keys simultaneously and analogously with pressing shift + >
 

Share this on

Tracking I/O hard disk server bottlenecks with iostat on GNU / Linux and FreeBSD

Tuesday, March 27th, 2012

Hard disk overhead tracking on Linux and FreeBSD with iostat

I've earlier wrote an article How to find which processes are causing hard disk i/o overhead on Linux there I explained very rawly few tools which can be used to benchmark hard disk read / write operations. My prior article accent was on iotop and dstat and it just mentioned of iostat. Therefore I've wrote this short article in attempt to explain a bit more thoroughfully on how iostat can be used to track problems with excessive server I/O read/writes.

Here is the command man page description;
iostatReport Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems

I will further proceed with few words on how iostat can be installed on various Linux distros, then point at few most common scenarious of use and a short explanation on the meaning of each of the command outputs.

1. Installing iostat on Linux

iostat is a swiss army knife of finding a server hard disk bottlenecks. Though it is a must have tool in the admin outfut, most of Linux distributions will not have iostat installed by default.
To have it on your server, you will need to install sysstat package:

a) On Debian / Ubuntu and other Debian GNU / Linux derivatives to install sysstat:

debian:~# apt-get --yes install sysstat

b) On Fedora, CentOS, RHEL etc. install is with yum:

[root@centos ~]# yum -y install sysstat

c) On Slackware Linux sysstat package which contains iostat is installed by default. 

d) In FreeBSD, there is no need for installation of any external package as iostat is part of the BSD world (bundle commands).
I should mention bsd iostat and Linux's iostat commands are not the same and hence there use to track down hard disk bottlenecks differs a bit, however the general logic of use is very similar as with most tools in BSD and Linux.

2. Checking a server hard disk for i/o disk bottlenecks on G* / Linux

Once having the sysstat installed on G* / Linux systems, the iostat command will be added in /usr/bin/iostat
a) To check what is the hard disk read writes per second (in megabytes) use:

debian:~# /usr/bin/iostat -m
Linux 2.6.32-5-amd64 (debian) 03/27/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
15.34 0.36 2.76 2.66 0.00 78.88
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 63.89 0.48 8.20 6730223 115541235
sdb 64.12 0.44 8.23 6244683 116039483
md0 2118.70 0.22 8.19 3041643 115528074

In the above output the server, where I issue the command is using sda and sdb configured in software RAID 1 array visible in the output as (md0)

The output of iostat should already be easily to read, for anyone who didn't used the tool here is a few lines explanation of the columns:

The %user 15.34 meaning is that 15.34 out of 100% possible i/o load is generad by system level read/write operations.
%nice – >Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%iowait – just like the top command idle it shows the idle time when the system didn't have an outstanding disk I/O requests.
%steal – show percentage in time spent in time wait of CPU or virtual CPUs to service another virtual processor (high numbers of disk is sure sign for i/o problem).
%idle – almost the same as meaning to %iowait
tps – HDD transactions per second
MB_read/s (column) – shows the actual Disk reads in Mbytes at the time of issuing iostat
MB_wrtn/s – displays the writes p/s at the time of iostat invocation
MB_read – shows the hard disk read operations in megabytes, since the server boot 'till moment of invocation of iostat
MB_wrtn – gives the number of Megabytes written on HDD since the last server boot filesystem mount

The reason why the Read / Write values for sda and sdb are similar in this example output is because my disks are configured in software RAID1 (mirror)

The above iostat output reveals in my specific case the server is experiencing mostly Disk writes (observable in the high MB_wrtn/s 8.19 md0 in the above sample output).

It also reveals, the I/O reads experienced on that server hard disk are mostly generated as a system (user level load) – see (%user 15.34 and md0 2118.70).

For all those not familiar with system also called user / level load, this is all kind of load which is generated by running programs on the server – (any kind of load not generated by the Linux kernel or loaded kernel modules).

b) To periodically keep an eye on HDD i/o operations with iostat, there are two ways:

– Use watch in conjunction with iostat;

[root@centos ~]# watch "/usr/bin/iostat -m"
Every 2.0s: iostat -m Tue Mar 27 11:00:30 2012
Linux 2.6.32-5-amd64 (centos) 03/27/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
15.34 0.36 2.76 2.66 0.00 78.88
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 63.89 0.48 8.20 6730255 115574152
sdb 64.12 0.44 8.23 6244718 116072400
md0 2118.94 0.22 8.20 3041710 115560990
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 55.00 0.01 25.75 0 51
sdb 52.50 0.00 24.75 0 49
md0 34661.00 0.01 135.38 0 270

Even though watch use and -d might appear like identical, they're not watch does refresh the screen, executing instruction similar to the clear command which clears screen on every 2 seconds, so the output looks like the top command refresh, while passing the -d 2 will output the iostat command output on every 2 secs in a row so all the data is visualized on the screen. Hence -d 2 in cases, where more thorough debug is necessery is better. However for a quick routine view watch + iostat is great too.

c) Outputting extra information for HDD input/output operations;

root@debian:~# iostat -x
Linux 2.6.32-5-amd64 (debian) 03/27/2012 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
15.34 0.36 2.76 2.66 0.00 78.88
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 4.22 2047.33 12.01 51.88 977.44 16785.96 278.03 0.28 4.35 3.87 24.72
sdb 3.80 2047.61 11.97 52.15 906.93 16858.32 277.05 0.03 5.25 3.87 24.84
md0 0.00 0.00 20.72 2098.28 441.75 16784.05 8.13 0.00 0.00 0.00 0.00

This command will output extended useful Hard Disk info like;
r/s – number of read requests issued per second
w/s – number of write requests issued per second
rsec/s – numbers of sector reads per second
b>wsec/s – number of sectors wrote per second
etc. etc.

Most of ppl will never need to use this, but it is good to know it exists.

3. Tracking read / write (i/o) hard disk bottlenecks on FreeBSD

BSD's iostat is a bit different in terms of output and arguments.

a) Here is most basic use:

freebsd# /usr/sbin/iostat
tty ad0 cpu
tin tout KB/t tps MB/s us ni sy in id
1 561 45.18 44 1.95 14 0 5 0 82

b) Periodic watch of hdd i/o operations;

freebsd# iostat -c 10
tty ad0 cpu
tin tout KB/t tps MB/s us ni sy in id
1 562 45.19 44 1.95 14 0 5 0 82
0 307 51.96 113 5.73 44 0 24 0 32
0 234 58.12 98 5.56 16 0 7 0 77
0 43 0.00 0 0.00 1 0 0 0 99
0 485 0.00 0 0.00 2 0 0 0 98
0 43 0.00 0 0.00 0 0 1 0 99
0 43 0.00 0 0.00 0 0 0 0 100
...

As you see in the output, there is information like in the columns tty, tin, tout which is a bit hard to comprehend.
Thanksfully the tool has an option to print out only more essential i/o information:

freebsd# iostat -d -c 10
ad0
KB/t tps MB/s
45.19 44 1.95
58.12 97 5.52
54.81 108 5.78
0.00 0 0.00
0.00 0 0.00
0.00 0 0.00
20.48 25 0.50

The output info is quite self-explanatory.

Displaying a number of iostat values for hard disk reads can be also achieved by omitting -c option with:

freebsd# iostat -d 1 10
...

Tracking a specific hard disk partiotion with iostat is done with:

freebsd# iostat -n /dev/ad0s1a
tty cpu
tin tout us ni sy in id
1 577 14 0 5 0 81
c) Getting Hard disk read/write information with gstat

gstat is a FreeBSD tool to print statistics for GEOM disks. Its default behaviour is to refresh the screen in a similar fashion like top command, so its great for people who would like to periodically check all attached system hard disk and storage devices:

freebsd# gstat
dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 10 0 0 0.0 10 260 2.6 15.6| ad0
0 10 0 0 0.0 10 260 2.6 11.4| ad0s1
0 10 0 0 0.0 10 260 2.8 12.5| ad0s1a
0 0 0 0 0.0 0 0 0.0 20.0| ad0s1b
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1c
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1d
0 0 0 0 0.0 0 0 0.0 0.0| ad0s1e
0 0 0 0 0.0 0 0 0.0 0.0| acd0

It even has colors if your tty supports colors 🙂

Another useful tool in debugging the culprit of excessive hdd I/O operations is procstat command:

Here is a sample procstat run to track (httpd) one of my processes imposing i/o hdd load:

freebsd# procstat -f 50404
PID COMM FD T V FLAGS REF OFFSET PRO NAME
50404 httpd cwd v d -------- - - - /
50404 httpd root v d -------- - - - /
50404 httpd 0 v c r------- 56 0 - -
50404 httpd 1 v c -w------ 56 0 - -
50404 httpd 2 v r -wa----- 56 75581 - /var/log/httpd-error.log
50404 httpd 3 s - rw------ 105 0 TCP ::.80 ::.0
50404 httpd 4 p - rw---n-- 56 0 - -
50404 httpd 5 p - rw------ 56 0 - -
50404 httpd 6 v r -wa----- 56 25161132 - /var/log/httpd-access.log
50404 httpd 7 v r rw------ 56 0 - /tmp/apr8QUOUW
50404 httpd 8 v r -w------ 56 0 - /var/run/accept.lock.49588
50404 httpd 9 v r -w------ 1 0 - /var/run/accept.lock.49588
50404 httpd 10 v r -w------ 1 0 - /tmp/apr8QUOUW
50404 httpd 11 ? - -------- 2 0 - -

Btw fstat is sometimes helpful in identifying the number of open files and trying to estimate which ones are putting the hdd load.
Hope this info helps someone. If you know better ways to track hdd excessive loads on Linux / BSD pls share 'em pls.
 

Share this on

diskinfo Linux hdparm FreeBSD equivalent command for disk info and benchmarking

Thursday, March 8th, 2012

FreeBSD Linux hdparm equivalent is diskinfo artistic logo

On Linux there is the hdparm tool for various hard disk benchmarking and extraction of hard disk operations info.
As the Linux manual states hdparmget/set SATA/IDE device parameters

Most Linux users should already know it and might wonder if there is hdparm port or equivalent for FreeBSD, the aim of this short post is to shed some light on that.

The typical use of hdparm is like this:

linux:~# hdparm -t /dev/sda8

/dev/sda8:
Timing buffered disk reads: 76 MB in 3.03 seconds = 25.12 MB/sec
linux:~# hdparm -T /dev/sda8
/dev/sda8:
Timing cached reads: 1618 MB in 2.00 seconds = 809.49 MB/sec

The above output here is from my notebook Lenovo R61i.
If you're looking for alternative command to hdparm you should know in FreeBSD / OpenBSD / NetBSD, there is no exact hdparm equivalent command.
The somehow similar hdparm equivallent command for BSDs (FreeBSD etc.) is:
diskinfo

diskinfo is not so feature rich as linux's hdparm. It is just a simple command to show basic information for hard disk operations without no possibility to tune any hdd I/O and seek operations.
All diskinfo does is to show statistics for a hard drive seek times I/O overheads. The command takes only 3 arguments.

The most basic and classical use of the command is:

freebsd# diskinfo -t /dev/ad0s1a
/dev/ad0s1a
512 # sectorsize
20971520000 # mediasize in bytes (20G)
40960000 # mediasize in sectors
40634 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
ad:4JV48BXJs0s0 # Disk ident.

Seek times:
Full stroke: 250 iter in 3.272735 sec = 13.091 msec
Half stroke: 250 iter in 3.507849 sec = 14.031 msec
Quarter stroke: 500 iter in 9.705555 sec = 19.411 msec
Short forward: 400 iter in 2.605652 sec = 6.514 msec
Short backward: 400 iter in 4.333490 sec = 10.834 msec
Seq outer: 2048 iter in 1.150611 sec = 0.562 msec
Seq inner: 2048 iter in 0.215104 sec = 0.105 msec

Transfer rates:
outside: 102400 kbytes in 3.056943 sec = 33498 kbytes/sec
middle: 102400 kbytes in 2.696326 sec = 37978 kbytes/sec
inside: 102400 kbytes in 3.178711 sec = 32214 kbytes/sec

Another common use of diskinfo is to measure hdd I/O command overheads with -c argument:

freebsd# diskinfo -c /dev/ad0s1e
/dev/ad0s1e
512 # sectorsize
39112312320 # mediasize in bytes (36G)
76391235 # mediasize in sectors
75784 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
ad:4JV48BXJs0s4 # Disk ident.

I/O command overhead:
time to read 10MB block 1.828021 sec = 0.089 msec/sector
time to read 20480 sectors 4.435214 sec = 0.217 msec/sector
calculated command overhead = 0.127 msec/sector

Above diskinfo output is from my FreeBSD home router.

As you can see, the time to read 10MB block on my hard drive is 1.828021 (which is very high number),
this is a sign the hard disk experience too many read/writes and therefore needs to be shortly replaced with newer faster one.
diskinfo is part of the basis bsd install (bsd world). So it can be used without installing any bsd ports or binary packages.

For the purpose of stress testing hdd, or just some more detailed benchmarking on FreeBSD there are plenty of other tools as well.
Just to name a few:
 

  • rawio – obsolete in FreeBSD 7.x version branch (not available in BSD 7.2 and higher)
  • iozone, iozone21 – Tools to test the speed of sequential I/O to files
  • bonnie++ – benchmark tool capable of performing number of simple fs tests
  • bonnie – predecessor filesystem benchmark tool to bonnie++
  • raidtest – test performance of storage devices
  • mdtest – Software to test metadata performance on filsystems
  • filebench – tool for micro-benchmarking storage subsystems

Linux hdparm allows also changing / setting various hdd ATA and SATA settings. Similarly, to set and change ATA / SATA settings on FreeBSD there is the:

  • ataidle

tool.

As of time of writting ataidle is in port path /usr/ports/sysutils/ataidle/

To check it out install it as usual from the port location:

FreeBSD also has also the spindown port – a small program for handling automated spinning down ofSCSI harddrive
spindown is useful in setting values to SATA drives which has problems with properly controlling HDD power management.

To keep constant track on hard disk operations and preliminary warning in case of failing hard disks on FreeBSD there is also smartd service, just like in Linux.
smartd enables you to to control and monitor storage systems using the Self-Monitoring, Analysisand Reporting Technology System (S.M.A.R.T.) built into most modern ATA and SCSI hard disks.
smartd and smartctl are installable via the port /usr/ports/sysutils/smartmontools.

To install and use smartd, ataidle and spindown run:

freebsd# cd /usr/ports/sysutils/smartmontools
freebsd# make && make install clean
freebsd# cd /usr/ports/sysutils/ataidle/
freebsd# make && make install clean
freebsd# cd /usr/ports/sysutils/spindown/
freebsd# make && make install clean

Check each one's manual for more info.

Share this on

How to reboot remotely Linux server if reboot, shutdown and init commands are not working (/sbin/reboot: Input/output error) – Reboot Linux in emergency using MagicSysRQ kernel sysctl variable

Saturday, July 23rd, 2011

SysRQ an alternative way to restart unrestartable Linux server

I’ve been in a situation today, where one Linux server’s hard drive SCSI driver or the physical drive is starting to break off where in dmesg kernel log, I can see a lot of errors like:

[178071.998440] sd 0:0:0:0: [sda] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
[178071.998440] end_request: I/O error, dev sda, sector 89615868

I tried a number of things to remount the hdd which was throwing out errors in read only mode, but almost all commands I typed on the server were either shown as missng or returning an error:
Input/output error

Just ot give you an idea what I mean, here is a paste from the shell:

linux-server:/# vim /etc/fstab
-bash: vim: command not found
linux-server:/# vi /etc/fstab
-bash: vi: command not found
linux-server:/# mcedit /etc/fstab
-bash: /usr/bin/mcedit: Input/output error
linux-server:/# fdisk -l
-bash: /sbin/fdisk: Input/output error

After I’ve tried all kind of things to try to diagnose the server and all seemed failing, I thought next a reboot might help as on server boot the filesystems will get checked with fsck and fsck might be able to fix (at least temporary) the mess.

I went on and tried to restart the system, and guess what? I got:

/sbin/reboot init Input/output error

I hoped that at least /sbin/shutdown or /sbin/init commands might work out and since I couldn’t use the reboot command I tried this two as well just to get once again:

linux-server:/# shutdown -r now
bash: /sbin/shutdown: Input/output error
linux-server:/# init 6
bash: /sbin/init: Input/output error

You see now the situation was not pinky, it seemed there was no way to reboot the system …
Moreover the server is located in remote Data Center and I the tech support there is conducting assigned task with the speed of a turtle.
The server had no remote reboot, web front end or anything and thefore I needed desperately a way to be able to restart the machine.

A bit of research on the issue has led me to other people who experienced the /sbin/reboot init Input/output error error mostly caused by servers with failing hard drives as well as due to HDD control driver bugs in the Linux kernel.

As I was looking for another alternative way to reboot my Linux machine in hope this would help. I came across a blog post Rebooting the Magic Wayhttp://www.linuxjournal.com/content/rebooting-magic-way

As it was suggested in Cory’s blog a nice alternative way to restart a Linux machine without using reboot, shutdown or init cmds is through a reboot with the Magic SysRQ key combination

The only condition for the Magic SysRQ key to work is to have enabled the SysRQ – CONFIG_MAGIC_SYSRQ in Kernel compile time.
As of today luckily SysRQ Magic key is compiled and enabled by default in almost all modern day Linux distributions in this numbers Debian, Fedora and their derivative distributions.

To use the sysrq kernel capabilities as a mean to restart the server, it’s necessery first to activate the sysrq through sysctl, like so:

linux-server:~# sysctl -w kernel.sysrq=1
kernel.sysrq = 1

I found enabling the kernel.sysrq = 1 permanently in the kernel is also quite a good idea, to achieve that I used:

echo 'kernel.sysrq = 1' >> /etc/sysctl.conf

Next it’s wise to use the sync command to sync any opened files on the server as well stopping as much of the server active running services (MySQL, Apache etc.).

linux-server:~# sync

Now to reboot the Linux server, I used the /proc Linux virtual filesystem by issuing:

linux-server:~# echo b > /proc/sysrq-trigger

Using the echo b > /proc/sysrq-trigger simulates a keyboard key press which does invoke the Magic SysRQ kernel capabilities and hence instructs the kernel to immediately reboot the system.
However one should be careful with using the sysrq-trigger because it’s not a complete substitute for /sbin/reboot or /sbin/shutdown -r commands.
One major difference between the standard way to reboot via /sbin/reboot is that reboot kills all the running processes on the Linux machine and attempts to unmount all filesystems, before it proceeds to sending the kernel reboot instruction.

Using echo b > /proc/sysrq-trigger, however neither tries to umount mounted filesystems nor tries to kill all processes and sync the filesystem, so on a heavy loaded (SQL data critical) server, its use might create enormous problems and lead to severe data loss!

SO BEWARE be sure you know what you’re doing before you proceed using /proc/sysrq-trigger as a way to reboot ;).

Share this on

Drunk

Monday, February 26th, 2007

I drinked 100 gr. of Rom and a beer after that and I got drunk. I have some problems with one of the servers in DBG. I hope the machine’s hard drive didn’t die. If the hdd is dead it would be very very bad for me. Tomorrow I have test in Management. As usual I haven’t study :]] I’m listening to Pantera and I want to drink more .. which is not good at all. END—–

Share this on