diskstat.pl

Author Wim Nelis
Compatibility Xymon 4.2
Requirements Perl, Linux
Download None
Last Update 2012-04-02

Script diskstat.pl is a client-side script for servers running Linux, which extracts disk I/O performance parameters from pseudo-file /proc/diskstats and reports those parameters to Xymon. It results in four graphs per monitored disk, showing the I/O request rate, the I/O throughput, the time needed per request and I/O queue length.

As most of the parameters in /proc/diskstats are ever-increasing counters, the reported values are effectivly the average value since the previous invocation of this script. Typically, this script will be invoked once every 5 minutes, thus most results will be 5-minute averages. In other words, the results cover the complete time the script is running, and do not deliver a periodic, statistical sample.

Script diskstat.pl is installed together with perl module diskstat.pm on the Xymon client. Script diskstat.pl sends all performance data of the selected disks and partitions to Xymon. The selection is defined in module diskstat.pm in hash %Disks. I did not find a way to map the name of the disk or partition onto it's mountpoint. Therefore the name of the mountpoint also needs to be specified. The data is sent to Xymon in NCV format. The name used has a special format, namely <Mountpoint>/<Variable>. In the name of the mountpoint, any forward slash is replaced by a comma. At the Xymon server, a script is invoked to handle the status message from script diskstat.pl and save the data in a RRD, one RRD per monitored disk. It uses <Mountpoint> to construct the name of the RRD, while <Variable> becomes the name of the dataset. At the generation of a graph, a custom script is invoked to extract <Mountpoint> from the name of the RRD and rebuild the original name.

At the client side, script diskstat.pl and module diskstat.pm need to be installed and one configuration file needs to be modified. At the server side, two scripts need to be installed and four configuration files of Xymon need to be modified.

Client side

Copy script diskstat.pl and module diskstat.pm to the server to be monitored, typically to subdirectory ~xymon/client/ext. Enter the name of the directory in the 'use lib' directive in script diskstat.pl, at line 17. Define in table %Disks in module diskstat.pm the disks and partitions to monitor, and the name of the mountpoint of each.

Edit the following section to reflect your environment and add it to the ~xymon/client/etc/tasks.cfg:

[diskstat]
  ENVFILE /path/to/xymon/client/etc/xymonserver.cfg
  CMD /path/to/xymon/client/ext/diskstat.pl
  LOGFILE /path/to/xymon/client/logs/diskstat.log
  INTERVAL 5m

Server side

At the Xymon server, the status message from script diskstat.pl is to be handled by script rrd_status.pl. This is accomplished by merging the following configuration into file ~xymon/server/etc/tasks.cfg:

[rrdstatus]
  ENVFILE /path/to/xymon/server/etc/xymonserver.cfg
  NEEDS xymond
  CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --extra-tests=diskstat --extra-script=/path/to/xymon/server/ext/rrd_status.pl --rrddir=$XYMONVAR/rrd

Install the following script, named rrd_status.pl, in directory ~xymon/server/ext. However, it could be that there is already a script with a similar purpose for other tests: in that case you should merge it with the script below.

Show rrd_status.pl ⇲

Hide rrd_status.pl ⇱

#!/usr/bin/perl
#
# This script handles a list of NCVs, send by a Xymon client, and prepares it to
# be stored in an RRA. This script is used in cases in which a fixed-size group
# of two or more values should be put together into a single RRA. The algorithm
# is specific for each test / client.
#
# This script is invoked with three parameters: the name of the host, the name
# of the test and the name of the file containing the message sent by the
# client, containing the NCVs to be handled.
#
use strict;
 
#
# Installation constants.
# -----------------------
#
# %Struct defines the datasets of the various tests.
#
my %Struct= (
        diskstat => [           # Must be sorted!
                "DS:IoQueue:GAUGE:600:0:U\n"  ,
                "DS:IoTime:DERIVE:600:0:U\n"  ,
                "DS:IoWTime:DERIVE:600:0:U\n" ,
                "DS:RdAmount:DERIVE:600:0:U\n",
                "DS:RdMerge:DERIVE:600:0:U\n"   ,
                "DS:RdRequest:DERIVE:600:0:U\n"   ,
                "DS:RdTime:DERIVE:600:0:U\n"  ,
                "DS:WrAmount:DERIVE:600:0:U\n",
                "DS:WrMerge:DERIVE:600:0:U\n"   ,
                "DS:WrRequest:DERIVE:600:0:U\n"   ,
                "DS:WrTime:DERIVE:600:0:U\n"   ]
   ) ;  # of %Struct
 
#
# Global variables.
# -----------------
#
my ( $HostName, $TestName, $FileName )= @ARGV ;
#
my %Var= () ;                           # Save area measurements
my ( $Line, @Line ) ;                   # List of values of one measurement
my $key ;                               # Loop control variable
 
 
#
# Main program.
# -------------
#
 
#
# Handle test "diskstat".
#
# An attempt has been undertaken to make this code a little bit more general.
# The name of an NCV should consist of two names separated by "/". The first
# name becomes (part of) the name of the RRA, the second name becomes the
# name of the DS. The DS-ses are written in sorted order.
#
if ( $TestName eq "diskstat" ) {
  open( FH, "<", $FileName )    or die ;
  while ( <FH> ) {
    chomp ;
    next                        unless m/^([\w\.\,-]+)\/(\w+)\s+:\s+(U|[\d\.]+)\s*$/ ;
    $Var{$1}{$2}= $3 ;
  }  # of while
  close( FH ) ;
 
  print @{$Struct{$TestName}} ;
  foreach $key ( sort keys %Var ) {
    @Line= () ;
    push @Line, $Var{$key}{$_}  foreach ( sort keys %{$Var{$key}} ) ;
    print "$TestName.$key.rrd\n" ;
    print join( ":", @Line ) . "\n" ;
  }  # of foreach
}  # of if
 
exit 0 ;

The results are displayed in 4 graphs, named [diskstat], [diskstat0], [diskstat1] and [diskstat2]. These graphs are multi-graphs and use script genlgt.pl to generate the graph title.

Copy script genlgt.pl to ~xymon/server/ext.

Show genlgt.pl ⇲

Hide genlgt.pl ⇱

#!/usr/bin/perl
#
# GENerate_Local_Graph_Title, genlgt
#
use strict ;
my $Title= '??' ;
my $Disk ;
 
if ( $ARGV[1]=~ m/^diskstat\d*$/ ) {
  $Title = "$ARGV[0] , " ;              # Hostname
  if ( $ARGV[3]=~ m/^diskstat\.(.+)\.rrd$/ ) {
    $Disk= $1 ;                         # Raw mountpoint name
    $Disk= '/'                  if $Disk eq ',root' ;
    $Disk=~ s/\,/\//g ;
  } else {
    $Disk= '??' ;
  }  # of else
  $Title.= "Disk statistics of \"$Disk\"  $ARGV[2]" ;
}  # of if
 
print $Title ;

Modify the directory names in the TITLE directives in the following graph definitions and add the result to ~xymon/server/etc/graphs.cfg.

[diskstat0]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Throughput [B/s]
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	DEF:RdAm@RRDIDX@=@RRDFN@:RdAmount:AVERAGE
	DEF:WrAm@RRDIDX@=@RRDFN@:WrAmount:AVERAGE
	LINE1:RdAm@RRDIDX@#@COLOR@:Read
	GPRINT:RdAm@RRDIDX@:MIN: Min\: %5.1lf %sB/s
	GPRINT:RdAm@RRDIDX@:MAX:Max\: %5.1lf %sB/s
	GPRINT:RdAm@RRDIDX@:AVERAGE:Avg\: %5.1lf %sB/s
	GPRINT:RdAm@RRDIDX@:LAST:Cur\: %5.1lf %sB/s\n
	LINE1:WrAm@RRDIDX@#@COLOR@:Write
	GPRINT:WrAm@RRDIDX@:MIN:Min\: %5.1lf %sB/s
	GPRINT:WrAm@RRDIDX@:MAX:Max\: %5.1lf %sB/s
	GPRINT:WrAm@RRDIDX@:AVERAGE:Avg\: %5.1lf %sB/s
	GPRINT:WrAm@RRDIDX@:LAST:Cur\: %5.1lf %sB/s\n

[diskstat1]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Time per request [s/r]
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	DEF:RdTim@RRDIDX@=@RRDFN@:RdTime:AVERAGE
	DEF:RdSys@RRDIDX@=@RRDFN@:RdRequest:AVERAGE
	DEF:RdMrg@RRDIDX@=@RRDFN@:RdMerge:AVERAGE
	DEF:WrTim@RRDIDX@=@RRDFN@:WrTime:AVERAGE
	DEF:WrSys@RRDIDX@=@RRDFN@:WrRequest:AVERAGE
	DEF:WrMrg@RRDIDX@=@RRDFN@:WrMerge:AVERAGE
	CDEF:RdTm@RRDIDX@=RdTim@RRDIDX@,1000,/,RdSys@RRDIDX@,RdMrg@RRDIDX@,+,/
	CDEF:WrTm@RRDIDX@=WrTim@RRDIDX@,1000,/,WrSys@RRDIDX@,WrMrg@RRDIDX@,+,/
	LINE1:RdTm@RRDIDX@#@COLOR@:Read
	GPRINT:RdTm@RRDIDX@:MIN: Min\: %5.1lf %ss/r
	GPRINT:RdTm@RRDIDX@:MAX:Max\: %5.1lf %ss/r
	GPRINT:RdTm@RRDIDX@:AVERAGE:Avg\: %5.1lf %ss/r
	GPRINT:RdTm@RRDIDX@:LAST:Cur\: %5.1lf %ss/r\n
	LINE1:WrTm@RRDIDX@#@COLOR@:Write
	GPRINT:WrTm@RRDIDX@:MIN:Min\: %5.1lf %ss/r
	GPRINT:WrTm@RRDIDX@:MAX:Max\: %5.1lf %ss/r
	GPRINT:WrTm@RRDIDX@:AVERAGE:Avg\: %5.1lf %ss/r
	GPRINT:WrTm@RRDIDX@:LAST:Cur\: %5.1lf %ss/r\n

[diskstat2]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Queue length []
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	-l 0
	DEF:IwTim@RRDIDX@=@RRDFN@:IoWTime:AVERAGE
	CDEF:Aql@RRDIDX@=IwTim@RRDIDX@,1000,/
	LINE1:Aql@RRDIDX@#@COLOR@:Queue length
	GPRINT:Aql@RRDIDX@:MIN:Min\: %5.1lf %s
	GPRINT:Aql@RRDIDX@:MAX: Max\: %5.1lf %s
	GPRINT:Aql@RRDIDX@:AVERAGE: Avg\: %5.1lf %s
	GPRINT:Aql@RRDIDX@:LAST: Cur\: %5.1lf %s\n

[diskstat]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Request rate [r/s]
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	DEF:RdRq@RRDIDX@=@RRDFN@:RdRequest:AVERAGE
	DEF:RdMr@RRDIDX@=@RRDFN@:RdMerge:AVERAGE
	DEF:WrRq@RRDIDX@=@RRDFN@:WrRequest:AVERAGE
	DEF:WrMr@RRDIDX@=@RRDFN@:WrMerge:AVERAGE
	LINE1:RdRq@RRDIDX@#@COLOR@:Read
	GPRINT:RdRq@RRDIDX@:MIN:   Min\: %5.1lf %sr/s
	GPRINT:RdRq@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:RdRq@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:RdRq@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n
	LINE1:WrRq@RRDIDX@#@COLOR@:Write
	GPRINT:WrRq@RRDIDX@:MIN:  Min\: %5.1lf %sr/s
	GPRINT:WrRq@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:WrRq@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:WrRq@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n
	LINE1:RdMr@RRDIDX@#@COLOR@:RdMerge
	GPRINT:RdMr@RRDIDX@:MIN:Min\: %5.1lf %sr/s
	GPRINT:RdMr@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:RdMr@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:RdMr@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n
	LINE1:WrMr@RRDIDX@#@COLOR@:WrMerge
	GPRINT:WrMr@RRDIDX@:MIN:Min\: %5.1lf %sr/s
	GPRINT:WrMr@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:WrMr@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:WrMr@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n

Define the diskstat graphs to be multi-graphs, which should show only one disk per graph, with the following modification in ~xymon/server/etc/cgioptions.cfg

CGI_SVC_OPTS="... --multigraphs=diskstat,diskstat0,diskstat1,diskstat2"

and the following modifications in ~xymon/server/etc/xymonserver.cfg:

TEST2RRD="...,diskstat"
GRAPHS="...,diskstat::1,diskstat0::1,diskstat1::1,diskstat2::1"

Finally, define that all diskstat graphs should be shown in the 'trends' column of the monitored host by adding the following directive in ~xymon/server/etc/hosts.cfg:

 <Ip> <Host> # ... TRENDS:*,diskstat:diskstat|diskstat0|diskstat1|diskstat2

diskstat.pl

Show diskstat.pl ⇲

Hide diskstat.pl ⇱

#!/usr/bin/perl -w
#
# This script determines some elementary disk statistics, which are (also)
# accessible via CLI command `iostat`. Pseudo file /proc/diskstats is used
# as primary source of information.
#
# This script does not have any memory by design. It only reports the values
# found in /proc/diskstat to Xymon, although it does convert some values to
# a more appropiate unit. In most cases, the difference with the value in
# the previous pass is of interest. These differences are computed by RRD,
# prior to saving the values.
#
# Written by W.J.M. Nelis, wim.nelis@nlr.nl, 201103
#
use strict ;
use POSIX qw/ strftime / ;		# Format time
use lib "/path/to/xymon/client/ext" ;
use diskstat ;				# Import list of devices to check
 
#
# Installation constants.
# -----------------------
#
# Define the parameters to reach the Xymon server.
#
my $XyDisp= $ENV{XYMONSERVERHOSTNAME} ;	# Name of monitor server
my $XySend= $ENV{XYMON} ;		# Monitor interface program
my $FmtDate= "%Y.%m.%d %H:%M:%S" ;	# Default date format
   $FmtDate= $ENV{XYMONDATEFORMAT} if defined $ENV{XYMONDATEFORMAT} ;
#
my $Server= `hostname` ;  chomp $Server ;
my $Now= strftime( $FmtDate, localtime ) ;	# Timestamp of tests
#
my $InpFil= '/proc/diskstats' ;		# Input file name
my $Test= 'diskstat' ;			# Name of test
#
# Define the name of each dataset and the unit conversion factor. As most
# of the datasets are defined to be of type DERIVED, the value passed to
# RRD must be an integer number. Thus the times expressed in [ms] are *not*
# converted to [s].
#
my @DS= ( 				# Define parameters of the datasets
    [ 'Name'     ,   1 ],		#  0 - Name of disk / partition
    [ 'RdRequest',   1 ],		#  1 - Read requests
    [ 'RdMerge'  ,   1 ],		#  2 - Read request merges
    [ 'RdAmount' , 512 ],		#  3 - Sectors read
    [ 'RdTime'   ,   1 ],		#  4 - Time spent reading [ms]
    [ 'WrRequest',   1 ],		#  5 - Write requests
    [ 'WrMerge'  ,   1 ],		#  6 - Write request merges
    [ 'WrAmount' , 512 ],		#  7 - Sectors written
    [ 'WrTime'   ,   1 ],		#  8 - Time spent writing [ms]
    [ 'IoQueue'  ,   1 ],		#  9 - Length of I/O queue
    [ 'IoTime'   ,   1,],		# 10 - Time spent for I/O [ms]
    [ 'IoWTime'  ,   1,]		# 11 - Time weighted with (9)
) ;
 
#
# Global variables.
#
my $Color= 'green' ;			# Xymon status
my $Result= '' ;			# Xymon status message
my %Stats= () ;				# Collected statistics
 
#
# Function ReadStatistics retrieves the disk usage statistics for the
# disks and partitions mentioned in %Disk.
#
sub ReadStatistics() {
  my $Disk ;			 	# Disk / partition name
  my @Field ;				# Statistics of one disk
 
 #
 # Preset all values to be reported. They get the value 'undefined'.
 #
  foreach my $Disk ( keys %Disks ) {
    $Stats{$Disk}= ['','U','U','U','U','U','U','U','U','U','U','U'] ;
  }  # of foreach
 
  if ( open(FH,'<',$InpFil) ) {
    while ( <FH> ) {
      chomp ;
      @Field= split ;			# Extract all fields
      next			unless defined $Disks{$Field[2]} ;
      if ( @Field != 14 ) {
	$Result.= "&yellow Format of statistics of $Field[2] is wrong\n" ;
	$Color  = 'yellow' ;
	delete $Stats{$Field[2]} ;	# Inhibit "missing info" warning
	next ;
      }  # of if
 
      splice @Field, 0, 2 ;		# Remove major and minor number
      $Stats{$Field[0]}= [$Disks{$Field[0]}{Name},@Field[1..11]] ;
    }  # of while
    close( FH ) ;
 #
 # See if statistics for each disk is found in the input file.
 #
    foreach $Disk ( keys %Stats ) {
      next			unless $Stats{$Disk}[0] eq '' ;
      $Stats{$Disk}[0]= $Disks{$Disk}{Name} ;
      $Result.= "&yellow No statistics found for $Disk\n" ;
      $Color= "yellow" ;
    }  # of foreach
 #
  } else {				# Open failed
    $Result= "&clear No statistics available: failed to open $InpFil\n" ;
    $Color = 'clear' ;
    return ;
  }  # of else
}  # of ReadStatistics
 
#
# Function BuildRrdData converts the list of values into a list of NCV
# patterns, which are recognised by Xymon and ultimately RRD. The names
# of the mountpoints are adapted to match the Xymon conventions.
#
sub BuildRrdData() {
  my $Disk ;				# Loop control variable
  my $Rra ;				# Part of name of RRA
  my $Val ;				# Value of one statistic
  my $ar ;				# Ref to array of measurements
 
  $Result.= "<!-- linecount=" . scalar(keys %Stats) . " -->\n" ;
  $Result.= "<!--\n" ;			# Hide measurements in HTML comment
  foreach $Disk ( sort keys %Stats ) {
    $ar= $Stats{$Disk} ;		# Ref to array
    $Rra= $$ar[0] ;			# Mountpoint of disk
    $Rra= '/root'		if $Rra eq '/' ;	# Special name for root
    $Rra=~ s/\//,/g ;			# '/' -> ','
 
    for ( my $i= 1 ; $i<= 11 ; $i++ ) {
      $Val= $Stats{$Disk}[$i] ;
      if ( $Val eq 'U' ) {
	$Result.= sprintf( "%s/%s : U\n", $Rra, $DS[$i][0] ) ;
      } else {
	$Val= $Val * $DS[$i][1] ;	# Unit conversion
	$Result.= sprintf( "%s/%s : %d\n", $Rra, $DS[$i][0], $Val ) ;
      }  # of else
    }  # of for
  }  # of foreach
  $Result.= '-->' ;
}  # of BuildRrdData
 
 
#
# Main program.
# -------------
#
if ( scalar(keys %Disks) ) {
  ReadStatistics ;			# Read disk statistics
  BuildRrdData ;			# Format measurements for Xymon / RRD
} else {
  $Result= "There are no disks defined to monitor\n" ;
}  # of else
 
$Result= "\"status $Server.$Test $Color $Now\n" .
	 "Disk and partition statistics\n\n" .
	 $Result . "\"\n" ;
`$XySend $XyDisp $Result` ;		# Inform Xymon

diskstat.pm

Show diskstat.pm ⇲

Hide diskstat.pm ⇱

package diskstat ;
#
# Specify the disks and partitions to be monitored, and the name of the
# associated mount-point. The latter is shown in the resulting graph.
#
require Exporter ;
 
our @ISA   = qw/ Exporter / ;
our @EXPORT= qw/ %Disks / ;
 
%Disks= (
	"dm-0" => { Name => "/"     },
) ;
 
1 ;

The combination of multigraphs and sub-graphs do not work well in Xymon. This problem becomes visible if you monitor more than one disk or partition. In column 'diskstat' there will be one graph per monitored partition. However, in column 'trends' the data of multiple partitions is shown in one graph.

An option is to rework this script to send a trends-message to Xymon in stead of a status message. An advantage is that there is no need any more for script rrd_status.pl, a disadvantage might be the lack of a column named 'diskstat'.

  • 2012-04-02
    • Initial release
  • monitors/diskstat.pl.txt
  • Last modified: 2012/04/12 07:16
  • (external edit)