diskstat.pl

Author	Wim Nelis
Compatibility	Xymon 4.2
Requirements	Perl, Linux
Download	None
Last Update	2019-02-12

Description

Script diskstat.pl is a client-side script for servers running Linux, which extracts disk I/O performance parameters from pseudo-file /proc/diskstats and reports those parameters to Xymon. It results in four graphs per monitored disk, showing the I/O request rate, the I/O throughput, the time needed per request and I/O queue length.

As most of the parameters in /proc/diskstats are ever-increasing counters, the reported values are effectively the average value since the previous invocation of this script. Typically, this script will be invoked once every 5 minutes, thus most results will be 5-minute averages. In other words, the results of this script cover the complete time the script is running. The results are thus not a periodic, statistical sample, each sample covering only a fraction of the time since the previous sample.

Script diskstat.pl is installed together with perl module diskstat.pm on the Xymon client. Script diskstat.pl sends all performance data of the selected disks and partitions to Xymon. The selection is defined in module diskstat.pm in hash %Disks. This list is also used to map the disk or partition name onto another name, typically the mount point. The data is sent to Xymon in Devmon format, which requires less configuration compared with the NCV format. At the generation of a graph, a custom script is invoked to extract mountpoint from the name of the RRD and rebuild the original name.

At the client side, script diskstat.pl and module diskstat.pm need to be installed and one configuration file needs to be modified. At the server side, one script needs to be installed and four configuration files of Xymon need to be modified.

Client side

Copy script diskstat.pl and module diskstat.pm to the server to be monitored, typically to subdirectory ~xymon/client/ext or /usr/lib/xymn/client/ext. Make sure that script diskstat.pl can be executed by user xymon. Enter the name of the directory in the 'use lib' directive in script diskstat.pl, at line 17. Define in table %Disks in module diskstat.pm the disks and partitions to monitor, and the name of the mount-point of each.

Edit the following section to reflect your environment and add it to ~xymon/client/etc/tasks.cfg or put it as a separate file named diskstat.cfg in subdirectory /usr/lib/xymon/client/etc/clientlaunch.d:

[diskstat]
  ENVFILE /path/to/xymon/client/etc/xymonserver.cfg
  CMD /path/to/xymon/client/ext/diskstat.pl
  LOGFILE /path/to/xymon/client/logs/diskstat.log
  INTERVAL 5m

Server side

At the Xymon server, it must be configured that this test uses the Devmon format. Therefore add the following two lines to ~xymon/server/etc/xymonserver.cfg or write them in a file named ~xymon/server/etc/xymonserver.d/diskstat.cfg:

TEST2RRD+=",diskstat=devmon"
GRAPHS+=",diskstat::1,diskstat0::1,diskstat1::1,diskstat2::1"

The results are displayed in 4 graphs, named [diskstat], [diskstat0], [diskstat1] and [diskstat2]. These graphs are multi-graphs and use script genlgt.pl to generate the graph title.

Copy script genlgt.pl to ~xymon/server/ext.

Show genlgt.pl ⇲

Hide genlgt.pl ⇱

#!/usr/bin/perl
#
# GENerate_Local_Graph_Title, genlgt
#
use strict ;
my $Title= '??' ;
my $Disk ;
 
if ( $ARGV[1]=~ m/^diskstat\d*$/ ) {
  $Title = "$ARGV[0] , " ;              # Hostname
  if ( $ARGV[3]=~ m/^diskstat\.(.+)\.rrd$/ ) {
    $Disk= $1 ;                         # Raw mountpoint name
    $Disk= '/'                  if $Disk eq ',root' ;
    $Disk=~ s/\,/\//g ;
  } else {
    $Disk= '??' ;
  }  # of else
  $Title.= "Disk statistics of \"$Disk\"  $ARGV[2]" ;
}  # of if
 
print $Title ;

Modify the directory names in the TITLE directives in the following graph definitions and add the result to ~xymon/server/etc/graphs.cfg.

[diskstat0]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Throughput [B/s]
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	DEF:RdAm@RRDIDX@=@RRDFN@:RdAmount:AVERAGE
	DEF:WrAm@RRDIDX@=@RRDFN@:WrAmount:AVERAGE
	LINE1:RdAm@RRDIDX@#@COLOR@:Read
	GPRINT:RdAm@RRDIDX@:MIN: Min\: %5.1lf %sB/s
	GPRINT:RdAm@RRDIDX@:MAX:Max\: %5.1lf %sB/s
	GPRINT:RdAm@RRDIDX@:AVERAGE:Avg\: %5.1lf %sB/s
	GPRINT:RdAm@RRDIDX@:LAST:Cur\: %5.1lf %sB/s\n
	LINE1:WrAm@RRDIDX@#@COLOR@:Write
	GPRINT:WrAm@RRDIDX@:MIN:Min\: %5.1lf %sB/s
	GPRINT:WrAm@RRDIDX@:MAX:Max\: %5.1lf %sB/s
	GPRINT:WrAm@RRDIDX@:AVERAGE:Avg\: %5.1lf %sB/s
	GPRINT:WrAm@RRDIDX@:LAST:Cur\: %5.1lf %sB/s\n

[diskstat1]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Time per request [s/r]
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	DEF:RdTim@RRDIDX@=@RRDFN@:RdTime:AVERAGE
	DEF:RdSys@RRDIDX@=@RRDFN@:RdRequest:AVERAGE
	DEF:RdMrg@RRDIDX@=@RRDFN@:RdMerge:AVERAGE
	DEF:WrTim@RRDIDX@=@RRDFN@:WrTime:AVERAGE
	DEF:WrSys@RRDIDX@=@RRDFN@:WrRequest:AVERAGE
	DEF:WrMrg@RRDIDX@=@RRDFN@:WrMerge:AVERAGE
	CDEF:RdTm@RRDIDX@=RdTim@RRDIDX@,1000,/,RdSys@RRDIDX@,RdMrg@RRDIDX@,+,/
	CDEF:WrTm@RRDIDX@=WrTim@RRDIDX@,1000,/,WrSys@RRDIDX@,WrMrg@RRDIDX@,+,/
	LINE1:RdTm@RRDIDX@#@COLOR@:Read
	GPRINT:RdTm@RRDIDX@:MIN: Min\: %5.1lf %ss/r
	GPRINT:RdTm@RRDIDX@:MAX:Max\: %5.1lf %ss/r
	GPRINT:RdTm@RRDIDX@:AVERAGE:Avg\: %5.1lf %ss/r
	GPRINT:RdTm@RRDIDX@:LAST:Cur\: %5.1lf %ss/r\n
	LINE1:WrTm@RRDIDX@#@COLOR@:Write
	GPRINT:WrTm@RRDIDX@:MIN:Min\: %5.1lf %ss/r
	GPRINT:WrTm@RRDIDX@:MAX:Max\: %5.1lf %ss/r
	GPRINT:WrTm@RRDIDX@:AVERAGE:Avg\: %5.1lf %ss/r
	GPRINT:WrTm@RRDIDX@:LAST:Cur\: %5.1lf %ss/r\n

[diskstat2]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Queue length []
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	-l 0
	DEF:IwTim@RRDIDX@=@RRDFN@:IoWTime:AVERAGE
	CDEF:Aql@RRDIDX@=IwTim@RRDIDX@,1000,/
	LINE1:Aql@RRDIDX@#@COLOR@:Queue length
	GPRINT:Aql@RRDIDX@:MIN:Min\: %5.1lf %s
	GPRINT:Aql@RRDIDX@:MAX: Max\: %5.1lf %s
	GPRINT:Aql@RRDIDX@:AVERAGE: Avg\: %5.1lf %s
	GPRINT:Aql@RRDIDX@:LAST: Cur\: %5.1lf %s\n

[diskstat]
	TITLE exec:/path/to/xymon/server/ext/genlgt.pl
	YAXIS Request rate [r/s]
	FNPATTERN ^diskstat\.(.+?)\.rrd$
	DEF:RdRq@RRDIDX@=@RRDFN@:RdRequest:AVERAGE
	DEF:RdMr@RRDIDX@=@RRDFN@:RdMerge:AVERAGE
	DEF:WrRq@RRDIDX@=@RRDFN@:WrRequest:AVERAGE
	DEF:WrMr@RRDIDX@=@RRDFN@:WrMerge:AVERAGE
	LINE1:RdRq@RRDIDX@#@COLOR@:Read
	GPRINT:RdRq@RRDIDX@:MIN:   Min\: %5.1lf %sr/s
	GPRINT:RdRq@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:RdRq@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:RdRq@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n
	LINE1:WrRq@RRDIDX@#@COLOR@:Write
	GPRINT:WrRq@RRDIDX@:MIN:  Min\: %5.1lf %sr/s
	GPRINT:WrRq@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:WrRq@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:WrRq@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n
	LINE1:RdMr@RRDIDX@#@COLOR@:RdMerge
	GPRINT:RdMr@RRDIDX@:MIN:Min\: %5.1lf %sr/s
	GPRINT:RdMr@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:RdMr@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:RdMr@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n
	LINE1:WrMr@RRDIDX@#@COLOR@:WrMerge
	GPRINT:WrMr@RRDIDX@:MIN:Min\: %5.1lf %sr/s
	GPRINT:WrMr@RRDIDX@:MAX:Max\: %5.1lf %sr/s
	GPRINT:WrMr@RRDIDX@:AVERAGE:Avg\: %5.1lf %sr/s
	GPRINT:WrMr@RRDIDX@:LAST:Cur\: %5.1lf %sr/s\n

Define the diskstat graphs to be multi-graphs with the following modification in ~xymon/server/etc/cgioptions.cfg:

CGI_SVC_OPTS="... --multigraphs=diskstat,diskstat0,diskstat1,diskstat2"

Finally, define that all diskstat graphs should be shown in the 'trends' column of the monitored host by adding the following directive in ~xymon/server/etc/hosts.cfg:

 <Ip> <Host> # ... TRENDS:*,diskstat:diskstat|diskstat0|diskstat1|diskstat2

Show diskstat.pl ⇲

Hide diskstat.pl ⇱

#!/usr/bin/perl -w
#
# This script determines some elementary disk statistics, which are (also)
# accessible via CLI command `iostat`. Pseudo file /proc/diskstats is used as
# primary source of information.
#
# This script does not have any memory by design. It only reports the values
# found in /proc/diskstat to Xymon, although it does convert some values to a
# more appropriate unit. In most cases, the difference with the value in the
# previous pass is of interest. These differences are computed by RRD, prior to
# saving the values.
#
# Written by W.J.M. Nelis, wim.nelis@nlr.nl, 201103
#
use strict ;
use Time::Piece ;			# Format time
use lib "/usr/lib/xymon/client/ext" ;
use diskstat ;				# Import list of devices to check
 
#
# Installation constants.
# -----------------------
#
# Define the parameters to reach the Xymon server.
#
my $XyDisp= $ENV{XYMSRV} ;		# Name of monitor server
my $XySend= $ENV{XYMON} ;		# Monitor interface program
my $FmtDate= "%Y.%m.%d %H:%M:%S" ;	# Default date format
   $FmtDate= $ENV{XYMONDATEFORMAT} if defined $ENV{XYMONDATEFORMAT} ;
my $XyTest= 'diskstat' ;		# Name of test
#
my $Server= `hostname` ;  chomp $Server ;
my $Now= localtime ;                    # Timestamp of tests
   $Now= $Now->strftime( $FmtDate ) ;
#
my $InpFil= '/proc/diskstats' ;		# Input file name
#
# Define the name of each data-set, the unit conversion factor and the RRD type.
# Note that RRD requires that the values of type 'DERIVE' are integer. If
# however floating point numbers are used, the RRD type must be set to
# 'DDERIVE'.
#
my @DS= ( 				# Define parameters of the datasets
    [ 'Name'     ,     1, ''        ],	#  0 - Name of disk / partition
    [ 'RdRequest',     1, 'DERIVE'  ],	#  1 - Read requests
    [ 'RdMerge'  ,     1, 'DERIVE'  ],	#  2 - Read request merges
    [ 'RdAmount' ,   512, 'DERIVE'  ],	#  3 - Sectors read
    [ 'RdTime'   , 0.001, 'DDERIVE' ],	#  4 - Time spent reading [ms]
    [ 'WrRequest',     1, 'DERIVE'  ],	#  5 - Write requests
    [ 'WrMerge'  ,     1, 'DERIVE'  ],	#  6 - Write request merges
    [ 'WrAmount' ,   512, 'DERIVE'  ],	#  7 - Sectors written
    [ 'WrTime'   , 0.001, 'DDERIVE' ],	#  8 - Time spent writing [ms]
    [ 'IoQueue'  ,     1, 'GAUGE'   ],	#  9 - Length of I/O queue
    [ 'IoTime'   , 0.001, 'DDERIVE' ],	# 10 - Time spent for I/O [ms]
    [ 'IoWTime'  , 0.001, 'DDERIVE' ]	# 11 - Time weighted with (9)
) ;
 
#
# Global variables.
#
my $Color= 'green' ;			# Xymon status
my $Result= '' ;			# Xymon status message
my %Stats= () ;				# Collected statistics
 
#
# Function ReadStatistics retrieves the disk usage statistics for the disks and
# partitions mentioned in %Disk.
#
sub ReadStatistics() {
  my $Disk ;			 	# Disk / partition name
  my @Field ;				# Statistics of one disk
 
 #
 # Preset all values to be reported. They get the value 'undefined'.
 #
  foreach my $Disk ( keys %Disks ) {
    $Stats{$Disk}= ['','U','U','U','U','U','U','U','U','U','U','U'] ;
  }  # of foreach
 
  if ( open(FH,'<',$InpFil) ) {
    while ( <FH> ) {
      chomp ;
      @Field= split ;			# Extract all fields
      next			unless exists $Disks{$Field[2]} ;
      if ( @Field != 14 ) {
	$Result.= "&yellow Format of statistics of $Field[2] is wrong\n" ;
	$Color  = 'yellow' ;
	delete $Stats{$Field[2]} ;	# Inhibit "missing info" warning
	next ;
      }  # of if
 
      splice @Field, 0, 2 ;		# Remove major and minor number
      $Stats{$Field[0]}= [$Disks{$Field[0]}{Name},@Field[1..11]] ;
    }  # of while
    close( FH ) ;
 #
 # See if statistics for each disk is found in the input file.
 #
    foreach $Disk ( keys %Stats ) {
      next			unless $Stats{$Disk}[0] eq '' ;
      $Stats{$Disk}[0]= $Disks{$Disk}{Name} ;
      $Result.= "&yellow No statistics found for $Disk\n" ;
      $Color  = "yellow" ;
    }  # of foreach
 #
  } else {				# Open failed
    $Result= "&clear No statistics available: failed to open $InpFil\n" ;
    $Color = 'clear' ;
    return ;
  }  # of else
}  # of ReadStatistics
 
#
# Function BuildRrdData reports the values in Devmon format to Xymon, causing
# Xymon to write the values to an RRD. The names of the mount-points are adapted
# to match the Xymon conventions.
#
sub BuildRrdData() {
  my $Rra ;				# Part of name of RRA
  my $Val ;				# Value of one statistic
  my $ar ;				# Ref to array of measurements
 
  $Result.= "<!-- linecount=" . scalar(keys %Stats) . " -->\n" ;
  $Result.= "<!--DEVMON RRD: diskstat 0 0\n" ;
  foreach my $ds ( @DS ) {
    next			if $$ds[2] eq '' ;
    $Result.= "DS:$$ds[0]:$$ds[2]:600:0:U " ;
  }  # of foreach
  chop $Result ;			# Remove trailing space
  $Result.= "\n" ;
 
  foreach my $Disk ( sort keys %Stats ) {
    $ar= $Stats{$Disk} ;		# Ref to array
    $Rra= $$ar[0] ;			# Mountpoint of disk
    $Rra= '/root'		if $Rra eq '/' ;	# Special name for root
    $Rra=~ s/\//,/g ;			# '/' -> ','
 
    $Result.= "$Rra " ;
    for ( my $i= 1 ; $i<= 11 ; $i++ ) {
      $Val= $Stats{$Disk}[$i] ;
      if ( $Val eq 'U' ) {
	$Result.= 'U:' ;
      } else {
	$Val= $Val * $DS[$i][1] ;	# Unit conversion
	if ( $DS[$i][2] eq 'DDERIVE' ) {
	  $Result.= sprintf( "%.3f:", $Val ) ;
	} else {
	  $Result.= sprintf( "%d:", $Val ) ;
	}  # of else
      }  # of else
    }  # of for
    chop $Result ;			# Remove trailing colon
    $Result.= "\n" ;
  }  # of foreach
  $Result.= '-->' ;			# End of Devmon format
}  # of BuildRrdData
 
 
#
# Main program.
# -------------
#
if ( scalar(keys %Disks) ) {
  ReadStatistics() ;			# Read disk statistics
  BuildRrdData() ;			# Format measurements for Xymon / RRD
} else {
  $Result= "There are no disks defined to monitor\n" ;
}  # of else
 
$Result= "\"status $Server.$XyTest $Color $Now\n" .
	 "Disk and partition statistics\n\n" .
	 $Result . "\"\n" ;
`$XySend $XyDisp $Result` ;		# Inform Xymon

Show diskstat.pm ⇲

Hide diskstat.pm ⇱

package diskstat ;
#
# Specify the disks and partitions to be monitored, and the name of the
# associated mount-point. The latter is shown in the resulting graph.
#
require Exporter ;
 
our @ISA   = qw/ Exporter / ;
our @EXPORT= qw/ %Disks / ;
 
%Disks= (
	"dm-0" => { Name => "/"     },
) ;
 
1 ;

The combination of multigraphs and sub-graphs do not work well in Xymon. This problem becomes visible if you monitor more than one disk or partition. In column 'diskstat' there will be one graph per monitored partition. However, in column 'trends' the data of multiple partitions is shown in one graph.

An option is to rework this script to send a trends-message to Xymon in stead of a status message. The lack of a column named 'diskstat' might be an advantage for some.