monitors:memstat

memstat.ksh

Author Vernon Everett
Compatibility Xymon, Solaris 10, Solaris 11, Solaris 11.2 (not tested on others)
Requirements Solaris 10, 11, 11.2, rrdtool 1.2.x or later
Download None
Last Update 2010-06-10 2013-01-03 2013-02-12 2014-11-06

Description

A simple graph to indicate what's using memory on Solaris 10. Can be set to alert.

Really cool for keeping an eye on the size of that ZFS cache and kernel memory usage.

The problem of the graphs not working for systems without ZFS has been resolved with SPLITNCV.

Installation

Client side

  • Copy memstat.ksh into xymon ~client/ext directory (usually in $XYMONCLIENTHOME/ext)
  • Copy getmemstat.ksh into xymon ~/client/ext directory,

For Solaris 11, or sites with restrictions on using setUID, ensure that the Xymon user has the ability to run getmemstat.ksh using sudo

Or

Set up getmemstat.ksh with setUID 

chown root getmemstat.ksh
chmod 4755 getmemstat.ksh

This is because getmemstat runs mdb, and requires root access to do so.

Server side

Add this to xymonserver.cfg 

TEST2RRD="blah,blah,blah,memstat=ncv"
GRAPHS="whatever was there, memstat::8"
SPLITNCV_memstat="*:GAUGE"

Add this to graphs.cfg [memstat] 

FNPATTERN memstat,(.*).rrd
TITLE Kernel Memory Allocation
YAXIS %
-l 0
-u 100
DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
AREA:p@RRDIDX@#@COLOR@:@RRDPARAM@:@STACKIT@
GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

Source

memstat.ksh

Show Code ⇲

Hide Code ⇱ 

#!/bin/ksh
TEMPFILE=$XYMONTMP/memstat.tmp
COLOUR=green
ALERT=0  # 1= alert on high usage
YELLOW=75
RED=90
USE_SUDO=yes
date > $TEMPFILE
echo >> $TEMPFILE
if [ "$USE_SUDO" = "yes" ]
then
   # You are probably on Solaris 11, and/or have set up sudo to grant Xymon user elevated access.
   sudo $XYMONCLIENTHOME/ext/getmemstat.ksh | egrep -v "temporary|^Guest" \
                                            | sed "s/ZFS Metadata/ZFS_Metadata/g" \
                                            | sed "s/ZFS File Data/ZFS          /g" > $TEMPFILE.data
else
   # We can assume you have another way of granting elevated access, like setUID
   $XYMONCLIENTHOME/ext/getmemstat.ksh | egrep -v "temporary|^Guest" \
                                       | sed "s/ZFS Metadata/ZFS_Metadata/g" \
                                       | sed "s/ZFS File Data/ZFS          /g" > $TEMPFILE.data
fi

cat $TEMPFILE.data  >> $TEMPFILE
if [ $ALERT -eq 1 ]
then
   cat $TEMPFILE.data | egrep "^Kernel|^Anon|^Exec" | cut -c52-56 | while read a
   do
      if [ $a -gt $YELLOW -a "$COLOUR" = "green" ]
      then
          COLOUR=yellow
      else
         [ $a -gt $RED ] && COLOUR=red
      fi
   done
fi
echo '<p><font size="1" color="black">' >> $TEMPFILE
cat $TEMPFILE.data | egrep -v -- "-----|Summary|^$|^Total|^Physical" \
                   | sed "s/^Free //g" \
                   | sed "s/[(|)|%]//g" \
                   | awk '{ print $1" : "$NF }' >> $TEMPFILE
echo '</font></p>' >> $TEMPFILE

rm $TEMPFILE.data
$XYMON $XYMSRV "status $MACHINE.memstat $COLOUR $(cat $TEMPFILE)"
rm $TEMPFILE

getmemstat.ksh

Show Code ⇲

Hide Code ⇱ 

#!/bin/ksh
echo "::memstat" | mdb -k

Known Bugs and Issues

  • None known, but please let me know if you find any.

To Do

  • Play around with SPLIT_NCV to avoid the “Graph doesn't work without ZFS issue documented above”
  • Might have a look at another script to assist identifying the memory hogs.

Credits

* Special thanks to Henrik not just for writing Xymon, but also showing me how to create stack graphs with SPLITNCV data.

Changelog

  • 2010-06-10
    • Initial release
  • 2013-01-03
    • The long-promised update to use SPLITNCV for graphing. This means we don't have to muck about adding extra fields to .rrd files for systems that are not using ZFS.
    • Updated variable names to be more Xymon
  • 2013-02-12
    • Changed the trends graph entry to memstat::8. Without it, in the trends column, the graph gets split into 2 graphs, which makes the visualisation of memory usage a little more difficult.
  • 2014-11-05
    • Changed script to give you option of running getmemstat.ksh as root using sudo or setUID
    • Change to cater for memstat/mdb changes in Solaris 11.2.
      Some of the updates look great, but having 2 ZFS values (one for ZFS Metadata and the other for ZFS File Data) although useful, broke graphing. It's fixed now.
      Also, please, can anybody tell me what the memstat value for “Guest” is? I suspect it's a memstat bug.
      The same for the “In temporary use” that we see sometimes.
      Yes, I am talking about the output of echo “::memstat” | mdb -k on Solaris 11.2.
  • monitors/memstat.txt
  • Last modified: 2014/11/06 04:30
  • by vernon