====== bb-zfs ======

^ Author         | [[everett.vernon@gmail.com| Vernon Everett ]]  |
^ Compatibility  | Xymon 4.2                                      |
^ Requirements   | ZFS                                            |
^ Download       | None                                           |
^ Last Update    | 2020-04-20                                     |

===== Description =====
An update to the BB ZFS monitor to graph capacity and keep tabs on snapshots as well

===== Installation =====
=== Client side ===
  * Copy bb-zfs.ksh to the hobbit/xymon ext directory (usually in HOBBITCLIENTHOME/ext)
  * Add the following to clientlaunch.cfg <code>
    [zfs]
        ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
        CMD $HOBBITCLIENTHOME/ext/bb-zfs.ksh
        LOGFILE $HOBBITCLIENTHOME/logs/zfs.log
        INTERVAL 5m
</code>
=== Server side ===
  * Add the following lines to hobbitgraph.cfg <code>
[zfs]
    FNPATTERN ^zfs,(.*).rrd
    TITLE ZFS Capacity
    YAXIS % Full
    DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
    LINE2:p@RRDIDX@#@COLOR@:@RRDPARAM@
    -u 100
    -l 0
    GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
    GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
    GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
    GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
</code>
  * In hobbitserver.cfg, add the following line
<code> GRAPHS="...,xymond,zfs"
TEST2RRD="...,xymond,zfs=ncv"
SPLITNCV_zfs="*:GAUGE"
</code>

===== Source =====
==== bb-zfs.ksh ====
<hidden onHidden="Show Code ⇲" onVisible="Hide Code ⇱">
<code>
#!/bin/ksh
# Revision History:
# 1. Mike Rowell <Mike.Rowell@Rightmove.co.uk>, original
# 2. Uwe Kirbach <U.Kirbach@EnBW.com>
# 3. T.J. Yang: add in some comments.
# 4. Vernon Everett <everett.vernon@gmail.com : Added check for old snapshots
#                                             : Added graphing data
# 5. FreeSoftwareServers <admin@freesoftwareservers.com>: 
# Update to new "zpool list -h" format
# Add Available/Free to table
# Full "zpool status" always displayed
# Set DEGRADED=RED
# DISKYELL=90 & DISKRED=95

DISKYELL=90
DISKRED=95
SNAPRED=90      # Days old
SNAPYELL=60     # Days old
SNAPCOL=true    # Set to true if snapshot age should effect test colour
CHECKSNAPS=true # Set to true to do snapshot checking
TEST="zfs"
DISPCOLOR="green"
FIRST_LINE="zfs - okay"
FIRST_LINE_HEALTH="okay"
FIRST_LINE_CAP="okay"
DATA=""

#What: beautify the page display by html code.
STRING="<table border=1 cellpadding=10><tr><th></th><th>Zpool Name</th><th>Status</th><th>Allocated</th><th>Free</th><th>Capacity</th></tr>"
#What: a loop to parse output of "zpool list -H" output.
# bash-3.00# zpool list
# NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
# mypool                 33.8G   84.5K   33.7G     0%  ONLINE     -
#####UPDATE####
#
#:# zpool list
#NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
#raid-z  18.1T  13.6T  4.54T         -     0%    74%  1.00x  ONLINE  -
# bash-3.00# zpool list -H
# mypool  33.8G   84.5K   33.7G   0%      ONLINE  -
# bash-3.00#


zpool list -H | while read name size alloc free expandsz frag cap dedup health altroot
do
  LINE_COLOR="green"

  if [ "${health}" == "ONLINE" ]; then
    HEALTH_COLOR="green"
  elif [ "${health}" == "DEGRADED" ]; then
    HEALTH_COLOR="red"
  elif [ "${health}" == "FAULTED" ]; then
    HEALTH_COLOR="red"
  fi


  CAP_COLOR="green"
  cap=`echo ${cap} | cut -d% -f1`
  if [ "${cap}" -ge $DISKYELL ]; then
    CAP_COLOR="yellow"
  elif [ "${cap}" -gt $DISKRED ]; then
    CAP_COLOR="red"
  fi
  DATA=$(echo "$name : $cap\n$DATA")

# Determine the line colours
  [ "$HEALTH_COLOR" == "yellow" -o "$CAP_COLOR" == "yellow" ] && LINE_COLOR="yellow"
  [ "$HEALTH_COLOR" == "red" -o "$CAP_COLOR" == "red" ] && LINE_COLOR="red"

# Determine the messages
  [ "$HEALTH_COLOR" == "red" -a "$FIRST_LINE_HEALTH" != "faulted" ] && FIRST_LINE_HEALTH="DEGRADED"
  [ "$HEALTH_COLOR" == "red" -a "$FIRST_LINE_HEALTH" != "DEGRADED" ] && FIRST_LINE_HEALTH="FAULTED"
  [ "$CAP_COLOR" == "yellow" -a "$FIRST_LINE_CAP" != "full" ] && FIRST_LINE_CAP="nearly full"
  [ "$CAP_COLOR" = "yellow" ] && FIRST_LINE_CAP="full"

#Determine the final colour status
  [ "$LINE_COLOR" == "yellow" -a "$DISPCOLOR" != "red" ] && DISPCOLOR="yellow"
  [ "$LINE_COLOR" == "red" ] && DISPCOLOR="red"

STRING="$STRING <tr><td>&${LINE_COLOR}</td><td>${name}</td><td>${health}</td><td>${alloc}</td><td>${free}</td><td>${cap} %</td></tr>"
done
DATA=$(echo "$DATA \n\n")

STRING="$STRING </table><br><br>"
STRING="$STRING`zpool status`"
FIRST_LINE="zfs - Health Report: $FIRST_LINE_HEALTH - Capacity Report: $FIRST_LINE_CAP"

# Snapshot check
if [ "$CHECKSNAPS" = "true" ]
then
   NOW=$(perl -e 'print time(), "\n" ')
   SNAPLIST=$(zfs list | grep @ |awk '{ print $1 }' \
                       | while read a
                         do
                            echo "$(zfs get -H -o name,value -p creation $a) $( zfs get -H -o value used $a)"
                         done)
   if [ -z "$SNAPLIST" ]
   then
      SNAPTABLE="<br>&green No snapshots found"
   else
      SNAPTABLE="<table border=1 cellpadding=10><tr><th></th><th>Snapshot</th><th>Age</th><th>Size</th></tr>"
      echo "$SNAPLIST" | while read SNAPSHOT CREATION SIZE
      do
         LINE_COLOR=green
         ((SNAPREDS=SNAPRED*43200))
         ((SNAPYELLS=SNAPYELL*43200))
         ((AGES=NOW-CREATION)) # AGES=Age in seconds
         [ $AGES -gt $SNAPYELLS ] && LINE_COLOR=yellow
         [ $AGES -gt $SNAPREDS ] && LINE_COLOR=red
         [ $AGES -gt 120 ] && ((AGE=AGES/60))&& AGE="$AGE Minutes"
         [ $AGES -gt 7200 ] && ((AGE=AGES/3600)) && AGE="$AGE Hours"
         [ $AGES -gt 172800 ] && ((AGE=AGES/86400)) && AGE="$AGE Days"
         SNAPTABLE="$SNAPTABLE <tr><td>&${LINE_COLOR}</td><td>${SNAPSHOT}</td><td>${AGE}</td><td>${SIZE}</td></tr>"
         if [ "$SNAPCOL" = "true" ] #Only if true will it effect test colour
         then
            if [ "$DISPCOLOR" != "red" ] # If it's already red, it's not getting any worse
            then
               [ "$LINE_COLOR" != "green" ] && DISPCOLOR=$LINE_COLOR
            fi
         fi
      done
      SNAPTABLE="$SNAPTABLE </table><br><br>"
   fi
   STRING="$STRING <br><br><B>SNAPSHOT STATUS</B> $SNAPTABLE"
fi
# What: Sent out the final bb message to hobbit server.
$BB $BBDISP "status $MACHINE.$TEST $DISPCOLOR `date` $FIRST_LINE $STRING"
$BB $BBDISP "data $MACHINE.$TEST $DISPCOLOR $(echo; echo "$DATA" ;echo;echo)"
</code>
</hidden>

===== Known  Bugs and Issues =====
None known at this stage.
===== To Do =====
Go for lunch.
===== Credits =====
I am eternally grateful to those that have assisted in creating this script. All I have done is add a little extra functionality to monitor snapshots and a little bit of graphing.

From the script, it appears credit is owed to 
  * Mike Rowell <Mike.Rowell@Rightmove.co.uk>, original
  * Uwe Kirbach <U.Kirbach@EnBW.com>
  * T.J. Yang: add in some comments.


===== Changelog =====

  * **2010-06-10**
    * Put it on Xymonton with snapshot feature
  * **2010-07-21**
    * Now with graphing goodness
  * **2011-05-14**
    * Fixed a minor code bug pointed out by Patrik Nilsson (Damn Quotes)\\ It appears there is a difference between THIS="THAT" and\\ <code> THIS="THAT" </code> At least if you cut and paste from this page there is. :-( \\ 
  * **2013-07-16**
    * Changed graphing FNPATTERN to **^**zfs,(.*).rrd\\ Without the ^ it will pick up and try and graph any rrd file with the string "zfs," in the file name. 
  * **2020-04-20**
    * Update to new "zpool list -h" format
    * Add Available/Free to table
    * Full "zpool status" always displayed
    * Set DEGRADED=RED
    * DISKYELL=90 & DISKRED=95