monitors:6540

6540.ksh

Author Vernon Everett
Compatibility Xymon 4.2 (Will work on others, but needs SPLITNCV for some graphs)
Requirements Solaris CAMS client - SUNWstkcam
Download None
Last Update 2010-09-08

We needed some visibility into what's happening on the 6540 without giving too many people access to CAMS. Slowly we added more functionality, and this is the result.

The script queries the CAMS application, so it might work on any other SUN storage array that is registered in CAMS.

If somebody wants to test this and give feedback, I would appreciate it.

Client side

Download 6540.ksh and put it into ~hobbit/client/ext on the CAMS client system. You will need to make sure you have the CAMS CLI installed on the box, and that hobbit has sudo rights to run the appropriate commands without password. Add this to sudoers

hobbit          hostname  =   (ALL)NOPASSWD: /opt/SUNWstkcam/bin/sscs

Add the following code to ~/hobbit/client/etc/clientlaunch.cfg

[6540]
      ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
      CMD $HOBBITCLIENTHOME/ext/6540.ksh
      LOGFILE $HOBBITCLIENTHOME/logs/6540.log
      INTERVAL 5m

Server side

Add this to TEST2RRD= in hobbitserver.cfg

cache=ncv,iops=ncv,reads=ncv,writes=ncv,cntrlr=ncv,details=ncv

Add this to GRAPHS= in hobbitserver.cfg

cntrlr,cache::7,iops::7,reads::7,writes::7,details
# The ::7 indicates the number of lines on a each graph. Edit to taste.

Add this to hobbitserver.cfg

NCV_cntrlr="*:NONE,ctrlA:GAUGE,ctrlB:GAUGE"
NCV_details="*:NONE,cache:GAUGE,read:GAUGE,write:GAUGE"
SPLITNCV_cache="*:GAUGE"
SPLITNCV_iops="*:GAUGE"
SPLITNCV_reads="*:GAUGE"
SPLITNCV_writes="*:GAUGE"

Add this to hobbitgraph.cfg

[cache]
    FNPATTERN cache,(.*).rrd
    TITLE Cache Hits Percentage
    YAXIS % Hits
    DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
    LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
    GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
    GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
    GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
    GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[iops]
    FNPATTERN iops,(.*).rrd
    TITLE Total IOs Per Second
    YAXIS IOPS
    DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
    LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
    GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
    GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
    GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
    GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[reads]
    FNPATTERN reads,(.*).rrd
    TITLE Total READS Per Second
    YAXIS Reads/s
    DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
    LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
    GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
    GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
    GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
    GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[writes]
    FNPATTERN writes,(.*).rrd
    TITLE Total WRITES Per Second
    YAXIS Writes/s
    DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE 
    LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
    GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
    GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
    GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
    GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[cntrlr]
      TITLE Controller IOPs
      YAXIS IOPS
      DEF:a=cntrlr.rrd:ctrlA:AVERAGE
      DEF:b=cntrlr.rrd:ctrlB:AVERAGE
      LINE2:a#FF0000:Controller A
      LINE2:b#0000FF:Controller B
      -l 0
      COMMENT:\n
      GPRINT:a:LAST:Controller A \: %8.1lf (cur)
      GPRINT:a:MAX: \: %8.1lf (max)
      GPRINT:a:MIN: \: %8.1lf (min)
      GPRINT:a:AVERAGE: \: %8.1lf (avg)\n
      GPRINT:b:LAST:Controller B \: %8.1lf (cur)
      GPRINT:b:MAX: \: %8.1lf (max)
      GPRINT:b:MIN: \: %8.1lf (min)
      GPRINT:b:AVERAGE: \: %8.1lf (avg)\n

[details]
      TITLE PERFORMANCE SUMMAY
      YAXIS Percent
      DEF:a=details.rrd:cache:AVERAGE
      DEF:b=details.rrd:read:AVERAGE
      DEF:c=details.rrd:write:AVERAGE
      AREA:b#0000FF:% Reads
      STACK:c#00FF00:% Writes
      LINE2:a#FF0000:Cache Hits
      -l 0
      -u 100
      COMMENT:\n
      GPRINT:a:LAST:Cache Hits \: %8.1lf (cur)
      GPRINT:a:MAX: \: %8.1lf (max)
      GPRINT:a:MIN: \: %8.1lf (min)
      GPRINT:a:AVERAGE: \: %8.1lf (avg)\n
      GPRINT:b:LAST:Reads     \: %8.1lf (cur)
      GPRINT:b:MAX: \: %8.1lf (max)
      GPRINT:b:MIN: \: %8.1lf (min)
      GPRINT:b:AVERAGE: \: %8.1lf (avg)\n
      GPRINT:c:LAST:Writes    \: %8.1lf (cur)
      GPRINT:c:MAX: \: %8.1lf (max)
      GPRINT:c:MIN: \: %8.1lf (min)
      GPRINT:c:AVERAGE: \: %8.1lf (avg)\n

Show Code ⇲

Hide Code ⇱

#!/usr/bin/ksh
SSCS="/opt/csw/bin/sudo /opt/SUNWstkcam/bin/sscs"
ARRAYS=$($SSCS list array | cut -d" " -f2)
export COLUMNS=255
TMPFILE=$BBTMP/6540.tmp
SITE=$($SSCS list site)
EVENTS=$($SSCS list event)
for ARRAY in $ARRAYS
do
   #OS_SUPPORT=$($SSCS list -a $ARRAY os_type)
   #TRAYS=$($SSCS list -a $ARRAY tray)
   #VDISKS=$($SSCS list -a $ARRAY vdisk)

   COLOUR=green
   ARRAY_PERF=$($SSCS list -a $ARRAY --type array_stats performance)
   IP=$($SSCS list -a $ARRAY registeredarray)
   CACHE=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $12}')
   READ=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $3}')
   WRITE=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $4}')
   echo > $TMPFILE
   echo "PERFORMANCE SUMMARY" >> $TMPFILE
   echo "$ARRAY_PERF" >> $TMPFILE
   echo >> $TMPFILE
   echo "FRU SUMMARY" >> $TMPFILE
   $SSCS list -d $ARRAY  -s fru >> $TMPFILE
   echo >> $TMPFILE
   $SSCS list -a $ARRAY registeredarray >> $TMPFILE
   echo >> $TMPFILE
   echo "$SITE" >> $TMPFILE
   echo "<font size =1 color=\"black\">" >> $TMPFILE
   echo "cache:$CACHE" >> $TMPFILE
   echo "read:$READ" >> $TMPFILE
   echo "write:$WRITE" >> $TMPFILE
   echo "</font>" >> $TMPFILE

   $BB $BBDISP "status $ARRAY.details $COLOUR $(date) $(cat $TMPFILE)"

   COLOUR=green
   LOCALLOG=$(echo "$EVENTS" | grep $ARRAY | sed '1!G;h;$!d')
   $BB $BBDISP "status $ARRAY.logs $COLOUR $(date) $(echo;echo "$LOCALLOG")"


   COLOUR=green
   DISK=$($SSCS list -a $ARRAY disk)
   DISKSUMMARY=$(echo "$DISK" | grep "Status" | grep -v "Optimal")
   if [ -n "$DISKSUMMARY" ]
   then
      # If there disks left after removing optimal disks, then it must non-green.
      COLOUR="yellow"
      # Now let's find out how non-green
      DISKSUMMARY=$(echo "$DISK" | grep "Status" | egrep -v "Optimal|Replaced")
      # Take out green and yellow, if we still have something, it must be red.
      [ -n "$DISKSUMMARY" ] && COLOUR=red
   fi
   echo "$DISK" | sed "s/Status: *Bypassed/\&red &/g;s/Status: *Failed/\&red &/g;s/Status: *Optimal/\&green &/g;s/Status: *Replaced/\&yellow &/g" > $TMPFILE.disk
   # We know most of these states. Bypassed is special. It is detailed under bug id 6371462
   # Search for it on sunsolve or here http://dlc.sun.com/pdf/819-6521-14/819-6521-14.pdf
   # Particularly useless workaround.
   # "Power off the arrays and set the link rate switch back to 2 Gb/s." WTF!
   $BB $BBDISP "status $ARRAY.disks $COLOUR $(date) $(echo; cat $TMPFILE.disk )"
   rm $TMPFILE.disk

   COLOUR=green
   FIRMWARE=$($SSCS list -a $ARRAY firmware)
   echo "$FIRMWARE" | grep "not at baseline" > /dev/null
   [ $? -eq 0 ] && COLOUR=yellow
   $BB $BBDISP "status $ARRAY.firmware $COLOUR $(date) $(echo;echo "$FIRMWARE" | \
                        sed "s/not at baseline.$/not at baseline. \&yellow/g")"
   COLOUR=green
   HOSTS=$($SSCS list -a $ARRAY host | while read a b
                                          do
                                             echo "<tr><td>$b</td></tr>"
                                          done)
   $BB $BBDISP "status $ARRAY.hosts $COLOUR $(date) $(echo;
                       echo "<table border=\"0\" cellpadding=\"2\"><tr><td><h3><b>Hosts\
                       </b></h3></td>$HOSTS</table>")"

   COLOUR=green
   HOSTGROUPS=$($SSCS list -a $ARRAY hostgroup|while read a b c
                                                  do
                                                      echo "<tr><td>$c</td></tr>"
                                                  done)
   $BB $BBDISP "status $ARRAY.hostgroups $COLOUR $(date) $(echo;\
                       echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Host Groups\
                             </b></h3></td>$HOSTGROUPS</table>") "

   COLOUR=red
   FCPORTS=$($SSCS list -a $ARRAY fcport)
   LINKSUP=$(echo "$FCPORTS" | grep "Link Status" | grep -v "Up")
   [ -z $LINKSUP ] && COLOUR=green
   $BB $BBDISP "status $ARRAY.fcports $COLOUR $(date) $(echo;echo "$FCPORTS")"

   COLOUR=green
   LICENSES=$($SSCS list -a $ARRAY license)
   $BB $BBDISP "status $ARRAY.licenses $COLOUR $(date) $(echo;echo "$LICENSES")"

   COLOUR=green
   REPL=$($SSCS list -a $ARRAY repset)
   echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Replication Set</b></h3></td>" > $TMPFILE
   echo "<td><h3><b>Consistency group</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Remote volume</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Replication peer</b></h3></td></tr>" >> $TMPFILE
   echo "$REPL" |  sed "s/^Replication set: /<tr><td>/g;
                        s/Consistency group: /<\/td><td>/g;\
                        s/Remote volume: /<\/td><td>/g;\
                        s/Replication peer: /<\/td><td>/g;\
                        s/$/<\/td><\/tr>/g" >> $TMPFILE
   echo "</table> " >> $TMPFILE
   $BB $BBDISP "status $ARRAY.replication $COLOUR $(date) $(echo;cat $TMPFILE)"
   rm $TMPFILE

   COLOUR=green
   SNAPSHOTS=$($SSCS list -a $ARRAY snapshot)
   [ -z $SNAPSHOTS ] && SNAPSHOTS="No snapshots found"
   $BB $BBDISP "status $ARRAY.snapshots $COLOUR $(date) $(echo;echo "$SNAPSHOTS")"

   COLOUR=green
   POOL=$($SSCS list -a $ARRAY pool)
   >$TMPFILE
   echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Pool</b></h3></td>" > $TMPFILE
   echo "<td><h3><b>Profile</b></h3></td><td><h3><b>Capacity</b></h3></td></tr>" >> $TMPFILE
   echo "$POOL" | sed "s/^Pool: /<tr><td>/g;
                       s/Profile: /<\/td><td>/g;\
                       s/Configured Capacity: /<\/td><td>/g;\
                       s/$/<\/td><\/tr>/g" >> $TMPFILE
   echo "</table> " >> $TMPFILE
   $BB $BBDISP "status $ARRAY.pools $COLOUR $(date) $(echo;cat $TMPFILE)"
   rm $TMPFILE

   COLOUR=green
   VOLUMES=$($SSCS list -a $ARRAY volume)
   >$TMPFILE
   echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Volume</b></h3></td>" > $TMPFILE
   echo "<td><h3><b>Type</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Pool</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Profile</b></h3></td></tr>" >> $TMPFILE
   echo "$VOLUMES" | sed "s/^Volume: /<tr><td>/g;
                          s/Type: /<\/td><td>/g;\
                          s/Pool: /<\/td><td>/g;\
                          s/Profile: /<\/td><td>/g;\
                          s/$/<\/td><\/tr>/g" >> $TMPFILE
   echo "</table> " >> $TMPFILE
   $BB $BBDISP "status $ARRAY.volumes $COLOUR $(date) $(echo;cat $TMPFILE)"
   rm $TMPFILE

   COLOUR=green
   >$TMPFILE
   export TERM=xterm-color
   echo "<table border=\"1\" cellpadding=\"10\"><tr>" > $TMPFILE
   echo "<td><h3><b>Name</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>FRU Type</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Alarm</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>State</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Status</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Firmware</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Unique ID</b></h3></td></tr>" >> $TMPFILE
   #FRU=$($SSCS list -d $ARRAY -t all fru)
   #echo "$FRU" > /tmp/fru2
   #$SSCS list -d $ARRAY -t all fru  > /tmp/fru
   $SSCS list -d $ARRAY -t all fru | egrep -v -- "^Name|Removed|-----" \
                                   | while read NAME B C D E F G
                                     do
                                        if [ "$E" == "OK" ]
                                        then
                                           E="$E &green"
                                        else
                                           if [ "$B" == "Battery" -o "$B" == "Disk" ]
                                           then
                                              E="$E &yellow"
                                              [ "$COLOUR" != "red" ] && COLOUR=yellow
                                           else
                                              E="$E &red"
                                              COLOUR=red
                                           fi
                                        fi
                                        echo "<tr><td>$NAME</td>" >> $TMPFILE
                                        echo "<td>$B</td>" >> $TMPFILE
                                        echo "<td>$C</td>" >> $TMPFILE
                                        echo "<td>$D</td>" >> $TMPFILE
                                        echo "<td>$E</td>" >> $TMPFILE
                                        echo "<td>$F</td>" >> $TMPFILE
                                        echo "<td>$G</td></tr>" >> $TMPFILE
                                     done
   echo "</table> " >> $TMPFILE
   $BB $BBDISP "status $ARRAY.fru $COLOUR $(date) $(echo;cat $TMPFILE)"

   COLOUR=red
   CONTROLLERS=$($SSCS list -a $ARRAY controller)
   CONSUMMARY=$(echo "$CONTROLLERS" | grep "Status" | grep -v "Optimal")
   [ -z "$CONSUMMARY" ] && COLOUR=green
   #$BB $BBDISP "status $ARRAY.cntrlr $COLOUR $(date) $(echo;echo "$CONTROLLERS")"
   HEADING="NAME^TOT IOPS^READ%^WRITE%^TOT DATA^AVG R SIZE^READS/s^PEAK R/s^AVG W SIZE^WRITES/s^PEAK W/s^CACHE%"
   CTRL_PERF=$($SSCS list -a $ARRAY --type controller_stats performance |  egrep -v "^NAME|^=====|^$" \
                                                                        | sed "s/[ \t][ \t]*/\^/g")
   echo "<table border=\"1\" cellpadding=\"10\"><tr>" > $TMPFILE
   echo "<td><h3><b>Stat</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Controller A</b></h3></td>" >> $TMPFILE
   echo "<td><h3><b>Controller B</b></h3></td>" >> $TMPFILE
   echo "</tr>" >> $TMPFILE
   for i in 2 3 4 5 6 7 8 9 10 11 12
   do
       echo "<tr>" >> $TMPFILE
       echo "<td>$(echo "$HEADING" | cut -d"^" -f$i)</td>" >> $TMPFILE
       echo "<td>$(echo "$CTRL_PERF" | grep ^A | cut -d"^" -f$i )</td>" >> $TMPFILE
       echo "<td>$(echo "$CTRL_PERF" | grep ^B | cut -d"^" -f$i)</td>" >> $TMPFILE
       echo "</tr>" >> $TMPFILE
   done
   echo "</table> " >> $TMPFILE
   echo "<font size =1 color=\"black\">" >> $TMPFILE
   echo ctrlA:$(echo "$CTRL_PERF" | grep ^A | cut -d"^" -f2 ) >> $TMPFILE
   echo ctrlB:$(echo "$CTRL_PERF" | grep ^B | cut -d"^" -f2 ) >> $TMPFILE
   echo "</font size>" >> $TMPFILE
   $BB $BBDISP "status $ARRAY.cntrlr $COLOUR $(date) $(echo;cat $TMPFILE;echo ;echo "$CONTROLLERS")"

#Performance Field descriptors of output from
# $SSCS list -a $ARRAY --type controller_stats performance
# $SSCS list -a $ARRAY --type array_stats performance
# $SSCS list -a $ARRAY --type volume_stats performance
# 1  NAME
# 2  TOT IOPS
# 3  READ %
# 4  WRITE%
# 5  TOT DATA
# 6  AVG R SIZE
# 7  READS/s
# 8  PEAK R/s
# 9  AVG W SIZE
# 10 WRITES/s
# 11 PEAK W/s
# 12 CACHE%

   # Cache Hits
   COLOUR=green
   CACHE=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$12 }')
   $BB $BBDISP "data $ARRAY.cache $COLOUR $(date) $(echo;echo "$CACHE")"
   # IOPS
   IOPS=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$2 }')
   $BB $BBDISP "data $ARRAY.iops $COLOUR $(date) $(echo;echo "$IOPS")"
   # READS
   READS=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$7 }')
   $BB $BBDISP "data $ARRAY.reads $COLOUR $(date) $(echo;echo "$READS")"
   # WRITES
   WRITES=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$10 }')
   $BB $BBDISP "data $ARRAY.writes $COLOUR $(date) $(echo;echo "$WRITES")"
done

Sometimes, the FRU listing

$SSCS list -d $ARRAY -t all fru 

wraps columns, causing the output to be unparsable, and making the resulting table look like absolute garbage.

* iSCSI support. We don't use it. Somebody want to add iSCSI support?

* SAS support. See iSCSI above.

* Get Sun/Oracle to fix a CAMS bug that sometimes causes controller B to show 0 for all values, even though there are plenty LUNs using it as their preferred controller. This shows up on the CAMS GUI too.

  • Update : 2010-07-21 Sun/Oracle BugID 6967353 created.

* Figure out why sometimes the FRU listing wraps columns. See Bugs above.

Can't really blame anybody except myself for this one.

  • monitors/6540.txt
  • Last modified: 2010/09/08 09:01
  • by 127.0.0.1