Author | Vernon Everett |
Compatibility | Xymon 4.2 (Will work on others, but needs SPLITNCV for some graphs) |
Requirements | Solaris CAMS client - SUNWstkcam |
Download | None |
Last Update | 2010-09-08 |
We needed some visibility into what's happening on the 6540 without giving too many people access to CAMS. Slowly we added more functionality, and this is the result.
The script queries the CAMS application, so it might work on any other SUN storage array that is registered in CAMS.
If somebody wants to test this and give feedback, I would appreciate it.
Client side
Download 6540.ksh and put it into ~hobbit/client/ext on the CAMS client system.
You will need to make sure you have the CAMS CLI installed on the box, and that hobbit has sudo rights to run the appropriate commands without password.
Add this to sudoers
hobbit hostname = (ALL)NOPASSWD: /opt/SUNWstkcam/bin/sscs
Add the following code to ~/hobbit/client/etc/clientlaunch.cfg
[6540]
ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
CMD $HOBBITCLIENTHOME/ext/6540.ksh
LOGFILE $HOBBITCLIENTHOME/logs/6540.log
INTERVAL 5m
Server side
Add this to TEST2RRD= in hobbitserver.cfg
cache=ncv,iops=ncv,reads=ncv,writes=ncv,cntrlr=ncv,details=ncv
Add this to GRAPHS= in hobbitserver.cfg
cntrlr,cache::7,iops::7,reads::7,writes::7,details
# The ::7 indicates the number of lines on a each graph. Edit to taste.
Add this to hobbitserver.cfg
NCV_cntrlr="*:NONE,ctrlA:GAUGE,ctrlB:GAUGE"
NCV_details="*:NONE,cache:GAUGE,read:GAUGE,write:GAUGE"
SPLITNCV_cache="*:GAUGE"
SPLITNCV_iops="*:GAUGE"
SPLITNCV_reads="*:GAUGE"
SPLITNCV_writes="*:GAUGE"
Add this to hobbitgraph.cfg
[cache]
FNPATTERN cache,(.*).rrd
TITLE Cache Hits Percentage
YAXIS % Hits
DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
[iops]
FNPATTERN iops,(.*).rrd
TITLE Total IOs Per Second
YAXIS IOPS
DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
[reads]
FNPATTERN reads,(.*).rrd
TITLE Total READS Per Second
YAXIS Reads/s
DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
[writes]
FNPATTERN writes,(.*).rrd
TITLE Total WRITES Per Second
YAXIS Writes/s
DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE
LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@
GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
[cntrlr]
TITLE Controller IOPs
YAXIS IOPS
DEF:a=cntrlr.rrd:ctrlA:AVERAGE
DEF:b=cntrlr.rrd:ctrlB:AVERAGE
LINE2:a#FF0000:Controller A
LINE2:b#0000FF:Controller B
-l 0
COMMENT:\n
GPRINT:a:LAST:Controller A \: %8.1lf (cur)
GPRINT:a:MAX: \: %8.1lf (max)
GPRINT:a:MIN: \: %8.1lf (min)
GPRINT:a:AVERAGE: \: %8.1lf (avg)\n
GPRINT:b:LAST:Controller B \: %8.1lf (cur)
GPRINT:b:MAX: \: %8.1lf (max)
GPRINT:b:MIN: \: %8.1lf (min)
GPRINT:b:AVERAGE: \: %8.1lf (avg)\n
[details]
TITLE PERFORMANCE SUMMAY
YAXIS Percent
DEF:a=details.rrd:cache:AVERAGE
DEF:b=details.rrd:read:AVERAGE
DEF:c=details.rrd:write:AVERAGE
AREA:b#0000FF:% Reads
STACK:c#00FF00:% Writes
LINE2:a#FF0000:Cache Hits
-l 0
-u 100
COMMENT:\n
GPRINT:a:LAST:Cache Hits \: %8.1lf (cur)
GPRINT:a:MAX: \: %8.1lf (max)
GPRINT:a:MIN: \: %8.1lf (min)
GPRINT:a:AVERAGE: \: %8.1lf (avg)\n
GPRINT:b:LAST:Reads \: %8.1lf (cur)
GPRINT:b:MAX: \: %8.1lf (max)
GPRINT:b:MIN: \: %8.1lf (min)
GPRINT:b:AVERAGE: \: %8.1lf (avg)\n
GPRINT:c:LAST:Writes \: %8.1lf (cur)
GPRINT:c:MAX: \: %8.1lf (max)
GPRINT:c:MIN: \: %8.1lf (min)
GPRINT:c:AVERAGE: \: %8.1lf (avg)\n
#!/usr/bin/ksh
SSCS="/opt/csw/bin/sudo /opt/SUNWstkcam/bin/sscs"
ARRAYS=$($SSCS list array | cut -d" " -f2)
export COLUMNS=255
TMPFILE=$BBTMP/6540.tmp
SITE=$($SSCS list site)
EVENTS=$($SSCS list event)
for ARRAY in $ARRAYS
do
#OS_SUPPORT=$($SSCS list -a $ARRAY os_type)
#TRAYS=$($SSCS list -a $ARRAY tray)
#VDISKS=$($SSCS list -a $ARRAY vdisk)
COLOUR=green
ARRAY_PERF=$($SSCS list -a $ARRAY --type array_stats performance)
IP=$($SSCS list -a $ARRAY registeredarray)
CACHE=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $12}')
READ=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $3}')
WRITE=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $4}')
echo > $TMPFILE
echo "PERFORMANCE SUMMARY" >> $TMPFILE
echo "$ARRAY_PERF" >> $TMPFILE
echo >> $TMPFILE
echo "FRU SUMMARY" >> $TMPFILE
$SSCS list -d $ARRAY -s fru >> $TMPFILE
echo >> $TMPFILE
$SSCS list -a $ARRAY registeredarray >> $TMPFILE
echo >> $TMPFILE
echo "$SITE" >> $TMPFILE
echo "<font size =1 color=\"black\">" >> $TMPFILE
echo "cache:$CACHE" >> $TMPFILE
echo "read:$READ" >> $TMPFILE
echo "write:$WRITE" >> $TMPFILE
echo "</font>" >> $TMPFILE
$BB $BBDISP "status $ARRAY.details $COLOUR $(date) $(cat $TMPFILE)"
COLOUR=green
LOCALLOG=$(echo "$EVENTS" | grep $ARRAY | sed '1!G;h;$!d')
$BB $BBDISP "status $ARRAY.logs $COLOUR $(date) $(echo;echo "$LOCALLOG")"
COLOUR=green
DISK=$($SSCS list -a $ARRAY disk)
DISKSUMMARY=$(echo "$DISK" | grep "Status" | grep -v "Optimal")
if [ -n "$DISKSUMMARY" ]
then
# If there disks left after removing optimal disks, then it must non-green.
COLOUR="yellow"
# Now let's find out how non-green
DISKSUMMARY=$(echo "$DISK" | grep "Status" | egrep -v "Optimal|Replaced")
# Take out green and yellow, if we still have something, it must be red.
[ -n "$DISKSUMMARY" ] && COLOUR=red
fi
echo "$DISK" | sed "s/Status: *Bypassed/\&red &/g;s/Status: *Failed/\&red &/g;s/Status: *Optimal/\&green &/g;s/Status: *Replaced/\&yellow &/g" > $TMPFILE.disk
# We know most of these states. Bypassed is special. It is detailed under bug id 6371462
# Search for it on sunsolve or here http://dlc.sun.com/pdf/819-6521-14/819-6521-14.pdf
# Particularly useless workaround.
# "Power off the arrays and set the link rate switch back to 2 Gb/s." WTF!
$BB $BBDISP "status $ARRAY.disks $COLOUR $(date) $(echo; cat $TMPFILE.disk )"
rm $TMPFILE.disk
COLOUR=green
FIRMWARE=$($SSCS list -a $ARRAY firmware)
echo "$FIRMWARE" | grep "not at baseline" > /dev/null
[ $? -eq 0 ] && COLOUR=yellow
$BB $BBDISP "status $ARRAY.firmware $COLOUR $(date) $(echo;echo "$FIRMWARE" | \
sed "s/not at baseline.$/not at baseline. \&yellow/g")"
COLOUR=green
HOSTS=$($SSCS list -a $ARRAY host | while read a b
do
echo "<tr><td>$b</td></tr>"
done)
$BB $BBDISP "status $ARRAY.hosts $COLOUR $(date) $(echo;
echo "<table border=\"0\" cellpadding=\"2\"><tr><td><h3><b>Hosts\
</b></h3></td>$HOSTS</table>")"
COLOUR=green
HOSTGROUPS=$($SSCS list -a $ARRAY hostgroup|while read a b c
do
echo "<tr><td>$c</td></tr>"
done)
$BB $BBDISP "status $ARRAY.hostgroups $COLOUR $(date) $(echo;\
echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Host Groups\
</b></h3></td>$HOSTGROUPS</table>") "
COLOUR=red
FCPORTS=$($SSCS list -a $ARRAY fcport)
LINKSUP=$(echo "$FCPORTS" | grep "Link Status" | grep -v "Up")
[ -z $LINKSUP ] && COLOUR=green
$BB $BBDISP "status $ARRAY.fcports $COLOUR $(date) $(echo;echo "$FCPORTS")"
COLOUR=green
LICENSES=$($SSCS list -a $ARRAY license)
$BB $BBDISP "status $ARRAY.licenses $COLOUR $(date) $(echo;echo "$LICENSES")"
COLOUR=green
REPL=$($SSCS list -a $ARRAY repset)
echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Replication Set</b></h3></td>" > $TMPFILE
echo "<td><h3><b>Consistency group</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Remote volume</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Replication peer</b></h3></td></tr>" >> $TMPFILE
echo "$REPL" | sed "s/^Replication set: /<tr><td>/g;
s/Consistency group: /<\/td><td>/g;\
s/Remote volume: /<\/td><td>/g;\
s/Replication peer: /<\/td><td>/g;\
s/$/<\/td><\/tr>/g" >> $TMPFILE
echo "</table> " >> $TMPFILE
$BB $BBDISP "status $ARRAY.replication $COLOUR $(date) $(echo;cat $TMPFILE)"
rm $TMPFILE
COLOUR=green
SNAPSHOTS=$($SSCS list -a $ARRAY snapshot)
[ -z $SNAPSHOTS ] && SNAPSHOTS="No snapshots found"
$BB $BBDISP "status $ARRAY.snapshots $COLOUR $(date) $(echo;echo "$SNAPSHOTS")"
COLOUR=green
POOL=$($SSCS list -a $ARRAY pool)
>$TMPFILE
echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Pool</b></h3></td>" > $TMPFILE
echo "<td><h3><b>Profile</b></h3></td><td><h3><b>Capacity</b></h3></td></tr>" >> $TMPFILE
echo "$POOL" | sed "s/^Pool: /<tr><td>/g;
s/Profile: /<\/td><td>/g;\
s/Configured Capacity: /<\/td><td>/g;\
s/$/<\/td><\/tr>/g" >> $TMPFILE
echo "</table> " >> $TMPFILE
$BB $BBDISP "status $ARRAY.pools $COLOUR $(date) $(echo;cat $TMPFILE)"
rm $TMPFILE
COLOUR=green
VOLUMES=$($SSCS list -a $ARRAY volume)
>$TMPFILE
echo "<table border=\"1\" cellpadding=\"10\"><tr><td><h3><b>Volume</b></h3></td>" > $TMPFILE
echo "<td><h3><b>Type</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Pool</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Profile</b></h3></td></tr>" >> $TMPFILE
echo "$VOLUMES" | sed "s/^Volume: /<tr><td>/g;
s/Type: /<\/td><td>/g;\
s/Pool: /<\/td><td>/g;\
s/Profile: /<\/td><td>/g;\
s/$/<\/td><\/tr>/g" >> $TMPFILE
echo "</table> " >> $TMPFILE
$BB $BBDISP "status $ARRAY.volumes $COLOUR $(date) $(echo;cat $TMPFILE)"
rm $TMPFILE
COLOUR=green
>$TMPFILE
export TERM=xterm-color
echo "<table border=\"1\" cellpadding=\"10\"><tr>" > $TMPFILE
echo "<td><h3><b>Name</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>FRU Type</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Alarm</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>State</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Status</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Firmware</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Unique ID</b></h3></td></tr>" >> $TMPFILE
#FRU=$($SSCS list -d $ARRAY -t all fru)
#echo "$FRU" > /tmp/fru2
#$SSCS list -d $ARRAY -t all fru > /tmp/fru
$SSCS list -d $ARRAY -t all fru | egrep -v -- "^Name|Removed|-----" \
| while read NAME B C D E F G
do
if [ "$E" == "OK" ]
then
E="$E &green"
else
if [ "$B" == "Battery" -o "$B" == "Disk" ]
then
E="$E &yellow"
[ "$COLOUR" != "red" ] && COLOUR=yellow
else
E="$E &red"
COLOUR=red
fi
fi
echo "<tr><td>$NAME</td>" >> $TMPFILE
echo "<td>$B</td>" >> $TMPFILE
echo "<td>$C</td>" >> $TMPFILE
echo "<td>$D</td>" >> $TMPFILE
echo "<td>$E</td>" >> $TMPFILE
echo "<td>$F</td>" >> $TMPFILE
echo "<td>$G</td></tr>" >> $TMPFILE
done
echo "</table> " >> $TMPFILE
$BB $BBDISP "status $ARRAY.fru $COLOUR $(date) $(echo;cat $TMPFILE)"
COLOUR=red
CONTROLLERS=$($SSCS list -a $ARRAY controller)
CONSUMMARY=$(echo "$CONTROLLERS" | grep "Status" | grep -v "Optimal")
[ -z "$CONSUMMARY" ] && COLOUR=green
#$BB $BBDISP "status $ARRAY.cntrlr $COLOUR $(date) $(echo;echo "$CONTROLLERS")"
HEADING="NAME^TOT IOPS^READ%^WRITE%^TOT DATA^AVG R SIZE^READS/s^PEAK R/s^AVG W SIZE^WRITES/s^PEAK W/s^CACHE%"
CTRL_PERF=$($SSCS list -a $ARRAY --type controller_stats performance | egrep -v "^NAME|^=====|^$" \
| sed "s/[ \t][ \t]*/\^/g")
echo "<table border=\"1\" cellpadding=\"10\"><tr>" > $TMPFILE
echo "<td><h3><b>Stat</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Controller A</b></h3></td>" >> $TMPFILE
echo "<td><h3><b>Controller B</b></h3></td>" >> $TMPFILE
echo "</tr>" >> $TMPFILE
for i in 2 3 4 5 6 7 8 9 10 11 12
do
echo "<tr>" >> $TMPFILE
echo "<td>$(echo "$HEADING" | cut -d"^" -f$i)</td>" >> $TMPFILE
echo "<td>$(echo "$CTRL_PERF" | grep ^A | cut -d"^" -f$i )</td>" >> $TMPFILE
echo "<td>$(echo "$CTRL_PERF" | grep ^B | cut -d"^" -f$i)</td>" >> $TMPFILE
echo "</tr>" >> $TMPFILE
done
echo "</table> " >> $TMPFILE
echo "<font size =1 color=\"black\">" >> $TMPFILE
echo ctrlA:$(echo "$CTRL_PERF" | grep ^A | cut -d"^" -f2 ) >> $TMPFILE
echo ctrlB:$(echo "$CTRL_PERF" | grep ^B | cut -d"^" -f2 ) >> $TMPFILE
echo "</font size>" >> $TMPFILE
$BB $BBDISP "status $ARRAY.cntrlr $COLOUR $(date) $(echo;cat $TMPFILE;echo ;echo "$CONTROLLERS")"
#Performance Field descriptors of output from
# $SSCS list -a $ARRAY --type controller_stats performance
# $SSCS list -a $ARRAY --type array_stats performance
# $SSCS list -a $ARRAY --type volume_stats performance
# 1 NAME
# 2 TOT IOPS
# 3 READ %
# 4 WRITE%
# 5 TOT DATA
# 6 AVG R SIZE
# 7 READS/s
# 8 PEAK R/s
# 9 AVG W SIZE
# 10 WRITES/s
# 11 PEAK W/s
# 12 CACHE%
# Cache Hits
COLOUR=green
CACHE=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$12 }')
$BB $BBDISP "data $ARRAY.cache $COLOUR $(date) $(echo;echo "$CACHE")"
# IOPS
IOPS=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$2 }')
$BB $BBDISP "data $ARRAY.iops $COLOUR $(date) $(echo;echo "$IOPS")"
# READS
READS=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$7 }')
$BB $BBDISP "data $ARRAY.reads $COLOUR $(date) $(echo;echo "$READS")"
# WRITES
WRITES=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$10 }')
$BB $BBDISP "data $ARRAY.writes $COLOUR $(date) $(echo;echo "$WRITES")"
done
Known Bugs and Issues
Sometimes, the FRU listing
$SSCS list -d $ARRAY -t all fru
wraps columns, causing the output to be unparsable, and making the resulting table look like absolute garbage.
* iSCSI support. We don't use it. Somebody want to add iSCSI support?
* SAS support. See iSCSI above.
* Get Sun/Oracle to fix a CAMS bug that sometimes causes controller B to show 0 for all values, even though there are plenty LUNs using it as their preferred controller. This shows up on the CAMS GUI too.
* Figure out why sometimes the FRU listing wraps columns. See Bugs above.
Can't really blame anybody except myself for this one.