====== 6540.ksh ====== ^ Author | [[ everett.vernon@gmail.com | Vernon Everett ]] | ^ Compatibility | Xymon 4.2 (Will work on others, but needs SPLITNCV for some graphs) | ^ Requirements | Solaris CAMS client - SUNWstkcam | ^ Download | None | ^ Last Update | 2010-09-08 | ===== Description ===== We needed some visibility into what's happening on the 6540 without giving too many people access to CAMS. Slowly we added more functionality, and this is the result. The script queries the CAMS application, so it might work on any other SUN storage array that is registered in CAMS. If somebody wants to test this and give feedback, I would appreciate it. ===== Installation ===== === Client side === Download 6540.ksh and put it into ~hobbit/client/ext on the CAMS client system. You will need to make sure you have the CAMS CLI installed on the box, and that hobbit has sudo rights to run the appropriate commands without password. Add this to sudoers hobbit hostname = (ALL)NOPASSWD: /opt/SUNWstkcam/bin/sscs Add the following code to ~/hobbit/client/etc/clientlaunch.cfg [6540] ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg CMD $HOBBITCLIENTHOME/ext/6540.ksh LOGFILE $HOBBITCLIENTHOME/logs/6540.log INTERVAL 5m === Server side === Add this to TEST2RRD= in hobbitserver.cfg cache=ncv,iops=ncv,reads=ncv,writes=ncv,cntrlr=ncv,details=ncv Add this to GRAPHS= in hobbitserver.cfg cntrlr,cache::7,iops::7,reads::7,writes::7,details # The ::7 indicates the number of lines on a each graph. Edit to taste. Add this to hobbitserver.cfg NCV_cntrlr="*:NONE,ctrlA:GAUGE,ctrlB:GAUGE" NCV_details="*:NONE,cache:GAUGE,read:GAUGE,write:GAUGE" SPLITNCV_cache="*:GAUGE" SPLITNCV_iops="*:GAUGE" SPLITNCV_reads="*:GAUGE" SPLITNCV_writes="*:GAUGE" Add this to hobbitgraph.cfg [cache] FNPATTERN cache,(.*).rrd TITLE Cache Hits Percentage YAXIS % Hits DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n [iops] FNPATTERN iops,(.*).rrd TITLE Total IOs Per Second YAXIS IOPS DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n [reads] FNPATTERN reads,(.*).rrd TITLE Total READS Per Second YAXIS Reads/s DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n [writes] FNPATTERN writes,(.*).rrd TITLE Total WRITES Per Second YAXIS Writes/s DEF:p@RRDIDX@=@RRDFN@:lambda:AVERAGE LINE1:p@RRDIDX@#@COLOR@:@RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:MIN: \: %5.1lf (min) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n [cntrlr] TITLE Controller IOPs YAXIS IOPS DEF:a=cntrlr.rrd:ctrlA:AVERAGE DEF:b=cntrlr.rrd:ctrlB:AVERAGE LINE2:a#FF0000:Controller A LINE2:b#0000FF:Controller B -l 0 COMMENT:\n GPRINT:a:LAST:Controller A \: %8.1lf (cur) GPRINT:a:MAX: \: %8.1lf (max) GPRINT:a:MIN: \: %8.1lf (min) GPRINT:a:AVERAGE: \: %8.1lf (avg)\n GPRINT:b:LAST:Controller B \: %8.1lf (cur) GPRINT:b:MAX: \: %8.1lf (max) GPRINT:b:MIN: \: %8.1lf (min) GPRINT:b:AVERAGE: \: %8.1lf (avg)\n [details] TITLE PERFORMANCE SUMMAY YAXIS Percent DEF:a=details.rrd:cache:AVERAGE DEF:b=details.rrd:read:AVERAGE DEF:c=details.rrd:write:AVERAGE AREA:b#0000FF:% Reads STACK:c#00FF00:% Writes LINE2:a#FF0000:Cache Hits -l 0 -u 100 COMMENT:\n GPRINT:a:LAST:Cache Hits \: %8.1lf (cur) GPRINT:a:MAX: \: %8.1lf (max) GPRINT:a:MIN: \: %8.1lf (min) GPRINT:a:AVERAGE: \: %8.1lf (avg)\n GPRINT:b:LAST:Reads \: %8.1lf (cur) GPRINT:b:MAX: \: %8.1lf (max) GPRINT:b:MIN: \: %8.1lf (min) GPRINT:b:AVERAGE: \: %8.1lf (avg)\n GPRINT:c:LAST:Writes \: %8.1lf (cur) GPRINT:c:MAX: \: %8.1lf (max) GPRINT:c:MIN: \: %8.1lf (min) GPRINT:c:AVERAGE: \: %8.1lf (avg)\n ===== Source ===== ==== 6540.ksh ==== #!/usr/bin/ksh SSCS="/opt/csw/bin/sudo /opt/SUNWstkcam/bin/sscs" ARRAYS=$($SSCS list array | cut -d" " -f2) export COLUMNS=255 TMPFILE=$BBTMP/6540.tmp SITE=$($SSCS list site) EVENTS=$($SSCS list event) for ARRAY in $ARRAYS do #OS_SUPPORT=$($SSCS list -a $ARRAY os_type) #TRAYS=$($SSCS list -a $ARRAY tray) #VDISKS=$($SSCS list -a $ARRAY vdisk) COLOUR=green ARRAY_PERF=$($SSCS list -a $ARRAY --type array_stats performance) IP=$($SSCS list -a $ARRAY registeredarray) CACHE=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $12}') READ=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $3}') WRITE=$(echo "$ARRAY_PERF" | grep -v "^$" | sed -n '$p' | awk '{ print $4}') echo > $TMPFILE echo "PERFORMANCE SUMMARY" >> $TMPFILE echo "$ARRAY_PERF" >> $TMPFILE echo >> $TMPFILE echo "FRU SUMMARY" >> $TMPFILE $SSCS list -d $ARRAY -s fru >> $TMPFILE echo >> $TMPFILE $SSCS list -a $ARRAY registeredarray >> $TMPFILE echo >> $TMPFILE echo "$SITE" >> $TMPFILE echo "" >> $TMPFILE echo "cache:$CACHE" >> $TMPFILE echo "read:$READ" >> $TMPFILE echo "write:$WRITE" >> $TMPFILE echo "" >> $TMPFILE $BB $BBDISP "status $ARRAY.details $COLOUR $(date) $(cat $TMPFILE)" COLOUR=green LOCALLOG=$(echo "$EVENTS" | grep $ARRAY | sed '1!G;h;$!d') $BB $BBDISP "status $ARRAY.logs $COLOUR $(date) $(echo;echo "$LOCALLOG")" COLOUR=green DISK=$($SSCS list -a $ARRAY disk) DISKSUMMARY=$(echo "$DISK" | grep "Status" | grep -v "Optimal") if [ -n "$DISKSUMMARY" ] then # If there disks left after removing optimal disks, then it must non-green. COLOUR="yellow" # Now let's find out how non-green DISKSUMMARY=$(echo "$DISK" | grep "Status" | egrep -v "Optimal|Replaced") # Take out green and yellow, if we still have something, it must be red. [ -n "$DISKSUMMARY" ] && COLOUR=red fi echo "$DISK" | sed "s/Status: *Bypassed/\&red &/g;s/Status: *Failed/\&red &/g;s/Status: *Optimal/\&green &/g;s/Status: *Replaced/\&yellow &/g" > $TMPFILE.disk # We know most of these states. Bypassed is special. It is detailed under bug id 6371462 # Search for it on sunsolve or here http://dlc.sun.com/pdf/819-6521-14/819-6521-14.pdf # Particularly useless workaround. # "Power off the arrays and set the link rate switch back to 2 Gb/s." WTF! $BB $BBDISP "status $ARRAY.disks $COLOUR $(date) $(echo; cat $TMPFILE.disk )" rm $TMPFILE.disk COLOUR=green FIRMWARE=$($SSCS list -a $ARRAY firmware) echo "$FIRMWARE" | grep "not at baseline" > /dev/null [ $? -eq 0 ] && COLOUR=yellow $BB $BBDISP "status $ARRAY.firmware $COLOUR $(date) $(echo;echo "$FIRMWARE" | \ sed "s/not at baseline.$/not at baseline. \&yellow/g")" COLOUR=green HOSTS=$($SSCS list -a $ARRAY host | while read a b do echo "$b" done) $BB $BBDISP "status $ARRAY.hosts $COLOUR $(date) $(echo; echo "$HOSTS

Hosts\

")" COLOUR=green HOSTGROUPS=$($SSCS list -a $ARRAY hostgroup|while read a b c do echo "$c" done) $BB $BBDISP "status $ARRAY.hostgroups $COLOUR $(date) $(echo;\ echo "$HOSTGROUPS

Host Groups\

") " COLOUR=red FCPORTS=$($SSCS list -a $ARRAY fcport) LINKSUP=$(echo "$FCPORTS" | grep "Link Status" | grep -v "Up") [ -z $LINKSUP ] && COLOUR=green $BB $BBDISP "status $ARRAY.fcports $COLOUR $(date) $(echo;echo "$FCPORTS")" COLOUR=green LICENSES=$($SSCS list -a $ARRAY license) $BB $BBDISP "status $ARRAY.licenses $COLOUR $(date) $(echo;echo "$LICENSES")" COLOUR=green REPL=$($SSCS list -a $ARRAY repset) echo "" > $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "$REPL" | sed "s/^Replication set: /

Replication Set

Consistency group

Remote volume

Replication peer

/g; s/Consistency group: /<\/td>/g;\ s/Remote volume: /<\/td>/g;\ s/Replication peer: /<\/td>/g;\ s/$/<\/td><\/tr>/g" >> $TMPFILE echo "
" >> $TMPFILE $BB $BBDISP "status $ARRAY.replication $COLOUR $(date) $(echo;cat $TMPFILE)" rm $TMPFILE COLOUR=green SNAPSHOTS=$($SSCS list -a $ARRAY snapshot) [ -z $SNAPSHOTS ] && SNAPSHOTS="No snapshots found" $BB $BBDISP "status $ARRAY.snapshots $COLOUR $(date) $(echo;echo "$SNAPSHOTS")" COLOUR=green POOL=$($SSCS list -a $ARRAY pool) >$TMPFILE echo "" > $TMPFILE echo "" >> $TMPFILE echo "$POOL" | sed "s/^Pool: /

Pool

Profile

Capacity

/g; s/Profile: /<\/td>/g;\ s/Configured Capacity: /<\/td>/g;\ s/$/<\/td><\/tr>/g" >> $TMPFILE echo "
" >> $TMPFILE $BB $BBDISP "status $ARRAY.pools $COLOUR $(date) $(echo;cat $TMPFILE)" rm $TMPFILE COLOUR=green VOLUMES=$($SSCS list -a $ARRAY volume) >$TMPFILE echo "" > $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "$VOLUMES" | sed "s/^Volume: /

Volume

Type

Pool

Profile

/g; s/Type: /<\/td>/g;\ s/Pool: /<\/td>/g;\ s/Profile: /<\/td>/g;\ s/$/<\/td><\/tr>/g" >> $TMPFILE echo "
" >> $TMPFILE $BB $BBDISP "status $ARRAY.volumes $COLOUR $(date) $(echo;cat $TMPFILE)" rm $TMPFILE COLOUR=green >$TMPFILE export TERM=xterm-color echo "" > $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE #FRU=$($SSCS list -d $ARRAY -t all fru) #echo "$FRU" > /tmp/fru2 #$SSCS list -d $ARRAY -t all fru > /tmp/fru $SSCS list -d $ARRAY -t all fru | egrep -v -- "^Name|Removed|-----" \ | while read NAME B C D E F G do if [ "$E" == "OK" ] then E="$E &green" else if [ "$B" == "Battery" -o "$B" == "Disk" ] then E="$E &yellow" [ "$COLOUR" != "red" ] && COLOUR=yellow else E="$E &red" COLOUR=red fi fi echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE done echo "

Name

FRU Type

Alarm

State

Status

Firmware

Unique ID

$NAME$B$C$D$E$F$G
" >> $TMPFILE $BB $BBDISP "status $ARRAY.fru $COLOUR $(date) $(echo;cat $TMPFILE)" COLOUR=red CONTROLLERS=$($SSCS list -a $ARRAY controller) CONSUMMARY=$(echo "$CONTROLLERS" | grep "Status" | grep -v "Optimal") [ -z "$CONSUMMARY" ] && COLOUR=green #$BB $BBDISP "status $ARRAY.cntrlr $COLOUR $(date) $(echo;echo "$CONTROLLERS")" HEADING="NAME^TOT IOPS^READ%^WRITE%^TOT DATA^AVG R SIZE^READS/s^PEAK R/s^AVG W SIZE^WRITES/s^PEAK W/s^CACHE%" CTRL_PERF=$($SSCS list -a $ARRAY --type controller_stats performance | egrep -v "^NAME|^=====|^$" \ | sed "s/[ \t][ \t]*/\^/g") echo "" > $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE for i in 2 3 4 5 6 7 8 9 10 11 12 do echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE echo "" >> $TMPFILE done echo "

Stat

Controller A

Controller B

$(echo "$HEADING" | cut -d"^" -f$i)$(echo "$CTRL_PERF" | grep ^A | cut -d"^" -f$i )$(echo "$CTRL_PERF" | grep ^B | cut -d"^" -f$i)
" >> $TMPFILE echo "" >> $TMPFILE echo ctrlA:$(echo "$CTRL_PERF" | grep ^A | cut -d"^" -f2 ) >> $TMPFILE echo ctrlB:$(echo "$CTRL_PERF" | grep ^B | cut -d"^" -f2 ) >> $TMPFILE echo "" >> $TMPFILE $BB $BBDISP "status $ARRAY.cntrlr $COLOUR $(date) $(echo;cat $TMPFILE;echo ;echo "$CONTROLLERS")" #Performance Field descriptors of output from # $SSCS list -a $ARRAY --type controller_stats performance # $SSCS list -a $ARRAY --type array_stats performance # $SSCS list -a $ARRAY --type volume_stats performance # 1 NAME # 2 TOT IOPS # 3 READ % # 4 WRITE% # 5 TOT DATA # 6 AVG R SIZE # 7 READS/s # 8 PEAK R/s # 9 AVG W SIZE # 10 WRITES/s # 11 PEAK W/s # 12 CACHE% # Cache Hits COLOUR=green CACHE=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$12 }') $BB $BBDISP "data $ARRAY.cache $COLOUR $(date) $(echo;echo "$CACHE")" # IOPS IOPS=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$2 }') $BB $BBDISP "data $ARRAY.iops $COLOUR $(date) $(echo;echo "$IOPS")" # READS READS=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$7 }') $BB $BBDISP "data $ARRAY.reads $COLOUR $(date) $(echo;echo "$READS")" # WRITES WRITES=$($SSCS list -a $ARRAY --type volume_stats performance | egrep -v "^$|====|^NAME" | awk '{ print $1" : "$10 }') $BB $BBDISP "data $ARRAY.writes $COLOUR $(date) $(echo;echo "$WRITES")" done ===== Known Bugs and Issues ===== Sometimes, the FRU listing $SSCS list -d $ARRAY -t all fru wraps columns, causing the output to be unparsable, and making the resulting table look like absolute garbage. ===== To Do ===== * iSCSI support. We don't use it. Somebody want to add iSCSI support? * SAS support. See iSCSI above. * Get Sun/Oracle to fix a CAMS bug that sometimes causes controller B to show 0 for all values, even though there are plenty LUNs using it as their preferred controller. This shows up on the CAMS GUI too. * **Update :** 2010-07-21 Sun/Oracle BugID 6967353 created. * Figure out why sometimes the FRU listing wraps columns. See Bugs above. ===== Credits ===== Can't really blame anybody except myself for this one. ===== Changelog ===== * **2010-07-12** * Initial release * **2010-09-08** * Updated to cater for the curious disk state "Bypassed" See bugId 6371462 here http://dlc.sun.com/pdf/819-6521-14/819-6521-14.pdf