====== flexlm.pl ====== ^ Author | [[ nelis@nlr.nl | Wim Nelis ]] | ^ Compatibility | Xymon 4.2 | ^ Requirements | Perl, unix | ^ Download | None | ^ Last Update | 2010-05-31 | ===== Description ===== Script flexlm.pl is a client-side Xymon extension, which monitors the status of the flexlm services and which measures the number of licences in use. Its main feature is that it does not depend on the names of the products or the names of their features. Even the graph definition does not depend on those names. Thus the measurement as well as the presented results will follow the changes in the set of products and features automatically. Script flexlm.pl generates two tests. The test "flexlm" shows the current status of each product on a server, as well as the users who use a license for a long period. The test "licences" shows the number of used and issued licenses for each {product,feature} combination. It will issue a warning (yellow status) if for at least one feature more than 90% of the licenses are in use. As there is one graph per {product,feature} pair, the number of features to be monitored should be limited in order to have a reasonable response. A reasonable upper limit is about 100 features. ===== Technical details ===== A challenge was to use only one short graph definition, which does show the names of the product and the feature. This is accomplished by using a two user-scripts at the Xymon server, one to write the data into the appropriate RRD and one to generate the title of a graph. The NCV data in test "licenses" uses names formatted like "./". The status-channel of test "licenses" is diverted to a script, which uses this naming convention to create one RRD per feature, named "licenses...rrd". When a graph is generated, a script is used to write " / " into the title of the graph. ===== Installation ===== === Client side === Copy script flexlm.pl, shown below in chapter "Source", into directory ~xymon/client/ext and set the permissions to "rwx" for the owner. A few installation constants within the script must be defined, while there are a few installation constants which might need modification. Hash %FlexServer defines the servers for which FLEXlm statistics are to be gathered as well as the name of those servers in Xymon. I've seen output of `lmstat-a` in which one server appeared both by (DNS) name and by IP address. Hash %FlexServer is used to map the IP address onto the host name. Scalar $lmstat defines the fully qualified name of program lmstat. In case not all licenses are shown in the output of `lmstat-a`, array @InpFil can be used to specify the port numbers which should be checked too. This array can also be used to monitor FLEXlm on servers on which it is not easy to install this script. Add the following section to the ~xymon/client/etc/clientlaunch.cfg: [flexlm] ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg CMD $HOBBITCLIENTHOME/ext/flexlm.pl LOGFILE $HOBBITCLIENTHOME/logs/flexlm.log INTERVAL 5m === Server side === Install the following script, named rrd_status.pl, in directory ~xymon/server/ext. However, it could be that there is already a script with a similar purpose for other tests: in that case merge it with the script below. #!/usr/bin/perl # # This script handles a list of NCV, send by a Xymon client and prepares it to # be stored in an RRA. This script is used in cases in which a fixed-size group # of two or more values should be put together into a single RRA. The algorithm # is specific for each test / client. # # This script is invoked with three parameters: the name of the host, the name # of the test and the name of the file containing the message sent by the # client, containing the NCV to be handled. # # Written by W.J.M. Nelis, nelis@nlr.nl, 2009.07 # use strict ; # # Installation constants. # ----------------------- # # %Struct defines the datasets of the various tests. # my %Struct= ( licenses => [ # Must be sorted! "DS:InUse:GAUGE:600:0:U\n", "DS:Total:GAUGE:600:0:U\n" ] ) ; # of %Struct # # Global variables. # ----------------- # my ( $HostName, $TestName, $FileName )= @ARGV ; # my %Var= () ; # Buffer area for the variables my ( $Line, @Line ) ; # List of values of one measurement my $key ; # Loop control variables # # Main program. # ------------- # # Handle test "licenses". # # An attempt has been undertaken to make this code a little bit more general. # The name of an NCV should consist of two names separated by "/". The first # name becomes (part of) the name of the RRA, the second name becomes the # name of the DS. The DS-ses are written in sorted order. # if ( $TestName eq 'licenses' ) { open( FH, '<', $FileName ) or die ; while ( ) { chomp ; next unless m/^([\w\.-]+)\/(\w+)\s+:\s+(\d+)\s*$/ ; $Var{$1}{$2}= $3 ; } # of while close( FH ) ; print @{$Struct{$TestName}} ; foreach $key ( sort keys %Var ) { @Line= () ; push @Line, $Var{$key}{$_} foreach ( sort keys %{$Var{$key}} ) ; print "$TestName.$key.rrd\n" ; print join( ":", @Line ) . "\n" ; } # of foreach } # of if exit 0 ; Script rrd_status.pl should be invoked whenever data for test "licenses" arrives. Change section [rrdstatus] in file ~xymon/server/etc/hobbitlaunch.cfg to something similar to the following: [rrdstatus] ENVFILE /home/xymon/server/etc/hobbitserver.cfg NEEDS hobbitd CMD hobbitd_channel --channel=status --log=$BBSERVERLOGS/rrd-status.log hobbitd_rrd --extra-tests=licenses --extra-script=/home/xymon/server/ext/rrd_status.pl --rrddir=$BBVAR/rrd The parameters extra-tests and extra-script are of interest. Change in file ~xymon/server/etc/hobbitserver.cfg the definitions of two variables to define this new test with NCV: TEST2RRD=....,licenses GRAPHS=....,licenses::1 Then the xymon daemon needs to be restarted. Add the following script, called gengt.pl, to ~xymon/server/ext. It is used to generate the title of a graph with license usage information. #!/usr/bin/perl # # GENerate_Graph_Title, gengt. # use strict ; my $Title; if ( $ARGV[1] eq 'licenses' ) { $Title = "Unknown license" ; # Default title $Title = "$1 / $2" if $ARGV[3]=~ m/^licenses\.(.+?)\.(.+)\.rrd$/ ; $Title.= " $ARGV[2]" ; # Datestring } # of if print $Title ; Finally, add the following section to ~xymon/server/etc/hobbitgraph.cfg: # # The total number and the number of issued licenses per product and feature # by a flexlm server. # [licenses] FNPATTERN ^licenses\..+\.rrd$ TITLE exec:/path/to/script/gengt.pl YAXIS License count [] -l 0 DEF:Tot@RRDIDX@=@RRDFN@:Total:AVERAGE DEF:Use@RRDIDX@=@RRDFN@:InUse:AVERAGE # DEF:Max@RRDIDX@=@RRDFN@:InUse:MAX # CDEF:Ext@RRDIDX@=Max@RRDIDX@,Use@RRDIDX@,- LINE1:Tot@RRDIDX@#FF0000:Total GPRINT:Tot@RRDIDX@:MIN: Min\: %5.1lf %s GPRINT:Tot@RRDIDX@:MAX:Max\: %5.1lf %s GPRINT:Tot@RRDIDX@:AVERAGE:Avg\: %5.1lf %s GPRINT:Tot@RRDIDX@:LAST:Cur\: %5.1lf %s\n LINE1:Use@RRDIDX@#00FF00:In use GPRINT:Use@RRDIDX@:MIN: Min\: %5.1lf %s GPRINT:Use@RRDIDX@:MAX:Max\: %5.1lf %s GPRINT:Use@RRDIDX@:AVERAGE:Avg\: %5.1lf %s GPRINT:Use@RRDIDX@:LAST:Cur\: %5.1lf %s\n # AREA:Ext@RRDIDX@#0000FF:Max Use:STACK # GPRINT:Max@RRDIDX@:MIN:Min\: %5.1lf %s # GPRINT:Max@RRDIDX@:MAX:Max\: %5.1lf %s # GPRINT:Max@RRDIDX@:AVERAGE:Avg\: %5.1lf %s # GPRINT:Max@RRDIDX@:LAST:Cur\: %5.1lf %s\n If you have defined the MAX consolidation function (CF) in the licenses.*.rrd files, you can uncomment the seven lines in section [licenses]. Then the maximum use will be shown too. ===== Source ===== ==== flexlm.pl ==== #!/usr/bin/perl -w # # flexlm : Retrieve the status of the Flex License Manager and report it to a # Xymon server. This script is to be run on each server (a client to Xymon), # at which the Flex License Manager is to be monitored. # # This Xymon client script generates two test-results. The first one is called # "flexlm". It contains the overall status of the FLEXlm: for each product the # global status is shown. This test will turn red if for at least one of the # products the FLEXlm service or the vendor daemon is not up. # # The second one is called "licenses". It shows per product and per feature the # number of issued licenses and the number of used licenses. These counters are # collected by Xymon in a RR database, giving an overview of the usage of the # licenses. This test will turn yellow if for at least one feature more than # 90% of the licenses are in use. This status is *not* propagated to the overall # status. # # This script is to be used in small FLEXlm environments, with a small number # of products and a small number of features. Typically, the number of {product, # feature} pairs should be less than about 100. For each pair one graph will be # generated in test "licenses"! This design has the huge benefit that it will # automatically follow the changes in the products and features without any # reconfiguration in this script and/or Xymon. # # Written by W.J.M. Nelis, nelis@nlr.nl, 200903. # use strict ; use POSIX qw/ strftime / ; use Time::Local ; # # Installation constants. # ----------------------- # # Define the parameters to contact the Xymon server. # my $XyDisp= $ENV{BBDISP} ; # Name of monitor server my $XySend= $ENV{BB} ; # Monitor interface program my $FmtDate= "%Y.%m.%d %H:%M:%S" ; # Default date format $FmtDate= $ENV{BBDATEFORMAT} if defined $ENV{BBDATEFORMAT} ; # # Define the list of servers for which FlexLM status information is to be # extracted. The name of the server as it appears in the `lmstat` output is # mapped onto the name to be used in Xymon. # my %FlexServer= ( => "" ) ; # # Define parameters for the local tests. $FlexlmCmd is the partial command # to fetch status information of FLEXlm. # my $lmstat= '/usr/local/bin/lmstat' ; # FQN of lmstat my $FlexlmCmd= "$lmstat -a" ; # FLEXlm status of "default" products # # Define the maximum lease time of a license. If a user uses a license longer # a warning message will be shown. The alert time is set to a relatively short # period: if the maximum lease time is exceeded but with less time than the # maximum alert time, the test status will be yellow. Typically, the alert # time interval is larger than twice the measurement interval and smaller than # the alert repeat time. # my $MaxLeaseTime= 12 * 3600 ; # Max lease time is 12 hours my $MaxAlertTime= 900 ; # Alert status for 15 minutes # # Define the list of licenses which are so-called system-licenses. Their usage # is almost always 100%, thus a warning (yellow status) is not appropiate for # these licenses. The name is case sensitive, and the modified feature name # should be used. Long usage of these system-licenses is not reported. # my %SysLics= ( # '.' => 0, ) ; # # Define the commands to read the current FLEXlm status information from # other FLEXlm servers. # my @InpFil= ( # "$lmstat -a -c \@", ) ; my @ColorOf= ( 'red', 'yellow', 'green' ) ; # Possible test status values # # Global variables. # ----------------- # my $Now= strftime( $FmtDate, localtime ) ; # Timestamp of tests my $ThisYear= (localtime)[5] + 1900 ; # This year my %Flexlm = () ; # Status of flexlm servers en licenses my $Server ; # Name of server running flexlm services my $LmPort ; # Identification of one flexlm service my ($hr,$fr) ; # References in and to a hash my %DSName ; # List of DS names # # Function min returns the lesser of two numeric values. # sub min($$) { my ($a,$b)= @_ ; return $a < $b ? $a : $b ; } # of min # # Function BuildDatasetName takes the name of a feature and generates a name # acceptable to Xymon / RRD from it. Only letters and digits are retained, and # the length is at most 32 characters. # sub BuildDatasetName($) { my $FN= lc $_[0] ; # Name of feature my $DN= '' ; # Function result: name of dataset $FN=~ tr/[a-z0-9\-_]//cd ; # Remove most funny characters my @F= split( /[-_]/, $FN ) ; # Remove "-" and "_" $DN.= ucfirst( $_) foreach ( @F ) ; $DN = substr( $DN, 0, 32 ) if length($DN) > 32 ; if ( defined $DSName{$DN} ) { $DSName{$DN}++ ; $DN.= $DSName{$DN} ; } else { $DSName{$DN}= 0 ; # Dataset name in use } # of else return $DN ; } # of BuildDatasetName # # Function GetDate builds an ISO-8601 formatted string showing date and time. # sub GetDate($) { my @Date= localtime( shift ) ; # Determine elements of date $Date[4]++ ; $Date[5]+= 1900 ; # Adjust month and year return sprintf( '%4d.%02d.%02d %02d:%02d', reverse @Date[1..5] ) ; } # of GetDate # # Function GetDayUTS determines the UTS of the first second of the given day. # sub GetDayUTS($$$) { return timelocal( 0, 0, 0, $_[2], $_[1], $_[0] ) ; } # of getDayUTS # # Function GetSecUTS determines the UTS of the fgiven time. The UTS of the # first second of the day is passed, as well as a string containing the current # time in the day. The result is the UTS as can be computed with timelocal. # sub GetSecUTS($$) { my $DayUTS= shift ; my @Time= $_[0]=~ m/^(\d\d):(\d\d):(\d\d)$/ ; return ($Time[0]*60 + $Time[1])*60 +$Time[2] + $DayUTS ; } # of getSecUTS # # Function GetLicenseUTS determines the start time of the use of a license. # Supplied parameters are month, day, hour and minute. # sub GetLicenseUTS($$$$) { my @ADate= ( 0, 0, 0, 0, 0, $ThisYear ) ; $ADate[4]= shift ; $ADate[4]-- ; # Month number $ADate[3]= shift ; # Day number $ADate[2]= shift ; # Hour $ADate[1]= shift ; # Minute my $ADate= timelocal( @ADate ) ; # Convert to UTS if ( $ADate > time ) { $ADate[5]-- ; # Adjust year number $ADate= timelocal( @ADate ) ; } #of if return $ADate ; } # of GetLicenseUTS # # Function ReadFlexlmStatus reads the status information delivered by one # invokation to `lmstat`, and saves this information in %Flexlm. # sub ReadFlexlmStatus($) { my $InpCmd= shift ; # Command to retrieve status information my $LmName ; # Name of one flexlm service my $Skip= 1 ; # Flag: do not interpret this server my $VDS = 0 ; # Flag: Vendor daemon status found my $AFeature ; # Name of a feature my $LicStart ; # Time at which a license was leased my @Lines= `$InpCmd` ; # Do supplied lmstat command foreach ( @Lines ) { chomp ; next if m/^\s*$/ ; if ( $Skip ) { next unless m/^\s*License server status: (\d+)\@([^\s]+)/ ; $Server= lc $2 ; # Name of server, OSI L3 address $LmPort= $1 ; # TCP/UDP port number, OSI L4 address next unless defined $FlexServer{$Server} ; $LmName= undef ; # Name of license $Skip= 0; # Interpret the next lines } else { if ( m/^\s*([^:]+): license server (\w+)/ ) { $Skip=1, next if lc $1 ne $Server ; $Server= $FlexServer{$Server} ; # Map hostname on Xymon name $Flexlm{$Server}{$LmPort}= {} ; $hr= $Flexlm{$Server}{$LmPort} ; $$hr{Name}= $LmName if defined $LmName ; $$hr{Status}= lc $2 ; # Save status of flexlm service } elsif ( m/^Vendor daemon status/ ) { $VDS= 1 ; } elsif ( $VDS ) { $VDS= 0 ; # Flag is true for one line only next unless m/^\s*(.*?):\s+(.*?)\s*$/ ; $$hr{Name}= $1 unless defined $LmName ; $$hr{Daemon}= $2 ; # Save status of vendor daemon } elsif ( m/^Users of ([^:]+):\s+\(Total of (\d+) lic.*? Total of (\d+) lic/ ) { $$hr{Feature}{$1}{Issued}= $2 ; # Total number of licenses $$hr{Feature}{$1}{Used} = $3 ; # Total number of licenses in use $$hr{Feature}{$1}{Locked}= [] ; # List of "locked" licenses $AFeature= $1 ; # Save feature name for "locked license" check } elsif ( m/^\s{4}(\w+) ([\w\-]+) .+ start [A-Z][a-z][a-z] (\d+)\/(\d+) (\d+):(\d+)/ ) { $LicStart= $$hr{Name} . '.' . BuildDatasetName( $AFeature ) ; # Internal name next if defined $SysLics{ $LicStart } ; # Skip system-lic $LicStart= GetLicenseUTS( $3, $4, $5, $6 ) ; # Time of lease if ( $LicStart < time - $MaxLeaseTime ) { push @{$$hr{Feature}{$AFeature}{Locked}}, [ "$1\@$2", $LicStart ] ; } # of if } elsif ( m/^\-\-\-\-\-/ ) { $Skip= 1 ; # Search for header line } # of elsif } # of else } # of foreach } # of ReadFlexlmStatus # # Function ShowGlobalStatus generates the overall status of the Flex License # Manager on this server. # sub ShowGlobalStatus() { my ($Color,$SvcColor,$AggColor) ; my %Output= () ; my @Sorted ; # Save area for a sorted list my $Result ; # Service status per server foreach $Server ( sort keys %Flexlm ) { $AggColor= 2 ; # Default value page color %Output= () ; foreach $LmPort ( keys %{$Flexlm{$Server}} ) { $hr= $Flexlm{$Server}{$LmPort} ; $fr= 'FLEXlm' ; # Name of license manager # # Build the status message showing the status of the flexlm service and the # vendor daemon. # $SvcColor= 2 ; # Default color for this product $Result = "Product $$hr{Name} at port $LmPort\n" ; $Color= $$hr{Status}=~ m/\bup\b/i ? 2 : 0 ; $SvcColor= $Color ; $AggColor= min( $AggColor, $Color ) ; $Result.= "&$ColorOf[$Color] $fr service status is $$hr{Status}\n" ; $Color= $$hr{Daemon}=~ m/\bup\b/i ? 2 : 0 ; $SvcColor= min( $SvcColor, $Color ) ; $AggColor= min( $AggColor, $Color ) ; $Result.= "&$ColorOf[$Color] Vendor daemon status is $$hr{Daemon}\n" ; # # Report the users which hold a license longer than $MaxLeaseTime seconds. # Only in the first 10 minutes after passing the threshold, the test status # will be yellow. In this way very long leases do not result in many alerts # from Xymon. # foreach my $feature ( sort keys %{$$hr{Feature}} ) { if ( @{$$hr{Feature}{$feature}{Locked}} ) { # # Sort the entries on the usage-time of the license, as it will show the # long-time users better. Moreover, by definition the status of the last # entry will be the one of the "highest" level. # @Sorted= sort { $$a[1] <=> $$b[1] } @{$$hr{Feature}{$feature}{Locked}} ; foreach ( @Sorted ) { $Color= (time - $$_[1] - $MaxLeaseTime) < $MaxAlertTime ? 1 : 2 ; $Result.= "&$ColorOf[$Color] User $$_[0] holds license " . "$$hr{Name}/$feature since " . GetDate($$_[1]) . "\n" ; } # of foreach $SvcColor= min( $SvcColor, $Color ) ; $AggColor= min( $AggColor, $Color ) ; } # of if } # of foreach $Result.= "\n" ; $Output{ $SvcColor . $$hr{Name} }= $Result ; } # of foreach $AggColor= $ColorOf[ $AggColor ] ; # Convert to name of color $Result = "\"status $Server.flexlm $AggColor $Now\n" . "Status of lm service / vendor daemon\n\n" ; $Result.= $Output{$_} foreach ( sort keys %Output ) ; $Result.= "\"\n" ; `$XySend $XyDisp $Result` ; # Inform Xymon } # of foreach } # of ShowGlobalStatus # # Function ShowOthProductStatus generates an overview of the license counts for # each {product,feature} pair known to FlexLM which are not reported by # ShowBigProductStatus. In case all licenses of a feature are in use, a # (yellow) warning is shown. # sub ShowOthProductStatus() { my $Color ; # Status of Xymon page my ($Product,$Feature) ; my $DS ; my %Output ; # Visible status messages my %Stats ; # License usage counts, invisible my $Result ; # Message for Xymon foreach $Server ( sort keys %Flexlm ) { $Color = 2 ; # Default value page color %Output= () ; # Clear message list %Stats = () ; # Clear license counts %DSName= () ; # Clear list of DS names foreach $LmPort ( keys %{$Flexlm{$Server}} ) { $hr=$Flexlm{$Server}{$LmPort} ; $Product= $$hr{Name} ; # Product name # # Build the statistics to be saved in an RR database. However if the licenses # for a feature are almost all in use, generate a (yellow) warning message. # $Result= '' ; foreach $Feature ( sort keys %{$$hr{Feature}} ) { $fr= $$hr{Feature}{$Feature} ; # Ref to feature description $DS= BuildDatasetName( $Feature ) ; # Build dataset name unless ( defined $$fr{Issued} and defined $$fr{Used} ) { print "$Now Statistics of $Server / $Product / $Feature are not defined\n" ; } # of unless if ( $$fr{Issued} > 2 and $$fr{Used} > 0.9 * $$fr{Issued} and not defined $SysLics{ $Product . '.' . $DS } ) { $Color= min( $Color, 1 ) ; $Result.= "&$ColorOf[1] $$fr{Used} out of $$fr{Issued} licenses " . "of $Product / $Feature are in use\n" ; } # of if $Stats{$Product . $DS}= sprintf( "%s : %d\n%s : %d\n", "$Product.$DS/Total", $$fr{Issued}, "$Product.$DS/InUse", $$fr{Used} ) ; } # of foreach $Output{$$hr{Name}}= $Result if $Result ; } # of foreach $Result = "\"status $Server.licenses $ColorOf[$Color] $Now\n" . "License counts of all products and features\n\n" ; $Result.= $Output{$_} foreach ( sort keys %Output ) ; $Result.= "" ; $Result.= "\"\n" ; `$XySend $XyDisp $Result` ; # Inform Xymon } # of foreach } # of ShowOthProductStatus # # MAIN PROGRAM. # ------------- # # Phase A: # The non-specific status command is used to retrieve statistics for those # products which are bound to the default port range. # ReadFlexlmStatus( $FlexlmCmd ) ; # # Phase B: # Fetch the statistics of other FLEXlm services. # ReadFlexlmStatus( $_ ) foreach ( @InpFil ) ; # # Phase C: # Remove those entries for which no status of the vendor daemon is known. # foreach $Server ( keys %Flexlm ) { foreach $LmPort ( keys %{$Flexlm{$Server}} ) { $hr= $Flexlm{$Server}{$LmPort} ; next if exists $$hr{Daemon} ; delete $Flexlm{$Server}{$LmPort} ; } # of foreach } # of foreach # # Phase D: # Inform Xymon. # ShowGlobalStatus ; # Show status of FLEXlm services ShowOthProductStatus ; # Show status of other products __END__ =head1 NAME flexlm.pl is a client-side Xymon extension, which monitors the status of the flexlm services and the number of licences in use on a number of flexlm servers. =head1 DESCRIPTION An important feature of script flexlm.pl is that it does not depend on the names of the licensed products or the names of theire features. Thus it will follow the changes in the set of products and features automatically. The script generates two tests. The test "flexlm" shows the current status of each product on a server, as well as the users who use a license for a long period. The test "licences" shows the the number of used and issued licenses for each {product,feature} combination. It will issue a warning (yellow status) if for at least one feature more than 90% of the licenses are in use. As there is one graph per {product,feature} pair, the number of features to be monitored should be limited in order to have a reasonable response. A reasonable upper limit is about 100 features. The status-channel is diverted to a script to handle the data about the license use. It creates one RRD per feature, named "licenses...rrd". It contains two DS, "Total" and "InUse", and 8 RRA's. Four RRA's are build using the AVERAGE CF, and 4 RRA's are build using the MAX CF. ===== Known Bugs and Issues ===== The name of the vendor daemon is used as the name of the product. In some cases, the name of the vendor daemon is not very descriptive. A table to map the name of the vendor daemon on the name of the product will improve the presentation of the results, but will make the script dependent on the names used. The script presented here is a stripped version of the script running at NLR. The published version misses two options. The first one is the use of a configuration file. This option is tightly bound with a naming convention in use at NLR. The second option is the measurement of the number of denied license requests. It depends on two features, namely the aforementioned naming convention and the availability of the debug log. ===== Changelog ===== * **2010-05-31** * Initial release