IT Operations Management (ITOM)
Showing results for 
Search instead for 
Do you mean 

Visualize OMi event pipeline performance in the OMi Performance Dashboard

Ian_Bromehead ‎05-19-2016 11:49 PM - edited ‎05-20-2016 02:39 PM

Stopwatch.jpg

The following is a guest post by Carol Park, R&D OpsBridge Specialist.

Do you want to track how long it takes to process your OMi events?

Do you want insight into the slowest steps in the event pipeline? 

Do you want to track this over time?

This article describes how to use Operations agent 12’s simplified custom data logging within an OMi policy to log event pipeline statistics to the performance data store on the Data Processing Server (DPS).

OMi event processing performance

The speed with which events pass through the OMi event pipeline varies based on many factors, as described in the OMi Performance and Sizing Guide.  In particular, OMi Event Processing Customization gives you the flexibility to insert custom Groovy scripts to manipulate or enrich events but if these scripts have slow performance characteristics, then that also affects the overall event processing rate.

The OMi JMX console (http://DPS:29922/invoke?operation=getReport&objectname=opr.store%3Aname%3DPipelineStatsMBean&value0=1&type0=java.lang.String) provides detailed event pipeline statistics, but it is a point-in-time snapshot.

EPI1.jpg

What would be really useful is to graph the data in OMi’s Performance Dashboard.

EPI2.jpg

To capture this data over time for historical graphing and reporting, you can create a policy that leverages the new Perl API capability of logging custom data.

Operations agent 12 custom data logging

One of the hidden gems of Operations agent 12.00 is the ability to log custom metrics to the performance data store using Perl.  It offers several advantages over using Data Source Integration (DSI):

  • Support for multi-instance* data
  • Support for 64-bit data types
  • Less effort, since there is no need to pre-create and compile a class specification file

* Multi-instance data refers to multiple data points submitted for a given metric at a given time.  For example, logging JVM heap usage for 3 different JVMs running on the same system.

Details on how to use the Perl API are well documented in the HPE Operations Agent User Guide.  The Perl script shown below is one example of how to use the API.

Create an OMi policy to log event pipeline statistics

Create a Scheduled Task policy to run every 5 minutes.  Select the Task Type “Perl Script” and enter the Perl script shown below.  The blue text shows the use of the new Operation agent 12 submittal API.

use strict;
use warnings;
use oaperlapi;

 

# Environment

my $omihome;
my $cmd;

 

if (defined $ENV{"TOPAZ_HOME"}) {
  $omihome = $ENV{"TOPAZ_HOME"};
} else {

  if ( $^O =~ /MSWin32/ ) {
    $omihome = "C:/HPBSM";
  } else {
    $omihome = "/opt/HP/BSM";
  }
}

if ( $^O =~ /MSWin32/ ) {
  $cmd = "$omihome/opr/support/opr-jmxClient.bat -u \"admin\" -p \"\" -s localhost:29622 -b opr.store:name=PipelineStatsMBean -m getReport -a 0";
} else {
  $cmd = "$omihome/opr/support/opr-jmxClient.sh -u \"admin\" -p \"\" -s localhost:29622 -b opr.store:name=PipelineStatsMBean -m getReport -a 0";
}

my @table;
my $x = -1;
my $y = 0;
my $step = "";

# Get event pipeline stats 

my $pipelineStats = qx{$cmd}; 

# Process interesting records into an array 

foreach (split(/\n/, $pipelineStats)) {
  next if /RMI URL/ || /\[main\]/;

  use feature qw(switch);
  given ($_)
  {
    when ( ! /\:/ && ! /^$/ ) {
      $y = 0;
      $x = $x + 1;
      $table[$x][$y] = $_;
      $y = $y + 1;
    }
    when ( /^Avg time/ || /^Max time/ || /^Last time/ ) {
      my @Line = split(': ', $_, 2);
      $table[$x][$y] = $line[1];
      $y = $y + 1;
    }

    when ( /^Bulk size/ ) {
      if($_ =~ m/Min:\s+(\S+)\s+Max:\s+(\S+)\s+Average:\s+(\S+)/) {
        $table[$x][$y] = $1;
        $table[$x][$y+1] = $2;
        $table[$x][$y+2] = $3;
      }
    }
  }
}

# Submit to the OA Perf Data Store via Perl APIs 

my $access = oaperlapi::OAAccess->new();
my $molist = oaperlapi::MetricObservationList->new();
my $interval = time;
my $mo = oaperlapi::MetricObservation->new($interval);

 for (my $x=0; $x < 19; $x++)
{
  $step = $table[$x][0];
  $mo->AddGauge("OMi:Pipeline:AvgTime",$step,"Average Time (ms)",$table[$x][1]+0);
  $mo->AddGauge("OMi:Pipeline:MaxTime",$step,"Max Time (ms)",$table[$x][2]+0);
  $mo->AddGauge("OMi:Pipeline:LastTime",$step,"Last Time (ms)",$table[$x][3]+0);
  $mo->AddGauge("OMi:Pipeline:BulkMin",$step,"Min Bulk Size ",$table[$x][4]+0);
  $mo->AddGauge("OMi:Pipeline:BulkMax",$step,"Max Bulk Size ",$table[$x][5]+0);
  $mo->AddGauge("OMi:Pipeline:BulkAvg",$step,"Average Bulk Size ",$table[$x][6]+0);
}
$molist->AddObservation($mo);
$access->SubmitObservations($molist);

 After saving the policy, all you need do is assign and deploy the policy to the DPS node.  After 5 to 10 minutes, you should see that it is logging metrics, like this:

ovcodautil -dumpds OMi
===
05/06/16 12:35:06 PM|Pipeline_Id                   |PipelineExit|
05/06/16 12:35:06 PM|AvgTime                       |      4.45|
05/06/16 12:35:06 PM|MaxTime                       |   5298.61|
05/06/16 12:35:06 PM|LastTime                      |      4.80|
05/06/16 12:35:06 PM|BulkMin                       |      1.00|
05/06/16 12:35:06 PM|BulkMax                       |    997.00|
05/06/16 12:35:06 PM|BulkAvg                       |      1.35|

Create the performance dashboard

Create the performance dashboard based on the metrics being logged.  Helpful tips:

While most pipeline steps are obvious, these few need some explanation.

Pipeline step

Description

PipelineEntry

EPI Step 1

ResolutionCompleted

EPI Step 2

PipelineExit

EPI Step 3

EventStore

Storing new events

EventUpdater

Storing event changes

 

The “Bulk” metrics refer to the number of events processed in a single bulk.  The higher the number the more efficient the event processing.  This number is influenced by change events.  For example, if you synchronize events between OMi and OM or another OMi server, then event changes will break up the bulk.

If you have made changes and want to start the counters afresh to establish new maximum and average values for each step, you can reset the event statistics via the JMX console:

http://DPS:29922/mbean?objectname=opr.store%3Aname%3DPipelineStatsMBean
Invoke the reset method.

Map the performance dashboard to the OMi Processing Server CI type

Since event processing is happening on the DPS, you might want to associate your new dashboard with the OMi Processing Server CI that is part of the OMi Deployment view.  Each time you select the OMi Processing Server CI in Performance Perspectives, you will see your OMi Event Pipeline Statistics dashboard.

  1. Navigate to Administration > Operations Console > Performance Dashboard Mappings.
  2. Select the OMi Processing Server CI type.
  3. Click Edit Mappings.
  4. Select your dashboard in the Add Dashboard list and click Done.
  5. Click Save

 

EPI3.jpg

 

This has been just one example of logging OMi metrics to Operations agent 12 for visualization in OMi Performance Dashboard.  By applying the same approach, you can log additional metrics such as the number of logged in users, CIs in the RTSM, event rate and more.  Please comment if you’d like a future blog on logging these or other metrics, or if you’d like more details around creating performance dashboards.

Try OMi now! OMi 10.10 comes pre-loaded with a number of Management Packs that you can try out without the hassle of getting management pack software or evaluation license.


 

Read more:

HPE Operations Bridge
HPE Live Network: Operations Bridge Evolution
HPE Live Network: Operations Manager i
HPE Live Network: OMi Management Pack development kit
HPE Live Network: HPE  SPI to MP Evolution Tool

 You can see demonstrations and find out more details of this and other features of our HPE Operations Bridge solution in our sessions and at our booths during HPE Discover Las Vegas.

Click on the image below or here to register.

Discover.jpg

 

About the Author

Ian_Bromehead

Comments
JimiT
on ‎09-07-2016 11:32 PM

Need to be careful with clear text password ;)

Events
June 6 - 8, 2017
Las Vegas, Nevada
Discover 2017 Las Vegas
Join us for HPE Discover 2017 in Las Vegas. The event will be held at the Venetian | Palazzo from June 6-8, 2017.
Read more
Apr 18, 2017
Houston, TX
HPE Tech Days - 2017
Follow a group of tech bloggers for a new HPE Tech Day, a full day of sessions about how to create a hybrid IT, from hyperconverged to Composable Infr...
Read more
View all
//Add this to "OnDomLoad" event