Digital Transformation
Showing results for 
Search instead for 
Did you mean: 

How to create a performance scorecard for the domain of IT automation


I’ve written before about the CIO whose remarks about performance management have really stayed with me. Basically, he told me that to get the business transformation and ROI that he wants from an IT toolset, it’s necessary to have a performance management system that measures the benefits that are delivered. In my last post, I talked about the importance of being able to measure the benefits of your business service management software. This week, I turn automation as a domain.


What objectives do customers have for purchasing automation software?

I asked HP’s customers this week to tell me why they invested in automation software as a category.  I was told that automation software enables them to perform functions that are time consuming, expensive, and in some case impossible to do well manually.  Put simply without automation software the following functions are difficult to achieve:


  • Fault and Event Management. These are foundational to proactive/predictive failure analysis and recovery.
  • End-to-End Configuration Management. If you cannot control your environment, it is difficult to comply against standards. And even worse, the chance of change or patch management succeeding is dramatically reduced as differences between dev or test environments and production systems increase the likelihood of failure.
  • Automated Patch Management and Deployment. This is a big deal on many fronts. Automation reduces the cost of patch management and helps ensure consistency. As well, automation is the basis of how you keep your configuration consistent across all environments. For this reason, it is also the basis of good security and configuration management.
  • Proactive Security and Vulnerability. Let’s face it most security indicators tend to lagging indicators. Security management is problem management at its best. But there are things that a CISO can do to ensure better control their IT environments and speed up response to undetected events. This is what automation software as a category enables.
  • Performance and capacity management. This is about enabling proactive/predictive performance management. This requires the ability to first see issues before they happen and fix things proactively. Second, this involves the ability to use data to determine the most effective surge plan between internal and external service providers—your cloud and service provider strategy.

These are all great objectives. And for purposes of this blog, let’s say these objectives are the reasons for purchasing the software.


What would go into an Automation Scorecard?

So if we follow the four-quadrant model for Norton Kaplan, what key performance indicators (KPIs) should be in the scorecard for the automation software purchase?




First Quadrant—IT Value

The first quadrant in a balanced scorecard focuses on IT value.  The first two KPIs on my list for this quadrant are the average time to configure and to provision a node. These go after the value side of at least four of the objectives discussed above. Another that fits here as well is the average time to run a policy check. The cost of doing this manually has to be very large and quality of the result low.  I care here about two things from a value prospective, the percent of non-encrypted traffic and the percent of failed business transactions. These say from a value perspective how well I am dealing with running my environment to take out cost and deliver better service quality.


Second Quadrant—Customer

Moving to the customer quadrant, this is where IT leaders can show that they are increasing value for IT’s customers. Since customers want their applications running well and securely, you should focus here on things here that should be seeing improvement or are demanded by customers. For customers, I would recommend you measure the percent of compliant nodes growing month over month because I want the potential of outage or a security issue going down. You would, also, want to see the mean time to recover from non-compliance going down. This says if something bad happens the lapse time is small.  Meanwhile, the average time to provision a node would tell you if your demand goes up that the IT system can grow to support it.  High on my list would clearly be the average time to deploy an application. As a customer, you want this to be as small as possible. And lastly, you want the frequency of policy checks to be high to ensure your IT remains secure.


Third Quadrant—Operational Excellence

For the operational excellence quadrant, there are many KPIs that make sense to track. However, if it is not clear, we want to measure improvement in all goal areas covered above. For this reason, I will list them by these topics. Many are technical but for customers they should be looking for a move into the green zone.


  • Fault and Event Management: Ratio of New vs. Dropped Nodes, Percent of Network Devices with Reachability Issues, Percent of Network Devices with Availability Issues, Percent of Utilization of Network Devices, and Network Latency.
  • End-to-End Configuration Management: Percent of Compliant Nodes, Average Time to Configure a Node, Average Time to Run Policy Check, and Average Time to Provision a Node. 
  • Automated Patch Management and Deployment: Percent of Successful Patches and Average Time to Patch a Node.
  • Proactive Security and Vulnerability: Percent of Incident Volume from Security Incidents, Percent of Compliant Nodes, Mean time to Recover from Non-Compliance, Average Time to Run Policy Check, Percent of Non-Encrypted Traffic, and Frequency of Policy Checks.
  • Performance and Capacity Management: Percent of Monitored Applications, Average Time to Provision a Node, Percent of Managed Nodes, and Percent of Network Devices with Availability Issue.

Fourth Quadrant—Future Orientation

In future orientation, we want KPIs that among other things measure improvement in the practice of Automation. For me, a few KPIs show future orientation:  Percent of Compliant Nodes, Percent of Managed Nodes, Percent of Successful Patches, and Percent of Monitored Applications. All of these should improve over a near term and a longer term. Together they are showing that I am achieving better control over my IT environment and establishing the type of future IT organization needed to succeed at the business of IT.


With this said, we would like to hear back from you. What measures would you put on a automation success scorecard? Where do you think we are missing something in the above list?


Related links:

 Feature:  Peak performance demands precision control

Solution page:  IT Performance Management

Operations Management software:

Video:  IT leader on importance of automation (links to:

Twitter: @MylesSuer

0 Kudos
About the Author


Mr. Suer is a senior manager for IT Performance Management. Prior to this role, Mr. Suer headed IT Performance Management Analytics Product Management including IT Financial Management and Executive Scorecard.

Jan 30-31, 2018
Expert Days - 2018
Visit this forum and get the schedules for online HPE Expert Days where you can talk to HPE product experts, R&D and support team members and get answ...
Read more
See posts for dates
HPE Webinars - 2018
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all