All About the Apps
Showing results for 
Search instead for 
Do you mean 

HP Diagnostics: Identifying Transaction Bottlenecks

BTopham ‎03-18-2013 01:41 PM - edited ‎06-09-2015 03:06 PM

This blog will present a real-life example of using HP LoadRunner and Diagnostics to find the method level bottleneck in a slow end user transaction during pre-production. Customers can use similar Diagnostics capability in production environment, when they are analyzing the application performance bottlenecks and root causes to poor application performance. This info can be channeled back to the development organization so that the application performance issues identified in production can be quickly addressed in a patch or new application version. This type of detail level info sharing on performance issues and collaboration to remedy them, will improve the application performance management at any customer organization.


Customer Challenge

The testing team at a US healthcare company uses HP LoadRunner to measure transaction performance of a mission critical composite application from the end user perspective. Backend components are performance tested using LoadRunner to make Web Service and database calls. While they are successful at identifying poorly performing transactions, they still struggle with identifying the root cause of observed transaction performance issues. This lack of visibility into transaction performance comes at a high price to the organization. Developers are not able to focus their performance tuning efforts on the right components. The test team struggles with finding granular component level performance timings through the transaction flow and fails to provide actionable information to the development team to respond in an agile fashion to application issues. HP J2EE/.NET Diagnostics is designed to address these issues in the development lifecycle and the conducted a brief proof-of-concept (POC) for the customer demonstrates the LoadRunner-Diagnostics integration and key capabilities it will provide.


Configuring LoadRunner-Diagnostics integration

See Joe Colantonio’s blog on setting up LoadRunner-Diagnostics integration:


Transaction Analysis

In the image below, we can see how the LoadRunner Controller UI controls the test execution. We can see the summary (1) and over-time (2) performance of the Login and Logout transactions as they are measured from a virtual user perspective for this specific load test scenario. Note that the “Run” tab (3) is selected below.

In the image below, we are looking at the contents of the “Diagnostics for J2EE/.NET” tab (1) in LoadRunner Controller UI. This UI provides visibility into the live performance of all the backend application components (Servlets, JSPs, EJBs, Heap, Layers, etc) for the active load test (2). We are actually viewing the Diagnostics UI embedded inside the LoadRunner Controller UI. This UI integration is unique to LoadRunner solution. The use of any alternative diagnostics solution would require the user to switch between the LoadRunner UI and an external UI constantly and try to baseline the measurement separately. This can be very time consuming and lead to misleading analysis.

As seen above, Diagnostics presents the latency of the Login and Logout transactions (3.1) within the JVM/CLR application servers. Diagnostics also presents the slowest 5 EJBs, Servlets, Web Service Operations, etc (3.2). In the load view (3.3), Diagnostics groups method level performance into logical layers of the application. The Probes view (3.4), displays the heap usage in the application servers under test. Customers prefer this view since it provides a single pane of glass into the various key backend performance indicators of the application under test. Users can double click on any chart to drill deeper into a specific area. In the above example, we double click on the transactions chart (3.5) to drill into the detailed view for transaction performance which is shown below.


In the image below, we have drilled into the Synthetic Transactions view in Diagnostics. We can see all performance metrics for incoming LoadRunner transactions aggregated over the active load test. Thanks to the out-of-the-box LR-Diagnostics integration, the user does not have to manually configure these transactions in Diagnostics. They are automatically seen in Diagnostics when you start your LoadRunner test scenario with Diagnostics probes. Other solutions require changes to test scripts in order to display this information in their own UI and will make the diagnostics and analysis more complex and time consuming task.


The above view tells us that the login transaction is taking a significant amount of time, 9.2 seconds. We can right click the Login transaction and look at its topology, as shown below. This tells us the application servers the Login transaction flows through.


Next, we drill into the application server requests invoked by this LoadRunner transaction. Note that this view is filtered down to requests for a specific transaction, Login in this case (1). The table lists the individual components involved and their statistics over the active load test.

The table above tells us that a single servlet (2) is causing most of the transaction latency. And we can see that latency is tied to CPU (3). Why is this request taking so much CPU? To answer this, we drill into the Java stack of servlet invocation instances (4) to identify methods consuming CPU.


In the image below, we are looking at the Java stack for a Servlet instance (1). Each red bar represents a method in the stack and the width reflects the method’s duration (2).  We can see that this instance had a total latency of 20.7 s of which 13.5 s was spent in CPU time (3).


In the image below, we are looking at the bottom portion of the above servlet instance. It shows us that the web service engine is spending most of the instance’s time in XML parsing related to the SOAP response from the downstream TIBCO service.


After analyzing a number of slow servlet instances, we found the SOAP SAX parsing was the actual bottleneck of customer’s slow transactions. This was exactly the type of unambiguous performance information that the test team wanted to provide their development team.


When a test team provides development teams with this level of visibility into transaction bottlenecks, it typically causes a shift in the relationship between the teams. The test team becomes a trusted advisor to the development team and contributes to the agility of the development process.


Similar sharing of diagnostics info can be achieved between the development and operations teams. The analysis example provided is one of the testimonials how the collaboration between application development, testing and operations can improve the application lifecycle management in a customer organizations today. This collaboration can be supported by the technology solutions like LoadRunner and Diagnostics, but will require much more active approach from the customer side to enable the collaborative processes between organization teams. This may require more of an organizational process development activity, rather merely using a technology or tool to enable the collaboration.


By: Petri Maanonen


About the Author


June 6 - 8, 2017
Las Vegas, Nevada
Discover 2017 Las Vegas
Join us for HPE Discover 2017 in Las Vegas. The event will be held at the Venetian | Palazzo from June 6-8, 2017.
Read more
Each Month in 2017
Software Expert Days - 2017
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all