LoadRunner and Performance Center
Showing results for 
Search instead for 
Do you mean 

How to select a performance load test solution?

SilviaS ‎08-24-2012 04:13 PM - edited ‎09-24-2015 01:43 PM

Decisions, decisions, decisions…

Our life is full of decisions; especially when we are purchasing a new solution to improve our work.

That is how Mark Tomlinson kicked-off the "Survey of performance testing tools" Online Summit event.


I had the opportunity to sit in this Summit provided by the Software Test Professionals (STP) organization, over the last few days. The event was very unique because STP was able to bring 9 different vendors to present their "performance testing solutions".


Initially, Mark spoke about how we are always deciding what we "purchase" and our personal criteria for it which is subjective; deciding for a candy, ice cream, car, house, etc.

When we need to choose an IT solution, "it is an objective choice", Mark said.

His intro was to help people that "personal preference" doesn't really play a part in a business decision about buying a commercial performance testing tool.


We also had Shane Evans, the HP LoadRunner Product Manager, where he showed yesterday how easy it is to script with the newest HP patented technology TruClient.


Well, back to the initial presentation from Mark, he gave very good tips about selecting a performance testing solution. The following is a combination of what I learned from the Summit together with my experience.


As a first step, define your requirements for performance load test. Define what would be needed in the short term and long term. Within this first step, review the following items:

  • Do you have a defined process regarding performance testing?

If not, it is time to create one or review what is or isn’t working in the current process.

  • Define what organizations will be using the solution
  • Define what resources will be part of the evaluation

Define a quick RACI table, with people who will responsible, accountable, consulted and informed in the selection process.

  • What are the technical requirements?

A suggestion here would be to create a document with a matrix of your requirements and rate them based on importance.



(Sample of a comparative matrix provided by Mark Tomlinson)


Also, collect some information from developers to understand the development environments used, also spend time with operations to understand how the scripts can be leveraged when moving the application to production and vice versa.

Mark Tomlinson mentioned some of the information to be captured, as described below:

Protocol support, bandwidth emulation, scripting language, OS Supportability, data formats, extensibility APIs, monitoring, Automation (CI/CD).

  • What integrations will be required

E.g.: Integration with requirement management tool may be needed or you may need to have the test results stored for audit purposes in a central repository


  • What are your success criteria once you implement the solution?             

Then as you have the information above clear, you can start selecting vendors.

Through the process of selecting a tool, have a project with a realistic timeframe and ensure to keep up with communications to the team.


  • Define the criteria for the vendors to present. Try to get similar demos from vendors, so you will be able to compare apples to apples.
  • Determine phases to evaluate and reduce the number of vendors by phase.
  • Be prepared to ask questions! Mark and Scott Barber did a great job with questions to all the vendors during this event. Open and closed questions.

See some of the examples of questions vendors were asked that were mentioned by Mark:

Open Questions - expands the conversation into more elaboration
Let the vendor lead the conversation

Let the vendor share their thoughts about your needs

- Tell me more about how this feature works
- Why is this feature important?
- And without this capability, what happens then?
- Describe a situation where this capability is used

Closed Questions - seeks specific, deterministic answers – confirmatory
Guide the conversation to what’s important

- Does your tool support technology X or Y?
- Can I load my own data into the tool?
- What OS platforms do your tools support, windows, Linux, Unix?


And for sure, take time to try out the product!


During the event I posted the following question in many social media areas and got the respective responses that can also be helpful to you.

What are the most important features that you look for when shopping for a performance load testing tool?

“As a user I can see features such as, 24/7 tool support, self-explanatory features, as much as supported inbuilt protocols (for Load Runner), majorly - tool must be user friendly.”

“As a tool user I look for features like UI, help, technical forum, capabilities. As a manager it is important that my team is not coming to me with any sort of issues with the tool.”

“Identify your own performance requirements, provide point values to the requirements, then evaluate the tools against those requirements and see which tool has the highest point value”


At the end of the event, there was a quick presentation by both Mark and Scott. Scott went through 10 very good tips for the whole process of selecting a tool. My 3 favorites were:

  • Start with a Business needs analysis
  • When selecting a solution, review the vendor capabilities do deliver. Remember that the vendor will be your partner and you must trust them.
  • Easy buttons are great, but don't sacrifice extensibility for ease of use


The event was finalized with a discussion panel with questions from customers to all vendors.

Shane Evans wrapped up with the final comment:

"Do your homework. Research online.  Talk to people.  Learn as much as you can.  Many people before you have had similar objectives when it comes to application performance validation. Chances are, someone has already found the answer you are looking for"


If you are STP member you will be able to listen to the whole Event.

Here are the links to the recording:  STP Online Summit: A Survey of Performance Testing Tools


Now it is your turn: What are the most important features that you look for when shopping for a performance load testing tool?


Looking forward to hear from you!

Have a nice weekend!

Silvia Siqueira

Learn more about HP LoadRunner


0 Kudos
About the Author


WW Sr Product Marketing Manager for Performance testing solutions

aash mangrulkar
on ‎08-26-2012 03:07 PM


Good information overall, and it sounds like it was a great event for folks looking to educated on performance testing tools and selection criteria. Honestly not much new to add (as usual, Mark covers off most relevant areas), but I would reiterate points that should be emphasized to anyone looking to establish a performance testing process and/or vendor assessments in the space.

  • Not to oversimplify, but KISS is a good rule to keep in mind when establishing and implementing any process. We live in a complex enough world, the processes by which we are governed should streamline this, versus further complicate the maze that we run through to get apps tested and out the door. Establish your success criteria, and build as simple a process around delivering that as possible. Consistency and predictability (of the app and the process) is way more important than a large number of ‘checkpoints’.
  • The ‘devops’ mention was a little underplayed here, but is something that MUST be taken into consideration, especially given the traction Continuous Delivery has across Enterprise IT and the capabilities that WE are bringing to that market.
  • An oldie but a goodie is the need to look at the entire application – infrastructure and network emulation/virtualization really needs to be included to have real confidence in how an app is going to behave in the wild.
  • Most customers are NOT where they want to be, regardless of how mature they feel they are in the QA arena. Even though integrations are usually ‘nice-to-have’, they should be mentioned for two reasons: 1) it paints the ‘desired state’ vision for where things need to go and 2) HP has them! (as opposed to having to cobble things together behind the scenes.
  • Last thing to include (under Shane’s ‘do your homework’ phrase) are reference customers and marketechture. Of course we lead with this, but I am unaware of anywhere else that you’ll find the dominance that a single vendor has like HP in performance testing. This one is worth emphasizing for sure.


27 Feb - 2 March 2017
Barcelona | Fira Gran Via
Mobile World Congress 2017
Hewlett Packard Enterprise at Mobile World Congress 2017, Barcelona | Fira Gran Via Location: Hall 3, Booth 3E11
Read more
Each Month in 2017
Software Expert Days - 2017
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all