Enterprise Services
Showing results for 
Search instead for 
Do you mean 

How to Pick the Right COTS Product

‎12-10-2013 03:51 AM - edited ‎09-30-2015 07:00 AM

  1. By: James (Jim) R. Hughes, Global Business Analysis Capability Leader, Hewlett Packard Company


Balancing rock.jpgAuthor’s note: My blogs address the challenges that limit applications solution delivery success, and how to overcome them.



If you wanted to purchase a wireless digital picture frame to place on the piano in your family room, you could perform an Internet search and find an assessment of competing products rated on different functions and features such as picture sharing (e.g., Facebook, Flickr), ease of use, display area, file compatibility, and technical specifications. This kind of assessment is very helpful when selecting inexpensive and widely available consumer electronics products. However, when it comes to selecting a COTS product, such as a payroll system or a CRM tool which will be become an integral part of your business operations, you won’t be able to use readily available product assessments. You will have to create your own process and assessment tool.


The important steps for selecting the right COTS product to meet your business needs, are:


  1. Identify Requirements – The requirements you document for selecting a COTS product will be identified from various sources, including what your current system does, the backlog of user requests, and function/feature lists from vendors. Be sure to include both functional and non-functional requirements (e.g. performance, security, cost per user, etc.). One mistake often made at this point is defining requirements at too low a level of detail. COTS vendors develop best-practice processes in their products through interaction with many customers, so you do not want to specify requirements at a level of detail that would cause disqualification of a good product. Your requirements should be straight-forward narrative statements of what you need, not use cases describing how the processing should work.
  2. Weight Requirements – Once you have a candidate set of requirements, classify each in terms of business priority and operational risk level and create a weighting for each. For example, what would be the consequences if a particular function was not available or failed to work properly. Keep the levels of priority and risk simple (i.e. a maximum of five). Ensure that the weightings have a reasonable distribution (e.g. using a ‘bell curve’ or quintiles), since all requirements cannot be weighted as ‘high.’ The weighting process will also help you keep the list of requirements to a reasonable number.  As a rule of thumb, if you have more than 250 requirements you probably have too many.
  3. Create an Evaluation Instrument – You need to create two parts for the evaluation instrument. One part will be supplied to the COTS vendors and the other will be for consolidating the vendors’ feedback. We usually create a spreadsheet for the vendors containing the requirements. We export the requirements from our management tool (HP ALM). The requirements are structured into a logical hierarchy. This spreadsheet allows the vendors to provide their compliance with each requirement, using a simple scoring scale:  0=no compliance, 1=limited compliance, 2=partial compliance, 3= full compliance.  The second part of the evaluation instrument provides for a consolidation of the vendors' reponses.
  4. Create a Candidate List – Identify a list of 5-10 candidate COTS products. Potential products can be identified by conducting searches on the Internet, reviewing industry analyst reports, or by attending conferences where vendors display their products.
  5. Request Self-Assessments – Request each vendor to conduct a self-assessment of their product against your requirements. Make sure that they do not know how you have weighted each requirement.
  6. Select a Short List – Consolidate the returned assessments and apply the weighting factors for each requirement to each vendor’s score for that requirement. Create sub-total scores for categories of requirements, and an overall score for each vendor’s COTS product.
  7. Conduct a Runoff – Select three (no more than five) of the products which score well against your requirements and ask each vendor to perform a series of demos against a scripted set of ‘day in the life’ scenarios which you have prepared. Have members of your team rate each vendor’s product using the same instrument that was used by the vendors. Consolidate your team members’ responses. Hopefully, you will find a clear winner. However, if two products score very close together, you can begin negotiations with one vendor and keep the other in reserve.

Other blog postings by Jim Hughes, which address the challenges that limit applications solution delivery success, and how to overcome them:



Blogs in the Producing Quality Software series by Jim Hughes


Other blogs by Jim Hughes:


About the Author


Jim-150X210.jpgJames (Jim) R. Hughes, Global Strategic Capability Leader, Hewlett Packard Company

Jim has been with HP for 33 years and currently leads a global Strategic Capabilities Management team, with a specific focus on Business Analysis and Configuration Management. Jim also manages a team within the US State, Local, and Education division of HP. He was a member of the IIBA committee that created a Business Analyst Competency Model and he participated in the development of IEEE standards. Jim graduated from Harvard University, the University of British Columbia, and Ottawa Theological Hall. He currently lives in Toronto, Canada.

0 Kudos
About the Author


Jim has been with HPE for almost 35 years and leads a matrixed global Strategic Capabilities Management team. His specific focus is on the Business Analysis capability. Jim is also a member of a team within the US Public Sector division of HPE, responsible for delivering application software to Federal and State governments. He was a member of the IIBA committee that created a Business Analyst Competency Model and he participated in the development of IEEE standards. Jim graduated from Harvard University, the University of British Columbia, and Ottawa Theological Hall. He lives in Toronto, Canada.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Aug 29 - Sep 1
Boston, MA
HPE Big Data Conference 2016
Attend HPE’s Big Data Conference to learn from peers in every industry and hear from Big Data experts and thought leaders in an exciting, energy fille...
Read more
Sep 13-16
National Harbor, MD
HPE Protect 2016
Protect 2016 is our annual conference and is the place to meet the world’s top information security talent, discuss new products and share information...
Read more
View all