All About the Apps
Showing results for 
Search instead for 
Do you mean 

For mobile application testing, do you head to the wild or to the lab?

AntoineAymer ‎05-14-2014 04:11 AM - edited ‎09-10-2015 11:57 AM

Please suggest a good testing tool for Mobile.”

LinkedIn is filled with such discussions. The assumption behind the question is that the traditional testing rules still apply. Of course, they do, and of course, the question is legitimate, but only to a certain extent.


As testers and QA professionals, we need to ensure that the content we deliver to our customers meets their needs regardless of the where, when, how and what they access.


As testing organizations, we should consider two types of practices:

in the wild testing and in the lab testing.


In the wild – or crowdsource - testing intermediaries offer to capture the “wisdom of crowds’’ (e.g. Wikipedia) by recruiting, training, certifying and managing demographic segmentation and provide both defect reports and recommendations as part of their value add.


Whereas “in-the-lab” testing is what you know as testing activities that take place through your organization (or outsourced) as part of your application lifecycle.


Is in-the-lab testing the only way?


The main challenge that in-the-lab practices fail to address is how to fit user experience into the overall testing effort. How to identify new defects beyond the scripted exploration? How to address research questions (e.g. A/B testing)?

Dev/Test staff is highly qualified to conduct complex and technical testing procedures, however routine processes can sometimes lead to overlooking potential rote defects.


Let’s diagnose this further, starting with the mobile testing disciplines:


Mobile testing discipline

In the wild

In the lab


  • Validation of functionality
    • Smoke / Regressions Testing
    • Offline access testing
    • Negative Testing
  • Peripheral Testing (think Nike Fuel Band)

Helps validate the perception of functionalities through a demographically segmented crowd of testers (pick the right testers) to mirror target users.

Can reproduce a very large number of scenarios.

Caution with using your own staff as repeated testing lead to fatigue, monotony and indifference

Caution with opening non-production application to outside and non-badge employees

Vital with continuous testing.

Ideally suited for regression testing.

Caution with costly onsite device management: think about a combination of on premise (core devices) and cloud deployment (scalability).

cannot reproduce every possible combination.

Check how HP Sprinter for Mobile accelerates manual testing

Non Functional

  • Performance testing
  • Varying network conditions
  • Different network Types
  • Services

Ability to gauge the level of performance that an app will have in 2G, 3G, and even 4G environments. Rather difficult to capture and report on.

Check how HP Network Virtualization for Mobile (Shunra) factors in network data in the context of mobile testing, and provides recommendations on code

Interoperability (IOP)

  • Voice / SMS interrupts
  • Notifications
  • Battery /Cable Removal

Tedious to reproduce (e.g. ask a friend to phone you, have a second phone in hand)

Great fit.

Automation using HP UFT Mobile addresses this use case.

Memory Leak

  • Memory Usage
  • Memory Leaks
  • Garbage Collection

Not fit for purpose


Installation Testing

  • New App Install
  • Uninstall and Reinstall
  • Upgrade testing



Automation using HP UFT Mobile fully addresses this use case

Context-driven Testing

  • Translation
  • Images and Text
  • Currencies, time zones etc.
  • Location-based services

Vital to validate appropriate terminology and accurate translation

Access to multiple environments at once (e.g. home, office, on the move)

Access to “dirty environments”

Can cover some aspects (e.g. currencies exchange rate, time zone)

Most in-the-lab solutions do not support GPS injection for context switching use cases

Performance Testing

  • CPU Usage testing
  • Network Usage
  • Activity Render time

Not fit for purpose, or difficult to coordinate


HP LoadRunner with HP Network Virtualization fully addresses this use case

Security Testing

  • OWASP Vulnerabilities
  • Dynamic Testing
  • Static Code Analysis
  • Data Encryption

Not fit for purpose


Usability and Compatibility Testing

  • User Experience
  • Competitive Analysis
  • Expert Review

Vital for validating the layout and design (A/B testing) according to best practices and industry standards.

Access to multiple environments at once (e.g. home, office, vacation)

Benefit from an outside UX expert

Limited to assess subjective thinking.



Overall, you have to have an in-the-lab testing practice. This is even more relevant as you shift testing left. Nothing will replace your expertise and your ability to use, reuse and scale your valuable business and testing knowledge.


And, as Meg Whitman once stated: “Sometimes, it’s a little bit like being a politician. We have work to do in understanding our users’ sentiments”.

In-the-lab practices will sometimes not suffice to address the numerous challenges the world poses.


However, do not fall for the easy to understand but flawed argument that in-the-wild testing is a cheap alternative to in-the-lab testing.

It is actually not an alternative at all but a complementary practice.


Don’t risk your “Brand” and create additional risks.


Think of how to use each practice type to best effect.

And watch your mobile application quality and customer satisfaction soar.




Want to learn more?

I encourage you to visit our Mobile Apps Solutions homepage to learn how HP solutions can help.


0 Kudos
About the Author


Maggie Milton
on ‎05-27-2014 06:26 AM

The main challenge that in-the-lab practices fail to address is how to fit user experience into the overall testing effort.

I fully agree with this statement, thorough testing in the lab can results in an excellent product, from a functional point of view, but you're only halfway through. You need not only a functional product, you need a user friendly one, and for this you have to go "in the wild".

Alis Nikol
on ‎07-22-2014 02:10 AM


Sometimes the bugs initially found in mobile app testing can appear small and insignificant, whereupon deeper investigation uncovers bigger problems.” I have been in situations where non-testers look at these “insignificant” probing but were surprised when I later on pointed out big issues after my investigation was through. Now you may choose “wild” or “lab”.

27 Feb - 2 March 2017
Barcelona | Fira Gran Via
Mobile World Congress 2017
Hewlett Packard Enterprise at Mobile World Congress 2017, Barcelona | Fira Gran Via Location: Hall 3, Booth 3E11
Read more
Each Month in 2017
Software Expert Days - 2017
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all