Digital Transformation
Showing results for 
Search instead for 
Did you mean: 

Big Data 2020 : Everything will be an experiment


Let’s jump again to the year 2020.

Business process flows are experiments - a business process is created, but as it gets used, the people using it realise that they could optimise it. So, they sit with a business process modeller and adjust their “business process experiment”.

Applications are experiments. Game designers create a game. Put it out there. Look at the touch streams and then modify the game. The game’s functionality is fluid.

Smart devices are experiments. The smart shopping trolley, the smart home security system, the smart cooker, the smart heating system, the smart water recycling system, the smart e-bike, the smart truck all contain applications. The applications are experiments - the app dev group puts out a version, measures the machine-to-machine data from the smart device and then adjusts their experiment.


smart fridge.jpg

Media is an experiment. Rock groups put out a mix and see what people think. Based on feedback, the mix and the song are adjusted. Broadcasting organisations put out the first few episodes of a series and then adjust based on customer feedback.

And even data analysis is an experiment. A 360-degree data analysis is created and released. Based on changing data feeds and “customer feedback”, the analysis is changed. (See my last blog post on the future of big data which talked a lot about this). 

We are talking here about applying the ideas of agility to business processes, to traditional applications, to embedded applications in smart devices, to media and to data analysis.

Putting in place the last piece in the agility jigsaw
Agility is a great idea. It allows us to put out functionality, see what our customers think of it, and then adjust accordingly. It very much views the application’s functionality as an experiment.

Agility started in development teams. They would create and test new functionality in “sprints”. These sprints would then be released into product. Or, to be more accurate, they would be given to the data center to be put into production.

Unfortunately, that is where the lofty goals of agility fall flat on their faces because production will often take a long time to put the new application ("yet another version of the application") onto their run-time systems. Which is why there is so much interest in “dev-ops” – the linking of development and operations using automation, sharing of applications models and even organizational alignment between development and operations.

What big data allows us to do is take agility and dev-ops one stage further, and to “close the loop” from release of a product to starting to work on the next version.

“Closing the loop” more quickly and more accurately
When I was a product manager, I worked with a really smart team keen to embrace all the latest application development thinking. They were big fans of beta testing. We would beta test release candidates with a number of customers. How did we decide which feedback to include, and which to keep for another day? A lot of weight was given to the a/ the loudness  and aggresiveness of the customer’s field team and b/ to the size of the customer. And so, the prioritization of feedback from the beta was down to what we in Britain term, “a wet finger in the air” (you wet your finger and then hold it up in the air. It tells you which way the wind is blowing. It’s used in Britain to describe a decision-making process that is some way from data-based!).

By 2020, I believe that business process owners, app owners, smart device owners and media owners will routinely use big data to quickly, accurately and objectively figure out what they need to change in their “experiments” - their process, their apps, their devices, and their media.


The diagram below shows what agility can be applied to - applications, smart devices, business processes, and media; and the data sources from which we can collect the data to “complete the agility loop” - machine to machine, touch streams, maintenance logs, etc.



completing agility.png


GSN is doing this today (aka The Game Show Network - a Sony e-Gaming company) is doing just this. In fact, I stole the phrase “Everything is an Experiment" from their Chief Data Monger, Portman Wills. creates a game version and then measures the touch-streams from those games. They use this data to quickly, and accurately, determine what they need to change in the game. They then repeat this cycle over and over.

The diagram below shows how quickly can go around their product loop from starting on one version to starting on the next.


gsn cycle timings.png


The times of each step in the game design and release can vary, but can actually whizz around the whole cycle in just seven days, basing their functionality decisions on factual, big data analysis.

Using “the crowd” to make decisions
In a previous blog post I talked about a company called Quirky, that uses “the crowd” to propose and vote on ideas for products. I think that this is the social media angle on “everything is an experiment”.


Everything I’ve talked about up till now has been about collecting structured data, typically lots of it, to determine how people are using your product.

Social media can also be used both to suggest ideas, and then to test the popularity of ideas. I word of caution. We can’t just accept everything that comes from social media, “in the raw”. For example, a gaming company that charges for their product will get skewed results because, if they could, everyone would want free gaming. The London Metropolitan Police were sent to the wrong areas of London during the 2012 riots in that city by false social media traffic deliberately posted to fool the Police.

That said, NASCAR is using social media monitoring to great effect. They change their “product” dynamically based upon twitter traffic. This is shown below : they analyze tweets for clusters around certain topics, and then they monitor the sentiment around each of these clusters, driving the information (their "product") they put out to race-day displays, broadcast partners and the fan site.


NASCAR diagram.png


And Quirky lives or dies based on the quality of the ideas and preference expressed by social media.


Management and the delusion of finality
About two years ago, I attended a management training for HP Software’s European management. In my workgroup were the European manager of support escalations and the European manager of contract admin. During the week, we discussed our jobs - warts and all. They both commented how they would love to innovate their business processes and could vastly improve them in doing so, but that those business processes were too rigid. “The process owners assume that the first version of their process is optimal, and won’t ever need to change”.

I think that this is a very, very important concept. As management gurus never tire of telling us, the only constant in this world is change. And, no-one is so omnipotent that their first version (or forth version, for that matter) attempt at something is going to be perfect and better than any competition.

So, if we want to thrive in a changing world with fast moving and aggressive competition, we need to stop assuming that once something is released, that is it. We need to move from “the delusion of finality”. Our business process can and should be viewed as an experiment. Our games and our mobile applications should be viewed as an experiments. The functionality of our smart devices should be viewed as an experiments.

When we let go of the delusion of finality, we next realise we need to setup a fast and accurate feedback mechanism from release of our product / process / media to work on the design of the next version.

My experience is that it’s management who most needs to let go of this delusion. After all, managers are measured on how much stuff they get their teams to do - how many things can be checked off the list. If we say, “we’ve checked this product release off the list, but it’s an experiment which we are now measuring”, this goes against the traditional measurement of management effectiveness.


Want more?

For a lovely graphical listing of all Big Data 2020-related postings, please go to my Big Data 2020 web page.


To find out what HP Big Data can do for your today, please go so our HP HAVEn page

Mike Shaw
Director Strategic Marketing

linkedin.gifMike Shaw

0 Kudos
About the Author


Mike has been with HPE for 30 years. Half of that time was in research and development, mainly as an architect. The other 15 years has been spent in product management, product marketing, and now, strategic marketing. .

Jan 30-31, 2018
Expert Days - 2018
Visit this forum and get the schedules for online HPE Expert Days where you can talk to HPE product experts, R&D and support team members and get answ...
Read more
See posts for dates
HPE Webinars - 2018
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all