Digital Transformation
Showing results for 
Search instead for 
Did you mean: 

Big Data 2020 : Predict and Adjust


Normal and abnormal patterns - us humans need to be good at telling the difference between the two in order to survive and thrive.


In our personal lives, we need to know if the stranger walking towards us in the street is behaving normally or abnormally. We need to know if the food we are eating is normal or abnormal because abnormal food can make us very ill. We need to know if our health is ok because if we feel “really, really odd”, it might be warning of a health problem.


When driving, we need to know if other drivers on the road are behaving normally or abnormally. My son is learning to drive. He was driving me the other day. At a round-about (a wonderful British invention to maintain the traffic flow at the meeting of many roads), I said to him, “watch the car on your right, it’s about to do something strange”. Sure enough, a few seconds later this car lurched towards us in an attempt to cut across and turn off the roundabout. My son said, “how did you know that was going to happen?” I couldn’t actually tell him, but obviously after 30 years of driving, my eyes took in the scene and my brain was able to predict, with a reasonably high probability, that we were about to get some anomalous behaviour.



At work, recognising normal and anomalous behaviours is important too. If you run European marketing for a fast moving consumer goods company, you need to know what normal and abnormal sales patterns look like because when you see an anomalous pattern, you will need to react.


Banks and benefits payments agencies need to understand normal and anomalous transaction patterns so that they can detect fraud. The internal compliance departments in banks, likewise.


Healthcare workers need to be able to recognise normal and anomalous patterns in their patients’ health. The security and operations management in IT departments, likewise.


In other words, humans are constantly scanning the systems with which they interact, looking for the emergence of anomalous behaviour patterns.


Prediction is another way in which big data can “augment humans”
We talked in the last section about how big data in 2020 would augment humans by providing them with information and recommendations based on their current situation, their current goals, and their preferences. In this section, we will look at how big data in 2020 will augment humans by helping them to quickly recognise the onset of abnormal behaviour patterns in the complex systems with which they interact.


Why humans need help with prediction
Why do humans need help to predict the onset of anomalous behaviours? Three reasons:


A: The systems that us humans create are becoming over more complex. Psychologists will tell you that humans are great at monitoring one variable, but are very poor at monitoring more than one variable at time. For example, ask a human to look for guns in hand luggage, and they do really well. Ask them to look for guns, liquids, knives and explosives, and they don’t do so well.

B: Monitoring is boring. It’s not a very productive use of the human brain. If we’re going to help humans, let’s do it in areas which humans find boring so that we can free up them up to do what computers can’t do.

C: Computers are fast and so, if they do the monitoring, maybe they can detect the onset of abnormal behaviour faster than humans.


Modelling and using prediction
HP Labs and others are thus researching the use of big data techniques to predict abnormal behaviours. Such “anomaly prediction systems” will work something like this …


1: Specify the data feeds
A subject matter expert (a marketing manager, a specialist doctor, a city transportation system expert, a security expert) will tell the system what data feeds to monitor. For our European marketing manager for fast moving consumer goods the system might monitor sales over time, geography, direct/indirect mix, weather, time of year, and pricing versus competition.


2: Model normal and anomaly patterns
The prediction system will take a guess at what are normal patterns and what is are anomalous patterns. The subject matter expert will then look at the system’s categorisation of patterns and decide if they agree with the conclusions. They may reclassify some anomalous patterns as in fact normal, and vice versa.


This step will create the model that the monitoring system will use, but rather than have the subject matter expert (or worse, a data scientist) enter hundreds of rules, the system takes the expert thru the modelling process in terms that they understand - what is normal and what is an anomaly.


There will be many normal patterns and many anomalous patterns. For example, if we are monitoring patients’ health, it is a “normal pattern” for them to be asleep, walking, running, playing squash, etc.


This interactive step can also throw up insights for the subject matter expert because the system might find an anomaly pattern that the expert had missed.


The prediction system’s initial analysis may also throw up relationships that the expert hadn’t seem before. When humans need to look at data, we typically use a tool like Microsoft Excel and we plot one variable against another. For example, as a European marketing manager, I might plot sales over of my products over time, by country. I might notice that certain countries show anomalous patterns. However, when the prediction system does its analysis, it might notice that there is in fact a stronger relationship between whether a region uses resellers or goes direct, than there is between countries and anomalies. So, rather than trying to figure out why certain countries cause anomalies, I could instead work out what it is about my indirect channel that is causing the anomalies.


3: The system monitors and predicts
The system now looks at the data feeds constantly looking for the emergence of the many anomalies that have been modelled. When it notices an anomaly starting to emerge, it flags this to the subject matter expert - the marketing manager, the healthcare professional, and so on.

P&A graphic.png


If someone came to you and said, “boss, you need to take a look at this. I think there’s something bad starting to happen”, your first question would be, “why do you think that something bad is going to happen?” It’s the same here - the expert is able to interrogate the prediction system’s logic to understand why an anomaly is flagged. This is, again, a chance for the system to learn -if it’s a “false positive”, then the system will learn more about the normal / anomalous models.


Could we then automate the response rather than simply alerting the human? Dr Ruth Bergman, director of HP Lab’s Analytics Core Technologies research and of HP Labs Israel, prefers to focus on the prediction of anomalies. "The problem is that because the systems are complex automation only works when it is tailored to the system. There are good automation solutions, but they are specific. What we can’t do is create generic automation solutions. But, we can create generic normal detectors and abnormal detectors, which, with just a little expert help, can be adapted to any domain".


Some examples of where we might use prediction technology
Let’s look at how this prediction technology might be used.


Proactive parts replacement is a big deal if you have expensive plant, for which downtime is hugely expensive. Data centres, oil rigs, large cargo ships, operations theatres, airplanes. Multi-variable prediction allows us to better pinpoint when we need to do parts replacement. 


oil rig.png


HP Labs is working on improving IT operations’ ability to predict and proactively avoid application performance problems, before customers notice.


The same HP Labs team is applying the technology to IT security. Like IT performance management, IT security anomaly detection is a multi-dimensional task, where those trying to breach security try their upmost to look normal, and so anomaly detection is quite a challenge.


Fraud detection in banks and benefits payments agencies and internal compliance monitoring in banks will also find predictive anomaly detection useful.


We have discussed above how it might be used by marketing organisations to monitor the sales of fast moving consumer goods.


When we get feeds from all of the transportation systems in a city, we can use prediction technology to manage to free-flow of travellers throughout the city.


And it could be used in healthcare. For example, patients in hospitals and in their own homes could be monitored, and their various health measurement feeds analysed for anomalies. 


medical predict.png


Monitoring and predicting anomalies is another example of how big data will be used in the future to augment humans.

Such prediction systems will be better at multi-dimensional monitoring and they will notice the emergence of anomalies faster than humans can.


But if subject matter experts have to sit with data scientists to create (and frequently amend) the rules that define normal and anomalous patterns, few prediction systems will ever be implemented. It is therefore vitally important that the modelling of normal and anomalous patterns is something that subject matter experts can do themselves.


Want more?
If you'd like to see more HP big data blog posts and other HP big data news, please go to our big data "" page, here.


If you'd to learn more about the Enteprise 2020 project, then please visit the its "" page, here


Mike Shaw
Director Strategic Marketing

linkedin.gifMike Shaw

0 Kudos
About the Author


Mike has been with HPE for 30 years. Half of that time was in research and development, mainly as an architect. The other 15 years has been spent in product management, product marketing, and now, strategic marketing. .

See posts for dates
See posts for locations
HPE at 2018 Technology Events
Learn about the technology events where Hewlett Packard Enterprise will have a presence in 2018.
Read more
See posts for dates
HPE Webinars - 2018
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all