Behind the scenes at Labs
Showing results for 
Search instead for 
Did you mean: 

HPE Haven OnDemand: To compare different types of anomalies, you have to stay SANE


cezanneaorg.gifPaul Cézanne, Apples and Oranges

By Curt Hopkins, Managing Editor, Hewlett Packard Labs

This is the first part of a two-part series on the contributions Hewlett Packard Labs has made to HPE’s newly-launched developer platform, Haven OnDemand. The second part will discuss category trend analysis.

When you enter a grocery store, you may notice how an apple gleams or how good an orange smells, or how fresh a head of lettuce appears (especially if you’re hungry). But when you talk about all that produce, you can’t use one measurement to register the overall “goodness” of the offerings. You can talk about freshness, taste, color, scent, but all of those qualities are measured against different scales.

The same is true when identifying anomalies. The difference is that tracking anomalies is much more important. It is integral to determining if anything is out of the ordinary with your system. Without anomaly detection there is, for instance, no cybersecurity; and if you have to use a proliferation of scales, it slows the process down enormously.

Thankfully, Ron Maurer, a Senior Researcher at Hewlett Packard Labs Israel in Haifa, has created a new and innovative statistical technology that allows the user to compare anomalies in disparate qualities on a shared scale. That technology is now one of the APIs developed by Labs that are available with HPE Haven OnDemand.

“HPE Haven OnDemand is the perfect platform to host these new APIs from Labs,” said Ruth Bergman, Director of Analytics at Labs, “and the fact that anyone can go signup on and seconds later can use the APIs directly from their browser is the ultimate proof that this technology is real and that it works!”

Apples and oranges

According to Doron Shaked, Research Manager and Principal Researcher at Hewlett Packard Labs Israel, corporate groups across the world have to deal with huge operational data sets; and as much as they work at reducing the size of the data they need to examine, it is usually still so large that human interaction is impractical.

“We cannot look at all the data,” said Shaked, “so we have built mechanisms to identify interesting events – anomalies – within that data. This way we can look first at the most severely anomalous data. Yet, until now there was no simple way to order anomalies by severity.

Now, you can use SANE, the “scalable anomaly exploration” technology that Maurer developed.

The data madhouse

SANE is not sui generis, according to Maurer.

“There are quite a few statistical tools you can use to compare apples to oranges,” he said, “but most tend to be very good at comparing deviations from some ‘normal’ average. Yet the comparison between deviations depends in intricate ways on properties of the ‘normal’ part of the data. By focusing instead on the statistics of the outliers, our method eliminates the sensitivity of the anomaly comparison to the ‘normal’ part, so our method has much broader applicability than traditional methods.”

To bring anomaly detection into a single scale, Maurer and his teammates had to invent a new statistical metrical core. They had to make this data madhouse SANE.

“With this method we even get outlier scores to combine to get an aggregate anomaly score per system - now you can compare not only the features like the smell and the shape, you can actually compare apples to oranges”, said Maurer.

SANE in the membrane

SANE, he said, “works beautifully.” What makes it so beautiful is that it works out of the box – “no setup required”. It learns everything it needs from the data, and adapts as more data is available. For an engineer (not to mention a business user), the fewer moving parts the better.

“SANE does not need constant tuning; you can add more data and more types of data and still get excellent results, everything becomes comparable.”

Maurer was surprised by how quickly he got SANE to work. However, it took quite some time to understand how to extend the core idea in order to handle more complexity. The technology is now accessible as part of HPE’s flagship online analytics platform, HPE Haven OnDemand; HPE’s IT Operations Management business unit is also exploring integration into the company’s premier data analytics tool, HPE Operations Analytics.

“We work hard, so for the user it’s easy.”

Illustration in the public domain, courtesy of Wikimedia Commons

0 Kudos
About the Author


Managing Editor, Hewlett Packard Labs