Protect Your Assets
Showing results for 
Search instead for 
Do you mean 

Big Data Security Analytics Part 1: Big Data is a Problem Statement

‎04-24-2014 08:44 AM - edited ‎06-09-2015 11:30 AM

The promise of Big Data

The volume of data being generated and persisted in our organizations is growing at a rapid speed. Traditional toolsets are not built to process the bulk and variety of data in meaningful ways. The panacea of big data is that we can use all of this increased knowledge to increase sales, enhance marketing efforts and reduce risks—all while increasing security within our operations. However, big data is a problem statement, not a solution.


The problem of Big Data

The problem of big data arose in the modern world of exponential data volume growth.  Simply defined, big data has three criteria, or the three V’s, to meet the popular convention:


  • Volume:  The terabytes, or even petabytes or more, of data being created daily.
  • Velocity:  The speed at which this data is generated but must also be processed.
  • Variety: Indicative of the various data sources producing this data. 

The problem set of dealing with the three “V’s” in security is to come up with meaningful, actionable, and timely security intelligence from it. The solution is security analytics.


Security analytics is the process of performing analysis on magnitudes of data, adding context to the information collected and then deriving answers and actionable knowledge. It is a process of analyzing the needles AND the haystacks.


The needle and the haystack

The needle in the haystack analogy is well known in the security industry. Tools and applications exist to attempt to find that one magical event or attack within a large stream or pool of events. The premise is that there is so much network and security data (“hay” in our analogy) that exists in organizations and the security analyst’s job is to find the indicators of compromise (“needles”) within that hay. A security information and event monitoring (SIEM) system enables you to take that security data and correlate it together in near real-time to identify threats. We pass a stream of “hay” into the SIEM and we find needles.


However, often times it is worth taking a step back and examining the haystack itself. This is where security can greatly benefit from security analytics. Let’s say we have found 10 actionable event "needles" per day via our SIEM and maintain a consistent 10 events every day for the last year.  Data analytics allows you to look at the entire haystack and ask the question "Given the size and makeup of the haystack, is finding 10 needles a day appropriate?" or even to ask questions such as "At what point is this haystack stale and not worth digging for needles in?"


Click here to learn more about HP HAVEn.


Check out part 2 of this BDSA blog series: Big Data Security Analytics Part 2: Security Analytics Results from a Combination of Tools

0 Kudos
About the Author


Nov 29 - Dec 1
Discover 2016 London
Learn how to thrive in a world of digital transformation at our biggest event of the year, Discover 2016 London, November 29 - December 1.
Read more
Each Month in 2016
Software Expert Days - 2016
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all