Protect Your Assets
Showing results for 
Search instead for 
Do you mean 

Big Data Security Analytics Part 1: Big Data is a Problem Statement

Kerry_Matre ‎04-24-2014 08:44 AM - edited ‎06-09-2015 11:30 AM

The promise of Big Data

The volume of data being generated and persisted in our organizations is growing at a rapid speed. Traditional toolsets are not built to process the bulk and variety of data in meaningful ways. The panacea of big data is that we can use all of this increased knowledge to increase sales, enhance marketing efforts and reduce risks—all while increasing security within our operations. However, big data is a problem statement, not a solution.


The problem of Big Data

The problem of big data arose in the modern world of exponential data volume growth.  Simply defined, big data has three criteria, or the three V’s, to meet the popular convention:


  • Volume:  The terabytes, or even petabytes or more, of data being created daily.
  • Velocity:  The speed at which this data is generated but must also be processed.
  • Variety: Indicative of the various data sources producing this data. 

The problem set of dealing with the three “V’s” in security is to come up with meaningful, actionable, and timely security intelligence from it. The solution is security analytics.


Security analytics is the process of performing analysis on magnitudes of data, adding context to the information collected and then deriving answers and actionable knowledge. It is a process of analyzing the needles AND the haystacks.


The needle and the haystack

The needle in the haystack analogy is well known in the security industry. Tools and applications exist to attempt to find that one magical event or attack within a large stream or pool of events. The premise is that there is so much network and security data (“hay” in our analogy) that exists in organizations and the security analyst’s job is to find the indicators of compromise (“needles”) within that hay. A security information and event monitoring (SIEM) system enables you to take that security data and correlate it together in near real-time to identify threats. We pass a stream of “hay” into the SIEM and we find needles.


However, often times it is worth taking a step back and examining the haystack itself. This is where security can greatly benefit from security analytics. Let’s say we have found 10 actionable event "needles" per day via our SIEM and maintain a consistent 10 events every day for the last year.  Data analytics allows you to look at the entire haystack and ask the question "Given the size and makeup of the haystack, is finding 10 needles a day appropriate?" or even to ask questions such as "At what point is this haystack stale and not worth digging for needles in?"


Click here to learn more about HP HAVEn.


Check out part 2 of this BDSA blog series: Big Data Security Analytics Part 2: Security Analytics Results from a Combination of Tools

0 Kudos
About the Author


27 Feb - 2 March 2017
Barcelona | Fira Gran Via
Mobile World Congress 2017
Hewlett Packard Enterprise at Mobile World Congress 2017, Barcelona | Fira Gran Via Location: Hall 3, Booth 3E11
Read more
Each Month in 2017
Software Expert Days - 2017
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all