Security Products
Showing results for 
Search instead for 
Do you mean 

Big Data Security Analytics Part 1: Big Data is a Problem Statement

Kerry_Matre ‎04-24-2014 08:44 AM - edited ‎06-09-2015 11:30 AM

The promise of Big Data

The volume of data being generated and persisted in our organizations is growing at a rapid speed. Traditional toolsets are not built to process the bulk and variety of data in meaningful ways. The panacea of big data is that we can use all of this increased knowledge to increase sales, enhance marketing efforts and reduce risks—all while increasing security within our operations. However, big data is a problem statement, not a solution.


The problem of Big Data

The problem of big data arose in the modern world of exponential data volume growth.  Simply defined, big data has three criteria, or the three V’s, to meet the popular convention:


  • Volume:  The terabytes, or even petabytes or more, of data being created daily.
  • Velocity:  The speed at which this data is generated but must also be processed.
  • Variety: Indicative of the various data sources producing this data. 

The problem set of dealing with the three “V’s” in security is to come up with meaningful, actionable, and timely security intelligence from it. The solution is security analytics.


Security analytics is the process of performing analysis on magnitudes of data, adding context to the information collected and then deriving answers and actionable knowledge. It is a process of analyzing the needles AND the haystacks.


The needle and the haystackHaystack.jpg

The needle in the haystack analogy is well known in the security industry. Tools and applications exist to attempt to find that one magical event or attack within a large stream or pool of events. The premise is that there is so much network and security data (“hay” in our analogy) that exists in organizations and the security analyst’s job is to find the indicators of compromise (“needles”) within that hay. A security information and event monitoring (SIEM) system enables you to take that security data and correlate it together in near real-time to identify threats. We pass a stream of “hay” into the SIEM and we find needles.


However, often times it is worth taking a step back and examining the haystack itself. This is where security can greatly benefit from security analytics. Let’s say we have found 10 actionable event "needles" per day via our SIEM and maintain a consistent 10 events every day for the last year.  Data analytics allows you to look at the entire haystack and ask the question "Given the size and makeup of the haystack, is finding 10 needles a day appropriate?" or even to ask questions such as "At what point is this haystack stale and not worth digging for needles in?"


Click here to learn more about HP HAVEn.


Check out part 2 of this BDSA blog series: Big Data Security Analytics Part 2: Security Analytics Results from a Combination of Tools

About the Author


Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
1-3 December 2015
Discover 2015 London
Discover 2015 in London, the ultimate showcase technology event for business and IT professionals to learn, connect, and grow.
Read more
November 2015
Software Online Expert Days
Join us online to talk directly with our Software experts.
Read more
View all