Protect Your Assets
Showing results for 
Search instead for 
Do you mean 

Big Data Security Analytics Part 5: The Challenges of Successful BDSA

‎05-22-2014 09:30 AM - edited ‎06-09-2015 11:27 AM

The path to real value in big data security analytics is not as smooth as it will be as the field matures. First and foremost, as an emerging capability, there is a cultural mandate that we are open to experimentation and that means being open to trying things that might fail. We need to go fast, fail and then adjust, which is the definition of agile rather than applying long waterfall processes. Here are some of the issues that teams commonly struggle with:


Data Quality – Sources of the data must provide consistent and quality data. Garbage in = garbage out. This is one of the most common issues, there is a large variation in how logs are configured and if they are not consistent and appropriately configured it can be very hard to analyze them effectively. 

Normalization – Data must be normalized so that it can be correlated with other data sources without dropping important contextual information. This is the reason the HP Security Analytics solution leverages the common event format (CEF) as this gives a common data model that makes structured analysis much easier. Trying to analyze data without normalizing it limits you you unstructured approaches which while they do exist that are not anywhere near as mature. 

Data Capture
– Proper data capture setting will ensure that not too little and not too much data is collected. Too much will dilute the effectiveness of the solution and too little will result in missed security findings. There is also a need to ensure you have complete capture within a specific log domain as any form of statistical analysis become highly suspect the moment you have non-representative samples. 

Security & Privacy - Captured data must comply with privacy regulations and it must be properly secured. Large data stores result in increased risk for an organization and a rich target for hackers. Protect this information carefully it aggregates much of your risk information. 

Capacity and Cost Planning – Make sure there is enough capacity for the data gathered and that the data is relevant and useful before you commit to storing it long term. There is a sense that more is better but in fact better is better and more is just more, which impact costs and ability to find malicious actors. 


And finally the biggest lesson... Never leave this to a research data scientist. They are focused on the structure and elegance of the problem and could care less about the actual answer. Always pair a domain expert with a data scientist so you bring both mindsets to the problem.


Click here to learn more about HP HAVEn.


Thank you Chris Calvert  for contributing this content.


Check out the next part of this series: Big Data Security Analytics Part 6: 3 Keys to Success

0 Kudos
About the Author


Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Jun 7-9
Las Vegas
Discover 2016 Las Vegas
Discover 2016 in Las Vegas, the ultimate showcase technology event for business and IT professionals to learn, connect, and grow.
Read more
Sep 13-16
National Harbor, MD
HPE Protect 2016
Protect 2016 is our annual conference and is the place to meet the world’s top information security talent, discuss new products and share information...
Read more
View all