Protect Your Assets
Showing results for 
Search instead for 
Do you mean 

Big Data Security Analytics Part 5: The Challenges of Successful BDSA

Kerry_Matre ‎05-22-2014 09:30 AM - edited ‎06-09-2015 11:27 AM

The path to real value in big data security analytics is not as smooth as it will be as the field matures. First and foremost, as an emerging capability, there is a cultural mandate that we are open to experimentation and that means being open to trying things that might fail. We need to go fast, fail and then adjust, which is the definition of agile rather than applying long waterfall processes. Here are some of the issues that teams commonly struggle with:


Data Quality – Sources of the data must provide consistent and quality data. Garbage in = garbage out. This is one of the most common issues, there is a large variation in how logs are configured and if they are not consistent and appropriately configured it can be very hard to analyze them effectively. 

Normalization – Data must be normalized so that it can be correlated with other data sources without dropping important contextual information. This is the reason the HP Security Analytics solution leverages the common event format (CEF) as this gives a common data model that makes structured analysis much easier. Trying to analyze data without normalizing it limits you you unstructured approaches which while they do exist that are not anywhere near as mature. 

Data Capture
– Proper data capture setting will ensure that not too little and not too much data is collected. Too much will dilute the effectiveness of the solution and too little will result in missed security findings. There is also a need to ensure you have complete capture within a specific log domain as any form of statistical analysis become highly suspect the moment you have non-representative samples. 

Security & Privacy - Captured data must comply with privacy regulations and it must be properly secured. Large data stores result in increased risk for an organization and a rich target for hackers. Protect this information carefully it aggregates much of your risk information. 

Capacity and Cost Planning – Make sure there is enough capacity for the data gathered and that the data is relevant and useful before you commit to storing it long term. There is a sense that more is better but in fact better is better and more is just more, which impact costs and ability to find malicious actors. 


And finally the biggest lesson... Never leave this to a research data scientist. They are focused on the structure and elegance of the problem and could care less about the actual answer. Always pair a domain expert with a data scientist so you bring both mindsets to the problem.


Click here to learn more about HP HAVEn.


Thank you Chris Calvert  for contributing this content.


Check out the next part of this series: Big Data Security Analytics Part 6: 3 Keys to Success

0 Kudos
About the Author


27 Feb - 2 March 2017
Barcelona | Fira Gran Via
Mobile World Congress 2017
Hewlett Packard Enterprise at Mobile World Congress 2017, Barcelona | Fira Gran Via Location: Hall 3, Booth 3E11
Read more
Each Month in 2017
Software Expert Days - 2017
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all