Digital Transformation
Showing results for 
Search instead for 
Did you mean: 

Big Data, dark data and the Internet of Things: Making sense of the data deluge


I’ve been watching the bad weather in North America and its knock-on effect of flooding in the UK, and it occurred to me that it parallels the increasing deluge of data that we’re collecting in data centres all over the world.


Opinions vary on just how much data will grow over the next few years, but 40 zettabytes  by 2020 seems like a nice round number.


What are we doing with all this data?

To date we’ve had data mining, analytics and Big Data, but no matter what the tag line is, we’re still only on the edge of the ability to gather the wealth of true information that’s held in these data sources. Having said that, progress to date has been impressive.


One example is the UK tax office HMRC. They’re using tools that gather and analyse data from multiple sources such as social media, contractor websites and company accounts in order to find evidence of tax fraud and corruption. HMRC expects to recoup £7 billion (about $ 11.65 billion) using these techniques.


In the same vein, the New York City Human Resources Administration uses a similar approach to verify eligibility for benefits, making sure people are who they say they are and they live where they say they live.


In the business world, the fashion retailer Guess? has found that using tools to significantly increase the speed at which they can analyse data has led to more staff asking a wider range of questions of the data than were ever considered before. This way, the company can run more efficiently while providing a better customer service.


Dark data

You really can’t have too many IT terms can you? Dark data, according to Gartner is “the information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes.”  Much like dark matter in science, which apparently is what most of the universe is made up of, dark data takes up most of the disk space in many organisations.


Internet of Things

So now we’ve recognised that not only is there a vast amount of data that is probably useful in some new way, which is held in a structured or semi-structure fashion, but also we have arguably even more data hanging around that apparently has no value.


And to add to this, there’s now all the talk about the Internet of Things. The idea here is to jam sensors into absolutely everything – even the cat – so that more and more data can be collected. This will enable businesses to know much more about things like workflow and supply chains, plus there’s the fridge that can automatically re-order your eggs and milk.


According to Mike McNamara, CIO of Tesco, “This will have an enormous impact on everything – on the operations, how we manage buildings, the lighting, the cooling, everything we do and absolutely it will have a huge impact on how we manage the supply chain. I will openly embrace it when it becomes widely available, as I think it will be hugely beneficial. I just don’t think it’s here yet.”


HP’s work in collaboration with Conservation International – HP Earth Insights – is working to process huge amounts of data collected from devices located in 16 tropical forests across the world. The analysis from HP Earth Insights is being used to develop policies regarding hunting and other causes of species loss in these ecosystems. Having the results of the analysis available in near real-time means that threat response can be proactive – no more waiting months to be able to decide what to do.


Google is spending some of its loose change in this area with a $3.2 billion acquisition of Nest, the maker of Internet-connected thermostats and smoke detectors. This is all part of a plan to develop the “conscious home.” With Google’s proven ability to get the most out of the Internet, this is certainly a move to watch.


Smart cities

The idea of the “smart city” is just an application of these ideas. Here the goal will be to have every city “asset” connected to the Internet – power, water, refuse, traffic, parking, the list goes on.

Analysing this kind of data can deliver some useful things to citizens, such as finding a parking spot or telling you what the level of pollution is today.


However, there is likely to be a great deal more that can be achieved through this data analysis. Given that the city is much like a large business, the ability to run the city more efficiently and cost-effectively is likely to be a huge driver to deliver this kind of system.


The need for speed

The common thread to all of this discussion is the need to analyse all of this data in real time. It’s no good having a nice car-parking app that tells me that five minutes ago there was a space on the London Road.


 So how to achieve this? It’s a combination of some clever technology that runs in big data centres, plus the use of all the compute power that we carry around in the form of smartphones and tablets. We have a phenomenal amount of processing resource available – it’s just a question of harnessing it in the most effective way.


Compute power is by no means the most important part of this puzzle – the software you use to perform the analysis is what makes the difference between information now or next week. In most “Big Data” analysis situations – whether it’s protecting endangered species or predicting shopper behaviour – it’s critical to have as close to real-time analysis as possible. Today we no longer need wait weeks for this information; platforms like HP’s HAVEn have the ability to deliver intelligence in moments.


What next?

Is Big Data on your agenda yet? How much dark data do you have in your organisation? What could you derive from your data that would save your company millions?


Next time, I’ll be looking at something a little more ‘now’: How to deliver apps in hours, not months.


Alastair Corbett leads HP’s UK&I Software Business Unit and has responsibility for its strategy, the promotion and selling of the IT Performance Suite and related services. Prior to this role, Alastair was responsible for defining the new sales strategy and go-to Market models for Worldwide Software Sales, and before that, he successfully led the Worldwide Services Operations team for HP Software. Alastair joined HP from Peregrine as a result of the acquisition in 2005, where he held the role of VP International Operations and was responsible for all Finance and Operations activities in EMEA and APJ. He also led the integration activity for EMEA, as well as leading the Sales Operations function.


Related links:

Blog - 2014: A game changing year for IT

Keeping the lights on in a competitive world

0 Kudos
About the Author


This account is for guest bloggers. The blog post will identify the blogger.

Jan 30-31, 2018
Expert Days - 2018
Visit this forum and get the schedules for online HPE Expert Days where you can talk to HPE product experts, R&D and support team members and get answ...
Read more
See posts for dates
HPE Webinars - 2018
Find out about this year's live broadcasts and on-demand webinars.
Read more
View all