Enterprise Services
Showing results for 
Search instead for 
Do you mean 

What you should know about investing in Big Data

‎05-01-2014 08:56 AM - edited ‎09-30-2015 07:03 AM

By:  Vilas Prabhu, Enterprise Architect, Hewlett Packard Company


Beer mugiStock_000002626099Small.jpgLate last year IDC published a maturity assessment for the Big Data and Analytics capability (IDC - "Maturity Model Benchmark: Big Data and Analytics in North America" # 245197, Dec 2013). According to their analysis, most organisations are at a “repeatable” maturity level when it comes to big data and analytics capability.


IDC report defines these maturity levels as follows:


Repeatable: Intentional, defined requirements and processes, unbudgeted funding, and project management and resource allocation inefficiency


Managed: Measured; project, process, and program performance measurement influences investment decisions and standards emerge


The assessment, although North American focused, also applies to most UK Big Data practitioner organisations. Organisations do get value for their Big Data efforts, but mostly by heroics of the people involved rather than institutionalized tools, methodologies and practices. 


There are reasons for that. One of them is, in the world of Big Data, everything is BIG.


The business problems that need to be tackled are big. The value that the solution will generate is big, the architectural effort is big and, when it comes to the IT that is needed (be it Compute, Storage or Network), the requirements are also big. It is therefore no surprise that the investments needed will be BIG.


This is where the trouble starts. Capabilities in corporations start small, prove their effectiveness and consolidate into a sustainable venture before they get any substantial investment. Given the nature of Big Data problems and solutions, it is not possible to start small. A first of a kind small Big Data solution (how’s that for an oxymoron?) would be more demanding of investment than a typical first of a kind solution. And this just gets you to “repeatable” level.


Without the further investments, the needed institutionalization cannot take place and Big Data capability maturity will remain at “repeatable” levels. A question that needs to be addressed at this point is: do we need to move to a “managed” maturity level for Big Data and Analytics capability?


To answer this question we need to consider a few things. Most important is to consider whether business value generated by using Big Data capability is a one-off (or infrequent) occurrence or recurring (or frequent) occurrence. One must also consider the integration capability with operational systems and its likelihood to cause disruptions to these operations.


If it generates value infrequently and not heavily integrated operationally, then the company may be better off staying at the “repeatable” level. On the other hand, if Big Data and Analytics capability can consistently generate significant business value and is heavily integrated operationally, then you need to move to a “managed” maturity level.


One example of such a situation is what is known as “closed loop” solutions. Such solutions employ Big Data and Analytics to make decisions which are fed back into the operational system in real time, allowing for operational systems to take calculated actions.  Let me illustrate with an example.


A company makes a semi-perishable product like beer. A Big Data solution can have a significant impact on its business. Such a solution can predict demand and shipping time in different geographies based on parameters such as current weather condition, current consumption, social happenings (e.g. Super bowl teams, match dates, fan engagement), road status coming from its lorries and suchlike. It can then feed these predictions into a production planning and logistics system so that the right quantity of beer is produced and shipped at right times to various stores without delay. It can even track these shipments and take corrective actions. This alleviates the need for excessive inventory. It will also insure the beer gets consumed instantly (provided the demand predictions were correct). The consumer will then be satisfied because they got their tasty, fresh beer when they wanted it, resulting in increased brand loyalty – a valuable business result.


In the above example, different parameters can generate fluctuations in demand. The ability to predict demand needs frequent tweaking of the solution for every new parameter. In such a case the capability needs to be institutionalised at a “managed” level. Anything else in such a complex environment, would be disastrous, as a company would not want to rely on individual heroics to make its operational systems work.


So the answer to our question is an emphatic yes, we do need to move Big Data and Analytics capability to a “managed“ maturity level.


The next question is how; how do we jump to a “managed” maturity level when it takes a huge investment just to reach a “repeatable” level?


Moving from a “repeatable” to a “managed” level of maturity requires action on a lot of fronts. It requires intent, data, people, processes and technological aspects of the capability to be addressed. It is a transformation which needs an enterprise architecture focused approach which I will address in my next blog. Stay tuned!


Previous blogs by Vilas Prabhu:

Related links: 

About the Author


Vilas Prabhu - cropped - master.pngVilas Prabhu, Enterprise Architect, Hewlett Packard Company

Vilas is a technology thought leader. He was instrumental in development and implementation of a Model Driven Architecture toolset for many large program engagements.  Vilas currently promotes architecture-led approaches for Enterprise’s Information Systems transformation and sustenance, specifically in the area of analytics and data management.

0 Kudos
About the Author


Nov 29 - Dec 1
Discover 2016 London
Learn how to thrive in a world of digital transformation at our biggest event of the year, Discover 2016 London, November 29 - December 1.
Read more
Each Month in 2016
Software Expert Days - 2016
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all