Enterprise Services
Showing results for 
Search instead for 
Do you mean 

What you should know about investing in Big Data

Apps_Services ‎05-01-2014 08:56 AM - edited ‎09-30-2015 07:03 AM

By:  Vilas Prabhu, Enterprise Architect, Hewlett Packard Company


Beer mugiStock_000002626099Small.jpgLate last year IDC published a maturity assessment for the Big Data and Analytics capability (IDC - "Maturity Model Benchmark: Big Data and Analytics in North America" # 245197, Dec 2013). According to their analysis, most organisations are at a “repeatable” maturity level when it comes to big data and analytics capability.


IDC report defines these maturity levels as follows:


Repeatable: Intentional, defined requirements and processes, unbudgeted funding, and project management and resource allocation inefficiency


Managed: Measured; project, process, and program performance measurement influences investment decisions and standards emerge


The assessment, although North American focused, also applies to most UK Big Data practitioner organisations. Organisations do get value for their Big Data efforts, but mostly by heroics of the people involved rather than institutionalized tools, methodologies and practices. 


There are reasons for that. One of them is, in the world of Big Data, everything is BIG.


The business problems that need to be tackled are big. The value that the solution will generate is big, the architectural effort is big and, when it comes to the IT that is needed (be it Compute, Storage or Network), the requirements are also big. It is therefore no surprise that the investments needed will be BIG.


This is where the trouble starts. Capabilities in corporations start small, prove their effectiveness and consolidate into a sustainable venture before they get any substantial investment. Given the nature of Big Data problems and solutions, it is not possible to start small. A first of a kind small Big Data solution (how’s that for an oxymoron?) would be more demanding of investment than a typical first of a kind solution. And this just gets you to “repeatable” level.


Without the further investments, the needed institutionalization cannot take place and Big Data capability maturity will remain at “repeatable” levels. A question that needs to be addressed at this point is: do we need to move to a “managed” maturity level for Big Data and Analytics capability?


To answer this question we need to consider a few things. Most important is to consider whether business value generated by using Big Data capability is a one-off (or infrequent) occurrence or recurring (or frequent) occurrence. One must also consider the integration capability with operational systems and its likelihood to cause disruptions to these operations.


If it generates value infrequently and not heavily integrated operationally, then the company may be better off staying at the “repeatable” level. On the other hand, if Big Data and Analytics capability can consistently generate significant business value and is heavily integrated operationally, then you need to move to a “managed” maturity level.


One example of such a situation is what is known as “closed loop” solutions. Such solutions employ Big Data and Analytics to make decisions which are fed back into the operational system in real time, allowing for operational systems to take calculated actions.  Let me illustrate with an example.


A company makes a semi-perishable product like beer. A Big Data solution can have a significant impact on its business. Such a solution can predict demand and shipping time in different geographies based on parameters such as current weather condition, current consumption, social happenings (e.g. Super bowl teams, match dates, fan engagement), road status coming from its lorries and suchlike. It can then feed these predictions into a production planning and logistics system so that the right quantity of beer is produced and shipped at right times to various stores without delay. It can even track these shipments and take corrective actions. This alleviates the need for excessive inventory. It will also insure the beer gets consumed instantly (provided the demand predictions were correct). The consumer will then be satisfied because they got their tasty, fresh beer when they wanted it, resulting in increased brand loyalty – a valuable business result.


In the above example, different parameters can generate fluctuations in demand. The ability to predict demand needs frequent tweaking of the solution for every new parameter. In such a case the capability needs to be institutionalised at a “managed” level. Anything else in such a complex environment, would be disastrous, as a company would not want to rely on individual heroics to make its operational systems work.


So the answer to our question is an emphatic yes, we do need to move Big Data and Analytics capability to a “managed“ maturity level.


The next question is how; how do we jump to a “managed” maturity level when it takes a huge investment just to reach a “repeatable” level?


Moving from a “repeatable” to a “managed” level of maturity requires action on a lot of fronts. It requires intent, data, people, processes and technological aspects of the capability to be addressed. It is a transformation which needs an enterprise architecture focused approach which I will address in my next blog. Stay tuned!


Previous blogs by Vilas Prabhu:

Related links: 

About the Author


Vilas Prabhu - cropped - master.pngVilas Prabhu, Enterprise Architect, Hewlett Packard Company

Vilas is a technology thought leader. He was instrumental in development and implementation of a Model Driven Architecture toolset for many large program engagements.  Vilas currently promotes architecture-led approaches for Enterprise’s Information Systems transformation and sustenance, specifically in the area of analytics and data management.

About the Author


Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
January 2016
Software Expert Days - 2016
Join us online to talk directly with our Software experts during the online Expert Days - see details below. Software experts do not monitor this foru...
Read more
See board event postings
Vivit Events - 2016
Learn about upcoming Vivit webinars and live events in 2016.
Read more
View all