Big Data
Showing results for 
Search instead for 
Do you mean 

Analytics for Human Information: New Top Ten Myths of Big Data - Myth #5

ChrisSurdak ‎10-28-2013 10:45 AM - edited ‎02-19-2015 01:38 PM

New Big Data Myth #5 is near and dear to me, as it in many ways is a review of my career in IT and consulting.  While to many people this myth may seem new, and in some ways it is, at its heart this myth is the same old paradigm wrapped up in shiny new rhetoric.  But no matter how many hashtags you apply to the newest coolest, technology, some business issues will come up again and again.


Big Data Myth #5: Big Data Requires the Hiring of Programming “Rock Stars”

Because of the newness of Big Data, there are relatively few people who understand and can implement the technologies that support it; technologies such as Hadoop.  Suddenly, there is a dramatic shortage of Hadoop developers, architects and programmers.  So much so that if you can both spell “Hadoop” and successfully log into LinkedIn, there’s likely a nice pay raise in your future.


These days, Hadoop literati are living the life.  These people are becoming “Rock Stars”.  Companies hold bidding wars to gain their service. They can ask for nearly endless concessions from companies that believe that their entire future depends upon having the best developers available. Followers of this blog know that I’ve written about Hadoop before.

For those of you who have been in IT world for more than ten years know that we have seen this all before. 


In the early 1990’s the client-server revolution was just hitting its stride.  At the time, C was the hot programming language of choice for client-server, and if you knew how to do anything in Lotus Notes you’re kids’ college tuition was in the bag. 


By 2000, we were getting into the “Java Rock Star” era, where Java-literate developers were bringing their dogs to work, having their employers lease BMW convertibles for them, receiving complementary shiatsu massages at their desks, and getting free beer from the work kitchenette.


Every technology has a life cycle. For those with the nerve, and occasionally the ability, to live at the leading edge of each cycle can reap the benefits.


Technology Drivers, Business Impacts

Programming rock stars aside, what is a bit different in this technology cycle versus earlier ones is the impact that Big Data is having on business people.  In the 1990’s you might not know how to write Office macros, or how to create Lotus Notes workflows, but you could still make effective use of the tools made by the developers.  In 2000, you might not have known what an Enterprise Java Bean was, or what they tasted like, but you could still help define what your corporate website should look like, and how it should function. 


Big Data is different.   In order to take advantage of the power of Big Data business people must be literate in the language of statistics, probabilities, Chi distributions and the like. If you are not, you may find yourself becoming marginalized in fairly short order.


Wherefore Art Thou, Data Scientist?

The Big Data boom has led to the new must-have moniker of “Data Scientist.”  I have yet to see a good definition of what this is, but if you can lay claim to this title do so, as soon as possible.  According to the previously-referenced study by McKinsey, the U.S. will have a shortage of “Data Scientists” to the tune of 150,000 – 200,000 by 2020.  Clearly, whatever a data scientist is, it would be a good thing to be one over the next decade.


Quick aside:  I have a sister who is a professional statistician, and who has been analyzing and acting upon customer website click-through data for nearly 15 years, yet she doesn’t call herself a “Data Scientist”. She may not know the first thing about Hadoop, but she can make statistical tools like “R” or SAS sing like Pavarotti. I’ll tell her it’s time to update her resume.


These people have been around for quite a while in a range of industries.  But, as businesses become ever-more dependent upon data-enhanced operations, such people will become critically important to their employers.  These data-literate, business savvy people are likely to become the new Big Data Rock Stars.  Once the technology systems are built and running, it is this group of people who will start to derive value from the systems’ use.


Further, it is this group of people that will determine the success or failure of Big Data efforts.  Companies that effectively use Big Data won’t be “data-driven.”  Rather, they will be “data-optimized.” With a wealth of data at your disposal, your business intuition will actually become more important.  Figuring out what questions should be asked and what the answers mean will be the true determinants of value.  So the data-literate business person, the “Data Scientist”, will become the real rock stars of the future.


As industry guru and Big Data advocate for Microsoft, Kate Crawford has said, “Correlation does not always indicate causation.” A Data Scientist is a person who understands exactly what Kate means by this statement, knows why understanding this distinction is so important, and can actually provide the expertise to figure out how this statement applies to a particular question.


So, by all means hire the best technology people that you possibly can in building out your Big Data platform.  But recognize that the minute your environment is up and running, you’ll need a different set of people with a whole new, and quite rare, set of skills in order to derive value from the technology and the data.  The Big Data revolution will indeed create a whole class of Axl Roses or Eddie Van Halens out there, but they’re more likely to be wearing cardigan sweaters than flip-flops and Grateful Dead T-shirts.


Find out more about HP’s Big Data Analytics platform.  


Click below to continue reading about The New Top Ten Myths of Big Data


0 Kudos
About the Author


Chris Surdak is a Subject Matter Expert on Information Governance, analytics and eDiscovery for HP Autonomy. He has over 20 years of consulting and technology experience, and holds a Juris Doctor from Taft University, an MS from the Wharton School at the University of Pennsylvania, a CISSP Master's Certificate from Villanova and a BS in Mechanical Engineering from Penn State. Chris is author of the Big Data strategy book, "Data Crush," which was recently nominated as International Book of the Year for 2014, by GetAbstract. Chris is also contributing editor and columnist for European Business Review magazine.

Emily Smith MA PMP
on ‎11-09-2013 07:06 AM

Nice post, Chris! Here is a recent related article:


Not much talk about the actual match process to align current "rock stars" with vacancies; though, tweaking her resume might work for your sister when trying to get a job. In fact, the match process might be an even bigger issue "in order to take advantage of the power of Big Data..."


I am looking forward to finding the real leaders in this space. To me, those who can exploit this value chain will emerge the true innovation leader.



June 6 - 8, 2017
Las Vegas, Nevada
Discover 2017 Las Vegas
Join us for HPE Discover 2017 in Las Vegas. The event will be held at the Venetian | Palazzo from June 6-8, 2017.
Read more
Each Month in 2017
Software Expert Days - 2017
Join us online to talk directly with our Software experts during online Expert Days. Find information here about past, current, and upcoming Expert Da...
Read more
View all