Grounded in the Cloud
cancel
Showing results for 
Search instead for 
Did you mean: 

My Cloud Learning Journey: Part 7 “Big Data, Cloud and Astral Ballet”

HaleyC

 

large (3).jpg

Guest Post by Tim Clayton, Marketing Business Services

A hundred billion suns swim silently through endless night. Galaxies by the million drift slowly apart. They will meet again one day.


 

Thousands of years ago….

A hunter, recognizably a man but not so far from an ape, makes his way home from a long day wandering in unfamiliar terrain. He is tired and hungry. He sits on a rock, lays down what little prey he has to take home to his family, and looks up to the sky. All at once he knows the way home.

Many years later…

We are still long before our digital era. In South America, thousands of years before it took that name, a Mayan carves the world’s first calendar into a piece of stone, carefully calculating the stars and their movement in the heavens.

Across the world, at roughly the same time…

Imhotep, the greatest mind in ancient Egypt, designs a pyramid with a starred ceiling to hang for eternity above the sarcophagus of the king.  


 It’s our natural bent as people to believe that the contemporary age is always the vanguard, and that all of our discoveries are new and were unimaginable to those that came before us. This may be true of the tools and technology we use (neither our Australopithecus man nor Imhotep could have comprehended a modern computer) but the study and search for patterns has always been the same.

We think that Big Data analytics—finding patterns in huge sets of structured and unstructured data—is a clever new device that we invented in just the last few years. In truth, it is as old as man. When that hunter wandered home alone, he looked to the sky and was faced with an infinite amount of data that was far too great for his mind to take in. In fact, the human eye can only see a fraction of the fraction that has reached us with its light. And yet he focused on the myriad stars, found a pattern that had meaning, and solved the problem that had presented itself to him.

Our tools may be different, but the analysis of unfathomable data sets in search of reason is as old as our own species.

It started with a Big Bang. We have reached the point of Big Data. It is just another stop along the way in the journey of man.

stars.jpg

I am talking to Sanjay Baronia from HPE Big Data Marketing about how the cloud and Big Data work together. Sanjay is another who has travelled a long way; he moved to Cambridge, Massachusetts, to study some 25 years ago and has settled for good. I asked him if he would do the same if he were a young man in India today and he is not sure if the need is the same. “We had good education in India but there were not the job opportunities that were in the US. Nowadays there is less migration. It is a very business-conducive environment.” That may be the case. India is holding on to its best people and that is for the good, although it seems to me that India’s loss with Sanjay is America’s gain, as his career has taken him through some tech giants and startups to his position in HPE.

“The term Big Data was one I first heard only about 4 years ago,” says Sanjay, “but we used to talk about a ‘data explosion’ when I worked at EMC. The real change in data amounts came at around the time of the whole Y2K thing. But the history of doing computational analytics with large data sets goes back to the fifties when Fair Issac were using analytics to do credit scores for lenders. Essentially, nothing has changed about the way those scores are measured today; the only difference is that Big Data allows companies to look at data from a range of sources. It is a much bigger scale, too.”

When Sanjay talks about a range of sources, what he means is that we are no longer just looking at bank balances. It is now possible for lenders to adjust a potential client’s credit score based upon any number of factors. “They could take into account how many people you have on your LinkedIn profile and in what positions those people are, to make a more accurate prediction of your earnings potential in five years,” says Sanjay. This is the kind of analysis that seems invasive and uncomfortable at first but may be in the borrower’s best interests in the long run.

The problems arise, like with all technology, when the tools are used in ways that seem to infringe upon people’s personal rights and freedoms. Sanjay cites the example of a firm that is using such tools to propagate information among the friends and acquaintances of non-paying debtors about their financial issues—to shame them into paying. This is where the lines are crossed and where the media uproar drowns out the good that Big Data is doing for all sizes of business.

“It doesn’t matter how big you are nowadays; Big Data is no longer the privy of large organizations,” says Sanjay. Even though majority of our customers are on premise but we see the biggest growth in the cloud for Big Data. Smaller companies are moving to the Cloud for analytics, because of agility, elasticity and cost.”

All any company really needs is a data scientist to feed in the algorithms and a platform on which to store and process all the data. For larger organizations, like most HPE Vertica customers, that is often still on premise servers, but cloud-hosted solutions mean that all companies can now follow the same simple path to gaining insight from unfathomably deep oceans of statistics:

 

bigdataflow.jpg

 When we talk about Big Data we are really only talking about a collection of things. Big Data itself is an idle pile of structured and unstructured information. The four orange boxes in this flow chart define the points where a human must use free thought and take meaningful action. They are the parts of this journey that are the same for our prehistoric hunter and for any modern-day business.

  • Identify the problem: For the hunter this is being lost, for a business it may mean making losses
  • Run the models: For our hunter this was raising his head to the stars, for a modern business it having a star data scientist create algorithms to process data
  • Interpret meaning: Once you see the picture, you need to understand what it means for you
  • Take action: Our hunter started walking. A modern businesses usually has a hundred strategy meetings and conference calls… then starts walking!

After Sanjay had explained all this to me, I had three key questions for him:

Can we remove the need for humans? Can’t the machines collect the data and write the models themselves? Can’t we create a system to instantly run questions and give us insight?

It is a long question with a short answer: No.

Sanjay doesn’t see the technology existing yet, and doesn’t think it is perhaps even the right way to go. “It is a step too far to take data scientists out of the loop. Humans have to have a problem to solve. We can only find what we are looking for. A business has to identify a need and then run the analytics to find if there are correlations that can help answer that problem. A machine would not know that the need existed. It would only see the data but not necessarily know that there is a problem. Human beings do not work well unless they are presented with a task to solve something, but we can use tools to very effectively solve those big questions.”

Is analyzing Big Data always going to lead to better business results?

Again, the answer seems to lie in that subtle distinction between humans and machines. To err is human. “One thing we have to make sure of is that our data is clean. Even when analyzing huge sets of data, there is a chance that bad inputs can lead to erroneous results,” says Sanjay.

The other problem comes with the issue of what action to take. Even when we have perfect visualizations from clean data, we need to interpret the results correctly. Any two people can look at seemingly obvious results and draw different conclusions about the best course of action. Or people may not act at all, even if they have the results that tell a clear story. Rich analytics delivered at scale should lead to better business.

There is no guarantee but organizations considering cloud as a deployment/consumption model for analytics should see positive results if they do the groundwork and get the right data onto the right platform and then make informed decisions to act.

With my mind on the distant past, I also want to hear about the future of Big Data and business.

Is there any aspect of business, indeed life on earth and beyond, that cannot be analyzed with Big Data analytics and somehow improved?

“It is hard to see where it won’t be,” Sanjay concedes. “It is in every industry already and businesses which use Big Data now will use it even more efficiently as they keep running the data and refining their results.” But Sanjay does not think that Big Data will produce results that we will implement in all aspects of our lives: “I think that there are aspects of our lives that Big Data has the capacity to improve but we, as humans, prefer to deal with using our own emotional intelligence—even if it goes against the data. We don’t want these things in our interpersonal relationships.”

Our hunter gatherer arrives home. He is exhausted but buoyed by his children who run into his arms. His wife takes the meat and begins to prepare a meal. He lays on the grass and looks up at the stars. They are there if he needs them, to help him find the way home—but for now they are just idle lights in the night sky. They are beautiful.

About the Author

HaleyC

I manage the HPE Helion social media brand accounts promoting the enterprise cloud solutions at HPE for hybrid, public, and private clouds.I have put my toes in the ocean of cloud evangelism for the enterprise IT industry. But my expertise is in Social Media and Digital Marketing.

Events
28-30 November
Madrid, Spain
Discover 2017 Madrid
Join us for Hewlett Packard Enterprise Discover 2017 Madrid, taking place 28-30 November at the Feria de Madrid Convention Center
Read more
HPE at Worldwide IT Conferences and Events -  2017
Learn about IT conferences and events  where Hewlett Packard Enterprise has a presence
Read more
View all