Big data is undergoing big change, but most companies are missing it or just grasping at the edges. My colleague Fatemeh Khatibloo and I have just completed an exhaustive study of the big data phenomenon. We found a familiar pattern: business confusion in the face of stern warnings about the dangers of big data and vendor-sponsored papers extolling its benefits. Here’s what we found hidden beneath the buzz:
As data explodes, so do old ways of doing business.
Everywhere we look, we find businesses using more diverse, messier, and larger data sets to stay competitive in the age of the customer — like the consumer goods firm that allocated marketing dollars based on flu trend predictions and the oil and gas companies that used weather data to predict iceberg flows and extend their drilling season. Savvy businesses find ways to turn more data into a competitive advantage. If your firm doesn’t get this, it won’t be pretty — starting in the not too distant future.
Technology managers and architects can’t afford to sit back and think that their Hadoop project will deliver everything the business needs. Nor can you afford to think that big data isn’t for you because you don’t have that much data. Why? Because “big data” is really the practices and technologies that close the gap between the available data and the ability to turn that data into business insight — insight that your firm needs to survive and thrive in the age of the customer. Four things to understand: