I met with a group of clients recently on the evolution of data management and big data.  One retailer asked, “Are you seeing the business going to external sources to do Big Data?”

My first reaction was, “NO!” Yet, as I thought about it more and went back to my own roots as an analyst, the answer is most likely, “YES!”

Ignoring nomenclature, the reality is that the business is not only going to external sources for big data, but they have been doing it for years.  Think about it; organizations that have considered data a strategic tool have invested heavily in big data going back to when mainframes came into vogue.  More recently, banking, retail, consumer packaged goods, and logistics have marquis case studies on what sophisticated data use can do. 

Before Hadoop, before massive parallel processing, where did the business turn?  Many have had relationships with market research organizations, consultancies, and agencies to get them the sophisticated analysis that they need. 

Think about the fact, too, that at the beginning of social media, it was PR agencies that developed the first big data analysis and visualization of Twitter, LinkedIn, and Facebook influence.  In a past life, I worked at ComScore Networks, an aggregator and market research firm analyzing and trending online behavior.  When I joined, they had the largest and fastest growing private cloud to collect web traffic globally. Now, that was big data.

Today, the data paints a split picture.  When surveying IT across various surveys, social media and online analysis is a small percentage of business intelligence and analytics that is supported.  However, when we look to the marketing and strategy clients at Forrester, there is a completely opposite picture. 

Big data is creating a bigger divide between the business and IT.  It could be skills. It could be technology. It could be IT’s ability to support the business when the business needs it.  It could be all of those things. What makes big data interesting is both the fact that you have access to more data and you have technology to make analysis faster and cheeper.  But, the argument is if it should be a core competency managed internally, or can big data still be delivered by research and consultancies?

My questions to you…Is big data outside of IT a good thing?  Is it necessary for now due to cost and skills?  Or, is this just smoke and mirrors?