I met with a group of clients recently on the evolution of data management and big data. One retailer asked, “Are you seeing the business going to external sources to do Big Data?”
My first reaction was, “NO!” Yet, as I thought about it more and went back to my own roots as an analyst, the answer is most likely, “YES!”
Ignoring nomenclature, the reality is that the business is not only going to external sources for big data, but they have been doing it for years. Think about it; organizations that have considered data a strategic tool have invested heavily in big data going back to when mainframes came into vogue. More recently, banking, retail, consumer packaged goods, and logistics have marquis case studies on what sophisticated data use can do.
Before Hadoop, before massive parallel processing, where did the business turn? Many have had relationships with market research organizations, consultancies, and agencies to get them the sophisticated analysis that they need.
Think about the fact, too, that at the beginning of social media, it was PR agencies that developed the first big data analysis and visualization of Twitter, LinkedIn, and Facebook influence. In a past life, I worked at ComScore Networks, an aggregator and market research firm analyzing and trending online behavior. When I joined, they had the largest and fastest growing private cloud to collect web traffic globally. Now, that was big data.
Today, the data paints a split picture. When surveying IT across various surveys, social media and online analysis is a small percentage of business intelligence and analytics that is supported. However, when we look to the marketing and strategy clients at Forrester, there is a completely opposite picture.
When I posted a blog on Don’t Establish Data Management Standards (it was also on Information Management's website as Data Management Standards are a Barrier) I expected some resistance. I mean, why post a blog and not have the courage to be provocative, right? However, I have to say I was surprised at the level of resistance. Although, I also have to point out that this blog was also one of the most syndicated and recommended I have had. I will assume that there is a bit of an agreement with it as well as I didn't see any qualifiers in tweets that I was completely crazy. Anyway, here are just a few dissenter comments:
“This article would be funny if it wasn't so sad...you can't do *anything* in IT (especially innovate) without standing on the shoulders of some standard.” – John O
“Show me data management without standards and good process to review and update them and I'll show you the mortgage crisis which developed during 2007.” – Jim F
“This article is alarmingly naive, detrimental, and counterproductive. Let me count the ways…” – Cynthia H
"No control leads to caos... I would be amused to watch the reaction of the ISO engineer while reading this article :)." - Eduardo G (I would too!)
After wiping the rotten tomatoes from my face from that, here are some points made that get to the nuance I was hoping to create a discussion on: