Just attended a Big Data symposium courtesy of IBM and thought I’d share a few insights, as probably many of you have heard the term but are not sure what it means to you.

No. 1: Big Data is about looking out of the front window when you drive, not the rearview mirror. What do I mean? The typical decision-making process goes something like this: capture some data, integrate it together, analyze the clean and integrated data, make some decisions, execute. By the time you decide and execute, the data may be too old and have cost you too much. It’s a bit like driving by looking out of your rearview mirror.

Big Data changes this paradigm by allowing you to iteratively sift through data at extreme scale in the wild and draw insights closer to real time. This is a very good thing, and companies that do it well will beat those that don’t.

No. 2: Big is not just big volume. The term “Big Data” is a misnomer and it is causing some confusion. Several of us here at Forrester have been saying for a while that it is about the four “V’s" of data at extreme scale - volume, velocity, variety and variability. I was relieved when IBM came up with three of them; variability being the one they left out.

Some of the most interesting examples we discussed centered on the last 3 V’s – we heard from a researcher who is collecting data on vital signs from prenatal babies and correlating changes in heart rates with early signs of infection. According to her, they collect 90 million data points per patient per day! What do you do with that stream of information? How do you use it to save lives? It is a Big Data Problem.

Greetings — thanks for taking the time to read my inaugural blog! Let me introduce myself by way of continuing a discussion that I started at Practicing EA and CIO.com on innovation and technology that I think strikes at the heart of our challenges as enterprise architects. It also provides a good context for my future research, which I discuss at the end.

Closing The Innovation Gap

In part 1 of this post, I claim that a gap opened while we were fighting the overly complex, expensive current state and trying to help our business partners innovate with new technology.

The gap – We cannot deliver new technology and innovation quickly or cheaply enough.

Shadow IT Is The Symptom, Not The Cause

• The Symptom – We often blame Shadow IT and manual workarounds for increases in complexity, reduction in quality of service, and obscuring true technology costs. These are symptoms of the problem, not the problem itself.
• The Cause – Business users know more about what they need and when they need it and are the most motivated to solve their problems now, not once the budget cycle gets around to funding a project. Central IT, where most EAs practice, is a knowledge store for designing enterprise-scale systems but is constrained in its ability to deliver.

Several recent reports on Forrester.com start with the sentence: "EA organizations often toil out of the limelight . . . " There are fewer and fewer reasons why this should be the case. 

We hear fewer stories of EA teams as purely "the standards police" or with "their heads in the clouds, not producing anything useful." We hear more and more stories of EA teams changing how business and IT plan, taking the lead in application simplification and rationalization, or being the broker for innovation. Infoworld and Forrester want to recognize these success stories with the 2011 Enterprise Architecture Award.

Our 2010 Award winners show the range of ways EA teams are making a difference:

• Discover Financial created an EA repository that aggregates information from its Service Catalog, Fixed Asset, PPM, and Business Goals to provide decision-making insights that saved more than $1M of avoided costs.
• Aetna used its Business Capability Map to combine more than 30 business unit strategies and road maps, highlighting common opportunities and gaps that it then used for its annual planning.

Forrester sees business empowerment — where business areas seek greater autonomy to address their own technology needs — as an inevitable trend. We’ve seen this before: New technology brings business areas new opportunities to improve their performance — from finance (PCs and spreadsheets) to marketing (web and eCommerce) to sales (PDAs). When this occurred, IT was unconnected to the frontlines of the business; IT’s technology was viewed as hard to use, and the result was business-initiated “shadow IT.”

At the recent Forrester Enterprise Architecture Forum in San Francisco, we offered attendees a copy of the new book Empowered, by Forrester analysts Josh Bernoff and Ted Schadler. To get a copy, attendees had to complete a two-question survey. The questions directly related to their readiness to support this round of business empowerment:

“On a scale of 1-5, where 1 = ‘This doesn’t sound like my company at all’ and 5 = ‘This sounds exactly like my company,’ please rate the following questions about your organization:

1. The EA function has close ties with business management.
2. Our technology strategy and standards allow for rapidly changing technologies.”