How To Have The BI Cake And Eat It Too: A (Or The) BI Prediction For 2014

Rather than going with the usual, ubiquitous, and often (yawn) repetitive “top 10 BI predictions” for the next year, we thought we’d try something different. After all, didn’t the cult movie Highlander prove beyond the shadow of a doubt that “in the end there will be only one”? And didn’t the Lord Of The Rings saga convince us that we need one prediction “to rule them all”? The proposed top BI prediction for 2014 rests on the following indisputable facts:
  • Business and IT are not aligned. Business and IT stakeholders still have a huge BI disconnect (after all these years — what a shocker!). This is not surprising. Business users mostly care about their requirements, which are driven by their roles and responsibilities, daily tasks, internal processes, and dealings with customers (who have neither patience nor interest in enterprises’ internal rules, policies, and processes). These requirements often trump IT goals and objectives to manage risk and security and be frugal and budget minded by standardizing, consolidating, and rationalizing platforms. Alas, these goals and objective often take business and IT in different directions.
  • Requirements are often lost in translation. Business and IT speak different languages. Business speaks in terms of customer satisfaction, improved top and bottom lines, whereas IT speaks in metrics (on a good day), star schemas, facts, and dimensions. Another consideration is that it’s human nature to say what we think others want to hear (yes, we all want our yearly bonus) versus what we really mean. My father, a retired psychiatrist, always taught me to pay less attention to what people say and pay more attention to what people actually do — quite handy and wise fatherly advice that often helps navigate corporate politics.
  • Shadow IT rules. As a result of business/IT disconnect, as much as 80% of all BI content is authored in shadow IT applications, mostly based on spreadsheets.
Now, here’s a dose of a sobering cold shower: A single enterprise BI platform based on a single version of the truth — a single enterprise data warehouse — will not solve this challenge. We all fought this battle in the 1990s and largely lost it. Time to embrace the pragmatic reality of the 21st century. Here it is in a nutshell:
So how do you have the BI cake and eat it too? How do you balance business users’ need to produce their own content with little dependence on bureaucratic IT processes while at the same time minimizing enterprise risk, achieving economies of scale, and getting rid of silos? We think we have an answer — maybe the answer. This is how Forrester proposes to address the dilemma (we already see this approach at a handful of customers with win-win for business and IT success stories). It’s a three-tier environment based on:
  • Individual BI sandboxes
    • Business users rule. Here they author and use about 50% of all BI content with no constraints or limitations. This is where data exploration, discovery, and what-if analyses happen.
    • Tools and technologies: desktop BI applications, in-memory BI applications, spreadsheets, cloud/self- provisioned sandboxes, and server-based sandboxes.
    • IT involvement is strictly limited to infrastructure and tools support plus monitoring to identify patterns, commonalities, and opportunities (using BI on BI) for productionalization. As IT succeeds in moving sandbox-based applications to production, that 50% number goes down.
    • Content produced here is used in individual tasks and low-risk applications.
  • Collaborative, shared, quasi-production BI environment
    • Business users share and collaborate on BI content — about 30% of all BI content — with their colleagues.
    • Tools and technologies: shared folders, portals, and data virtualization.
    • IT steps up its monitoring and now watches for red flags (too much data, too many users, too critical or risky applications, etc.) and opportunities (using BI on BI) to productionalize BI content. As IT succeeds in moving business user-authored BI applications to production, that 30% number goes down.
    • Content produced here is shared within departments and workgroups. Low-risk, low-criticality decisions can be made based on this content.
  • Production environment
    • Business uses and authors BI content — about 20%  of all enterprise BI content — within the limitations and constraints of the enterprise data model, standards, policies, rules, guidelines, etc.
    • Tools and technologies: EDW and enterprise BI platforms.
    • Owned, run, and managed by IT. As IT succeeds in moving sandbox and shared BI content to production, that 20% number goes up.
A step-by-step process would look like this:
  1. A business user builds a spreadsheet (or a model using a desktop in-memory BI tool) that pulls data from multiple data sources (via direct database connections and/or by exporting data from operational apps), creates a pivot table, refreshes it on a daily basis, and uses the model in daily operational tasks.
  2. IT flags that spreadsheet as part of the BI ecosystem, but does not take any action yet, as it’s still a standalone, single-user application that is not setting off any red flags.
  3. Colleagues notice the application and ask the business user to share it with them. The business user saves the spreadsheet/model to a shared folder and his/her colleagues are now leveraging the results in their daily routines too.
  4. IT monitors the newly shared content for patterns such as data complexity, frequency of usage, number of users sharing the content, etc.
  5. When/if some of the established thresholds are reached, IT offers to help automate the process. As a first step, rather than manually importing data sources and manually refreshing the pivot table, IT hooks up the model to all relevant data sources via data virtualization.
  6. When/if the usage patterns of the application indicate that it’s a candidate for production, IT steps into action. It documents the model and the process, and puts it into the request queue. When its time comes, IT moves and integrates data sources into the EDW and gives the model a production look and feel.
  7. One morning, without having been dragged into endless requirements gathering meetings, the team of business users is pleasantly surprised that their homegrown app is now part of the production enterprise BI environment.

Didn’t we just have our BI cake and eat it too? Yes, we did, because:

  • A business user created BI content on his/her own schedule without any constraints and limitations. He/she did not have to waste their valuable time (often aware from customer interactions) in requirements-gathering meetings, steering committees, and building permit processes.
  • IT productionalized the content based on the actual model that’s been battle tested and not on vague and imprecise requirements.
Brilliant, huh? But this is our first attempt to document and understand this process — we don’t yet know very well all of the possible ramifications and implications. We welcome all constructive criticism, suggestions, best practices, lessons learned, etc.


Strong effort in right direction

Hi Boris,

Thanks for sharing this. My initial thoughts:

1) Shared ownership is not only possible, but may likely be required. In studying public info on anti-terrorism and other turf battles that were at least partial causal to fatal events-- people and organizations, the old corp ownership model required tweaking. Fractional ownership of RE doesn't quite capture it --we need far more detailed entity relationship management in the digital world, but it was a big step that allowed a great deal just as pay as you go in cloud computing and SaaS has appeal.

Accountability is necessary --more than less IMO, but let's face it -- ability to work on same team across departments for the overall mission of the org should be the priority--incentives should be structured accordingly. More on the overlap of BI and culture within convergence later..

2) For a recent case I was involved with we had a strong BI team that wanted to pilot Kyield, but they didn't want to pay for the cost of doing so. No surprise there--they cited regulatory and policy challenges, but as Jamie Dimon was quoted today in WSJ as saying --paraphrase--on regulatory reform-- "for too long we pushed this issue too far down in the organization". Bullseye -- that issue and those you discuss here are very directly related and essential to get right-- the CEO, risk committee, COO, CFO etc. should be very involved -- especially in areas where turf is a serious risk. Align the interests for xxx sake.

3) A related issue to ownership and turf battles is protectionist models that are misaligned--including the business analyst who may be too biased or conflicted, the more publicized CIO / IT alignment problem, and increasingly the emerging data scientist problem. The latter actually is quite old-- I saw it with credit report models more than a decade ago that were as much influenced by ideology bent towards financial engineering than credit and data science--and certainly not designed for banks who kept skin in the game--but today I see more evidence of bias towards OSS and certain types of projects to optimize personal short term gain. While we all appreciate the security challenge in terms of jobs and careers of late, extortion is dangerous. The service model is I think in need of maturing around systems or the downside risk is high. I'm seeing service models & cultures ruin critical projects, some before they get started.

4) If you get the NLP and architecture right--that is with good balance and adaptiveness, the business analysts and IT folks along with everyone else in the organization can continue to focus on their top priorities rather than playing coder. As we witnessed with OC, it can be pretty ugly when an organization wants to play with the complexity of enterprise computing who don't have a clue.

CEOs need to ask some hard questions still about what business they are in--and want to be in, core competencies, and do a much better job of tapping the depth of R&D out in the market place, much of which was very carefully designed to extend depth and efficiency they can't do internally--often for very good reasons like it isn't their business or focus.

Realize I drifted slightly but gets into the bigger sand box issues that have been on my mind and in some cases in my face of late:)


I really like seeing the

I really like seeing the increasing regularity of these types of articles. I think it reflects a necessary evolution in thinking - one that recognises reality, and doesn't attempt to deny or obfuscate.
I'd be interested to know if anyone disagrees with this in general terms - especially if they live in a world with TM1, or visualisation tools like Tableau.
I also thought your 7 step example was useful in explaining the (sometimes abstract) concepts.

Getting past Step 1 and then past Step 2

I think this does make sense, provided that the Business User can perform Step 1 and that IT can perform Step 2.

Let me explain:

I have often been in the position of straddling the two worlds - interpreting business requirements and then translating them into something that IT can action.

Getting beyond Step 1:
The projects and interactions that I enjoyed the most were those with Business Users that were able to provide real "working" models (e.g. spread sheets) with "real" numbers, "real" results and "real" data sources. If IT could provide the individual BI sandboxes with the necessary connectivity to the back-end data sources / systems, then these Business Users would be very successful. (These Business Users were also generally fairly IT-literate though - either as a result of working in IT or as a result of IT being part of their education.)

The considerations/stumbling blocks would be:
- Whose budget will fund these BI sandboxes and technology?
- Whose resources will be used to set up and support the BI sandboxes?
- What connectivity to back-end data sources can be provided?
- How much of the Business User's time will be spent populating these models/spread sheets?

In my "real world" I, more often than not, had Business Users requiring data from fairly complex data sources such as SAP ERP and from "non-existent" data sources i.e. data that had to be manually maintained because it was not stored in any existing source system, as well as Business Users that conceptually knew what they needed but did not know where/how to find the underlying data or how all the "pieces" tied together.

Getting beyond Step 2:
The success of both Step 2 and Step 4 would rely on the Business User making use of environments that can in fact be monitored by IT.

Probably as a result of the frustration of getting anything done via IT, the Business Users that are able to construct their own "real" models often progress to Step 5 without IT even knowing about it. They then approach IT because their model becomes a trusted source of business information, which they create on their own workstations and share via email (or memory sticks if there is a cap on email size). They also approach IT because they then need to have it automated, probably because they are wrestling with one of the "stumbling blocks" of Step 1.

To an extent you summarized what I said above in your last two bullet points.

Indeed, I think your 7 step process is brilliant!

If we can find the "magic" that lets us succeed in Step 1 and Step 2, then we're on our way!

Consider the "Data Scientist" role: it would be great if there were enough people that had all the knowledge and skills required by the Data Scientist role - but more often than not we would need a team of two people with complementary skills and knowledge that work together to satisfy the one role (like a successful marriage?).

Perhaps something similar needs to happen, in general, for successful BI? IT takes care of infrastructure - with the necessary budget and resources. And the "BI" resources which still report into IT move across to Business - and work as a team with the Business Users i.e. they become a Business resource paid for by Business.

Stephen Covey's sixth habit of highly effective people: Synergize! Combine the strengths of people through positive teamwork, so as to achieve goals no one person could have done alone.

That would be brilliant BI!