We all know that the war of fighting the proliferation of spreadsheets (as BI or as any other applications) in enterprises has been fought and lost. Gone are the days when BI and performance management vendors web sites had “let us come in and help you get rid of your spreadsheets” message in big bold letters on front pages. In my personal experience – implementing hundreds of BI platforms and solutions – the more BI apps you deliver, the more spreadsheets you end up with. Rolling out a BI application often just means an easier way for someone to access and export data to a spreadsheet. Even though some of the in memory analytics tools are beginning to chip away at the main reasons why spreadsheets in BI are so ubiquitous (self service BI with no modeling or analysis constraints, and little to no reliance on IT), the spreadsheets for BI are here to stay for a long, long, long time.
With that in mind, let me offer a few best practices for controlling and managing (not getting rid of !) spreadsheets as a BI tool:
Create a spreadsheet governance policy. Make it flexible – if it’s not, people will fight it. Here are a few examples of such policies:
- Spreadsheets can be used for reporting and analysis that support processes that do not go beyond individuals or small work groups vs. cross functional, cross enterprise processes
- Spreadsheets can be used for reporting and analysis that are not part of mission critical processes
Whoever said BI market is commoditizing, consolidating and getting very mature? Nothing can be farther from the truth. On the buy side, Forrester still sees tons of less-than-successful BI environments, applications and implementations as demonstrated by Forrester's recent BI Maturity survey. On the vendor/sell side, Forrester also sees a flurry of activity from the startups, small vendors and large, leading BI vendors constantly leapfrogging each other with every major and minor release.
In terms of the amount of BI activity that Forrester sees from our clients (from inquiries, advisories and consulting) there’s no question that SAP BusinessObjects and IBM Cognos continue to dominate client interest. Over the past couple of years Microsoft has typically taken the third place, SAS fourth place and Oracle the distant fifth. But ever since Siebel and Hyperion acquisitions, the landscape has been changing, and we now often see Oracle jumping into third place, sometimes leapfrogging even Microsoft in the levels of monthly interest from Forrester clients.
I recently asked my Twitter followers if they had good examples of queries, business questions that SQL can't do. It turns out a better question is "what SQL can't do easily", so I thought I'd share with everyone what I heard and found. Seth Grimes was the first one to provide an excellent answer with some informative examples - thank you, Seth! I also found very useful articles on typical SQL challenges such as avoiding multiple duplicate sets in your SQL results, and why NULLs create tons of headaches for SQL coders.
There's also a typical SQL challenge with ragged, sparse, unbalanced hierarchies and dimensions. For example, a retail store, a wholesaler or a distributor with thousands of products, and a manufacturer with thousands of parts often struggle with dissimilar data. A pencil in an office supply store does not have the same descriptive attributes (lead type, for example) as a calculator (scientific, financial, etc.) or an office chair (number of wheels, etc.). Or a tire in a car manufacturing supply chain does not have any common descriptive elements (rubber grade, width-to-height ratio) with gear boxes (automatic vs. manual, 4 or 5 speed, gear-to-gear ratios, etc). When looking for correlation between two entities (for example, what is a potential product quality issue that is making my sales go down?) in cases with disparate, dissimilar products (as in retail products or manufacturing parts), the same SQL query cannot work for all products or parts. One would be forced to write multiple SQL queries for each product or part type to find such a sales/quality relationship.
IBM announced its intentions to acquire Coremetrics, a leading Web analytics vendor, as BI megavendors continue to round out their BI portfolios (the other leading vendor in the space, Omniture, was recently picked up by Adobe). Good move, IBM. Web analytics can't really continue to exist in a silo. In order to get truly complete 360-degree view of customers, prospects and products, one needs to combine Web analytics data with ERP, CRM, HR, Financials and other transactional and analytical data sets. Currently, there are no off-the-shelf solutions that do that - it's pretty much the realm of customized offerings and systems integration. If IBM can indeed plug Web analytics into its data integration, data warehouse and BI products and solutions, it'd be quite a differentiated offering. Other large BI vendors, like Microsoft, Oracle and SAP will probably pick up one of the remaining Web analytics vendors Nedstat, Unica and Webtrends sometime soon.
I know many of you already know my position on this, but I thought I'd get it out in the open and challenge all of you with a controversial discussion. In my definition – and believe it, I am fighting and defending it every day – analytics has always been, and will always be, part of BI. What many of the vendors and analysts describe as "the new age of analytics" I built at Citibank in the early '80s and then built in about 50+ enterprises in the '90s at PwC. I think the effort of trying to differentiate analytics from BI is a vendor-invented hype, since many BI vendors are running out of ways to differentiate themselves (and incorrectly so: see the next paragraph, and many other next-gen BI trends). I also disagree with the “old BI = bad”, “new analytics = good” premise that I see in many analysts' papers. You and I know that you can’t build analytics (OLAP, advanced analytics, etc.) without basic ETL, DW, MDM, etc. So nothing’s really changed as far as I am concerned: we are still fighting the same battles – silos, data quality, etc.
Defining a successful BI strategy is a lot more than gathering requirements and selecting a vendor. While it’s been a subject of many books, I know few of you have time to read them, so here’s a short version.
First defining what BI is and what it is not. Is it just reporting, analytics and dashboards? Or does it involve ETL, DW, portal, MDM, etc., as well?
If the former, you then need to define linkages, dependencies, overlaps and integration with all of the latter (including - very importantly - integration and coordination with the higher level enterprise architecture efforts). If latter, it’s a whole different subject. You then really do need to read a few thick books.
Ensure senior business executive commitment and top down mandate. If you cannot get that, do not proceed until you do. Two ways to “sell BI” to them (even though that’s not a good position to be in):
Educate them on BI ROI. Here's where you'd build a high level BI business case.
I get this request almost on a weekly basis: "Boris, my BI vendor is offering me the following discount, is it a good deal or not?" The first question is what are you comparing it to? It reminds me of an old joke: Q. How much is 5 times 5. A. Depends on whether you're buying or selling. Many of the vendors do not publish or reveal list prices, or even if they do, they are revealed only under NDA to each client, so good luck comparing what the vendor told you and what they told another client. So what ARE you comparing it to?
Another problem, IMHO, is that many of the vendors muddy the waters with CPU based prices, clock speed based prices, etc. Yes, CPU, server, core based prices make sense if you are growing and want to lock in a good deal now, before you grow and expand. But in the end, you, the buyer, still need to figure out how much the software costs you per seat, per user. So with both of these challenges in mind I looked through my 20+ years of notes on BI contracts and per seat license costs and came up with the following. Notice, an interesting X-factor (obviously, I fixed the numbers a bit to have it look nicely like that):
BI output consumer, no interactivity $300
BI output consumer, with light (sort, filter, rank) interactivity $600 (or 2x)
BI output consumer with heavy interactivity (interactive dashboards, search, etc.) $1,200 (or 4x)
My friend and highly respected colleague, Wayne Eckerson from TDWI, posted a great article called “Purple BI People”. In the article he described some of the best practices for business and IT alignment, and cross-functional skills sets needed for successful and effective BI professionals. Wayne, I loved the blue cow analogy, you know that I always think in metaphors, analogies, similies and associations. But, while I completely agree with Wayne in his near term assessment, best practices and recommendations, I would like to suggest another long term point of view.
Can business and IT ever align on BI? Can business ever be satisfied with IT for delivering successful and effective BI applications? Is there such a thing as BT (Business Technology, the phrase that Forrester coined and promotes) in BI?
I used to think we could deliver on that promise. Not so sure it’s that straightforward now. Just look at some of the hopelessly diametrically opposing business and IT priorities. I hear the following complaints from my clients day in and day out:
Business is all about revenue generation. While IT can support that, much more often cost cutting is IT's highest priority.
Business wants solutions now. Not tomorrow. IT needs to go through due diligence of testing and approving BI applications. Right now, on demand does not sit well with IT.
Business wants to react to constantly changing BI requirements. IT has to plan.
Business sometimes is willing to do something “quick and dirty” – even at the expense of potentially jeopardizing accuracy and adherence, compliance with standards. IT is all about compliance and sticking with standards.
I have long resisted and will continue to resist for the foreseeable future any notions that the BI market is commoditizing. A single simple look at the BI maturity in enterprises and next gen BI technologies is a simple proof that we are far, very far, from any kinds of commoditization. Consolidation is quite a different story. Last week's SAP acquisition of Sybase and my roaming the exhibitor / partner floor at SAPPHIRE in Orlando are two more proofs. On a huge SAPPHIRE exhibition floor I could count software partners by the number of the fingers on my hands. Why? Because everyone who matters has been acquired by a competitor! Most of the exhibitors were management consultancies, systems integrators and other SAP implementation partners. Hence, a lesson to independent BI vendors: offer your own full BI stack or position yourself for an acquisition. No other long-term options in my mind.
But as always I welcome all and any comments and opposing views.