Business intelligence (BI) practitioners have always thought of the world as data-centric. Data integration, data warehouses, data marts, reports, and query builders were always about data. BI has traditionally excelled at answering questions like "what happened" or even "why did it happen" but always fell short on "what do I do about it" and fell short of the next logical steps which traditionally have been the realm of business process management (BPM) and business rules engines (BRE). This data-centric view of the world turns out to be plain wrong. The world is much more process and rules-centric. We run many processes every time we come to the office, these processes generate data, which in turn trigger rules, and in turn generate more data output that is being consumed by processes in an endless loop.
Why is BI TEI (Total Economic Impact) so elusive? Recently I reached out to all major BI software vendors and asked them to provide a customer reference who's willing to stand up and confirm a hard $ return on investment from BI implementation. Guess how many takers I got? None. Yes many are willing to point to expected savings and benefits, but no one's gone back and calculated the actual results. Why? It is definitely very complex. For example:
Make sure you account for both direct and indirect costs.
Direct costs are the obvious expenses and capital expenditures associated with BI software, hardware and consulting services. A good rule of thumb is to expect to pay $5-$7 dollars for system integration and management consulting for every $1 you pay for software. And don't forget to include the costs of training and on-going support.
Indirect costs are for software/hardware/services for non-BI specific components which are nevertheless necessary to achieve a successful BI implementation: data quality, master data management, metadata implementation, portals, collaboration, knowledge management and many others. The indirect costs are not as easy to quantify. For example, do you attribute the cost of implementing a data quality solution to the BI initiative? Most likely your data quality problems exist in your sources, so one might think it should be a separate effort. However, very often you identify data quality problems when you build your first BI solution, so there may be a tendency to bundle in these costs into the BI project. As a result, these indirect costs are notoriously difficult to identify and negotiate (with other stakeholders), but nevertheless they are a major component of the total cost.
I remember my days as a PricewaterhouseCoopers consultant in the late 90's and early 2000, when the company was awash in HP acquisition rumors and then later discussions of the failed transaction. IBM beat HP and picked up a gem — PwC in these days was hard to beat in many areas, especially in business intelligence management consulting offerings. HP then went on an picked up a much smaller BI boutique Knightsbridge. Now that IBM is acquiring Cognos, will HP follow the same fate and acquire smaller Information Builders, Microstrategy or Actuate? There's also still SAS that would give HP a complete BI stack, but as we know acquiring the world's largest privately held company can be a financial and cultural fit nightmare (plus a rumored $20B or more than 10x revenues price tag is hard for anyone to swallow). That's why I thought that HP's potential acquisition of Cognos + Informatica + Teradata could've given HP best of breed components in all areas of BI stack. But just like with PwC, HP will now have to pursue smaller, more niche BI opportunities.
Back to IBM. Well, not so fast. Back to IBM and SAP. In my opinion, IBM/COGN and SAP/BOBJ deals are defensive moves since both companies have been telling us for years that they prefer to grow their BI portfolios organically, with smaller tuck-in acquisition. However, organic growth is not happening fast enough, and giving in to sideway pressures from Oracle (with two top of the line BI products from Siebel and Hyperion) and upward pressures from Microsoft (after Proclarity acquisition and with significant Performance Point market momentum), IBM and SAP had no choice but to react.
SAP and Business Objects today announced that the companies have reached an agreement for SAP to acquire Business Objects for approximate sum of slightly above 4.8 billion euro. Forrester has been predicting this continued market consolidation for some time — see our Microsoft Buys Proclarity and Oracle Buys Hyperion research documents, as well as a couple of my earlier blogs on the subject. SAP must be feeling a lot of pain and pressure to make such a significant move — SAP executives have been telling the world for years that they prefer small, tuck-in acquisitions. The deal though does make a lot of sense. In one transaction SAP gets the best of breed set of BI tools with full BI stack capabilities, everything from data integration tools like ETL and data quality to reporting, OLAP, dashboards, text analytics and many others. This deal has multiple implications to enterprise software users, especially for those 30-40% from the common SAP/BOBJ customer base:
Business Objects users will gain from SAP’s domain expertise. In the era of increasingly commoditized products and services, domain expertise and industry specific solutions are key differentiating factors for any enterprise software vendor.
Yet another rumor crossed the wires today about Business Objects getting ready for a takeover. These rumors have come up before, and it may be nothing else but a routine exercise that BO and other vendors go through to test the market periodically.
As I already blogged back in March '07, there's no denying however that the business intelligence (BI) and business performance solutions (BPS) markets are consolidating. Last year Microsoft bought ProClarity; this year Oracle bought Hyperion, Business Objects acquired Carthesis, SAP acquired Outlooksoft, and most recently Cognos acquired Applix.
I predict that within 2-3 years there will be five major BI vendors carving up the BI and BPS market: Microsoft, IBM, SAP, Oracle and HP. There'll be much M&A activity seen from these vendors in the near future.
Here's how I see it playing out:
HP will probably make the next big move by acquiring somebody like Business Objects, Cognos or SAS. HP is pushing into BI market very strongly: It repositioned NeoView as a strong data warehouse (DW) platform and bought Knightsbridge — leading BI boutique strategy and implementation consultancy. It'd make a lot of sense for HP to pick up BO or SAS, since in one transaction HP would get the entire BI and PM suite. Cognos would be the second choice for them, since Cognos does not have operational ETL and data quality offerings.
SAP told us repeatedly that they can't justify very large acquisitions — it doesn't fit their model. So Microstrategy, Actuate or Information Builders would be more obvious pick up choices for them.
I sincerely hope that Oracle is more than busy with Hyperion integration and more than set in the BI and BPS markets for a while. I don't see the next big move coming from them.
Today Cognos announced its intention to acquire Applix, Inc for a cash tender offer of $17.87 per share, or approximately $339 million.
This acquisition is primarily about performance management, and I quote my colleague, Paul Hamerman: “This continues a trend of rapid consolidation in the business performance solutions space, following acquisitions of pure plays by larger BI and ERP vendors. Applix has been a stellar performer as a flexible platform for customers to rapidly implement applications for planning, budgeting, forecasting and business performance analytics. Although Cognos has been a leader in performance management solutions, the Applix technology gives it an opportunity to refresh its offerings and more aggressively sell to the midmarket.”
The BI side of this acquisition is all about TM1 OLAP technology. Cognos’s Powerplay OLAP product has been trailing Microsoft SQLServer Analysis Services and Hyperion (recently acquired by Oracle) Essbase in market adoption. With TM1, a memory-based OLAP cube, Cognos achieves several objectives:
Leapfrogs Microsoft and Oracle with one of the fastest read/write OLAP technologies on the market.
Provides an enterprise grade in-memory OLAP alternative to fast-growing and increasingly popular QlikTech, which is more of a departmental solution.
Provides additional midmarket entry points for Cognos
The future of the rest of the Applix BI products — reporting, querying, dashboards, and data integration — is quite murky. Cognos will face the same challenge as Oracle now does with the Hyperion acquisition: Establishing a strategy and a road map of overlapping product portfolio integration or sunsetting/retirement.
I get many questions on dashboards and scorecards and the role these tools play in BI (Business Intelligence). If we use Forrester’s definition of BI — a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information — then we see that dashboards are just the tip of the BI iceberg. One cannot build “just a dashboard”, without considering, architecting and implementing many other necessary BI layers and components such as data integration (ETL, data quality, etc), analytics (OLAP), metrics management, and many supporting components such as collaboration, knowledge management, metadata and master data management, and others. So that’s the first key takeaway: do not be fooled by 2nd tier dashboard vendor claims that one can implement an enterprise wide dashboard easily and inexpensively.
Let’s start with definitions, since I see the terms dashboards and scorecards used interchangeably:
Dashboards are just one style of interactive user interface, designed to deliver historical, current, and predictive information typically represented by key performance indicators (KPIs) using visual cues to focus user attention on important conditions, trends and exceptions.
Scorecards are a type of a dashboard which link KPIs to goals, objectives and strategies. Many scorecards follow a certain methodology, such as Balanced Scorecard, Six Sigma, Capability Maturity Models, etc.
Other types of dashboards include Business Activity Monitoring (BAM) dashboards and visualizations of data / text mining operations.
There’s tug of war going on in the world of BI. On the one hand we have IT whose mission it is to manage and protect enterprise information assets, and on the other side there are end users who just want the data when they want it, and in the shape and form that they want it, without any limitations.
Traditional, mainstream BI vendors have catered primarily to IT target audience. These vendors will disagree, but take one look at their complex architectures, multiple layers and components, integration and support requirements, and you can’t help but agree that these are IT tools that can be used to create end user applications.
On the other hand I am seeing am emergence of smaller BI vendors that cater directly to the end users. They pitch simplicity, flexibility and little or no reliance on IT. True, these vendors do not have large enterprise functions like metadata, semantic layers, robust security and scalability, so I do not see them as enterprise-level, but rather departmental, focused solutions. Yet, the appeal to end users is undeniable.
Finding a compromise – satisfying all typical IT requirements, while empowering the end users - remains an elusive goal, and hence an opportunity for all BI vendors.
I’d like to hear what my colleagues out there think about the convergence of structured and unstructured data business intelligence. Here are the intersects as I see them. I see two types of BI paradigms emerging in the future:
Structured OLAP will continue to be just that – structured, as far as the process and UI are concerned. However, to become more effective, we will need to bring unstructured data into the analysis, in a way that is transparent to the end user. For example, as we are creating customer segmentation analysis for a marketing campaign, in addition to structured data such as customer demographics and prior buying behavior, we’d want to bring in comments hidden in customer email and voice mail requests. In an ideal environment, the OLAP engine will automatically match these emails to a customer dimension and quantify and qualify comments into star schema facts (number of requests) and dimensions (request types).
Combination of search and light-weight query used for ad-hoc research and analysis. Here, a familiar search text box should be the main UI, however, the engine should be smart enough to a) quantify and qualify unstructured results into facts and dimensions – a so called guided search, or b) recognize that the request is actually about data stored in a structured repository and automatically return search results via OLAP, cross tab or tabular report format.
For years I've been predicting that relational DBMS will run out of steam when it comes to effectively managing and manipulating very large, heterogeneous (structured + unstructured) data sets for business intelligence. First, RDBMS were never designed and optimized for unstructured data (not just XML, which is structured data in my definition, but truly unstructured text pages). Second, there's just too much overhead and cost in RDBMS for handling OLTP functions. The result: search index DBMS will be king in BI and DSS in the future.
Today’s announcements that Microsoft may be buying Yahoo came several weeks early. On May 17th I would’ve gone on the record at Forrester IT Forum in Nashville by saying the following, and I quote from my presentation paper: “DBMS/BI vendor may buy a search company, to address the trend of increasing importance of unstructured data in BI and to obtain an early leading position in the space. I know it should be Oracle or IBM, but it probably won’t, since these guys will never admit that their relational DBMS cannot do something. Microsoft is a more likely contender since they know they won’t leapfrog IBM or Oracle in relational DBMS and they could use this opportunity to stick one to Google too.”
I thought Microsoft would buy somebody like Fast Search, but I guess that was too small for them.