In-memory analytics are all abuzz for multiple reasons. Speed of querying, reporting and analysis is just one. Flexibility, agility, rapid prototyping is another. While there are many more reasons, not all in-memory approaches are created equal. Let’s look at the 5 options buyers have today:
1. In-memory OLAP. Classic MOLAP cube loaded entirely in memory
Vendors: IBM Cognos TM1, Actuate BIRT
Fast reporting, querying and analysts since the entire model and data are all in memory.
Ability to write back.
Accessible by 3rd party MDX tools (IBM Cognos TM1 specifically)
Requires traditional multidimensional data modeling.
Limited to single physical memory space (theoretical limit of 3Tb, but we haven’t seen production implementations of more than 300Gb – this applies to the other in-memory solutions as well)
2. In-memory ROLAP. ROLAP metadata loaded entirely in memory.
Speeds up reporting, querying and analysis since metadata is all in memory.
Not limited by physical memory
Only metadata, not entire data model is in memory, although MicroStrategy can build complete cubes from the subset of data held entirely in memory
Requires traditional multidimensional data modeling.
3. In memory inverted index. Index (with data) loaded into memory
Vendors: SAP BusinessObjects (BI Accelerator), Endeca
Fast reporting, querying and analysts since the entire index is in memory
Less modelling required than an OLAP based solution
Q2: Do you provide all components necessary for an end to end BI environment (data integration, data cleansing, data warehousing, performance management, portals, etc in addition to reports, queries, OLAP and dashboards)?
If a vendor does not you'll have to integrate these components from multiple vendors.
A number of clients ask me "how many people do you think use BI". Not an easy question to answer, will not be an exact science, and will have many caveats. But here we go:
First, let's assume that we are only talking about what we all consider "traditional BI" apps. Let's exclude home grown apps built using spreadsheets and desktop databases. Let's also exclude operational reporting apps that are embedded in ERP, CRM and other applications.
Then, let's cut out everyone who only gets the results of a BI report/analysis in a static form, such as a hardcopy or a non interactive PDF file. So if you're not creating, modifying, viewing via a portal, sorting, filtering, ranking, drilling, etc, you probably do not require a BI product license and I am not counting you.
I'll just attempt to do this for the US for now. If the approach works, we'll try it for other major regions and countries.
Number of businesses with over 100 employees (a reasonable cut off for a business size that would consider using what we define as traditional BI) in the US in 2004 was 107,119
US Dept of Labor provides ranges as in "firms with 500-749 employees". For each range I take a middle number. For the last range "firms with over 10,000" I use an average of 15,000 employees.
This gives us 66 million (66,595,553) workers employed by US firms who could potentially use BI
Next we take the data from our latest BDS numbers on BI which tell us that 54% of the firms are using BI which gives us 35 million (35,961,598) workers employed by US firms that use BI
The world is changing. The traditional lines of demarcation between IT and business, developers and end users, producers and consumers of info no longer work. But every time I attempted to create a matrix of BI personas in the new world, I ended up with so many dimensions (business vs. IT, consumers vs producers, strategic vs tactical vs operational decisions, departmental vs. line of business vs enterprise cross functional roles, running canned reports vs. ad-hoc queries, and many others, i ended up with something quite unreadable. But there still has to be something that on the one hand shows the realities of the new BI world, yet something that fits onto a single PPT. Here's my first attempt at it (click on the small image to see the full one).
In this diagram I attempt to show
Who's consuming vs. producing the information, how heavy or light that task is. What's interesting is that all our research shows is that most of the BI personas now are both consumers and producers of info.
Who's using what style of BI as in reports, queries, dashboards and OLAP
Who is using BI only as reports and dashboards embedded in enterprise apps (such as ERP, CRM, others), which usually means canned reports and prebuilt dashboards, vs BI as a standalone app
Who's using non traditional BI apps, such as the ones allow you to explore (vs just report and analyze) and allow you to perform that analysis without limitations of an underlying data model
Who's a producer and a consumer of advanced analytics
And finally show the level of reliance on IT by every group
As always, all comments, suggestions and criticism are very welcome! HD
Over the last 25 years in the business I heard my share of BI horror stories: “we have over 20 different BI tools”, or “we have a few thousand reports in our BI application”. BI is very much a self fulfilling prophecy – “build it, and they will come”. As we popularize BI, and as technology becomes more scalable, more stable, more function rich and user-friendly - BI spreads like wildfire and often becomes uncontrollable.
I can’t help but to quote from one of my favorite books by a British author Jerome K. Jerome “Three Men In A Boat, To Say Nothing Of A Dog”. One of the reasons I love the book, in addition to it being one of the funniest novels I ever read, is that I can almost always find a very relevant humorous quote to just about any life or business situation. At the beginning of the book three British gentlemen are planning a vacation on a river boat. As they plan for how much food and supplies they should carry, they quickly realize that there isn’t a boat big enough to fit the dimensions of the Thames river to carry all that junk.
“How they pile the poor little craft mast-high with fine clothes and big houses; with useless servants, and a host of swell friends that do not care twopence for them, and that they do not care three ha'pence for; with expensive entertainments that nobody enjoys, with formalities and fashions, with pretence and ostentation, and with - oh, heaviest, maddest lumber of all! - the dread of what will my neighbour think, with luxuries that only cloy, with pleasures that bore, with empty show that, like the criminal's iron crown of yore, makes to bleed and swoon the aching head that wears it!
We all struggle with complexity of designing, building and maintaining BI apps. Why? Among many other reasons, the simplest one is that there's just too many components involved. Just to name a few
Data modeling (star schemas, cubes)
Delivery (portals, schedulers, emails, etc)
For years there were many attempts to automate some of these steps via metadata. So rather than than coding source to target SQL transformations or DDL for DW generation vendors came up with, what I know call "1st generation" metadata driven BI tools, such as
ETL tools where metadata auto-generated SQL scripts for data extraction, loading and transformation
BI tools where metadata auto-generated SQL for queries
Data modeling tools where metadata auto-generated logical data models and DDL for physical data models
But, the "2nd generation" metadata driven BI apps (note apps vs tools now) do much more. For example, they:
Use metadata to generate multi vendor apps (like BalancedInsight, Kalido and BIReady do), and having a single place where changes can be made
Use metadata to generate all three (ETL SQL, BI SQL, DW DDL, like Cognos, Wherescape, BIReady do), and having a single place where changes to all 3 can be made
Using metadata to generate report layouts (like Cognos does)
After the recent board changes the strategy will change too
After the recent board changes at SAP the message we could read in most news was like ‘new board – old strategy’. Along with the board changes SAP did not announce (yet) any significant strategic changes. But what good is it to change the board and leave everything else as is?
The recent SAP board changes are just the visible tip of the iceberg of much deeper changes SAP will and has to go through to renew itself as a leading IT vendor. Below are 10 predictions for changes in SAP’s strategic direction I expect within the next 10+ months:
1. More SAP Board Changes Will Come
Additional board changes will further strengthen the product & technology focus and competence within the SAP board. See also Forrester’s blog on the recent SAP board changes: SAP CEO Resigns – Long Live The Co-CEOs
2. Business ByDesign Will Get Back Into SAP’s Strategic Center
Business ByDesign will become again the corner stone of SAP’s growth strategy and the successful introduction will mark a ‘make it or break it’ milestone for SAP.
3. SAP Announces The Next-Generation ERP
SAP will announce a next-generation ERP solution to regain leadership in its core business area and it will likely be based on the ByDesign platform.
4. SAP Changes Its Cloud Strategy
SAP will rework its whole On-Demand strategy and will unify and align all components based on the ByDesign platform. See also Forrester’s recent blog on SAP’s On-Demand strategy: SAP Is Skydiving Into The Clouds.
When a user of a BI application complains about the application not being useful - something that I hear way too often - what does that really mean? I can count at least 11 possible meanings, and potential reasons:
1. The data is not there, because
It's not in any operational sources, in which case the organization needs to implement a new app, a new process or get that data from an outside source
It is in an operational source, but not accessible via the BI application.
The data is there, but
2. It's not usable as is, because
There are no common definitions, common metadata
The data is of poor quality
The data model is wrong, or out of date
3. I can't find it, because I
Can't find the right report
Can't find the right metadata
Can't find the data
I don't have access rights to the data I am looking for
4. I don't know how to use my application, because I
Was not trained
Was trained, but the application is not intuitive, user friendly enough
5. I can't/don'thave time do it myself - because I just need to run my business, not do BI !!! - and
My colleague, Holger Kisker, just posted a very insightful blog on the convergence of BI and BPM technologies. Yes, Holger, BPM vendors definitely have some BI capabilities. And so do some search vendors like Attivio, Endeca and Microsoft FAST Search. And so do some middleware vendors like TIBCO, Vitria and Software AG. And so do rules vendors like FairIsaac, PegaSystems. Should I go on? I have a list of hundreds of vendors that "say" they are a BI vendor.
But it’s not that simple. First of all, let’s define BI. In the last BI Wave we defined BI as “a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insights and decision-making”. To provide all these capabilities a vendor should have most of the necessary components such as data integration, data quality, master data management, metadata management, data warehousing, OLAP, reporting, querying, dashboarding, portal, and many, many others. In this broader sense only full BI stack vendors such as IBM, Oracle, SAP, Microsoft, SAS, TIBCO and Information Builders qualify.
Even if we define BI more narrowly as the reporting and analytics layer of the broader BI stack, we still want to include capabilities such as 11 ones we use to rate BI vendors in the BI Waves:
When business processes finally become intelligent
Over the past several months I have done a lot of research on the BI market, the trends and the vendor landscape. There is a clear indication that BI solutions are becoming more sophisticated, more intelligent and – more integrated into other applications to enhance the performance of the application supported business processes.
Very recently now, in discussions with BPM vendors like IDS Scheer, HandySoft and many others it became very eminent that from the other side, BPM solutions are moving steadily into the field of Business Intelligence too. The world of BPM and BI solutions are converging to bring intelligent business processes to the market – eventually. However, today we are still some steps away from this picture and the convergence of BPM and BI will likely proceed in smaller steps are outlined in the below BI-BPM convergence model.
Today several BPM vendors have actively integrated business intelligence capabilities into their solutions. Larger ones like IDS Scheer have developed their own analytics while smaller vendors like HandySoft are using OpenSorce components offered by JasperSoft and other OpenSource BI vendors. The integration offers users new and consistent insights along the whole business process. A user in this context means both:
a) Business users that are part of the business process get access to relevant information and reports that increase the efficiency of the process, and
b) Business process owners get an insightful analytics of the process metadata to be able to further enhance and streamline the process.