In-memory analytics are all abuzz for multiple reasons. Speed of querying, reporting and analysis is just one. Flexibility, agility, rapid prototyping is another. While there are many more reasons, not all in-memory approaches are created equal. Let’s look at the 5 options buyers have today:
1. In-memory OLAP. Classic MOLAP cube loaded entirely in memory
Vendors: IBM Cognos TM1, Actuate BIRT
Fast reporting, querying and analysts since the entire model and data are all in memory.
Ability to write back.
Accessible by 3rd party MDX tools (IBM Cognos TM1 specifically)
Requires traditional multidimensional data modeling.
Limited to single physical memory space (theoretical limit of 3Tb, but we haven’t seen production implementations of more than 300Gb – this applies to the other in-memory solutions as well)
2. In-memory ROLAP. ROLAP metadata loaded entirely in memory.
Speeds up reporting, querying and analysis since metadata is all in memory.
Not limited by physical memory
Only metadata, not entire data model is in memory, although MicroStrategy can build complete cubes from the subset of data held entirely in memory
Requires traditional multidimensional data modeling.
3. In memory inverted index. Index (with data) loaded into memory
Vendors: SAP BusinessObjects (BI Accelerator), Endeca
Fast reporting, querying and analysts since the entire index is in memory
Less modelling required than an OLAP based solution
Q2: Do you provide all components necessary for an end to end BI environment (data integration, data cleansing, data warehousing, performance management, portals, etc in addition to reports, queries, OLAP and dashboards)?
If a vendor does not you'll have to integrate these components from multiple vendors.
Guesstimates are often essential for market sizing and trending. To be useful, especially where primary data are lacking, they demand a valid conceptual framework.
Like you, I’m looking forward to the responses to Boris Evelson’s quick Web-based survey, which you can access from his most recent blogpost.It’s always a challenge to assess how truly pervasive BI is—and pervasive it could potentially become.
To generate a valid first approximation, Boris scoped his blog comments and quick survey to “traditional BI” applications (i.e., historical reporting, query, dashboarding). He scoped his estimate only to large enterprise and midmarket firms (i.e., those with 100 or more employees) and only to BI usage in the US.
In order to keep this task manageable, Boris excluded some use cases that are often included in the “traditional BI” category: spreadsheets and other “homegrown” analytics apps; BI embedded in line-of-business apps; and non-interactive, static, published BI outputs. He leveraged both public and Forrester-gathered primary datato gauge how many actual and potential BI users there might be.
Scoping it as he did, Boris estimated that slightly more than 1.5 million people in the US are using traditional BI applications, which is between 2-3 percent of the employees of BI-implementing firms. He suspects the actual percentage might be as high as 6-8 percent of employees, but he’s not sure. That’s why he’s running the Web-based quick survey.
A number of clients ask me "how many people do you think use BI". Not an easy question to answer, will not be an exact science, and will have many caveats. But here we go:
First, let's assume that we are only talking about what we all consider "traditional BI" apps. Let's exclude home grown apps built using spreadsheets and desktop databases. Let's also exclude operational reporting apps that are embedded in ERP, CRM and other applications.
Then, let's cut out everyone who only gets the results of a BI report/analysis in a static form, such as a hardcopy or a non interactive PDF file. So if you're not creating, modifying, viewing via a portal, sorting, filtering, ranking, drilling, etc, you probably do not require a BI product license and I am not counting you.
I'll just attempt to do this for the US for now. If the approach works, we'll try it for other major regions and countries.
Number of businesses with over 100 employees (a reasonable cut off for a business size that would consider using what we define as traditional BI) in the US in 2004 was 107,119
US Dept of Labor provides ranges as in "firms with 500-749 employees". For each range I take a middle number. For the last range "firms with over 10,000" I use an average of 15,000 employees.
This gives us 66 million (66,595,553) workers employed by US firms who could potentially use BI
Next we take the data from our latest BDS numbers on BI which tell us that 54% of the firms are using BI which gives us 35 million (35,961,598) workers employed by US firms that use BI
What’s the perfect database management system (DBMS) architecture, where analytics is concerned? It seems as if everybody in business intelligence (BI), data warehousing (DW), and related areas has their own opinion on this topic. In fact, it’s more than just opinions. In the database wars, we have huge communities of vendors and users with substantial investments in one or more approaches—from traditional relational DBMSs to column-oriented, in-memory, dimensional, inverted indexing, and other approaches.
In the analytic database wars, new architectures are springing up everywhere, each with its own devotees and differentiators, and each with a go-to-market message that takes potshots at established approaches. The “No SQL” movement is just the latest coalition to emerge in this titanic struggle, and is essentially a loose coalition of diverse approaches rallying around a common theme: that traditional RDBMSs and their SQL-based access languages are unfit, or at least, maladapted to the new world of cloud, social networking, and Web-oriented analytics applications.
DW industry analysts such as myself are of course embedded in these wars—we’re the neutral observers often caught in the crossfire. Of course, this is not a new battle. The columnar database industry has been around for many years and positioned itself as the chief high-performance, low-footprint analytics contender to traditional row-based RDBMSs. And columnar has steadily encroached on relational’s turf, not only through established columnar-based DW vendors such as Sybase, Vertica, and ParAccel, but also through recent adoption (albeit in limited fashion) by the likes of Oracle, SAP, and others into their analytics architectures.
Self-service analytics is one of my core coverage focus areas. It applies not just to business intelligence (BI) but also to advanced analytics.
When, a few months ago, I uttered the immortal phrase “roll over rocket scientists,” I was referring more specifically to the need for pervasive self-service tools for predictive analytics and data mining (PA/DM). Considering that my recently published Forrester Wave on PA/DM Solutions primarily addressed the traditional requirements of “rocket scientist” experts in statistical analysis, I did not put a huge emphasis on data mining features geared to business analysts, subject matter experts, and other “non-technical” information workers.
As I’ve stated in that blogpost and the follow-on podcast, the core problem with today’s PA/DM offerings is that many of them are power tools, not solutions that have been designed for the mass business market. Vendors such as SAS Institute, IBM/SPSS, KXEN, Oracle, Portrait Software, Angoss, FICO, and TIBCO Spotfire provide data mining specialists with feature-rich algorithm-powered solutions for modeling, scoring, regression, and other core PA/DM functions. Their core, traditional user base consists of statisticians, mathematicians, and other highly educated analytics professionals.
Social networks have their foundations in the space-time continuum—you know, the funky coordinate system that Einstein was so keen about.
Social network analysis is all about looking for patterns of “proximity” among people, considered in their cultural capacities as influencers and followers, innovators and imitators, first-movers and late adopters. Down deep, I consider social network analysis an important new branch of decision support systems as a discipline. The core question is: What unique situational chemistry causes various people, individually or collectively, to make various decisions at various places and times?
That’s where space and time enter the social network analysis equation. It’s not enough that I look up to your shining example and take my lead from what you say and do. It’s just as important that we be in the same city, neighborhood, or room. More than that, it’s important that you and I actually cross paths in order for you to actively influence me to buy that latte, or for you to calm me down and thereby stop me from storming out the door and severing my relationship with a retailer who has ignored my complaints one time too many.
What the business world needs now is a bigger, badder, more powerful social media dashboard for customer relationship management (CRM). It almost goes without saying that TweetDeck just won’t cut it.
Ideally, the social media dashboard would provide a CRM-integrated interface for monitoring what customers are saying about you in Twitter, Facebook, and other communities. It would also allow you to aggregate high-level customer satisfaction metrics; to flag smouldering issues surrounding defective products and poor customer service; to respond inline through these channels; and to escalate issues internally to the appropriate parties. In other words, it would be, per my colleagues Bill Band and Natalie Petouhoff, a true “customer business intelligence (BI)” dashboard.
As you develop your company’s social CRM strategy, you must provide social media dashboards to all roles that participate in the customer lifecycle. Whether you’re a brand manager who simply wants to listen into social networks to track awareness, sentiment, and propensities, or a sales person who is interested in identifying and qualifying leads, or a customer service rep who wants to interact closely with established customers, a social media dashboard will soon become a core productivity tool.
It would be an understatement to say that data management is a hot topic today. Master data management, data quality management, metadata management, data integration and data governance have all emerged as high priorities for many global IT organizations. Often times, these data management efforts are paired with investments in business intelligence and facilitated by data warehousing strategies.
Once the strategy, business case, and supporting architectures and organizations are defined (no easy task in and of itself), the next inevitable question is then, which vendors should IT leaders partner with to enable these strategies? There are pure play and best of breed MDM, data quality, BI and DW vendors that offer unbiased, agnostic approaches, eliminating any vendor lock-in or reliance on database platform or enterprise applications. On the other hand, a single platform vendor can offer better ease of integration with existing IT infrastructure than the best of breed alternatives.
These considerations lead us to a major platform vendor, like SAP. Similar to its mega-platform competitors, IBM and Oracle, SAP offers a deep and wide set of data management, BI and data warehousing solutions that promise not only integration within these products, but more notably - across its broader product portfolio of enterprise applications.
It was my pleasure to participate in the latest DM Radio podcast panel yesterday. Eric Kavanaugh and Jim Ericson always do a fine job of organizing these events, and, with their stellar industry panels and fun “morning drive-time crew” on-air patter, they keep it lively. And these guys actually know a thing or two about information management.
The latest DM Radio panel was right in my core coverage area. They called it their “Third Annual Appliance Showdown.” That got to me to thinking: early 2008 (when they held their first) was also when Forrester began our coverage of data warehousing (DW) appliances, starting with publication of my report “Appliance Power: Crunching Data Warehousing Workloads Faster And Cheaper Than Ever.” When I published that report, DW appliances were still not quite in the enterprise mainstream, because they were still regarded by enterprise IT as, in the words of Kavanaugh, an “adjunct” to the enterprise DW (EDW) for fast table scans and query processing, rather than as platform that could scale to support all EDW functions.