It’s an understatement to say companies are drowning in digital information. Since the death of the floppy disk and the rise of networked computing, barriers to creating and sharing information have steadily come down. Combined with increased digitization paper-laden business processes, most companies find themselves struggling to harness the volume and diversity of information on their networks for business benefit. What’s startling is just how little progress we've made in maximizing the value and minimizing risks associated with the digital content and data we collect.
Any discussion of information governance always brings me back to this depressing little anecdote:
"Monday September 8, 2008, is a day that the executives at United Airlines will remember. The company’s stock fell 76 percent to $3 by 11:08 a.m. when trading was halted. The decline was not the result of a pending acquisition or recent financial results announced by the company. Instead, an article that said “UAL Declares Bankruptcy” appeared on the South Florida Sun Sentinel Web site that Sunday, got picked up by Google News, and then quickly summarized and republished to Bloomberg by a reporter tasked with summarizing stories about companies in distress. Then the trading began and the stock collapsed. The problem: the article was from 2002, not 2008."
In-memory analytics are all abuzz for multiple reasons. Speed of querying, reporting and analysis is just one. Flexibility, agility, rapid prototyping is another. While there are many more reasons, not all in-memory approaches are created equal. Let’s look at the 5 options buyers have today:
1. In-memory OLAP. Classic MOLAP cube loaded entirely in memory
Vendors: IBM Cognos TM1, Actuate BIRT
Fast reporting, querying and analysts since the entire model and data are all in memory.
Ability to write back.
Accessible by 3rd party MDX tools (IBM Cognos TM1 specifically)
Requires traditional multidimensional data modeling.
Limited to single physical memory space (theoretical limit of 3Tb, but we haven’t seen production implementations of more than 300Gb – this applies to the other in-memory solutions as well)
2. In-memory ROLAP. ROLAP metadata loaded entirely in memory.
Speeds up reporting, querying and analysis since metadata is all in memory.
Not limited by physical memory
Only metadata, not entire data model is in memory, although MicroStrategy can build complete cubes from the subset of data held entirely in memory
Requires traditional multidimensional data modeling.
3. In memory inverted index. Index (with data) loaded into memory
Vendors: SAP BusinessObjects (BI Accelerator), Endeca
Fast reporting, querying and analysts since the entire index is in memory
Less modelling required than an OLAP based solution
Q2: Do you provide all components necessary for an end to end BI environment (data integration, data cleansing, data warehousing, performance management, portals, etc in addition to reports, queries, OLAP and dashboards)?
If a vendor does not you'll have to integrate these components from multiple vendors.
On March 25, 2010 TIBCO Software announced that they acquired Netrics, a small, private data matching vendor. TIBCO and Netrics had a pre-existing OEM relationship that was originally announced in June 2009, where TIBCO embedded the Netrics match engine into its Collaborative Information Manager (CIM) master data management (MDM) solution.
Netrics differentiates its advanced matching engine by describing how it “Matches data based on a mathematical model that mimics human perception of similarity, identifying hidden relationships in the data.” The Netrics matching engine includes a self-learning capability that improves the confidence in its matches over time by also evaluating manual matches made by business users. Netrics business and technology approach to this market made it a ripe (and obvious) acquisition target since it developed the match engine to be completely embeddable in existing applications with the vast majority of its revenue coming from OEM and SI channels. In addition to TIBCO, Netrics current MDM OEM partners also include Data Foundations and Kalido. This bodes well for TIBCO’s ability to further integrate these capabilities into its product portfolio.
This is not really a new blog post. It's a relatively recent post that didn't manage to make it over from my independent blog. I wanted to be sure it made it to my Forrester blog because I will have lots of publications and posts on information architecture coming up and this was a post on my first piece in this series. So here's the original post:
In January, the lead-off piece that introduces my research thread on information architecture hit our web site. It’s called Topic Overview: Information Architecture. Information architecture (IA) is a huge topic and a hugely important one, but IA is really the worst-performing domain of enterprise architecture. Sure, even fewer EA teams have a mature — or even active — business architecture practice, but somehow I’m inclined to give that domain a break. Many, if not most, organizations have just started with business architecture, and I have a feeling business architecture efforts will hit practical paydirt fairly quickly. I’m expecting to soon hear more and more stories of architects relating business strategy, goals, capabilities, and processes to application and technology strategies, tightly focusing their planning and implementation on areas of critical business value, and ultimately finding their EA programs being recognized for having new relevance, all as a result of smart initial forays into business architecture in some form.
A number of clients ask me "how many people do you think use BI". Not an easy question to answer, will not be an exact science, and will have many caveats. But here we go:
First, let's assume that we are only talking about what we all consider "traditional BI" apps. Let's exclude home grown apps built using spreadsheets and desktop databases. Let's also exclude operational reporting apps that are embedded in ERP, CRM and other applications.
Then, let's cut out everyone who only gets the results of a BI report/analysis in a static form, such as a hardcopy or a non interactive PDF file. So if you're not creating, modifying, viewing via a portal, sorting, filtering, ranking, drilling, etc, you probably do not require a BI product license and I am not counting you.
I'll just attempt to do this for the US for now. If the approach works, we'll try it for other major regions and countries.
Number of businesses with over 100 employees (a reasonable cut off for a business size that would consider using what we define as traditional BI) in the US in 2004 was 107,119
US Dept of Labor provides ranges as in "firms with 500-749 employees". For each range I take a middle number. For the last range "firms with over 10,000" I use an average of 15,000 employees.
This gives us 66 million (66,595,553) workers employed by US firms who could potentially use BI
Next we take the data from our latest BDS numbers on BI which tell us that 54% of the firms are using BI which gives us 35 million (35,961,598) workers employed by US firms that use BI
It would be an understatement to say that data management is a hot topic today. Master data management, data quality management, metadata management, data integration and data governance have all emerged as high priorities for many global IT organizations. Often times, these data management efforts are paired with investments in business intelligence and facilitated by data warehousing strategies.
Once the strategy, business case, and supporting architectures and organizations are defined (no easy task in and of itself), the next inevitable question is then, which vendors should IT leaders partner with to enable these strategies? There are pure play and best of breed MDM, data quality, BI and DW vendors that offer unbiased, agnostic approaches, eliminating any vendor lock-in or reliance on database platform or enterprise applications. On the other hand, a single platform vendor can offer better ease of integration with existing IT infrastructure than the best of breed alternatives.
These considerations lead us to a major platform vendor, like SAP. Similar to its mega-platform competitors, IBM and Oracle, SAP offers a deep and wide set of data management, BI and data warehousing solutions that promise not only integration within these products, but more notably - across its broader product portfolio of enterprise applications.
In the past couple of months I've been working on a document called 'Information Management For Market Researchers', released earlier this month to our dedicated Forrester Market Research Leadership Board Members. Although I can't share all lessons learned with you yet, there are a couple of insights I'd like to bring to your attention.
The most important outcome from my interviews with market researchers and knowledge managers is that a culture of sharing creates better products and helps companies be more successful innovators. Simply said: to innovate, knowledge from various departments needs to come together, irrespective of role or rank.
The world is changing. The traditional lines of demarcation between IT and business, developers and end users, producers and consumers of info no longer work. But every time I attempted to create a matrix of BI personas in the new world, I ended up with so many dimensions (business vs. IT, consumers vs producers, strategic vs tactical vs operational decisions, departmental vs. line of business vs enterprise cross functional roles, running canned reports vs. ad-hoc queries, and many others, i ended up with something quite unreadable. But there still has to be something that on the one hand shows the realities of the new BI world, yet something that fits onto a single PPT. Here's my first attempt at it (click on the small image to see the full one).
In this diagram I attempt to show
Who's consuming vs. producing the information, how heavy or light that task is. What's interesting is that all our research shows is that most of the BI personas now are both consumers and producers of info.
Who's using what style of BI as in reports, queries, dashboards and OLAP
Who is using BI only as reports and dashboards embedded in enterprise apps (such as ERP, CRM, others), which usually means canned reports and prebuilt dashboards, vs BI as a standalone app
Who's using non traditional BI apps, such as the ones allow you to explore (vs just report and analyze) and allow you to perform that analysis without limitations of an underlying data model
Who's a producer and a consumer of advanced analytics
And finally show the level of reliance on IT by every group
As always, all comments, suggestions and criticism are very welcome! HD
Over the last 25 years in the business I heard my share of BI horror stories: “we have over 20 different BI tools”, or “we have a few thousand reports in our BI application”. BI is very much a self fulfilling prophecy – “build it, and they will come”. As we popularize BI, and as technology becomes more scalable, more stable, more function rich and user-friendly - BI spreads like wildfire and often becomes uncontrollable.
I can’t help but to quote from one of my favorite books by a British author Jerome K. Jerome “Three Men In A Boat, To Say Nothing Of A Dog”. One of the reasons I love the book, in addition to it being one of the funniest novels I ever read, is that I can almost always find a very relevant humorous quote to just about any life or business situation. At the beginning of the book three British gentlemen are planning a vacation on a river boat. As they plan for how much food and supplies they should carry, they quickly realize that there isn’t a boat big enough to fit the dimensions of the Thames river to carry all that junk.
“How they pile the poor little craft mast-high with fine clothes and big houses; with useless servants, and a host of swell friends that do not care twopence for them, and that they do not care three ha'pence for; with expensive entertainments that nobody enjoys, with formalities and fashions, with pretence and ostentation, and with - oh, heaviest, maddest lumber of all! - the dread of what will my neighbour think, with luxuries that only cloy, with pleasures that bore, with empty show that, like the criminal's iron crown of yore, makes to bleed and swoon the aching head that wears it!