Transform Business Processes Through Business Analytics

co-authored by Clay Richardson, James Kobielus, Craig Le Clair

Boris, Clay, Jim, and Craig get together in this podcast to talk about their upcoming Forrester IT Forum presentation.

http://www.forrester.com/role_based/images/author/imported/forresterDotCom/Podcasts/BPA/Evelson_Kobielus_Richardson_Le%20Clair_Transform%20Business%20Processes%20Through%20Business%20Analytics_050510.mp3

Categories:

Want to know what Forrester's lead data analysts are thinking about BI and the data domain?

By Boris Evelson

What is BI? There are two prevailing definitions out there – broad and narrow. The broad definition (using our own) is that BI is a set of methodologies, processes, architectures, and technologies that transform raw data into meaningful and useful information used to enable more effective strategic, tactical, and operational insight and decision-making. But if we stick to this definition then shouldn’t we include data integration, data quality, master data management, data warehousing and portals in BI? I know lots of folks would disagree and fit these into data management or information management segments, but not BI.

Then, the narrow definition is used when referring to just the top layers of the BI architectural stack such as reporting, analytics and dashboards. But even there, as Jim Kobielus and I discovered as we were preparing to launch our BI TechRadar 2010 research, we could count over 20 (!) product categories such as Advanced Analytics, Analytical Performance Management, Scorecards, BI appliances and BI SaaS, BI specific DBMS, BI Workspaces, Dashboards, Geospatial analytics, Low Latency BI, Metadata Generated BI Apps, Non modeled exploration and In-memory analytics, OLAP, Open Source BI and SaaS BI, Packaged BI Apps, Process / Content Analytics, Production reports and ad-hoc query builders, Search UI for BI, Social Network / Media Analytics, Text analytics, Web Analytics.

 

To make matters worse, some folks out there are now trying to clearly separate BI and analytics, by trying to push a “core, traditional BI is commoditized, analytics is where differentiation is today” message. Hmmm, I thought I was building analytical apps using OLAP starting back in the early 80’s.

 

Read more

Not all BI self service capabilities are created equal

By Boris Evelson

There’s a lot of hype out there by many vendors who claim that they have tools and technologies to enable BI end user self service. Do they? When you analyze whether your BI vendor can support end user self service, consider the following types of “self service” and related BI tool requirements:

#1. Self service for average, casual users.

  • What do these users need to do?
    • Run and lightly customize canned reports and dashboards
    • Run ad hoc queries
    • Add calculated measures
    • Collaborate
    • Fulfill their BI requirements with little or no training (typically one needs search-like, not point-and-click UI for this)
  • What capabilities do they need for this?
    • Report and dashboard templates
    • Customizable prompts, sorts, filters, and ranks
    • Report, query, dashboard building wizards
    • Portal
    • Semantic layer (not all BI vendor have a rich semantic layer)
    • Prompting for columns (not all BI vendors let you do that)
    • Drill anywhere  (only BI vendors with ROLAP and multisourcing / data federation provide this capability)

#2. Self service for advanced, power users

  • What do these users need to do?
    • Perform what-if scenarios (this often requires write back, which very few BI vendors allow)
    • Add metrics, measures, and hierarchies not supported by the underlying data model (typically one needs some kind of in-memory analytics capability for this)
    • Explore based on new (not previously defined) entity relationships (typically one needs some kind of in-memory analytics capability for this)
    • Not knowing exactly what one is looking for (typically one needs search-like UI for this)
Read more

BI on BI

By Boris Evelson

How do you know if your BI application has high, low or no ROI? How do you know that what the business users requested last month and you spent countless of hours and sleepless nights working on is actually being used? How do you know if your BI applications are efficient and effective? I don't have all the answers, but here's what I recommend.

Start with collecting basic data about your BI environment. The data model (hint, it's a classical multidimensional model exercise) should have the following components:

  •  Requests (these should be available from your help desk and project/portfolio management applications), such as
    • User provisioning
    • New applications
    • New data sources
    • Data model changes
    • New/changed metrics
    • New/changed reports
    • New report delivery options
  • Usage (these should be available from your DBMS and BI apps log files or from www.appfluent.com or www.teleran.com) by
    • Person
    • Time of day
    • Database
    • BI application
    • Report
    • Index
    • Aggregate
    • KMI/KPM
  • Track additional events like
    • Application usage vs. using application/report just to download or export data
    • Incomplete/cancelled queries
Read more

I forget: what's in-memory?

By Boris Evelson
 

In-memory analytics are all abuzz for multiple reasons. Speed of querying, reporting and analysis is just one. Flexibility, agility, rapid prototyping is another. While there are many more reasons, not all in-memory approaches are created equal. Let’s look at the 5 options buyers have today:
 

1. In-memory OLAP. Classic MOLAP cube loaded entirely in memory

Vendors: IBM Cognos TM1, Actuate BIRT
Pros

  • Fast reporting, querying and analysts since the entire model and data are all in memory.
  • Ability to write back.
  • Accessible by 3rd party MDX tools (IBM Cognos TM1 specifically)

Cons

  • Requires traditional multidimensional data modeling.
  • Limited to single physical memory space (theoretical limit of 3Tb, but we haven’t seen production implementations of more than 300Gb – this applies to the other in-memory solutions as well)

 

2. In-memory ROLAP. ROLAP metadata loaded entirely in memory.

Vendors: MicroStrategy
Pros

  • Speeds up reporting, querying and analysis since metadata is all in memory.
  • Not limited by physical memory

Cons

  • Only metadata, not entire data model is in memory, although MicroStrategy can build complete cubes from the subset of data held entirely in memory
  • Requires traditional multidimensional data modeling.

 

3. In memory inverted index. Index (with data) loaded into memory

Vendors: SAP BusinessObjects (BI Accelerator), Endeca

Pros

  • Fast reporting, querying and analysts since the entire index is in memory
  • Less modelling required than an OLAP based solution
Read more

Three Top Questions To Ask a BI Vendor

By Boris Evelson

 

An editor from a leading IT magazine asked me this question just now, so I thought I'd also blog about it. Here it goes:

 

Q1: What are the capabilities of your services organization to help clients not just with implementing your BI tool, but with their overall BI strategy.

 

The reason I ask this as a top question, is that most BI vendors these days have modern, scalable, function rich, robust BI tools. So a real challenge today is not with the tools, but with governance, integration, support, organizational structures, processes, etc – something that only experienced consultants can help with.
 
Q2:  Do you provide all components necessary for an end to end BI environment (data integration, data cleansing, data warehousing, performance management, portals, etc in addition to reports, queries, OLAP and dashboards)?
 
If a vendor does not you'll have to integrate these components from multiple vendors.
 
Read more

Number of people using BI

By Boris Evelson

 

A number of clients ask me "how many people do you think use BI". Not an easy question to answer, will not be an exact science, and will have many caveats. But here we go:

 

  1. First, let's assume that we are only talking about what we all consider "traditional BI" apps. Let's exclude home grown apps built using spreadsheets and desktop databases. Let's also exclude operational reporting apps that are embedded in ERP, CRM and other applications.
  2. Then, let's cut out everyone who only gets the results of a BI report/analysis in a static form, such as a hardcopy or a non interactive PDF file. So if you're not creating, modifying, viewing via a portal, sorting, filtering, ranking, drilling, etc, you probably do not require a BI product license and I am not counting you.
  3. I'll just attempt to do this for the US for now. If the approach works, we'll try it for other major regions and countries.
  4. Number of businesses with over 100 employees (a reasonable cut off for a business size that would consider using what we define as traditional BI) in the US in 2004 was 107,119
  5. US Dept of Labor provides ranges as in "firms with 500-749 employees". For each range I take a middle number. For the last range "firms with over 10,000" I use an average of 15,000 employees.
  6. This gives us 66 million (66,595,553) workers employed by US firms who could potentially use BI
  7. Next we take the data from our latest BDS numbers on BI which tell us that 54% of the firms are using BI which gives us 35 million (35,961,598) workers employed by US firms that use BI
Read more

Who are the BI Personas?

The world is changing. The traditional lines of demarcation between IT and business, developers and end users, producers and consumers of info no longer work. But every time I attempted to create a matrix of BI personas in the new world, I ended up with so many dimensions (business vs. IT, consumers vs producers, strategic vs tactical vs operational decisions, departmental vs. line of business vs enterprise cross functional roles, running canned reports vs. ad-hoc queries, and many others, i ended up with something quite unreadable. But there still has to be something that on the one hand shows the realities of the new BI world, yet something that fits onto a single PPT. Here's my first attempt at it (click on the small image to see the full one).

 


In this diagram I attempt to show

  • Who's consuming vs. producing the information, how heavy or light that task is. What's interesting is that all our research shows is that most of the BI personas now are both consumers and producers of info.
  • Who's using what style of BI as in reports, queries, dashboards and OLAP
  • Who is using BI only as reports and dashboards embedded in enterprise apps (such as ERP, CRM, others), which usually means canned reports and prebuilt dashboards, vs BI as a standalone app
  • Who's using non traditional BI apps, such as the ones allow you to explore (vs just report and analyze) and allow you to perform that analysis without limitations of an underlying data model
  • Who's a producer and a consumer of advanced analytics
  • And finally show the level of reliance on IT by every group

As always, all comments, suggestions and criticism are very welcome! HD

333 Rule To Keep Your BI Apps In Check

Over the last 25 years in the business I heard my share of BI horror stories: “we have over 20 different BI tools”, or “we have a few thousand reports in our BI application”. BI is very much a self fulfilling prophecy – “build it, and they will come”. As we popularize BI, and as technology becomes more scalable, more stable, more function rich and user-friendly  - BI spreads like wildfire and often becomes uncontrollable.

I can’t help but to quote from one of my favorite books by a British author Jerome K. Jerome “Three Men In A Boat, To Say Nothing Of A Dog”. One of the reasons I love the book, in addition to it being one of the funniest novels I ever read, is that I can almost always find a very relevant humorous quote to just about any life or business situation. At the beginning of the book three British gentlemen are planning a vacation on a river boat. As they plan for how much food and supplies they should carry, they quickly realize that there isn’t a boat big enough to fit the dimensions of the Thames river to carry all that junk.

“How they pile the poor little craft mast-high with fine clothes and big houses; with useless servants, and a host of swell friends that do not care twopence for them, and that they do not care three ha'pence for; with expensive entertainments that nobody enjoys, with formalities and fashions, with pretence and ostentation, and with - oh, heaviest, maddest lumber of all! - the dread of what will my neighbour think, with luxuries that only cloy, with pleasures that bore, with empty show that, like the criminal's iron crown of yore, makes to bleed and swoon the aching head that wears it!

Read more

Next gen of metadata driven BI apps

We all struggle with complexity of designing, building and maintaining BI apps. Why? Among many other reasons, the simplest one is that there's just too many components involved. Just to name a few

  • Data sourcing
  • Data extraction
  • Data integration
  • Data cleansing
  • Data aggregation
  • Data modeling (star schemas, cubes)
  • Metrics management
  • Queries
  • Reports
  • Dashboards
  • Alerts
  • Delivery (portals, schedulers, emails, etc)

For years there were many attempts to automate some of these steps via metadata. So rather than than coding source to target SQL transformations or DDL for DW generation vendors came up with, what I know call "1st generation" metadata driven BI tools, such as

  • ETL tools where metadata auto-generated SQL scripts for data extraction, loading and transformation
  • BI tools where metadata auto-generated SQL for queries
  • Data modeling tools where metadata auto-generated logical data models and DDL for physical data models

But, the "2nd generation" metadata driven BI apps (note apps vs tools now) do much more. For example, they:

  • Use metadata to generate multi vendor apps (like BalancedInsight, Kalido and BIReady do), and having a single place where changes can be made
  • Use metadata to generate all three (ETL SQL, BI SQL, DW DDL, like Cognos, Wherescape, BIReady do), and having a single place where changes to all 3 can be made
  • Using metadata to generate report layouts (like Cognos does)
Read more