Do You Have BI On BI?

BI is used to build, report, and analyze business performance metrics and indicators. What about measuring the performance of BI itself? How do you know if you have a high-performing, widely used BI environment? Is your opinion based on qualitative “pulse checks” or is it based on quantitative metrics? BI practitioners who preach to their business counterparts to run their business by the numbers need to eat their own dog food: run their BI environment, platforms, and apps by the numbers. For example, do you know:

  • How many reports and queries do end users create by themselves versus how many IT creates? That's a great efficiency metric.
  • How many clicks within a dashboard does it take to find an answer to a question? That’/s another great efficiency metric.
  • How long does each user stay within each report? Do they just run and print the reports, or export the data to Excel, or do they really slice, dice, and analyze the information? That’s a good example of how effective your BI environment is.
  • Do you see any patterns in BI usage? User by user, department by department, or line of business by line of business?
  • How many reports, queries, and other objects are being used, how many are shelfware (not being used)? How often are people using the ones that are being used?

I find surprisingly little research out there on this subject, so we plan to publish a report on it. With that in mind, I’d like to solicit your input. If you have some experience, specific use cases, and best practices, please share them with me. For example:

  • I named a few metrics above, but I am sure we can come up with dozens more. Do you have any good examples?
  • Can these metrics be organized by categories, like efficiency metrics or effectiveness metrics?
  • Do you have any benchmarks that show what the targets/goals for these metrics should be?
  • If you are a BI software vendor, do you build such functionality into your product?
  • If you are a BI systems integrator, do you have such a service offering for your clients?
  • Can you share any best practices on how to collect, organize, present, and manage these metrics?
  • Do you agree that BI on BI (performance management of BI) should be a key function of BI competency centers/centers of excellence?
  • I am only aware of two vendors — Appfluent and Teleran — that provide this type of a solution across multiple heterogeneous BI vendors, applications, and databases. Are there others?

I’m sure that I’ve missed some questions I should be asking — so I’d appreciate any comments and any contributions to this post.

Comments

Knowledge worker metrics--to and from

Boris, early in Kyield's voyage nearly a decade ago now in R&D, we incorporated two key ingredients into this mix, one of which you covered in Future of BI report a year ago, and this being the second.

As I think you are aware, I've been a proponent of extending BI functionality to and from entire workforce, or as much as possible-- really since the very earliest capabilities came online in the mid 1990s. Many good reasons for it--from innovation to security, productivity and crisis prevention. Based on really ancient good management principle of all employees being the "eyes and ears" of the organization, with the potential to better align incentives towards corp needs and mission.

As part of our architecture, the rich meta data in documents and communications is combined with data from PC/smart devices, analytics, and our performance ratios with self-managed functionality in the digital workplace-through our patented individual module (within org parameters set in corp 'CKO' engine), resulting in performance metrics not just on knowledge workers in the organizations, but through them metrics on the organization itself that are otherwise not captured with traditional BI methods- financials, for example. Each entity can visualize their own metrics and set goals, with privacy and security tailored to each situation.

I am not claiming that all functionality is the same on all devices--they aren't, and regulatory/culture/policy issues also vary widely around the world, which is only one reason why our system is continuously adaptive through data management rather than code work. Other reasons include innovation/differentiation/competitive advantage/TCO.

When we first produced a demo presentation- maybe 7 years ago now, quite a bit of evolution needed to take place in productivity suites, HPC, smart devices, and DBs, but we were (finally) able to confirm sufficient functionality to meet min requirements in 2011, which is why we moved to pilot phase in 2012. There is still a fair amount of data prep and integration work involved, but much less so than most other enterprise systems, particularly given the potential ROI in corp and human performance, higher level governance, and more informed and timely decisions.

A challenging metric to capture, but far more doable with such a system, is organization-wide (BI) awareness in NRT. To some not all that important--to others it can be the difference between survival or not, and leadership. Hope this helps--MM

BI should make the work of the user effective

Drucker's "Management: Tasks, Responsibilities and Practices" talks about the effectiveness of tools for a user.

BI can be seem as a kind of control tool for the knowledge worker. A control system is effective when it only alerts the worker of the exceptions or important events that requires human intervention. This idea applies in finance (low revenue, high spending), health care (deviation of a disease from the pattern), etc. This is efectiveness. It makes BI agile for the worker.