Number of People Using Advanced Analytics

Guesstimates are often essential for market sizing and trending. To be useful, especially where primary data are lacking, they demand a valid conceptual framework. 

Like you, I’m looking forward to the responses to Boris Evelson’s quick Web-based survey, which you can access from his most recent blogpost.It’s always a challenge to assess how truly pervasive BI is—and pervasive it could potentially become.

To generate a valid first approximation, Boris scoped his blog comments and quick survey to “traditional BI” applications (i.e., historical reporting, query, dashboarding). He scoped his estimate only to large enterprise and midmarket firms (i.e., those with 100 or more employees) and only to BI usage in the US.

In order to keep this task manageable, Boris excluded some use cases that are often included in the “traditional BI” category: spreadsheets and other “homegrown” analytics apps; BI embedded in line-of-business apps; and non-interactive, static, published BI outputs. He leveraged both public and Forrester-gathered primary data to gauge how many actual and potential BI users there might be.

Scoping it as he did, Boris estimated that slightly more than 1.5 million people in the US are using traditional BI applications, which is between 2-3 percent of the employees of BI-implementing firms. He suspects the actual percentage might be as high as 6-8 percent of employees, but he’s not sure. That’s why he’s running the Web-based quick survey.

On a related note, I’m curious how many actual and potential users there are for advanced analytics applications. Of course, that’s entirely outside the scope of Boris’ focus, but very much part of the “BI writ large.” If traditional BI is far from pervasive in the business world, then advanced analytics—which I scope to predictive modeling, data mining, text analytics, in-database analytics, and complex event processing (CEP)—is a mere drop in the bucket.

Interestingly, the Q4 2009 Forrester survey gives us useful fodder for such an estimate. It shows that, where reporting tools are concerned, 74 percent of enterprises either already use them or are planning to implement them in 2010. For predictive analytics and data mining (PA/DM), the equivalent number is 31 percent; for text analytics and CEP it’s 19 percent; and for in-database analytics, 7 percent. Considering that PA/DM is the most widely adopted advanced analytics solution category, I’d estimate that perhaps 25 percent of the companies that responded use or plan soon to implement one or more of these types of solutions.

That would mean approximately one in three companies implementing traditional BI also uses advanced analytics. In other words, if, say 3 percent of employees in BI-implementing firms use traditional BI, that would correspond to 1 percent of those firms’ employees using advanced analytics (for the time being, let’s not concern ourselves with the overlap, within each firm, among users of traditional BI, on the one hand, and advanced analytics, on the other).

As a back-of-envelope mental calculation, it appears that Boris’ numbers correspond to roughly 300 traditional-BI users per company for those that deploy the technology. From statistics gathered for my recently published Forrester Wave on PA/DM Solutions, I know that there are an average 10-20 users (i.e., statistical modeling specialists, primarily) per PA/DM-implementing enterprise. Let’s take the midpoint on the latter range: 15 PA/DM users per firm. Hence, we’re looking at roughly 5 percent of BI-using employees in “BI writ large” user companies that use PA/DM. If we adjust this to include the smaller numbers of text mining, CEP, and in-database analytics users, we might guesstimate 6-7 percent of employees in “BI writ large” user organizations are also currently doing advanced analytics.

But, and here's a thought that sprung to mind upon eyeballing Boris' guesstimates, it would appear that the potential number of advanced analytics users per company could be three times higher. Remember, from up above, the 3:1 ratio of traditional BI users to advanced analytics users, derived from last fall's Forrester survey findings? In other words, multiplying 3 by the 6-7 percent advanced-anaytics penetration ratio (as a percentage of traditional-BI users) that I derived from my Wave, we get a potential advanced-analytics penetration of 18-21 percent (as a percentage of traditional-BI users). Another way of looking at this is that the potential average number of advanced analytics users per BI-using organization could grow from 15 to 45. That, of course, assumes that the users we surveyed deliver on their plans to implement advanced analytics in the coming year.

Thanks for hanging in there with me while I did the math. How does that look to you? It's just a first rough estimate. I haven’t done a web survey on any of this. I’m not yet sure that I would be able to define and frame the questionnaire in such a way as to elicit valid responses.

As I said, I’m waiting for the results of Boris’ more narrowly scoped survey before deciding how to proceed on this.

Comments

Jim, thanks for another

Jim, thanks for another insightful and thought-provoking post.

While I agree that the market for users of *stand alone* predictive analytics and data mining products may be limited to more highly trained users, I think we should also talk how many people *should be* benefitting from leveraging them. As you correctly pointed out, current tools in the market require statistical training in order to use them properly and that represents a small sub-segment of the BI population. To that end, calling such tools “advanced analytics” is not unreasonable.

However, consider how many potential users exist in an organization if such powerful capabilities are deployed behind easy to understand and consume dashboards and applications! A dashboard that not only depicts historical summary of trends (like traditional BI products do) but also projects that trend into the future using powerful statistics and computational models. For this use-case, I actually believe the term “advanced” is misleading. These more general business users may not even know they are using any advanced features, but rather may click a button that says “project this trend” or “compare the similarities between these two customer groups.” Deploying these powerful underlying “advanced analytics” capabilities like these to mainstream users will enable new audiences to answer questions they could not in the past.

With this new consideration in mind, I’d argue that the potential audience is actually *more significant* than your estimates for traditional BI populations. Think about users that do not currently turn to traditional BI products for problem solving, yet who benefit immensely from analytical or predictive rigor embedded into their applications. Geologists investigating oil and gas deposits, clinicians running clinical trials, equities traders digging into market trends. Few of these users would consider turning to a traditional BI product, yet clearly are running applications that can and do benefit from embedding more analytical capabilities.

Making predictive analytics more mainstream is our mission and the real driver behind our recent TIBCO Spotfire Analytics 3.1 release. By front-ending powerful analytics with highly visual and interactive dashboards and applications, we believe we can reach vast populations that would never have leveraged such tools to-date. Maybe it’s time to find a better term than “advance analytics”...

Keep up the great coverage!

Mark Lorion
http://spotfire.tibco.com/

Thanks...but "should" not a predictive call

Mark:

Thanks for the positive and detailed feedback. However, I was simply estimating current and potential usage of advanced analytics--not making an imperative "should use" call. As regards the scoping of "advanced analytics," it's essentially a catchall for any analytics app/approach that goes beyond traditional analytics (i.e., what we all call "BI") to focus on the future (i.e., predictive modeling, what-if analysis, etc.) and/or present (complex event processing, etc.). To the extent the industry puts those into the hands of business analysts and every other information worker, the scope of traditional analytics tool (i.e., what we all call "BI") will continue to expand. The definition of "advanced analytics" wil continue to change over time.

Jim