By Boris Evelson

Many years ago as I started researching and analyzing the differences between major BI vendors, one criterion that I always used was whether these vendors ate their own dog food. In other words, did a vendor executive team use the same solutions for data collection, building metrics and dashboards to run their own companies that they also tried to sell to their clients? Those who did tended to score higher in my evaluations.

The same guiding principle is applicable to Forrester: you have to eat your own dog food in order to convince the clients to buy your products and services. Hence, our methodologies, such as Forrester Waves are completely open and transparent (thank you, Doug Henschen, for recognizing this in your recent blog), and we encourage our clients to challenge us on every point made in our Waves.

I do get lots of requests to explain the way I rate this or that vendor in my latest BI Wave. While I will typically try to support and explain the way I scored each vendor, I also encourage our clients to just download our Wave spreadsheet evaluation matrix and put in their own scores. No matter how objective and transparent a Forrester analyst tries to be, there’s always a small element of a personal subjective opinion built into every evaluation. But a) that opinion is fully presented and described in detail in the Wave spreadsheet matrix, and b) clients that disagree with the rating are free to just change scores and weights in the model and come up with their own vendor ratings. In fact, in addition to just using our Waves for their primary purpose, there are several other ways Wave models can be utilized:

• Customizing a Wave. Put in your own weights and scores and come up with a customized rating that is specific to your enterprise requirements, culture, priorities and standards.
• Creating a Wave of Waves. If one is looking beyond just one product capability from a certain vendor, our clients typically combine results from our, say, BI, ETL, CRM, database, etc Waves to come up with the overall evaluation of the vendor product portfolio. After normalizing the scores from multiple Waves (a relatively painless process) one can fairly effortlessly come up with vendor ratings across multiple product lines.
• Comparing apples to apples. Certain market sub segments have their own individual challenges, and that’s why some analysts choose to do multiple Waves for a single market segment. For example, I felt that the differences in corporate and product strategy between closed source and open source BI vendors were significant enough to warrant evaluating them in two different Waves.  However, while some of the criteria around open source projects vs. corporate strategy were obviously quite different, we kept almost all of the current product offering criteria exactly the same across both Waves. This means that with just a bit of a criteria mapping exercise (a trivial task) one can compare product capabilities of closed source and open source BI vendors.
• Evaluating vendors not covered in our Waves. The benefit of having an open and transparent methodology also carries a cost: Waves take a lot of effort (at least 2 resources for 4-6 months) to execute. That’s the main reason we often include fewer vendors in our Waves than we’d like to. However, there’s nothing stopping our clients from using the Wave model to do their own evaluations. For example, my upcoming report “BI Belt Tightening In Tough Economic Times” will have a review of over 10 BI SaaS and over 20 small BI vendors. While we are not planning to do a more formal Wave based evaluation of most of these vendors in the near future, I encourage our clients to use the latest BI Wave model to build their own evaluations. With all of the evaluation criteria pre-built and battle tested, it should save anyone a major part of the evaluation effort.

Research reports are great to kick start an IT initiative, but our clients also need so called “last mile” or “2nd phase” decision aids and accelerators. Recognizing this need we will continue to come up with more of such evaluation and selection decision tools for our clients. One example is our recently published Business Intelligence Data Architecture Decision Tool, which, unlike our Waves that are used to guide you through a vendor selection process, can be used to make vendor neutral analytical data architecture decision. In addition to BI, we plan to leverage this new decision tool for other enterprise architecture segments.