- Forrester Councils
- Councils Overview
- log in
Posted by Tom Grant on March 8, 2011
Democracy is great, right? We'd all prefer to have direct participation in the decisions that affect our lives, from which multimillionaire will represent us common folk to which features we'd like to see in the next version of Microsoft Office. (Please, please, PowerPoint team, just copy Keynote's auto-align feature already.) The more voting we do, the more we feel that civilization has advanced, and the better the quality of the products or politicians we get.
Polls Are Valuable But Inadequate
In recent years, application development teams have grown increasingly open minded, and in many cases even enthusiastic, about voting or polls as a prioritization mechanism. Worried that your requirements rely too heavily on interviews with a potentially unrepresentative sample of users? Take a quick poll to get a more accurate estimate of real demand for the work you might do.
As important as polls can be, they're not perfect. Even if you don't go crazy with how you use the results, the quality of these findings depends a great deal on the questions you ask. Even if your survey question kung fu is great, other risks exist, such as the unfortunate tendency of people with no opinion answering the question anyway because they don't want to appear foolish.
One peril that holds special relevance to application development (or product development in general) is the missing part of the sample. By their nature, polls omit the customers you think you should have but don't yet have.
Polls Provide A Megaphone For Customers You Already Know
From first-hand experience on development teams, and second-hand experience as a researcher, I've seen the "missing customer" problem in action. But how bad is this problem? Since I have practically no ability to resist simulation exercises, I whipped together a simple model of the costs of relying exclusively on polls. (Click here to see the spreadsheet I developed if you want to scrutinize the calculations or play around with the numbers yourself.) I tried to keep the scenario relatively simple but still reflective of the real choices that development teams face.
OK so far? While the Category D ("Dearly Like To Win") customers are missing, some of the other customers want the same features, albeit with greater or lesser interest than the Category D people. This kind of overlap is part of requirements and prioritization exercises all the time: for example, while salespeople might be interested in an improved pipeline report in a CRM system, sales managers are even happier to see it developed.
Using the numbers I plugged into the model, here are the votes:
Prioritization now seems easy-peasy: Build Feature #3, then #2, then #1.
If Want To Serve A New Audience, Be Prepared To Invert Priorities
While intrigued by the results, your management is still anxious about reaching the Category D ("Dearly Love To Have") customers. Therefore, to triangulate around the truth, you invest in another mechanism for collecting data from all four customer categories. Let's posit, for reasons that will be clear later, that this second mechanism is a serious game.
This second tool turns out to be amazingly successful, capturing the exact demand for each feature from within each customer segment. (In reality, no tool is perfect, but our point here is to compare the poll results to the actual demand.) We'll use the same method for calculating votes, only in this case, it includes how people in Category D would have voted, given the opportunity. Here are the results:
Now that we understand what the Category D customers want, our prioritization is exactly the reverse of what it would have been, based on voting alone. And no, I did not construct this model to engineer this result.
Prioritization Shouldn't Be Completely By The Numbers
There are other reasons for expanding your inputs into the prioritization process beyond polls. Votes express demand without explaining it. If 1,193 customers vote for a feature, what have you really learned? Do all of them desire it with the same ardor? Do they want to see it designed in the same way?
Context is everything, which is why serious games are an important supplement to other techniques for requirements and prioritization. Read the chat logs in a Buy A Feature session. Talk to participants during a session of Prune The Product Tree. These exercises strike the happy medium between the breadth of measurement that polls provide and the depth of understanding you get from interviewing a few stakeholders.
[P.S. Please, feel free to rip apart the spreadsheet on which I based this analysis.]
Lead BT Transformation
Develop customer-obsessed strategies to drive growth »
Forrester's CX Index
Predict how actions to improve CX will affect revenue performance.
Measure the customer experiences that matter most »