Posted by Brad Bortner on October 9, 2009
Now that solutions are finally available to "clean up" panels, will buyers at last insist that providers use them, or will a myopic focus on price continue to be the rule when on-line sample is used?
The use of online panels for market research was highly contentious at one time. Traditional researchers thought that they would be prone to the same problems as traditional off-line panels: filled with respondents that are not representative of the general population motivated by the desire to earn money taking surveys.
Despite these misgivings, online panels have taken off in the US (and are coming on strong in Europe.) Why? Because they allowed research to be conducted in one third the time at one fifth the cost. This allowed buyers to say to themselves, "ok, maybe its not quite as project-able, but give the savings it's worth it." Also, many panel vendors claimed to be doing something special to ensure that their panels were better, and indeed several of them did.
Since there was no way to actually effectively evaluate the quality of one panel vs. another, though, panel quality often took a back seat to cost for many buyers. If it's hard to measure if one panel is more high quality than another, and even harder to judge if insights that are, say 20% less accurate, are directly responsible for bad business decisions, it was very easy for research buyers to ignore the whole quality issue. A whole generation of researchers has now grown up in the US who have always thought that online research is the default, and would only turn to other, more expensive, modes if for some reason it absolutely could not deliver.
This lack of transparency about panel quality is finally ending. Solutions such as MarketTools' True Sample (that utilizes a combination of machine fingerprinting and list matching to make sure that respondents truly exist and only answer a survey once), as well as a wide variety of machine fingerprinting solutions such as Optimus and RelevantID can not only show how "dirty" a panel is but offer solutions to clean out the dirt. The result? Third party measurements of the quality of panels, and an ability for buyers to insist that cleaning occur. Many panel vendors are also publicly redoubling their panel hygiene efforts.
The open question is whether one of these approaches will become industry dominant, or whether each major panel vendor will insist on owning the solution for their own panel. One industry dominant solution might be best for highest quality, since the more sample goes through a single cleaning process, the "better" the cleaning process becomes (especially when several panels are used for one study), but panel vendors are understandably loath to play with hygiene companies they consider to be competitors.
The final answer to this will depend on what research buyers demand, if anything, and how compelling the information they get is about the superiority of one hygiene approach vs. another.
One thing is clear, however, the next 12 months will be very telling about the future on online panel quality.
Search Forrester's Blogs
Forrester's CX Index
Predict how actions to improve CX will affect revenue performance.
Measure the customer experiences that matter most »
Free On-Demand and Live Events
Latest events from Forrester analysts, online and in person »