The Online Panel Quality Debate Continues

Last week, I was in LA, hosting a session on online panel quality at Forrester’s Marketing Forum. I discussed the past, present, and future of online panel quality with Steve Schwartz from Microsoft, Maria Cristina Gomez from Procter & Gamble, and Frank Findley from ARS Group.

Online panel quality is still a major issue in the industry. The whole discussion started in 2006 with a speech by Kim Dedeker -- at that time, the VP of global consumer and market knowledge at Procter & Gamble. In it, she publicly expressed her concerns about online panel quality, how it affected their research results, and, as a result, the credibility of market research. In her speech, she stressed that, in her opinion, the industry – both research suppliers and clients – needed to focus on how to improve the overall quality of research. Her appeal to the industry was very successful. Many other research buyers weighed in with their stories, and the research providers took up the challenge. Since then, many initiatives have started, such as the ARF’s Foundation of Quality and ESOMAR’s 26 questions, as well as more technology-driven approaches like Peanut Labs’ Optimus and MarketTools’ TrueSample.

However, the truth is that while all these initiatives have helped give insight into the problems that exist and have resulted in some improvements within the participating panels, they don’t solve the problems across panels. One of the panel members shared a story that illustrated how the current initiatives don’t solve the problem. For a project with very low incidence, they received 170 completes in just a couple of days. After the initial excitement, they became somewhat suspicious, but at first glance, the results looked perfectly fine: different IP addresses, no speeding, no straightliners, and logic answers. It wasn’t until it held an in-depth investigation that the vendor discovered that all the IP addresses led to the same location in China -- an “intellectual sweatshop” that had earned quite a few dollars in just one night!

The panel was quite clear that something has to change, and it needs to change fast! The industry can’t take another four years to implement improvements; there’s too much at stake. Companies like Microsoft and Procter & Gamble are continuing to lead the efforts to set guidelines for the entire industry, and they’ve recently formulated a number of requirements that they believe the industry needs to commit to, such as transparency, audits, objectivity, and reporting:

P&G and Microsoft are looking for partners to help them drive this process, because four years into this battle, they realize more than ever that only a combined effort will make a difference. In the meantime, they are already implementing these stricter guidelines: Only panel partners who align with the requirements mentioned above qualify for projects with them.

I’d love to hear your thoughts: Will you back up P&G and Microsoft’s initiatives, or are the current quality guidelines serving your needs well enough?