Online research: Who are we talking about here?

There has been a lot of discussion and chatter around social market research (SMR) lately, fueled in part bythe social sessions at the MRA conference a couple of weeks ago. We’ve had social on our radar here at Forrester for awhile and my colleague Tamara Barber has done a great job looking at social market research and its opportunities and challenges. Some of the issues around social MR are hot topics for online research in general, representivity being one of the key ones. Just who are we gathering data on? Whether we’re talking about new social methods or tried and true online panels, the question is still relevant.

The topic of representivity in online panels surfaced a few years ago as MR professionals began to examine and question the data coming back from online surveys. Vendors began to address client concerns through a variety of approaches like MarketTools' TrueSampleand Peanut Labs’ Optimus initiatives. Some clients seemed appeased by the measures, but the debate has continued to rage within the MR industry -- and rightfully so.

Research Magazine recently included an article interviewing Adrian Sanger, Tim Britton, Jeffrey Henning, and Terry Sweeney on the issue of online panels and the representative nature of the research being drawn from such studies. The interviews are very insightful and touch upon some highly relevant issues for market researchers dealing with online research. As Terry says:

 “ ...there’s a bias from the recruitment methodology all the way down to technology issues, bandwidth issues, things like that. We try to educate clients on the biases that could exist within the sample population.”

 Just who are we reporting data on when we survey online panels? As market researchers, are we clearly communicating the issues about representivity and non-response so that our end clients know the true value of the data? As we move into the world of hybrid methodologies, where online panels are not only used for quantitative research, but also serve as sampling base for online communities, these issues will become even more important. It's our responsibility to know exactly what the data is representative of and so, as an industry, we need to continue these conversations about the quality of online panels. This also means understanding everything possible about the vendors you’re buying sample from. Where do they recruit from? What are the panel demographics? How often are respondents contacted? How much time do the respondents spend online? How are they incented? These initial questions will start to shed light on who your survey takers are and will help you choose a supplier that most closely matches the type of respondents you need to reach.

Moving forward I'm going to focus some of my research on issues like online panels, panel quality, and MR vendors. I look forward to having conversations (and debates!) with many of you on the current and future landscape of these issues. If you're an MR vendor that wants to discuss the specific steps you're taking to address the online panel issue, feel free to connect with me and we can set up a time for a briefing. In the meantime, happy researching!

Comments

Education, Transparency and Consistency are Key

Great post on a very important and relevant topic, Jackie. This issue is particularly top-of-mind for me because this week, MarketTools hosted its fourth TrueSample Quality Council with delegates from the biggest research buyers and suppliers in the industry. Interestingly, a similar sentiment to yours was shared by all. In particular, there were three main themes that emerged:

1. Education -- How do we educate research buyers on what questions to ask of their suppliers? Without knowing all of the possible causes of compromised data quality and how to prevent them, how can a buyer evaluate a supplier's ability to deliver high-quality insights? The Advertising Research Foundation has made great strides in this area with their Quality Enhancement Process, but there still seems to be a lot of work we can do in this area to create standards against which suppliers can be measured apples to apples.

2. Transparency -- How can we make a shift toward complete transparency and openness in our industry? Buyers can ask questions, but if suppliers aren't willing to 'pull back the curtain' and be transparent about their quality assurance processes and, more importantly, the results and impact of these processes, then Buyers aren't able to effectively assess a supplier's quality. Many suppliers have adopted third-party certification systems, such as TrueSample as a way to provide this transparency and meet buyer's quality requirements. Systems like these provide a consistent and auditable quality assurance process that allows buyers to feel confident in the accuracy of their research. But the challenge for suppliers now is determining which third-party certification system to choose. As they strive to meet the demands of their clients, suppliers have to choose which basket to put their eggs in, and hope that they choose well.

3. Consistency -- For buyers, this may be the most critical issue of all. How can they blend sample from multiple suppliers for a project if each supplier is using a different method for ensuring quality? What impact does this variance have on the reliability of the results? As buyers go through the effort of defining quality standards and requirements for their business, they want to ensure that all of their suppliers are meeting those requirements and are using consistent methods for doing so. If supplier A defines an unengaged respondent in one way, but Supplier B defines it another, there is no way to effectively measure or audit the effectiveness of quality assurance processes. This is an issue that must be resolved.

All in all, I think the industry is slowly reaching agreement that we need to have some industry-wide online research quality standards put into place. Suppliers who want to be held accountable, can provide third-party evidence that they are meetings these standards and can provide statistics and reports demonstrating their value and impact. And at the end of the day, the suppliers who deliver will win the business.