Some time back, Vision Critical's Springboard America team began dabbling in using non-traditional sources of sample such as rivers and re-routed panel samples. We wondered what the quality of these sources was, and how they compared to our Springboard America panel.
We wanted to know about the quality of these sources for a number of reasons:
1) Young adults are becoming increasingly more difficult to find and river sources and re-routed panel samples can exponentially increase the feasibility of studies with this challenging audience - who are becoming an endangered species on traditional online access panels.
2) Given the high quality of VC Market Panels, we needed to assure our clients that using these sources when blended with our own panels, still provides them a quality sample frame for their studies when using non-traditional online sources and other panels.
3) Lastly, we take great pride in how we recruit, manage and engage with our panels, so we took the opportunity to test our theory that our own Springboard America panel was the highest quality source in the industry.
To test this out, we conducted a five minute survey on April 12, 2012 and November 4-7, 2011 with a sample of about 500 people per sample source, per wave. We employed sample from Vision Critical's Springboard America panel, and four other leading sample providers.
We have anonymised the other sample providers as "Panel B", "River C", "Panel D" and "River E". The questionnaire included items from the US 2010 General Social Survey, as well as questions about snack consumption and a concept test for a snack food.
In our test, we focused on a few key areas such as straight-lining, differences between the sources in how respondents answered all questions and purchase intent.
Our conclusions were that sample matters, a lot. Springboard America has fewer drop-outs, notably better data quality, and is "right down the middle" when it comes to comparison to the other panels on the findings, unlike some panels which show real variation - particularly on the concept test.
Further, what was surprising to us at VC Springboard America was that our river sources were comparable on the most part with other access panels in the industry, across two separate tests of our quality measures, in two different time periods. This makes us feel comfortable that employing non-traditional online sources for blending outside Springboard America is just a comparable quality as employing one of the many industry mega-panels.
Bottom line; don't treat your sample as just a means to collecting data. Quality matters, and knowing how to manage both panels and use of other sources is key to a successful project.
You can view the two presentations that summarize the research results below:
Part 1 - Does Sample Source Matter?
Part 2 - Does sample Source Matter?