Upload
vcuniversity
View
932
Download
1
Embed Size (px)
DESCRIPTION
A comparison of some leading sample sources including river sample and router sample.
Citation preview
Does the Sample Source Matter? Wave 2 of a comparison of leading sample sources
Sept 07, 2012
Sample is the soil and water of research. The quality of it determines the value of what you produce.
We compared other sample sources that VC
regularly uses
We conducted a five minute survey April 12, 2012 with a sample of about 500 people each, using sample from Vision Critical’s Springboard America, and four of the leading sample providers that we have anonymised as “Panel B”, “River C”, “Panel D” and “River E”. The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI)
Methodology cont.
The questionnaire included items from the US 2010 General Social Survey, as well as a questions about snack consumption and a concept test for a snack food. We counted the number of statistically significant differences between the panels (95% CI), and looked at drop-out rates, straight lining, and the use of “don’t know”.
Sample Quality: Dropout rate during the survey Drop-outs as a percent of all who started the survey
4%
0
8% 10%
7%
16%
0
5% 5% 4%
5%
9%
19%
2%
0%2%4%6%8%
10%12%14%16%18%20%
SBA -Targeted
SBA -Routed
Lightspeed SSI ResearchNow
River UnitedSample
Nov-11 Apr-12
Panel E
Panel D
Panel C
Panel B
Data quality: straight-lining
0.5%
6.5%
7.9%
3.9%
7.8%
0.7%
5.5% 5.6%
4.8%
2.7% 2.1%
3.1%
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
SBA SBA Routed Lightspeed SSI ResearchNow
SB River uSamp
NA
Panel B
Panel C
Panel D
Panel E
Data quality: use of “don’t know”
Average percent who choose the option "don’t know", "can't choose", "not sure", "prefer not to say".
4%
0
6% 6%
4%
6%
0
4% 4%
5% 4%
4% 4%
3%
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
SBA -Targeted
SBA -Routed
Lightspeed SSI ResearchNow
SB River uSamp
Nov-11 Apr-12
Panel B
Panel C
Panel D
Panel E
Differences between SBA and other panels in
how people answered all questions
Count of the number of statistically significant differences (95% CI) as a percent of the total number of questions
4%
2%
3%
7%
4%
2%
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
SBA Routed Lightspeed SSI ResearchNow
uSamp RiverPanel B
Panel C
Panel D
Panel E
Expected by chance
Differences between SBA and other panels in
how people answered all questions
While, for the most part, these differences appear within the realm of random, there were some differences that seem to be very real. For example, in both April and November Panel B respondents were less likely to purchase snack products and had lower purchase intent scores for a snack concept.
Differences between panels in how people
answered the purchase intent question
Which statement below best describes how likely you would be purchase this snack if it were available today?—top 2 box: definitely and probably
34%
0
20%
33%
41%
29%
0
31%
39%
30%
44% 46%
40% 44%
0%5%
10%15%20%25%30%35%40%45%50%
SBA -Targeted
SBA -Routed
Lightspeed SSI ResearchNow
River UnitedSample
Nov-11 Apr-12
Panel B
Panel C
Panel D
Panel E
Conclusions
Sample matters. A lot. Springboard America has fewer drop-outs, notably better data quality, and is “right down the middle” when it comes to comparison to the other panels on the findings, unlike some panels which show real variation—particularly on the concept test. SBA also shows consistency in how people answered, as illustrated by reaction to the concept.
Conclusions
The same cannot be said for the other panels, however. They showed considerable variation between time 1 and time 2 in how they reacted to the snack concept, which was a non-trendy concept. This may be a concern for tracking. I will dig deeper into the comparison between waves in another report.