View
3
Download
0
Category
Preview:
Citation preview
Do Response Rates Matter inOnline Panels?
Representativity at Different Levels ofCumulative Response Rates
Johan MartinssonUniversity of Gothenburgwww.lore.gu.se
About response rates
• RR used to be the main indicator of data quality
• Perhaps today more questionable (e.g. Groves &Peytcheva 2008)
• Moving to online surveys and panels, even more so
• Are participation rates for online panels meaningful?
www.mod.gu.sewww.mod.gu.se
Yeager et al. 2011, pp. 731
www.mod.gu.sewww.mod.gu.se
0
5
10
15
20
25
30
35
40
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
No
ofsu
rvey
s
Participation rate (%)
Participation rates in 163 surveys from theCitizen Panel at the University of Gothenburg
The most important predictor is the demographiccomposition of the invited sample
www.mod.gu.se
Moving to more controlled comparison: theprobability based part of the Citizen Panel
• The emerging indicator is now Cumulative ResponseRate for probability based panels
• CRR = Recr.Rate * Profile rate * Participation rate
• Both recruitment rates and CRRs are usually low
• And it is not clear how important they are
www.mod.gu.se
Briefly about the Citizen Panel and LORE
• Started in 2010 with support from the University ofGothenburg
• The Citizen Panel has approx. 9,000 members fromprobability based recruitments from population samples
• And approximately 57,000 members from opt-inrecruitments
• You can read more at www.lore.gu.se
www.mod.gu.se
Response rates in web panels
• In 2012 LORE conducted a large probability recruitmentwith an experimental design
• This results in variation in recruitment rates
www.mod.gu.sewww.mod.gu.se
Recruitment rates in 9 different treatment groups
www.mod.gu.se
Questions
• What happens to response rates and CRR over time ina panel?
• How are they related to accuracy/representativity?
www.mod.gu.se
How to examine representativeness/accuracy
• We calculate average absolute deviations from a goodbenchmark
• 6 demographic indicators– Gender, age, country of birth, marital status,
education, labour market situation– These benchmarks come from Statistics Sweden– Four of these also register data in the sample
www.mod.gu.se
How to examine representativeness/accuracy
• 2 political indicators (quasi benchmarks)
– Interest in politics (from the SOM Institute survey,mail, n=6000)
– Vote intention (Statistics Sweden, RDD, n=9000)
www.mod.gu.se
www.mod.gu.se
1. What happens to CRR over time in a panel?
www.mod.gu.se
www.mod.gu.se
• Initial span 8.4 pct
• After two yrs 2.7 pct
• Higher RR => higherattrition
• Incentives recruitment=> higher attrition
www.mod.gu.se
2. How are RR and CRR related toaccuracy/representativity?
• First, we examine the accuracy at the recruitment
• Next, accuracy over time in the panel
www.mod.gu.se
2. How are RR and CRR related toaccuracy/representativity?
• At recruitment we have 11 treatment groups withdifferent recruitment rates between 6 and 21 percent
• We examine the average the AAD for the 8 indicators
www.mod.gu.se
All 8 indicators
r=-.61
www.mod.gu.se
r=-.30
All 8 indicators
www.mod.gu.se
Vote intention (8 parties)
r=-.44
www.mod.gu.se
Vote intention (8 parties)
r=-.14
www.mod.gu.se
2. How are RR and CRR related toaccuracy/representativity?
• We observe a weak correlation at the recruitment stage
• But what about later on? For that purpose we have asmaller set of groups that allow strict comparisons
• We examine representativeness 2 years later (8 panelwaves later)
www.mod.gu.sewww.mod.gu.se
Standard postcards: all 8 indicators
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 9.8Wave 1 4.7 7.3Wave 8 2.7 4.8
www.mod.gu.sewww.mod.gu.se
Standard postcards: all 8 indicators
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 7.2 9.8 6.9Wave 1 4.7 7.5 7.3 6.6Wave 8 2.7 8.4 4.8 8.2
www.mod.gu.sewww.mod.gu.se
Standard postcards: 2 political indicators
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 6.6 9.8 6.0Wave 1 4.7 7.6 7.3 6.4Wave 8 2.7 10.1 4.8 8.5
www.mod.gu.sewww.mod.gu.se
Incentive postcards: all 8 indicators
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 14.2Wave 1 6.4 9.5Wave 8 3.6 5.2
www.mod.gu.sewww.mod.gu.se
Incentive postcards: all 8 indicators
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 6.5 14.2 6.9Wave 1 6.4 7.7 9.5 7.4Wave 8 3.6 9.2 5.2 8.6
www.mod.gu.sewww.mod.gu.se
Incentive postcards: 2 political indicators
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 4.3 14.2 5.9Wave 1 6.4 4.8 9.5 6.4Wave 8 3.6 7.0 5.2 9.7
www.mod.gu.sewww.mod.gu.se
Did you notice something peculiar?
Standard cardno reminder
Standard card1 reminder
CRR AAD CRR AADWave 0 5.8 7.2 9.8 6.9Wave 1 4.7 7.5 7.3 6.6Wave 8 2.7 8.4 4.8 8.2
www.mod.gu.sewww.mod.gu.se
Incentive cardno reminder
Incentive card1 reminder
CRR AAD CRR AADWave 0 9.3 6.5 14.2 6.9Wave 1 6.4 7.7 9.5 7.4Wave 8 3.6 9.2 5.2 8.6
Did you notice something peculiar?
www.mod.gu.se
Summing up
• High initial recruitment rates deteriorate substantiallyover time and differences diminish
• Higher cumulative response rates consistently yieldlower average errors ...– ... when comparing over time within a recruitment
cohort– ... and when comparing between cohorts at same
wave / panel age• But panel attrition seems to play a part independent of
cumulative response rates (because not at random)• Overall, correlations btw CRR and accuracy seem low• Caveats: small samples for accuracy, unweighted data,
limited variation in recruitment rates and CRRs
www.mod.gu.se
Read more at
www.lore.gu.se
Laboratory of Opinion Research(LORE)
Recommended