Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
The Effect of Survey Mode on Nonresponse Bias
and Measurement Bias. A Validation Approach
Antje Kirchner
Barbara Felderer
AAPOR, Boston
May17th, 2013
2 / 15 Research Interests Study Details Results Conclusions
Research Gap
Only a few web survey mode comparison studies use validation data …
aggregate level: Sanders et al. 2007 (voting behavior); Stephenson/Crête 2011
(voting behavior); Vannieuwenhuyze et al. 2010 (political attitudes and voting)
individual level: McCabe et al. 2002 (alcohol & drug use); Sax et al. 2003 (university
issues); Kreuter et al. 2008 (academic performance); Dillman et al. 2008 (long
distance service); Sakshaug et al. 2010 (academic performance)
And those typically focus on …
specific subpopulations, i.e. limited generalizability.
mean statistics.
only one error source.
and neglect the analyses of costs per error source.
3 / 15 Research Interests Study Details Results Conclusions
Research Question
1) How do nonresponse bias and measurement bias compare across survey
modes and how do they interact?
Disentangling:
nonresponse bias
measurement bias
total bias
for socio-demographic and substantive variables across modes using
validation data from the sampling frame.
2) How does adjusting for nonresponse (e.g. raking or post-stratification)
affect total bias?
3) What cost savings can be achieved through the use of web surveys
compared to the telephone survey?
4 / 15 Research Interests Study Details Results Conclusions
Study Details
Survey Data CATI Web
IAB Project Team S. Eckman, B. Felderer, A. Kirchner, F. Kreuter, J. Sakshaug
Project Title “Work and Consumption in Germany”
Stratified SRS Target persons who are either
1) Unemployed
2) Receipt of Welfare Benefits
3) At least two employment spells
Gross Sample N = 12,400 N = 11,838
Completes N = 2,400
(RR1=19.35%)
N = 1,082
(RR1= 9.01%)
Fieldwork 08/2011-10/2011 02/2012-04/2012
Advance Letter Paper-based Paper-based (tel. option)
3€ incentive (Amazon voucher)
reminder 2 weeks later
Questionnaire Employment Histories
Add. Experiments Filter Format
Consent
MTMM etc.
Filter Format
Consent
Incentive Experiment
5 / 15 Research Interests Study Details Results Conclusions
Study Details
Administrative data from the registers of the Federal Employment
Agency (FEA) comprising employer reports to the social security
system
e.g. to administer benefit claims and payment
includes information on age, sex, education, status of benefit receipt or
income
Assumption of gold standard, i.e. true parameters (means and
distribution), of those information known to be
relevant to payments (pensions, welfare etc.)
accurate and complete
6 / 15 Research Interests Study Details Results Conclusions
Bias Assessment: Analyses
Administrative Data Survey Data
Population
Sample
Respondents Respondents Item Respondents
Nonresponse Measurement
Bias Bias
Total Bias
respsvyy ,respadmy ,
fsadmy ,
7 / 15 Research Interests Study Details Results Conclusions
Items under Study
Characteristics Statistics Nonresponse
Bias
Measure-
ment Bias
Total
Bias
Socio-
Demography:
Gender Proportion
Age Mean/
Distribution
Nationality Proportion Measurement inequality of
survey and admin. data Education Proportion
Substantive:
Current Employment (Overall: Other, Marginal and Regular
Employment)
Proportion
Current Receipt of
Unemployment Benefits II (UB II)
Proportion No survey data
Past Receipt of Unemployment
Benefits II (UB II)
Proportion
Labor Income from Current Job Mean/
Distribution
8 / 15 Research Interests Study Details Results Conclusions
Socio-Demographic Variables
9 / 15 Research Interests Study Details Results Conclusions
Relative Total Bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
Female
Mean Age (Years)
-60 -40 -20 0 20
Socio-Demography: Relative Total Bias (in %)
CATI Web
10 / 15 Research Interests Study Details Results Conclusions
Substantive Variables
11 / 15 Research Interests Study Details Results Conclusions
Relative Total Bias (except for UB II: only preliminary for web!)
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
Regular empl.
Marginal empl.
Mean Inc. (Euro)
Past UB II
-50 0 50
Substantive Items: Relative Total Bias (in %)
CATI Web
12 / 15 Research Interests Study Details Results Conclusions
Distributions
13 / 15 Research Interests Study Details Results Conclusions
Relative Total Bias: Income Distributions (only preliminary for web)
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
Mean Inc. (Euro)
Low Inc.
Middle Inc.
High Inc.
-40 -20 0 20 40
Income: Relative Total Bias (in %)
CATI Web
Measurement bias differs significantly (Two-sample Kolmogorov-Smirnov test: CATI p < 0.01; web p < 0.05) LINK
14 / 15 Research Interests Study Details Results Conclusions
Conclusions
Nonresponse bias:
in telephone mode largely driven by noncontact
larger for web compared to telephone
Measurement error bias:
for sensitive and socially desirable items smaller in web compared to
telephone
Total bias:
Larger total bias in web compared to telephone
Driven mostly by nonresponse bias
Reinforced by (smaller) measurement error
15 / 15 Research Interests Study Details Results Conclusions
Limitations and Future Work
Limitations:
Measurement equivalence survey and administrative data
New data release in summer
Stratified SRS
Future work:
Inclusion of measures of overall representativity: R-Indicators
Adjustment for Nonresponse and Sample Composition
Pooling the data
Costs per error source LINK
17 / 15 Research Interests Study Details Results Conclusions
Bibliography
Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J. & B.L. Messer (2009) Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Social Science Research 38: 1-18.
Duffy, B., Smith, K., Terhanian, G. & J. Bremer (2005) Comparing data from online and face-to-face surveys. International Journal of Market Research 47(6): 615-639.
Kreuter, F., Presser, S. & R. Tourangeau (2008) Social desirability in CATI, IVR, and web surveys. The effects of mode and question sensitivity. Public Opinion Quarterly 72(5): 847-865.
Lugtig, P., Lensvelt-Mulders, G.J.L.M., Frerichs, R. & A. Greven (2011) Estimating nonresponse bias and mode effects in a mixed-mode survey. International Journal of Market Research 53(5): 2-16.
McCabe, S.E., Boyd, C.J., Couper, M.P., Crawford, S. & H. D’Arcy (2002) Mode effects for collecting alcohol and other drug use data: Web and U.S. mail. Journal of Studies on Alcohol 63(3): 755-761.
Sakshaug, J., Yan, T. & R. Tourangeau (2010) Nonresponse error, measurement error, and mode of data collection. Tradeoffs in a multi-mode survey of sensitive and non-sensitive items. Public Opinion Quarterly 74(5): 907-933.
Sanders, D., Clarke, H.D., Stewart, M.C. & P. Whiteley (2007) Does mode matter for modeling political choice? Evidence from the 2005 British election study. Political Analysis 15: 257-285.
Sax, L.J, Gilmartin, S.K. & A.N. Bryant (2003) Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education 44(4): 409-432.
Stephenson, L.B. & J. Crête (2011) Studying political behavior: A comparison of internet and telephone surveys. International Journal of Public Opinion Research 23(1): 24-55.
Vannieuwenhuyze, J., Loosveldt, G. & G. Molensberghs (2010) A method for evaluating mode effects in mixed-mode surveys. Public Opinion Quarterly 74(5): 1027-1045.
18 / 15 Research Interests Study Details Results Conclusions
6 Appendix: Response Rates
Sample CATI Web
Gross Sample 12,400 11,836
Net Sample (w. tel. nb. / correct address)
9,332 10,525
Completes 2,400 1,082
Response Rate (RR1) 19.35 % 9.01 %
19 / 15 Research Interests Study Details Results Conclusions
Appendix: Stratification
BEH
2 employers
Empl.
last 10 years
LEH (UB I)
since 1/1/2001
LHG (UB II)
spell in 2010
20 / 15 Research Interests Study Details Results Conclusions
Appendix: Sample Composition
Stratum CATI Web Total
UB I %
3,600 29.03
4,277 36.13
7,876 32.50
UB II %
3,800 30.65
4,218 35.63
8,017 33.08
Empl. %
5,000 40.32
3,343 28.24
8,342 34.42
Total %
12,400 100.00
11,838 100.00
24,235 100.00
21 / 15 Research Interests Study Details Results Conclusions
What We (Think) We Know
(Validation) Studies* Comparing Web Mode to…
* non-exhaustive overview
** also considering causal inference, not just assessment of proportions
CATI: Computer Assisted Telephone Interview NR: Nonresponse (Bias)
IVR: Interactive Voice Recognition ME: Measurement (Error/Bias)
CAPI: Computer Assisted Personal Interview TE: Total (Survey) Error/Bias
Modes Item NR ME TE
w.
Valid.
Data
Aggregate Level (Frame Data, other External
Data such as Gov. Data, Large
Nat. Surveys, ‘Comparative
Sample’ etc.)
Sanders et al. 2007**
Stephenson/Crête 2011**
Vannieuwenhuyze et al.
2010
CAPI
CATI
CAPI, CATI
Voting Behavior
Voting Behavior
Political Variables (voting etc.)
Individual Level (Frame Data)
McCabe et al. 2002
Sax et al. 2003
Kreuter et al. 2008
Dillman et al. 2009
Sakshaug et al. 2010
CATI, IVR
CATI, IVR, Mail
CATI, IVR
Alcohol & Drug Use
University Issues
Academic Performance
Long Distance Service
Academic Performance
w/o.
Valid.
Data
Experimental
Approach (Mode Switch)
Dillman et al. 2008
…
s.a.
…
s.a. s.a.
s.a.
s.a.
Quasi-Exp. Approach/
Stat. Modeling (Prop. Score Matching, Test-
Retest, Weighting etc.)
Lugtig et al. 2011
Duffy et al. 2005
…
CATI
CAPI
…
Environ. knowledge
Voting, Activism, Cholesterol
22 / 15 Research Interests Study Details Results Conclusions
Administrative data
Nonrespondents: date of last interview
Respondents: date of interview
(web: income and employment data only available of two months
prior to survey, i.e. Dec. 2011)
Standard Errors
Bootstrap: with 5000 reps (for relative bias),
C.I. according to quantiles (0.025, 0.975)
Total Bias
unadjusted for nonresponse
Analyses: Assumptions
23 / 15 Research Interests Study Details Results Conclusions
Nonresponse Bias
1
Measurement Bias
2
Total Bias
3
Bias Estimates for Mean Statistics
100*)(
)(.
100*)(
)(.
100*)(
)(.
,
,,
,
,,
,
,,
fsadm
fsadmrespsvy
te
respadm
respadmrespsvy
me
fsadm
fsadmrespadm
nr
y
yyybiasrel
y
yyybiasrel
y
yyybiasrel
data frame :admdata;survey :svysample full :fs ;respondent :resp
24 / 15 Research Interests Study Details Results Conclusions
Implications for Survey Costs
CATI
Web with mailing
Web w/o mailing
Costs per interview (completed)
completed interviews
Costs
pe
r in
terv
iew
(com
ple
ted
)
25 / 15 Research Interests Study Details Results Conclusions
ncrefnr
ncrefnr
ncrefnr
ncrefnr
ncrefnr
ncrefnr
ncrefnr
Female
Mean Age (Years)
Edu. unknown
Edu. low. sec.
Edu. up. sec.
Edu. tert.
Nationality
-50 0 50 100 150
Socio-Demography: Relative Nonresponse Bias (in %)
CATI Web
4 Results: Relative Nonresponse Bias
(by type of nonresponse: noncontact and refusal)
nc: noncontact
ref: refusal
nr: nonresponse
26 / 15 Research Interests Study Details Results Conclusions
nc
ref
nr
nc
ref
nr
nc
ref
nr
nc
ref
nr
nc
ref
nr
nc
ref
nr
Employed:
Regular empl.
Marginal empl.
Mean Inc. (Euro)
Currently UB II
Past UB II
-40 -20 0 20 40
Substantive Items: Relative Nonresponse Bias (in %)
CATI Web
4 Results: Relative Nonresponse Bias
(by type of nonresponse: noncontact and refusal)
nc: noncontact
ref: refusal
nr: nonresponse
27 / 15 Research Interests Study Details Results Conclusions
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
Mean Age (Years)
<=30
30-39
40-49
50-59
>=60
-60 -40 -20 0 20 40
Age: Relative Total Bias (in %)
CATI Web
4 Results: Relative Nonresponse Bias
(by type of nonresponse: noncontact and refusal)
nc: noncontact
ref: refusal
nr: nonresponse
28 / 15 Research Interests Study Details Results Conclusions
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
Mean Inc. (Euro)
Low Inc.
Middle Inc.
High Inc.
-40 -20 0 20 40
Income: Relative Total Bias (in %)
CATI Web
4 Results: Relative Nonresponse Bias
(by type of nonresponse: noncontact and refusal)
nc: noncontact
ref: refusal
nr: nonresponse
29 / 15 Research Interests Study Details Results Conclusions
Relative Total Bias: Age Distributions
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
nonresponse
meas. bias
tot. bias
Mean Age (Years)
<=30
30-39
40-49
50-59
>=60
-60 -40 -20 0 20 40
Age: Relative Total Bias (in %)
CATI Web
30 / 15 Research Interests Study Details Results Conclusions
Measurement Bias: Age Distributions
Two-sample Kolmogorov-Smirnov test:
D = 0.000, p-value = 1 D = 0.000, p-value = 1 alternative hypothesis: two-sided alternative hypothesis: two-sided
31 / 15 Research Interests Study Details Results Conclusions
Measurement Bias: Income Distributions (only preliminary for web)
Two-sample Kolmogorov-Smirnov test:
D = 0.0721, p-value < 0.05 D = 0.0642, p-value < 0.01 alternative hypothesis: two-sided alternative hypothesis: two-sided
32 / 15 Research Interests Study Details Results Conclusions
Relative Nonresponse Bias
33 / 15 Research Interests Study Details Results Conclusions
Absolute Nonresponse Bias
34 / 15 Research Interests Study Details Results Conclusions
Relative Total Bias
35 / 15 Research Interests Study Details Results Conclusions
Absolute Total Bias
36 / 15 Research Interests Study Details Results Conclusions
1 The Question
Disentangling
nonresponse bias and measurement bias
comparing web and telephone mode
using validation data,
ideally from the sampling frame
(government data, large nat. surveys, etc.)