Upload
krikor
View
43
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Mixed Modes and Measurement Error. Gerry Nicolaas. Mixed Modes and Measurement Error. Funded under the ESRC Survey Design and Measurement Initiative 3-year contract starting 1 Oct 2007 Collaboration between academics and data collection organisation. Research Team. - PowerPoint PPT Presentation
Citation preview
Mixed Modes and Measurement Error
Gerry Nicolaas
Mixed Modes and Measurement Error• Funded under the ESRC Survey Design and
Measurement Initiative• 3-year contract starting 1 Oct 2007 • Collaboration between academics and data
collection organisation
Research Team• National Centre for Social Research (NatCen)
Gerry NicolaasSteven Hope
• Institute for Social & Economic Research (ISER)Peter LynnAnnette JäckleAlita NandiNayantara Dutt
• Freelance Survey Methods ConsultantPam Campanelli
Background
• Interviewer-administeredface-to-face interview (PAPI, CAPI)telephone interview (PAPI, CATI)
• Self-administeredpostal questionnaireself-completion questionnaires with interviewer
present (PAPI, CASI, A-CASI)IVR and TDE in telephone surveysweb/email
Modes of data collection:
Choice of mode = trade-off
Quality
Time Cost
Coverage error
Non-responseerror
Measurement error
Recording error
Samplingerror
Differences across modes• coverage error
different members of the population have a zero chance of inclusion depending on the mode
• non-response errordifferential non-response bias
• measurement errorrespondents give different answers in different
modes
Increasing use of mixed modes:• To maximise response rates (unit & item)
offer choice of mode to respondentsuse alternative mode among non-respondents
• To reduce measurement errorself-completion modes for sensitive items within
a f2f or tel interview• To reduce costs
sequentially with cheapest mode firstdifferent waves of a longitudinal study
Mixing modes:• Different modes to collect different data items from
the same respondentse.g. CAPI with CASI module data comparability not affecteddata quality may be improved
• Different modes to collect the same data from different respondents e.g. tel follow-up among postal non-responderse.g. panel survey with wave 1=CAPI & wave 2=CATIpotential for mode effects
For an overview of using mixed modes in surveys:
• Edith D. de Leeuw (2005) “To Mix or Not to Mix Data Collection Modes in Surveys”, Journal of Official Statistics, 21(2), pp. 233–255
• One of Edith’s conclusions: Hardly any theoretical or empirical knowledge is
available on how to design optimal questionnaires for mixed-mode data collection.
Study Design
Main Objective
• Practical advice on how to improve portability of questions across modesWhich mode combinations are likely to produce
comparable responses?Which types of questions are more susceptible
to mode effects?
Research Design• A literature review & framework of mixed modes
develop a theoretical framework identify gaps in evidence base and formulate
hypotheses to address gaps• Quantitative data analysis
test hypotheses using existing datasets and new experimental data
• Cognitive interviewingexplore how respondents process questions in
different modes
The Quantitative Data• Existing datasets, e.g.
1999 Welsh Assembly Election Study2005 Social Capital SurveyEuropean Social Survey mode experiments2006 Health Survey for England London boost
• New experimental datafollow-up surveys to BHPS & NatCen Omnibusfocus on f2f, tel and web comparisons
NatCen Omnibus Survey• Two rounds of face-to-face data collection
Jul/Aug 2008 and Sep/Oct 2008• Follow-up surveys after 6 months
Omnibus respondents who agreed to follow-up and web access
Random allocation = 400 f2f, 400 tel, 400 web• Cognitive interviews after 6 months
Purposively selected sample of 36 respondents from follow-up surveys
British Household Panel Study• BHPS Wave 18 (all f2f)
Sep 2008 – Dec 2008• Follow-up surveys after 6 months
BHPS respondents who agreed to follow-up and have web access
Random allocation = 400 tel, 400 webNo separate f2f data collection at this stage
• BHPS Wave 19 (all f2f)Sep 2009 – Dec 2009
20 BHPSquestions
20 BHPSquestions
20 BHPSquestions
Another 40questions
20 BHPSquestions
20 BHPSquestions
20 BHPSquestions
20 BHPSquestions
20 BHPSquestions
Another 40questions
Another 40questions
Another 40questions
Another 40questions
Other modules
Other BHPS
questions
Other BHPS questions
NatCen Omnibus
BHPS W18
BHPS W19 (12 months after W18)
F2F interview after 6 months
Tel interview after 6 months
Web q’naire after 6 months
Tel interview after 6 months
Web q’naire after 6 months
Cognitive interviews after 6 months
Key Features of Design• Repeated measures
enables estimation of mode effects in measures of change
• Random allocation to modes • Compare ‘seasoned’ panel members with ‘fresh’
survey sample members• Cost-efficient design to collect very rich
experimental data
Limitations• Restricted to respondents with web access
primarily relying on randomisation within the sample as our basis for inference
relatively broad basis for extrapolation to general population compared to other mixed mode studies
• BHPS f2f follow-up is 12 months later rather than 6 monthsovercome by comparing data from Omnibus and its f2f
follow-up after 6 months with data from BHPS and its f2f follow-up after 12 months
Literature Review
Aims of literature review
• A review of the evidence of differences in measurement due to mode for:Different types of questionsDifferent combinations of modes
• Identify gaps (mode pairs and question types) in the evidence base
• Formulate hypotheses to address these gaps
The literature review• Initially over 700 papers identified• Currently screening papers for relevance and
summarising relevant papers• Criteria for inclusion:
comparison of 2 or more modesmodes of survey data collectionmeasurement error
Classify existing evidence:• The question
question type (e.g. attitude, behaviour, other factual) question format (e.g closed/open, scale, # categories) task difficulty sensitivity of question
• The mode comparison interviewer presence (face-to-face, telephone, none) delivery of question (visual, aural) response list (visual, aural) recording of responses (oral, written)
Continued on next slide
Classify existing evidence:• The results
Hypotheses tested Indicators and statistical methods Results
Synthesis of literature• Causes of differential measurement error
e.g. interviewer presence, cognitive task• Nature of differential measurement error
e.g. social desirability bias, survey satisficing• Magnitude of differential measurement error
Causes of Mode Effects on Measurement (Roberts, Jäckle & Lynn; 2006)
Comprehension Retrieval Judgement Response
Sufficient Effort?
Willingness to disclose?
Task difficulty R motivation R ability* *
Stimulus: Cognitive task
Interviewer presence:Pace, non-verbal communication, multitasking
Interviewer presence: Anonymity vs. rapport
Privacy/legitimacy
Shortcuttin
g
Social desirability
bias
Initial observations from lit review:• Many experiments are not theory-driven
Focus is on descriptive comparisons of response distributions across modes
Lack of generalisable inferences about causal mechanisms
• Many papers provide insufficient information about the questions and modes being testedquestion type & format, sensitivity of question, task
difficulty interviewer presence, delivery of question and
response options, recording of responses
Next steps
Next steps:
• Complete the literature review and develop theoretical framework
• Identify gaps in the evidence base• Design extra 40 questions for mixed modes
experiment
RC-33 Conference in Naples, 1-5 Sept 2008• Research team is chairing a session on Mixed
Modes and Measurement Error• Steven Hope presenting a paper on preliminary
results of literature review