Report on the 2008 International Total Survey Error Workshop (ITSEW)

Presented at Q2008 Conference

Rome, Italy July 11, 2008

Presented by Paul Biemer

Acknowledge

Other Organization Committee members:

Roeland Beerten, ONSLilli Japec, Statistics SwedenMary Mulry, Census BureauAlan Karr, National Institute for Statistical ScienceJerry Reiter, Duke UniversityClyde Tucker, BLSBrian Meekins, BLS

Outline

• Definition of Total Survey Error• Highlights of the Recent Workshop on Total

Survey Error Workshop• Other Emerging Areas of Research• Future Directions of Research on TSE

Purpose of the ITSEWs

• To stimulate new research on total survey error (TSE)• Provide a forum for researchers to exchange ideas on

current research projects• Initiate new projects to advance the TSE field• To generate interest in the field among students and new

researchers

Organization of the Workshops

• Attendance by invitation only – although no one has ever been refused a seat at the workshop

• Focus is on on-going research and undeveloped ideas– Midcourse research– Ideas still in incubation– Exploratory research

• ITSEW series began 2005 in Washington, DC• ITSEW 2009 will be held in Tälberg, Sweden• National Institute of Statistical Science (NISS) provides

funding and staff support• Support from other organizations is welcome

2008 Total Survey Error WorkshopTheme: Multiple Error Sources and their Interactions

• Held in Research Triangle Park, NC on June 2-4, 2008• Organizing committee:

– Paul Biemer, RTI and UNC-CH– Roeland Beerten, ONS– Lilli Japec, Statistics Sweden– Mary Mulry, Census Bureau– Alan Karr, NISS– Jerry Reiter, Duke University– Clyde Tucker, BLS– Brian Meekins, BLS

• Sponsored by the National Institute for Statistical Science, RTI International and the Survey Research Methods Section of ASA

27 Presenters at the 2008 ITSEW

Joop HoxAnnica IsakssonPeter LundquistNoelle Molinari Donsig Jang Scott Fricker Ting YanSteven Cohen Wendy Hicks Barry Schouten Emilia PeychevaKristen OlsonJohn Dixon

Paul Biemer

Catharine Burt

Andy Peytchev

Brian Meekins

Mary Mulry

Roger Tourangeau

Jerry Reiter

Lars Lyberg

Rita Thissen

Oztas Ayhan,

Mojca Bavdaz,

Steven Machlin

Eric Slud

Total Survey Error

• Sampling Error• Nonsampling Error

– Specification– Nonresponse– Noncoverage– Measurement– Data Processing

Mean Squared

Error (MSE)

TSE: the Concept

• Includes both measurement, analysis and reporting of multiple sources of error

–Quantify the major components of survey error for a survey–Assess contributions of each source to TSE–Combine these to estimate total MSE

• Does not include error reduction methods –E.g., questionnaire design, nonresponse reduction methods, etc.

Arguments for Estimating TSE

• To correct inferences based upon faulty theory

– Biased estimates– Biased measures of uncertainty

• To optimize the design of future surveys• To provide quality declarations to users

Arguments Against Estimating TSE

• Cost• Intractability• Inaccessible methodology• Lack of motivation• Fear

Topic Coverage

Coverage

Sampling

Nonresponse

.

Instrument

Mode

Interviewer

Respondent

.

Processing

0 2 4 6 8 10 12 14 16 18

Highlights

• Interaction between nonresponse and measurement error– Effect on measurement error of nonresponse followup– Statistical models for simulating the measurement error and

nonresponse interaction– Meta-analysis of literature on nonresponse-measurement error

interaction– Empirical studies of the relationship between nonresponse and

measurement error– Causal factors influencing both response propensity and

measurement error– Nonresponse propensity and under-reporting of sensitive topics

Other common themes

• Frame stratification errors and sampling errors• Frame coverage errors and measurement errors (e.g., cell

phone samples and measurement error)• Interviewer effects on nonresponse and measurement

errors

Gaps in the Research

• Data processing errors– Questionnaire design and editing errors– Information content and coding errors

• Establishment surveys• Specification error vs. measurement error

– i.e., measuring the wrong concept accurately vs. measuring the right concept inaccurately

• Need for meta-analysis of existing studies• Better indicators of data quality

– Schouten, Bethlehem, et al

• More attention to costs of error reduction methods

Gaps in the Research (cont’d)

• Methods for developing countries• TSE issues in multi-cultural surveys• Methods for estimating variances that reflect multiple error

sources, not just sampling errors• Standardized indicators for comparing the quality of

survey data• A common language for TSE

Need for Data

• How can “true values” be obtained?• Paradata for TSE analysis

– E.g. level of effort to obtain a response in the broadest sense, condition or value of housing, encounters with gatekeepers

– need a systematic / theory-based framework – quality of paradata / “metadata for paradata”

• Data sets for TSE analysis are scarce

Emerging Research Areas

• Interviewer effects in surveys and their effects on TSE• Methods for predicting poor reporters (i.e., bad

respondents)• Language translation issues for TSE• Completing the feed-back loop between data processors

and survey designers• Methods for estimating the nonresponse-response error

interaction

Some Initiatives Underway for ITSEW 2009

• Lars Lyberg is organizing ITSEW 2009 in Talberg, Sweden for June 14-17, 2009

• Biemer and Lyberg will edit a special issue of Public Opinion Quarterly in 2010 on TSE.

• Special issue on TSE is also being planned for the Journal of Official Statistics in 2010.

• Alan Karr has offered NISS services to host a moderated Wikipedia on Total Survey Error

• Special Topic Sessions for ITSEW 2009– Nonresponse and measurement error interaction and practical

implications (Paul Biemer)– Shared data sets for TSE research (Barry Schouten)– Coverage error (Stephanie Eckman)– POQ and JOS have offered special issues in TSE

Some Initiatives Underway for ITSEW 2009 (cont’d)

Some Issues in Research on Nonresponse-Response Error Interaction

• Is there an interaction between the propensity to respond and the propensity to respond correctly?

• Are reluctant respondents more likely to be misclassified by survey questions?

• What can be done to obtain accurate data from reluctant respondents?

• If so, what are some strategies for nonresponse followup that minimize total survey error?

• What research designs allow analyst to separate nonresponse error and measurement error?

• What models are helpful for understanding the linkage between measurement error and nonresponse error?

P εMeasurement Process Model

ZεZp T

YP ε

Common Cause Model

Z T

Y P εTrue Value

Model

T

Y

Example: Kristen Olsen’s “Causal” Models

Summary

• Papers from the ITSEW 2008 will be available by August 15 on the NISS website

• Call for papers for ITSEW 2009 will be announced in September

• ITSEW 2009 will be held June 14-17 in Sweden• Papers that examine the interaction between two or more

error sources are encouraged.