1

Coverage Rates and Coverage BiasWhen Interviewers Create Frames

Stephanie EckmanJoint Program in Survey MethodologyUniversity of Maryland

June 15, 2009 @ NCHS

2

Interviewers as Source of Error

Interviewers contribute to nonresponse, measurement error

Create frames in area-probability surveys Housing unit listing Missed housing unit procedure Household rostering Screening for eligibility

2

3

Literature Review

Previous work concentrates on: How many people are missed? What kinds of people are missed?

Errors by respondents Definitional Motivated

3

4

Research Questions

Compare listing methods Scratch: start with blank frame Update: start with address list

Mechanisms of lister error How are errors made? Incentives of interviewers

Bias & variance

4

5

Hypotheses on Mechanisms

What makes listing easier or more comfortable for the lister? Race or language match between

residents and lister High crime areas Driving

5

6

Hypotheses on Mechanisms

Confirmation bias in dependent listing Failure to add

HUs missing from list Failure to delete

Inappropriate units on list

6

7

Hypotheses on Mechanisms

Undercoverage and nonresponse Anecdotal evidence Hainer 1987 (CPS) Difficult to test

7

8

Pretest of Methods

Pretest with masters students 14 segments in SE Michigan 2 listings: traditional, dependent

Overall disagreement: 12% Evidence of confirmation bias

FTA: 13% less likely to add FTD: 11% less likely to delete

8

9

Census Dataset: Lister Agreement

2 identical listings 211 blocks Dependent on MAF, where available

Overall: 79% agreement

9

10

Census: Block Agreement Rates

10

11

Census Dataset: Drawbacks

Lister characteristics not available No manipulation to test for

confirmation bias

11

12

NSFG Data Collection

National Survey of Family Growth Three listings of 49 segments

1st listing by project 2nd listing: traditional 3rd listing: dependent

Manipulate input: add & delete lines

12

13

Paper 2: Mechanisms of Lister Error

Undercoverage and overcoverage rates Overall By housing unit and block

characteristics By listing method

Test hypotheses

13

14

Paper 3: Coverage Bias

Does lister error lead to bias in survey estimates?

2 sources of data on undercovered NSFG response data (27% selected) Experian data (60% match)

14

15

Paper 3: Estimating Bias

Direct estimates Re-estimate key variables without

cases undercovered by 1 or 2 listers Indirect estimates

Correlation between listing propensity & key variables

16

Questions and Concerns

Analysis of repeated listings Latent class analysis?

Interviewer debriefing What do I want to know?

NR and coverage Available datasets? How to test for this?

17

Thank You

Stay tuned for results next year

seckman@survey.umd.edu

17

18

Pretest: FTA Conf Bias

D listers missed 11 suppressed lines T listers missed only 4

Difference-in-differences estimate Suppression leads to 13% decrease in inclusion

18

HUs D listed T listed

Suppressed HUs 58 47 54

19

Pretest: FTD Conf Bias

D listers confirmed 4 added lines All in multi-unit buildings

T listers included only 1 (??)

Difference-in-differences estimate Bad lines in D lead to 11% increase in inclusion

19

HUs D listed T listed

Added HUs 24 4 1

20

Difference-in-Differences Estimate

Lines D listed T listedUnmanipulated HUs 732

655 (89%)

645 (88%)

Added HUs 244

(17%)1

(4%)

(0.17-0.89) – (0.04-0.88) = -0.13

rounding

20