Response Rates and Results of the Advance Letter Experiment

2004 RRFSS Workshop

Toronto, June 23, 2004

David A. Northrup, Renée Elsbett-Koeppen and Andrea Noack

ISR, York University

Outline

general comments on response rates how response rates are calculated a very brief history of response rates what strategies have/are being put in

place to deal with declining response rates

Outline (continued)

response rates and RRFSS what did it take to get the 62% rate for

2003 RRFSSnumber of callsrefusal conversions

results of the advance letter experiment

Calculating Response Rates

Completions / estimate of number of eligible households (HH) eligible HHs include completions, refusals,

callbacks, and a % of the “never answered”

ISR method same as BRFSS, aka “CASRO 3”

RRFSS 2003 = 62%, exclude callbacks = 71%

Response Rates for American Election Study

0%

20%

40%

60%

80%

100%

1952 1956 1960 1968 1972 1976 1980 1984

Response Rates for BRFSS

0%

20%

40%

60%

80%

1994 1995 1996 1997 1998 1999 2000 2001 2002

medianmean

Strategies for Improving Response Rates

interviewer training increase call attempts “convert” refusalsuse advance letterspayments (as a lottery, to completers, to the

whole sample)

Data Collection at ISR for RRFSS Response Rates

minimum number of 14 calls (more when there is reason to think extra calls might obtain a completion) limitation of one month sample release costs about 3 to 7 points on response rate

at least one attempt to convert almost all refusals

390,106

RRFSS: Fun with Numbers 1 (2003 Data)

number of calls: 390,106percent of interviews completed first call:

21number of interviews completed on the

10th or subsequent calls: 3,158number of interviews completed after a

refusal: 2,678

RRFSS: Fun with Numbers 2 (2003 Data)

average number of calls per completed interview: 4.65

most calls made for a single completion: 33 (for two (different) interviews)

response rate if 10 plus calls and refusal conversions are dropped: 48.2%

number of complaints about interviewer calling registered at ISR: 13

Characteristics of Refusers:2003 RRFSS Data

variable standard converted

mean age 47.99 53.86

education: > than high school (%)

university (%)

16.7

47.8

23.0

39.2

% employed 61.7 52.4

% saying health fair or poor 11.8 16.5

% doctor told high blood pressure 22.3 27.1

% smoke 100 cigarettes 52.5 55.0

# of cases: standard = 24,700, converted = 2,640all differences significant

Characteristics of Easy and Hard to Reach: 2003 RRFSS Data

variable easy to reach

hard to reach

mean age 50.43 43.50

education: > than high school (%)

university (%)

18.6

45.6

12.5

52.8

% employed 54.8 73.6

% saying health fair or poor 13.8 8.1

% doctor told high blood pressure 25.0 16.8

% smoke 100 cigarettes 52.7 50.6# of cases: easy = 17,000, hard = 3,150all differences significant

Letter Experiment: 1

six Health Units participated (Durham, London, Grey Bruce, Halton, Waterloo, Sudbury)

test two versions of letter: ISR and HU needed to work with our monthly target and

wanted to acknowledge random variation in response rates per HU per month

used sample “replicates” to implement experiment

Letter Experiment: 2

Month one: replicate 1, ISR letter; replicate 2, HU letter; replicate 3, and 4 (when used), control group changed presentation in months 2 and 3 copy of letter at the end of this set of

handouts exactly the same text, different letterhead,

signature & envelope Except Halton

Letter Experiment: 3

survey introduction exactly the same except one additional sentence “Recently, we sent a letter to your

household about an important research project.”

questions about the letter the same except Durham

Why the Letter Might Improve Response Rates to RDD Surveys

reduces the possibility that the telephone call catches people by surprise

increases legitimacy of research project in the eye of the potential respondents

demonstrates social value improves the confidence of the interviewer

Why Advance Letters Might Not Improve Response Rates to

RDD Surveys

letter does not reach, or is not read by, respondent

ceiling effects survey topic & subpopulations they give “timid” participants a chance to

prepare to say “no”

Response Rates for Months 1 & 2 of the Experiment

66%62% 63%

70%

30%

50%

70%

letter no letter ISR letter HU letter

p value =.035 (for letter (1,200) versus no letter (1,345))p value =.025 (ISR (600) versus HU (600))

Response Rates Months 1 & 2: All Six Health Units

0%

20%

40%

60%

80%

Durham London GreyBruce

Halton Waterloo Sudbury

none

ISR

HU

See next slide for numbers & p values

RR by HU and Treatment treatment, RR (%) P value

HU

none

ISR

HU

none/ISR

none/ HU

ISR/ HU

Durham 58.9 63.6 65.3 .422 .275 .803

London 57.8 64.5 69.2 .254 .048 .475

Grey 70.5 73.2 78.7 .618 .128 .367

Halton 57.5 57.0 68.7 .925 .053 .087

Waterloo 64.7 60.2 65.8 .441 .851 .416

Sudbury 69.8 63.2 71.4 .255 .776 .213

Number of cases per HU: ISR = 50, HU = 50, none = 100

Mean Calls per Completion

mean # of calls P value

None

ISR

HU

none/ISR

none/ HU

ISR/ HU

Durham 5.89 5.69 5.57 .825 .721 .905

London 6.41 6.06 4.43 .751 .049 .060

Grey 4.36 4.58 4.69 .763 .622 .896

Halton 5.18 6.42 6.53 .157 .124 .935

Waterloo 4.82 6.42 4.38 .049 .510 .047

Sudbury 5.95 5.14 5.42 .445 .597 .809

Number of cases per HU: ISR = 50, HU = 50, none = 100

Mean Calls per Completion by Letter Status

0

1

2

3

4

5

6

# of calls

letter no letter ISR HU not seeletter

sawletter

Letter/no letter p = .890, ISR/HU p = .230, not see/saw p = .001

% of First Call Attempts Leading to Completions & Refusals

0%

5%

10%

15%

letter no letter ISR letter HU letter

comps

ref

letter/no letter p =.204, ISR/HU p =.008

At the Start of the Interview

35%

27%

38%

0%

20%

40%

R told Interviewerthey saw letter

R asked aboutletter

"standard" surveyintroduction

Awareness of Letter

Variable (based on 602 cases) total

R indicated saw letter at intro 30

R indicated letter came to house 21

total respondents aware of letter 51

personally read the letter 40

got more info (web site, 1-800) 1

letter made a lot of difference to decision to participate

26

Data Characteristics

Variable ISR(n=300)

HU(n=300)

none(n=620)

P

year of birth 1955 1954 1954 .820

male (%) 58 56 57 .800

employed (%) 59 59 60 .627

health excellent 22 22 22 .951

smoked at least 100 cigarettes (%)

50 56 56 .197

Costs: Month One

cost of materials: $314; staff cost: $1,919 total: $2,233per case cost: $3.62buys 72 interviews or 12 per HUneed to estimate savings from making

fewer calls, and making fewer refusal conversion calls

Conclusions HU letter (seems to) increase response and

warrants consideration as a tool to improve RRFSS response rates

affect on variable distributions minimal, but small sample size limits scope of examination

social-political distance between respondent and sender probably matters

letters may have value other than just increasing response rates

Questions