View
216
Download
1
Category
Tags:
Preview:
Citation preview
Response Rates and Results of the Advance Letter Experiment
2004 RRFSS Workshop
Toronto, June 23, 2004
David A. Northrup, Renée Elsbett-Koeppen and Andrea Noack
ISR, York University
Outline
general comments on response rates how response rates are calculated a very brief history of response rates what strategies have/are being put in
place to deal with declining response rates
Outline (continued)
response rates and RRFSS what did it take to get the 62% rate for
2003 RRFSSnumber of callsrefusal conversions
results of the advance letter experiment
Calculating Response Rates
Completions / estimate of number of eligible households (HH) eligible HHs include completions, refusals,
callbacks, and a % of the “never answered”
ISR method same as BRFSS, aka “CASRO 3”
RRFSS 2003 = 62%, exclude callbacks = 71%
Response Rates for American Election Study
0%
20%
40%
60%
80%
100%
1952 1956 1960 1968 1972 1976 1980 1984
Response Rates for BRFSS
0%
20%
40%
60%
80%
1994 1995 1996 1997 1998 1999 2000 2001 2002
medianmean
Strategies for Improving Response Rates
interviewer training increase call attempts “convert” refusalsuse advance letterspayments (as a lottery, to completers, to the
whole sample)
Data Collection at ISR for RRFSS Response Rates
minimum number of 14 calls (more when there is reason to think extra calls might obtain a completion) limitation of one month sample release costs about 3 to 7 points on response rate
at least one attempt to convert almost all refusals
390,106
RRFSS: Fun with Numbers 1 (2003 Data)
number of calls: 390,106percent of interviews completed first call:
21number of interviews completed on the
10th or subsequent calls: 3,158number of interviews completed after a
refusal: 2,678
RRFSS: Fun with Numbers 2 (2003 Data)
average number of calls per completed interview: 4.65
most calls made for a single completion: 33 (for two (different) interviews)
response rate if 10 plus calls and refusal conversions are dropped: 48.2%
number of complaints about interviewer calling registered at ISR: 13
Characteristics of Refusers:2003 RRFSS Data
variable standard converted
mean age 47.99 53.86
education: > than high school (%)
university (%)
16.7
47.8
23.0
39.2
% employed 61.7 52.4
% saying health fair or poor 11.8 16.5
% doctor told high blood pressure 22.3 27.1
% smoke 100 cigarettes 52.5 55.0
# of cases: standard = 24,700, converted = 2,640all differences significant
Characteristics of Easy and Hard to Reach: 2003 RRFSS Data
variable easy to reach
hard to reach
mean age 50.43 43.50
education: > than high school (%)
university (%)
18.6
45.6
12.5
52.8
% employed 54.8 73.6
% saying health fair or poor 13.8 8.1
% doctor told high blood pressure 25.0 16.8
% smoke 100 cigarettes 52.7 50.6# of cases: easy = 17,000, hard = 3,150all differences significant
Letter Experiment: 1
six Health Units participated (Durham, London, Grey Bruce, Halton, Waterloo, Sudbury)
test two versions of letter: ISR and HU needed to work with our monthly target and
wanted to acknowledge random variation in response rates per HU per month
used sample “replicates” to implement experiment
Letter Experiment: 2
Month one: replicate 1, ISR letter; replicate 2, HU letter; replicate 3, and 4 (when used), control group changed presentation in months 2 and 3 copy of letter at the end of this set of
handouts exactly the same text, different letterhead,
signature & envelope Except Halton
Letter Experiment: 3
survey introduction exactly the same except one additional sentence “Recently, we sent a letter to your
household about an important research project.”
questions about the letter the same except Durham
Why the Letter Might Improve Response Rates to RDD Surveys
reduces the possibility that the telephone call catches people by surprise
increases legitimacy of research project in the eye of the potential respondents
demonstrates social value improves the confidence of the interviewer
Why Advance Letters Might Not Improve Response Rates to
RDD Surveys
letter does not reach, or is not read by, respondent
ceiling effects survey topic & subpopulations they give “timid” participants a chance to
prepare to say “no”
Response Rates for Months 1 & 2 of the Experiment
66%62% 63%
70%
30%
50%
70%
letter no letter ISR letter HU letter
p value =.035 (for letter (1,200) versus no letter (1,345))p value =.025 (ISR (600) versus HU (600))
Response Rates Months 1 & 2: All Six Health Units
0%
20%
40%
60%
80%
Durham London GreyBruce
Halton Waterloo Sudbury
none
ISR
HU
See next slide for numbers & p values
RR by HU and Treatment treatment, RR (%) P value
HU
none
ISR
HU
none/ISR
none/ HU
ISR/ HU
Durham 58.9 63.6 65.3 .422 .275 .803
London 57.8 64.5 69.2 .254 .048 .475
Grey 70.5 73.2 78.7 .618 .128 .367
Halton 57.5 57.0 68.7 .925 .053 .087
Waterloo 64.7 60.2 65.8 .441 .851 .416
Sudbury 69.8 63.2 71.4 .255 .776 .213
Number of cases per HU: ISR = 50, HU = 50, none = 100
Mean Calls per Completion
mean # of calls P value
None
ISR
HU
none/ISR
none/ HU
ISR/ HU
Durham 5.89 5.69 5.57 .825 .721 .905
London 6.41 6.06 4.43 .751 .049 .060
Grey 4.36 4.58 4.69 .763 .622 .896
Halton 5.18 6.42 6.53 .157 .124 .935
Waterloo 4.82 6.42 4.38 .049 .510 .047
Sudbury 5.95 5.14 5.42 .445 .597 .809
Number of cases per HU: ISR = 50, HU = 50, none = 100
Mean Calls per Completion by Letter Status
0
1
2
3
4
5
6
# of calls
letter no letter ISR HU not seeletter
sawletter
Letter/no letter p = .890, ISR/HU p = .230, not see/saw p = .001
% of First Call Attempts Leading to Completions & Refusals
0%
5%
10%
15%
letter no letter ISR letter HU letter
comps
ref
letter/no letter p =.204, ISR/HU p =.008
At the Start of the Interview
35%
27%
38%
0%
20%
40%
R told Interviewerthey saw letter
R asked aboutletter
"standard" surveyintroduction
Awareness of Letter
Variable (based on 602 cases) total
R indicated saw letter at intro 30
R indicated letter came to house 21
total respondents aware of letter 51
personally read the letter 40
got more info (web site, 1-800) 1
letter made a lot of difference to decision to participate
26
Data Characteristics
Variable ISR(n=300)
HU(n=300)
none(n=620)
P
year of birth 1955 1954 1954 .820
male (%) 58 56 57 .800
employed (%) 59 59 60 .627
health excellent 22 22 22 .951
smoked at least 100 cigarettes (%)
50 56 56 .197
Costs: Month One
cost of materials: $314; staff cost: $1,919 total: $2,233per case cost: $3.62buys 72 interviews or 12 per HUneed to estimate savings from making
fewer calls, and making fewer refusal conversion calls
Conclusions HU letter (seems to) increase response and
warrants consideration as a tool to improve RRFSS response rates
affect on variable distributions minimal, but small sample size limits scope of examination
social-political distance between respondent and sender probably matters
letters may have value other than just increasing response rates
Questions
Recommended