Embed Size (px)
Citation preview
SHOW Me the Money?Effects of Preincentives, Differential Incentives,
and Envelope Messaging in an ABS Mail Survey
Kristen Cyffka1,2, Jennifer Dykema1, John Stevenson1, Kelly Elver1, Karen Jaques1
University of Wisconsin Survey Center
© 2012. Materials may not be reproduced without permission of the author.
1University of Wisconsin Survey Center2Department of Statistics, University of Wisconsin-Madison
AAPOR
May 18, 2012
Background
• RDD telephone surveys are no longer a viable mode for many studies
• Decline in last 2 decades in response rates
• Inadequate coverage due to cell phones
• Researchers are increasingly looking to alternative designs
• Return to mail surveys due to the DSF and ABS
University of Wisconsin Survey Center
• Return to mail surveys due to the DSF and ABS
• Effective ways to increase response rates in mail surveys (Edwards et al 2002, Church 1993):
• More interesting vs. less interesting questionnaire
• Recorded delivery vs. standard
• Monetary incentive vs. no incentive
2
Research Questions
• How much of an increase will a $5 pre-incentive yield over a $2 pre-incentive?
• Can we attract initial nonresponders with a second incentive?
University of Wisconsin Survey Center
• What effect will messages on the front of the envelope have on response rates?
• What is the most cost-effective incentive package we can give to increase response rates?
3
Study: Survey of the Health of Wisconsin (SHOW)
SHOW 1 SHOW 2
Year 2011 2012
Contacts Initial packetReminder postcardSecond packet
Initial packetReminder postcardSecond packet
Sampled cases 2,608 2,616
University of Wisconsin Survey Center 4
Completed surveys 1,566 1,440
Response rate (RR1) 66.9% 66.8%
Experimental conditions
Pre-incentive
Second incentive
Envelope message
Study: Survey of the Health of Wisconsin (SHOW)
SHOW 1 SHOW 2
Experimental conditions
Pre-incentive - $2- $5
- $2
Second incentive - $0- $2
- $0- $5
Envelope message - No message - No message
University of Wisconsin Survey Center 5
Envelope message - No message- Monetary message
- No message- Monetary message- Health message
Envelope Message: SHOW 1
University of Wisconsin Survey Center 6
Response Rates: Envelope Messaging
57% 57%
11%9%
50%
60%
70%
Resp
on
se R
ate
SHOW 1
After 2nd SAQ
67% 66%
University of Wisconsin Survey Center 7
0%
10%
20%
30%
40%
No Message Money
Resp
on
se R
ate
After 2nd SAQ
After 1st SAQ
p=0.600
Envelope Message: SHOW 2
University of Wisconsin Survey Center 8
Response Rates: Envelope Messaging
54%52% 51%
14%14% 15%
50%
60%
70%
Resp
on
se R
ate
SHOW 2
69% 66%66%
57% 57%
11%9%
50%
60%
70%
Resp
on
se R
ate
SHOW 1
67% 66%
University of Wisconsin Survey Center 9
0%
10%
20%
30%
40%
No Message Health Money
Resp
on
se R
ate
p=0.4370%
10%
20%
30%
40%
No Message Money
Resp
on
se R
ate
p=0.600
Response Rates: Pre-Incentive
53%
61%11%
9%
50%
60%
70%
Re
sp
on
se
Ra
teSHOW 1
64%70%
University of Wisconsin Survey Center 10
0%
10%
20%
30%
40%
$2 $5
Re
sp
on
se
Ra
te
p=0.005
Response Rates: After All Incentives
53%51%
11%
18%
50%
60%
70%
Resp
on
se R
ate
SHOW 2
64% 69%
54%52%
60%62%
11%10%
9%9%
40%
50%
60%
70%
Resp
on
se R
ate
SHOW 1
66%63%
69% 70%
University of Wisconsin Survey Center 11
0%
10%
20%
30%
40%
Resp
on
se R
ate
p=0.015
$2$0
$2$5
Pre:2nd:
0%
10%
20%
30%
40%
Resp
on
se R
ate
p2nd=0.734
$2$0
$2$2
$5$2
$5$0
Pre:2nd:
pAll =0.030
Response Rates: Message and Incentive
67%70%
62%
70%
64%68%
50%
60%
70%
SHOW 2
p=0.141
University of Wisconsin Survey Center 12
0%
10%
20%
30%
40%
50%
No Message
Health
Money
$2$0
$2$5
Pre:2nd:
Costs per Complete by Incentive
SHOW 1 Pre:2nd:
$2$0
$2$2
$5$0
$5$2
Total Number of Completes 383 371 403 423
Response Rates 65.7% 63.1% 69.0% 70.6%
Cost Per Complete $23.31 $26.05 $26.67 $26.28
University of Wisconsin Survey Center 13
SHOW 2 Pre:2nd:
$2$0
$2$5
Total Number of Completes 669 741
Response Rates 64.4% 69.3%
Cost Per Complete $25.49 $28.37
Other Analyses in SHOW 1
Data quality
• Item-missing data: no difference between treatments
Differential response
• No difference between treatments in:
University of Wisconsin Survey Center
• No difference between treatments in:• Gender
• Age
• Race
• Education
• Employment
• County
14
Discussion
• Envelope messaging
• Effects on response rates
• Future study
• Incentives
University of Wisconsin Survey Center
• Incentives
• Effects on response rates
• Pre-incentives
• Second incentives
• Implications on cost
15
Thank You!
For copies of this presentation or more information, contact:
Kristen Cyffka
University of Wisconsin Survey Center
Please visit us at:www.uwsc.wisc.edu
16
Response Rates: After All Treatments
53%56%
61% 59%
53% 51%
62% 61%10%
13% 6%11%
10% 10%
9% 8%
40%
50%
60%
70%
80%
SHOW 1
University of Wisconsin Survey Center 17
0%
10%
20%
30%
40%
Message None Message None Message None Message None
$2 $2 $5 $5 $2 $2 $5 $5
$0 $0 $0 $0 $2 $2 $2 $2
Response Rates: After All Treatments
55%50%
54% 53% 54%
47%
12%12% 10%
17% 17%20%
40%
50%
60%
70%
80%
SHOW 2
University of Wisconsin Survey Center 18
0%
10%
20%
30%
40%
None Health Money None Health Money
$2 $2 $2 $2 $2 $2
$0 $0 $0 $5 $5 $5
Response in Initial Nonresponders
25%
30%
35%
40%R
esp
on
se R
ate
SHOW 2: Second Incentive
23%
38%
University of Wisconsin Survey Center 19
0%
5%
10%
15%
20%
$0 $5
Resp
on
se R
ate
p=0.000
Cost analysis – What we included
We included:
• Mailings
• Postage, printing of all materials
• Stuffing
• Incentives
• Data entry
University of Wisconsin Survey Center
• Data entry
We did not include:
• Survey development and layout of SAQ
• Programming of data entry instrument
• Project management
• Sample management
• Data delivery
20
