Embed Size (px)
Citation preview
1
2
The Effects of The Effects of Survey Design Features and Survey Design Features and
Economic Conditions on Economic Conditions on Business Survey Response RatesBusiness Survey Response Rates
Diane K. WillimackU.S. Census Bureau
Economic Census
10
11
3
Outline
Motivation– Economic Census
Conceptual framework for nonresponse in establishment surveys
Data analysis, results, and conclusions– Number of pages– Economic conditions
4
U.S. Economic Census
Establishment level data Collected every 5 years
– Reference year ends in 2 or 7– Collection year is the following year
5
U.S. Economic Censuscontinued
Self-administered– Paper mail-out/mail-back– 2002: Electronic via downloadable software
Tailored by industry– 2002: 520 different questionnaire versions
6
Motivation
Changes in format affecting the 2002 Economic Census– Paper size: Legal Letter– Page layout: 2 columns 1 column
Number of pages increased
Response rates decline???
Thank you
for completing
the 2002
Econom
ic Census
8
The Data
1992, 1997, and 2002 Economic Censuses– Check-in rates
• Receipts + 2nd time Undeliverables Mail Counts
– 1st week of June, 1993, 1998, 2003
Unit of analysis = questionnaire versionquestionnaire version n = 1459 questionnaire versions
9
Check-In Rates by Year,1st week of June of the Collection Year
Census Year
Number of Pages
(Range)
Mean Check-in Rates Across
All Versions
1992 2 – 14 77.34%
1997 2 – 16 69.50%
2002 3 – 27 65.19%
14
Research Question
1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
15
A Conceptual Framework forBusiness Survey Participation
Out of Survey Organization Control Under Survey Organization Control
External Environment Survey Design
The Business
ResponseBurden
BusinessGoals
Survey Participation
DecisionSource: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.
The Respondent
16
Defining the Model
Concept
(External Environment;
Characteristics of the Business;
Characteristics of the Respondent;
Survey Design)
Measure
Economic Conditions %GDP, %Employment
Industrial Sector
----------------------
#pp for the form, availability of electronic mode
Response Rate = f
17
The Model
Concept
(External Environment;
Characteristics of the Business;
Characteristics of the Respondent;
Survey Design)
Measure
Economic Conditions %GDP, %Employment
Industrial Sector
----------------------
#pp for the form, availability of electronic mode
Response Rate = f
18
Main Effects ModelMain Effects Model:What is the relationship, regardless of
the census year?
Response Rate = a + b (#pages)
+ [effects of economic variables] + gi (Sector*) + h (Mode)
+ [error term]
* The omitted variable among the Sector dummies is Mining.
19
Care in Interpreting Results
Behavioral model– Explanatory, NOT predictiveNOT predictive– Estimated coefficients
• Relative magnitude• Direction (+ or -)• Significance
20
Main Effects Model
Estimated Coefficient
Significantly different from 0? p-value
Intercept 81.8181.81 YesYes <.0001<.0001
# Pages -1.13-1.13 YesYes <.0001<.0001
Availability of Electronic Mode -6.91-6.91 YesYes <.0001<.0001
21
Research Question
“Does 2002 differ from previous censuses?”
H0: 1992 = 1997 = 2002
Pairwise comparisons
22
Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ
by census year?
= a + b92(#pp) + cc9292(#pp)(#pp)2 2 + d+ d9292 (#pp) (#pp)33
+ b97(#pp) + cc9797(#pp)(#pp)2 2 + d+ d9797 (#pp) (#pp)33
+ b02(#pp) + cc0202(#pp)(#pp)2 2 + d+ d0202 (#pp) (#pp)33
+ [effect of economic variables]
+ gi (Sector*)
+ h (Mode)
+ [error term]
Response Rate
* The omitted variable among the Sector dummies is Mining.
23
Does the effect of pages differ by census years?
Testing HTesting H00: 1992 = 1997 = 2002: 1992 = 1997 = 2002
Reject H0F-stat
p-value
#pp H0: b92 = b97 = b02 Yes 27.23p<.0001
(#pp)2 H0: c92 = c97 = c02 Yes 18.28p<.0001
(#pp)3 H0: d92 = d97 = d02 Yes 12.04p<.0001
24
Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ
by census year?
Page Term Year
Estimated Coefficient
Differ from 1997?
Differ from 2002?
#pp 1992 -9.94
1997 -11.92
2002 -5.78
(#pp)2 1992 1.14
1997 1.23
2002 0.33
(#pp)3 1992 -0.04
1997 -0.04
2002 -0.01
25
Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ
by census year?
Page Term Year
Estimated Coefficient
Differ from 1997?
Differ from 2002?
#pp 1992 -9.94 p=.0028 p<.0001
1997 -11.92 p<.0001
2002 -5.78
(#pp)2 1992 1.14
1997 1.23
2002 0.33
(#pp)3 1992 -0.04
1997 -0.04
2002 -0.01
26
Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ
by census year?
Page Term Year
Estimated Coefficient
Differ from 1997?
Differ from 2002?
#pp 1992 -9.94 p=.0028 p<.0001
1997 -11.92 p<.0001
2002 -5.78
(#pp)2 1992 1.14 p=.5278 p<.0001
1997 1.23 p<.0001
2002 0.33
(#pp)3 1992 -0.04
1997 -0.04
2002 -0.01
27
Interaction Effects Model:Interaction Effects Model:Does the effect of pages differ
by census year?
Page Term Year
Estimated Coefficient
Differ from 1997?
Differ from 2002?
#pp 1992 -9.94 p=.0028 p<.0001
1997 -11.92 p<.0001
2002 -5.78
(#pp)2 1992 1.14 p=.5278 p<.0001
1997 1.23 p<.0001
2002 0.33
(#pp)3 1992 -0.04 p=.7564 p<.0001
1997 -0.04 p<.0001
2002 -0.01
28Number of Pages
Re
sp
on
se
Ra
te
Estimated ModelEstimated Model19921992
29Number of Pages
Re
sp
on
se
Ra
te
Estimated ModelEstimated Model19971997
30Number of Pages
Re
sp
on
se
Ra
te
Estimated ModelEstimated Model20022002
32
Research Questions
1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
33
Research Question #1 Conclusion
Negative effect– #pages response rates – Caution: Behavioral model vs. Controlled
Experiment• Cannot identify causality• Cannot isolate factors
34
Research Question #1 Conclusion
continued
Reject H0: 1992 = 1997 = 2002
– 2002 differed from the previous censuses (per results from pairwise comparisons)
Negative effect of #pages on economic census response appears to have diminished over time
35
A Conceptual Framework forBusiness Survey Participation
Out of Survey Organization Control Under Survey Organization Control
External Environment Survey Design
The Business
ResponseBurden
BusinessGoals
Survey Participation
DecisionSource: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.
The Respondent
36
Household SurveyLiterature
Better economic times refusal rates increased (in a household survey)– Harris-Kojetin and Tucker, 1999, “Exploring the
Relation of Economic and Political Conditions with Refusal Rates to a Government Survey,” JOS, 15:167-184
37
Research Questionscontinued
1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
2) Did economic conditions affect economic census response rates?
38
The Model
Concept
(External Environment;
Characteristics of the Business;
Characteristics of the Respondent;
Survey Design)
Measure
Economic Conditions Economic Conditions %%GDP, %GDP, %EmploymentEmployment
Industrial Sector
----------------------
#pp for the form, availability of electronic mode
Response Rate = f
39
Main Effects ModelMain Effects Model:What is the relationship, regardless of
the census year?
Response Rate = a + b (#pages)
+ e (%GDP)
+ f (%Employment)
+ gi (Sector*) + h (Mode)
+ [error term]
* The omitted variable among the Sector dummies is Mining.
40
Main Effects Model
Estimated Coefficient
Significantly different from 0? p-value
%GDP -0.12-0.12 NoNo 0.41930.4193
%Employment -1.11-1.11 YesYes <.0001<.0001
41
Interaction Effects Model:Interaction Effects Model:Do the effects of economic conditions
differ by census year?
= a + [effect of page variables]
+ e+ e92 92 (%(%GDP)GDP) + f+ f9292 (% (%Employment)Employment)
+ e+ e97 97 (%(%GDP)GDP) + f+ f9797 (% (%Employment)Employment)
+ e+ e02 02 (%(%GDP)GDP) + f+ f0202 (% (%Employment)Employment)
+ gi (Sector*)
+ h (Mode)
+ [error term]
Response Rate
* The omitted variable among the Sector dummies is Mining.
42
Do the effects of economic conditions differ by census year?
Testing HTesting H00: 1992 = 1997 = 2002: 1992 = 1997 = 2002
H0: 1992 = 1997 = 2002 Reject H0F-stat
(p-value)
%%GDPGDP No 0.65(0.5209)
%%EmploymentEmployment No 1.25(0.2861)
43
Research Questions
1) Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
2) Did economic conditions affect economic census response rates?
44
Research Question
3) Does the effect of number of pages on response rates vary with economic conditions?
45
Mixed Effects Models
+ j+ ji i (#pp) * (%(#pp) * (% GDP) GDP)
+ k+ kii (#pp) (#pp)22 * (% * (% GDP) GDP)
+ m+ mii (#pp) * (% (#pp) * (% Emply) Emply)
+ n+ nii (#pp) (#pp)22 * (% * (% Emply) Emply)
+
Response Rate
= a
+ bi (#pp)
+ ci (#pp)2
+ di(#pp)3
+ ei (% GDP)
+ fi (% Emply)
+ g (Sector*)
+ h (Mode)
* The omitted variable among the Sector dummies is Mining.
For each census year (i = 1992, 1997, 2002):
46
Summary of Results Under the Mixed Effects Model
Variables that become non-significant non-significant (with p-values)– (#pp)2 for 1992 (0.1159)– (#pp)3 for 2002 (0.3753)
47
Summary of Results Under the Mixed Effects Model
continued
%GDP (0.0282)(#pp) * (%GDP) (0.0103)(#pp)2 * (%GDP) (0.0087)
%Emply (0.0322)(#pp) * (%Emply ) (0.0117)(#pp)2 * (%Emply ) (0.0082)
Economic variables that are significantsignificant (all are for 2002)(all are for 2002) (with p-values)
48
What was happening in the U.S. economy in 2001-2003?
Economic recovery – beginning to emerge from a recession.
49
Economic Trends 1991-2003
-2.00
-1.00
0.00
1.00
2.00
3.00
4.00
5.00
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003
Year
Per
cent
Cha
nge
GDP Employment
50
What was happening in the U.S. economy in 2001-2003?
Economic recovery – beginning to emerge from a recession. GDP increasing sharply Employment declining, “bottoming out”
Unstable economy– Sharp opposite movements in GDP and
Employment
51
What is the nature of the interaction between economic
conditions and #pages? GDP and Employment effects tend
to be offsetting. GDP moderates the page effect on
response rates. Emply exacerbates the page effect on
response rates. Magnitude of Employment effect is
larger.
53
What is the nature of the interaction between economic
conditions and #pages? continued
Employment effect
Number of pages
Re
spo
nse
Ra
tes
54
Economic Effects Postscript: Economic Theory of Economic Theory of Lagged ExpectationsLagged Expectations
In-depth examination of effects of quarterly employment change on response rates– Statistically significant impact of changes
4 quarters and 3 quarters prior to beginning of economic census data collection.
55
Interpreting the Economic Effect
Unstable economy– Employment decline and GDP growth
Uncertainty– Riskier to divert resources to non-
productive activities
Cost-benefit– Survey response results in a cost with no
associated productive benefit.
56
Interpreting the Economic Effect continued
The bottom line– In a risky economic environment…– Employees are too valuable to devote to
filling out forms.– Forms that appear longer (#pages) are
presumed to take longer and be more costly to complete…
– Response rates are in jeopardy!
57
Summary:Number of Pages
Negative effect– #pages response rates – Cubic functional relationship
Negative effect of #pages on economic census response appears to have diminished over time
58
Summary:Economic Conditions
Evidence that economic conditions interact with form length to affect response rates– Mainly through changes in employment
Survey response = non-productive cost– Corroborates qualitative research findings
