Embed Size (px)
Citation preview
Aggregate and Systemic Components of Risk
in Total Survey Error Models
John L. EltingeU.S. Bureau of Labor Statistics
International Total Survey Error Workshop
June 20, 2011
Acknowledgements and Disclaimer
The author thanks Ted Chang, Moon Jung Cho, Mike Davern, John Dixon, Malay Ghosh, Jeffrey Gonzalez, Rachel Harter, Bill Iwig, Mike Horrigan, Brandon Kopp, Phil Kott, Bill Mockovak, Danny Pfeffermann, Jay Ryan, George Stamas and Michael Sverchkov for many helpful discussions of the topics considered in this paper.
The views expressed here are those of the author and do not necessarily reflect the policies of the U.S. Bureau of Labor Statistics.
2
Overview
I. Survey Risk, Data Quality and Total Survey Error Models
II. Aggregate and Systemic Components of Risk
III. Impact of, and Recovery from, Systemic Errors
IV. Adjusting Design Features to Account for Systemic Error Components
3
I. Survey Risk, Data Quality and Total Survey Error Models
A. Survey Risk
1. Degradation of one or more dimensions of survey quality
2. Six dimensions from Brackstone (1999, Survey Methodology)
Accuracy, Timeliness, Relevance, Interpretability, Accessibility, Coherence
4
I. Survey Risk, Data Quality and Total Survey Error Models (Continued)
B. Total Survey Error Model: Detailed decomposition of the “accuracy” component of quality
(Estimator) – (True value) = (frame error)+ (sampling error)+ (incomplete data effects)+ (measurement error)+ (processing effects)
5
II. Aggregate and Systemic Components of Survey Risk
A. Types of Survey Risk: For degradation in any component of data quality (and especially for TSE components), distinguish between aggregate risk and systemic risk
1. Aggregate risk: Combined effects of a large
number of (infinitesimal) random events
a. Superpopulation effects
b. Local imperfections in frame, listing operationc. Selection mechanismd. Incomplete datae. Measurement error 6
II. Aggregate and Systemic Components of Survey Risk
(Continued)
2. Systemic risk: Large shocks with widespread impact on our data collection and estimation work
Ex: Change in administrative record system used to produce frame, auxiliary data
Ex: Change in field management; training problems
Ex: Errors in programming collection instrument
7
II. Aggregate and Systemic Components of Survey Risk
(Continued)
Additional Examples:
Ex: Lack of fit in implicit or explicit models used for imputation, measurement error adjustments
Ex: Failure to identify important sources of nonsampling error in preliminary lab studies, pilot tests, field operations
Ex: Problems in implementation of rules for edit, imputation, allocation
8
II. Aggregate and Systemic Components of Survey Risk
(Continued)
B. Historical Pattern:
1. Research community has focused on characterization, modeling and
management of aggregate risk
2. Survey managers often worry as much, or more, about systemic risk
9
A. Origins of Systemic Errors:
Mixed linear or hierarchical Bayes models
Note especially that many systemic components riskill affect many (or all) data collected
II. Aggregate and Systemic Components of Survey Risk (Continued)
C. Potential models for systemic risk:
1. Generalized mixed linear models, e.g.,
Y_ij = X_ij A + Z_ij B + u_i + e_ij
X: Design variables
Z: Observed paradata
A, B: Coefficient vectors
u_i: Regional or other coarse random effects
e_ij: Deviation specific to unit j in region I
Reflect systemic effects in coefficients A, B or coarse random effect u_i 10
II. Aggregate and Systemic Components of Survey Risk (Continued)
2. Hierarchical Bayes models
3. Life-testing models for identification of software problems
4. For all potential models, issues with
a. Model identification information
b. Limited effective sample sizes
11
III. Impact of, and Recovery from, Systemic Errors
A. Four Possible Outcomes from Systemic Error
1. Perfect resilience
2. Substantial degradation in quality with slow recovery
3. Moderate degradation, again with slow recovery
4. Substantial degradation with rapid recovery (cf. literature on “recoverable computing”)
12
Perfect Resilience
0
25
50
75
100
Time
Pe
rfo
rma
nc
e
X
X
Not Robust
0
25
50
75
100
Time
Pe
rfo
rma
nc
e
X
X
Limited Maximum Risk
0
25
50
75
100
Time
Pe
rfo
rma
nc
e
X
X
Rapid Recovery
0
25
50
75
100
Time
Per
form
ance
X
X
III. Impact of, and Recovery from, Systemic Errors
(Continued)
B. Features of outcomes in the four graphs arising from:
1. Population characteristics (e.g., decay rate of systemic shocks)
2. Survey design (design features to buffer systemic shocks)
3. Ad hoc direct intervention14
IV. Adjusting Design Features to Account for Systemic Error Components
A. Potential Goals:
1. Prevent systemic errors from arising
2. Limit the effects of systemic errors in current production sample (cf. discussion
of weighting methods that are robust against model misspecification)
3. Timely identification of, and direct adjustment for, systemic errors (cf. responsive or multi-phase designs)
15
IV. Adjusting Design Features to Account for Systemic Error Components (Continued)
B. Tools to Prevent, or Reduce the Impact of, Systemic Errors
(cf. “The Three Es” of Risk Management)
1. Enforcement- Formal management directives- Incentive structures in
organization 16
IV. Adjusting Design Features to Account for Systemic Error Components (Continued)
2. Education: Training of methodologists, managers, field personnel
3. Engineering: Modification of Methodological and Technological Features of Survey Processes
17
V. Closing Remarks
A. Survey Risk: Likelihood and impact of failure in one or more dimensions of survey quality, including TSE components
B. Important for methodologists to study both aggregate and systemic components of risk
C. Consider design work to identify and ameliorate systemic components
18
V. Closing Remarks (Continued)
D. Discussion Questions
1. Have you (or your survey organization) encountered forms of systemic errors?
2. If so:
a. What are their dominant features?
b. What models have you used to describe the errors, and their effect on data quality?
c. What changes in design or estimation methods have you used to ameliorate their effects?
19
Contact Information
John L. EltingeAssociate Commissioner
Office of Survey Methods Researchwww.bls.gov
