Upload
rian
View
31
Download
0
Embed Size (px)
DESCRIPTION
The SIPP Event History Calendar Field Test: Analysis Plans and Preliminary Report. Jeff Moore Statistical Research Division, U.S. Census Bureau Jason Fields Housing and Household Economic Statistics Division, U.S. Census Bureau ASA/SRM SIPP Working Group Meeting September 16, 2008. - PowerPoint PPT Presentation
Citation preview
The SIPP Event History Calendar Field Test:
Analysis Plans and Preliminary Report Jeff Moore
Statistical Research Division, U.S. Census Bureau
Jason FieldsHousing and Household Economic Statistics Division, U.S. Census Bureau
ASA/SRM SIPP Working Group MeetingSeptember 16, 2008
Overview Background:
- SIPP “re-engineering”- event history calendar (EHC) methods
Goals and Design of the EHC Field Test
Evaluation Plans
Preliminary Results [not yet available]
SIPP Re-EngineeringImplement Improvements to SIPP
- reduce costs- reduce burden- improve processing system- modernize instrument- expand/enhance use of admin
records
Key Design Change: Annual Interviewing
EHC Interviewing (1)
Human Memory- structured/organized- links and associations
EHC Exploits Memory Structure- links between to-be-recalled events- coherence, consistency, sequence
EHC Encourages Active Assistance to Rs
EHC Interviewing (2)
Evaluation: EHC vs. Q-List Comparisons- various methods- in general: positive data quality results
BUT, Important Research Gaps- data quality for need-based programs?- extended reference period?
Field Test Goals & DesignBasic Goal:Can an EHC interview collect data of comparable(or better) quality than standard SIPP?
- month-level data- one 12-month ref pd interview vs. three 4-month ref pd interviews- especially for need-based programs
Basic Design:EHC re-interview of SIPP sample HHs
Design Details (1)
Main Sample:SIPP Wave 10-11-12 Interview Cases
- reported on CY-2007 via SIPP [Fig. 1]
Supplemental Sample:SIPP Wave 8 Sample Cut Cases
- dropped from SIPP in 2006; “unprimed”
EHC Re-Interview in 2008, about CY-2007
Design Details (2)
Two Sites- Illinois (all)- Texas (4 metro areas)
N = 1,945 Addresses- cooperating HHs in SIPP
Sample Distribution:ILLINOIS (n = 914) TEXAS (n = 1,031)
W10-11-12 Sample Cut W-10-11-12 Sample Cut
487 427 609 422
Design Details (3)
Administrative Records(for some characteristics, and with R approval)
- Medicare- Social Security retirement, disability- SSI- TANF- Food Stamps- [Medicaid?]
Design Details (4)
EHC Questionnaire [handout]- paper-and-pencil- 12-month, CY-2007 reference period- selected SIPP topics (“domains”)- start with landmark events- within domains, anchor on “now”- month-level (at least) detail
Sample of Addresses, Not People- post-interview clerical match to SIPP
Design Details (5)
$40 Incentive, Non-Contingent
Same Response Rules as SIPP- EHC interview for all adults (15+)- self-response preferred (proxy permitted)
Field Staff: Census Bureau FRs- most with some interview experience- ~1/3 with SIPP experience- 3-day training on EHC methods
Design Details (6a)
Field Period: Mid-April thru Late June 2008
Outcomes:
- 1,627 HH interviews
- 3,318 individual EHC interviews
- 2,747 EHC Rs matched to SIPP
Design Details (6b)
ILLINOIS TEXAS TOTAL
W-10-11-12
Sample Cut
W-10-11-12
Sample Cut
W-10-11-12
Sample Cut
Address Sample(1,945) 487 427 609 422 1,096 849
Completed HH Interviews
(1,627)
417(91%)
347(89%)
518(91%)
345(92%)
935(91%)
692(91%)
Completed Individual EHC Interviews
(3,318)
866(99%)
707(98%)
1,056(99%)
689(99%)
1,922(99%)
1,396(99%)
Interviewed Adults Matched to SIPP
(2,747)
767(89%)
588(83%)
890(84%)
502(73%)
1,657(86%)
1,090(78%)
Evaluation Plans (1)
Compare SIPP and EHC Survey Reports - same people- same time period- same characteristics
Data Quality Comparison using Admin Records(later)
Evaluation of “Priming” Bias
Evaluation Plans (2)
Other Evaluations- R debriefing form- FR “case report” debriefing form- FR debriefing focus groups- interview observations
Focus on EHC Interview Process
Compare SIPP/EHC Reports (1a)
SIPP Report
No Yes
EHC No a bReport Yes c d
2x2 Consistency Table for “Participation”(Employed? Enrolled? Insured? etc.)
- for each characteristic- for each month of CY-2007- unweighted / unedited data
Compare SIPP/EHC Reports (1b)
SIPP Report
No Yes
EHC No a bReport Yes c d
b=c equivalent data quality (high if (b+c)/N~0; low if (b+c)/N is large)
b>c EHC “underreporting” (rel. to SIPP)
b<c SIPP “underreporting” (rel. to EHC)
Compare SIPP/EHC Reports (1c)
SIPP Report
No Yes
EHC No a bReport Yes c d
Patterns of Consistency/Inconsistency- b>c for most months? b<c? mixed?- early months vs. late months?
Compare SIPP/EHC Reports (2a)
Total Reported Months of “Participation”- by Qtr / combined Qtrs / whole year
SIPP Participation Months – Q(n)
0 1 2 3
EHCParticipation
Months – Q(n)
0
1
2
3
Compare SIPP/EHC Reports (2b)
Patterns of Off-Diag Clustering Across Time- above for most Qtrs? below? mixed?- early Qtrs vs. late Qtrs?
SIPP Participation Months – Q(n)
0 1 2 3
EHCParticipation
Months – Q(n)
0
1
2
3
Compare SIPP/EHC Reports (2c)
Patterns of Off-Diag Clustering Across Time- above for most Qtrs? below? mixed?- early Qtrs vs. late Qtrs?
# Reporting At Least 1 Month of Participation
SIPP Participation Months – Q(n)
0 1+
EHCParticipation
Months – Q(n)
0
1+
Compare SIPP/EHC Reports (3)
Other “Participation” Comparisons:
- ANY need-based program participation? (by month / Qtr / combined Qtrs / year)
or- ANY health insurance coverage
[etc.]
- alignment/sequencing across domains (e.g., moves & jobs, employment & health insurance, etc.)
Compare SIPP/EHC Reports (4a)
Month-to-Month Transitions (yesno; noyes)
SIPP’s Staggered Interview Design:- each month-pair is a “seam” for ¼ sample- each month-pair is off-seam for ¾ sample
Compare Reporting of Transitions
Jan-Feb
Feb-Mar
Mar-Apr
Apr-May
May-Jun
Jun-Jul
Jul-Aug
Aug-Sep
SIPP – Seam Cases
SIPP – Off-Seam Cases
EHC
Compare SIPP/EHC Reports (4b)
Seam Bias:- too much Δ across interview “seams”- too little Δ within a single interview
EHC Δ rates below SIPP’s (seam), and above SIPP’s (off-seam) Improved Quality
Jan-Feb
Feb-Mar
Mar-Apr
Apr-May
May-Jun
Jun-Jul
Jul-Aug
Aug-Sep
SIPP – Seam Cases (++)
SIPP – Off-Seam Cases (- -)
EHC (0)
Compare SIPP/EHC Reports (5a)
Income Amount Report Comparisons- unemployment benefits- disability income ($)- workers’ comp- Social Security ($)- Medicare Part B deduction ($)- TANF ($)- Food Stamps ($)- SSI ($)
($)=admin records
Compare SIPP/EHC Reports (5b)
$$ Comparison is Less Straightforward
Continuous $$ Variable - arbitrary definition(s) of “agreement”- disagreements are directional
Limited to “Yes/Yes” Cases
Compare SIPP/EHC Reports (5c)
$$ Reporting Comparisons- mean amount (EHC; SIPP; difference)
- levels of correspondence(e.g., ±5%; ±5-10%; ±10-25%; ±25-50%; >±50%)
- direction of differences($EHC > $SIPP; $EHC=$SIPP (±1%); $EHC < $SIPP)
- timing of amount changes
Assessment of “Priming” (6a)
W-10-11-12 Rs Provide CY-2007 Data Twice- first SIPP, then EHC
Are Their EHC Reports Biased?- e.g., more accurate EHC response- could bias field test interpretation
Control Group: W-8 Sample Cut- last SIPP response in Jun-Sep 2006- “unprimed” re: CY-2007 (not SIPP content)
Assessment of “Priming” (6b)
Compare Distributions for Key Characteristics- e.g., monthly “participation” reports- weighted (sub-sampling; attrition)
Similarity of Profiles Extent/Nature of Priming Bias
Admin Records for Some Characteristics- meaning of distribution differences- may also reveal hidden quality diffs
Guidance, Questions, Advice…Questions?
Thoughts/Comments...?- on the evaluation approach?
- about additional analyses?
- about how to weigh evidence from the field test in deciding whether or not to adopt a 12-month EHC?
Thank you very much!