Embed Size (px)
Citation preview
1
Understanding Event History Calendars(Update on Reengineering SIPP)
COPAFSMarch 5, 2010
David JohnsonJason Fields
US Census Bureau
2
The Evolution of a Phoenix
3
Congresswomen Maloney hails 25th Anniversary of SIPP, Census Bureau’s
Survey of Income and Participation "October, 2008 marks 25 years of SIPP data collection. The vital
data collected by career professionals at the Census Bureau
allows for the evaluation of the efficiency and effectiveness
of government programs and gives us a more robust picture
of how well we are doing as a nation in helping families
progress through tough economic challenges"
“The SIPP allows Congress to allocate scarce government
resources and save tax dollars. It’s fitting that during this
national economic crisis we draw attention to this important
diagnostic tool which helps us understand how we can best
provide assistance to families in need.”
4
The Unique Value of SIPP
• To provide a nationally representative sample for evaluating:– annual and sub-annual dynamics of income
– movements into and out of government transfer programs
– family and social context of individuals and households
– interactions between these items
5
National Academy of Sciences National Research Council
Committee on National Statistics
Panel ReportReengineering the Survey of Income and
Program Participation
Constance F. Citro and John Karl Scholz, Editors
July 2009
6
CNSTAT Report – Importance of SIPP
Conclusion 2-1:The Survey of Income and Program Participation is a
unique source of information for a representative sample of household members on the intrayear dynamics of income, employment, and program eligibility and participation, together with related demographic and socioeconomic characteristics. This information remains as vital today for evaluating and improving government programs addressed to social and economic needs of the U.S. population as it did when the survey began 25 years ago.
7
Our work on SIPP Improvements
• Improve Processing System and Collection Instrument
• Develop Event History Calendar (EHC) Instrument
• Examine use of administrative records data to supplement and evaluate survey data
• Continue meetings with stakeholders, development of survey content, and use of reimbursable supplements
8
Statements from the CNSTAT Report:
On EHC methodology• As discussed in Belli (1998), in an event history
calendar, “respondents are encouraged to consider various events that constitute their personal pasts as contained within broader thematic streams of events. Not only can respondents not the interrelationship of events within the same themes (top-down and sequential retrieval) but, depending on which themes are represented by the calendar, respondents can also not the interrelationships among events that exist with different themes (parallel retrieval).”
9
Timeline for SIPP Development
2008 paper EHC
2007 2008 2009 2010 2011 2012 Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan 2013
2009 SIPP Re-EngineeringInstrument Dev.
Processing andEvaluation
SIPP 2008 Panel – Waves 1 – 10 collection
Eval. Analysis
Systems Tests -Preparation
2009 Re-engineered SIPP automated
Prototype Reference Period
Field A
ctivities
2012/13 SIPP Re-EngineeringInstrument Refinement
Systems Tests -Preparation
2013 Reengineered SIPP Reference Period
Field A
ctivities
SIPP 2008 Panel – Waves 1 – 13 data release
Data GapSIPP 2004
2nd automated prototype
Reference Period
Field A
ct.
SIPP 2004 Panel data release
Waves 11 – 13
Re-engineered SIPP – Progress Update –
COPAFSMarch 5, 2010
Current SIPP BasicsNational panel survey – Since 1984 with sample size between
about 11,000 and 45,000 interviewed households
The duration of each panel varies from 2½ yrs to 4 yrs
The SIPP sample is a multistage-stratified sample of the U.S. civilian noninstitutionalized population
The survey uses a 4-month recall period – 3 interviews / year
The sample is divided into 4 rotation groups for monthly interviewing
Interviews are conducted by personal visit and by decentralized telephone
EHC Interviewing
Human Memory- structured/organized- links and associations
EHC Exploits Memory Structure- links between the occurrence and timing of events
EHC Encourages Active Assistance to Rs- flexible approach to help elicit an
autobiographical “story”
Evaluations of EHC Methods
Many EHC vs. “Q-List” Comparisons- various methods- in general: positive data quality results
BUT, Important Research Gaps- data quality for need-based programs?- comparison to 4-month reference period?
SIPP Re-engineering Field Test Plans
- Proof of concept test - - 2008 paper and pencil reinterview test
- EHC CAPI test- - 2010 Integrated Blaise and C# instrument prototype
- CAPI Revised test- - 2011 Test improvements to the wave 1 instrument, training,
and expand sample to all regional offices.
- - 2012 Test wave 2 concepts and instrument, examine movers and attrition issues, and refine training procedures.
2008 Paper Field Test Goals & Design
Basic Goal:Can an EHC interview collect data of comparablequality to standard SIPP?
- month-level data- one 12-month ref pd interview vs. three 4-month ref pd interviews- especially for need-based programs
Basic Design:EHC re-interview of SIPP sample HHsApproximately 2000 HHs in IL and TX
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • equivalent data quality•
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
SSI -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- no “main effect” for method (SIPP = EHC)
- no significant method difference in any month
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• Medicare; Social Security;
WIC (TX); Food Stamps (IL)
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• reduced EHC data quality, but • not due to longer recall period
•
Results Summary
15.0%
17.5%
20.0%
22.5%
25.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % …EHC % …
SOCIAL SECURITY -- % Covered in Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- significant “main effect” for method (SIPP > EHC)
- method difference is constant across months
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• Medicare; Social Security;
WIC (TX); Food Stamps (IL)
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only• Food Stamps (TX); TANF (TX);• employment; school enrollment
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only• EHC data quality may suffer due
– to longer recall period
Results Summary
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yes
EHC % yes
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports
Analysis Summary
- no significant “main effect” for method
- BUT significant variation by month --
JAN-MAY: SIPP > EHC
later months: no difference (reversal?)
0.0%
2.5%
5.0%
7.5%
10.0%
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
% Y
es
SIPP % yesEHC % yes
FOOD STAMPS (Texas Only) -- % Participation for Each Month of CY2007 According to the SIPP and EHC Reports and ADRECS
•3 Patterns:
• 1. EHC = SIPP All Year • SSI; WIC (IL)
• 2. EHC < SIPP All Year• Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only• Food Stamps (TX); TANF (TX);• employment; school enrollment
Results Summary
2008 Paper Field Test Overall Summary
Successful “Proof of Concept”
Overwhelming Finding: SIPP-EHC Agreement
Valuable Lessons to Inform Next Test- larger, broader sample- “correct” timing of field period- automated questionnaire
Specific Data Comparisons are Instructive
2010 EHC CAPI Test Goals & DesignBasic Goal:Can a CAPI Event History Calendar interview be:
- developed in-house (new/unknown requirements)- integrated with survey management systems- administered by regular field staff interviewers
Develop and test new processing system
Determine the comparability of data collected:- month-level data- (1) 12-month ref pd intvw vs. (3) 4-month ref pd intvws- especially for need-based programs
Basic Design:8000 HHs interviewed in 10 states limited to high poverty strata that
can be matched to areas in 2008 Panel SIPP data.Recording 400-500 of the completed interviews
2010 EHC CAPI Instrument
Current StatusInterviewing in 6 of 12 regional office areas (10 States)
Interviews continue through March 13.
Recording 400-500 of the completed interviews – transcription will begin shortly.
Reviewing and making changes to content and design.
Reviewing and revising training materials and methods
Planning for 2013 Production implementation
LANDMARKSRESIDENCES
PRESENCE OF PARENT - MOMPRESENCE OF PARENT - DAD
REFERENCE YEAR 20XX INTERVIEW YEAR 20XX+1
INTV
SCHOOL ENROLLMENT – (CURRENTLY ENROLLED fill)JOBS/BUSINESSES – (fill name of emp/bus from ehc as entered)+ Job/Bus 2 – (fill name of emp/bus from ehc as entered)+ Job/Bus 3 – (fill name of emp/bus from ehc as entered)+ Job/Bus 4 – (fill name of emp/bus from ehc as entered)+ Job/Bus 5 – (fill name of emp/bus from ehc as entered)ALL OTHER WORK FOR PAYTIME NOT WORKING
MARITAL STATUS – (CURRENT MARITAL STATUS fill)
SSI - SUPPLEMENTAL SECURITY INCOMESNAP - FOOD STAMPS/SUPPLEMENTAL NUTRITION ASSISTANCE PRGMTANF - TEMPORARY ASSISTANCE FOR NEEDY FAMILIESGA – GENERAL ASSISTANCE PROGRAM
WIC – WOMEN, INFANTS AND CHILDREN PROGRAM
HEALTH INSURANCE – PRIVATE COVERAGE+ ADDITIONAL PRIVATE COVERAGEMEDICARE COVERAGEMEDICAID COVERAGE – (fill interview state program name)MILITARY COVERAGE – (VA, CHAMPUS, CHAMPVA)
OTHER HEALTH INSURANCE COVERAGE
TIME WITHOUT HEALTH INSURANCE COVERAGE
Topic – (FILL TOPIC LABEL)* INTRO TO TOPIC QUESTION FILL – ASKS ABOUT STATUS NOW OR DIRECTS FR TO PROCEED [ ] (1) YES/PROCEED (2) NO (3) REFUSED
* FOLLOW-UP ASKS ABOUT EVER DURING REFERENCE YEAR OR DIRECTS FR TO PROCEED [ ] (1) YES/PROCEED (2) NO (3) REFUSED
* IDENTIFY NEW PERIOD OF TIME [ ] 0-NEW [F4 – EDIT SPELL] [CTRL-X DELETE SPELL]
* WHEN DID THIS PERIOD OF (TOPIC FILL) END? [ ] CHOOSE MONTH * WHEN DID THIS PERIOD OF (TOPIC FILL) BEGIN? [ ] CHOOSE MONTH
ANY MORE PERIODS OF (TOPIC FILL) DURING REFERENCE YEAR? [ ] (1) YES (2) NO
RECORDING ‘NO’ ADDS STRIKEOUT TO ANY UNASSIGNED MONTHS IN THIS TOPIC – ANSWERING ‘YES’ CYCLES BACK TO NEW PERIOD – UPON SPELL COMPLETION YOU EXIT THE DETAILED QUESTIONS TO THE ‘ANY MORE’ ITEM.
DEPENDENT DATA FILL UP TO LAST WAVES ‘INTV’ MONTH
FILL’S ARE USED TO LABEL INTERVIEW MONTH STATUS OR NAMES FROM JOBS/PROGRAM NAMES
(FILL NAME OF PERSON FOR THIS EHC)
Mockup (example of possible changes) for 2011 EHC CAPI Instrument
DATA ARE RECORDED UP TO THE MONTH OF INTERVIEW, WE ALLOW UP TO 6 MONTHS FROM THE END OF THE CALENDAR YEAR TO CONDUCT INTERVIEWS – MONTHS BEYOND THE INTERVIEW ARE GREY’ED OUT.
IF YES TO ‘NOW’ INITIALIZE FIRST PERIOD AND FILL INTERVIEW MONTH IN THE ‘TO’ BOX AND SET FOCUS ON THE ‘FROM’ BOX
Assessing Users’ Needs
URL: http://www.census.gov/sipp
