Upload
giovanna-golden
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
WRECMay 29, 2014Washington, D.C.
Learning “What Works” in Career Pathways Programming: The ISIS Evaluation
. David JudkinsAbt Associates, Inc
Abstract
2
Introductory review of new methodology for the study of the mediating pathways in the context of randomized experiments of social interventions– Pathways may contain both exposure to program
components and early outcomes of that exposure– Other talks in this session talk about randomizing to
program components– This talk is about how to analyze when randomization
of components is not selected as the evaluation methodology
Plans to apply this new methodology to ISIS
What is ISIS? Random assignment evaluation of nine “career pathways”
programs– Impact, implementation, cost-benefit studies
Evaluation funded by ACF; Abt Associates heading research team– Additional funding for program enhancements/scale up from
Open Society Foundations, Joyce Foundation, Kresge Foundation, Meadows Foundation, OFA/ACF (3 HPOG sites)
RA still underway; early impact results in 2-3 years Programs vary, but all promote access to and completion of
post-secondary education, targeting low-income, low-skilled adults and youth
Conducting nine separate studies– Overlap in research questions, measurement, analysis plans
3
Core Ideas of Career Pathways Programs
Address the wide range of skill and other needs of economically disadvantaged participants
Key “inputs” or elements: Assessment (academic and non-academic); basic and vocational skills training; supports; employment connections
Create manageable, well-articulated training steps Provide credentials valued in high demand
occupations/sectors Build effective partnerships
– Community colleges, employers, CBOs, WIBs
4
Challenge: Getting inside the “Black Box”
How to determine more effective program components?– Counseling– Tutoring– Financial assistance– Internships
How to determine intermediate outcomes on vital causal paths?– Self-efficacy– Strength of social network– Basic numeracy and literacy skills– Stress management skills
5
Benefits
Effective components are built into future interventions Ineffective components are dropped New components are invented to impact vital early
outcomes even more strongly Future evaluation findings can be obtained more quickly
if good early indicators are available
6
Familiar but Inadequate
Baron and Kenny (1986) mediation triangle
The indirect effect of treatment on Y mediated by M estimated as ab.
The direct effect of treatment on Y is d.
7
1 1
2 2
i i i
i i i i
M aT e
Y dT bM e
Why Do We Need Better Methodology?
Baron and Kenny assume independence of errors in two equations.
This will be violated if there are any common causes of the two errors such as:– Measured baseline characteristics – Unmeasured baseline characteristics– Other measured mediators– Other unmeasured mediators
Also, Baron and Kenny does not generalize well to categorical outcomes
8
Pearl’s Method
Judea Pearl has developed a much more powerful and general method.
However, he developed it in a graphical framework unnatural to statisticians in the Neyman-Rubin tradition.
Recently, Kosuke Imai and co-authors have recast it in the language and traditions of Neyman-Rubin.
Dramatically increased popularity Still very difficult literature to penetrate Sketch some features today We plan to use on ISIS
9
Pearl’s Method Capabilities
10
Challenge Pearl can Handle
Confounding due to measured moderators
Confounding due to other measured mediators
Confounding due to unmeasured moderators
Confounding due to unmeasured other mediators
Nonnormal outcomes such as binary outcomes
ISIS Nomenclature
Blend of nomenclatures proposed by Pearl and by Imai and coworkers.
Key is the concept of forcing an environmental stimulus while “blocking” one type of response to the stimulus
Three potential outcomes per personY0: person randomly assigned to controlY1: person randomly assigned to treatment and no changes are blockedY2: person randomly assigned to treatment but change in mediator M is “blocked”
11
Three Potential Outcomes
12
Example
Y0 = person’s degree attainment at 36 months if assigned to control
Y1 = person’s degree attainment at 36 months if assigned to treatment and no “natural” changes are “blocked”
Y2 = person’s degree attainment at 36 months if assigned to treatment but somehow we blocked change to self-confidence
13
Potential Outcomes Framework for Mediation
Total effect = average of Y1-Y0 Indirect effect mediated by M = average of Y1-Y2 Direct effect = average of Y2-Y0 Then total = indirect + direct Continuing example:
– Y1-Y0 = total effect of treatment on degree attainment– Y1-Y2 = indirect effect of treatment on degree attainment
via boosted self-confidence– Y2-Y0 = direct effect of treatment on degree attainment
Note: no need to reference linear models to define the estimands
14
Mediating Pathways
In ISIS, we are interested in multiple mediating pathways– Trying to decide which are the vital pathways for successor
programs to emulate Multiple mediation can take two forms:
– Serial (T causes M1 causes M2 causes Y)– Parallel (T causes both M1 and M2, both of which jointly
cause Y) Pearl’s framework flexible enough to handle
simultaneous parallel and serial mediation
15
Service Exposure and Early Outcomes
Both are mediators Let D1 and D2 be exposure levels to two program
components Let W1 and W2 be two early outcomes Treatment causes D1 and D2 to change, which causes W1 and
W2 to change (in addition to direct effects of T on them) All of which lead to changes in Y
16
Example Graph for Parallel and Serial Mediation
17
T
D1
D2
W1
W2
YX
Estimation Process
Build a series of models for every mediator and outcome in the system in terms of causally prior variables
Draw simulated values from system for each pathway of interest– What happens if some services are blocked and
some early outcomes are blocked Pathways with highest simulated Y values are the
most promising for future program developers to consider
18
Assumptions
Strong assumptions required Considerable debate about best way to express
them Oversimplified version
– Variation in M within each level of Y– No unmeasured prime joint causes of M and Y– This includes post-randomization latent outcomes
19
Example Violation
M = Self-confidence Y = Degree attainment Academic skill at unmeasured intermediate point
mediates treatment effects on both self-confidence and degree attainment
Easy to come to the mistaken conclusion that shortcut methods work that build self-confidence through means other than skill improvement
Need to measure skill to prevent this incorrect mediational inference
20
Challenges
Curse of dimensionality Need to identify graphs to be tested that correspond
to program-specific theories of change Also need to believe that we have measured the
necessary set of variables to unconfound the mediator(s) and outcome(s) in the graph of interest
21
Additional Information
David JudkinsPrincipal ScientistAbt [email protected]
Websitewww.projectisis.org
Brendan KellyFederal Co-Project Officer, ISISACF/[email protected]
Molly IrwinFederal Co-Project Officer, ISISACF/[email protected]