Embed Size (px)
DESCRIPTION
Citation preview
Slide 1
Linking Evaluation and Cost-Benefit Analysis
in Criminal Justice: A Practical Introduction
March 24, 2011
Elizabeth Drake, Senior Research Associate, Washington State Institute for Public Policy
Valerie Levshin, Policy Analyst, Cost-Benefit Analysis Unit, Vera Institute of Justice
Slide 2
Welcome
Linking Evaluation and Cost-Benefit Analysis
in Criminal Justice: A Practical Introduction
The webinar will start at 2pm.
Call 1-866-740-1260 for the audio portion of the
webinar.
Slide 3
Linking Evaluation and Cost-Benefit Analysis
Criminal Justice: A Practical Introduction
Valerie Levshin
Elizabeth Drake
Slide 4
The Cost-Benefit Knowledge Bank for Criminal Justice (CBKB) is a
project of the Vera Institute of Justice funded by the U.S.
Department of Justice’s Bureau of Justice Assistance.
• Website (cbkb.org)
• Cost-Benefit Analysis Toolkit
• Snapshots of CBA Literature
• Podcasts, Videocasts, and Webinars
• Roundtable Discussions
• Community of Practice
Slide 5
What You Will Learn Today
• Why evaluation is an important element of a cost-benefit analysis (CBA).
• Different ways to assess program/policy impacts.
• What meta-analysis is and how it can be used in
a CBA.
Slide 6
Today’s Agenda
Introduction and Housekeeping 5 minutes
The Role of Evaluation in CBA 5 minutes
Ways to Assess Program/Policy Impacts 10 minutes
Meta-Analysis and CBA 30 minutes
Q & A 10 minutes
Slide 7
Housekeeping items
Questions
Use the chat feature to send us your
questions at any time during the webinar.
We will address your questions after
each section of the presentation.
Slide 8
Housekeeping items
Webinar support and troubleshooting
Call: (800) 843-9166
Email: [email protected]
This webinar is being recorded
The recording and PowerPoint will be posted to cbkb.org
Slide 9
The Role of Evaluation in
Cost-Benefit Analysis
Slide 10
What is Cost-Benefit Analysis (CBA)?
• A tool to assess the pros and cons of policies and programs
• A method for finding out what will achieve the greatest net benefit to society
• An approach to policymaking
Slide 11
CBA in Five Steps
1. Determine the impact of the initiative
2. Determine whose perspectives matter
3. Measure costs
4. Measure benefits (in dollars)
5. Compare costs and benefits
Slide 12
Evaluation and CBA
• CBA is only possible if there is information about program/policy impact.
• If you don’t know the outcome, you can’t measure the benefits and some of the costs.
• You need to evaluate the initiative or draw on research to predict program/policy outcomes.
Slide 13
Ways to Assess Program and
Policy Impacts
Slide 14
Ways to Assess Program/Policy Impacts
• Evaluate the initiative
Comparison group design
Random assignment design
• Draw on evaluations of similar initiatives
Literature-based estimate
Meta-analysis
Slide 15
Comparison Group Design
Compare the outcomes for people in your program to a similar group of people not in your program
For example: matching groups, pre-post.
Need to match groups to make sure they’re similar.
Need to consider outside factors (such as the economy) that
could influence the outcome.
If the groups are similar, and other factors are ruled out, then
we can assume that the program affected the outcome.
Slide 16
Comparison Group Example
Example: CBA of Washington’s Drug Offender Sentencing Alternative (DOSA)
WSIPP’s evaluation compared the
recidivism rates before/after DOSA was implemented in 1999.
Evaluation results: DOSA reduced recidivism for drug offenders,
not drug-involved property offenders.
CBA results: benefits > costs for drug offenders.
Benefit / Cost ratio = $7.25 - $9.94 for drug offenders
Benefit / Cost ratio = $0.93 for drug-involved property offenders.
Slide 17
Random Assignment Design
Conduct a randomized assignment study to assess the
impact
Participants are randomly assigned into program or “control”
groups.
Best way to create very similar groups, where the only difference
between them is program participation.
Differences in outcomes can be attributed directly to the
program.
e.g., if program participants have lower recidivism rates that
non-participants, then we know the program reduced recidivism
rates.
Slide 18
Random Assignment Example
Example: CBA of the Center for
Employment Opportunity (CEO)
MDRC random assignment evaluation
showed that CEO reduced recidivism
rates.
CBA showed that recidivism reduction
generated taxpayer, victim and offender benefits.
Benefit / Cost ratio = about 3 to 1
Slide 19
Make a Literature-Based Estimate
• Review the literature to determine the impact of similar initiatives
• Example: CBA of Raising the Age in North Carolina
How will trying 16- and 17-yr-olds in the juvenile instead of the adult
system affect their recidivism rates?
6 studies show that the recidivism rates are 0%-50% lower than in
the adult system.
CBA assumes that trying youth in the juvenile system will reduce
recidivism rates by 10%.
Sensitivity analysis shows how using a different recidivism reduction
affects CBA results.
Slide 20
Meta-Analysis
• Review the literature to estimate the average effect of a program/policy on outcomes
• Review all evidence
• Give more weight to results of stronger evaluations
• Details are up next
Slide 21
Which Approach to Use?
• Depends on time, resources, staff expertise
• Aim for stronger research designs: the more rigorous the evaluation, the more accurate the CBA results
• Random assignment, comparison group evaluation are sometimes unfeasible; need to draw on the literature instead
Random assignment
Comparison group
Meta-analysis
Literature-based estimate
Increasing
difficulty,
time, and
expertise
Slide 22
Questions
Slide 23
Meta-Analysis and CBA
Slide 24
Given the Current Level of
Credible Research, What Don’t
We Know?
Benefits to
Crime Victims (of the reduction
in crime)
Benefits to
Taxpayers (of the reduction
in crime)
Costs (marginal program
cost, compared to
the cost of
alternative)
Benefits (total)
Minus
Costs
(per participant)
(2) (3) (4) (5)
Programs for People in the Adult Offender SystemVocational education in prison -9.0% (4) $8,114 $6,806 $1,182 $13,738Intensive supervision: treatment-oriented programs -16.7% (11) $9,318 $9,369 $7,124 $11,563General education in prison (basic education or post-secondary) -7.0% (17) $6,325 $5,306 $962 $10,669Cognitive-behavioral therapy in prison or community -6.3% (25) $5,658 $4,746 $105 $10,299Drug treatment in community -9.3% (6) $5,133 $5,495 $574 $10,054Correctional industries in prison -5.9% (4) $5,360 $4,496 $417 $9,439Drug treatment in prison (therapeutic communities or outpatient) -5.7% (20) $5,133 $4,306 $1,604 $7,835Adult drug courts -8.0% (57) $4,395 $4,705 $4,333 $4,767Employment and job training in the community -4.3% (16) $2,373 $2,386 $400 $4,359Electronic monitoring to offset jail time 0% (9) $0 $0 -$870 $870Sex offender treatment in prison with aftercare -7.0% (6) $6,442 $2,885 $12,585 -$3,258Intensive supervision: surveillance-oriented programs 0% (23) $0 $0 $3,747 -$3,747Washington's Dangerously Mentally Ill Offender program -20.0% (1) $18,020 $15,116 n/e n/eDrug treatment in jail -4.5% (9) $2,481 $2,656 n/e n/eAdult boot camps 0% (22) $0 $0 n/e n/eDomestic violence education/cognitive-behavioral treatment 0% (9) $0 $0 n/e n/eJail diversion for mentally ill offenders 0% (11) $0 $0 n/e n/eLife Skills education programs for adults 0% (4) $0 $0 n/e n/e
Programs for Youth in the Juvenile Offender SystemMultidimensional Treatment Foster Care (v. regular group care) -22.0% (3) $51,828 $32,915 $6,945 $77,798Adolescent Diversion Project (for lower risk offenders) -19.9% (6) $24,328 $18,208 $1,913 $40,623Family Integrated Transitions -13.0% (1) $30,708 $19,502 $9,665 $40,545Functional Family Therapy on probation -15.9% (7) $19,529 $14,617 $2,325 $31,821Multisystemic Therapy -10.5% (10) $12,855 $9,622 $4,264 $18,213Aggression Replacement Training -7.3% (4) $8,897 $6,659 $897 $14,660Teen courts -11.1% (5) $5,907 $4,238 $936 $9,208Juvenile boot camp to offset institution time 0% (14) $0 $0 -$8,077 $8,077Sex offender cognitive-behavioral treatment -10.2% (5) $32,515 $8,377 $33,064 $7,829Restorative justice for low-risk offenders -8.7% (21) $4,628 $3,320 $880 $7,067Interagency coordination programs -2.5% (15) $3,084 $2,308 $205 $5,186Juvenile drug courts -3.5% (15) $4,232 $3,167 $2,777 $4,622Regular surveillance-oriented parole (v. no parole supervision) 0% (2) $0 $0 $1,201 -$1,201Juvenile intensive probation supervision programs 0% (3) $0 $0 $1,598 -$1,598Juvenile wilderness challenge 0% (9) $0 $0 $3,085 -$3,085Juvenile intensive parole supervision 0% (10) $0 $0 $6,460 -$6,460Scared Straight +6.8% (10) -$8,355 -$6,253 $58 -$14,667Counseling/psychotherapy for juvenile offenders -18.9% (6) $23,126 $17,309 n/e n/eJuvenile education programs -17.5% (3) $41,181 $26,153 n/e n/eOther family-based therapy programs -12.2% (12) $15,006 $11,231 n/e n/eTeam Child -10.9% (2) $5,759 $4,131 n/e n/eJuvenile behavior modification -8.2% (4) $19,271 $12,238 n/e n/eLife skills education programs for juvenile offenders -2.7% (3) $6,441 $4,091 n/e n/eDiversion progs. with services (v. regular juvenile court) -2.7% (20) $1,441 $1,034 n/e n/eJuvenile cognitive-behavioral treatment -2.5% (8) $3,123 $2,337 n/e n/eCourt supervision vs. simple release without services 0% (8) $0 $0 n/e n/eDiversion programs with services (v. simple release) 0% (7) $0 $0 n/e n/eJuvenile intensive probation (as alternative to incarceration) 0% (5) $0 $0 n/e n/eGuided Group Interaction 0% (4) $0 $0 n/e n/e
Prevention Programs (crime reduction effects only)Nurse Family Partnership-Mothers -56.2% (1) $11,531 $8,161 $5,409 $14,283Nurse Family Partnership-Children -16.4% (1) $8,632 $4,922 $733 $12,822Pre-K education for low income 3 & 4 year olds -14.2% (8) $8,145 $4,644 $593 $12,196Seattle Social Development Project -18.6% (1) $1,605 $4,341 n/e n/eHigh school graduation -10.4% (1) $1,738 $2,851 n/e n/eGuiding Good Choices -9.1% (1) $570 $2,092 n/e n/eParent-Child Interaction Therapy -3.7% (1) $268 $784 n/e n/e
Program types in need of additional research & development before we can conclude they do or do not reduce crime outcomes:
CommentCase management in the community for drug offenders 0% (13) Findings are mixed for this broad grouping of programs.
COSA (Faith-based supervision of sex offenders) -22.3% (1) Too few evaluations to date.
Day fines (compared to standard probation) 0% (1) Too few evaluations to date.
Domestic violence courts 0% (2) Too few evaluations to date.
Faith-based programs 0% (5) Too few evaluations to date.
Intensive supervision of sex offenders in the community 0% (4) Findings are mixed for this broad grouping of programs.
Medical treatment of sex offenders -21.4% (1) Too few evaluations to date.
Mixed treatment of sex offenders in the community 0% (2) Too few evaluations to date.
Regular parole supervision vs. no parole supervision 0% (1) Too few evaluations to date.
Restorative justice programs for lower risk adult offenders 0% (6) Findings are mixed for this broad grouping of programs.
Therapeutic community programs for mentally ill offenders -20.8% (2) Too few evaluations to date.
Work release programs (from prison) -4.3% (4) Too few recent evaluations.
Dialectical Behavior Therapy 0% (1) Too few evaluations to date.
Increased drug testing (on parole) vs. minimal drug testing 0% (1) Too few evaluations to date.
Juvenile curfews 0% (1) Too few evaluations to date.
Juvenile day reporting 0% (2) Too few evaluations to date.
Juvenile jobs programs 0% (3) Too few recent evaluations.
Juvenile therapeutic communities 0% (1) Too few evaluations to date.
Mentoring in juvenile justice 0% (1) Too few evaluations to date.
Programs needing more research for youth in the juvenile offender system
Effect on Crime
Outcomes Percent change in crime
outcomes, & the number of
evidence-based studies on
which the estimate is based
(in parentheses)
(1)
Programs needing more research for people in the adult offender system
Exhibit 4
Reducing Crime With Evidence-Based Options: What Works, and Benefits & Costs
Washington State Institute for Public Policy
Estimates as of October, 2006
.
Notes:
"n/e" means not estimated at this time.
Prevention program costs are partial program costs, pro-rated to
match crime outcomes.
Benefits and Costs(Per Participant, Net Present Value, 2006 Dollars) WSIPP
“Consumer Reports” Lists:
What Works?
What Doesn’t?
What Can Give Washington
Taxpayers a Good Return
(Better Outcomes) for Their Money?
Washington legislature has asked WSIPP this question:
Are There Evidence-Based Policy Options That Improve Public Outcomes,
but at Less Cost?
Washington State Institute for Public Policy (WSIPP)
Slide 25
What I’ll cover today
• Meta-analytical concepts Research design quality
and discount factors
• Cost-benefit concepts Resources we monetize
What I won’t cover today
• Cost-benefit data and
calculations
• Meta-analytic calculations
Meta-Analysis and CBA: Overview
Slide 26
Meta-Analysis
Slide 27
Empirical Summarization of a Set of Literature
• Meta-analysis produces an average effect on something.
• The unit of measurement is an effect size, which measures the
degree to which a program has been shown to change an
outcome for program participants relative to a comparison
group.
• Not all research is of equal quality, and this greatly influences
the confidence that can be placed in the results of a study.
• A meta-analysis is only as good as the selection and coding
criteria used to conduct the study.
What is Meta-Analysis?
Slide 28
1. Search Criteria (published and unpublished sources)
2. Comparison group studies
• no single group, pre/post research designs
3. Intent-to-treat sampling procedures
• Completers only = bias treatment effect
4. Crime outcomes
• Prefer dichotomous outcomes
• Longest follow-up period
• Felony convictions
Meta-Analytic Procedures: Important Criteria to Determine Inclusion of Studies
Slide 29
An adjustment factor is assigned to the results of individual
effect sizes based on our judgment concerning the research
design quality.
Rating Research Design Quality Discount
Multiplier
5 Random assignment None
4 Random assignment with issues .75
3 Studies that attempt to statistically control for
un observed factors (e.g., regression
discontinuity or natural experiment)
.75
2 Well done comparison group study with many
controls
.625
1 Less well implemented with some covariates .5
Meta-Analytic Procedures: Standards of Rigor and Adjustments to Effect Sizes
Slide 30
Discount for:
• Not “Real World” evaluations
• Lab settings
• Evaluation researcher is the program developer
• Weak outcome measure (i.e., incarceration)
Internally consistent set of procedures.
Meta-Analytic Procedures: Standards of Rigor and Adjustments to Effect Sizes
Slide 31
Questions
Slide 32
Meta-Analysis and
Cost-Benefit
Slide 33
Once we have an effect size, how much does it cost to buy
that effect size, and what’s it worth to achieve it?
Effect size Relative to base
population
CJS resource
response and
victimizations
incurred
Cost-benefit
results
Cost-Benefit Procedures
Slide 34
Adult Offenders
(Draft 2010 Results)
Change In
Crime (# of EB Studies)
Benefits Minus Costs,
per-person, life cycle (Probability: you lose $)
Aggression Repl. Trng (wf) -9% (4) $12,900 (<1%) Functional Family Thpy (wf) -14% (8) $23,000 (<1%)
Juvenile Offenders
Results: What Works to Reduce Crime?
Correctional Education -5% (13) $7,700 (<1%) Cog-Behavioral Treatment -4% (27) $7,100 (<1%)
ISP: surveillance -2% (23) -$2,900 (≈53%)
ISP: treatment -18% (11) $6,200 (≈13%)
Slide 35
Results, results, and more results.
1. Meta-analysis and cost-benefit analysis can inform
stakeholders where resources are best utilized
2. But results will only tell you the average effect
3. Follow up with an outcome evaluation to ensure you
are getting the results you expect
An example in Washington.
Pulling It All Together
Slide 36
Questions
Slide 37
Wrap-Up
Slide 38
Recap of Today’s Webinar
You learned:
• Why evaluation is an important element of a cost-benefit analysis (CBA).
• Different ways to assess program/policy impacts.
• How meta-analysis can be used in a CBA.
Slide 39
Follow-up
Please complete the evaluation form as you leave this training.
To receive information and notifications about upcoming webinars and other events
• Visit the Cost-Benefit Knowledge Bank for Criminal Justice at http://cbkb.org.
• Subscribe to receive updates from CBKB.
• Follow us on Twitter at http://www.twitter.com/CBKBank.
The next webinar will focus on discussing cost-benefit results with the media. Stay tuned for updates.
Slide 40
This project is supported by Grant No. 2009-MU-BX K029 awarded by the Bureau of
Justice Assistance. The Bureau of Justice Assistance is a component of the Office of
Justice Programs, which also includes the Bureau of Justice Statistics, the National
Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, and
the Office of Sex Offender Sentencing, Monitoring, Apprehending, Registering, and
Tracking. Points of view or opinions in this document are those of the author and do
not represent the official position or policies of the United States Department of
Justice.
Slide 41
Contact Information
Elizabeth Drake
(360) 586-2767
Valerie Levshin
(212) 376-3062
http://www.cbkb.org
Slide 42
Thank you!
