What Works: Evidence-Based Practice in Juvenile Delinquency Dispositions

James C. (Buddy) HowellCo-Director, North Carolina Evidence-Based

Juvenile Justice Project

Institute of Government UNC Chapel HillMarch 29, 2006

Ways to Use Evaluation Research to Improve Program

Practice• Evaluate each program to provide feedback

on implementation and outcomes.

• Implement a “model” program that evaluation has shown to be effective; monitor implementation compliance.

• Follow “best practices” guidelines for effective programs that have been derived from evaluation research; monitor compliance.

An Introduction to Evidence-Based Practice

What do the following terms mean?

Evidence-based programsResearch-based programsBest practices

They all mean the same thing: Programs that are based on scientifically sound research that shows what programs or specific services effectively reduce problem behaviors.

NC’s Comprehensive Strategy for Juvenile Delinquency

Problem Behavior > Noncriminal Misbehavior > Delinquency > Serious, Violent, and Chronic Offending

PreventionTarget Population: At-Risk Youth

Preventing youth from becoming

delinquent by focusing prevention

programs on at-risk youth

Graduated SanctionsTarget Population: Delinquent Youth

Improving the juvenile justice

system response to delinquent

offenders through a system of

graduated sanctions and a

continuum of treatment

alternatives

> > > > > >Programs for All Youth

Programs for Youth at Greatest Risk

Immediate Intervention

Intermediate Sanctions

Community Confinement

Training Schools

Aftercare

The Scope of North Carolina’sJuvenile Delinquency Problem

Because only a small fraction of adjudicated youths in NC are serious, violent, and chronic offenders (see next slide), the delinquency problem is quite manageable.

• We must use structured decision making tools,

• To build a continuum of effective programs

• With evidence-based services.

Non-Serious Non-ViolentNon-Chronic62%

Serious or Violent

28%

Chronic

10%

Violent

2%

C & V0.5%

Source: NC DJJDP Risk Assessment Data FY01-02 to FY 02-03 (N=17,645)

Juvenile Offender Court CareersNorth Carolina

AUTHORITY CONFLICT PATHWAY(before age 12)

OVERT PATHWAY COVERT PATHWAY(before age 15)

AGE OF ONSET:LATE

EARLY

%BOYS/GIRLS:FEW

MANYStubborn Behavior

Defiance/Disobedience

AUTHORITYAVOIDANCE

(truancy, running away,

staying out late)

MINOR AGGRESSION (bullying, annoying others)

PHYSICAL FIGHTING(physical fighting,gang fighting)

VIOLENCE(rape, attack,strong-arm, homicide)

MINOR COVERT BEHAVIOR (shoplifting, frequent lying)

PROPERTY DAMAGE(vandalism, fire-setting)

MODERATELYSERIOUSDELINQUENCY(fraud, pick-pocketing)

SERIOUS DELINQUENCY (auto theft, burglary)

Developmental Pathways to Serious and Violent Behavior*

*Loeber©

Graduated Sanctions Component

Driven by data from assessment of offender’s risk factors and treatment needs (Structured Decision Making tools)

The first objective is to ensure public safety by restricting offender’s freedom to commit offenses.

The second objective is to achieve a good match between offender characteristics and their treatment needs.

A Model of Graduated Sanctions

Increasing Sanctions

Decreasing Sanctions

Diversion

Youth Court

Probation

Intensive PS

CB Resid.

Residential Placement

Intensive PS

Probation

Group Counseling

Mentoring

Day/EveReport.

Key DJJDP SDM Tools• DJJDP has a validated risk assessment

instrument

• DJJDP has an effective needs/strengths assessment instrument

• The JJ Reform Act provided a Disposition Matrix

• The Disposition Matrix and risk assessment instrument are functioning well in guiding offender placements

Risk Assessment in Juvenile Justice

NCCD

What is risk assessment?

• A statistical procedure for estimating the probability that a “critical” event will occur at some time in the future.

• In the juvenile justice system, the “critical” event is generally a new delinquent offense committed by a juvenile offender who has been placed on probation or parole supervision.

Needs Assessment in Juvenile Justice

NCCD

A needs assessment is intended to do the following:

• Provide an overview of the level of seriousness of the juvenile offender’s treatment needs

• Provide information to assist court counselors in developing comprehensive treatment plans

• Comprehensive supervision and service plans establish specific objectives, time frames, and offender’s needs to be addressed by the responsible agencies/persons.

Disposition Matrix

• A disposition matrix organizes sanctions and programs by risk level and offense severity.

• It places offenders along a continuum of programs and sanctions

• Research shows that a reliable risk assessment instrument predicts differential recidivism rates at various risk levels.

Key Points of the Disposition Matrix

• Low risk offenders are placed in community programs with minimal supervision

• Medium risk offenders are typically placed in more structured community programs with intensive probation supervision

• High risk offenders may be placed in Youth Development Centers

North Carolina Offender Disposition Matrix

Risk Level

Offense Low Medium High

Violent Level 2 or 3 Level 3 Level 3

Serious Level 1 or 2 Level 2 Level 2 or 3

Minor Level 1 Level 1 or 2 Level 2

       

Level 1 Community

Level 2 Intermediate

Level 3 Commitment to Youth Development Center

 

 

  

Dispositional Levels Risk Level by Disposition 

Low Medium High Total 

  %% % %

Level 1 – Community 65% 31% 3% 100%

Level 2 – Intermediate 27% 47% 26% 100%

Level 3 – Commitment 7% 23% 70% 100%

Protective Supervision 47% 49% 4% 100%

Total 49% 38% 14% 100%

Disposition of NC Court Referrals by Risk Level

46%

40%

14%

0%

10%

20%

30%

40%

50%

Low Medium High

Risk Levels of Adjudicated Juvenile OffendersNorth Carolina, 2003

(N=3722) (N=3197) (N=1115)

20%

34%

45%

0%

10%

20%

30%

40%

50%

Low Medium High

Recidivism by Risk Level in North Carolina (percent with new court complaints)

Admissions to North Carolina YouthDevelopment Centers 1998-2003

122

712

469

95

619

438

112

560

239

108

434

78

110

343

47

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5

NC Youth Development Center Admissions (1998-2002)

Violent Serious Minor Offenses

A Practical Approach To Evaluating and Improving

Juvenile Justice Programs Utilizing The Standardized

Program Evaluation Protocol

The Lipsey-Howell-Tidd Project

What is meta-analysis?

“Meta” means beyond, more comprehensive

A synthesis of evaluation results

A technique for coding, analyzing, and summarizing

research evidence; or a sophisticated way to extract,

analyze, and summarize the findings of a collection of

related research studies

Current database on the effects of intervention with juvenile offenders

Offenders N of Studies

Pre-adjudication (at-risk) 178

Court supervision (probation) 216

Committed 90

Aftercare 25

Total 509

From the bottom up: Evaluation study of intervention effects on recidivism

Recidivism Results from more than 500 Delinquency Outcome Studies (Lipsey, 2002)

A Surprising Revelation to SomeMost everyday (“practical”) juvenile justice programs reduce recidivism! (see the following table)

• The challenge is this. Half of the evaluated everyday JJ programs do not produce meaningful reductions in recidivism.

• To improve this situation, JJ practitioners must use evidence-based program services and in configurations that, on average, produce larger reductions in recidivism .

Practical programs with different numbers of favorable features

Key Meta-Analysis Findings

• Research shows that, on average, most juvenile justice programs reduce recidivism.

• Most JJ programs developed by practitioners also reduce recidivism, but very little.

• Some programs have large effects but implementation is equally important.

• There is enough research to characterize best practice for most programs, but not all.

An important finding about factors associated with program outcomes

Implementation is as important as treatment type

Reduction in Recidivism from .50 Control Group Rate

The Standardized Program Evaluation Protocol

(SPEP)

What is it? A practical method for evaluating juvenile justice and delinquency prevention programs against best practices.

The SPEP provides a scheme (protocol) for assigning points to programs according to how closely their characteristics match those associated with the best outcomes in research.

Four Main Characteristics Of Effective Program Services

1. The Program Type (primary service)2. Supplemental Services3. Amount of Service4. Characteristics of Clients

These are the four sections of the Standardized Program Evaluation Protocol (SPEP) program rating instrument

Point system based on what the research shows as “best practice”

“Ideal” “Good”

Mor

e P

oint

s =

Les

s D

elin

quen

cy

Primary Service

Supplemental Services

Amount of Service

Characteristics of Clients

Room for Improvement

Prevention Programs(Primary Service Types)

Parent training/counselingInterpersonal skills trainingTutoring

Group counselingDrug/alcohol therapy/counselingEmployment related

Individual counselingMentoringFamily counseling

Effective – above average--60 points

Effective – about average--50 points

Effective – below average--40 points

Court Delinquency Supervision Programs(Primary Service Types)

Family counselingTutoringMentoring

Parent training/counselingInterpersonal skills trainingDrug/alcohol therapy/counseling

Individual counselingGroup counselingEmployment-relatedRestitution

Effective – above average--60 points

Effective – about average--50 points

Effective – below average--40 points

Programs for Committed Youths(Primary Service Types)

EFFECTIVE--ABOVE AVERAGE• Behavior management • Cognitive-behavioral therapy • Employment/job training • Interpersonal skills training

EFFECTIVE– ABOUT AVERAGE• Family counseling• Group counseling• Individual counseling

EFFECTIVE-- BELOW AVERAGE• Career/vocational counseling• Tutoring/remedial education

Example of Best Practice Rating System Based on Meta-Analysis Results: NC SPEP for Court

Supervision Cases

Service mix:Primary &supplementaryservices

Amount ofservicecompleted

Risk & ageof juvenilesserved

Pointsallocated inproportionto strengthof relationshipto recidivismfound inmeta-analysis

Data onproviderperformancederived fromclient trackingforms

Total points=match with“best practice”

Expected Recidivism with Features of Effective Court Delinquency Programs

Comparable Juveniles not in a Program 40%

Average Supervision Program in Database 34%

Effective, Above Average Service (AAS) 32%

AAS +Best Supplemental Service (BSS) 28%

AAS +BSS+Optimal Service Amount (OSA) 24%

AAS +BSS+OSA+Appropriate Clients 21%

Continuum Building

Even if well implemented and effective, a single “model” program will do little to strengthen the overall continuum of program options.

The SPEP can be used to evaluate and improve routine programs spanning the continuum from prevention to post-release supervision.

At the same time, program structures must be recognized for their value in a continuum of program services and structures.

Arrest

Counsel & release

Diversion;Informal probation

Probation

Incarceration

Level of Supervision

InterventionPrograms

RecidivismOutcomes

Program A

Program B

Program C

Program D

Program E

Program F

U%U%

V%V%

W%W%

X%X%

Y%Y%

Z%Z%

TotalTotalReoffenseReoffense

RateRate

Pre

ven

tio

n P

rog

ram

s

T%T%

Risk assessmentand risk-based

dispositionsEffectiveprograms

Needs assessment;match needsto program

Minimizereoffending

What the SPEP is Not

• It is not a whole blueprint for a program. It measures only the delinquency reduction potential a program type has, on average, based on prior research.

• It does not provide a treatment plan for individual clients, only a framework within which treatment can be planned.

Key Reference WorksAvailable at: http://www.vanderbilt.edu/cerm/

Howell, J. C., & Lipsey, M. W. (2004). A practical approach to linking graduated sanctions with a continuum of effective programs. Juvenile Sanctions Center Training and Technical Assistance Bulletin Vol. 2(1), 1-10. Reno, NV National Council of Juvenile and Family Court Judges.

Howell, J. C., & Lipsey, M. W. (2004). A practical approach to evaluating and improving juvenile justice programs. Juvenile and Family Court Journal , 55(1), 35-48.

Lipsey, M. W. (2002). Meta-analysis and program outcome evaluation. Socialvetenskaplig Tidskrift, 9(2-3), 194-208.

Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.