The Standardized Program Evaluation Protocol ~SPEP~

11

The Standardized Program Evaluation The Standardized Program Evaluation ProtocolProtocol

~SPEP~~SPEP~

Understanding the Tool Understanding the Tool

and and

Program Improvement Program Improvement ImplicationsImplications

Division of Adult Correction and Juvenile Justice

2

Historical Context of North Carolina’s Historical Context of North Carolina’s Juvenile Justice Evidence–Based Juvenile Justice Evidence–Based

PracticesPractices►First State-wide community-based First State-wide community-based

alternatives (CBA) ~ late 1970salternatives (CBA) ~ late 1970s►First “wraparound” model in the US via First “wraparound” model in the US via

Willie M. Program~ 1979 class action Willie M. Program~ 1979 class action lawsuitlawsuit

►1998 NC Juvenile Justice Reform Act ~ 1998 NC Juvenile Justice Reform Act ~ Second state to mandate evidence-Second state to mandate evidence-based servicesbased services

3


PracticesPractices► First state to fully adopt the OJJDP First state to fully adopt the OJJDP

Comprehensive Strategy for Serious, Comprehensive Strategy for Serious, Violent, and Chronic Juvenile OffendersViolent, and Chronic Juvenile Offenders

► First legislated statewide, county-level First legislated statewide, county-level Juvenile Crime Prevention Councils (also first Juvenile Crime Prevention Councils (also first statewide use of a risk factor assessment statewide use of a risk factor assessment beginning in the late 1990’s)beginning in the late 1990’s)

► Most effective statewide use of a juvenile Most effective statewide use of a juvenile offender risk and needs assessment offender risk and needs assessment instrument along with a disposition matrix to instrument along with a disposition matrix to reduce confinement (Nov, 2001)reduce confinement (Nov, 2001)

4


PracticesPractices►First state to begin implementation of First state to begin implementation of

a state-wide gang prevention and a state-wide gang prevention and intervention initiative (Community-intervention initiative (Community-Based Youth Gang Violence Prevention Based Youth Gang Violence Prevention Project) 2009 (Use of Recovery Act Project) 2009 (Use of Recovery Act funds) (THE first U. S. experiment to funds) (THE first U. S. experiment to fund statewide Comprehensive Gang fund statewide Comprehensive Gang Model programming!)Model programming!)

5

Historical Context of Historical Context of SPEPSPEP in in NCNC

►1998 NC Juvenile Justice Reform Act 1998 NC Juvenile Justice Reform Act required a statewide evaluationrequired a statewide evaluation of of programs funded through the JCPCprograms funded through the JCPC

►The The Standardized Program Evaluation Standardized Program Evaluation Protocol ~SPEP~Protocol ~SPEP~ was developed was developed specifically for this purpose in NC. 2006specifically for this purpose in NC. 2006

►First statewide Level II services based First statewide Level II services based on SPEP primary service types (2010)on SPEP primary service types (2010)

6

Most (57%) JJ programs reduce recidivism: Most (57%) JJ programs reduce recidivism: Outcomes of 556 studies (Dr. Mark Lipsey, Outcomes of 556 studies (Dr. Mark Lipsey,

2002)2002)

77

The SPEP ProcessThe SPEP Process

Key Steps to SPEP ScoreKey Steps to SPEP Score

8

MATCH AGAINST RESEARCH-BASED

CATEGORIES

IDENTIFYSERVICES

DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE

SPEP

SCORE

EVALUATE/PROGRAM IMPROVEMENT

SPEP SCORING PROCESS

IDENTIFY/ MATCHSERVICES

DATA (DEMOGRAPHIC,RISK,QUALITY, QUANTITY FOR EACH SERVICE)

1010

Getting Started…….Getting Started…….

Classification of Program Classification of Program InterventionIntervention

11

SPEP 101 DefinitionsSPEP 101 Definitions

► Structure or FormatStructure or Format: Setting within which : Setting within which the program services are deliveredthe program services are delivered Example: Residential, YDC, Day TreatmentExample: Residential, YDC, Day Treatment

► ProgramProgram:: Services that are delivered within Services that are delivered within the context of a structure or format.the context of a structure or format.

► ServiceService:: The treatment ingredient of the The treatment ingredient of the program. Services target key criminogenic program. Services target key criminogenic risk factors and treatment needs and are risk factors and treatment needs and are direct interventions with the juvenile or with direct interventions with the juvenile or with others (peers or family) in order to alter others (peers or family) in order to alter juvenile behavior that leads to reduced juvenile behavior that leads to reduced recidivism.recidivism.

12

Identification of “Philosophies” Identification of “Philosophies” toward Altering Juvenile toward Altering Juvenile

BehaviorBehavior► Program Services Fall into 3 Broad Program Services Fall into 3 Broad

Categories:Categories: External Control TechniquesExternal Control Techniques

► Behavioral change through instilling discipline, fear, or Behavioral change through instilling discipline, fear, or detection of bad behavior in the absence of treatmentdetection of bad behavior in the absence of treatment

Therapeutic TechniquesTherapeutic Techniques► Behavioral change through improved skills, Behavioral change through improved skills,

relationships, insight.relationships, insight.► These are the only services for which SPEP ratings are These are the only services for which SPEP ratings are

completed.completed.

OtherOther :Services for which there is insufficient :Services for which there is insufficient research to estimate the effects on recidivismresearch to estimate the effects on recidivism

13

Program Types Sorted by Program Types Sorted by General Approach: Average General Approach: Average

Recidivism EffectRecidivism Effect

Controlapproaches

Multiple services

Counseling*

Skill building**

Restorative

Surveillance

Deterrence

Discipline

-10 -5 0 5 10 15

% Recidivism Reduction from .50 Baseline

Controlapproaches

Therapeuticapproaches

14

Further Sorting by Intervention Further Sorting by Intervention Type within, e.g., Type within, e.g., Counseling Counseling

ApproachesApproaches

Mixed w/referrals

Mixed

Peer

Group

Family crisis

Family

Mentoring

Individual

0 5 10 15 20 25


15

Further Sorting by Intervention Further Sorting by Intervention Type within, e.g., Type within, e.g., Skill-building Skill-building

ApproachesApproaches

Job related

Academic

Challenge

Social skills

Cognitive-behavioral

Behavioral

0 5 10 15 20 25 30


16


►Primary ServicePrimary Service:: Main focus of the programMain focus of the program Most time/effort spent on thisMost time/effort spent on this Recognized as the dominant theme Recognized as the dominant theme Other program elements support its Other program elements support its

effectivenesseffectiveness

17


► Supplemental ServiceSupplemental Service:: Other distinct serviceOther distinct service Less time/effort spent on thisLess time/effort spent on this Serves to reinforce or complement the primary Serves to reinforce or complement the primary

serviceservice Note: Supplemental service can be deemed as a Note: Supplemental service can be deemed as a

“qualifying” one in order to receive additional “qualifying” one in order to receive additional points on the SPEP tool OR points may be points on the SPEP tool OR points may be automatic based on lack of evidence in the automatic based on lack of evidence in the research base to couple a supplemental service research base to couple a supplemental service with the identified primary service.with the identified primary service.

WHAT IS THE PROGRAM’S PRIMARY SERVICE?

IN WHICH OF THE 5 SERVICE GROUPS DOES THE SERVICE FALL?

IS THERE A QUALIFYING SUPPLEMENTAL SERVICE ?

19

Things to Consider………Things to Consider………

► JCPCs will need to fund structures or formats. JCPCs will need to fund structures or formats. (Teen Court, Structured Day Programs)(Teen Court, Structured Day Programs)

► It is important that an analysis of key therapeutic It is important that an analysis of key therapeutic element (s) within the structures be explored. element (s) within the structures be explored.

► JCPCs need to fund JCPCs need to fund Therapeutic Therapeutic Approaches/InterventionsApproaches/Interventions

► JCPCs need to understand the SPEP process in JCPCs need to understand the SPEP process in order to understand SPEP scores and their order to understand SPEP scores and their implications for program improvement. implications for program improvement.

2020

DATADATA

Examining the Service Quality, Amount Examining the Service Quality, Amount of Service (Duration and Contact of Service (Duration and Contact

Hours), and Risk Level Hours), and Risk Level

21

Data Used for SPEP ScoresData Used for SPEP Scores

►Quality of ServiceQuality of Service:: SPEP Quality of Services ChecklistSPEP Quality of Services Checklist~ ~

Completed with the Area Consultant Completed with the Area Consultant Examines the following:Examines the following:

►Written program protocol ~ program manual; Written program protocol ~ program manual; written protocol that describes how the written protocol that describes how the intended services is deliveredintended services is delivered

►Staff training /staff retentionStaff training /staff retention►Monitoring of Program effectiveness, protocol Monitoring of Program effectiveness, protocol

and corrective action to correct deviationsand corrective action to correct deviations

SPEP Quality of Services Checklist Score

23


►Amount of ServiceAmount of Service: : (NCALLIES)(NCALLIES) Duration of ServiceDuration of Service

►Target number of weeks specified for each Target number of weeks specified for each service typeservice type

►% of youth who received targeted weeks of % of youth who received targeted weeks of serviceservice

►Maximum number of points ~10Maximum number of points ~10 Contact HoursContact Hours

►Target number of hours specified for each Target number of hours specified for each service typeservice type

►% of youth who received targeted hours of % of youth who received targeted hours of serviceservice

►Maximum number of points ~10Maximum number of points ~10

Data generated from NC ALLIES

25


► Risk level of Youth Served:Risk level of Youth Served: Program’s use of a validated risk assessment Program’s use of a validated risk assessment

instrument/tool to provide juvenile risk level data instrument/tool to provide juvenile risk level data on each juvenile receiving serviceson each juvenile receiving services

► North Carolina’s Assessment of Juvenile Risk of Future North Carolina’s Assessment of Juvenile Risk of Future OffendingOffending ~ Validated tool currently being used by ~ Validated tool currently being used by Juvenile Court Services. This risk assessment tool has Juvenile Court Services. This risk assessment tool has been selected for use by program providers. The been selected for use by program providers. The department will train providers on its use.department will train providers on its use.

► NC ALLIES will be modified with the addition of a NC ALLIES will be modified with the addition of a Risk Risk Level field.Level field. Risk Level Scores from referrals from Risk Level Scores from referrals from Juvenile Court would be entered as well as risk level Juvenile Court would be entered as well as risk level scores generated by the program.scores generated by the program.

Future modifications will include incorporating the risk Future modifications will include incorporating the risk assessment tool into NCALLIES. assessment tool into NCALLIES.

Data Generated via a Validated Risk Assessment Tool and entered into NC ALLIES

27

The SPEP SCOREThe SPEP SCORE

►Examines how a specific program is Examines how a specific program is performing compared to the effective performing compared to the effective practice for that service type in the practice for that service type in the researchresearch

►Provides Provides opportunities for program opportunities for program improvementimprovement for programs that fall for programs that fall short of the evidence-based effective short of the evidence-based effective practice profiles.practice profiles.

28

SSystematicystematic

CCorrectionorrection to to

OOptimizeptimize

RRecidivism reductionecidivism reduction

EEffectivelyffectively

29

Scoring = Improvement Scoring = Improvement OpportunityOpportunity

► Initial ScoreInitial Score

► Understanding the ScoreUnderstanding the Score

► Implications for ImprovementImplications for Improvement

► Program Improvement PlanProgram Improvement Plan

► Subsequent ScoringSubsequent Scoring

► RepeatRepeat

30

Program ImprovementProgram Improvement► Program Improvement Plans address the Program Improvement Plans address the

areas needing improvement as identified areas needing improvement as identified through the program’s SPEP score.through the program’s SPEP score.

► Timeframes for program improvement will Timeframes for program improvement will be establishedbe established

► Monitoring of the status of program Monitoring of the status of program improvement will be incorporated into the improvement will be incorporated into the JCPC monitoring/planning processes and JCPC monitoring/planning processes and Consultant Program MonitoringConsultant Program Monitoring

► JCPCs would continue to evaluate the JCPCs would continue to evaluate the effectiveness of their funded programs effectiveness of their funded programs through general oversight and supporting through general oversight and supporting program improvement plans.program improvement plans.

3131

Questions?Questions?

Ronald TillmanRonald TillmanCentral Area ConsultantCentral Area Consultant

[email protected]

Division of Adult Correction and Juvenile Justice

Documents

The Standardized Program Evaluation Protocol ~SPEP~