Summary Report Evaluation of the CYSA/TANF Program in California Doug Calderwood Edwin Kitzes Cliff Howard Stoycho Ivanoff PADM 522 Presentation – November

Summary ReportEvaluation of the CYSA/TANF

Program in California

Doug CalderwoodEdwin KitzesCliff Howard

Stoycho Ivanoff

PADM 522 Presentation – November 20th, 2012

Evaluation of Federal Assistance• Chief Probation Officers of California sponsored an

evaluation of the implementation and impact of California Youth Services Act (CYSA) distribution of federal assistance administered the Temporary Assistance for Needy Families (TANF) to California county probation departments (CPDs).

• 3 year Evaluation conducted by the RAND corporation.

Howard

Four Goals of Evaluation• provide timely feedback to Chief

Probation Officers of California on CYSA/TANF implementation,

• document how county probation departments (CPDs) have used CYSA/TANF funds

• assess impact of local interventions, and • draw lessons for improving overall

program design and operation

Howard

Evaluation Sought Answers to 5 Questions

1. What programs were implemented?2. What CYSA/TANF services were provided?3. What CYSA/TANF services and programs were

being provided in the juvenile halls and camps/ranches?

4. What was the impact of CYSA/TANF at the individual and system level?

5. What was the CYSA/TANF funding environment and what were county claiming experiences?

Howard

Probation Departments Administered at the County Level – 58 Counties

Howard

California County Probation Department Funding

State and Federal Funds

Howard

California funding for Probation Facilities

• California county probation departments (CDPs) started receiving federal funds under Title IV-A-EA of the Social Security Act in July 1993 to reimburse institutional costs for juvenile camps and ranches and many child welfare services.

• Ended on December 31, 1995 with the passage of Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) (a.k.a. welfare reform act)

Howard

California County Probation Departments State and Federal Funding

Howard

Sequence of federal assistanceTitle IV-A-EA to TANF

Title IV-A-Emergency Assistance Jul 1993 -Dec 1995PRWORA TANF July 1997

Howard

TANF Replaces Title IV-A-EA Funding• PRWORA instituted Temporary Assistance to

Needy Families (TANF) which became effective July 1, 1997

• TANF is a block grant to states to target needy families – it replaced previous welfare programs

• TANF for California County Probation Departments was around $168M annually during the time of this study.

• CPDs went for 18 months w/o Title IV-A-EA federal assistance

Howard

Intent of CYSA• Provide a continuum of family-focused, case-

specific services in a community-based setting that addresses the full spectrum of child and family needs, including services provided in county-operated residential care facilities

• Programs were to focus more on prevention/intervention when possible

Howard

Goals of CYSA• Legislation had 3 main goals– Keep probation youths from further crime– Help probation and at-risk youths to develop skills to

avoid dependence on public assistance– Help achieve 4 overarching federal TANF goals

• Provide assistance to families so youths may be cared for in their homes

• Reduce dependence of needy parents on government benefits by promoting job preparation, work and marriage

• Encourage formation/maintenance of two-parent families• Prevent/reduce incidence of out-of-wedlock pregnancies

Howard

CYSA Program Planning Procedure• Establish local planning council to advise the county Chief

Probation Officer including agencies– County departments of health, mental health, probation, child

protective services and education– Local school districts– City/county law enforcement agencies– Community organizations that serve at-risk youth– One or more youth who are at-risk– Parents of family of at-risk youth

• Chief Probation Officer prepares the written plan• County Board of Supervisors must approve the CYSA/TANF

programs Howard

5 Key Principles CYSA/TANF• be oriented toward principles of personal

responsibility and self-reliance• use available community resources to the

extent possible• be based on case plans that consider

family concerns, priorities, and resources• be family-focused• address identified immediate needs, as

well as underlying risk factors contributing to problems that are more pervasive and recurrent in nature

Howard

CYSA/TANF Goals pursued by providing a spectrum of 23 authorized services

CYSA/TANF Services Areas Emergency shelter Parent peer support

After-care servicesFamily crisis intervention Parenting skills

Anger management Family mentoring Pre-vocational trainingAvailability of community services Gang intervention Respite careCase management Home detention Sex/health educationCounseling, monitoring, and treatment

Individual, family, group counseling

Social responsibility training

Drug/alcohol education Life skills counseling

Therapeutic day treatment

Educational advocacyMental health assessment

Transportation to needed services

Howard

Spectrum of Juvenile Cases5 categories from least to most serious

• At risk youth – failing academically, poverty, disabled, disadvantaged, certain ethnic minorities

• WIC * 601 – accused of status offense (an offense that’s not applicable to adult)

• WIC * 602 – accused of misdemeanor or felony• Wardship – under court supervision, placed at

home or at a relative’s house on probation • Custody – under court supervision, sentenced

from 0-12 months in juvenile camp or ranch* WIC - California Welfare and Institutions Code

Howard

Change Model from Chen

Intervention Determinants Outcome23 authorized service categories such as: • After-care services• Anger management• Availability of community

services• Case management• Counseling, monitoring,

and treatment• Drug/alcohol education• Educational advocacy• Emergency shelter• Family crisis intervention• Family mentoring• Gang intervention• Home detention

Function of Juvenile Case - Includes things like:• Home/shelter conditions• Family/home conditions• Exposure to:o Drugs/alcoholo Gang violenceo Physical vulnerabilitieso Verbal abuse• Coping abilities• Decision making abilities• Access to harm others• Academic success level• Food availability

Function of Case –Goals include:• Recovery from

transgression• Avoid repeat offense• Live at home• Not dependent on

public assistance• Develop Job skills• Stable 2-parent families• Avoid out-of-wedlock

pregnancies• Protect community• Academic success

Howard

Review of Evaluation Scope,Program Goals, Authorized Services

• CYSA has 6 goals• California has 58 counties• Legislation authorizes 23 service categories• Juvenile cases span at least 5 broad categories of

seriousness (at-risk youth to detention facilities)• Service centers include multiple government and

community facilities and organizations • CYSA/TANF funding is combined with several

other county, state, and federal sourcesHow do you evaluate this?

Howard

Four Components of Evaluation• statewide implementation surveys fielded to

all 58 counties (Years 1 and 3)• an 11-county process study (Years 1 and 2)• an outcomes/impact analysis (Years 1 and 2)• a funding/claim submission analysis (Year 3)

5 questions to be addressed by evaluation1. What programs were implemented?2. What CYSA/TANF services were provided?3. What CYSA/TANF services and programs were being provided in the

juvenile halls and camps/ranches?4. What was the impact of CYSA/TANF at the individual and system level?5. What was the CYSA/TANF funding environment and what were county

claiming experiences?Howard

Data Collected in the Evaluation• Programs implemented under CYSA/TANF (Cliff)• CYSA/TANF services provided (Edwin)• CYSA/TANF services provided in the juvenile

halls and camps/ranches (Edwin)• The impact of CYSA/TANF at the individual and

system levels (Doug)• The CYSA/TANF funding environment and

experiences (Doug)• Summary, Conclusions, and Observations

(Stoycho)Howard

Distribution of Program Funds Along the Continuum of Options

Programs ImplementedHoward

custody

multiplemultiple

custody

Distribution of Program Funds Along Target Population

601 truan

ts

601 runaway

adults


602s

Percent of Counties with Programs That Target Youths and Families Along the Continuum of Options (N=55)


Key Programs in 6 Various Counties

Juvenile assessment centerFamily services program

Personal/social responsibility Juvenile Hall programCamps servicesVoucher program

Medium-level custody programJuvenile Hall Youth Center ServicesBoot camp program

Youth and family resources centersJuvenile Hall unitDiversion program for at-risk youths

Multiagency comprehensive community-based programAftercareDrug Court

Prevention network

Programs Implemented

County B

County E

County D

County G

County C

County F

Howard

Services Areas Categorized into Four Groupings

Formal Treatment Life Skills DevelopmentCounseling, monitoring, treatmentIndividual, family counselingDrug/alcohol educationMental health assessmentTherapeutic day treatmentMental health assessmentFamily crisis intervention

Family mentoringAnger managementLife skills counselingSex/health educationGang interventionParenting skillsEducational advocacy

Coordination OtherAvailability of community servicesCase managementTransportation to needed service

Home detentionAfter-care servicesEmergency shelter

Howard

Types of CYSA/TANF Services Outsourced (N=24)

Programs Implemented Howard

Number of Youths and Families Served in 44 Reporting Counties (June 2002) (N=44)

HowardPrograms Implemented

CYSA/TANF Mediating Variables• Implemented during a time when juvenile

programs were being overhauled• Comprehensive integrated planning was

encouraged• Multiagency planning bodies existed already• Needs assessments had already been done• Built on existing CPD relations with service

providers


CYSA/TANF Program Moderators

• Difficulty to implement multiservice delivery model

• Turf concerns, philosophical differences, conflicts between service agencies

• Procedures for claiming funds time consuming• CYSA implementation guides were vague• Difficult to evaluate qualifications of some service

providers• Difficult to write performance-based contracts• Staff shortages in early yearsPrograms Implemented

Howard

CYSA/TANF Services Provided

Edwin Kitzes

Services ProvidedKitzes

From the Chen Reading• Assessment-Oriented Process Evaluation• Primary for External Stakeholders to

evaluate how well a program is being implemented

• Evaluate three main program purposes:1) Meeting Accountability Needs2) Meeting Program Improvement Needs3) Provide Interpreting Outcome

Evaluation

Services Areas Categorized into Four Groupings

Formal Treatment Life Skills DevelopmentCounseling, monitoring, treatmentIndividual, family counselingDrug/alcohol educationMental health assessmentTherapeutic day treatmentMental health assessmentFamily crisis intervention

Family mentoringAnger managementLife skills counselingSex/health educationGang interventionParenting skillsEducational advocacy

Coordination OtherAvailability of community servicesCase managementTransportation to needed service

Home detentionAfter-care servicesEmergency shelter

Howard

Services ProvidedHow was the data gathered?

• Year 1 Survey – before/after – all programs

• Year 3 Survey – Counties primary programs/services provided

• 58-Counties• Compare pre- and post-CYNA/TANF• Compare different size counties, big vs.

small• ‘Real Change’ vs. adjustments of claims


58 Counties in California

Howard

Formal Treatment Services

46 Counties 12 Counties

Before | After


Coordination Services


Life Skills Development


Other Services


Services ProvidedSummary

• Majority of funds used to ‘Enhance existing services’

• Larger counties were more likely to enhance service vs. smaller counties which used funds to add new services.

Kitzes

Services ProvidedYear 3


Services Provided


Types of CYSA/TANF Services Outsourced (N=24)

Programs Implemented Howard

Spectrum of Juvenile Cases6 categories from least to most serious

• At risk youth – failing academically, poverty, disabled, disadvantaged, certain ethnic minorities

• WIC * 601 – accused of status offense (an offense that’s not applicable to adult)

• WIC * 602 – accused of misdemeanor for felony• WIC* 777 – accused of probation violation• Wardship – under court supervision, placed at home or

at a relative’s house on probation • Custody – under court supervision, sentenced from 0-12

months in juvenile camp or ranch* WIC - California Welfare and Institutions Code

Howard

Services Provided


Services ProvidedYear 3 Summary

• 11 of the 23 services were provided in more than 75% of the counties.

• There was “real change” made in the delivery of the services due to CYSA/TANF funds - virtually unanimous in all counties.


• Larger counties were more likely to enhance service vs. smaller counties which used funds to add new services.

Kitzes

CYSA/TANF Services Provided inJuvenile Halls and Camps/Ranches

Services in halls/campsKitzes

Juvenile Hall, Orange County

Juvenile Hall, San Deigo County

http://www.youtube.com/watch?v=QTI2u3bmKFQ

Juvenile Halls and Camps/Ranches

http://www.youtube.com/watch?v=gyaO634JWpU

http://www.youtube.com/watch?v=QTI2u3bmKFQ



Services ProvidedHow was the data gathered?

• Year 2 Survey – In-depth survey of 11 Counties

• Purpose: gather general information: to gather information on services provided in Halls/Camps/Ranches

• And results of CYSA/TANF funds‘Real Change’ vs. adjustments of claims

• Compare pre- and post-CYNA/TANFServices Provided

Kitzes

Services provided in the juvenile halls and camps/ranches


1,50026



KitzesServices in halls/camps


Services ProvidedYear 2 Summary Halls

• Relative short stays in juvenile halls; often occupied at daily capacity.

• Claiming strategies were driven to maximize Federal revenues received.

• There was “real change” made in the delivery of the services due to CYSA/TANF funds.

• Formal treatment, Life Skills Development, but few Coordination and Other Services


Kitzes

Services ProvidedYear 2 Summary Camps/Ranch

• Mirrored Juvenile Hall results with the exception of more services provided

• AGAIN, • Claiming strategies were driven to maximize

Federal revenues received.• There was “real change” made in the delivery

of the services due to CYSA/TANF funds.

Kitzes

CYSA/TANF

Doug Calderwood

Individual and Systemic Impact

Funding Environment and Experiences

CYSA/TANF Systemic and Individual Impact

• System Evaluation– Year 1 and 3 surveys– Chiefs’ assessments– Evaluation Team’s

assessment

• Individual Evaluation– Four County Programs’

outcomes

Calderwood

The Multi-Goal, Theory-Driven Approach to Evaluation

Huey T. Chen & Peter H. Rossi• In one field after another, evaluation

researchers find that the programs in place or contemplated have few or no effects of the sort intended by their designers.

(1980, p. 106)

Calderwood

“…[S]tatements of program goals tend to be vague and/or stated in terms that do not lend themselves easily to precise measurement. …[I]n practice one is able to define only a few measurable, specific outcome variables (goals) …typically quite narrow compared to the original social problem that the social program aims to alleviate.”

(Chen & Rossi, 1980, p. 109)

Calderwood

“Clearly the problems with the conventional paradigm are not to be remedied by using rigorous research designs since there would be even less chance of finding effects using more powerful designs.”

(Chen & Rossi, 1980, p. 109)

Calderwood

System—Year 1 and 3 Surveys

• “CYSA Legislation hoped to achieve specific outcomes” (Turner, Davis, Steinberg & Fain, p. 51)– Effects on juvenile justice services (Table 5.1)– Moderate to large impact (2.9 to 3.9 out of 5)

• “CYSA Legislation’s goal was to…avoid dependency…and reduce…future criminal activity” (Turner et. al., p. 52)– Effects on target population (Table 5.2)– Positive to extremely positive impact (3.4/5 to 4.6/5)

Calderwood

System—Chief’s Assessment• Separate survey of Chiefs in year 3 included

two questions:– Additional activities enabled?• Addition and enhancement of specific services• Base funding for custody programs• Expansion of range of services

– Impact if funding went away?• Services would be cut

Calderwood

System—Evaluation Team

• CYSA/TANF:– Filled an important funding gap– Supported a shift from suppression,

enforcement and monitoring to rehabilitation and therapy

– Increased system-wide collaboration and planning

– Enabled viability of probation in county level children’s programs

Calderwood

Individual—Four County Programs

• Research criteria for selection– Distinct CYSA/TANF Programs – Comparison group available– Identifiable CYSA/TANF clients– County collects client characteristics, services

delivered and outcomes• Inclusivity selection criteria– Counties: Northern/Southern and Large/Small– Programs: Field/Institutional

Calderwood

County K: Adult TANF Caseload Program

• TANF-eligible parolees received added family services

• Case load limits excluded some eligible—the control group

• TANF and control had equal risks, TANF had more strengths (housing, family stability and food)

• TANF had fewer probation violations, more technical violations

Calderwood

County F:Juvenile Assessment Center Program

• Assessment, case planning and follow-up– 2,437 youths served from 2/98 to 5/02

• Historical control group – 787 youths from ’97

• Recidivism was less for program participants at 6, 12 and 18 months

• Time before first subsequent referral was also longer

Calderwood

County H:Placement Readiness Recidivism Program

• Youth with mental illness in custody got weekly independent therapy, theme groups and daily recreation

• 200 youths participated from 6/99 to 3/02• Historical control group of 200 youths who

would have been eligible• Participants has lower recidivism but worse

behavior while in custody

Calderwood

County D:Youth and Family Resource Centers

• Day-reporting and treatment centers for “8%” youth—a profile of a small proportion accounting for 50% of repeat offenses

• Control group met profile but were not assigned

• Results showed little difference for those in the program

Calderwood

CYSA/TANFFunding Environment and Experiences

• Probation Funding Environment

• Claiming Strategies

• Leveraging Strategies

• Fiscal Pressures

Calderwood

California County Probation Departments State and Federal Funding

Howard

Funding Environment

• CPD funding is about 10% CYSA/TANF (more in small counties)

• Title IV-E funds can’t be used in custody settings, so TANF is more heavily claimed there.

• This is part of the difficulty evaluating “pure” CYSA/TANF programs

Calderwood

Funding Environment

• About 1/3 of counties experienced fiscal pressure—withholding or re-appropriation of CYSA/TANF funds

• Chiefs of CPDs felt CYSA/TANF funding was extremely important overall

• CYSA/TANF money replaced lost Title IV-A-EA funds

Calderwood

CYSA/TANF Program Complications

• 23 services in four broad categories• 58 counties setting up programs• CYSA seeks to bridge TANF and Juvenile Justice• TANF funding often used to cover programs

defunded when Title IV-A-EA went away• It is unlikely that local programs have clear

program theories• Programs have little traditionally measurable

effectCalderwood

Documents

Summary Report Evaluation of the CYSA/TANF Program in California Doug Calderwood Edwin Kitzes Cliff Howard Stoycho Ivanoff PADM 522 Presentation – November