Measuring Child Welfare Agency Performance: Advantages and Challenges of State, County,

& University Collaboration

National Association of Welfare Research and Statistics45th Annual WorkshopMadison, Wisconsin

August 31, 2005

Kelly CrossSan Bernardino County HS

Terry V. ShawCenter for Social Services ResearchUniversity of California at Berkeley

The Performance Indicators Project is funded by the California Department of Social Services and the Stuart Foundation

Measuring Child Welfare Outcomes

Child In Child OutA bunch of stuff happens

*adapted from Lyle, G. L., & Barker, M.A. (1998) Patterns & Spells: New approaches to conceptualizing children’s out of home placement experiences. Chicago: American Evaluation Association Annual Conference

Government Performance Results Act of 1993 (GPRA)

Annual Outcomes Report to Congress mandated by Adoption and Safe Families Act (ASFA) of 1997

Statewide Data Indicators in Child and Family Services Reviews -- a subset of the Annual Outcomes—from National Child Abuse and Neglect Data System (NCANDS) and Adoption and Foster Care Analysis and Reporting System (AFCARS)

California Child Welfare System Improvement and Accountability Act (AB636) became law in 2001 and went into effect in January 2004

Outcomes, outcomes, everywhere

Quarterly distribution of county specific outcome

indicators data• Includes national standards (from AFCARS),

but also draws heavily on previous work done by CWDA and UCB using entry cohort measures

• Mirrors Family to Family Outcomes

• Retains key process measures (e.g., child visits, time to investigation)

Statewide Data Indicators from

AFCARS Stability Of Foster Care

Placement Length Of Time To

Reunification Foster Care Re-entries Length Of Time To Adoption

Why do we use entry cohort measures in addition

to measures from AFCARS?

Who is in AFCARS? AFCARS contains data on children in

foster care during a federal fiscal year

Each reporting period’s submission is a separate dataset. Reporting periods are linked together by the Children’s Bureau to form the annual databases. ANNUAL DATABASES ARE NOT LINKED TO EACH OTHER.

11/02 11/03 11/04

Data snapshots can be biased

Source: Aron Shlonsky, University of Toronto (formerly at CSSR)

California EXAMPLE: Age of Foster Children

(2003 first entries, 2003 exits, July 1 2003 caseload)(2003 first entries, 2003 exits, July 1 2003 caseload)

22

31

22 20

54

30

24 2219

5

24 24

32

16

0

10

20

30

40

50

<1 1-5 6-10 11-15 16+

Age in Years

Perc

ent

EntriesExitsPoint in Time

California EXAMPLE:Median Length of Stay in Months

(1998-2000 first entries, 2001 first spell exits, July 1 2002 first spell caseload)

1821

13

22

14

2126

17 1821

2528

2419

46

0

10

20

30

40

50

All Kin Foster FFA Group

Placement Type

Mon

ths

Entries

Exits

Point inTime

• Federal Measure: Of all children who were adopted during the year, what % had been in care for less than 24 months? (national standard = 32%)

• State enriched: Of all children entering care for the first time, what % are adopted in less than 24 months? (we do not have state standards)

Percent of children exiting care to finalized adoption in less than 24 months

(32% National Standard)

• Baseline: 100 kids exiting to adoption, 33 of them within 24 months=33%. Substantial conformity achieved!

Two pronged approach (1) Faster adoption for 100 children, 50 of them within 24 months=50%,

(2) adoptions for 100 kids in long term care

• 2 years later: 200 kids exiting to adoption, 50 within 24 months=25%. Substantial conformity NOT achieved!?!

Are you getting better or worse? Data from the Multi State Data Archive

Adoption within 24 Months

State A

0%

5%

10%

15%

20%

25%

30%

1990

1991

1992

1993

1994

1995

1996

1997

Ado

pted

FederalState

year

Source: Chapin Hall Center for Children

Why don’t we have state standards ?

The Cycle of Experiences in the Child Welfare The Cycle of Experiences in the Child Welfare SystemSystem

CounterbalancedCounterbalancedIndicators ofIndicators of

SystemSystemPerformancePerformance

PermanencyPermanencyThroughThrough

Reunification,Reunification,Adoption, orAdoption, orGuardianshipGuardianship

LengthLengthOf StayOf Stay

StabilityStabilityOf CareOf Care

Rate of Referrals/Rate of Referrals/Substantiated ReferralsSubstantiated Referrals Home-BasedHome-Based

Services vs.Services vs.Out-of-HomeOut-of-Home

CareCare

Positive Positive AttachmentsAttachments

To Family,To Family,Friends, andFriends, andNeighborsNeighbors

Use of LeastUse of LeastRestrictiveRestrictive

Form of CareForm of Care

Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill:  Jordan Institute for Families,

Reentry to CareReentry to Care

Lack of understanding about the limitations of the national standards, and pressure to

achieve “substantial conformity” (pass), could

drive changes in policy and practice that may not be best

for children and families.

Kelly CrossSan Bernardino County HS

AB636 Components

• Quarterly distribution of county specific outcome indicators data• County Self Assessment• Peer Quality Case Review• County Self Improvement Plan• Continuous monitoring of outcomes

Limitations of Administrative Data

• Adapting a case management system into a mechanism for tracking longitudinal outcomes.

• A three-year cycle of outcome evaluation•Still are not ready to set a baseline•Must refine measurement methods, data clean up and training.

Examples of County Work to Examine Data and Improve Practice

• Changing policy•Social worker monthly contacts•Out of home abuse.

• Confounding policy and data issues•ICWA status•Recurrence of maltreatment

• Associated referrals• “Substantial risk,” & “Sibling abused, child at

risk”

Examples of County Work to Examine Data and Improve Practice

continued

• Complex outcomes—Required health visits•One of several complex measures in development

• Identify CHDP standards• Allow data entry lag• Revise code to account for different time

periods when children are in compliance

• An example of improved practice—Reentry•Examining data helped target those children

most at risk of returning to care after reunification

•Expanded use of Public Health Nurses

AB636=State / County Partnership

• Shifts focus from process measured compliance to outcome based review system, but requires linking outcomes to related processes.

• Data are our friends, not our dictators. • Requires county collaboration with community partners (SIPs signed by Boards of Supervisors).

• Promotes sharing of promising practices among counties.

More Advantages of University Involvement

• Participate regularly on state and county workgroups and committees.

• Share programming code and seek input from county partners on its continual improvement.

• Ensure availability to answer ad hoc

questions

Data and Policy Committees

• Continual refining of measurement process requires both Data and Policy Committees.

• Policy committee—interprets regulatory implications and decides general structure that measurement will conform to (e.g., inclusion of guardianship in measures).

• Data committee—determines specific data

collection & analysis steps necessary to implement measurement guidelines decided by policy group.

Examples of County Work to Examine Data and Improve Practice

continued

• Complex outcomes—Required health visits•One of several complex measures in development

• Identify CHDP standards• Allow data entry lag• Revise code to account for different time

periods when children are in compliance

• An example of improved practice—Reentry•Examining data helped target those children

most at risk of returning to care after reunification

•Expanded use of Public Health Nurses

UCB Website

cssr.berkeley.edu(Child Welfare Services Reports)

includes

age, ethnicity, gender breakoutskin vs non-kin

for all AB636 measures and more

use “Datadude” to examine performance over time

Lessons Learned

•It takes time.

•Keep accurate records of development & policy decisions. •Participant turn over (state, county, & university)

•Discovering new populations

State Websites http://www.dss.cahwnet.gov/cdssweb/

(Child Welfare Systems Improvements)

http://www.childsworld.ca.gov/CDSSCounty_1954.htm

(AB636 Quarterly Reports)

Kelly Cross kcross@hss.sbcounty.gov

(909) 388-0174