1 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Welcome! We will begin our webinar at 9:00 am PT

1National CASA/Child Trends Performance MeasurementLori Delale-O’Connor and Sarah Catherine Williams

Welcome!

• We will begin our webinar at 9:00 am PT

• As you log on, do not be surprised if you don’t hear anyone else; participants are placed on mute in order to limit background noise

National CASA Performance Measurement

Lori Delale-O’ConnorSarah Catherine Williams

Child Trends

www.childtrends.org Twitter/childtrends www.facebook.com/childtrends:

April/May 2014

http://www.childtrends.org/

http://www.twitter.com/childtrends

http://www.twitter.com/childtrends

http://www.facebook.com/childtrends


Presenter Information

Lori Delale-O’ConnorScientist

Child Trends Having difficulty with the sound? Please send us a message in the chat box.


Roadmap

Part I: Introductions and Overview of the National CASA project In-depth interviews Survey development, testing, and fielding

Part 2: Survey Results Survey completion demographics Survey data Potential “priority areas” identified

Part 3: Process of Becoming Evidenced-Based What it means to be evidence-based “Arrow” and process National CASA’s work to becoming evidence-based


What you will walk away with…….

• Knowledge about the current data collection activity of CASA/GAL programs across the country

• Understanding of what it means to be “Evidence Based” – and why it matters for your CASA/GAL program

• Insight into how National CASA is moving forward with performance measurement and working to build an evidence base


Part I: Introductions and Overview

Child Trends’ Mission

To improve outcomes for children by providing research and analysis to the people and institutions whose decisions and actions affect children.


7


http://www.childtrends.org/_catdisp_page.cfm?LID=2002





Research Areas

8

FATHERHOOD AND PARENTING

EARLY CHILDHOOD

CHILD WELFAREINDICATORS

EDUCATION (K-12)

DATA DEVELOPMENT

YOUTH DEVELOPMENT FERTILITY AND FAMILY STRUCTURE


Develop and track data on children and youth Help to design, implement, and evaluate child

and youth-serving programs Analyze and synthesize data and evidence Provide training and technical assistance

A key focus of our work is identifying evidence-based programs, practices, and policies

Child Trends’ Work

9


National CASA/Child Trends Performance Measurement

Goal: Develop key measures to help National CASA better understand performance and move toward developing an evidence base.

Phase 1: Scan of current data collection and use (Sept-March) In-depth interviews Survey development & testing Survey fielding

Phase 2: Presentation of findings (April-May)Phase 3: Recommendation of performance measures (May-June)


Part II: Survey Findings


Survey Completion OverviewCategory % Category %Level Urbanicity

local 89 rural 50state 9 suburban/mixed 36state – direct advocacy 2 urban 14

Age (in years) Program Model

10 or fewer 20 friend of the Court 5611 – 20 36 guardian ad litem 2821 or more 44 friend of the Court/Attorney team 10

Size (# of children served) Type of Agency

50 or less 24 independent non-profit 6251 – 300 59 part of a government agency 21301 or more 18 part of a non-profit umbrella agency 16


Topics Addressed in the Survey

• Data collection responsibility and burden • Data collection systems

• Systems used• Meeting needs

• Data use • Priorities for future data collection


Data collection and level of burdenType of staff % Hours per month

CASA supervisors/coordinators 70 20

CASA administrators 56 21

Volunteer 16 4

Assistant or support staff 9 31

Other 5 19

CASA supervisors/coordinators

CASA administrators

volunteers

assistant/support staff

2

2

14

7

15

14

14

12

54

48

49

38

21

28

18

31

8

9

6

12

Don't know Very burdensome Somewhat burdensom Not very burdensome Not burdensome at all


Data collection systems and satisfaction Current Data System % Current Data System %

CASA Manager 39 Other 4

Microsoft Excel 36 Optima 4

COMET 16 CASA Tracker 4

State system 12 Own system 2

Efforts to Outcomes (ETO) 6 Microsoft Access 2

CASA Manager

COMET

Excel

State System

1

2

10

27

20

24

55

59

60

56

35

15

18

20

Not at all Not very well Somewhat well Very well


Data use Activity %

Reporting requirements 90

Demonstrate effectiveness 73

Make program improvements 62

Fundraising 58

Service improvements 43


Data support needs

Area of desired support %

Training on how to analyze data 53

Guidance on collecting data 46

Funding for data collection/analysis software 44

Guidance in identifying outcomes 44

Funding for personnel 40


Data priorities

Domain %

Permanency 60

Well-being 44

Placements 44

Education 39

Mental health 31


Participant Feedback


Participation

What do you think about the priority areas identified in terms of their alignment with CASA?

• I think they are an accurate reflection of CASA’s mission and the reach of a volunteer.

• I think one or more of them are outside of CASA’s mission and the reach of a volunteer.

• I think one or more of them are outside of CASA’s mission, but a volunteer may impact these areas anyway.


Participation

Do you/local programs you work with already collect data on measures in the priority areas?

• Yes, I/they collect data on ALL of these measures.• Yes, I/they collect data on SOME of these measures.• No, I/they do not collect data on ANY of these measures.


Participation

What potential challenges do you see to getting data on measures in the priority areas?

• My program/the programs I work with does not have access to the data needed for these measures.

• My program/the programs I work with would need to administer additional assessments (i.e., mental health assessments/screenings) in order to capture data on these measures.

• I am not clear on what type of data are needed for these measures.• Programs may interpret the meaning of these measures differently,

and therefore data quality would be inconsistent.


Participation

What potential challenges do you see to reporting on measures in the identified priority areas?

• It is a challenge to get the data for these measures, and therefore my program/the programs I work with would most likely be unable to report on these measures.

• I/the programs I work with can get the data on these measures, but I am not confident I/they would be able to report on them.

• There is a danger that programs could interpret the measures differently, and therefore the data would not be as useful.


Part III: Becoming Evidence Based


Evidence-based programs: – Interventions that have been found to have positive effects

or impacts on targeted outcomes Evidence-based practices:– Intervention strategies or program practices that have

been found to improve outcomes for children and youth Evidence-based policies:– Regulations and laws that have been found to have

positive effects on child and youth outcomes

What We Mean by “Evidence-Based”?

25


High

Low

Quasi-experimental

Experimental

Replicated Experimental

Evidence-informed, non-experimental

Levels of Evidence

26


Collect Data on Performance & Outcome Measures

Conduct Needs

Assessment

Identify Your Population

Becoming Evidence-Based

Targeting

Conduct a Quasi-Experimental Outcomes Evaluation

Conduct an Implementation Evaluation

Select Intervention, Develop Logic Model &

Identify Indicators

Implement Program/Approach & Conduct Ongoing Performance

Management

Conduct a Randomized-Controlled Impact Evaluation

27


Participant Feedback


Participation

Where is your CASA/GAL program/the programs that you work with on the arrow?

• We/They are not on the arrow at all.• We/They are in the preliminary stages of performing a needs

assessment and identifying target population.• We/They are in the more secondary stage of developing a logic model

and identifying indicators.• We/They are in the conducting ongoing performance management

phase.• We/They are in the more advanced stage of conducting an

implementation evaluation, quasi-experimental evaluation, or a randomized control trial (RCT).


Food for thought—post-webinar survey question

Where do you see your CASA/GAL program/the programs you work with being able to move on the arrow?

• Not any further than we/they currently are.• Performing a needs assessment and identifying target

population.• Developing a logic model and identifying indicators.• Conducting ongoing performance management.• Conducting an implementation evaluation, quasi-

experimental evaluation, or a randomized control trial (RCT).


Building the Evidence Base


• Set up a clear path to success or a roadmap for achieving program goals

• Provide a visual test of whether inputs, and activities align with outputs and outcomes

• Help to get everyone “on the same page”

• Facilitate performance management and evaluation

Benefits of Logic Models

32

33

OutcomesOutputsActivities

Inputs Outputs Outcomes

Program Component

Inputs Outputs Outcomes

Program Mission

Program Component

Logic Model Example

Inputs

Activities Activities

34

Logic Model Example

Outcomes Outcomes Outcomes

Outputs

Inputs

Activities

Inputs

ActivitiesActivities

OutputsOutputs


Outcomes and Outputs

Outcomes: The observed changes in attitudes, knowledge, or behavior among program

participants that result from a program / service / initiative.

Output: Measures of evidence of program implementation or quality.


Outputs are sometimes confused with outcomes

Outputs Child Outcomes

Numbers of volunteersNumber of children served

Changes in healthChanges in behaviorChanges in knowledgeChanges in placementChanges in permanency


What exactly is impact? Researcher definition of impact: Linking (through study/research) the

observed outcomes to a program / service / initiative. Does the program lead to outcomes? Can we control for as many other explanations as possible?

Impact could also mean the very long term changes attributable to a program / service / initiative. Child welfare advocates – do children who return home to their

parents STAY with their parents? Or do they re-enter foster care at a later date or repeatedly?

Impact could also mean the collective outcomes across multiple programs / services / initiatives. Does the same program offered in multiple locations have outcomes

that can be aggregated across the locations?


How does measuring impact differ?

Court Appointed Special Advocates

Program Outcome(s)

Decrease in length of stay in

foster care

Decrease in trauma

symptoms

Change in social worker at public child

welfare agency

Initiative at local

child welfare agency

Alternative programming

Are other conditions or programs causing these changes?


National CASA’s next steps for consideration…

• Developing a logic model• Identifying a small set of “key measures” for all CASA/GAL

programs• Addressing the evaluation challenges of networks structured

like National CASA• Conveying potential benefits and collaborating across the

network to build an evidence base


Contact Information

Lori Delale-O’ConnorE-mail: [email protected]

Sarah Catherine WilliamsE-mail: [email protected]

mailto:[email protected]

mailto:[email protected]

www.childtrends.org

Research in the Service of Children

56


Documents

1 National CASA/Child Trends Performance Measurement Lori Delale-O’Connor and Sarah Catherine Williams Welcome! We will begin our webinar at 9:00 am PT