SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

What We Have Learned?Research & Evaluation Presentation

Charles Smith, Stephen Baker, Neil NaftzgerLeadership InstituteThursday, May 9

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

• What does the research say about how quality afterschool programming makes a difference for youth?

• Why should we invest in evaluating the quality of afterschool programs?

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

Four Continuous Improvement Practices4

Organization Level Setting

(Plus 10 hours of TA/coaching for site managers to implement the four CI practices)

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

Theory of Change Template for Afterschool Outcomes8

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

l

2012-2013 Dissemination

Pol

icy

Set

ting

Org

aniz

atio

n S

ettin

gP

oint

of S

ervi

ce

Set

ting

>21,125 StaffEstimate based on mean of 6.5

staff per site in YPQI Study Sample

85 Networks/ Systems

>276,250 Child & Youth

Estimate based on mean daily attendance of 85 youth per day in YPQI

Study Sample

>3250 Sites

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

System Accountabilities: Higher Stakes

Objective Data Publicity Action

Improved Outcome

s

Higher Stakes Accountabilities

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

Objective Data

Meaningful Informatio

n

Action/ Expertise

Improved Outcomes

Lower Stakes Accountabilities

Interpretive Community

•Team Self Assessment •Review external scores

Team Planning and Implementing

•Improvement planning•Performance coaching

Higher Stakes Accountabilities

System Accountabilities: Lower Stakes

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

System Supports for CI Practices

Policy: Eligibility, Targeting, Low/high stakes

Training, TA & Coaching

EvaluationExternal Raters, Program Evaluation

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

Program Objectives

N=100 site managers

Academic Support

Social/civic development

Artistic development

Sports & Recreation

Other (computer skills,

self-esteem, parent workshop, etc.)

Objective 1 (%) 56 29 2 6 7Objective 2 (%) 63 18 3 11 4Objective 3 (%) 27 32 10 27 5

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

Content Offerings at Baseline

 Percentage of sites

(N=87 site managers)Example program offerings

Leadership 97 Planning team for events, youth advisory boardReading 96 Vowels, spellingLife Skills 95 Discuss race and culture tensionArt 93 Scrapbooking, clayPhysical Fitness 91 Walleyball, gymTechnology/Computers 90 Typing & navigating skills, video productionMath 89 Ratios, counts re: food drive donationsCommunity Service 89 Christmas gifts to those in sheltersSports 86 Basketball, baseballCreative Writing 78 JournalingCooking 77 Recipes and cookie dipScience 76 Laws of motion, inertia experimentsDance 71 Hip Hop classMusic 71 History of Pop Music, Guitar lessonsTheater 69 Rehearsed 3 acts for musical play AnniePoetry 49 Forms of poetry lessonBuilding/Shop 35 Lego robotics, building a tower out of paper

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

Conclusions from Qualistar Evaluation

Third, the study examined the relationships between the Q-QRIS star ratings, the individual Q-QRIS component measures that yield those ratings, and two commonly used measures of adult-child interaction. (Adult-child interaction is generally agreed to be the most important aspect of childcare quality.) The study found that the star ratings and the Q-QRIS components are generally unrelated to measures of staff -child interaction, but the lack of results may reflect the fact that these latter quality measures were collected in only one classroom per provider.

Fourth, according to the logic model underlying QRISs, an improved child-care environment, characterized by more responsive caregiving and enriched content, will lead to better outcomes for children. The researchers examined relationships between the star ratings, Q-QRIS components, and child cognitive and social outcome measures. The researchers found few relationships between individual Q-QRIS components and child outcomes and virtually none between star ratings and child outcomes.

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

What We Have Learned?Research & Evaluation Presentation

Charles Smith, charles@cypq.org Leadership InstituteThursday, May 9