Using Cost-Effective Processes to Develop Large …...Using Cost-Effective Processes to Develop...

Using Cost-Effective Processes to Develop Large-Scale Data-Driven Continuous Improvement Systems for Local Programs

Laurie A. Van Egeren, Jamie Wu, Michigan State University Angelina Garner, Charles Smith, David P. Weikert Center for Youth Program Quality

American Evaluation Association Minneapolis, MN October 26, 2012

21st Century Community Learning Centers

Michigan 21st CCLC

In 2011-2012:

TACSS is Quality Assurance for: • 320 elementary, middle and

high schools sites • 40,000+ students • Over $50M investment

Evaluation Scope in 2003

Al I modern domestic dogs are descend ants of the Gray Wolf

Evaluation Scope Now

Continuous Improvement

Assess

Plan Improve

Hmprowment

Quality Improvement Support System

MDE State Education Agency

MSU State Evaluator

TACSS Program Quality

Improvement

Standard Indicators of Quality

Cost-Effective Data Reports

Cost-Effective Data Reports

Capacity-Building for Data Use

1. Standard quality indicators

Identify Indicators MDE

Literature TACSS

Advisory MSUBoard

I.II 1. Instructional Context

2. Organizational Context

3. Positive Relationships

Select Data Sources

Youth survey Parent survey Staff survey

Supervisor survey

Administrator report

Observational program self-assessment

Attendance/ activity data

(web)

School outcomes data

Comparability: Weighted 10- pt scale

Measure Weight Academic activity participation 1.5 Homework help/tutoring participation for academically at-risk students

1.5 Academic enrichment participation 1.5 Activities informed by grade-level content standards 1 Student reports of academic support quality 1.5 Academics is top priority .5 Supervisor connection to school-day content 1 Staff connection to school-day content 1.5 Total 10

10

Instructional Context State Grantee

5.1 6.5 4.7 4.6 5.94.4 6.9 4.3 4.1 5.20

2

4

6

8

Enrollment/ Academic Enrichment Connections Instructional Retention Content Content to School Quality

0

2

4

6

8

10 10-pt scale for comparability

II State

Grantee – State Comparisons

State Grantee

Grantee – State Comparisons

Site variations at a glance

Go deeper – comparisons on measures State Org Sites

Go deeper – comparisons on measures

0-10 pt indicator score

Measure scores however defined

Go deeper – comparisons on measures Sites

Uh oh…

Even deeper – item data for sites

1-gradle stud1ents at thi.s s ite and statewide w·ho agre 1ed or ng.a.gement in the program.

ent of Students who Agrered or S

Your si'te Statewide 46°/o 77°/o 50°/o 23°/o s1°10

62°/o 77°/o

Yo1ur site co . a .. to

state Very· low

Very· low

Very low

Very· low

!·nts_ Statewide numbers are for students in the san1e grade' se1Ved b ·

Uh oh…

MI 21•1 Century Commu nity Lellrning Ce nters Leading Indicator s Report

Citi' P ublic Schoolc Peril 7, 2010

Overa• Ft lcture Compared to :State

• -O rtp11iut ional Cont ext

si.t~itv lt!roimt~ Gtart.. St• SUl'f eo,,w,, .. ,>11, rttentOI' ll'III\IQf!n'el'tAH.IWII e:>Pffl'"(e/ ~·ene,i

•="1

'

Instructional Context

Acade mic Instructio n 5 .6 5 .3 5.2

Cc nnection to schoo lb 7l.°A. 80% Yes Formal poUcy t'or connecting with l '1ach u s

129( "" No

FUl -time Sit e ( oord inato? 631< 100% Yes Ac~d"mic ~ct ivity

811< 70% 67% participatiOnc

Provi~ion of homework hel p( 43% 63% 67%

Provis ion ot a cademic 53% 47% 5 1%

~nrich~ntc

Provis ion of t utorin, ~ 11% 0% 0% A.t:i::1Je111k.")b lup pli1..11ilyd 861< 77% 60%

Ce rtif ieo teache rs provide 4:J':-E °" ~

ct1,;dJ l:'111i1,;,up pur l<

St 11 riP nt rf'pnn,; n f ;tr:MPmic 53% 59% 50%

support quality'

5 .8 , .. ... 5 .1

Yes Yes Yes No

No Ne No No

Yes Yes Yes Yes

89" 82% LS% 761<

89% 71% 36% 60%

"" 78% 33% 68% -Q)(, 0% 0% 0)(; n. ...................... _ ... _.,..r .......................................... .. -....---........ --~-~ .... ......

701< 50% 100% 67%

m, "" ~ "" 81% 50% 27% 88%

\l- ...... - - ..,.~-n.. .............. ,._...,,~.-1:-................................ . =..----....---.......... --..---.-_ ....... ,_.... .... ,...,_. ......... _ _,, __

inoo•--•-• .-..:'.='--·.:::::~------::. ... ..;::.-.--

o..---.c:..,,c..~ ... o.--n...w. ................. .,, ...... t" ................. _ .... ____ ......... ==-~...._ ................ _,---.,., .. ____

'"'*" ..,.,._"'___... ... 0.- ...._...___..._

- ~-NM~ ·-·--· -~--==:-s:... ... .:=..--.. ~-.. :'-..--., .. -----..--~----··-.....-

Grantee Summary

Site Comparisons

Site Details

2. Cost-effective local report production

Assumption: You’re analyzing

data anyway

Process 1. Collect data 2. Develop report template in Word

3. Analyze data to match [decisions] 4. Create excel or .csv file of data 5. Use Word mail merge to populate

reports (tweak if necessary) 6. Voila!

Step 1: Collect Data

Youth survey Parent survey Staff survey

Supervisor survey

Administrator report

Observational program self-assessment

Attendance/ activity data

(web)

School outcomes data

Step 1: Collect Data

Youth survey Parent survey Staff survey

Supervisor survey

Administrator report

Observational program self-assessment

Attendance/ activity data

(web)

School outcomes data

MI 21•1 Century Commu nity Lellrning Ce nters Leading Indicator s Report

Citi' P ublic Schoolc Peril 7, 2010

Overa• Ft lcture Compared to :State

• -O rtp11iut ional Cont ext

si.t~itv lt!roimt~ Gtart.. St• SUl'f eo,,w,, .. ,>11, rttentOI' ll'III\IQf!n'el'tAH.IWII e:>Pffl'"(e/ ~·ene,i

•="1

'

Instructional Context

Acade mic Instructio n 5 .6 5 .3 5.2

Cc nnection to schoo lb 7l.°A. 80% Yes Formal poUcy t'or connecting with l '1ach u s

129( "" No

FUl -time Sit e ( oord inato? 631< 100% Yes Ac~d"mic ~ct ivity

811< 70% 67% participatiOnc

Provi~ion of homework hel p( 43% 63% 67%

Provis ion ot a cademic 53% 47% 5 1%

~nrich~ntc

Provis ion of t utorin, ~ 11% 0% 0% A.t:i::1Je111k.")b lup pli1..11ilyd 861< 77% 60%

Ce rtif ieo teache rs provide 4:J':-E °" ~

ct1,;dJ l:'111i1,;,up pur l<

St 11 riP nt rf'pnn,; n f ;tr:MPmic 53% 59% 50%

support quality'

5 .8 , .. ... 5 .1

Yes Yes Yes No

No Ne No No

Yes Yes Yes Yes

89" 82% LS% 761<

89% 71% 36% 60%

"" 78% 33% 68% -Q)(, 0% 0% 0)(; n. ...................... _ ... _.,..r .......................................... .. -....---........ --~-~ .... ......

701< 50% 100% 67%

m, "" ~ "" 81% 50% 27% 88%

\l- ...... - - ..,.~-n.. .............. ,._...,,~.-1:-................................ . =..----....---.......... --..---.-_ ....... ,_.... .... ,...,_. ......... _ _,, __

inoo•--•-• .-..:'.='--·.:::::~------::. ... ..;::.-.--

o..---.c:..,,c..~ ... o.--n...w. ................. .,, ...... t" ................. _ .... ____ ......... ==-~...._ ................ _,---.,., .. ____

'"'*" ..,.,._"'___... ... 0.- ...._...___..._

- ~-NM~ ·-·--· -~--==:-s:... ... .:=..--.. ~-.. :'-..--., .. -----..--~----··-.....-

Grantee Step 2: Develop report Summary template

Site Comparisons

Site Details

Step 3: Analyze for report – Decisions! Indicator MI Org 1 2 3

1.4 Connection to School Day 4.7 4.1 4.4 6.9 4.2

Formal policy for connecting with school daya,b

69% 75% 75% 75% 75%

Supervisor communication with schoole 46% 11% 0% 0%

Staff communication with schoold 27% 21% 40% 25%

School investment in programb 61% 80% Yes Yes Yes

Step 3: Analyze for report – Decisions! Indicator MI Org 1 2 3

1.4 Connection to School Day 4.7 4.1 4.4 6.9 4.2

Formal policy for connecting with school daya,b

69% 75% 75% 75% 75%

Supervisor communication with schoole

46% 11% 0% I

in

mpute to get dicator

0%

Staff communication with schoold 27% 21% 40% score 25%

School investment in programb 61% 80% Yes Yes Yes

Step 3. Analyze for report – Decisions!

What is minimum N

(varies)

How determine cut-offs?

Use syntax!

Step 4. Create Excel or .csv file

Step 5. Mail merge Excel file into template

Step 5. Mail merge Excel file into template

3. Capacity-building for data use

MICHIGAN~

EdDeparonenc1t~~ ._.,, ) uca ton

A Decade in the Making

Why TACCS?

TACSS Goals

1. Grow a culture of performance accountability

2. Develop a low-stakes infrastructure for continuous quality improvement

3. Improve overall quality of 21st CCLC services and start up for new sites

4. Improve the instructional quality for young people

Important Concepts Underpinning TACSS P

OLI

CY

SE

TTIN

G

OR

GA

NIZ

ATIO

N S

ETT

ING

INS

TRU

CTI

ON

AL

SE

TTIN

G

Instructional Quality

Management Skills for Continuous Quality Improvement

Low Stakes Accountability

TA/Coach Values & Methods

Important Concepts Underpinning TACSS Management Skills for CIP

PLAN

IMPROVE ASSESS

• Lead a team to • Carry out plan to assess the quality improveof instruction • Lead team to create instructional quality

• Provide real-time an improvement • Monitor progressstaff performance plan based on data and repeat feedback • Select align

methods training for direct staff

Important Concepts Underpinning TACSS High Stakes Accountability Policy

Objective Data Publicity Action Improved

Outcomes

Important Concepts Underpinning TACSS

Objective Data

MeaningfulInformation

Action/ Expertise

Improved Outcomes

Low Stakes Accountabilities

LearningCommunity

ImprovementEfforts

Low Stakes Accountability Policy for CQI

TACSS Project Model in Detail 5-year project

5.5 FTEs (1 manager, 4 TA/Coaches, 1 support staff)

1 PTE (Contract Coach) Regional

TA Coaches

Improve Service Quality

& Child Outcomes

The TACSS Model Comprehensive Support Sequence

MDE Kickoff Event Introductory

meeting with Grantee

Data Profile assembled

Onsite visit, data profile review, and

prep for TAPlanning Day

Director Interview Regarding CQI Practices & red

flag issues

Team Self Assessment of

Instructional Quality

Planning with Data sessions; Develop

TA-Plan

Maintenance of TA Plan with on-going TA/coach support

Data Driving the System Leading Indicators to Program Improvement

Grantee Profile

Site Profiles

Site Detail

Technical Assistance Plan

• Co-created • Linear/sequential • Accountability • Intentionality • Scheduling • Use of Data to drive decision

making • Living/Working document

Core & Supplemental Services Menu

The TACSS Model

POLICY CONTEXT

ORGANIZATIONAL CONTEXT

INSTRUCTIONAL SETTING

Hig

her I

nten

sity

.

.

.

.

.

Comprehensive Supports are Multi-level

. School district and union issues around staffing

Understanding vendor and partnership relationships

Training for Conflict Resolution

Support continuation grant /renewal

Support program self-assessment

Site visits to provide quality coaching

Lower Intensity

TACSS Calendar Year July Aug Sept Oct Nov Dec Jan Feb March April May June July Aug

Letters Coach's reflection letters Personal invitation to kick off orientation TACSS orientation at Kick off Introductory TACSS meeting

TA Planning Self Assessment process support ( YPQA, PIP) External Assessment scheduling/observation Leading Indicator Introduction/review (PD) External Assessment review Data planning session (support PD to lead staff) Mission is Possible professional development opportunity Monthly follow up communications

To Sum Up •Leading Indicators = roadmap to quality program

•Founded in mass-reported data •Decisions about changes are driven by data

•Technical assistance supports programs to use that data in ways they identify

Questions…