Measuring Implementation of a Coaching-based Professional

Development ProgramDouglas Powell

Purdue University

Levels of Implementation in PD

• Teacher: To what extent do teachers implement the curriculum or instructional practices promoted in a PD program?

• PD: To what extent are features of a PD program implemented as designed?

• PD outcome research typically gives more attention to fidelity at the teacher level.

PD Program Implementation Data Are Integral to:

• determining whether PD components occurred as planned

• specifying what happened in individualized work with teachers

• examining “what works?” in multi-component PD programs

• improving PD by systematically identifying features that are challenging to implement

Classroom Links to Early Literacy

• Early literacy PD program examined in a randomized controlled trial with 88 Head Start teachers and 759 children (Powell, Diamond, Burchinal, & Koehler, 2010)

• Individualized coaching with teachers for one semester: on-site visit or technologically-mediated (remote)

• Positive PD effects on:– classroom supports for early literacy and language

development (d = 0.92)– children’s letter knowledge, blending skills, writing, and

concepts about print (ds = 0.17 to 0.29)• No differential effects of remote vs. on-site coaching

PD Components

• Content focus on children’s oral language and code-focused skills introduced in 2-day workshop

• Coaching protocol: observe-assess-recommend sequence

• On site coaching condition– Coach observed for about 90 min, consulted with teacher

for about 30 min• Remote coaching condition

– Coach feedback on teacher-submitted videotape of instruction, including links to video exemplars

– Teacher use of case-based hypermedia resource (97 video exemplars, 33 articles)

Data Sources for Examining PD Implementation

• Records of on-site coaching sessions– signed by teacher and coach

– randomly-selected 20% of sessions were audiotaped

• Records of teacher videotape submissions

• Transcripts of coach feedback on teacher-submitted videotapes

• Web usage logs (remote coaching)– browser actions with case-based hypermedia resource

Overview of Implementation Findings

• Frequency of participation in PD– On-site coaching

• 7.07 (SD = 0.35) sessions • ave length of 180 (SD = 31.9) min, including 35.5 min in

consultation– Remote coaching

• 7.0 (SD = 1.53) sessions (videotape submission, feedback)• ave videotape length of 15.45 (SD = 6.05) min • Coaches provided feedback on ave of 5.80 (SD = 2.92) segments

– Teachers in both conditions viewed on average a similar number of demonstrations of evidence-based practice

• Remote: 17.03 (SD = 24.86) video exemplars • On-site: 10.92 (SD = 5.92) video exemplars, 5.93 (SD = 7.96) coach

modeling in classroom

Overview of Implementation Findings

• Content focus of coaching– 36% oral language– 60% code-focused (36% letter knowledge, 24% phonological

awareness)

– 4% of sessions had relatively equal focus– No differences between on-site and remote conditions in

main content focus (p = .11)– similar number of different literacy topics pursued in on-

site (M = 7.9) and remote (M = 7.7) – 1.7% of all on-site sessions (n = 288) and 2.9% of feedback

in all remote sessions (n = 301) included one or more coach recommendations not related to PD content

A Closer Look at Teacher Use of Hypermedia Resource (remote coaching)

• Visits to content pages were unevenly distributed across literacy topics – 9 of 16 cases received majority of all page visits

• Selective engagement of resources– Actual page visits exceeded expected page visits for

content re: individualization and responsiveness

– 49.8% of video clips viewed by teachers were also suggested by their coach (embedded links in feedback on teacher-submitted tape)

– Limited use of articles

Some Implications & Interpretations

• Use of existing records to measure program implementation is economical and informative

• A focus on literacy was maintained• Technologically-mediated coaching can approximate

a traditional on-site approach (but with more variability in frequency of participation)

• Content emphases in coaching may explain child outcomes (code-focused but not oral language effects)

• Measuring teacher reactions to coach feedback is challenging yet potentially important

For Further Information

• Powell, D. R., Diamond, K. E., Burchinal, M. R., & Koehler, M. J. (2010). Effects of an early literacy professional development intervention on Head Start teachers and children. Journal of Educational Psychology, 102, 299-312.

• Powell, D. R., Diamond, K. E., & Koehler, M. J. (2010). Use of a case-based hypermedia resource in an early literacy coaching intervention with pre-kindergarten teachers. Topics in Early Childhood Special Education, 29, 239-249.

• powelld@purdue.edu or kdiamond@purdue.edu• Supported by grant R305M040167 from the Institute of

Education Sciences, U.S. Department of Education