Measuring Implementation and Intervention Fidelity in Scaling Up: Processes, Tools, and Benefits Carol M. Trivette, Ph.D. Karen Blase, Ph.D. Frank Porter

Measuring Implementation and Intervention Fidelity in Scaling

Up:Processes, Tools, and Benefits

Carol M. Trivette, Ph.D.

Karen Blase, Ph.D.

Frank Porter Graham Child Development InstituteUNC at Chapel Hill. North Carolina

OSEPProject

Directors ConferenceJuly 2012

Center for Early Literacy Learning Center on Everyday Child Language Learning

Orelena Hawks Puckett InstituteAsheville and Morganton, North Carolina

Through the CELL and SISEP Lens….

GOALS: Describe differences between intervention and implementation processesDescribe the differences between intervention fidelity and implementation fidelityIllustrate how a “cascading logic model” details implementation strategies and outcomes at different levelsConnect implementation fidelity to improved intervention fidelity.

Intervention Practices and Fidelity

Intervention practices include methods and strategies used by intervention agents (teachers, therapists, clinicians, parents, etc.) to affect changes or produce desired outcomes in a targeted population or group of recipients (e.g., children with disabilities).

Therefore, intervention fidelity refers to the degree to which evidence-based intervention practices are used as intended by early childhood practitioners, teachers, parents, or other intervention agents and have expected or intended.

Dunst, C. J., Trivette, C. M., McInerney, M., Holland-Coviello, R., Masiello, T., Helsel, F., & Robyak, A. (2008). Measuring training and practice fidelity in capacity-building scaling-up initiatives. CELLpapers, 3(1), 1-11. Available at http://www.earlyliteracylearning.org/cellpapers/cellpapers_v3_n1.pdf.

Implementation Practices and Fidelity

Implementation practices include methods and procedures used by implementation agents (trainers, coaches, instructors, supervisors, etc.) to promote interventionists’ use of evidence-based intervention practices.

Therefore implementation fidelity refers to the degree to which coaching, in-service training, instruction, or any other kind of evidence-based professional development practice is implemented as intended and has the effect of promoting the adoption and use of evidence-based intervention practices.

Trivette, C. M., & Dunst, C. J. (2011, August). Implementation with fidelity: How to get changes in early childhood classroom practices. Paper presented at the Global Implementation Conference, Washington, DC.

© Fixsen & Blase, 2008

All Students & Families

All Students & Families

School Teachers and

Staff

School Teachers and

Staff

StateDepartment Leadership

StateDepartment Leadership

Implementation-Skilled Workforce

Develop staff with special implementation skills

Re

-Pu

rpo

se

Regional Implementation

Teams

Regional Implementation

Teams

Adult interactions produce Student benefits

Population Intervention

Strategies (WHAT)

Intervention

Outcomes

All students and/or young children

Teachers and staff fully and effectively use evidence-based practices

Improved academic, behavioral outcomes

SISEP Cascading TA Logic Model


Strategies (WHAT)

Intervention

Outcomes




Populations Implementation Strategies

(HOW)

Implementation Outcomes



Strategies (WHAT)

Intervention

Outcomes





(HOW)


Teachers and Staff Provision of skillful, timely selection, training, coaching, performance assessments in supportive administrative environments organized by Districts, Schools, or Regions

Teachers and staff competently and confidently use evidence-based practices

~ Intervention Fidelity~


Intervention Fidelity: The degree to which evidence-based intervention practices are used as intended by early childhood practitioners, teachers, parents, or other intervention agents and have expected or intended. Dunst, C. J., Trivette, C. M., McInerney, M., Holland-Coviello, R., Masiello, T., Helsel, F., & Robyak, A. (2008).


Strategies (WHAT)

Intervention

Outcomes





(HOW)





Building and District staff Provision of services from the Regional Implementation Team to LEA (District and School staff)

Skillful, timely selection, training, coaching, performance assessments and supportive administrative environments for teachers and staff

~Implementation Fidelity~


Implementation Fidelity: The degree to which coaching, in-service training,instruction, or any other kind of evidence-based professional development practice is implemented as intended and has the effect of promoting the adoption and use of evidence-based intervention practices. Trivette, C. M., & Dunst, C. J. (2011, August). Implementation with fidelity: How to get changes in early childhood classroom practices. Paper presented at the Global Implementation Conference, Washington, DC.


Strategies (WHAT)

Intervention

Outcomes





(HOW)





Local and District staff Provision of services from the Regional Implementation Team to LEA (District and School staff)


~ Implementation Fidelity

Regional Implementation Team Members

Skillful leadership by State Transformation Specialists, supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams.

Timely and skillful provision of services by Regional Implementation Teams to LEA



Strategies (WHAT)

Intervention

Outcomes





(HOW)





Building, District, Regional Staff Provision of services from the Regional Implementation Team to LEA (District and School staff) and Regional entities



Leadership & use of “active implementation frameworks” by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop Regional Implementation Teams.

Timely and skillful provision of services by Regional Implementation Teams to LEA, Regional Entities

State Transformation Specialists

State Management Team

1st Generation Regional Implementation Team

Intensive technical assistance by SISEP to

develop implementation capacity at multiple levels

Skillful leadership & use of “active implementation” frameworks by State Transformation Specialists, and supportive administrative environments developed by State Management Team to develop multiple Regional Implementation Teams.


Strategies (WHAT)

Intervention

Outcomes





(HOW)

















Strategies (WHAT)

Intervention

Outcomes





(HOW)
















Implementation Drivers Best Practices & DCA

Implementation Drivers Best Practices & DCA

Regional Implementation Team Capacity Assessment (RIT CA)

State Capacity Assessment (SCA & RIT CA)

SCA & RIT CA

All Measures Above

RIT Capacity Assessment

Implementation and Intervention

No intervention practice, no matter its evidence base, is likely to be adopted and used if the implementation methods used to teach or train practitioners to use the practice are not themselves effective.

Therefore concern for the characteristics of implementation practices that are associated with optimal learner and practitioner outcomes should be of paramount importance as part of implementation research.

Dunst, C. J., & Trivette, C. M. (2009). Let's be PALS: An evidence-based approach to professional development. Infants and Young Children, 22(3), 164-175. doi:10.1097/IYC.0b013e3181abe16

Center for Early Literacy LearningCenter on Early Childhood Language Learning

CELL – OSPE TA&D Project – Children 0-3 with disabilities• Developed evidence-based early literacy practices for young

children with disabilities (web site)• Worked with 6 states to scale up these early literacy practices• Worked with state teams to develop training• State teams trained trainers or teachers or practitioners• Teachers and parents provided intervention with children

CECLL – OSEP Model Demonstration – Children 0-3 with disabilities

• Provided training to Part C EI providers in 3 states• Part C provided coaching with parents• Parents provided intervention to children

Tiered Approach to Training to Promote Adoption and Sustained Use of CELL

One

Many

StateResource

(Leadership) Team

Regional Resource(Cohorts) Team

Level I

Local/Regional TrainersLevel II

End Users – Teachers or ParentsInterventions to Children

Research Synthesis of Adult Learning Studiesa

• Research synthesis of studies of accelerated learning, coaching, guided design, and just-in-time training

• 58 randomized control design studies

• 2,095 experimental group participants and 2,213 control or comparison group participants

• Combination of studies in university and nonuniversity settings

• Learner outcomes included learner knowledge, practices, skills, attitudes, and self-efficacy beliefs

• The influence of the adult learning methods on the learner outcomes was estimated by weighted Cohen’s d effect sizes for the differences on the post test scores for the intervention vs. nonintervention group participants

a Dunst, C.J., Trivette, C.M., & Hamby, D.W. (2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. International Journal of Continuing Education and Lifelong Learning, 3(1), 91-112.

Characteristics Used to Code and Evaluate the Implementation Studiesa

Planning

Introduce Engage the learner in a preview of the material, knowledge, or practice that is the focus of instruction or training

Illustrate Demonstrate or illustrate the use or applicability of the material, knowledge, or practice for the learner

Application

Practice Engage the learner in the use of the material, knowledge, or practice

Evaluate Engage the learner in a process of evaluating the consequence or outcome of the application of the material, knowledge, or practice

Deep UnderstandingReflection Engage the learner in self-assessment of his or her acquisition of

knowledge and skills as a basis for identifying “next steps” in the learning process

Mastery Engage the learner in a process of assessing his or her experience in the context of some conceptual or practical model or framework, or some external set of performance standards or criteria

a Donovan, M. et al. (Eds.) (1999). How people learn. Washington, DC: National Academy Press.

Most Effective Adult Learning Method Practices

Characteristic Practice Mean Effect SizeIntroduction Out-of-class learner activities/self-instruction 0.64

Classroom/workshop presentations 0.63

Pre-class learner exercises 0.54

Illustration Trainer role playing/simulations 0.55

Learner informed input 0.53

Practicing Real-life learner application 0.94

Real-life learner application/role playing 0.86

Evaluation Self assessment of strengths/weaknesses 0.94

Reflection Identify performance-improvement goals 1.27

Journaling/behavior suggestions 0.82

Mastery Standards-based assessment 0.86

Cumulative Effects of Different Combinations of the Most Effective Adult Learning Method Practices

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 2 3 4 5 6

MEA

N E

FFEC

T SI

ZE

NUMBER OF PRACTICES

0 1 2 3 4 5

NUMBER OF ADULT LEARNING METHOD PRACTICES

PALS(Participatory Adult Learning Strategy)

PLAN

APPLICATIONRECYCLE Active

Learner Involvement

Reflection and Mastery

Practice and Evaluate

Introduce and Illustrate

Identify Next Steps in the

Learning Process

INFORMED UNDERSTANDING

CELL Trainer Feedback Form

Content Items1. The importance of literacy-rich environments was well explained by the trainer.2.The key characteristics of responsive teaching were described and illustrated in ways that made the instructional practice easy to understand.Training-Method Items3.The importance of active participation of trainees as a method was made explicitly clear.4.The training method was described in enough detail to understand its key elements.Practice Items5.A sufficient amount of time was devoted to each component of the CELL literacy model.6.The activities for engaging trainees in learning the CELL literacy practices were especially helpful.

Hypothesized RelationshipsAmong the Fidelity Measures

• Variations in implementation fidelity should be related to variations in intervention fidelity. Tests of the hypothesis include evaluation of the relative importance of the quantity and quality of implementation fidelity, and the interactions between the types and elements of fidelity.

• Variations in intervention fidelity should be related to variations in practice outcomes. Tests of the hypothesis include evaluation of the relative importance of the quantity and quality of intervention fidelity, and the interactions between the types and elements of fidelity.

26

Relationship Between Implementation Fidelity and Child Communication Development

PurposeEvaluate the relationship between variations in the fidelity

of parents’ use of the CECLL practices and changes in the children’s early communication development

ParticipantsSixty-three infants and toddlers 8 to 40 months of age and

their parents

Outcome MeasuresFidelity: A weekly parent-completed scale of child participation

in and the characteristics of everyday learning activitiesChild Outcome: Early Communication Development Scale

27

CECLL StaffTraining of

Practitioners

PractitionerImplementation

with Parent

Parent Use of the

Intervention With the Child

Enhanced Language Learning

ImplementationFidelity

Intervention Fidelity

ChildOutcomes

Hypothesized Relationships Between the Implementation and Intervention Fidelity Measures and Child Outcomes

28

Parent-Completed Fidelity Measure

The scale used to assess intervention fidelity captured two aspects of the use of the CECLL practices:

•Quantity of child participation in everyday activities (number of settings, number of activities within settings, frequency of participation)

•Quality of child participation in everyday activities (development-enhancing characteristics of the child learning experiences and opportunities)

29

Influences of Fidelityon Child Communication Development

5

6

7

8

9

10

6 12 18 20 24 30 36

CHIL

D C

OM

MU

NIC

ATI

ON

LO

GIT

SCO

RE

CHILD AGE (Months)

Low Fidelity

High Fidelity

5

6

7

8

9

10

6 12 18 20 24 30 36

CHIL

D C

OM

MU

NIC

ATI

ON

LO

GIT

SCO

RE

CHILD AGE (Months)

Low Fidelity

High Fidelity

Cohen’s d = 0.62 for differences in the slopes

Summary

• Intervention Fidelity Matters to Achieve Outcomes

• Implementation Fidelity Matters to Achieve Intervention Fidelity

• Multiple levels of the system need to be impacted to achieve these outcomes

• Cascading logic models and measures at every level can help develop effective implementation strategies and intervention outcomes.

Documents

Measuring Implementation and Intervention Fidelity in Scaling Up: Processes, Tools, and Benefits Carol M. Trivette, Ph.D. Karen Blase, Ph.D. Frank Porter