A Procedure for Assessing Fidelity of Implementation in Experiments Testing Educational...

A Procedure for Assessing Fidelity of Implementation

in Experiments Testing Educational Interventions

Michael C. Nelson1, David S. Cordray1, Chris S. Hulleman2, Catherine L. Darrow1, & Evan C.

Sommer1

1Vanderbilt University, 2James Madison University

Purposes of Paper:

To argue for a model-based approach for assessing implementation fidelity

To provide a template for assessing implementation fidelity that can be used by intervention developers, researchers, and implementers as a standard approach.

Presentation Outline

I. What is implementation fidelity?

II. Why assess implementation fidelity?

III. A five-step process for assessing

implementation fidelity

IV. Concluding points

A Note on Examples:

• Examples are drawn from our review of (mainly) elementary math intervention studies, which we are currently deepening and expanding to other subject areas

• Examples for many areas are imperfect or lacking

• As our argument depends on having good examples of the most complicated cases, we appreciate any valuable examples to which you can refer us (michael.nelson@vanderbilt.edu.)

What Is Implementation Fidelity?

What is implementation fidelity?

Implementation fidelity is the extent to which the intervention has been implemented as expected

Assessing fidelity raises the question: Fidelity to what?

Our answer: Fidelity to the intervention model.

Background in “theory-based evaluations” (e.g., Chen, 1990; Donaldson & Lipsey, 2006)

Why Assess Implementation Fidelity?

Fidelity vs. the Black Box

The intent-to-treat (ITT) experiment identifies the effects of causes:

Assignment to Condition

Treatment “Black Box”

Intervention’s Causal

ProcessesOutcomes

Outcome Measures

Control “Black Box”

Business As UsualCausal

ProcessesOutcomes

Outcome Measures

Fidelity vs. the Black Box

…While fidelity assessment “opens up” the black box to explain the effects of causes:

Intervention “Black Box”

Intervention Component

Mediator OutcomeAssignment to Condition

FidelityMeasure 1

FidelityMeasure 2

Outcome Measure

Fidelity assessment allows us to:

Determine the extent of construct validity and external validity, contributing to generalizability of results

For significant results, describe what exactly did work (actual difference between Tx and C)

For non-significant results, it may explain why beyond simply “the intervention doesn’t work”

Potentially improve understanding of results and future implementation

Limitations of Fidelity Assessment:

Not a causal analysis, but it does provide evidence for answering important questions

Involves secondary questions

Involves secondary questions Field is still developing and validating methods

and tools for measurement and analysis

Involves secondary questions Field is still developing and validating methods

and tools for measurement and analysis Cannot be a specific, one-size-fits-all approach

A Five Step Process for Assessing Fidelity of Implementation

1. Specify the intervention model2. Identify fidelity indices3. Determine index reliability and validity4. Combine fidelity indices*5. Link fidelity measures to outcomes*

*Not always possible or necessary

Step 1: Specify the Intervention Model

The Change Model

• A hypothetical set of constructs and relationships among constructs representing the core components of the intervention and the causal processes that result in outcomes

The Change Model

• Should be based on theory, empirical findings, discussion with developer, actual implementation

The Change Model

• Should be based on theory, empirical findings, discussion with developer, actual implementation

• Start with Change Model because it is sufficiently abstract to be generalizable, but also specifies important components/processes, thus guiding operationalization, measurement, and analysis

Change Model: Generic Example

Teacher training in use of educational

software

Teachers assist students in using

educational software

Improved student learning

Intervention Component

Mediator Outcome

Change Model: Project LINCS

Adapted from Swafford, Jones, and Thornton, 1997

Instruction in student

cognition of geometry

Instruction in geometry

Increase in teacher

knowledge of student

cognition

Increase in teacher

knowledge of geometry

Improved teacher

instructional practice

The Logic Model

The set of resources and activities that operationalize the change model for a particular implementation

The Logic Model

A roadmap for implementation

The Logic Model

A roadmap for implementation

Derived from the change model with input from developer and other sources (literature, implementers, etc.)

Logic Model: Project LINCS

Research seminar on van Hiele

Geometry content course

Increase in teacher knowledge of student

cognition

Increase in teacher knowledge of geometry

Instruction in student cognition of

geometry

Improved teacher

instructional practice

How it is taught

Characteristics teachers display

What is taught

A Note on Models and Analysis:

Recall that one can specify models for both the treatment and control conditions.

The “true” cause is the difference between conditions, as reflected in the model for each.

Using the change model as a guide, one may design equivalent indices for each condition to determine the relative strength of the intervention (Achieved Relative Strength, ARS).

This approach will be discussed in the next presentation (Hulleman).

Steps 2 and 3: Develop Reliable and Valid Fidelity Indices and Apply to the Model

Examples of Fidelity Indices

Self-report surveys Interviews Participant logs Observations Examination of permanent products created

during the implementation process

Index Reliability and Validity

Both are reported inconsistently Report reliability at a minimum, because

unreliable indices cannot be valid Validity is probably best established from pre-

existing information or side studies

Index Reliability and Validity

Both are reported inconsistently Report reliability at a minimum, because

unreliable indices cannot be valid Validity is probably best established from pre-

existing information or side studies We should be as careful in measuring the

cause as we are in measuring its effects!

Selecting Indices

• Guided foremost by the change model: identify core components as those that differ significantly between conditions and upon which the causal processes are thought to depend

Selecting Indices

• Use the logic model to determine fidelity indicator(s) for each change component

Selecting Indices

• Use the logic model to determine fidelity indicator(s) for each change component

• Base the number and type of indices on the nature and importance of each component

Selecting Indices: Project LINCS

Change Model Construct

Logic Model Components

Indicators Indices

Geometry content course

None; Proposed: Teacher attendance, content delivery

None; Proposed: Head count, observation

Instruction of student cognition of geometry

Research seminar van Hiele model

None; Proposed: Teacher attendance, content delivery

None; Proposed: Head count, observation

Increase of teacher knowledge of geometry

None Ability to apply geometry knowledge

Pre/post test of geometry knowledge

Increase of teacher knowledge of student cognition

None Ability to describe student cognition

Pre/post test of van Hiele levels

Improved teacher instructional practice

What is taught Alignment of lesson content with van Hiele levels

Observations

How it is taught

Particular instructional behaviors of teachers

Observations

Characteristics teachers display

Reflecting knowledge of student cognition in planning

Lesson plan task

Step 4: Combining Fidelity Indices*

Why Combine Indices?

*May not be possible for the simplest models *Depends on particular questions

Why Combine Indices?

*May not be possible for the simplest models *Depends on particular questions

Combine within component to assess fidelity to a construct

Combine across components to assess phase of implementation

Combine across model to characterize overall fidelity and facilitate comparison of studies

Some Approaches to Combining Indices:

• Total percentage of steps implemented• Average number of steps implemented

HOWEVER: These approaches may underestimate or overestimate the importance of some components!

• Weighting components based on the intervention model

• Sensitivity analysis

MAP Example

Weighting of training sessions for the MAP intervention

Cordray, et al (Unpublished)

TrainingSession

Month Content Initial Weight

Adjusted Weight

Session 1 September Administration .25 .10

Session 2 October Data use .25 .30

Session 3 November Differentiated Instruction

.25 .50

Session 4 May Growth and planning

.25 .10

Step 5: Linking Fidelity Measures to Outcomes*

Linking Fidelity and Outcomes

• *Not possible in (rare) cases of perfect fidelity (no covariation without variation)

• *Depends on particular questions• Provide evidence supporting the model (or

not)• Identify “weak links” in implementation• Point to opportunities for “boosting” strength• Identify incorrectly-specified components of

the model

Assessment to Instruction (A2i)

Teacher use of web-based software for differentiation of reading instruction

Professional developmentStudents use A2i Teachers use A2i recommendations for grouping and lesson planningStudents improve learning

Measures: Time teachers logged in, observation of instruction, pre/post reading

(Connor, Morrison, Fishman, Schatschneider, and Underwood, 1997)

Assessment to Instruction (A2i)

Used Hierarchical Linear Modeling to analyze Overall effect size of .25 Tx vs. C Pooling Tx+C, teacher time using A2i accounted

for 15% of student performance Since gains were greatest among teachers who

both attended PD and were logged in more, concluded both components were necessary for outcome

(Connor, Morrison, Fishman, Schatschneider, and Underwood, 1997)

Some Other Approaches to Linking from the Literature

• Compare results of hypothesis testing (e.g., ANOVA) when “low fidelity” classrooms are included or excluded

• Correlate overall fidelity index with each student outcome

• Correlate each fidelity indicator with the single outcome

• Calculate Achieved Relative Strength (ARS) and use HLM to link to outcomes

Concluding points

In Summary:

If we do not know what we are testing, we cannot know what the results of our tests mean.

In Summary:

Model-based (change and logic) assessment answers the question “Fidelity to what?”

In Summary:

There is a need for a systematic approach to fidelity assessment, which we describe

In Summary:

Most useful when research designs are able to incorporate this process from early stages

In Summary:

Most useful when research designs are able to incorporate this process from early stages

Additional examples and refinement of measurement and analytical tools are needed

References

Chen, H.T. (1990). Theory-driven evaluation. Thousand Oaks, CA: Sage Publications.Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ:

Erlbaum.Connor, C. M., Morrison, F. M., Fishman, B. J., Schatschneider, C., & Underwood, P. (2007). Algorithm-

guided individualized reading instruction. Science, 315, 464-465.Cook, T. (1985). Postpositivist critical multiplism. In R. L. Shotland & M. M. Marks (Eds.), Social

science and social policy (pp. 21-62). Beverly Hills, CA: Sage.Cordray, D.S. (2007) Assessing Intervention Fidelity in Randomized Field Experiments. Funded Goal

5 proposal to Institute of Education Sciences. Cordray, D.S., Pion, G.M., Dawson, M., and Brandt, C. (2008). The Efficacy of NWEA’s MAP Program.

Institute of Education Sciences funded proposal.Donaldson, S.I., & Lipsey, M.W. (2006). Roles for theory in contemporary evaluation practice:

Developing practical knowledge. In I. Shaw, J.C. Greene, & M.M. Mark (Eds.), The Handbook of Evaluation: Policies, Programs, and Practices (pp. 56-75). London: Sage.

Fuchs, L.S., Fuchs, D., and Karns, K. (2001). Enhancing kindergarteners’ mathematical development: Effects of peer-assisted learning strategies. Elementary School Journal, 101, 495-510.

Fuchs, L. S., Fuchs, D., Yazdian, L, & Powell, S. R. (2002). Enhancing First-Grade Children's Mathematical Development with Peer-Assisted Learning Strategies. School Psychology Review, 31, 569-583.

Gamse, B.C., Jacob, R.T., Megan, H., Boulay, B., Unlu, Fatih, Bozzi, L., Caswell, L., Rodger, C., Smith, W.C., Brigham, N., and Rosenblum, S. (2008). Reading First Impact Study Final Report. Washington, D.C.: Institute of Education Sciences.

References

Ginsburg-Block, M. & Fantuzzo, J. (1997). Reciprocal peer tutoring: An analysis of teacher and student interactions as a function of training and experience. School Psychology Quarterly, 12, 1-16.

Holland, P.W. (1986). Statistics and causal inference. Journal of the American Statistical Association.81(396), 945-960.

Hulleman, C. S., & Cordray, D. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Intervention Effectiveness, 2(1), 88-110.

Hulleman, C.S., Cordray, D.S., Nelson, M.C., Darrow, C.L., & Sommer, E.C. (2009, June). The State of Treatment Fidelity Assessment in Elementary Mathematics Interventions. Poster presented at the annual Institute of Education Sciences Conference, Washington, D.C.

Institute of Education Sciences (2004). Pre-doctoral training grant announcement. Washington, DC: US Department of Education.

Knowlton, L.W. and Phillips, C.C. (2009). The Logic Model Guidebook: Better Strategies for Great Results. Washington, D.C.: Sage.

Kutash, K., Duchnowski, A. J., Sumi, W. C., Rudo, Z. & Harris, K. M. (2002). A school, family, and community collaborative program for children who have emotional disturbances. Journal of Emotional and Behavioral Disorders, 10(2), 99-107.

McIntyre, L.L., Gresham, F.M., DiGennaro, F.D., and Reed, D.D. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis 1991-2005. Journal of Applied Behavior Analysis. 40, 659-672.

References

Michalopoulos, C. (2005). Precedents and Prospects for Randomized Experiments. In H.S. Bloom (Ed.) Learning More from Social Experiments, (pp. 1-36). New York, NY: Russell Sage Foundation.

Noell, G.H., Witt, J.C., Slider, N.J., Connell, J.E., Gatti, S.L., Williams, K.L., Koenig, J.L. & Resetar, J.L. (2005). Treatment Implementation Following Behavioral Consultation in Schools: A Comparison of Three Follow-up Strategies. School Psychology Review, 34(1), 87-106.

O'Donnell, C. L. (2008). Defining, Conceptualizing, and Measuring Fidelity of Implementation and Its Relationship to Outcomes in K-12 Curriculum Intervention Research. Review of Educational Research, 78(1), 33-84.

Shadish, W.R., Cook, T.D., and Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York, NY: Houghton Mifflin Company.

Swafford, J.O., Jones, G.A., and Thornton, C.A. (1997). Increased Knowledge in Geometry and Instructional Practice. Journal for Research in Mathematics Education, 28(4), 467- 483.

Trochim, W. and Cook, J. (1992). Pattern matching in theory-driven evaluation: A field example from psychiatric rehabilitation. In H. Chen and P.H. Rossi (Eds.) Using Theory to Improve Program and Policy Evaluations. Greenwood Press, New York, 49-69.

A Procedure for Assessing Fidelity of Implementation in Experiments Testing Educational...

Documents

Fidelity Technology Innovators Class of the Fidelity ... · Fidelity Technology Innovators Class of the Fidelity Capital Structure Corp. ... 2019 2018) Investments at fair value through

Low-Fidelity Prototyping · Fidelity in Prototyping Fidelity refers to the level of detail High fidelity? – Prototypes look like the final product Low fidelity? – Artists renditions

Supplemental Information: Fidelity Go Fidelity ......Supplemental Information: Fidelity Go ... 3. Fidelity Go® and Fidelity® Personalized Planning & Advice Brochure Supplement 4

Sketching UX: Low-Fidelity Method; High-Fidelity Results

Social Experiments in␣ Housing - HUDUser.gov · Social Experiments in Housing ... ment about the degree to which the policy interventions had impacts—that is, ... 15 265 179 16

Learning Causal Graphs with Small Interventions · directions is by performing experiments, the so-called interventions. An intervention requires mod- ... Bayesian network using a

FIDELITY INVESTMENTS Fidelity Viewpoints® Webcast: April 22, 2013

Fidelity International’s UK Gender Pay Gap Report ……through a series of data-led interventions on the recruitment, development, inclusion and retention of female talent. Fidelity’s

Very Brief Interventions Study: Potential efficacy, fidelity, feasibility and acceptability of techniques to promote physical activity in primary care

11: Quantifying Prediction Fidelity in Ocean Circulation ...experiments or ﬁeld observations. APPROACH. Our approach to uncertainty quantiﬁcation (UQ) is built on applying PC expansions

Targeted and Intensive Interventions: Assessing Process (Fidelity)

Recent Progress in Advanced Wind Tunnel Boundary ...•High fidelity CFD correction may be needed for complex, unconventional or special ... Computational Modeling for Validation Experiments

Fidelity Monthly Income Class of the Fidelity Capital

RTI Attendance and Intervention Record (AIR) · Intervention Fidelity Statement: I certify that the above noted strategies/interventions were conducted as described. Please initial

Assessing biodiversity outcomes from waterpoint interventions in … · 2011. 10. 16. · impact) design where the ‘before’ experiments surveyed responses to sustained grazing

A Procedure for Assessing Intervention Fidelity in ...itp.wceruw.org/documents/Nelson_2012.pdf · A Procedure for Assessing Intervention Fidelity in Experiments Testing Educational

05- Methods and Experiments - CUPScups.cs.cmu.edu/courses/ups-sp15/Lecture05.pdfIterative prototyping is crucial! High-fidelity, “Wizard of Oz,” low-fidelity 10 Paper prototypes

1 Ecodevelopmental and Systemic Modeling and Implementing High Fidelity Interventions in Real World Settings Guillermo Prado 1,2, Hilda Pantin 1, Seth

Analyzing Social Experiments as Implemented: A Reexamination …ftp.iza.org/dp5095.pdf · Social experiments can produce valuable information about the e ectiveness of interventions

8. Interventions II · When delivered as designed and researched –fidelity is extremely important! ... Washington, DC: American Psychological Association. mjansen@bayviewbehavioral.org