Laying the Foundation for Scaling Up During Development

Definition

• Fidelity of implementation is: • the extent to which a program (including its

content and process) is implemented as designed;

• how it is implemented (by the teacher);• how it is received (by the students); • how long it takes to implement (duration); and,• what it looks like when it is implemented (quality).

Motivation: What problems exist?

• Teachers have difficulty teaching with fidelity when creativity, variability, and local adaptations are encouraged.

• Developers often fail to identify the critical components of an intervention.

• Researchers often fail to measure whether components are delivered as intended.

• What are the critical components of the program? If the teacher skips part of the program, why does that happen, and what effect will it have on outcomes?

• Is the program feasible (practical) for a teacher to use? Is it usable (are the program goals clear)? If not, what changes should I make to the program? What programmatic support must be added?

• What ancillary components are part of the program (e.g., professional development) and must be scaled-up with it?

What questions about fidelity of implementation are asked during Development studies?

What questions about fidelity of implementation are asked during Efficacy studies?

• How well is the program being implemented in comparison with the original program design?

• To what extent does the delivery of the intervention adhere to the program model originally developed?

• How can effective programs be scaled up across many sites (i.e., if implementation is a moving target, generalizability of research may be imperiled)?

• How can we gain confidence that the observed student outcomes can be attributed to the program?

• Can we gauge the wide range of fidelity with which an intervention might be implemented?

Source: Lynch, O’Donnell, Ruiz-Primo, Lee, & Songer, 2004.

What questions about fidelity of implementation are asked during Scale-up studies?

Definition: Efficacy Study

• Efficacy is the first stage of program research following development . Efficacy is defined as “the ability of an intervention to produce the desired beneficial effect in expert hands and under ideal circumstances” (RCTs) (Dorland’s Illustrated Medical Dictionary, 1994, p. 531).

• Failure to achieve desired outcomes in an efficacy study "give[s] evidence of theory failure, not implementation failure" (Raudenbush, 2003, p. 4).

• Internal validity - determines that the program will result in successful achievement of the instructional objectives, provided the program is “delivered effectively as designed” (Gagne et al., 2005, p. 354).

• Efficacy entails continuously monitoring and improving implementation to ensure the program is implemented with fidelity (Resnick et al., 2005).

• Explains why innovations succeed or fail (Dusenbury et al., 2003);

• Helps determine which features of program are essential and require high fidelity, and which may be adapted or deleted (Mowbray et al., 2003).

Definition: Efficacy Studies

• Interventions with demonstrated benefit in efficacy studies are then transferred into effectiveness studies.

• Effectiveness study is not simply a replication of an efficacy study with more subjects and more diverse outcome measures conducted in a naturalistic setting (Hohmann & Shear, 2002).

• Effectiveness is defined as “the ability of an intervention to produce the desired beneficial effect in actual use” under routine conditions (Dorland, 1994, p. 531) where mediating and moderating factors can be identified (Aron et al., 1997; Mihalic, 2002; Raudenbush, 2003; Summerfelt & Meltzer, 1998).

Definition: Scale-up Study

• External validity – fidelity in effectiveness studies helps to generalize results and provides “adequate documentation and guidelines for replication projects adopting a given model” (Mowbray et al, 2003; Bybee, 2003; Raudenbush, 2003).

• Role of developer and researcher is minimized. • Focus is not on monitoring and controlling levels of

fidelity; instead, variations in fidelity are measured in a natural setting and accounted for in outcomes.

Definition: Scale-up Studies

O’Donnell (2008): Steps in Measuring Fidelity

1. Start with curriculum profile or analysis; review program materials and consult with developer. Determine the intervention’s program theory. What does it mean to teach it with fidelity?

2. Using developer’s and past implementers’ input, outline critical components of intervention divided by structure (adherence, duration) and process (quality of delivery, program differentiation, participant responsiveness) and outline range of variations for acceptable use.

O’Donnell, C. L. (2008).Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84.

O’Donnell (2008): Steps in Measuring Fidelity

3. Develop checklists and other instruments to measure implementation of components (in most cases unit of analysis is the classroom).

4. Collect multi-dimensional data in both treatment and comparison conditions: questionnaires, classroom observations, self-report, student artifacts, interviews. Self-report data typically yields higher levels of fidelity than observed in the field.

5. Adjust outcomes if fidelity falls outside acceptable range.

O’Donnell, C. L. (2008).Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84.

Know when & how to use fidelity data• Development - Use fidelity results to inform revisions. Decide now

what components are required to deliver the intervention as intended when implemented at scale.

• Efficacy - Monitor fidelity and relate it to outcomes to gain confidence that outcomes are due to the program (internal validity).

• Replication - Determine if levels of fidelity and program results under a specific structure replicate under other organizational structures.

• Scale-up - Understand implementation conditions, tools, and processes needed to reproduce positive effects under routine practice on a large scale (external validity). Are methods for establishing high fidelity financially feasible?

Bottom Line: If the intervention can be implemented with adequate fidelity under conditions of routine practice and yield positive results, scale it up.

Source: O’Donnell, 2008