Conceptualizing Fidelity in Tiered Models: A ... Fidelity in Tiered Models: A Multidimensional Framework ... is different from "business as ... predictive utility/validity for all

Conceptualizing Fidelity in Tiered Models: A Multidimensional

Framework

Susan M. Sheridan, PhDNebraska Center for Research on Children, Youth,

Families and Schools

Supported by IES Grant #s R324A090075 and R305F050284

Work presented here reflective of collaborations with colleagues Lisa Knoche, Mary Abbott, Judy Carta, Michelle

Swanger-Gagne, and Andy Garbacz

Why are We Concerned with Why are We Concerned with Fidelity Assessment?Fidelity Assessment?

Our ability to conclude that an intervention is effective (i.e., outcome is a function of the IV) requires knowledge about implementation

There is a positive relationship between implementation fidelity and the reliability/quality of our conclusions

Failure to document fidelity results in evaluation of interventions as described or desired, rather than delivered◦

Potential for “Type III error”

Presenter

Presentation Notes

Type III error: unreliable results with little to no bearing on actual intervention effects

Guiding Assumptions for PreschoolGuiding Assumptions for Preschool-- 3T Fidelity3T Fidelity

• Fidelity is a multi-dimensional construct that concerns both what is delivered, how

it is implemented, whether

it is received

by its intended recipients, and its appropriateness

based on student needs.

• Fidelity assessment considers both delivery by the treatment agent (i.e., what, how, when, where) and uptake/response (match) by its intended recipients (i.e., students at each tier).

• In three-tiered models, it also concerns the fidelity with which student data are collected and used, decisions are made, and interventions/strategies are developed based on the data.

Assessing Fidelity within Preschool Assessing Fidelity within Preschool 3T Programs3T Programs

Fidelity dimensions

Structural elements:◦

Instructional context/setting (where?)◦

Instructional content (what?)

Process elements:◦

Problem solving and decision making

Multidimensional Framework for Fidelity Multidimensional Framework for Fidelity Assessment in PreAssessment in Pre--3T: Language and Literacy Focus3T: Language and Literacy Focus

Dimensions of FidelityDimensions of Fidelity ((CordrayCordray

& & PionPion, 2006; Dane & Schneider, 1998), 2006; Dane & Schneider, 1998)

• Dosage –

overall amount of intervention delivered and received by participant (“strength of intervention”)

• Adherence –

the implementation of intervention strategies as designed by developers

• Quality --

qualitative aspects or effectiveness with which strategies are delivered (the “competence of treatment agents)

• Participant Response --

the extent to which the participant is engaged in and receptive to the intervention

• Program Differentiation --

the extent to which the intervention is different from "business as usual" or an alternative intervention

Structural ElementsStructural Elements

Context (Instructional Setting) --

the

discrete structural settings within preschool programs designed for the majority of academic instruction and learning to occur, including instructional strategies that are utilized: ◦

Story Time◦

Center Time◦

Large Group◦

Small Group

Structural ElementsStructural Elements

Skill Domain (Instructional Content) --

Skill

areas predictive of early literacy development:◦

Print Awareness

◦

Alphabet Knowledge◦

Oral Language◦

Phonological Awareness

Process ElementsProcess Elements• The problem solving/decision making process is a

separate aspect of, and potential source of variance in, the tiered intervention sequence.

• Targeted and individualized tiers need to be concerned with fidelity of decision making and problem solving processes as unique facets of the intervention.

• To conclude that a tiered model is efficacious at bringing about change in a child, evidence of fidelity of both content

(what strategies are delivered) and

process

(how decisions about individualization are made) is necessary.

ChallengeChallenge……• Each aspect of fidelity addresses a

fundamentally important, albeit different, issue.

• Each instructional context poses different instructional demands and targeted opportunities for learning.

• Each skill domain is important to early literacy development.

• How do we capture each dimension, context, skill domain, and process element in a way that is practical

and sound?

Putting it TogetherPutting it TogetherFidelity

DimensionWithin Context/Setting Skill/Domain

Adherence Overall percentage of instructional strategies met within each setting

Percent of relevant instructional strategies met targeting skill within setting

Quality Overall quality ratings of each instructional strategy within each setting

Quality ratings of relevant strategies targeting skill within setting

Student Responsiveness

Proportion of students engaged and participating in tasks/activities within settings aimed at targeting skill domains

Dosage Individual student present x strategy

Individual student present x quality of relevant strategies met within setting

Presenter

Presentation Notes

Still very much a trial period for our conceptulization and definitions

Putting it TogetherPutting it Together

Within settings:◦

Assess adherence

to evidence-based instructional strategies

Adherence Instructional Strategy (Story Time)

Y N ST 1. Introduces the book through reading of title, author, or illustrator.

Y N ST 2. Discusses or demonstrates of concepts about print (one or

more)

(e.g., text contains letters, words, sentences; reading progresses left to

right, top to bottom, finger tracing along text; etc).

Y N ST 3. Uses facial expressions and voice to capture children's attention as

appropriate to the text by using different tones for characters or

modulating voice to emphasize words/facts (e.g., for a non‐fiction book

emphasizing words, and with a fiction book, changes voices for

characters).

Y N ST 4. Calls attention to novel vocabulary words that children may not

know by repeating, providing a definition or a brief explanation;

vocabulary words are discussed when preparing to read and/or reading

books aloud (charts and displays are not required).

Putting it Together in Pre3TPutting it Together in Pre3TFidelity








Dosage Individual student present x adherence of instructional strategy


Presenter

Presentation Notes


Within settings:◦

Rate instructional strategy delivery on quality

Strategy (Story Time) Adherence Quality

0 1. 2. 3.

ST 1. Introduces the book through

reading of title, author, or illustrator.0. FAILS to meet

adherence

criteria

1. Introduces the book by ONLY

mentioning 2 of 3 of the title,

author, and illustrator.

2. Introduces the book through

reading of title, author, and

illustrator.

3. Introduces the book through

reading of title, author, and

illustrator and checks for

understanding of the terms.

ST 2. Discusses or demonstrates of

concepts about print (one or more)

(e.g., text contains letters, words,

sentences; reading progresses left to

right, top to bottom, finger tracing

along text; etc).

0. FAILS to meet

adherence

criteria

1. Sporadically discusses or

demonstrates of concepts

about print (e.g., traces finger

along one or two pages of text

but not most of the pages).

2. Consistently discusses or

demonstrates concepts about

print (e.g., consistently uses

finger tracing along text).

3. Discusses AND demonstrates

of concepts about print (one or

more) (e.g., uses, finger tracing

along text and tells or asks

children why she is moving

tracing long text ).

ST 3. Uses facial expressions and voice

to capture children's attention as

appropriate to the text by using

different tones for characters or

modulating voice to emphasize

words/facts (e.g., for a non‐fiction book

emphasizing words, and with a fiction

book, changes voices for characters).

0. FAILS to meet

adherence

criteria

1. Fails to appropriately use

facial expressions. E.g., Uses

facial expressions that are

inappropriate for preschool

children or that don't match

the tone of the book.

3.For the most part, uses facial

expressions and voice to capture

children's attention as




words/facts

4. Strongly exhibits consistent

facial expressions and voice to

capture children's attention as




words/facts

ST 4. Calls attention to novel vocabulary

words that children may not know by

repeating, providing a definition or a

brief explanation; vocabulary words are

discussed when preparing to read

and/or reading books aloud (charts and

displays are not required).

0. FAILS to meet

adherence

criteria

1.Fails to call attention to

novel vocabulary words that

children may not know by

repeating, providing a

definition or a brief

explanation OR fails to discuss

vocabulary words when

preparing to read.

1.Misses numerous appropriate

opportunities but does call

attention to novel vocabulary

words AND discusses

vocabulary words when

preparing to read and/or

reading books aloud.

3. Consistently calls attention to

novel vocabulary words providing

a definition or a brief explanation

AND discusses vocabulary words

when preparing to read and/or

reading books aloud.

Presenter

Presentation Notes

Same items, but we rate them in terms of adherence (yes/no) AND quality (1-3)











Presenter

Presentation Notes


Weighted Language and Literacy Skills Across the Day

Story Time: 20% Print Awareness, 80% oral language (additional points when Alphabetic Knowledge and

Phonological Awareness are part of the lesson plan and appropriate for the book to be read).

Print AwarenessST 1. Introduces the book through reading of title, author, or illustrator.

Oral LanguageST 4. Calls attention to novel vocabulary words that children may not know by repeating, providing a definition or

a brief explanation; vocabulary words are discussed when preparing to read and/or reading books aloud (charts and

displays are not required).ST 5. Calls attention to pictures and asks children to name objects in pictures or describe what they see.ST 6. Asks open ended questions (e.g., “what if”, “where have you seen”, “how would”) to encourage discussion of

facts in the book (nonfiction), details, plot and/or characters (fiction), or topic and/or rhyming (poetry).ST 7. Uses “think‐alouds”

or summarizing, or prior knowledge. (e.g., What do you think is

happening here? How is

Nora feeling? Look at Nora’s face. I think she looks sad.) to support literal and/or inferential comprehension.

Alphabet Knowledge

(Note: AK is not expected to occur during story reading unless

the lesson plan states that the

teacher plans to use the book as a tool learning or reinforcing AK; e.g., ABC book. Score as NA if not applicable.)

Phonological Awareness

(Note: PA is not expected to occur during story reading unless the book lends its self to

rhyming or alliteration. Score as NA if not applicable.)

• Skill Domain--

Weight adherence and quality ratings

by setting

Presenter

Presentation Notes

Again, same items. However, not all items share the same predictive utility/validity for all skills being targeted. This weighting of adherence/quality within settings is an attempt to introduce sensitivity into the measurement. Strategies used within each setting, such as story time, are thought to contribute more to certain domains, such as oral language. Oral language strategies thus get more “credit” during story time to assess the relative attention given to that skill.











Presenter

Presentation Notes


Student Responsiveness Student Responsiveness ––

Rate within Setting and Weight by DomainRate within Setting and Weight by Domain1 2 3 4

Not at All

Not Very

Somewhat

Completely

1: Students not at all responsive/engaged. Students are completely unengaged/No evidence of student engagement. Fewer than 25% of students are engaged during the majority of the observational period.

2: Students not very responsive/engaged. There is little evidence of engagement. Between 25 and 50% of the students engaged during the majority of the observational period.

3: Students somewhat responsive/engaged.

There is moderate evidence of engagement. Between 50 and 75% of the students are engaged during the majority of the observational period.

4: Students are completely responsive/engaged. There is much evidence of engagement. Between 75 and 100% of the students are engaged during the observational period.

Behavioral indicators of engagement during Story Time include any of the following:

Listening to story, conversation, or instruction

Alert body language

Raising hand

Vocalizing in manner consistent with group during choral responding

Responding to discussion questions

Use the names of the characters/relate to another story that was

read

Asking questions

Using own experiences to talk about the story

Pointing out objects/actions in the picture

Pointing out letters/sight words

Process Elements of PreProcess Elements of Pre--3T3T

Use of data to move students into more intensive and individualized levels of intervention requires complex decision-making

Development of individualized plans is based on a problem-solving consultation model, involving specification of the problem, frequent assessment, development of an evidence-based intervention that can be delivered effectively, and ongoing evaluation

Problem-solving components include frequent assessment of target skills/behaviors; equivalence between progress monitoring tools and targeted assessments are untested

Presenter

Presentation Notes

P-s model components such as assessment and evaluation are part of the 3-t approach (progress monitoring ) – but are the PM tools going to be sensitive to the individualized plans?

Challenge: Psychometric Qualities Challenge: Psychometric Qualities of Fidelity Measuresof Fidelity Measures

Reliability (interrater, internal consistency, stability)

Validity (convergent, predictive)

Sensitivity of fidelity criteria (utility in predicting outcomes)

Sensitivity of scale (ability to capture ‘true’

implementation; ability to

differentiate high from low/ “good”

from “bad”)

Representativeness of occasion

Presenter

Presentation Notes

Reliabiity across respondents – interrater agreement, percentage agreement, intraclass correlation, Pearson r Internal consistency – assesses internal structure assesses of measures; Cronbach’s alpha; cluster factor analysis Stability of measures – stability of scores over time based on standard deviation Convergent validity – agreement b/t 2 different sources of information about an intervention (e.g., r’s b/t scores on different fidelity measures such as permanent products and observations) Predictive validity– relationship b/t measure and expected outcomes for participants – are we measuring the right things? What predicts outcomes? Sensitivity – some intervention components don’t warrant direct measurement or require very expensive techniques such as full day observatin (e.g., behaviors that earn long-term back up reinforcers delivered at the end of a day; attainment of goals over days or a week)

Challenge: Analytic IssuesChallenge: Analytic Issues

How do we best understand and use fidelity information in studies on intervention efficacy?◦

Indexing fidelity and creating composites/ scores◦

Mediation/Moderation◦

“Grain size”◦

Cost effectiveness

Implementation fidelity as an independent variable

Presenter

Presentation Notes

Mediation – what is the mechanism of change? Can quality of implementation (i.e., practice) be the pathway through which interventions have their effect? Moderation – under what conditions are effects seen? For example, effects prevalent when intervention is recd by child at a certain threshold level or dose? Mediation / Moderation – some aspects seem more conducive to conceptualize as mediators (e.g., quality) and others as moderators (e.g., dosage) – we are just starting to think through these issues Grain size – whereas the strength of the measure and the resulting total score is determined in part by the consistency of the items, we expect that the relationship b/t implementation fidelity and intervention outcomes is more a function of adherence to meaningful intervention protocols over time (dosage issue)

Lessons LearnedLessons Learned

Streamlining is critical in assuring the tool is practical and useful

Needs in research and practice are unique; for example:◦

Evidence of implementation via paper-pencil records and self-reports are reliable and yield results similar to direct measures (Sheridan et al., 2009) but on a narrow range of dimensions◦

Video records provide rich and flexible data collection across fidelity dimensions, but are expensive to collect and interpret (Knoche et al., in press)

Presenter

Presentation Notes

Research does not equal practice in this area

Documents

Conceptualizing Fidelity in Tiered Models: A ... Fidelity in Tiered Models: A Multidimensional Framework ... is different from "business as ... predictive utility/validity for all