Upload
karen-umstead-med-bcba
View
75
Download
0
Embed Size (px)
Citation preview
Treatment Integrity: What it is and How to do it
Presented by:Karen Umstead-Yosmanovich, M.Ed, B.C.B.A, LBS
Beautiful Minds of Princeton“Teach, Reach, & Expand Potential”
www.beautifulmindsofprinceton.com [email protected]
1
Who am I?Who am I ?
– Teacher (Special/General Education) in private and public schools
– Board Certified Behavior Analyst (BCBA)– Executive Director of BMOP– Adjunct Professor
• Populations worked with– Schools– Agencies– Community Providers– Families
2
Beautiful Minds of Princeton“Teach, Reach, & Expand Potential”
For more information:Call: 1-800-675-2709
Email: kyosmanovich@ beautifulmindsofprinceton.com
or Visit us: www.beautifulmindsofprinceton.com
3
Who are you• BCBA• University/Researcher• Teacher• Child Study• Related Services• Administrator• Parent
4
Warm Up ActivityWhat do you know: (thumbs up, in middle, down)Definition of treatment integrityMulti-dimensional approach to treatment integrity Barriers to treatment integrityVariables or interventions that increase treatment integrityWhat do you want to know (why are you here)More info on defining integrityDealing with barriersSpecific variables or interventions to increase integrity
5
Our Objectives
• Participants will be able to: Label and define multidimensional components of
treatment integrity, including related variables Identify at least four methods of assessing
treatment integrity Label at least 3 solutions to common challenges to
high treatment integrity in school and home settings
6
Proposed definition
• Multidimensional construct (from several fields)• Not as simple as “Did they do it?”• Not just percentage of components implemented as planned
• Content related (quantity)– “How much of the intervention was implemented?”• Process related (quality)– “How well was the intervention implemented?”
7
Process-related
8
Creating a Treatment Integrity Tool
• Define treatment integrity • Pick an appropriate assessment method • Delineate the intervention into operational
steps • Decide how intervention steps will be rated
and the measure will be summed • Determine how often treatment integrity will
be assessed
9
(Collier-Meek, Fallon, Sanetti, & Maggin, 2013; Gresham, 1989; Gresham, 2013; Sanetti, Fallon, & Collier-Meek, 2011; Sheridan, Swanger-Gagne, Welch, Kwon, & Garbacz, 2009)
From PRIME: Planning Realistic Implementation and Maintenance by Educators
Content-Related
• Adherence– This is the most common one we are used to– Requires operational definitions of the intervention
steps• Exposure
– Is it being delivered as often as it supposed to?– For the length of the time it is supposed to happen?
• Another view or key term in research literature– Errors of Omission (not doing something)
10
Content-Related• Program Differentiation
– Just like we consider it when doing research– Especially important when comparing two approaches or
programs or when doing a component analysis of a multi-component plan
– Often times implementers start to incorporate other elements into existing treatment
– Need to understand the “MUST do” versus “Would LIKE to do”
• Another view or key term in research literature– Errors of Commission (doing something wrong)
11
12From PRIME: Planning Realistic Implementation and Maintenance by Educators
ExampleNET (natural environment teaching)- Procedures for Early, Intermediate, & Advanced Students
Staff:____________________ Student:____________________ Observer:____________________ Date:____________________ Section 1: Setting Section 4: Behaviors ____Area is sanitized ____Staff uses appropriate antecedent strategies ____Materials needed are organized and ready ____Staff takes accurate behavior data ____Variety of reinforcers are available ____Staff implements behavior interventions correctly ____NET lesson plan/targets are updated and visible ____Staff pairs social reinforcement with tangible reinforcers Section 2: NET Teaching Procedures Section 5: Data Collection ____Staff follows student’s MO ____Begins session with manding ____Any necessary mand data sheets are updated & ready ____Appropriate level of enthusiasm ____Mand data is taken ____Staff’s voice is natural ____ Behavior data sheets are updated & ready ____Staff provides adequate number of manding trials ____Behavior data is taken ____Staff fades prompts as quickly as possible ____NET lesson plans/target data sheets are updated & ____Staff follows appropriate variable ratio of reinforcement (VR) schedule ready ____Number of demands is faded in ____Data on targets is taken ____Difficulty of demands is faded in ____Staff generalizes skills already taught at IT across: ____Environments ____People ____Stimuli ____Staff mixes verbal operants ____Staff uses errorless teaching with appropriate time delay prompts: ____0 second time delay for new targets ____2-3 second time delay for acquisition or maintenance targets ____Staff uses transfer trials (re-present Sd after error and prompts) ____Staff differentially reinforces novel behaviors/appropriate play skills ____Skills are taught to fluency ____Extinction is used for problem behavior Rating: 1= Yes/Consistently 2= Sometimes/Inconsistent 3= No/Not Occurring 4= N/A Not Applicable 13
From PATTAN Resource Guide for Autism Initiative
14
Natural Environment Teaching Evaluation form Teacher: ______________________________ Date: ___________________________ Observer: _____________________________ Activity: _________________________
Area 1: Organization Area 4: Reinforcement ____ Instructional area is neat and clean ____ Sr+ Reinforcer competes with Sr-/SrA+ ____ All materials needed are organized and ready ____ Pairs social reinforcement w/ tangible items ____ Begins promptly/avoids wasted time ____ Lesson plan current and visible Area 5: Behavior Management ____ Correctly implements extinction procedures Area 2: Instructional Delivery ____ Maintains composure during procedures ____ Follows EO of student ____ Accurately records behavior data ____ Begins NET session with manding ____ Implements effective antecedent interventions ____ Positioned at child’s eye level ____ Appropriate level of enthusiasm Area 6: Error Correction ____ Mixes verbal operants ____ Re-presents after error w/ Sd and 0 sec time delay ____ Uses errorless teaching with appropriate time delay ____ Returns to target several trials later ____Averages 4-5 responses per minute ____ Requires correct response ____Spends recommended time per day in NET ____ Uses non-verbal stimulus to evoke responses Area 3: Data Collection Additional Comments: __________________________ ____ NET data sheet is available and data is taken ____ Data is graphed by verbal operant ______________________________________________ ______________________________________________ Rating: 1 = consistently 2 = Sometimes/inconsistently 3 = Not occurring N/A = Not applicable Responses Across Verbal Operants: 3 Minute Sample Mands Tacts Receptive Intraverbals Motor Imitation Echoics
Responses Per Minute: 1 Minute Timing 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 18 20
From PATTAN Resource Guide for Autism Initiative
Process-Related
• Harder to operationally define (should still attempt to though)
• Qualitative • Quality of delivery
– How well was the treatment delivered– Can tie into some of the content-related elements– Often times a rating scale
• Participants responsiveness15
16From PRIME: Planning Realistic Implementation and Maintenance by Educators
Participant Responsiveness
• Implementer and intended receipt of intervention• Enlisting help to ensure integrity
– Fruit helper of the day (study on increasing fruit)• Implementers wanted to quit, were able to identify solution by
incorporating kids
• Treatment acceptability research has not shown a functional relationship between acceptability of intervention and integrity of implementation
• Responsiveness relates engagement in the program/intervention vs. acceptability
17
Example: Usage Rating Profile
18Chafouleas, S.M., Briesch, A.M., Neugebauer, S. R., & Riley-Tillman, T. C. (2011). Usage Rating Profile – Intervention (Revised). Storrs, CT: University of Connecticut.
Example: Usage Rating Profile
• Factor I: ACCEPTABILITY Items - 1, 7, 9*, 11, 12, 18, 21, 22, 23
• Factor II: UNDERSTANDING Items – 4, 6, 25 • Factor III: HOME SCHOOL COLLABORATION Items – 5,
15, 28 • Factor IV: FEASIBILITY Items – 3, 8, 13, 17, 19*, 27 • Factor V: SYSTEM CLIMATE Items – 10, 14, 16, 20, 26 • Factor VI: SYSTEM SUPPORT Items – 2, 24, 29
19Chafouleas, S.M., Briesch, A.M., Neugebauer, S. R., & Riley-Tillman, T. C. (2011). Usage Rating Profile – Intervention (Revised). Storrs, CT: University of Connecticut.
Intervention characteristicsConsider for implementation, include… •Perception of intervention benefit, as compared to current practice •Intervention complexity •Time or duration required •Materials and resources required •Number of interventionists required •Compatibility •Rate of behavior change •Extent to which the intervention can be adapted
20
Sanetti & Kratochwill (2009)
From PRIME: Planning Realistic Implementation and Maintenance by Educators
Please do the following
• Draw a 2 dimensional picture that represents a plane with a parabola and include a line indicating the vertex
• In a vociferous manner, avow \hu̇-ˈrā\ • Extend your adductor pollicis while
simultaneously lifting your flexor digitorum superficialis above your ossicles
21
22
Interventionist Characteristics
• Comprehension– does the implementer actually know what the plan is saying?
• Feasibility – can the implementer actually do the components of the plan?
• Acceptability – will the plan itself and related student behavior be considered appropriate to the implementer and others in the setting?
23
Readability and Comprehension
• Ease of understanding or comprehension • National Average is 8th grade reading level
– Know your audience and it’s background• SMOG formula (McLaughlin 1969)• Readability Assessment Instrument (RAIN)
(Singh, 2003)
24Singh et al 2009
RAIN Suggestions
• Text must have titles and subtitles with highlighting
• Introductory or summary statement;• Pronoun references and connectives (e.g.
conjunctions) must be clear• New words highlighted and accompanied by a
definition or synonym• Use 12 – 14 pt print size
25Singh et al 2009
Suggestions• Can check readability with Microsoft Word• Under Options for Spelling/Grammar
– Check readability– Run spelling and grammar check– At the end it will give you a reading level and ease of reading
• Get feedback from people who will implement– Change wordings if need be or clarify
• If you must use technical jargon, pair it with easy to understand terms or definitions
26
Don’t forget to consider culture
• Your OWN – Ex: Your priorities versus others (the case of eating
with forks and knives versus hands)• Organizational
– Ex: We can’t allow that here• Implementers
– Ex: What does agreement looking (e.g. nodding head does not indicate agreement in some cultures, but means I heard you)
27
Collecting Data
• Several ways to do this:– Direct observation– Permanent Product– Self-monitoring, self-reporting, behavioral
interviews
There are Pros and Cons to each approach!
28
Direct observation
Pros Cons
•See exactly what is being done•Most accurate representation•Can account for program differentiation issues or external variables
•Reactivity•Time consuming•May be cost prohibitive (funding source)
29
Permanent ProductPros Cons
•Less chance of reactivity•More efficient•Potentially more accurate than other assessments
•You don’t see what is going on•Not all interventions result in a permanent product
30Wilkinson 2007
Self-monitoring, self-reporting, and behavioral interviews
Pros Cons•Not as time consuming for the evaluator•Treatment implementers take ownership for their actions
•Honesty?•Don’t actually see what is happening
31
Self Monitoring Example
32
Self Monitoring Example
33
Treatment Integrity Tools
• Treatment Monitoring Interview– Semi-structured interview meant to increase the
strength of a plan by providing direct support and performance feedback.
• Treatment Integrity Planning Protocol– Standardized process for planning and creating a
treatment integrity assessment
34
True or False?
• Better treatment integrity results in better outcomes.
35
True or False?
• Better treatment integrity results in better outcomes.– It depends!– There are other factors involved in determining
outcomes other than treatment integrity.– How much treatment integrity is needed depends
on the intervention itself – some interventions require more stringent treatment integrity than others.
36
True or False?
• Treatment integrity must remain high at all times for an intervention to be successful.
37
True or False?
• Treatment integrity must remain high at all times for an intervention to be successful.– This would be nice, BUT not necessarily true.
• Treatment drift happens.– Don’t think of treatment integrity as an all or
nothing.• We can view it as a percentage (25%, 50%, 100%)
38
Plan from the start • When you design the plan/intervention,
consider how you can easily adapt materials to check for treatment integrity
• Determine what aspects of integrity you want to check
• Determine a schedule of assessment
39
Determine barriers to treatment
• What are the barriers to treatment• What can be done to address the barrier ahead of time
– Staff says “I don’t have time to do prize box at end of day”» Can an assistant do it? Is there another time/way for it to be
done? Can the student go to another person for prize box (think of level 2/secondary level interventions in PBS)?
• Consider best practices and work from there– This is what best practice would say…now how can we make it
feasible– Start from a positon of strength and then negotiate – CONSIDER THE FUNCTION!
40
Training
• Know your audience• Direct methods
– Modeling, role –playing, rehearsal and feedback• Indirect methods
– Didactic instruction– Written instruction
41
Video clips for training OR feedback
• Online• Youtube• Made by you
– Example- easily add “captions” to video and share with staff on secure drive
• Used a combination of short-video clips (3-5 min) and feedback– Incorporated video-self monitoring, Teacher Performance Rate and Accuracy Scale or
Form (TPRA), and mentoring/feedback from “expert” – Slim, Lina, "Exploring A Staff Training Model For Enhancing Post-Training Procedural Integrity And
Staff Performance Outcomes When Working With Children Diagnosed With ASD" (2015). Seton Hall University Dissertations and Theses (ETDs). Paper 2058.
42
Staff Training • Important to distinguish staff’s ability to perform skill versus
verbally describing skill or enhancing knowledge• EBP consists of performance-based and competency based
strategies– Performance refers to trainer and trainee actions during training– Competency refers to continuing training until mastery criterion is met
• Training is data-based• Essentially being referred to in literature as behavioral skills
training (we can’t cover more specifics, but online handouts include more slides at end of presentation). BST should also be covered in supervision training for BCBAs
43Parson et al 2012
Initial training is just the start
• We cannot train and hope– Several studies showed that post initial training-
high treatment integrity lasted 0-10 days. • How often has a plan worked great initially
and then you hear about problems 2 months later?
• Treatment drift
44
Performance Feedback
• Consider:– What?
• Specify exactly what feedback will be about.– Where?– When?
• Immediately? Daily? Weekly?– How?
• Verbal, written, graphic
• Try an incorporate preferences when possible45
School Challenges • Using negative reinforcement to increase treatment
integrity (teachers avoided consultant meeting by achieving integrity)
• Meeting summaries– Write up any decisions/changes that were agreed upon at
the meeting.– Include action steps ((who will do what by when (and if
need be how))– Delineate roles and responsibilities– Role/Job analysis
46
School Challenges
• Buy-in and accountability– Engage student, various staff members, and parents
as much as possible• REINFORCE!!!! • By engaging others, individuals more hesitant may get
“swept” along • Have the student help with elements of intervention (e.g.
graphing own data = math practice)
• Start with one routine or teacher then expand– Prevent-Teach Reinforce example
47
School Challenges
• Permanent product methods to assist/augment direct observation.
• Write treatment integrity checks into plan. Prepare and include sample forms with your assessment/BIP.
• Using Google Docs to assess other elements
48
School Challenges
• Connect with other resources that can support intervention– Autism Internet Modules– Evidence Based Practice Briefs from National
Professional Development Center on Autism Spectrum Disorders
– PBIS • Internal coaches
– Pyramidal approach to training• Train the Trainer model
49
Scheduling
• Mini-training series (start of day or end of day required time by contract)
• Augment with video-taping and sharing securely– Could even be used by implementer for self-
assessment – Could be used by consultant to model intervention,
including important elements (captioning)
50
Scheduling
• Secure conferencing• Utilize auxiliary resources
– Mentor Teacher– School Psychologist– Case Manager– Administration
51
Home Challenges
• Why do we face treatment integrity issues in the home?– Parents may not agree on what is the best approach
OR – Who has more energy to tough it out?– Disconnect between parents and grandparents
52Gulick (2014)
From Improving Treatment Integrity in Home-based Settings
How do we get better treatment integrity in the home:
– We have to show that our approach will ultimately be more EFFICIENT
– We address the various factors of parental resistance
• Faulty rules – self-generated or found in media– Consider teaching parent behavior that is
necessary to implementing a treatment plan BEFORE having the parents do it.
• Shaping
53Gulick (2014)From Improving Treatment Integrity in the Home-based Setting
Specifically…
• We can collaborate with school to see how they are doing things– Videos– Observation sessions– If there is a home program aspect, have them come
in and demonstrate skills to parents/families
54
Beautiful Minds of Princeton“Teach, Reach, & Expand Potential”
For more information:Call: 1-800-675-2709
Email: kyosmanovich@ beautifulmindsofprinceton.com
or Visit us: www.beautifulmindsofprinceton.com
55© BMOP 2010 www.beautifulmindsofprinceton.com
ReferencesAllen, K.D., & Warzak, W.J.(2000). The problem of parental nonadherence in clinical behavior analysis: Effective treatment is not enough. Journal of Applied Behavior Analysis, 33(3), 373-391.Collier-Meek, M. (2013). Increasing parents’ treatment integrity to a homework intervention through conjoint behavioral consultation and performance feedback (Doctoral Dissertation). Retrieved from http://digitalcommons.ucon.edu/dissertationsCourtemanche, A., Sheldon, J., Sherman, J., Schroeder, S., Bell, A., & House, R. (2014). Assessing the effects of a staff training package on the treatment integrity of an intervention for self-injurious behavior. Journal of Developmental and Physical Disabilities, 26(4), 371-389. doi: 10.1007/s10882-014-9372-6.Dart, E. H., Cook, C.R., Collins, T.A., Gresham, F.M., & Chenier, J.S. (2012). Test driving interventions to increase treatment integrity and student outcomes. School Psychology Review, 41(4), 467-481.De Fazio, C.M., Fain, A.C., & Duchaine, E.L. (2011). Using treatment integrity in the classroom to bring research and practice together. Beyond Behavior, 20(1), 45-49.DiGennaro-Reed, F.D., Reed, D.D., Baez, C.N., & Maguire, H. (2011). A parametric analysis of errors of commission during discrete-trial training. Journal of Applied Behavior Analysis, 44(3), 611-615.Fiske, K.E. (2008). Treatment integrity of school-based behavior analytic interventions: A review of the research. Behavior Analysis in Practice, 1(2), 19-25.Fryling, M.J., Wallace, M.D., & Yassine, J.N. (2012). Impact of treatment integrity on intervention effectiveness. Journal of Applied Behavior Analysis, 45(2), 449-453.Gresham, F.M., Gansle, K.A., & Noell, G.H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26(2), 257-263.
56
ReferencesGulick, R.F (2014). Improving treatment integrity in home-based services [PowerPoint Slides]. Retrieved from http://www.achievemntctr.org/files/abai-presentation-2014-gulick.pdfHagermoser Sanetti, L.M., Fallon, L.M., & Collier-Meek, M.A. (2013). Increasing teacher treatment integrity through performance feedback provided by school personnel. Psychology in the Schools, 50(2), 134-150. doi: 10.1002/pits.21664.Hagermoser Sanettin, L.M., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4(1), 95-114.Hagermoser Sanetti, L.M., Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445-459.Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2011). An evaluation of the Treatment Integrity Planning Protocol and two schedules of treatment integrity: Impact on implementation and report accuracy. Journal of Educational and Psychological Consultation, 21(4), 284-308. doi: 10.1080/10474412.2011.620927.Kuhn, S.A.C., Lerman, D.C., & Vorndan, C.M. (2003). Pyramidal training fro families of children with problem behavior. Journal of Applied Behavior Analysis, 36(1), 77-88.McIntyre, L.L., Gresham, F.M., DiGennaro, F.D., & Derek, D.R. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis. Journal of Applied Behavior Analysis, 40(4). 659-672.
57
ReferencesPage, T.J., Iwata, B.A., & Reid, D.H. (1982). Pyramidal training: A large-scale application with instructional staff. Journal of Applied Behavior Analysis, 15(3), 335-351.Plavnick, J.B., Ferreri, S.J., & Maupin, A.N. (2010). The effects of self- monitoring on the procedural integrity of a behavioral intervention for young children with developmental disabilities. Journal of Applied Behavior Analysis, 43(2), 315-320.Sanetti, L.M.H. (2012). Assessing and promoting high levels of treatment integrity of school-based Interventions [PowerPoint Slides]. Retrieved from: http://www.cber.uconn.edu/up-content/uploads/2012/02/NE-PBIS-2011-10-03-11.pdfSanetti, L.M.H., Collier-Meek, M.A., Byrne, D.C., Byron. J.R., Wisniewsk, A., & Dobey, L. (2014). Research-based strategies to assess and promote educators’ intervention implementation [PowerPoint Slides]. Retrieved from http://www.primeimplementation.com/wp-content/uploads/Promoting-TI-Miniskills_NASP2014.pdfSanetti, L., Williamson, K., Byron, J., Long, A., Kim, J., & Kratochwill, T. (2014). Defining and quantifying treatment integrity: Conceptual and methodological barriers to assessment and interpretation [PowerPoint Slides]. Retrieved from http://www.primeimplementation.com/wp-content/uploads/Measuring-TI_NASP2014.pdfSchulte, A.C., Easton, J.E., & Parker, J. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38(4), 460-475.
58
ReferencesSolomon, B.G., Klein, S.A., & Politylo, B.C. (2012). The effect of performance feedback on teachers' treatment integrity: A meta-analysis of the single-case literature. School Psychology Review, 41(2), 160-175.St. Peter Pipkin, C., Vollmer, T. R., & Sloman, K. N. (2010). Effects of treatment integrity failures during differential reinforcement of alternative behavior: A translational model. Journal of Applied Behavior Analysis, 43(1), 47-70.Vollmer, T.R., Sloman, K.N., St. Peter Pipkin, C. (2008). Practical implications of data reliability and treatment integrity monitoring. Behavior Analysis in Practice, 1(2), 4-11.
59
Staff Training • Step 1: Describe target skill (instruction)
– Include rationale as to why important– Behaviorally define target skill
• Include performance checklist of necessary staff actions
• Step 2: Succinct Written Description– This is not the same as the formal behavior plan– This is essentially just the basics and a cheat sheet– Can refer back to background documents like FBA and
full BIP– Keep it simple and watch the language used
60Parson et al 2012
Staff Training • Step 3 Demonstrate the skill (model)
– Can use role-play or video model– One person is staff, One is targeted individual– Make sure well-scripted/planned
• Can use cue cards to prompt people to perform certain actions (e.g. make an error, no response, touch only the left side, etc)
– Stop and freeze at certain points to describe what is being done and why
61Parson et al 2012
Staff Training • Step 4 Trainee practices skill (rehearsal)
– Role-play– Practice is critical
• Step 5 Provide performance feedback during practice– Supportive
• Describe EXACTLY what was done correctly – Corrective
• Describe EXACTLY what was incorrect• Providing instruction on how to correct error
62Parson et al 2012
Staff Training • Consider giving feedback at the end rather than in
the middle. Can be a matter of preference though• Step 6 Repeat steps 4 and 5 until mastery• On the job training
– Remember initial acquisition does not mean that skills will be maintained or generalized
– Some “drift” has been noted in as little as 10 days• Consider effectiveness, efficiency, and
acceptability of staff training
63Parson et al 2012