Embed Size (px)
Citation preview
Single-Case Research Designs:Training Protocols in Visual Analysis
Wendy MachalicekUniversity of Oregon
www.uoecs.org
Acknowledgement:
Rob HornerTom Kratochwill
Joel LevinChris SwobodaGeorge Sugai
Hariharan SwaminathanDan Maggin
Specific Goals
Review training protocols for competency and fluency in visual analysis
www.singlecase.org
Single-Case Research Designs:Training Protocols in Visual Analysis
Wendy MachalicekUniversity of Oregon
www.uoecs.org
Acknowledgement:
Rob HornerTom Kratochwill
Joel LevinChris SwobodaGeorge Sugai
Hariharan SwaminathanDan Maggin
Specific Goals
Review training protocols for competency and fluency in visual analysis
www.singlecase.org
Context: Is visual analysis reliable across analysts?• Some have found visual analysis to be unreliable, but these studies
have been conducted with novice analysts• e.g. Liberman, Yoder, Reichow, & Wolery (2010); Ottenbacher (1993)
• When analysts are trained and experienced, we see high agreement • Kahng et al. (2010)
WWC White Paper
• Single-Case Intervention Research Design Standards• • Thomas R. Kratochwill
• University of Wisconsin-Madison
• • John H. Hitchcock
• Ohio University
• • Robert H. Horner
• University of Oregon
• • Joel R. Levin
• University of Arizona
• • Samuel M. Odom
• University of North Carolina at Chapel Hill
• • David M. Rindskopf
• City University of New York
• • William R. Shadish
• University of California Merced
Available atwww.uoecs.org“Materials”
What is needed
• Better standards for training in single-case methods
• More precision in RFA stipulations• Establishes expectations for reviewers
• Better standards for reviewing single-case• Journal editors• Grant reviewers
• Development of meta-analysis technology
Evaluate the Design
Meets Design Standards Meets with Reservations Does Not Meet Design Standards
Evaluate the Evidence
Strong Evidence Moderate Evidence No Evidence
Effect-Size Estimation
Social Validity Assessment
Does the Design allow
Documentation of Experimental
Control ?
Do the data document
Experimental Control ?
Is the effect something we should care
about?
Stop
Stop
Visual analysis of single case designs• Determine viability of design first① Assess Baseline② Assess the within phase pattern of each phase③ Assess the basic effects with each phase comparison④ Assess the extent to which the overall design documents experimental
control• E.g. Three demonstrations of basic effect, each at a different point in time
• Assess effect size• Assess social validity
Assessing within phase “pattern” and Between phase “basic effect”• Level• Trend• Variability +• Overlap• Immediacy of Effect• Consistency across similar phases_________________________________________• Other: vertical analysis; intercept gap
Within Phase
Between
Phases
Effective training protocols
Fisher, Kelley, & Lomas (2003) • Train practitioners to use dual-criterion method with AB graphs• Baseline split-middle and mean level lines projected onto treatment
phase• Specified proportion of data must fall above/below both lines
• Visual analysis improved when graphs included criterion lines, have written rules, and brief verbal instruction/modeling training provided
Effective training protocols
Stewart, Carr, Brandt, & McHenry (2007)• Train practitioners to use (1) 8-min recorded lecture, demos,
conservative dual-criterion method with AB graphs & (2) 12-min recorded lecture• Lecture alone didn’t improve accuracy
• Visual analysis improved to 80% accuracy when graphs included criterion lines, have written rules, and brief verbal instruction/modeling training provided
Effective training protocolsWolfe & Slocum (2015)• Random assignment of
participants to (a) computer-based instruction, (b) 36 min recorded lecture or (c) control condition• AB graphs varied on trend
and level
Training Protocol for Visual Analysis of Single-case data • Development of visual analysis graphs
• Rob Horner, Bruce Wampold, Richard Freund, Rick Albin, Tom Kratochwill, Chris Swoboda• Level, trend, variability, overlap• Purpose is to emphasize interaction
• Examine and score each graph (ABAB, MBD, ATD graph sets)• Use ALL data in the graph• Combine assessment of:
• Level, trend, variability, overlap, immediacy of effect, consistency of data patterns in similar phases.• Use scoring metric:
• 1 = no functional relation• 2,3,4 = considerable weaknesses • 5 = moderate functional relation (publishable)• 7 = strong functional relation
• Compare scores with Excel file• Median scores from five “expert” Single-case Researchers
http://www.singlecase.org
Using www.singlecase.org• Anyone, any time may use the website to assess visual analysis
competency• For Instructors• Contact Rob with course code, and dates needed
• Set up “coordinator”• Students enter code and name• You can download their scores, number of sessions, time
• For Research• Inter-rater agreement with visual analysis• Training protocols for teaching visual analysis.
Visual analysis of single case designs
• Determine viability of design first• Assess Baseline• Assess the within phase pattern of each phase• Assess the basic effects with each phase comparison• Assess the extent to which the overall design documents experimental
control• E.g. Three demonstrations of basic effect, each at a different point in time
• Assess effect size• Assess social validity
Screen shot of results
Screen shot of results
Summary• Visual Analysis of Single Case Designs• Determine viability of design first• Assess Baseline• Assess the within phase pattern of each phase• Assess the basic effects with each phase comparison• Assess the extent to which the overall design documents experimental
control• E.g. Three demonstrations of basic effect, each at a different point in time
• Assess effect size• Assess social validity
YOUR TURN: The full set of
140 graphs
Design activity
• Design a single-case research or Group experimental design evaluating use of www.singlecase.org or another training protocol to improve participant acquisition, fluency of visual analysis in single-case research
