View
218
Download
3
Category
Preview:
Citation preview
Non-Overlap Methods in Single Case Research Methodology
Erin E. Barton, PhD, BCBA-D
Non-overlap Methods
1. PND
2. PEM (ECL)
3. PEM-T
4. PAND
5. R-IRD
6. NAP
7. Tau-U
Rationale: Non-overlap Methods
1. Need to aggregate across studies to determine evidence for practice
Meta-analysis is a well established practice for group experiments
– Magnitude – Aggregate findings – Moderator analyses
Rationale: Non-overlap Methods
1. Need to aggregate across studies to determine evidence for practice
– Many have argued if SCRD will not be included in reviews of evidence-based practices unless an effect size estimator is used
– They are often left out of reviews in disciplines outside of special education
Rationale: Non-overlap Methods
2. Emphasis on “effect sizes” in education research to quantify the magnitude
– Standardized effect sizes are particularly valued– Reviewers compare results across studies having
different outcome measures that otherwise could not be easily compared
Rationale: Non-overlap Methods
3. Meta-analytic techniques are compromised when data are serially dependent – On a single individual– Using the same definitions– Using the same data collection procedures– In the same context – Under the same procedures– Often with short intervals between observations
Rationale: Non-overlap Methods
4. Current meta-analytic techniques are inappropriate for aggregating SCR data
The data patterns must shift consistently in the predicted (therapeutic) direction with each change in experimental condition
Replication logic is used to make judgments about functional relations
The design needs an adequate number of replications of the experimental conditions (internal validity)
Rationale: Non-overlap Methods
5. Non-parametric techniques needed
• Short data sets or few data points• Non-normal or unknown distributions• Unknown parameters
PND: Percent of Non-overlapping Data
PND: Percent of Non-overlapping Data
• One of the oldest of the overlap methods (Scruggs, & Mastropieri, 1998; Scruggs, Mastropieri, & Casto, 1987)
• Used extensively• Easily calculated• Does not assume data are independent
Calculating PND
1. Identify the intended change
2. Drawing a straight line from the highest (or lowest) point in Phase A and counting the number of data point in Phase B above the line
3. Quotient = # above the line / total number in Phase B X 100
Interpreting PND
70% is effective, 50% to 70% is questionable effectiveness, and <50% is no observed effect
(Scruggs & Mastropieri, 1998)
Practice: Calculating PNDSchilling (2004), David
In Seat
A1 to B1:
B1 to A2:
A2 to B2:
Engaged
A1 to B1:
B1 to A2:
A2 to B2:
A1 to B1: 22%
B1 to A2: 100%
A2 to B2: 100%
AVERAGE: 74%
Practice: Calculating PNDSchilling (2004), David
In Seat Engaged
A1 to B1: 100%
B1 to A2: 100%
A2 to B2: 100%
AVERAGE: 100%
Practice: Calculating PNDVaughn (2002)
Disruptive Behavior
Arrival: 100%
Mealtime: 75%
Departure: 100%
AVERAGE: 92%
Engaged
Arrival: 100%
Mealtime: 75%
Departure: 100%
AVERAGE: 92%
Outliers
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
1
2
3
4
5
6
7
8
Days
CB
Rat
e p
er M
inu
te
PND = 0%
PND = 0%
PND = 0%
Functional Relation?
Trends
1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627280123456789
10
Days
CB
Rat
e pe
r M
inut
e
PND = 0%
PND = 0%
PND = 0%
Functional Relation?
Magnitude
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
2
4
6
8
10
12
Days
CB
Rat
e p
er M
inu
te
PND = 100%
PND = 100%
PND = 100%
Functional Relation?
Outliers
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
1
2
3
4
5
6
7
8
Days
CB
Rat
e p
er M
inu
te
PND = 0%
PND = 0%
PND = 0%
Functional Relation?
BUT NO ONE USES A-B-A-B…..
You might be thinking….
0
2
4
6
8
10
12
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
2
4
6
8
10
12
Days
Fu
nct
ion
al R
elat
ion
?
PND = 68%
PND = 43%
PND = 86%Soc
ial I
nitia
tions
to
Pee
rs
PND = 68%
PND = 43%
PND = 86%
PND = 66%
TREND?
0
2
4
6
8
10
12
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
2
4
6
8
10
12
Days
Fu
nct
ion
al R
elat
ion
?
PND = 0%
PND = 0%
PND = 0%Soc
ial I
nitia
tions
to
Pee
rs
PND = 0%
PND = 0%
PND = 0%
PND = 0%
MAGNITUDE?
0
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28012345678
Days
Fu
nct
ion
al R
elat
ion
?
PND = 100%
PND = 100%
PND = 100%
Soc
ial I
nitia
tions
to
Pee
rs
WHAT DOES THAT MEAN?
You SHOULD be thinking….
PND Flaws
• Compared with consensus visual analysis, PND resulted in an error in about 1 of 5 condition changes—high rate of errors (Wolery, Busick, Reichow, & Barton, 2010)
• Compromised by:– Longer data sets, # of
data points – Variability – Outliers – Trends
Should not be used (Brossart et al., 2013; Kratochwill et al., 2010; Parker & Vannest, 2009)
PND Flaws
Is it an Effect Size?
Replication
Magnitude
Can it supplement visual analysis?
1. Level
2. Trend
3. Variability
4. Immediacy
5. Overlap
6. Consistency Vertical analysis
PAND: Percent of All Non-overlapping Data
PAND: Percent of All Non-overlapping Data
• Not compromised by serial dependency or other data assumptions
• Percentage of data remaining after determining the fewest data points that must be removed to eliminate all between-phase overlap
Calculating PAND
1. Count the total number of data points in comparison
2. Identify how many need data points need to be removed to eliminate overlap
3. Count the number of remaining data points
4. Divide count in step 3 by count in step 1
5. X by 100
Practice: Calculating PANDSchilling (2004), David
In Seat
A1 to B1: 94%
B1 to A2: 100%
A2 to B2: 100%
AVERAGE: 98%
(PND Average was 74%)
Engaged
A1 to B1: 100%
B1 to A2: 100%
A2 to B2: 100%
AVERAGE: 100%
(PND Average was 100%)
Practice: Calculating PANDVaughn (2002)
Disruptive Behavior
Arrival: 100%
Mealtime: 85%
Departure: 100%
AVERAGE: 95%
PND was 92%
Engaged
Arrival: 100%
Mealtime: 85%
Departure: 100%
AVERAGE: 95%
PND was 92%Adding more data points
Extinction burst Variability
0
2
4
6
8
10
12
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
2
4
6
8
10
12
Days
PAND = 96%
PAND = 93%
PAND = 93%Soc
ial I
nitia
tions
to
Pee
rs
PAND = 96%
PAND = 93%
PAND = 93%
PAND = 94%
MAGNITUDE?
0
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28012345678
Days
PAND = 100%
PAND = 100%
PAND = 100%
Soc
ial I
nitia
tions
to
Pee
rs
PAND Flaws
Is it an Effect Size?
Replication
Magnitude
Can it supplement visual analysis?
1. Level
2. Trend
3. Variability
4. Immediacy
5. Overlap
6. Consistency Vertical analysis
Should not be used (Brossart et al., 2013; Manolov et al., 2010)
PEM: Percent Exceeding the Median (Ma, 2006)
PEM: Percent Exceeding the Median (Ma, 2006)
• Not compromised by serial dependency and other data assumptions
• Designed to eliminate problem with baseline datum point being at floor or ceiling
– Designed to not rely on the most extreme datum point
– Less influenced by variability in baseline
Calculating PEM
1. Drawing a line at the median of Phase A data through Phase B data
2. Count the number of data points in Phase B above (or below) the line and divide by the total number of data points in Phase B
PEM Flaws
• Compared with consensus visual analysis, PEM resulted in an error in about 1 of 6 condition changes—high rate of errors (Wolery et al., 2010)
Should not be used (Parker, Vannest, & Davis, 2011)
PEM Flaws
Is it an Effect Size?
Replication
Magnitude
Can it supplement visual analysis?
1. Level
2. Immediacy
3. Overlap
4. Consistency Vertical analysis
PEM-T: Percent Exceeding the Median Trend Line
ECL: Extended Celeration Line
PEM-T: Percent Exceeding the Median Trend Line (Wolery et al., 2010)
• Not compromised by serial dependency and other data assumptions
• Designed to eliminate problem with baseline datum point being at floor or ceiling
– Designed to not rely on the most extreme datum point
– Less influenced by variability in baseline– Less influenced by trends in data
Calculating PEM-T
1. Graph data on semi-logarithmic chart
2. Calculate and draw a split middle line of trend estimation for Phase A data and extend it through Phase B
3. Count # of Phase B data points above/below the split middle line of trend estimation
4. Divide count from Step 4 by # data points in Condition 2 and multiply quotient by 100
PEM-T Flaws
• Compared with consensus visual analysis, PEM-T resulted in an error in about 1 of 8 condition changes—high rate of errors (Wolery et al., 2010)
PEM-T Flaws
Is it an Effect Size?
Replication
Magnitude
Can it supplement visual analysis?
1. Level
2. Trend
3. Variability
4. Immediacy
5. Overlap
6. Consistency Vertical analysis
So what, right?
Social Stories for Children with ASD
• 20 studies met design standards with or without reservation, which exceeded the minimum number of five studies set by the WWC as needed to be represented across studies.
• Across 3 research groups
Qi & Barton, under review
Social Stories for Children with ASD
• Using Non-overlap indices: – 28 participants (51%) had a PND score higher
than 70; 41 (75%) had a PEM score higher than 70; 40 (73%) had a PEM-T score higher than 70; and 50 (91%) had a PDO2 score higher than 70.
• Using visual analyses: – only 13 participants were included across the
one study that provided strong evidence and in the six studies that provided moderate evidence.
Qi & Barton, under review
Social Stories for Children with ASD
• Based on visual analysis, social stories interventions were not considered an EBP according to WWC criteria.
• Based on non-overlap indices, social stories interventions were considered an EBP according to WWC criteria.
Qi & Barton, under review
R-IRD: Robust Improvement Rate Difference
R-IRD: Robust Improvement Rate Difference
• Not compromised by serial dependency and other data assumptions
• Not about rate • IRD calculation begins as PAND, but in a second
step converts the results to two improvement rates (IR), for phase A and B respectively.
• The two IR values are finally subtracted to obtain the “Improvement Rate Difference” (IRD)
• R-IRD requires rebalancing (by hand) of a 2 x 2 matrix
R-IRD: Robust Improvement Rate Difference
• The original IRD article recommended that in the first step, data point ‘removal’ “should be balanced across the contrasted phases” (Parker et al., 2009, p. 141) for more robust results.
• A better robust IRD solution was later described and formalized as “Robust IRD” (R-IRD).
• R-IRD requires rebalancing (by hand) of a 2 x 2 matrix
• IRD is interpreted as the difference in the proportion of high or “improved” scores between phases B and A.
R-IRD: Robust Improvement Rate Difference
• The superior robust version of IRD (R-IRD) requires that quadrants be balanced.– Balancing when a large number of data points
are be removed arbitrarily from one side and a few from the other…
– Does not allow bias in removal of data points from A versus B, as some datasets provide two or more equally good removal solutions.
Calculating R-IRD
1. Determine the fewest data points that must be removed to eliminate overlap
2. Balance quadrant W and Z
3. Then balance Y = A – Phase A: W / (W + Y)
4. Then balance X = B – Phase B: X / (X + Z)
5. R - IRD = B – A
Calculating R-IRD
http://www.singlecaseresearch.org/calculators/ird
Flaws: Length of data can impact (Brossart et al., 2013; Manolov et al., 2011)
NAP: Non-overlap of All Pairs
NAP: Non-overlap of All Pairs
• The percentage of data that improve from A to B or operationally, the percentage of all pairwise comparisons from Phase A to B which show improvement or growth (Parker & Vannest, 2009)
Calculating NAP
1. NAP begins with all pairwise comparisons (#Pairs = nA × nB) between phases.
2. Each paired comparison has one of three outcomes: improvement over time (Pos), deterioration (Neg), or no change over time (Tie).
3. NAP is calculated as (Pos + .5 × Tie) / #Pairs.
Practice: Calculating NAP
1 2 3 4 5 6 7 8 9 10 11 12 130
1
2
3
4
5
6
7
8
Phase A: 0 4 3 0 0Phase B: 5 2 3 5 3 5 6 7
1 2 3 4 5 6 7 8 9 10 11 12 13012345678
# of Pairs = 5*8 = 40#Pos = 34, #Neg = 4, #Tie = 2NAP = (#Pos + .5*#Ties)/#PairsNAP = (34 + .5*2)/40NAP = .875
N=5 N=8
NAP Flaws
Is it an Effect Size?
Replication
Magnitude
Can it supplement visual analysis?
1. Level
2. Trend
3. Variability
4. Immediacy
5. Consistency
6. Overlap Vertical analysis
Tau-U
Extension of NAP – but can control for trend.
Tau-U:(Kendall’s Tau + Mann-Whitney U)
• NAP’s major limitation of insensitivity to data trend led to development of a new index that integrates non-overlap and trend: TauU (Parker, Vannest, Davis, & Sauber, 2011).
• Melding KRC and MW-U are transformations of one another and share the same S sampling distribution
• The Tau-U score is not affected by the ceiling effect present in other non-overlap methods, and performs well in the presence of autocorrelation.
• NAP is percent of non-overlapping data, whereas TauU is percent of non-overlapping minus overlapping data.
• Can control for baseline trends
Calculating Tau-U
Simplest Tau (non-overlap only)• Conduct the same pairwise comparisons
(nA × nB = #Pairs) across phases as is NAP, resulting in a Pos, Neg, or Tie for each pair
• The Tau simple non-overlap form (not considering trend) is Tau = (Pos - Neg) / Pairs
• Tau-U can control for baseline trend
Practice: Calculating TauU
1 2 3 4 5 6 7 8 9 10 11 12 130
1
2
3
4
5
6
7
8
Phase A: 0 4 3 0 0Phase B: 5 2 3 5 3 5 6 7
1 2 3 4 5 6 7 8 9 10 11 12 13012345678
# of Pairs = 5*8 = 40#Pos = 34, #Neg = 4, #Tie = 2Tau-U = (#Pos - #Neg)/#PairsTau-U = (34 - 4)/40Tau-U = .75
N=5 N=8
Practice
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
1
2
3
4
5
6
7
8
Days
CB
Rat
e p
er M
inu
te
Tau-U = -.82
Tau-U = .79Tau-U = -.81
78
67
56 pairs, 2+, 6=, 48-
48 pairs, 40+, 6=, 2-
42 pairs, 1+, 6=, 35-
Calculating Tau-USchilling (2004), David
In Seat
A1 to B1: 1 (0.54, 1.46)
B1 to A2: -1 (-1.59, -.41)
A2 to B2: 1 (.36, 1.64)
PAND Average: 98%
(PND Average was 74%)
Engaged
A1 to B1: .83 (.37, 1.29)
B1 to A2: -1 (-1.59, -.41)
A2 to B2: 1 (.36, 1.64)
PAND Average: 100%
(PND Average was 100%)
Calculating Tau-UVaughn (2002)
Disruptive Behavior
Arrival: -1 (-1.708, -0.23)
Mealtime: -.67 (-1.44, .11)
Departure: -1 (-1.78, -.23)
PAND was 95%
PND was 92%
Engaged
Arrival: 1 (.292, 1.71)
Mealtime: .83 (.06, 1.61)
Departure: 1 (.23, 1.76)
PAND was 95%
PND was 92%
0
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
2
4
6
8
Days
Tau-U = 1.0S
ocia
l Ini
tiatio
ns t
o P
eers
Tau-U = 1.0
Tau-U = 1.0
0
1
2
3
4
5
6
7
8
0
1
2
3
4
5
6
7
8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280
2
4
6
8
Days
Tau-U = 1.0S
ocia
l Ini
tiatio
ns t
o P
eers
Tau-U = 1.0
Tau-U = 1.0
Tau-U Flaws
Is it an Effect Size?
Replication
Magnitude
Can it supplement visual analysis?
1. Trend
2. Level
3. Variability
4. Overlap
5. Immediacy
6. Consistency Vertical analysis
Tau-U is the recommended non-
overlap index
Summary
• Complete non-overlap measures offer the most robust option (NAP, Tau-U)– Complete measures equally emphasize all scores– Incomplete measures emphasize particular scores
(e.g., median)
Summary
• Determination of evidence-based practice does not need to involve summary statistic
• Dynamic, flexible nature of SCRD allow for ongoing decision making while maintaining experimental control
• Replication logic can not be ignored
Summary
• Visual Analysis is complex and involves more than overlap– Graphing is important!
• Effect sizes, non-overlap measures should not take the place of visual analysis
Continue to:
1. Determine if the design supports the demonstration of a functional relation and meets design standards
2. Use systematic visual analysis to determine if data support a functional relational
– Not just behavioral change or effect– Report protocol used and perhaps training of VAs– Include predicted data pattern in RQ
3. Consider magnitude and social validity
4. If using an effect size estimator, test and report assumptions
Develop:
Effect sizes that are: • Is consistent with single case research
design logic • Synthesis of SC studies with similar IVs
and DVs• Synthesis rigorous, experimental studies
using SCR and RCTs
ReferencesBroissart, D. F., Vannest, K. J., Davis, J. L., & Patience, M. A. (2014). Incorporating nonoverlap indices with visual analysis for quantifying intervention effectiveness in single-case experimental designs. Neuropsychological Rehabilitation: An International Journal, 24, 464-491.
Ma, H. H. (2006).An alternative method for quantitative synthesis of single-subject research: Percentage of datapoints exceeding the median. Behavior Modification, 30, 598–617.
Parker, R., & Vannest, K. J. (2008). An improved effect size for single case research: Non-overlap of all pairs (NAP). Behavior Therapy, 40, 357-67.
Parker, R. I., Vannest, K. J., & Brown, L. (2009). The improvement rate difference for single case research. Exceptional Children, 75, 135–150.
Parker, R. I., Vannest, K. J., & Davis, J. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322.
Wolery, M., Busick, M., Reichow, B., & Barton, E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18-28.
Recommended