Non-Overlap Methods in Single Case Research Methodology Erin E. Barton, PhD, BCBA-D

Non-Overlap Methods in Single Case Research Methodology

Erin E. Barton, PhD, BCBA-D

Non-overlap Methods

1. PND

2. PEM (ECL)

3. PEM-T

4. PAND

5. R-IRD

6. NAP

7. Tau-U

Rationale: Non-overlap Methods

1. Need to aggregate across studies to determine evidence for practice

Meta-analysis is a well established practice for group experiments

– Magnitude – Aggregate findings – Moderator analyses


1. Need to aggregate across studies to determine evidence for practice

– Many have argued if SCRD will not be included in reviews of evidence-based practices unless an effect size estimator is used

– They are often left out of reviews in disciplines outside of special education


2. Emphasis on “effect sizes” in education research to quantify the magnitude

– Standardized effect sizes are particularly valued– Reviewers compare results across studies having

different outcome measures that otherwise could not be easily compared


3. Meta-analytic techniques are compromised when data are serially dependent – On a single individual– Using the same definitions– Using the same data collection procedures– In the same context – Under the same procedures– Often with short intervals between observations


4. Current meta-analytic techniques are inappropriate for aggregating SCR data

The data patterns must shift consistently in the predicted (therapeutic) direction with each change in experimental condition

Replication logic is used to make judgments about functional relations

The design needs an adequate number of replications of the experimental conditions (internal validity)


5. Non-parametric techniques needed

• Short data sets or few data points• Non-normal or unknown distributions• Unknown parameters

PND: Percent of Non-overlapping Data

PND: Percent of Non-overlapping Data

• One of the oldest of the overlap methods (Scruggs, & Mastropieri, 1998; Scruggs, Mastropieri, & Casto, 1987)

• Used extensively• Easily calculated• Does not assume data are independent

Calculating PND

1. Identify the intended change

2. Drawing a straight line from the highest (or lowest) point in Phase A and counting the number of data point in Phase B above the line

3. Quotient = # above the line / total number in Phase B X 100

Interpreting PND

70% is effective, 50% to 70% is questionable effectiveness, and <50% is no observed effect

(Scruggs & Mastropieri, 1998)

Practice: Calculating PNDSchilling (2004), David

In Seat

A1 to B1:

B1 to A2:

A2 to B2:

Engaged

A1 to B1:

B1 to A2:

A2 to B2:

A1 to B1: 22%

B1 to A2: 100%

A2 to B2: 100%

AVERAGE: 74%

Practice: Calculating PNDSchilling (2004), David

In Seat Engaged

A1 to B1: 100%

B1 to A2: 100%

A2 to B2: 100%

AVERAGE: 100%

Practice: Calculating PNDVaughn (2002)

Disruptive Behavior

Arrival: 100%

Mealtime: 75%

Departure: 100%

AVERAGE: 92%

Engaged

Arrival: 100%

Mealtime: 75%

Departure: 100%

AVERAGE: 92%

Outliers

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

1

2

3

4

5

6

7

8

Days

CB

Rat

e p

er M

inu

te

PND = 0%

PND = 0%

PND = 0%

Functional Relation?

Trends

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627280123456789

10

Days

CB

Rat

e pe

r M

inut

e

PND = 0%

PND = 0%

PND = 0%


Magnitude

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

2

4

6

8

10

12

Days

CB

Rat

e p

er M

inu

te

PND = 100%

PND = 100%

PND = 100%


Outliers

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

1

2

3

4

5

6

7

8

Days

CB

Rat

e p

er M

inu

te

PND = 0%

PND = 0%

PND = 0%


BUT NO ONE USES A-B-A-B…..

You might be thinking….

0

2

4

6

8

10

12

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

2

4

6

8

10

12

Days

Fu

nct

ion

al R

elat

ion

?

PND = 68%

PND = 43%

PND = 86%Soc

ial I

nitia

tions

to

Pee

rs

PND = 68%

PND = 43%

PND = 86%

PND = 66%

TREND?

0

2

4

6

8

10

12

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

2

4

6

8

10

12

Days

Fu

nct

ion

al R

elat

ion

?

PND = 0%

PND = 0%

PND = 0%Soc

ial I

nitia

tions

to

Pee

rs

PND = 0%

PND = 0%

PND = 0%

PND = 0%

MAGNITUDE?

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28012345678

Days

Fu

nct

ion

al R

elat

ion

?

PND = 100%

PND = 100%

PND = 100%

Soc

ial I

nitia

tions

to

Pee

rs

WHAT DOES THAT MEAN?

You SHOULD be thinking….

PND Flaws

• Compared with consensus visual analysis, PND resulted in an error in about 1 of 5 condition changes—high rate of errors (Wolery, Busick, Reichow, & Barton, 2010)

• Compromised by:– Longer data sets, # of

data points – Variability – Outliers – Trends

Should not be used (Brossart et al., 2013; Kratochwill et al., 2010; Parker & Vannest, 2009)

PND Flaws

Is it an Effect Size?

Replication

Magnitude

Can it supplement visual analysis?

1. Level

2. Trend

3. Variability

4. Immediacy

5. Overlap

6. Consistency Vertical analysis

PAND: Percent of All Non-overlapping Data

PAND: Percent of All Non-overlapping Data

• Not compromised by serial dependency or other data assumptions

• Percentage of data remaining after determining the fewest data points that must be removed to eliminate all between-phase overlap

Calculating PAND

1. Count the total number of data points in comparison

2. Identify how many need data points need to be removed to eliminate overlap

3. Count the number of remaining data points

4. Divide count in step 3 by count in step 1

5. X by 100

Practice: Calculating PANDSchilling (2004), David

In Seat

A1 to B1: 94%

B1 to A2: 100%

A2 to B2: 100%

AVERAGE: 98%

(PND Average was 74%)

Engaged

A1 to B1: 100%

B1 to A2: 100%

A2 to B2: 100%

AVERAGE: 100%


Practice: Calculating PANDVaughn (2002)

Disruptive Behavior

Arrival: 100%

Mealtime: 85%

Departure: 100%

AVERAGE: 95%

PND was 92%

Engaged

Arrival: 100%

Mealtime: 85%

Departure: 100%

AVERAGE: 95%

PND was 92%Adding more data points

Extinction burst Variability

0

2

4

6

8

10

12

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

2

4

6

8

10

12

Days

PAND = 96%

PAND = 93%

PAND = 93%Soc

ial I

nitia

tions

to

Pee

rs

PAND = 96%

PAND = 93%

PAND = 93%

PAND = 94%

MAGNITUDE?

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28012345678

Days

PAND = 100%

PAND = 100%

PAND = 100%

Soc

ial I

nitia

tions

to

Pee

rs

PAND Flaws


Replication

Magnitude


1. Level

2. Trend

3. Variability

4. Immediacy

5. Overlap


Should not be used (Brossart et al., 2013; Manolov et al., 2010)

PEM: Percent Exceeding the Median (Ma, 2006)

PEM: Percent Exceeding the Median (Ma, 2006)

• Not compromised by serial dependency and other data assumptions

• Designed to eliminate problem with baseline datum point being at floor or ceiling

– Designed to not rely on the most extreme datum point

– Less influenced by variability in baseline

Calculating PEM

1. Drawing a line at the median of Phase A data through Phase B data

2. Count the number of data points in Phase B above (or below) the line and divide by the total number of data points in Phase B

PEM Flaws

• Compared with consensus visual analysis, PEM resulted in an error in about 1 of 6 condition changes—high rate of errors (Wolery et al., 2010)

Should not be used (Parker, Vannest, & Davis, 2011)

PEM Flaws


Replication

Magnitude


1. Level

2. Immediacy

3. Overlap


PEM-T: Percent Exceeding the Median Trend Line

ECL: Extended Celeration Line

PEM-T: Percent Exceeding the Median Trend Line (Wolery et al., 2010)


• Designed to eliminate problem with baseline datum point being at floor or ceiling

– Designed to not rely on the most extreme datum point

– Less influenced by variability in baseline– Less influenced by trends in data

Calculating PEM-T

1. Graph data on semi-logarithmic chart

2. Calculate and draw a split middle line of trend estimation for Phase A data and extend it through Phase B

3. Count # of Phase B data points above/below the split middle line of trend estimation

4. Divide count from Step 4 by # data points in Condition 2 and multiply quotient by 100

PEM-T Flaws

• Compared with consensus visual analysis, PEM-T resulted in an error in about 1 of 8 condition changes—high rate of errors (Wolery et al., 2010)

PEM-T Flaws


Replication

Magnitude


1. Level

2. Trend

3. Variability

4. Immediacy

5. Overlap


So what, right?

Social Stories for Children with ASD

• 20 studies met design standards with or without reservation, which exceeded the minimum number of five studies set by the WWC as needed to be represented across studies.

• Across 3 research groups

Qi & Barton, under review


• Using Non-overlap indices: – 28 participants (51%) had a PND score higher

than 70; 41 (75%) had a PEM score higher than 70; 40 (73%) had a PEM-T score higher than 70; and 50 (91%) had a PDO2 score higher than 70.

• Using visual analyses: – only 13 participants were included across the

one study that provided strong evidence and in the six studies that provided moderate evidence.



• Based on visual analysis, social stories interventions were not considered an EBP according to WWC criteria.

• Based on non-overlap indices, social stories interventions were considered an EBP according to WWC criteria.


R-IRD: Robust Improvement Rate Difference



• Not about rate • IRD calculation begins as PAND, but in a second

step converts the results to two improvement rates (IR), for phase A and B respectively.

• The two IR values are finally subtracted to obtain the “Improvement Rate Difference” (IRD)

• R-IRD requires rebalancing (by hand) of a 2 x 2 matrix


• The original IRD article recommended that in the first step, data point ‘removal’ “should be balanced across the contrasted phases” (Parker et al., 2009, p. 141) for more robust results.

• A better robust IRD solution was later described and formalized as “Robust IRD” (R-IRD).

• R-IRD requires rebalancing (by hand) of a 2 x 2 matrix

• IRD is interpreted as the difference in the proportion of high or “improved” scores between phases B and A.


• The superior robust version of IRD (R-IRD) requires that quadrants be balanced.– Balancing when a large number of data points

are be removed arbitrarily from one side and a few from the other…

– Does not allow bias in removal of data points from A versus B, as some datasets provide two or more equally good removal solutions.

Calculating R-IRD

1. Determine the fewest data points that must be removed to eliminate overlap

2. Balance quadrant W and Z

3. Then balance Y = A – Phase A: W / (W + Y)

4. Then balance X = B – Phase B: X / (X + Z)

5. R - IRD = B – A

Calculating R-IRD

http://www.singlecaseresearch.org/calculators/ird

Flaws: Length of data can impact (Brossart et al., 2013; Manolov et al., 2011)

NAP: Non-overlap of All Pairs

NAP: Non-overlap of All Pairs

• The percentage of data that improve from A to B or operationally, the percentage of all pairwise comparisons from Phase A to B which show improvement or growth (Parker & Vannest, 2009)

Calculating NAP

1. NAP begins with all pairwise comparisons (#Pairs = nA × nB) between phases.

2. Each paired comparison has one of three outcomes: improvement over time (Pos), deterioration (Neg), or no change over time (Tie).

3. NAP is calculated as (Pos + .5 × Tie) / #Pairs.

Practice: Calculating NAP

1 2 3 4 5 6 7 8 9 10 11 12 130

1

2

3

4

5

6

7

8

Phase A: 0 4 3 0 0Phase B: 5 2 3 5 3 5 6 7

1 2 3 4 5 6 7 8 9 10 11 12 13012345678

# of Pairs = 5*8 = 40#Pos = 34, #Neg = 4, #Tie = 2NAP = (#Pos + .5*#Ties)/#PairsNAP = (34 + .5*2)/40NAP = .875

N=5 N=8

NAP Flaws


Replication

Magnitude


1. Level

2. Trend

3. Variability

4. Immediacy

5. Consistency

6. Overlap Vertical analysis

Tau-U

Extension of NAP – but can control for trend.

Tau-U:(Kendall’s Tau + Mann-Whitney U)

• NAP’s major limitation of insensitivity to data trend led to development of a new index that integrates non-overlap and trend: TauU (Parker, Vannest, Davis, & Sauber, 2011).

• Melding KRC and MW-U are transformations of one another and share the same S sampling distribution

• The Tau-U score is not affected by the ceiling effect present in other non-overlap methods, and performs well in the presence of autocorrelation.

• NAP is percent of non-overlapping data, whereas TauU is percent of non-overlapping minus overlapping data.

• Can control for baseline trends

Calculating Tau-U

Simplest Tau (non-overlap only)• Conduct the same pairwise comparisons

(nA × nB = #Pairs) across phases as is NAP, resulting in a Pos, Neg, or Tie for each pair

• The Tau simple non-overlap form (not considering trend) is Tau = (Pos - Neg) / Pairs

• Tau-U can control for baseline trend

Practice: Calculating TauU

1 2 3 4 5 6 7 8 9 10 11 12 130

1

2

3

4

5

6

7

8

Phase A: 0 4 3 0 0Phase B: 5 2 3 5 3 5 6 7

1 2 3 4 5 6 7 8 9 10 11 12 13012345678

# of Pairs = 5*8 = 40#Pos = 34, #Neg = 4, #Tie = 2Tau-U = (#Pos - #Neg)/#PairsTau-U = (34 - 4)/40Tau-U = .75

N=5 N=8

Practice

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

1

2

3

4

5

6

7

8

Days

CB

Rat

e p

er M

inu

te

Tau-U = -.82

Tau-U = .79Tau-U = -.81

78

67

56 pairs, 2+, 6=, 48-

48 pairs, 40+, 6=, 2-

42 pairs, 1+, 6=, 35-

Calculating Tau-U

www.singlecaseresearch.org

http://www.singlecaseresearch.org/

Calculating Tau-USchilling (2004), David

In Seat

A1 to B1: 1 (0.54, 1.46)

B1 to A2: -1 (-1.59, -.41)

A2 to B2: 1 (.36, 1.64)

PAND Average: 98%


Engaged

A1 to B1: .83 (.37, 1.29)

B1 to A2: -1 (-1.59, -.41)

A2 to B2: 1 (.36, 1.64)

PAND Average: 100%


Calculating Tau-UVaughn (2002)

Disruptive Behavior

Arrival: -1 (-1.708, -0.23)

Mealtime: -.67 (-1.44, .11)

Departure: -1 (-1.78, -.23)

PAND was 95%

PND was 92%

Engaged

Arrival: 1 (.292, 1.71)

Mealtime: .83 (.06, 1.61)

Departure: 1 (.23, 1.76)

PAND was 95%

PND was 92%

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

2

4

6

8

Days

Tau-U = 1.0S

ocia

l Ini

tiatio

ns t

o P

eers

Tau-U = 1.0

Tau-U = 1.0

0

1

2

3

4

5

6

7

8

0

1

2

3

4

5

6

7

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 280

2

4

6

8

Days

Tau-U = 1.0S

ocia

l Ini

tiatio

ns t

o P

eers

Tau-U = 1.0

Tau-U = 1.0

Tau-U Flaws


Replication

Magnitude


1. Trend

2. Level

3. Variability

4. Overlap

5. Immediacy


Tau-U is the recommended non-

overlap index

Summary

• Complete non-overlap measures offer the most robust option (NAP, Tau-U)– Complete measures equally emphasize all scores– Incomplete measures emphasize particular scores

(e.g., median)

Summary

• Determination of evidence-based practice does not need to involve summary statistic

• Dynamic, flexible nature of SCRD allow for ongoing decision making while maintaining experimental control

• Replication logic can not be ignored

Summary

• Visual Analysis is complex and involves more than overlap– Graphing is important!

• Effect sizes, non-overlap measures should not take the place of visual analysis

Continue to:

1. Determine if the design supports the demonstration of a functional relation and meets design standards

2. Use systematic visual analysis to determine if data support a functional relational

– Not just behavioral change or effect– Report protocol used and perhaps training of VAs– Include predicted data pattern in RQ

3. Consider magnitude and social validity

4. If using an effect size estimator, test and report assumptions

Develop:

Effect sizes that are: • Is consistent with single case research

design logic • Synthesis of SC studies with similar IVs

and DVs• Synthesis rigorous, experimental studies

using SCR and RCTs

ReferencesBroissart, D. F., Vannest, K. J., Davis, J. L., & Patience, M. A. (2014). Incorporating nonoverlap indices with visual analysis for quantifying intervention effectiveness in single-case experimental designs. Neuropsychological Rehabilitation: An International Journal, 24, 464-491.

Ma, H. H. (2006).An alternative method for quantitative synthesis of single-subject research: Percentage of datapoints exceeding the median. Behavior Modification, 30, 598–617.

Parker, R., & Vannest, K. J. (2008). An improved effect size for single case research: Non-overlap of all pairs (NAP). Behavior Therapy, 40, 357-67.

Parker, R. I., Vannest, K. J., & Brown, L. (2009). The improvement rate difference for single case research. Exceptional Children, 75, 135–150.

Parker, R. I., Vannest, K. J., & Davis, J. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322.

Wolery, M., Busick, M., Reichow, B., & Barton, E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18-28.

Documents

Non-Overlap Methods in Single Case Research Methodology Erin E. Barton, PhD, BCBA-D