Making Data Driven Decisions: Cut points, Curve Analysis and Odd Balls Laura Lent IU 13 Pennsylvania

Making Data Driven Decisions:Cut points, Curve Analysis and Odd Balls

Laura Lent

IU 13

Pennsylvania

District-Wide RtI Development

• Essential Question: How do you use data to facilitate systemic paradigm shifts?

District Demographics

• 3200 students total• 5 Elementary Buildings, 2 Secondary• 4 K-4 buildings ranging from 110 to 500

students• 1 5-6 building of 630 students• 25-35% FRL• <15% minority• Equivalent of 2.5 full-time school

psychologists

PSSA Scores: Reading

• Warning=(W), School Improvement 1=SI 1

Grade 2006 2007 2008 2009

3 75 80 85 81

4 70 74 72 75

5 59 66 (W) 60 (SI 1) 62 (SI 1)

6 69 73 70 72

Intensive Reform Focus

• 5th/6th grade building the “identified patient” due to School Improvement status.

• Building has 5 teacher teams of a pair of 5th grade teachers with a pair of 6th grade teachers.

• Two 5th grade classrooms allocated as “ELM” or Essentials of Literacy and Math.

• Each team has a learning support teacher assigned to it who does inclusion during social studies and science.

• No universal screening. No consistent reading instructional practices.

Data Collection

• Use of CBMs for Universal Screening and Progress Monitoring including:

• Early Literacy, Early Numeracy, MAZE, R-CBM, Math Applications and Single Digit Computation.

• K Screening: Use of Individual Growth and Development Indicators (IGDIs)-Picture Naming and Rhyming and Early Numeracy in August to form heterogeneous classes.

Benchmark Data Analysis

• Move from DIBELS to AIMSweb allowed for determination of local benchmark criteria to be used in the reporting system.

• Question: How do we set benchmarks that are sensitive and specific to PSSA performance?

ROC Curve Analysis

• Receiver Operating Characteristic (ROC)

• Statistical Evaluation Process to identify benchmarks by identifying cut scores that are the most sensitive without sacrificing specificity.

• Sensitivity: Fewest False Negatives

• Specificity: Fewest False Positives

Scatterplot Interpretation of Screening Results

False Positives/

“Happy Surprises”

Adapted from Silberglitt (2009).

True Negatives

True Positives False Negatives/

“Unhappy Surprises”

PS

SA

P

rofi

cien

t

R-CBM

Low Risk

Benefits of ROC Curve Analysis

• Re-set Benchmarks for Greater Classification Accuracy.

• Minimize the “word caller” hysteria.

• Fringe Benefits: “Odd Ball” or Outlier Score Investigation.

Acknowledgement

Thanks to Dr. Edward Shapiro and Dr. Gini Hampton, Center for Promoting Research to Practice, Lehigh University, for sharing the proceeding slides of ROC curve analysis for this district.

ROC Curve Analysis

• R-CBM District Benchmarks- 25th %tile

Grade Score - WCPM

3 80

4 104

5 114

6 130

Probability Outcomes based on AIMSweb (25th %tile) Spring Benchmarks

• Grade 4 = 104     

Adv/Proficient

Basic/Below Basic Total

RCBM BMK=104 Low Risk Count 172 28 200% within RCBM BMK=104

86.0% 14.0% 100.0%

% of Total 68.3% 11.1% 79.4%Some Risk Count 15 20 35

% within RCBM BMK=104

42.9% 57.1% 100.0%

% of Total 6.0% 7.9% 13.9%At Risk Count 1 16 17

% within RCBM BMK=104

5.9% 94.1% 100.0%

% of Total .4% 6.3% 6.7%Total Count 188 64 252

% within RCBM BMK=104

74.6% 25.4% 100.0%

% of Total 74.6% 25.4% 100.0%

Probability Outcomes based on Reset R-CBM Spring Benchmarks

• Grade 4 = 117 Reading PSSA Level

Adv/Prof Basic

Below Basic Total

RCBM ROC B=117

Low Risk Count 143 18 0 161

% within RCBM ROC B=117 88.8% 11.2% .0% 100.0%

% of Total 56.7% 7.1% .0% 63.9%

Some/At Risk

Count 45 27 19 91

% within RCBM ROC B=117

49.5% 29.7% 20.9% 100.0%

% of Total 17.9% 10.7% 7.5% 36.1%

Total Count 188 45 19 252

% within RCBM ROC B=117 74.6% 17.9% 7.5% 100.0%

% of Total 74.6% 17.9% 7.5% 100.0%

Probability Outcomes based on AIMSweb (25th %tile) Spring Benchmarks

• Grade 5 = 114     

Adv/Proficient

Basic/Below Basic Total

RCBM BMK=114 Low Risk Count 167 72 239% within RCBM BMK=114

69.9% 30.1% 100.0%

% of Total 58.0% 25.0% 83.0%Some Risk Count 7 20 27

% within RCBM BMK=114

25.9% 74.1% 100.0%

% of Total 2.4% 6.9% 9.4%At Risk Count 2 20 22

% within RCBM BMK=114

9.1% 90.9% 100.0%

% of Total .7% 6.9% 7.6%Total Count 176 112 288

% within RCBM BMK=114

61.1% 38.9% 100.0%

% of Total 61.1% 38.9% 100.0%

Probability Outcomes based on Reset R-CBM Spring Benchmarks

• Grade 5 = 140 Reading PSSA Level

Adv/Prof Basic

Below Basic Total

RCBM Bmk=140 Gr 5

Low Risk Count 135 29 7 171

% within RCBM Bmk=140 Gr 5

78.9% 17.0% 4.1% 100.0%

% of Total 46.9% 10.1% 2.4% 59.4%

Some/At Risk

Count 41 40 36 117

% within RCBM Bmk=140 Gr 5

35.0% 34.2% 30.8% 100.0%

% of Total 14.2% 13.9% 12.5% 40.6%

Total Count 176 69 43 288

% within RCBM Bmk=140 Gr 5

61.1% 24.0% 14.9% 100.0%

% of Total 61.1% 24.0% 14.9% 100.0%

Probability Outcomes based on AIMSweb (25th %tile) Spring Benchmarks

• Grade 6 = 130

     Adv/Proficient

Basic/Below Basic Total

RCBM BMK=130 Low Risk Count 176 39 215

% within RCBM BMK=130

81.9% 18.1% 100.0%

% of Total 66.2% 14.7% 80.8%Some Risk Count 11 25 36

% within RCBM BMK=130

30.6% 69.4% 100.0%

% of Total 4.1% 9.4% 13.5%At Risk Count 0 15 15

% within RCBM BMK=130

.0% 100.0% 100.0%

% of Total .0% 5.6% 5.6%Total Count 187 79 266

% within RCBM BMK=130

70.3% 29.7% 100.0%

% of Total 70.3% 29.7% 100.0%

Probability Outcomes based on Reset R-CBM Spring Benchmarks

• Grade 6 = 154     

Adv/ProficientBasic/Below

Basic TotalRCBM Bmk=154 Gr 6 Low Risk Count 144 23 167

% within RCBM Bmk=154 Gr 6 86.2% 13.8% 100.0%

% of Total 54.1% 8.6% 62.8%

Some/At Risk Count 43 56 99

% within RCBM Bmk=154 Gr 6 43.4% 56.6% 100.0%

% of Total 16.2% 21.1% 37.2%

Total Count 187 79 266

% within RCBM Bmk=154 Gr 6 70.3% 29.7% 100.0%

% of Total70.3% 29.7% 100.0%

ROC Curve Analysis to Reset Cut Points

• Reset Spring Targets: R-CBM Proficient

Grade Spring R-CBMScore

Spring R-CBM

Percentile3 93 17th

4 117 37th

5 140 42nd

6 154 39th

What to do about those odd balls?

To investigate the scores that were false positives and false negatives or “happy surprises” and “unhappy surprises” the following variables were examined:

• Current and historical PSSA performance• Demographic characteristics including

ELL, Economic Disadvantage (ED), IEP status, gender and teacher assignment.

False Positive Results

• Students who were “at risk” or below the 10th percentile on the spring R-CBM but scored Proficient or above on PSSA.

Grade 3 4 5 6

Number of Scores

0 1 2 0

“Extreme” False Negative Results

or Very Unhappy Surprises

• Students who were “low risk” on spring RCBM and Below Basic on PSSA.

Grade 3 4 5 6

Number of False Positives

4 1 17 6

Who are these students?

• 5th Grade (n=17):– 11 Male, 6 Female– 7 with IEPs, 4 ED, 1 ELL– 3 scored Proficient in 4th grade– Teacher distribution: all had at least 1, three teachers

held 3 or more.• 6th Grade (n=6):

– 5 Female, 1 Male– 1 with IEP, 1 ELL– None scored Proficient in 5th grade

“Near”False Negative Results

• Students who scored “low risk” on RCBM and Basic on PSSA.

Grade 3 4 5 6

Number of False Positive Scores

13 27 55 33

Who are these kids?

• Grade 3:– Evenly distributed across schools (n=3, n=5, n=5)– 3 students ED, 3 students IEP, 1 ELL

• Grade 4:– Evenly distributed across schools (n=7, n=10, n=8)– 5 students “retainees”, 5 ED

• *12/23 were students who previously scored Proficient!

• 5th Grade:

• 12/55 retentions, 12 ED (1 overlap), 2 IEP

• 31 fell from Proficient to Basic

• 2 fell from Advanced to Basic

• *33/47 or 70% were students who previously scored Proficient!

• 6th Grade:

• 6 were retainees, 5/ED, 4/IEP, 0 ELL

• 11/33 or 33% fell from Proficient to Basic!

Teacher Distribution: 5th Grade

• Total # False Negatives =72

• Of the 12 teachers, “F” had the most false positives outside of the combined “ELM” classes.

T A B C D E F G H ELM

Total#FalseNeg.

3 6 5 4 3 10 5 5 22

I

5

J

4

Teacher Distribution: 6th Grade

• Total Number of False Negatives=33

T A B C D E F G H I#

False

Neg

1 6 2 2 4 8 4 5 2

Local Impact of “Odd Ball” Analysis

• Consensus-Building: • Administration conceded that major infrastructure and

implementation changes were needed. Staff agreed.

• Infrastructure:• 5/6 teams were restructured into 5th and 6th only teams. Weaker

teams paired with stronger and 5th grade classrooms moved closer to the office.

• Implementation:• 6 full days professional development on core reading instruction.• Administrators attend all inservice trainings and all team meetings.• R-CBM targets re-set for 09-10 to attempt to better identify students

who need targeted intervention. • Targeted intervention provided daily for 45 minutes by all classroom

teachers and interventionists. Groups matched to identified need.

Questions for Further Analysis

• Do the False Negatives fail to identify those with comprehension problems or fail to identify those who are failing to receive adequate instruction?

• Would MAZE identify the same sample of False Negatives as “at risk”?

• Would a diagnostic or benchmark measure based on grade level standards have identified these students as “at risk”?