42510011 0010 1010 1101 0001 0100 1011

Making MAP More MeaningfulDavid Dreher, Project Coordinator

Dr. Kathryn Sprigg, Assistant DirectorOffice of Accountability, Highline Public Schools

Dr. Sandra L. Hunt , Literacy CoachBeverly Park Elementary, Highline Public Schools

4251

0011 0010 1010 1101 0001 0100 1011

Overview

• The needs of the data users

• The objectives of the data producers

• The products

• The process

• The implementation

• The results

• The future

4251

0011 0010 1010 1101 0001 0100 1011

What is MAP• Measures of Academic Progress

– Developed by the Northwest Evaluation Association – Norm-referenced assessment – Computerized and adaptive– Performance is reported as a RIT score

• The RIT Scale – Uses individual item difficulty values to estimate

student achievement – A RIT score has the same meaning regardless of grade

level– Equal interval scale

• Highline Public Schools– Three testing windows per year (Fall, Winter, Spring)– Test students in the areas of math and reading– Test students in grades 3-10

4251

0011 0010 1010 1101 0001 0100 1011

The Needs of the Data User• Building staff were saying things

like . . .– “How can we use MAP data to help us

make decisions?” – “How do MAP and WASL

performance compare?”– “I want to know what a student’s

history is with MAP.”– “What is a RIT score?”– “Giving me a RIT score is like telling

me the temperature in Celsius!”

4251

0011 0010 1010 1101 0001 0100 1011

The Objectives for Us

• Include more historical data in reports.

• Make the data more accessible. – Put MAP scores in context with WASL

scores. – Provide indication of a student’s

likelihood of meeting standard.

4251

0011 0010 1010 1101 0001 0100 1011

Some General Challenges

• Fear of Numbers – The products generated had to be fairly

simple to explain and understand.

• Availability of Time– Because it had to be there yesterday it

has to be fairly simple for us to produce.

4251

0011 0010 1010 1101 0001 0100 1011

The Products• Fall Predictions

– Our “best guess” about each student’s performance on the upcoming WASL.

– Used for• Identifying level of risk for not meeting standard • School- and District- level WASL forecasts

• Benchmark, Strategic, Intensive (BSI) Updates – “Status update” produced after each testing window.– “ Coarse filter” based only on MAP.

• Cut Score Document– A “quick reference table” that could be used to help

put a MAP score in context.

4251

0011 0010 1010 1101 0001 0100 1011

Making The Predictions• Snooped and found the best

indicators of WASL success

• Applied linear regression models to generate WASL scores for each student

• Examined the predicted WASL scores

4251

0011 0010 1010 1101 0001 0100 1011

Snooping (Reading)R-Values

WASL 2007 Reading Scale vs.

Grade

3 4 5 6 7 8 10

WASL Reading 2006   0.730 0.769 0.764 0.745 0.755  

WASL Reading 2005       0.727      

WASL Reading 2004             0.723

MAP-R Spring 2007 0.792 0.778 0.799 0.774 0.744 0.726 0.755

MAP-R Winter 2007 0.782 0.782 0.796 0.767 0.780 0.711 0.786

MAP-R Fall 2006 0.769 0.750 0.793 0.768 0.768 0.755 0.806

High MAP-Read (F06, W07, S07) 0.804 0.779 0.816 0.784 0.791 0.746 0.804

High MAP-Read + High MAP-Math 0.814            

High MAP-Read +WASL06 Read scale   0.803 0.835 0.824 0.816  0.780  

High MAP-Read + WASL 04_Rscale             0.787

4251

0011 0010 1010 1101 0001 0100 1011

Snooping (Math)R-values

WASL 2007 Math vs.Grade

3 4 5 6 7 8 10

WASL Math 2006   0.801 0.840 0.875 0.873 0.890  

WASL Math 2005       0.603      

WASL Math 2004             0.856

MAP-Math Spring 2007 0.817 0.862 0.862 0.865 0.888 0.896 0.828

MAP-Math Winter 2007 0.809 0.851 0.860 0.863 0.883 0.906 0.856

MAP-Math Fall 2006 0.794 0.817 0.848 0.833 0.879 0.907 0.865

High MAP-Math (F06, W07, S07) 0.832 0.877 0.879 0.878 0.902 0.915 0.879

High MAP-Math + High MAP-Read 0.846            

High MAP-Math + WASL06 Math scale   0.892 0.896 0.910 0.921 0.930  

Highest MAP-M + WASL04 Math scale             0.908

4251

0011 0010 1010 1101 0001 0100 1011

What we learned by snooping. . .• Correlations were generally good.

– Reading R-value range: 0.711 - 0.835– Math R-value range: 0.603 - 0.921

• Correlations in math were stronger than in reading.

• “Highest MAP” consistently correlated better than any single MAP score.

• Correlations were generally strongest when Highest MAP and WASL 2006 factors were combined.

4251

0011 0010 1010 1101 0001 0100 1011

Regression ModelsFor students with both MAP and 2006 WASL scores (~95%)

WASL 2007 = b0 + b1*Highest MAP + b2*WASL 2006

For students that only had MAP score(s) (~3%)WASL 2007 = b0 + b1*Highest MAP

For students that only had WASL 2006 score (~2%)

WASL 2007 = b0 + b1*WASL 2006

Where: Highest MAP = The student’s highest score on MAP

from the Fall 2006, Winter 2007, or Spring 2007 windows.

Typically Spring 2007.

WASL 2006 = The student’s raw score from the 2006 WASL Spring testing.

4251

0011 0010 1010 1101 0001 0100 1011

Prediction ModelsFor students with both MAP and 2007 WASL scores

WASL 2008 = b0 + b1*Projected MAP + b2*WASL 2007

For students with only MAP score(s)WASL 2008 = b0 + b1*Projected MAP

For students with only WASL 2007 score

WASL 2008 = b0 + b1* WASL 2007

Where: Projected MAP = Projected Spring 2008 MAP score based

on the student’s highest score on MAP from the Winter 2007, Spring 2007 or Fall 2008 windows.

WASL 2007 = The student’s raw score from the 2007 WASL Spring testing.

4251

0011 0010 1010 1101 0001 0100 1011

Projecting MAP to Spring

• For the models with “Projected MAP” as one of the factors individual student performance on MAP in the Spring of 2008 was projected. – The amount of expected growth added to

a student’s Highest MAP score came from NWEA’s Growth Study

4251

0011 0010 1010 1101 0001 0100 1011

Example of Projection and Prediction7th Grade Student in Reading

MAP Scores

Winter 2006 Spring 2007 Fall 2007 Highest MAP

210 212 216 216

Expected MAP Growth

3

WASL 2007Projected MAP

Spring 2008

392 219

Predicted

Prediction Model 2008 WASL

Score 2008 WASL

Range

MAP and WASL 399 392 - 406

MAP Only 402 395 - 409

WASL Only 397 390 - 404

4251

0011 0010 1010 1101 0001 0100 1011

WASL Prediction Range

• Constructed using the SEM values reported in the 2001 WASL Technical Reports.

• Predicted Range = Predicted WASL Score +/- SEM

Grade Level SEM – Reading SEM – Math

3, 4, 5, 6 7 12

7,8 7 16

10 10 12

4251

0011 0010 1010 1101 0001 0100 1011

Examining the Predictions

• What are the predictions saying about how we might do in 2008? – “Forward look”

• How would we have done if we had predicted 2007 WASL scores in the fall of 2006? – “Backward look”

4251

0011 0010 1010 1101 0001 0100 1011

What are the predictions saying about how we might do in 2008?

WASL Reading Trend with 2008 Projections

50

55

60

65

70

75

80

2004 2005 2006 2007 2008

% M

eeti

ng

Sta

nd

ard

4th

5th

6th

4251

0011 0010 1010 1101 0001 0100 1011

What are the predictions saying about how we might do in 2008?

WASL Math Trend with 2008 Projections

20

25

30

35

40

45

50

2004 2005 2006 2007 2008

% M

eeti

ng

Sta

nd

ard

4th

5th

6th

4251

0011 0010 1010 1101 0001 0100 1011

What are the predictions saying about how we might do in 2008?

7th Grade WASL Reading Trend by School Including Predictions

30%

40%

50%

60%

70%

80%

2004 2005 2006 2007 2008

Per

cen

t M

eeti

ng

Sta

nd

ard

Cascade

Chinook

Pacific

Sylvester

4251

0011 0010 1010 1101 0001 0100 1011

What are the predictions saying about how we might do in 2008?

7th Grade WASL Math Trends by School Including Predictions

20%

25%

30%

35%

40%

45%

50%

55%

2004 2005 2006 2007 2008

Per

cen

t M

eeti

ng

Sta

nd

ard

Cascade

Chinook

Pacific

Sylvester

4251

0011 0010 1010 1101 0001 0100 1011

What are the predictions saying about how we might do in 2008?

10th Grade WASL Reading Trend by School with Predictions

40%

50%

60%

70%

80%

90%

100%

2004 2005 2006 2007 2008

% M

eeti

ng

Sta

nd

ard

Highline

Rainier

Aviation

4251

0011 0010 1010 1101 0001 0100 1011

What are the predictions saying about how we might do in 2008?

10th Grade WASL Math Trend by School with Predicitons

30%

40%

50%

60%

70%

80%

90%

100%

2004 2005 2006 2007 2008

% M

eeti

ng

Sta

nd

ard

Highline

Rainier

Aviation

4251

0011 0010 1010 1101 0001 0100 1011

Looking BackwardsHow would we have done predicting the 2007 WASL?

• Successful prediction – Accurately predicting whether a student

would or would not meet standard on the WASL

• Unsuccessful prediction – Predicted to meet standard and did not

• “false positive” (the kind we don’t want)

– Predicted not to meet standard but did • “false negative” (the kind we are okay with)

4251

0011 0010 1010 1101 0001 0100 1011

Looking Backwards - Math

GradeSuccessfulPrediction

(%)

Unsuccessful Prediction

False Positives(%)

False Negatives (%)

3 80.8 7.2 12.0

4 87.3 6.3 6.4

5 86.8 5.2 8.0

6 86.8 4.8 8.4

7 89.8 4.5 5.7

8 89.0 5.3 5.7

10 86.9 5.5 7.6

District 86.8 5.5 7.7

4251

0011 0010 1010 1101 0001 0100 1011

Looking Backwards - Math  Actual Met – Predicted to Meet

Grade Number of Students Percent of Total

3 -55 -4.8

4 -1 -0.1

5 -33 -2.8

6 -45 -3.6

7 -13 -1.2

8 -5 -0.4

10 -19 -2.1

District -171 -2.2

4251

0011 0010 1010 1101 0001 0100 1011

Looking Backwards - Reading

GradeSuccessfulPrediction

(%)

Unsuccessful Prediction

False Positives(%)

False Negatives(%)

3 80.5 9.1 10.4

4 83.1 9.0 7.9

5 81.8 10.3 8.0

6 83.1 7.8 9.0

7 82.5 7.3 10.2

8 81.0 8.3 10.7

10 87.7 8.2 4.2

District 82.7 8.6 8.8

4251

0011 0010 1010 1101 0001 0100 1011

Looking Backwards - Reading  Actual Met - Predicted to Meet

Grade Number of Students Percent of Total

3 -15 -1.3

4 14 1.2

5 27 2.3

6 -15 -1.2

7 -32 -2.9

8 -27 -2.5

10 35 4.0

District -13 -0.2

4251

0011 0010 1010 1101 0001 0100 1011

The Implementation• Fall Predictions

– Rolled out at Fall Math Summit

• Cut Scores– Released in November 2007

• BSI Status Updates– Delivered in February 2008

• Use of the information was determined within each building by principals, coaches, and teachers.

4251

0011 0010 1010 1101 0001 0100 1011

The Results• How good were the predictions?

– We won’t know how good they are until after we get our WASL results.

• Come see our Fall WERA presentation!

• Did the products “work” for the end user.– Feedback has been fairly limited – Most feedback has been positive– Some feedback says more work is still

needed.

4251

0011 0010 1010 1101 0001 0100 1011

The Future• Check the predictions after WASL

results are released

• Continue to refine the products to make them “work” for the end user – “job security”

4251

0011 0010 1010 1101 0001 0100 1011

Work in ProgressCut Scores Document

• Predictions “Roll Out”

• Cut Score Table

• Augmented with BSI graph

• NWEA’s recently released Cut Scores document

4251

0011 0010 1010 1101 0001 0100 1011

Email to Principals• If the prediction range is:

– Entirely below 400 (ex.: 380-396): student has less than a 20% chance on the WASL this spring unless we accelerate their learning.

– Straddles 400 (ex.: 396-410): student has basically a coin-flip chance on the WASL, even if their prediction is above 400.

– Entirely above 400 (ex.: 408-424): student has more than an 80% chance on the WASL in the spring, IF they continue to progress.

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

42510011 0010 1010 1101 0001 0100 1011

Contact Information

David Dreher, Project Coordinator

Dr. Kathryn Sprigg, Assistant Director

Office of Accountability, Highline Public Schools

www.hsd401.org

206-433-2334