View
29
Download
0
Category
Tags:
Preview:
DESCRIPTION
Making MAP More Meaningful. David Dreher, Project Coordinator Dr. Kathryn Sprigg, Assistant Director Office of Accountability, Highline Public Schools Dr. Sandra L. Hunt , Literacy Coach Beverly Park Elementary, Highline Public Schools. Overview. The needs of the data users - PowerPoint PPT Presentation
Citation preview
42510011 0010 1010 1101 0001 0100 1011
Making MAP More MeaningfulDavid Dreher, Project Coordinator
Dr. Kathryn Sprigg, Assistant DirectorOffice of Accountability, Highline Public Schools
Dr. Sandra L. Hunt , Literacy CoachBeverly Park Elementary, Highline Public Schools
4251
0011 0010 1010 1101 0001 0100 1011
Overview
• The needs of the data users
• The objectives of the data producers
• The products
• The process
• The implementation
• The results
• The future
4251
0011 0010 1010 1101 0001 0100 1011
What is MAP• Measures of Academic Progress
– Developed by the Northwest Evaluation Association – Norm-referenced assessment – Computerized and adaptive– Performance is reported as a RIT score
• The RIT Scale – Uses individual item difficulty values to estimate
student achievement – A RIT score has the same meaning regardless of grade
level– Equal interval scale
• Highline Public Schools– Three testing windows per year (Fall, Winter, Spring)– Test students in the areas of math and reading– Test students in grades 3-10
4251
0011 0010 1010 1101 0001 0100 1011
The Needs of the Data User• Building staff were saying things
like . . .– “How can we use MAP data to help us
make decisions?” – “How do MAP and WASL
performance compare?”– “I want to know what a student’s
history is with MAP.”– “What is a RIT score?”– “Giving me a RIT score is like telling
me the temperature in Celsius!”
4251
0011 0010 1010 1101 0001 0100 1011
The Objectives for Us
• Include more historical data in reports.
• Make the data more accessible. – Put MAP scores in context with WASL
scores. – Provide indication of a student’s
likelihood of meeting standard.
4251
0011 0010 1010 1101 0001 0100 1011
Some General Challenges
• Fear of Numbers – The products generated had to be fairly
simple to explain and understand.
• Availability of Time– Because it had to be there yesterday it
has to be fairly simple for us to produce.
4251
0011 0010 1010 1101 0001 0100 1011
The Products• Fall Predictions
– Our “best guess” about each student’s performance on the upcoming WASL.
– Used for• Identifying level of risk for not meeting standard • School- and District- level WASL forecasts
• Benchmark, Strategic, Intensive (BSI) Updates – “Status update” produced after each testing window.– “ Coarse filter” based only on MAP.
• Cut Score Document– A “quick reference table” that could be used to help
put a MAP score in context.
4251
0011 0010 1010 1101 0001 0100 1011
Making The Predictions• Snooped and found the best
indicators of WASL success
• Applied linear regression models to generate WASL scores for each student
• Examined the predicted WASL scores
4251
0011 0010 1010 1101 0001 0100 1011
Snooping (Reading)R-Values
WASL 2007 Reading Scale vs.
Grade
3 4 5 6 7 8 10
WASL Reading 2006 0.730 0.769 0.764 0.745 0.755
WASL Reading 2005 0.727
WASL Reading 2004 0.723
MAP-R Spring 2007 0.792 0.778 0.799 0.774 0.744 0.726 0.755
MAP-R Winter 2007 0.782 0.782 0.796 0.767 0.780 0.711 0.786
MAP-R Fall 2006 0.769 0.750 0.793 0.768 0.768 0.755 0.806
High MAP-Read (F06, W07, S07) 0.804 0.779 0.816 0.784 0.791 0.746 0.804
High MAP-Read + High MAP-Math 0.814
High MAP-Read +WASL06 Read scale 0.803 0.835 0.824 0.816 0.780
High MAP-Read + WASL 04_Rscale 0.787
4251
0011 0010 1010 1101 0001 0100 1011
Snooping (Math)R-values
WASL 2007 Math vs.Grade
3 4 5 6 7 8 10
WASL Math 2006 0.801 0.840 0.875 0.873 0.890
WASL Math 2005 0.603
WASL Math 2004 0.856
MAP-Math Spring 2007 0.817 0.862 0.862 0.865 0.888 0.896 0.828
MAP-Math Winter 2007 0.809 0.851 0.860 0.863 0.883 0.906 0.856
MAP-Math Fall 2006 0.794 0.817 0.848 0.833 0.879 0.907 0.865
High MAP-Math (F06, W07, S07) 0.832 0.877 0.879 0.878 0.902 0.915 0.879
High MAP-Math + High MAP-Read 0.846
High MAP-Math + WASL06 Math scale 0.892 0.896 0.910 0.921 0.930
Highest MAP-M + WASL04 Math scale 0.908
4251
0011 0010 1010 1101 0001 0100 1011
What we learned by snooping. . .• Correlations were generally good.
– Reading R-value range: 0.711 - 0.835– Math R-value range: 0.603 - 0.921
• Correlations in math were stronger than in reading.
• “Highest MAP” consistently correlated better than any single MAP score.
• Correlations were generally strongest when Highest MAP and WASL 2006 factors were combined.
4251
0011 0010 1010 1101 0001 0100 1011
Regression ModelsFor students with both MAP and 2006 WASL scores (~95%)
WASL 2007 = b0 + b1*Highest MAP + b2*WASL 2006
For students that only had MAP score(s) (~3%)WASL 2007 = b0 + b1*Highest MAP
For students that only had WASL 2006 score (~2%)
WASL 2007 = b0 + b1*WASL 2006
Where: Highest MAP = The student’s highest score on MAP
from the Fall 2006, Winter 2007, or Spring 2007 windows.
Typically Spring 2007.
WASL 2006 = The student’s raw score from the 2006 WASL Spring testing.
4251
0011 0010 1010 1101 0001 0100 1011
Prediction ModelsFor students with both MAP and 2007 WASL scores
WASL 2008 = b0 + b1*Projected MAP + b2*WASL 2007
For students with only MAP score(s)WASL 2008 = b0 + b1*Projected MAP
For students with only WASL 2007 score
WASL 2008 = b0 + b1* WASL 2007
Where: Projected MAP = Projected Spring 2008 MAP score based
on the student’s highest score on MAP from the Winter 2007, Spring 2007 or Fall 2008 windows.
WASL 2007 = The student’s raw score from the 2007 WASL Spring testing.
4251
0011 0010 1010 1101 0001 0100 1011
Projecting MAP to Spring
• For the models with “Projected MAP” as one of the factors individual student performance on MAP in the Spring of 2008 was projected. – The amount of expected growth added to
a student’s Highest MAP score came from NWEA’s Growth Study
4251
0011 0010 1010 1101 0001 0100 1011
Example of Projection and Prediction7th Grade Student in Reading
MAP Scores
Winter 2006 Spring 2007 Fall 2007 Highest MAP
210 212 216 216
Expected MAP Growth
3
WASL 2007Projected MAP
Spring 2008
392 219
Predicted
Prediction Model 2008 WASL
Score 2008 WASL
Range
MAP and WASL 399 392 - 406
MAP Only 402 395 - 409
WASL Only 397 390 - 404
4251
0011 0010 1010 1101 0001 0100 1011
WASL Prediction Range
• Constructed using the SEM values reported in the 2001 WASL Technical Reports.
• Predicted Range = Predicted WASL Score +/- SEM
Grade Level SEM – Reading SEM – Math
3, 4, 5, 6 7 12
7,8 7 16
10 10 12
4251
0011 0010 1010 1101 0001 0100 1011
Examining the Predictions
• What are the predictions saying about how we might do in 2008? – “Forward look”
• How would we have done if we had predicted 2007 WASL scores in the fall of 2006? – “Backward look”
4251
0011 0010 1010 1101 0001 0100 1011
What are the predictions saying about how we might do in 2008?
WASL Reading Trend with 2008 Projections
50
55
60
65
70
75
80
2004 2005 2006 2007 2008
% M
eeti
ng
Sta
nd
ard
4th
5th
6th
4251
0011 0010 1010 1101 0001 0100 1011
What are the predictions saying about how we might do in 2008?
WASL Math Trend with 2008 Projections
20
25
30
35
40
45
50
2004 2005 2006 2007 2008
% M
eeti
ng
Sta
nd
ard
4th
5th
6th
4251
0011 0010 1010 1101 0001 0100 1011
What are the predictions saying about how we might do in 2008?
7th Grade WASL Reading Trend by School Including Predictions
30%
40%
50%
60%
70%
80%
2004 2005 2006 2007 2008
Per
cen
t M
eeti
ng
Sta
nd
ard
Cascade
Chinook
Pacific
Sylvester
4251
0011 0010 1010 1101 0001 0100 1011
What are the predictions saying about how we might do in 2008?
7th Grade WASL Math Trends by School Including Predictions
20%
25%
30%
35%
40%
45%
50%
55%
2004 2005 2006 2007 2008
Per
cen
t M
eeti
ng
Sta
nd
ard
Cascade
Chinook
Pacific
Sylvester
4251
0011 0010 1010 1101 0001 0100 1011
What are the predictions saying about how we might do in 2008?
10th Grade WASL Reading Trend by School with Predictions
40%
50%
60%
70%
80%
90%
100%
2004 2005 2006 2007 2008
% M
eeti
ng
Sta
nd
ard
Highline
Rainier
Aviation
4251
0011 0010 1010 1101 0001 0100 1011
What are the predictions saying about how we might do in 2008?
10th Grade WASL Math Trend by School with Predicitons
30%
40%
50%
60%
70%
80%
90%
100%
2004 2005 2006 2007 2008
% M
eeti
ng
Sta
nd
ard
Highline
Rainier
Aviation
4251
0011 0010 1010 1101 0001 0100 1011
Looking BackwardsHow would we have done predicting the 2007 WASL?
• Successful prediction – Accurately predicting whether a student
would or would not meet standard on the WASL
• Unsuccessful prediction – Predicted to meet standard and did not
• “false positive” (the kind we don’t want)
– Predicted not to meet standard but did • “false negative” (the kind we are okay with)
4251
0011 0010 1010 1101 0001 0100 1011
Looking Backwards - Math
GradeSuccessfulPrediction
(%)
Unsuccessful Prediction
False Positives(%)
False Negatives (%)
3 80.8 7.2 12.0
4 87.3 6.3 6.4
5 86.8 5.2 8.0
6 86.8 4.8 8.4
7 89.8 4.5 5.7
8 89.0 5.3 5.7
10 86.9 5.5 7.6
District 86.8 5.5 7.7
4251
0011 0010 1010 1101 0001 0100 1011
Looking Backwards - Math Actual Met – Predicted to Meet
Grade Number of Students Percent of Total
3 -55 -4.8
4 -1 -0.1
5 -33 -2.8
6 -45 -3.6
7 -13 -1.2
8 -5 -0.4
10 -19 -2.1
District -171 -2.2
4251
0011 0010 1010 1101 0001 0100 1011
Looking Backwards - Reading
GradeSuccessfulPrediction
(%)
Unsuccessful Prediction
False Positives(%)
False Negatives(%)
3 80.5 9.1 10.4
4 83.1 9.0 7.9
5 81.8 10.3 8.0
6 83.1 7.8 9.0
7 82.5 7.3 10.2
8 81.0 8.3 10.7
10 87.7 8.2 4.2
District 82.7 8.6 8.8
4251
0011 0010 1010 1101 0001 0100 1011
Looking Backwards - Reading Actual Met - Predicted to Meet
Grade Number of Students Percent of Total
3 -15 -1.3
4 14 1.2
5 27 2.3
6 -15 -1.2
7 -32 -2.9
8 -27 -2.5
10 35 4.0
District -13 -0.2
4251
0011 0010 1010 1101 0001 0100 1011
The Implementation• Fall Predictions
– Rolled out at Fall Math Summit
• Cut Scores– Released in November 2007
• BSI Status Updates– Delivered in February 2008
• Use of the information was determined within each building by principals, coaches, and teachers.
4251
0011 0010 1010 1101 0001 0100 1011
The Results• How good were the predictions?
– We won’t know how good they are until after we get our WASL results.
• Come see our Fall WERA presentation!
• Did the products “work” for the end user.– Feedback has been fairly limited – Most feedback has been positive– Some feedback says more work is still
needed.
4251
0011 0010 1010 1101 0001 0100 1011
The Future• Check the predictions after WASL
results are released
• Continue to refine the products to make them “work” for the end user – “job security”
4251
0011 0010 1010 1101 0001 0100 1011
Work in ProgressCut Scores Document
• Predictions “Roll Out”
• Cut Score Table
• Augmented with BSI graph
• NWEA’s recently released Cut Scores document
4251
0011 0010 1010 1101 0001 0100 1011
Email to Principals• If the prediction range is:
– Entirely below 400 (ex.: 380-396): student has less than a 20% chance on the WASL this spring unless we accelerate their learning.
– Straddles 400 (ex.: 396-410): student has basically a coin-flip chance on the WASL, even if their prediction is above 400.
– Entirely above 400 (ex.: 408-424): student has more than an 80% chance on the WASL in the spring, IF they continue to progress.
4251
0011 0010 1010 1101 0001 0100 1011
4251
0011 0010 1010 1101 0001 0100 1011
42510011 0010 1010 1101 0001 0100 1011
Contact Information
David Dreher, Project Coordinator
Dr. Kathryn Sprigg, Assistant Director
Office of Accountability, Highline Public Schools
www.hsd401.org
206-433-2334
Recommended