CAREER- AND COLLEGE-READY CUT SCORES FOR MEAP AND MME
Presentation to the Michigan State Board of EducationSeptember 13, 2011
2
CONTRIBUTORS
ACT Research & Development Unit National Center for Educational
Achievement Michigan Technical Advisory
Committee MDE/BAA Measurement Research
& Psychometrics Unit
3
ISSUES TO ADDRESS
Identify a score for Proficient on the MME that represents being on track for career and college
Identify a score for Proficient on the highest grade level of MEAP that represents being on track to high school success
Identify a score for Proficient on the lower grade levels of MEAP that represent being on track to success in the next higher grade
Identify a score in each grade level of MEAP or MME that represents attainment of Partial Proficiency
Identify a score in each grade level of MEAP or MME that represents attainment of Advanced skills
4
DEFINING ON TRACK FOR CAREER AND COLLEGE
Assumed that if a student is on track to success in college, then the student is also on track to success in technical career training Therefore, focus on success in college Included 2-year college programs (including job training
programs at 2-year institutions) Success could mean many things:
A, B, or C in college? In a 2-year or 4-year college?
Conducted analyses of all of these scenarios Conducted analyses only of academic success, not of
every factor that leads students to be successful in college
5
DEFINING COLLEGE SUCCESS
2-year versus 4-year colleges Separate analyses were run regarding college
success in 2-year and 4-year institutions The cut scores identified for 2 year versus 4
year institutions were within measurement error of each other
Therefore, all final analyses combined all students from 2-year and 4-year colleges into a single group
6
DEFINING COLLEGE SUCCESS
Separate analyses were run using students achieving an A versus B versus C in their first credit-bearing freshman courses
A and C analyses did not produce usable results
Therefore, all final analyses used the criterion of B or better as the measure of college success
This is also the criterion for success used by ACT in its college readiness benchmarking study
7
TYPES OF ANALYSES CONDUCTED
Three types of analyses conducted Logistic Regression (LR) Signal Detection Theory (SDT) Equipercentile Cohort Matching (ECM)
8
IDENTIFYING PROFICIENT (ON TRACK) CUT SCORES ON MME
Started with data from Michigan Public Institutions of Higher Education Identified appropriate credit-bearing freshman
courses against which to analyze the relationship between MME scores and course grades
Thanks to 2- and 4-year institutions for providing the data
Thanks to President’s Council for providing listings of courses appropriate to tie to MME scores
Using those data, conducted SDT analyses to connect MME to college freshman grades
9
IDENTIFYING PROFICIENT (ON TRACK) CUT SCORES ON MEAP
Used SDT to map backward from 11th grade MME to 7th and 8th grade MEAP Identified the score on MEAP that would
maximize the consistent classifications from MEAP to MME
Used SDT to map from 7th grade MEAP to all other MEAP grades Identified the score on each MEAP grade
that would maximize consistent classifications from grade to grade
10
ACHIEVEMENT LEVELS
Recommend retaining the labels for the four performance categories Not Proficient (Considered “Off Track”) Partially Proficient (Considered “Off
Track”) Proficient (Considered “On Track”) Advanced (Considered “On Track”)
11
IDENTIFYING THE OTHER CUT SCORES SDT can also identify scores on the MME scale that
give certain probabilities of obtaining a B or better in the first credit bearing freshman course
Identified two probabilities that have strong meaning and give cut scores sufficiently far from the MME Proficient (On Track) cut scores. 1/3 probability of B or better (Partially Proficient cut score) 2/3 probability of B or better (Advanced cut score)
These cut scores were identified using SDT for the MME These cut scores were also mapped back to the MEAP
using ECM
12
RECOMMENDED CUT SCORES FOR MEAP AND MME MATHEMATICS
Grade
Partially Proficien
tProficien
tAdvance
d11 1093 1116 1138
8 809 830 865
7 714 731 776
6 614 629 675
5 516 531 584
4 423 434 470
3 322 336 371
13
IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME MATHEMATICS (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)
3 4 5 6 7 8 110%
10%20%30%40%50%60%70%80%90%
100%
37.3% 41.2% 39.1% 39.0% 38.5% 39.7% 36.9%
27.7% 22.3% 22.3% 24.7% 25.4%31.1% 35.1%
32.7% 32.1% 34.1% 33.2% 32.3%24.5% 22.1%
2.3% 4.5% 4.5% 3.0% 3.9% 4.6% 5.9%
AdvancedProficientPartially ProficientNot Proficient
Grade
Perc
en
t in
Cate
gory
14
MATHEMATICS PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)
3 4 5 6 7 8 110
102030405060708090
100
95 91
80 84 8578
52
35 37 39 36 3629 28
With Current Cut ScoresWith Recommended New Cut Scores
Grade
Perc
en
t M
eeti
ng
Pro
-fi
cie
ncy T
arg
ets
15
APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME MATHEMATICS TESTS WITH CURRENT AND RECOMMENDED CUT SCORES
3 4 5 6 7 8 110
102030405060708090
100
3429
3935 32 35
42
72
58 61 58 60 59 58 With Current Cut ScoresWith Recommended New Cut Scores
Grade
Ap
pro
xim
ate
Perc
en
t C
orr
ect
Passin
g S
core
16
RECOMMENDED CUT SCORES FOR MEAP AND MME READING
Grade
Partially Proficien
tProficien
tAdvance
d11 1081 1108 1141
8 796 818 853
7 698 721 760
6 602 619 653
5 501 521 565
4 395 419 478
3 301 324 364
17
IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME READING (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)
3 4 5 6 7 8 110%
10%20%30%40%50%60%70%80%90%
100%
13.1% 13.3% 14.8% 18.7% 20.7%14.8% 16.0%
23.5% 23.0% 19.7%17.9%
23.4%28.8% 30.3%
54.7% 56.7%53.5% 44.3%
42.1% 46.8% 41.1%
8.6% 7.1% 12.0%19.2% 13.8% 9.7% 12.6%
AdvancedProficientPartially ProficientNot Proficient
Grade
Perc
en
t in
Cate
gory
18
READING PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)
3 4 5 6 7 8 110
102030405060708090
100
87 84 85 8479 82
6363 64 66 6456 57 54 With Current Cut
ScoresWith Recommended New Cut Scores
Grade
Perc
en
t M
eeti
ng
Pro
-fi
cie
ncy T
arg
ets
19
APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME READING TESTS WITH CURRENT AND RECOMMENDED CUT SCORES
3 4 5 6 7 8 110
102030405060708090
100
45 45 45 42
5245
51
67 64 64 61
7064
57 With Current Cut ScoresWith Recommended New Cut Scores
Grade
Ap
pro
xim
ate
Perc
en
t C
orr
ect
Passin
g S
core
20
RECOMMENDED CUT SCORES FOR MEAP AND MME SCIENCE
Grade
Partially Proficien
tProficien
tAdvance
d11 1106 1126 1144
8 826 845 863
5 526 553 567
21
IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME SCIENCE (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)
5 8 110%
10%20%30%40%50%60%70%80%90%
100%
51.7%59.1%
46.6%
30.9%26.6%
27.4%
9.7%10.2%
17.3%
7.7% 4.1% 8.7%
AdvancedProficientPartially ProficientNot Proficient
Grade
Perc
en
t in
Cate
gory
22
SCIENCE PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)
5 8 110
102030405060708090
100
78 78
61
17 14
26
With Current Cut ScoresWith Recommended New Cut Scores
Grade
Perc
en
t M
eeti
ng
Pro
-fi
cie
ncy T
arg
ets
23
APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME SCIENCE TESTS WITH CURRENT AND RECOMMENDED CUT SCORES
5 8 110
102030405060708090
100
4840
46
8375
65
With Current Cut ScoresWith Recommended New Cut Scores
Grade
Ap
pro
xim
ate
Perc
en
t C
orr
ect
Passin
g S
core
s
24
RECOMMENDED CUT SCORES FOR MEAP AND MME SOCIAL STUDIES
Grade
Partially Proficien
tProficien
tAdvance
d11 1097 1129 1158
9 899 928 960
6 593 625 649
25
IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME SOCIAL STUDIES (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)
6 9 110%
10%20%30%40%50%60%70%80%90%
100%
13.7%23.4%
18.0%
58.1% 43.3%
40.1%
25.4% 30.3%
32.5%
2.9% 3.1%9.5%
AdvancedProficientPartially ProficientNot Proficient
Grade
Perc
en
t in
Cate
gory
26
SOCIAL STUDIES PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)
6 9 110
102030405060708090
100
75 74 78
2833
42
With Current Cut ScoresWith Recommended New Cut Scores
Grade
Perc
en
t M
eeti
ng
Pro
-fi
cie
ncy T
arg
ets
27
APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME SOCIAL STUDIES TESTS WITH CURRENT AND RECOMMENDED CUT SCORES
5 8 110
102030405060708090
100
43 41 39
66 64 63With Current Cut ScoresWith Recommended New Cut Scores
Grade
Ap
pro
xim
ate
Perc
en
t C
orr
ect
Passin
g S
core
28
REPORTING IMPLICATIONS
MDE staff will apply the cut scores retroactively as if the new cut scores had been in place Applied to four years of data Within one month of approval
No retroactive accountability implications for schools, just to allow for following trends over time
29
CONTACT INFORMATION
Joseph Martineau Executive Director Bureau of Assessment & Accountability Michigan Department of Education