Presentation to the Michigan State Board of Education September 13, 2011

CAREER- AND COLLEGE-READY CUT SCORES FOR MEAP AND MME

Presentation to the Michigan State Board of EducationSeptember 13, 2011

2

CONTRIBUTORS

ACT Research & Development Unit National Center for Educational

Achievement Michigan Technical Advisory

Committee MDE/BAA Measurement Research

& Psychometrics Unit

3

ISSUES TO ADDRESS

Identify a score for Proficient on the MME that represents being on track for career and college

Identify a score for Proficient on the highest grade level of MEAP that represents being on track to high school success

Identify a score for Proficient on the lower grade levels of MEAP that represent being on track to success in the next higher grade

Identify a score in each grade level of MEAP or MME that represents attainment of Partial Proficiency

Identify a score in each grade level of MEAP or MME that represents attainment of Advanced skills

4

DEFINING ON TRACK FOR CAREER AND COLLEGE

Assumed that if a student is on track to success in college, then the student is also on track to success in technical career training Therefore, focus on success in college Included 2-year college programs (including job training

programs at 2-year institutions) Success could mean many things:

A, B, or C in college? In a 2-year or 4-year college?

Conducted analyses of all of these scenarios Conducted analyses only of academic success, not of

every factor that leads students to be successful in college

5

DEFINING COLLEGE SUCCESS

2-year versus 4-year colleges Separate analyses were run regarding college

success in 2-year and 4-year institutions The cut scores identified for 2 year versus 4

year institutions were within measurement error of each other

Therefore, all final analyses combined all students from 2-year and 4-year colleges into a single group

6

DEFINING COLLEGE SUCCESS

Separate analyses were run using students achieving an A versus B versus C in their first credit-bearing freshman courses

A and C analyses did not produce usable results

Therefore, all final analyses used the criterion of B or better as the measure of college success

This is also the criterion for success used by ACT in its college readiness benchmarking study

7

TYPES OF ANALYSES CONDUCTED

Three types of analyses conducted Logistic Regression (LR) Signal Detection Theory (SDT) Equipercentile Cohort Matching (ECM)

8

IDENTIFYING PROFICIENT (ON TRACK) CUT SCORES ON MME

Started with data from Michigan Public Institutions of Higher Education Identified appropriate credit-bearing freshman

courses against which to analyze the relationship between MME scores and course grades

Thanks to 2- and 4-year institutions for providing the data

Thanks to President’s Council for providing listings of courses appropriate to tie to MME scores

Using those data, conducted SDT analyses to connect MME to college freshman grades

9

IDENTIFYING PROFICIENT (ON TRACK) CUT SCORES ON MEAP

Used SDT to map backward from 11th grade MME to 7th and 8th grade MEAP Identified the score on MEAP that would

maximize the consistent classifications from MEAP to MME

Used SDT to map from 7th grade MEAP to all other MEAP grades Identified the score on each MEAP grade

that would maximize consistent classifications from grade to grade

10

ACHIEVEMENT LEVELS

Recommend retaining the labels for the four performance categories Not Proficient (Considered “Off Track”) Partially Proficient (Considered “Off

Track”) Proficient (Considered “On Track”) Advanced (Considered “On Track”)

11

IDENTIFYING THE OTHER CUT SCORES SDT can also identify scores on the MME scale that

give certain probabilities of obtaining a B or better in the first credit bearing freshman course

Identified two probabilities that have strong meaning and give cut scores sufficiently far from the MME Proficient (On Track) cut scores. 1/3 probability of B or better (Partially Proficient cut score) 2/3 probability of B or better (Advanced cut score)

These cut scores were identified using SDT for the MME These cut scores were also mapped back to the MEAP

using ECM

12

RECOMMENDED CUT SCORES FOR MEAP AND MME MATHEMATICS

Grade

Partially Proficien

tProficien

tAdvance

d11 1093 1116 1138

8 809 830 865

7 714 731 776

6 614 629 675

5 516 531 584

4 423 434 470

3 322 336 371

13

IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME MATHEMATICS (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)

3 4 5 6 7 8 110%

10%20%30%40%50%60%70%80%90%

100%

37.3% 41.2% 39.1% 39.0% 38.5% 39.7% 36.9%

27.7% 22.3% 22.3% 24.7% 25.4%31.1% 35.1%

32.7% 32.1% 34.1% 33.2% 32.3%24.5% 22.1%

2.3% 4.5% 4.5% 3.0% 3.9% 4.6% 5.9%

AdvancedProficientPartially ProficientNot Proficient

Grade

Perc

en

t in

Cate

gory

14

MATHEMATICS PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)

3 4 5 6 7 8 110

102030405060708090

100

95 91

80 84 8578

52

35 37 39 36 3629 28

With Current Cut ScoresWith Recommended New Cut Scores

Grade

Perc

en

t M

eeti

ng

Pro

-fi

cie

ncy T

arg

ets

15

APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME MATHEMATICS TESTS WITH CURRENT AND RECOMMENDED CUT SCORES

3 4 5 6 7 8 110

102030405060708090

100

3429

3935 32 35

42

72

58 61 58 60 59 58 With Current Cut ScoresWith Recommended New Cut Scores

Grade

Ap

pro

xim

ate

Perc

en

t C

orr

ect

Passin

g S

core

16

RECOMMENDED CUT SCORES FOR MEAP AND MME READING

Grade

Partially Proficien

tProficien

tAdvance

d11 1081 1108 1141

8 796 818 853

7 698 721 760

6 602 619 653

5 501 521 565

4 395 419 478

3 301 324 364

17

IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME READING (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)

3 4 5 6 7 8 110%

10%20%30%40%50%60%70%80%90%

100%

13.1% 13.3% 14.8% 18.7% 20.7%14.8% 16.0%

23.5% 23.0% 19.7%17.9%

23.4%28.8% 30.3%

54.7% 56.7%53.5% 44.3%

42.1% 46.8% 41.1%

8.6% 7.1% 12.0%19.2% 13.8% 9.7% 12.6%


Grade

Perc

en

t in

Cate

gory

18

READING PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)

3 4 5 6 7 8 110

102030405060708090

100

87 84 85 8479 82

6363 64 66 6456 57 54 With Current Cut

ScoresWith Recommended New Cut Scores

Grade

Perc

en

t M

eeti

ng

Pro

-fi

cie

ncy T

arg

ets

19

APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME READING TESTS WITH CURRENT AND RECOMMENDED CUT SCORES

3 4 5 6 7 8 110

102030405060708090

100

45 45 45 42

5245

51

67 64 64 61

7064

57 With Current Cut ScoresWith Recommended New Cut Scores

Grade

Ap

pro

xim

ate

Perc

en

t C

orr

ect

Passin

g S

core

20

RECOMMENDED CUT SCORES FOR MEAP AND MME SCIENCE

Grade

Partially Proficien

tProficien

tAdvance

d11 1106 1126 1144

8 826 845 863

5 526 553 567

21

IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME SCIENCE (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)

5 8 110%

10%20%30%40%50%60%70%80%90%

100%

51.7%59.1%

46.6%

30.9%26.6%

27.4%

9.7%10.2%

17.3%

7.7% 4.1% 8.7%


Grade

Perc

en

t in

Cate

gory

22

SCIENCE PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)

5 8 110

102030405060708090

100

78 78

61

17 14

26


Grade

Perc

en

t M

eeti

ng

Pro

-fi

cie

ncy T

arg

ets

23

APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME SCIENCE TESTS WITH CURRENT AND RECOMMENDED CUT SCORES

5 8 110

102030405060708090

100

4840

46

8375

65


Grade

Ap

pro

xim

ate

Perc

en

t C

orr

ect

Passin

g S

core

s

24

RECOMMENDED CUT SCORES FOR MEAP AND MME SOCIAL STUDIES

Grade

Partially Proficien

tProficien

tAdvance

d11 1097 1129 1158

9 899 928 960

6 593 625 649

25

IMPACT OF RECOMMENDED CUT SCORES FOR MEAP AND MME SOCIAL STUDIES (FROM SPRING 2011 MME AND FALL 2010 MEAP DATA)

6 9 110%

10%20%30%40%50%60%70%80%90%

100%

13.7%23.4%

18.0%

58.1% 43.3%

40.1%

25.4% 30.3%

32.5%

2.9% 3.1%9.5%


Grade

Perc

en

t in

Cate

gory

26

SOCIAL STUDIES PASSING RATES (WITH CURRENT AND RECOMMENDED CUT SCORES)

6 9 110

102030405060708090

100

75 74 78

2833

42


Grade

Perc

en

t M

eeti

ng

Pro

-fi

cie

ncy T

arg

ets

27

APPROXIMATE PERCENT CORRECT SCORES NEEDED TO PASS THE FALL 2010 MEAP AND SPRING 2011 MME SOCIAL STUDIES TESTS WITH CURRENT AND RECOMMENDED CUT SCORES

5 8 110

102030405060708090

100

43 41 39

66 64 63With Current Cut ScoresWith Recommended New Cut Scores

Grade

Ap

pro

xim

ate

Perc

en

t C

orr

ect

Passin

g S

core

28

REPORTING IMPLICATIONS

MDE staff will apply the cut scores retroactively as if the new cut scores had been in place Applied to four years of data Within one month of approval

No retroactive accountability implications for schools, just to allow for following trends over time

29

CONTACT INFORMATION

Joseph Martineau Executive Director Bureau of Assessment & Accountability Michigan Department of Education

[email protected]

Documents

Presentation to the Michigan State Board of Education September 13, 2011