1

Measure Up!Benchmark Assessment

Quality Assurance Process

RCANSeptember 10, 2010

2

Measure Up! ObjectiveTo monitor and improve Benchmark Assessments in order to attain the most accurate possible measurement of student achievement with respect to California Content Standards Tests.

Measure Up! Components• Content—Structure & Course Guides• Predictability

– Correlation of Benchmark Exams to CST scores– Association of Benchmark Exams with CST Performance Levels

• Item Analysis– Difficulty– Discrimination– Representative of CST items

Evolving

4

PSUSD Benchmark Structure

CST

~153 ID

60 to 75 Items

Blueprint Aligned

Benchmark#1

~45 ID

~20-35 Items

1st 45 ID Paced

Standards

Partial Match to

CST

Benchmark#2

~90 ID

~20-35 Items

2nd 45 ID Paced

Standards

Partial Match to

CST

Benchmark#3

~135 ID

~20-35 Items

3rd 45 ID Paced

Standards

Partial Match to

CST

5

2008-2009

Algebra I(8th Grade)Aggregation of 3 Benchmarks

2008-209

Algebra I(8th Grade)

CST

Predictability

6

Prof = 38

38

Algebra I BM 08-09 (8th Grade)

Alge

bra

I CST

200

9 (8

th G

rade

)

7

N = 23

N = 119 N = 125

N = 51

N = 5

8

Comparison of 2008-2009 Benchmark Exams with 2009 CST Performance Bands for Grade 8 Algebra I

using STAR Performance Band Cut Points

1 5 38

189

3

32 56

11

14

50

54

3

5

32

18

1

1-Far BelowBasic

2-Below Basic 3-Basic 4-Proficient 5-Advanced

Benchmark Exams Performance Levels

CS

T P

erfo

rman

ce L

evel

s

5-Advanced

4-Proficient

3-Basic

2-Below Basic

1-Far Below Basic

N = 9

N = 70

N = 123

N = 100

N = 21

9

2008-2009

ELA 10th GradeAggregation of 3 Benchmarks

2008-209

ELA 10th Grade

CST

Predictability

10ELA 10th Gd BM 08-09

ELA

10th

Gd

CST

2009

Prof = 53

53

11

N = 26

N = 151

N = 326

N = 249

N = 36

12

Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 10 English Language Arts

using STAR Performance Band Cut Points

18 28 21

5143

7

50

162

30

7

107

85

31

76

35

41-Far Below

Basic2-Below Basic 3-Basic 4-Proficient 5-Advanced

Benchmark Exams Performance Levels

CS

T P

erf

orm

an

ce

Lev

els

5-Advanced

4-Proficient

3-Basic

2-Below Basic

1-Far Below Basic

6 Basic6 Below Basic

7 Proficient

4 Far Below Basic

2 Basic6 Below Basic

2 Advanced

N = 26

N = 138

N = 364

N = 202

N = 58

13

2008-2009

US History11th GradeAggregation of 3 Benchmarks

2008-209

US History11th Grade

CST

Predictability

14

r = .46

US His 11th Gd BM 08-09

US

His

11th

Gd

CST

2009

Prof = 40

63

15

N = 68

N = 116

N = 142

N = 72

N = 7

16

Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 11 US History

using STAR Performance Band Cut Points

48

19 2410

25

24 18

12

25

2644

23

15

30

28

5

5

13

2

3

1-Far BelowBasic

2-Below Basic 3-Basic 4-Proficient 5-Advanced

Benchmark Exams Performance Levels

CS

T P

erfo

rman

ce L

evel

s

5-Advanced

4-Proficient

3-Basic

2-Below Basic

1-Far Below Basic

2 Below Basic2 Far Below Basic

2 Basic

N = 100

N = 84

N = 121

N = 86

N = 14

17

2008-2009

Science8th GradeAggregation of 3 Benchmarks

2008-209

Science8th Grade

CST

Predictability

18

r = .77

Science 8th Gd BM 08-09

Scie

nce

8th G

d CS

T 20

09

Prof = 39

44

19

N = 84

N = 307

N = 494

N = 275

N = 24

20

Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 8 Science

using STAR Performance Band Cut Points

6430 13

67

89

3515

31

110

77

20

6

59

154

111

10

40

154

91

1

5

1-Far BelowBasic

2-Below Basic 3-Basic 4-Proficient 5-Advanced

Benchmark Exams Performance Levels

CS

T P

erfo

rman

ce L

evel

s

5-Advanced

4-Proficient

3-Basic

2-Below Basic

1-Far Below Basic

1 Basic1 Below Basic

N = 168

N = 293N = 319

N = 301

N = 103

21

2008-2009

Math6th GradeAggregation of 3 Benchmarks

2008-209

Math6th Grade

CST

Predictability

22

r = .84

Math 6th Gd BM 08-09

Mat

h 6th

Gd

CST

2009

Prof = 43

48

23

N = 164

N = 393

N = 460

N = 224

N = 31

24

Comparison of 2008-2009 Benchmark Exams with2009 CST Performance Bands for Grade 6 Math

using STAR Performance Band Cut Points

41 57

42

202

89

116

246

43

9129

142

83

416 4 4

3

1-Far BelowBasic

2-Below Basic 3-Basic 4-Proficient 5-Advanced

Benchmark Exams Performance Levels

CS

T P

erfo

rman

ce L

evel

s

5-Advanced

4-Proficient

3-Basic

2-Below Basic

1-Far Below Basic

N = 98

N = 384

N = 472

N = 272

N = 45

2 Basic

8 Proficient5 Advanced

25

2008-2009

ELA4th GradeAggregation of 3 Benchmarks

2008-209

ELA4th Grade

CST

Predictability

26

r = .80

Prof = 44

102

27

N = 88

N = 176

N = 334

N = 388

N = 93

28

Comparison of 2008-2009 Benchmark Exams with 2009 CST Performance Bands for Grade 4 English Language Arts

using STAR Performance Band Cut Points

25 27 8

1842

54

13

39

167

62

8

106

167

31

23

103

169

1

7

7

1-Far BelowBasic

2-Below Basic 3-Basic 4-Proficient 5-Advanced

Benchmark Exams Performance Levels

CS

T P

erfo

rman

ce L

evel

s

5-Advanced

4-Proficient

3-Basic

2-Below Basic

1-Far Below Basic1 Proficient

1 Far Below Basic

N = 51

N = 115

N = 358N = 346

N = 209

29

Item Level Analysis

30

Item Difficulty

• The p value for any item– percentage of correct answers– usually in decimal form

• Ideally p value range is .30 to .80 for most items• For example

– p value of .28 = 28% of the test takers got the item right– p value of .75 = 75% of the test takers got the item right– P value of .95 = 95% of the test takers got the item right

31

Item Difficulty MonitoringItem Number N Students Standard p Value

1 853 NS1.2 .37

2 853 NS1.2 .58

3 853 NS1.2 .77

4 853 MG2.1 .25

5 853 MG2.1 .35

6 853 MG2.1 .40

7 853 PS3.3 .43

8 853 PS3.3 .59

9 853 PS3.3 .75

10 853 PS3.3 .95

32

Item Discrimination

• Is item “discriminating” appropriately between higher & lower scoring students

• Discrimination Index (DI) = difference between how upper half and lower half of students score on an item

• DI ranges between -1 and +1• We want items to discriminate positively

33

Item Discrimination MonitoringItem

NumberN

Students Standard p Value DI Max DI

1 853 NS1.2 .37 .45 .742 853 NS1.2 .58 .53 .843 853 NS1.2 .77 .30 .464 853 MG2.1 .25 -.12 .505 853 MG2.1 .35 .10 .706 853 MG2.1 .40 .62 .807 853 PS3.3 .43 .40 .868 853 PS3.3 .59 .58 .829 853 PS3.3 .75 .41 .50

10 853 PS3.3 .95 .09 .10

34

2008-2009

7th Grade Math3rd Benchmark

2009-2010

7th Grade Math3rd Benchmark

Representative of CST Items and our continual Revision Process

35

7NS1.6

2008-2009 Item

p = .33DI = .34

36

RTQ for 7NS1.6

CST = 1RTQ = 1

37

7NS1.6

2009-2010 Item

p = .42DI = .31

38

Additionally…

• Item #3 replaced (as #3) with item modeled after RTQ # 16 (7NS1.7*)—CST = 5, RTQ = 7

• Item #15 replaced (as #13) with item modeled after RTQ #47 (7AF1.5)

• Item #s 23 & 24 replaced (as #s 21 & 22) with items modeled after RTQ #s 89 & 88 (7MG3.4*)

39

Measure Up! Next Steps

• Benchmark Exam Structure• Institutionalize System• CAHSEE• Writing Prompt• Discrimination• Distractor Shaping

40

Questions?