Curriculum-Embedded Performance Assessments as a

Measure of Student Learning and

GrowthRUTH CHUNG WEI

STANFORD UNIVERSITY

JUNE 2014

Stanford Center for Assessment, Learning, & Equity

Mathematics Design

Collaborative

Mathematics Design CollaborativeFormative Assessment Lessons

RESEARCH QUESTIONS1. What is the relationship between

performance on curriculum embedded performance tasks to other on-demand measures of student learning (e.g, performance tasks, standardized large-scale assessments)?

2. What is the feasibility of applying growth models to the performance task scores to assess growth in student performance?

Stanford Center for Assessment, Learning, & Equity

DATA SOURCES – LDC STUDY

LDC Tasks

On Deman

d Pre-test

On Deman

d Post-test

District Admin Data

State A(7 districts) 30 teachers(ELA, History, Science)

1 Task Spring 2013

CRESST writing

task

CRESST writing

task

State tests,SAT/ACT,

Demographics,

Attendance,District-level

data,Links to School,

Class, andTeacher

State B(1 district)30 teachers(ELA, History, Science)

1 TaskSpring 2013

CRESST writing

task

CRESST writing

task

DATA SOURCES – MDC STUDY

MDC Study•1 large urban school district

(about 30 Algebra teachers within one school network)

• 8 unit performance tasks• 1 summative performance-based

assessment • District administrative data

(concurrent and prior achievement, demographics, attendance)Stanford Center for Assessment, Learning, & Equity

Preliminary Results LDC Performance Tasks• Can they be scored

reliably?• Are they comparable?

Stanford Center for Assessment, Learning, & Equity

Generalizability Study LDC Scoring Rubrics

Number of RatersDIMENSIONS 1 2 3 4Focus 0.92 0.96 0.97 0.98Controlling Idea 0.86 0.92 0.95 0.96Reading Research 0.83 0.91 0.94 0.95Development 0.89 0.94 0.96 0.97Organization 0.83 0.90 0.93 0.95Conventions 0.69 0.82 0.87 0.90Content Understanding 0.87 0.93 0.95 0.97Total Score 0.91 0.96 0.97 0.98

Estimated Reliability of the ELA Explanatory Task as a Function of Number of Raters Used to Calculate Scores

DISENTANGLING SOURCES OF ERROR VARIANCE

Source

FocusControl-ling Idea

Reading

Research

Dev’t Org.Conve

n-tions

Content

Und.

student

37.85 (91%)

37.52 (86%)

32.28 (83%)

37.7 (88%)

32.92 (81%)

23.66 (60%)

30.48 (86%)

rater0.39 (1%)

0 (0%)

0 (0%)

0.2 (0%)

0.99 (2%)

5.15 (13%)

0.59 (2%)

Error (s x r)

3.36 (8%)

6.29 (14%)

6.58 (17%)

4.8 (11%)

6.93 (17%)

10.68 (27%)

4.41 (12%)

Source Table for the ELA Explanatory Task

Stanford Center for Assessment, Learning, & Equity

Work in Progress Good to Go Exemplary

Task Clarity &

Coherence

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

Content• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

Text(s)• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

Student Product

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

• –––––––––––––––––––––• ––––––––––––––• –––––––––––––––––––––––

LDC TASK JURYING RUBRIC

COMPARABILITY OF LDC TASKS?

4 5 6 7 8 9 10 110

2

4

6

8

10

12

2

810

6 6

21

Distribution of Total Scores on LDC Task Jurying Rubric

Total Score

Fre

qu

en

cy

COMPARABILITY OF LDC TASKS?

Work In Progress

Good to Go Exemplary0

5

10

15

20

2521

12

2

Distribution of Holistic Ratings on LDC Task Jurying Rubric

Holistic Score

Fre

qu

en

cy

RELATIONSHIP BETWEEN TASK QUALITY SCORE AND STUDENT PERFORMANCE – ELA TASKS Total Task

Quality Score

Holistic Task Rating

Total LDC Essay Score

Pearson Correlation .135* .367**

Sig. (2-tailed) .022 .000

 N 290 290Stanford Center for Assessment, Learning, & Equity

DIFFERENCES IN MEAN LDC ESSAY SCORES BY HOLISTIC TASK QUALITY RATING – ELA TASKSHolistic

Task Quality Rating N

Mean LDC

Essay Score

Std. Dev.

Std. Error Min Max

Work in Progress 159 12.9 5.062 0.401 7 28Good to Go 131 17.31 6.135 0.536 7 28Exemplary 0Total 290 14.89 5.979 0.351 7 28Difference in Mean LDC Essay Score between groups is significant at the 0.000 level.

CORRELATION BETWEEN LDC ESSAY SCORES AND CRESST ON-DEMAND TESTS – ELA TASKS

 

Pre Test

Essay

Pre Test Total

Post Test

Essay

Post Test Total

Total LDC

Essay Score

Total LDC Essay Score

Pearson Correlation

.191** .170** .278** .199**

Sig. (2-tailed) .001 .004 .000 .001

N 278 287 278 282Post Test Essay

Pearson Correlation

.692** .796** .848** .199**

Sig. (2-tailed) .000 .000 .000 .001

N 335 348 343 282

ANALYSES TO BE COMPLETED

• Correlational studies of other on-demand assessments with curriculum-embedded assessments

• Growth modeling on longitudinal student achievement data

• Introducing performance assessment data as part of the growth model -- either as a predicted outcome or as a predictor