Using Growth Models to Monitor School Performance Over Time: Comparing NCE, Scale and Scores on NRTs and SBTs

American Educational Research AssociationAnnual Meeting

March, 2008

Pete Goldschmidt, Kilchan Choi, Felipe Martinez, and John Novak

Using Growth Model Value Added estimates, do inferences about school change

Examine the role of the metricNCE vs Scale Scores on a Vertically equated assessment.

Examine the role of switching Assessment type NRT vs SBT

Introduction

Summary Parameter Estimates Compared

4

Estimated Initial Status

Residual Initial Status

Estimated Growth

Value Added

Summary of Estimates Compared Using Rank Order Correlations

5

Also compare school ranks based on the residual Initial Status and Value Added estimates

SAT-9 Reading Achievement NCE SS NCE SS NCE SS

Mean Initial status (g000)

Student Predictors

Special Education (010) -0.47 -0.44 -0.47 -0.44 -0.47 -0.44

Low SES (020) -0.36 -0.4 -0.35 -0.4 -0.35 -0.39

LEP (030) -0.34 -0.35 -0.33 -0.34 -0.32 -0.33

Minority (040) -0.48 -0.54 -0.48 -0.54 -0.48 -0.53

Girl (050) 0.1 0.1 0.1 0.1 0.1 0.1

School Predictors

LAAMP Effect (001) 0.03 0.04 0.02 0.03 0.02 0.02

Minority (002) -0.01 -0.01 -0.01 -0.01 -0.01 -0.01

Low (003) 0.13 0.1 0.17 0.15 0.2 0.17

Mean Growth (g100) 0.07 0.64 0.07 0.63 0.07 0.63

Student Predictors

Special Education (110) 0 -0.03 0 -0.03 0 -0.03

Low SES (120) 0.05 0.06 0.05 0.06 0.05 0.06

LEP (130) 0.07 0.07 0.07 0.07 0.07 0.07

Minority (140) -0.03 -0.02 -0.03 -0.02 -0.03 -0.02

Girl (150) 0.01 0.01 0.01 0.01 0.01 0.01

School Predictors

LAAMP Effect (101) 0.01 0.01 0.01 0.01 0.01 0.01

Minority (102) 0.11 0.14 0.12 0.15 0.12 0.16

Low (103) -0.08 -0.08 -0.08 -0.08 -0.08 -0.08

25% 50% 75%

Summary of Results Describing SAT-9 Reading Achievement

6

Sample Test Type Initial Status Growth Growth

R25 Read 0.988 0.936 0.806

Math 0.987 0.963 0.863

R50 Read 0.99 0.932 0.798

Math 0.988 0.964 0.87

R75 Read 0.991 0.932 0.798

Math 0.989 0.964 0.871

0.931

0.929

0.935

0.932

Kendall (Tau) Correlation

Initial Status

0.925

0.925

Spearman Correlation

Correlations Between Value added estimates for NRT for models without student covariates

8

Sample Test Type Initial Status Growth Growth

R25 Read 0.964 0.914 0.779

Math 0.975 0.955 0.849

R50 Read 0.97 0.91 0.775

Math 0.978 0.956 0.857

R75 Read 0.974 0.908 0.776

Math 0.981 0.955 0.857

Spearman Correlation Kendall (Tau) Correlation

Initial Status

0.857

0.905

0.887

0.872

0.898

0.881

Correlations Between Value added estimates for NRT for models with student covariates

9

-0.89

-0.88

-0.87

-0.86

-0.85

-0.84

-0.83

-0.82

-0.81

-0.80

-0.79

0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.70 0.71 0.72 0.73

Efect Size of Growth (scale scores)

Rela

tive B

ias

in G

row

th

Comparison of Relative Bias to the Effect Size of Growth

13

Correlations between School Means by Year:

NRT1 and SBT2

Year

2002

(SAT-CST) 2003

(CAT-CST) 2004

(CAT-CST) Reading 0.971 0.967 0.960 Math 0.949 0.955 0.887

1 NRT consists of SAT9 and CAT6 2 SBT is California Standards Test

Estimated effects of student characteristics in initial Status and Growth

for NRT and SBT Reading Mathematics NRT SBT NRT SBT Mean Initial status (g000) 0.17 -0.13 0.14 0.07

Girl (g010) 0.07 0.15 -0.09 -0.14 Special Education (g020) -0.70 -0.72 -0.71 -0.74 Low SES (g030) -0.47 -0.60 -0.46 -0.54 LEP (g040) -0.14 -0.15 0.04 0.01 Minority (g050) -0.30 -0.31 -0.30 -0.37

Mean Growth (g100) -0.26 0.13 -0.20 -0.01 Girl (g110) 0.09 0.01 0.02 0.05 Special Education (g120) 0.07 0.05 0.03 0.10 Low SES (g130) -0.08 0.00 -0.03 -0.01 LEP (g140) 0.04 0.06 0.02 0.06 Minority (g150) -0.01 0.02 0.01 0.07

Correlations among Value Added estimates based on:

NRT1 and SBT2 Spearman Correlation Kendall (Tau) Correlation

Sample Test Type Initial Status Growth Initial Status Growth

Model 1 -Unconditional Growth

Read 0.979 0.548 0.880 0.340

Math 0.966 0.793 0.830 0.627

Model 2 – Growth with Covariates

Read 0.954 0.740 0.781 0.534

Math 0.933 0.808 0.772 0.659

1 NRT consists of SAT9 and CAT6 2 SBT is California Standards Test

Mathematics

School context and inferences

While individual student characteristics’ impact differ depending on assessment used (though not metric) -particularly for growth,

School enrollment characteristics have virtually no impact inferences between NRT and SBT.

Mathematics

Reading

Mathematics

Relationship between missing scores and school performance

Performance based on Percent MissingSchool Means

NRT -0.42 *CST -0.42 *

Value AddedNRT 0.2CST 0.05

* p < .05

Relationship between missing scores and school performance

Summary – the scale

Using a relative scale for monitoring individual achievement growth when the assessment is vertically equated – significantly under-estimates growth.

Using a relative scale for monitoring school performance based on growth when the assessment is vertically equated – yield very consistent results to using an absolute scale.

No patterns as to where deviations may occur.

Summary – the assessment

Individual results between NRT and CST highly correlated in each year.

Individual student characteristics affect relative performanceAttempting to become more egalitarian?

School results fairly consistent in Mathematics, but not in Language Arts

School characteristics have virtually no impact on changes in inferences or rankings of schools.

Summary – the method

Means highly correlated with student background

Means inversely correlated to misingness

VA added estimates based on individual growth substantively less related to student background

VA added estimates based on individual growth substantively less related to missingness