Re-examining Prediction of Freshman Grade-Point Average in the CUNY system

Daniel Koretz Harvard Graduate School of Education

CUNY Graduate Center

March 26, 2015

Thanks

• To CUNY and staff for providing CUNY data and assistance—especially

−David Crook −Colin Chellman − Zun Tang

(2)

(3)

Three research topics

1. How do Regents scores compare to SAT scores as predictors of freshman GPA (FGPA) in Senior and Comprehensive colleges? − Potential for different score inflation from coaching

2. How do the does the prediction of FGPA differ within and between schools?

3. How are the benefits distributed? Which types of students and schools win and lose?

Status of work

• Study 1: working paper completed and available

• Study 2: results presented today are preliminary; we expect a working paper by summer

• Study 3: just started; we expect to complete a working paper by fall

• Conducted for 2010 and 2011 cohorts; 2010 reported here

(4)

Background

• Validation studies typically predict FGPA from high school grade point average (HSGPA) and SAT (or ACT) scores

• Use a single-level student regression model − Effects of differences among schools confounded

with differences among students within schools − Problematic—for example, between-school

differences in grading standards

• Done campus-by-campus because of between-campus differences in grading standards

(5)

Study 1: comparing Regents to SAT scores

• Used a traditional single-level model: regressed FGPA on HSGPA, scores, and the combination of the two

• All predictors were standardized for comparability

• Analyzed SAT and Regents (Math A and ELA) separately, and SAT and Regents together

• Initially conducted analysis separately by campus − Results differed among campuses − But unnecessary for overall results in the CUNY data

• Final analyses were using data pooled across campuses

(6)

Primary conclusions • HSGPA predicts substantially better than either set of scores

− Larger difference than in national studies may reflect more refined CUNY HSGPA measure

• Adding scores to HSGPA improves aggregate prediction slightly

• Regents and SAT scores provide similar aggregate prediction

• Subject-specific and composite scores provide similar aggregate prediction

• Adding a second test has only trivial effects on aggregate prediction

(7)

Simple correlations, CUNY and CEEB

(8)

CEEB correlations from Kobrin, J. L. et al. (2008), Validity of the SAT for Predicting First-Year College Grade-Point Average. NY: CEEB, RR 2008-5.

CUNY CEEBHigh School GPA 0.50 0.36SAT Total Score 0.37 0.32SAT Math 0.35 0.26SAT Critical Reading 0.31 0.29Regents Math 0.36Regents English 0.35

OLS regressions, composite scores

(9)

SAT RegentsHSGPA 0.42*** 0.39***SAT Total 0.19***Regents Total 0.19***R 2 0.28 0.27

OLS regressions, subject scores

(10)

SAT RegentsHSGPA 0.42*** 0.39***SAT Math 0.10***SAT Critical Reading 0.12***Regents Math 0.11***Regents ELA 0.13***R 2 0.28 0.28

Mean FGPA by Mean HSGPA

(11)

Study 2: predictive relationships within- and between schools

• Issue: prediction among students within high schools may differ from predictions between schools

• Hypothesis: because of between-high-school differences in grading standards:

− Predictive power of HSGPA will be weaker between schools than within

− Predictive power of test scores will be comparable or greater between schools than within

(12)

Modeling approach

• Two-level random-coefficients, fixed-slopes models, school-mean-centered, with aggregates entered as predictors at the school level

• Student level: estimates within-school relationships, pooled across high schools

• School level: estimates relationships between school means on predictors and school mean FGPA

(13)

Example of two-level model

(14)

Let : G = grades S = scores i index individuals j index schools

𝑌𝑌𝑖𝑖𝑖𝑖 = 𝛽𝛽0𝑖𝑖 + 𝛽𝛽10𝐺𝐺𝑖𝑖𝑖𝑖 + 𝛽𝛽20𝑆𝑆𝑖𝑖𝑖𝑖 + 𝜀𝜀𝑖𝑖𝑖𝑖 𝛽𝛽0𝑖𝑖 = 𝛾𝛾00 + 𝛾𝛾01�̅�𝐺∙𝑖𝑖 + 𝛾𝛾02𝑆𝑆∙̅𝑖𝑖 + 𝑢𝑢𝑖𝑖

Results: single composite variables

• Included only one of the three predictors: SAT composite, Regents composite, or HSGPA

• All three variables showed stronger prediction between schools than within

• That is, a 1-unit difference between two students in one school had a smaller effect on FGPA than a 1-unit difference in school means

(15)

Two-level regressions, single predictor

(16)

HSGPA SAT RegentsStudent-LevelHS GPA 0.39***SAT Total 0.27***Regents Total 0.32***

School-LevelAverage HS GPA 0.49***Average SAT Total 0.36***Average Regents Total 0.39***

Results: HSGPA and composite scores together

• Similar results for SAT and Regents

• As predicted, HSGPA predicts less strongly between schools than within

• Composite scores predict much more strongly between schools than within

(17)

Two-level regressions, composite scores

(18)

SAT RegentsStudent-LevelHS GPA 0.37*** 0.35***SAT Total 0.05***Regents Total 0.09***

School-LevelAverage HS GPA 0.24*** 0.20***Average SAT Total 0.27***Average Regents Total 0.30***

Two-level regressions, subject scores

• Again, similar results for both tests

• Again, HSGPA predicts less well between schools than within

• Pattern shown by composite scores is more a result of math:

− Verbal scores predict similarly or better between schools than within

−Math scores predict only between schools

(19)

Two-level regressions, subject scores

(20)

SAT RegentsStudent-LevelHS GPA 0.38*** 0.36***SAT Math -0.01SAT Verbal 0.07***Regents Math 0.00Regents English 0.09***

School-LevelAverage HS GPA 0.24*** 0.19***Average SAT Math 0.18***Average SAT Verbal 0.12***Average Regents Math 0.25***Average Regents English 0.09**

Study 3

• Results of the first two studies show that different prediction models will rank students and schools differently but don’t identify winners and losers

• Study 3 will examine which types of students win and lose with different prediction approaches

(21)

Implications

• Understanding the predictive value of scores and grades requires contrasting prediction within- and between schools

• Need to distinguish two questions: − Aggregate strength of prediction −Who benefits and loses from different sets of

predictors

• Value of scores may be less the improvement in aggregate prediction than leveling the playing field between high schools

(22)

Future directions

• Examine who wins and loses (Study 3) −Results of studies 1 and 2 show that the choice of

predictors should matter

• Replicate for new Common Core tests. May differ because of:

−Different content and difficulty − Initially, less opportunity for score inflation

• Improve analytical methods to address problematic distributions of key variables

(23)

Supplementary slides

24

Campus-specific results

(25)

Baruch Brooklyn City Hunter John Jay Queens York Medgar Evers

NYCCT Staten Island

HSGPA 0.490*** 0.258*** 0.371*** 0.454*** 0.265*** 0.328*** 0.337*** 0.411*** 0.312*** 0.395***SAT Math -0.012 -0.021 0.017 0.028 0.077* 0.036 0.075 0.1 0.090** -0.02SAT CR 0.124*** 0.117** 0.06 0.079** 0.113*** 0.102* 0.039 0.011 0.091*** 0.128***Regents Math 0.127*** 0.037 -0.037 0.058 0.027 -0.001 -0.002 -0.025 0.089** 0.137***Regents ELA -0.002 0.096* 0.126*** 0.108*** 0.097** 0.097* 0.111** 0.067 0.027 0.055R 2 0.31 0.14 0.18 0.28 0.15 0.19 0.15 0.21 0.19 0.28N 817 704 953 1,128 1,243 766 878 514 1,749 1,315

Example of problematic distribution: FGPA by HSGPA

(26)

Student-level correlations among predictors

(27)

HSGPA SAT Regents HSGPA 1SAT total 0.43 1Regents total 0.56 0.74 1