The Impact of The Impact of Including Predictors Including Predictors and Using Various and Using Various

Hierarchical Linear Hierarchical Linear Models on Evaluating Models on Evaluating

School Effectiveness in School Effectiveness in MathematicsMathematics

Nicole Traxel & Cindy WalkerNicole Traxel & Cindy Walker

University of Wisconsin - University of Wisconsin - MilwaukeeMilwaukee

April 14, 2009April 14, 2009The Milwaukee Mathematics Partnership (MMP) is supported by the National Science Foundation under Grant No. 0314898.

IntroductionIntroduction Value added modelsValue added models

Fair and accurate way to assess the effectiveness Fair and accurate way to assess the effectiveness of schoolsof schools

Determine how much value a school adds to Determine how much value a school adds to student learning by examining student progress student learning by examining student progress over timeover time

Operational definition: Effectiveness=growthOperational definition: Effectiveness=growth Hierarchical linear models can be used to Hierarchical linear models can be used to

implement a value added accountability implement a value added accountability systemsystem Hierarchical because students nested within Hierarchical because students nested within

schoolsschools Can determine how much of growth can be Can determine how much of growth can be

attributed to the student and to the schoolattributed to the student and to the school

Types of Hierarchical Types of Hierarchical Linear ModelsLinear Models

Several different hierarchical linear models Several different hierarchical linear models can be used to assess school effectiveness, can be used to assess school effectiveness, so which is best?so which is best? 2-level hierarchical model predicts final 2-level hierarchical model predicts final

achievement from initial achievement. Can achievement from initial achievement. Can include student level and school level include student level and school level predictors of achievement, not growthpredictors of achievement, not growth

2-level growth model predicts change in test 2-level growth model predicts change in test scores from one year to the next. Can include scores from one year to the next. Can include student level and school level predictors of student level and school level predictors of growth, not achievementgrowth, not achievement

3-level individual growth model predicts 3-level individual growth model predicts achievement and change over time. Can include achievement and change over time. Can include student level and school level predictors of student level and school level predictors of growth and achievementgrowth and achievement

Research QuestionsResearch Questions

Do effectiveness rankings differ Do effectiveness rankings differ depending on which type of model is depending on which type of model is used?used?

Does predictor significance remain Does predictor significance remain constant across model types?constant across model types?

Does including predictors change Does including predictors change effectiveness rankings of school?effectiveness rankings of school?

Sample & MeasuresSample & Measures 7,232 students from 128 school from a 7,232 students from 128 school from a

large urban school district in the Midwestlarge urban school district in the Midwest 33rdrd to 4 to 4thth grade grade 87% minority, 79% receive free/reduced lunch87% minority, 79% receive free/reduced lunch

Mathematics scores on a state mandated Mathematics scores on a state mandated standardized teststandardized test

Math Focus score for each school – “There Math Focus score for each school – “There is a strong focus on increasing student is a strong focus on increasing student achievement in mathematics at my school.”achievement in mathematics at my school.” Gain in Math Focus calculated by subtracting Gain in Math Focus calculated by subtracting

Math Focus score from 1Math Focus score from 1stst year from Math year from Math Focus score from 2Focus score from 2ndnd year year

The Models That Were The Models That Were FitFit

2-Level2-Level 2-Level Gain2-Level Gain 3-Level3-Level

Initial ScoreInitial Score NoneNone Initial ScoreInitial Score

Student RaceStudent Race Student RaceStudent Race Student RaceStudent Race

Student SESStudent SES Student SESStudent SES Student SESStudent SES

Student Race & Student Race & SESSES

Student Race & Student Race & SESSES

Student Race & Student Race & SESSES

Student & School Student & School RaceRace

Student & School Student & School RaceRace

Student & School Student & School RaceRace

Student & School Student & School SESSES

Student & School Student & School SESSES

Student & School Student & School SESSES

Student & School Student & School Race & SESRace & SES

Student & School Student & School Race & SESRace & SES

Student & School Student & School Race & SESRace & SES

Note: Initial Score was included as a student level covariate in all 2-level and 3-level models, but not in the 2-level gain models.

ComparisonsComparisons

Predictor significance across modelsPredictor significance across models Effectiveness rankings across Effectiveness rankings across

predictors being included within each predictors being included within each model typemodel type

Effectiveness rankings across model Effectiveness rankings across model types for models including only initial types for models including only initial score or no predictorsscore or no predictors

Effectiveness rankings across model Effectiveness rankings across model types for models including only initial types for models including only initial score or no predictors validated using score or no predictors validated using gain in Math Focus scoregain in Math Focus score

Predictor SignificancePredictor Significance Student-level SES and Student-level Student-level SES and Student-level

Race were significant predictors of the Race were significant predictors of the average achievement of students within average achievement of students within schools but not of the average growth of schools but not of the average growth of students within schoolsstudents within schools

School-level Race and School-level SES School-level Race and School-level SES were significant predictors of the were significant predictors of the average achievement among schools but average achievement among schools but not of the average growth among schoolsnot of the average growth among schools But not when both were included, due to But not when both were included, due to

collinearitycollinearity

Predictors Do Not Change Predictors Do Not Change EffectivenessEffectiveness2-Level Model 2-Level Gain Model 3-Level Model

Student Level Race 0.999 1.000 1.000

Student Level SES 0.991 1.000 0.999

Student Level Race & SES

0.991 0.999 0.999

Student & School Level Race

0.996 0.999 0.998

Student & School Level SES

0.989 0.999 0.998

Student & School Level Race & SES

0.988 0.998 0.997

Spearman Correlations between the null model and models including predictors for each model type.

Correlations Among Model Correlations Among Model Types (Null Models)Types (Null Models)

2-level and 2-level gain models 2-level and 2-level gain models rr = .090 = .090

2-level and 3-level models2-level and 3-level models rr = .101 = .101

2-level gain and 3-level models2-level gain and 3-level models rr = .993 = .993

Therefore, rankings from 2-level Therefore, rankings from 2-level gain and 3-level models are very gain and 3-level models are very similar to one anothersimilar to one another

Validating Effectiveness Validating Effectiveness RankingsRankings

Pearson correlations between gain Pearson correlations between gain in math focus and effectiveness in math focus and effectiveness estimates from null models of each estimates from null models of each model type.model type. 2-level: 2-level: rr = .034, = .034, pp = .747 = .747 2-level gain: 2-level gain: rr = .200, = .200, pp = .055 = .055 3-level: 3-level: rr = .177, = .177, pp = .089 = .089

Conclusions, Part OneConclusions, Part One

Including predictors, even if they are Including predictors, even if they are significant, does not change significant, does not change effectiveness estimateseffectiveness estimates

Effectiveness estimates from null 2-level Effectiveness estimates from null 2-level model were different from effectiveness model were different from effectiveness estimates from null 2-level gain and 3-estimates from null 2-level gain and 3-level modelslevel models

Effectiveness estimates from 2-level gain Effectiveness estimates from 2-level gain and 3-level models were very similarand 3-level models were very similar

Conclusions, Part TwoConclusions, Part Two

Correlation between 2-level gain and Correlation between 2-level gain and 3-level models and gain in math 3-level models and gain in math focus had higher magnitudes than focus had higher magnitudes than correlation between 2-level model correlation between 2-level model and gain in math focusand gain in math focus

Effectiveness estimates from 2-level Effectiveness estimates from 2-level gain and 3-level models are more gain and 3-level models are more valid than those from 2-level modelvalid than those from 2-level model

So which model type should So which model type should I use?I use?

3-level model has several advantages 3-level model has several advantages over 2-level gain modelover 2-level gain model Includes ALL available data-all Includes ALL available data-all

participants with at least one participants with at least one observation are includedobservation are included

Can include many years of dataCan include many years of data Provides more information (growth and Provides more information (growth and

achievement estimates)achievement estimates)