School Accountability and School Choice: Effects on Student Selection across Schools

Comments:Eric J. Brunner

University of Connecticut

Overview

• In May of 1999, Florida introduced the A+ Accountability Plan which assigned letter grades, based on student achievement and a set of other factors, to all schools in the state.

• Cassandra and David examine the effect of this accountability plan on student sorting across elementary schools and the subsequent effect of that sorting on school peer composition.

• Main Findings:– Schools that were awarded an “A” grade in the initial year of the program

witnessed a significant increase in the proportion of high-SES students attending those schools post policy.

– The impact of receiving an “A” grade is strongest within school districts that allow for intra-district choice and those that offer families a greater variety of schools to choose from.

– Little evidence that the accountability plan affected student sorting across schools receiving other grades.

Overview

• Interesting and policy relevant paper that attempts to add to the growing body of literature on the intended and unintended effects of school accountability programs.

• To date, literature on school accountability has focused primarily on:

1) The impact of accountability programs on student achievement– Carnoy and Loeb (2002), Hanushek and Raymond (2005), Jacob (2005), Figlio

and Rouse (2006), West and Peterson (2006), Rockoff and Turner (2010), and Chakrabarti (2013).

2) How accountability programs affect the behavior of school administrators and teachers

– Cullen and Reback 2002, Jacob 2005, Figlio 2006, Figlio and Getzler 2006, Figlio 2006, Figlio and Winicki 2005, Jacob 2005, Jacob and Levitt 2003, Neal and Schanzenbach 2010, Reback 2008).

Overview

• What has been largely overlooked in this literature is the potential effect of school accountability programs on student sorting across schools and hence the peer composition of schools.

• This paper adds to the existing literature by addressing that issue.

Frequent Grade Reassignments

• As noted by Figlio and Lucas (2004), there was a significant amount of churn in the grades assigned to schools in the years immediately surrounding the implementation of the policy.

– “Over 80 percent of schools receiving a grade of “A” in 1999, received a grade of “B” or worse in 2000 or 2001, and nearly 40 percent of 1999’s “A” schools received a grade of “C” or worse in 2000 or 2001.”

Number of Different Grades Received between 1999 and 2001

Number of Grades A B C1 19.6 10.1 31.22 59.6 54.5 56.23 11.8 35.4 12.6

School Rating in May 1999

Frequent Grade Reassignments

• Authors correctly anchor their analysis by using only the 1999 grades to identify the effect of the A+ program on student sorting – using actual grades would certainly be problematic since those grades are endogenous.

• However, using the 1999 grade as the “treatment” potentially introduces significant measurement error due to the amount of grade reassignment that went on in subsequent years.

• Two ways to deal with this issue. The first (and one use by the authors) is to simply estimate a reduced form model based on the original assignment of grades.

• Other is to use actual grades but instrument for those grades using the initial (and plausibly exogenous) original assignment of grades.

Frequent Grade Reassignments

• Need to discuss impact of potential measurement error on results.

• Given that grades are assigned in May or June, measurement error is most likely worst in 2001. • In 1999, parents only saw 1999 grade if anything, and in 2000,

parents probably made school choice decisions based on 1999 grades (since 2000 grades not released until June 2000). In 2001, parents most likely based decisions on 2000 grades.

• Expect 2001 estimates to suffer from attenuation bias.

Identification Strategy

• Main analysis is based on “event study” model of the following form:

• Should we be controlling for district specific trends?– School districts in Florida are county-based and thus large, could add county-

by-year fixed effects to insulate estimates from district-specific trends in unobservables.

• As noted by Wooldridge (2002) among others, for T > 2, the first difference estimator is more efficient than the fixed effects estimator when the errors exhibit serial correlation. – Move to a first-difference specification.

• Should be ?– Variables that determine grades in period t are endogenous thus only use pre-

program determinants of grades as controls.

Identification Strategy

• In short, I might suggest a baseline specification of the following form:

,

where is a set of district-by-year fixed effects and is a vector of the variables that determine grades measured in the pre- policy base-year of 1998.

Segregation Analysis

• Why the focus on cities as opposed to school districts for this analysis?

• Left wondering what else might have been going on at same time that could also be driving these results. A couple of obvious candidates are: – Charter schools were introduced in 1996 and 201 were in operation by 2001-02

with most of those being introduced right around the time of the introduction of the accountability plan

– Controlled open enrollment was beginning to take off in several counties– Opportunity Scholarships were introduced in 1999.

• Some discussion of these possible confounding effects is probably worthwhile.

General Point

• In the introduction and perhaps in several other places in the paper (e.g. conclusion) the authors might want to highlight exactly what the question is they are attempting to answer.

• The question is really whether parents respond to information that is

provided by a letter grade that is above and beyond the information that is conveyed through the variables that go into determining the grade.

• It is the above and beyond part that I think gets lost as the reader moves through the paper. Really the question is whether highly salient information matters even if that information is already built into school quality but perhaps hard to obtain.