DATA ANALYTICS TO SUPPORT K-12 EDUCATION: 25 YEARS OF RESEARCH AND NATIONWIDE IMPLEMENTATION October...
28
DATA ANALYTICS TO SUPPORT K-12 EDUCATION: 25 YEARS OF RESEARCH AND NATIONWIDE IMPLEMENTATION October 20, 2014 Robert H. Meyer, Research Professor, WCER and the La Follette School of Public Affairs, Director of Value- Added Research Center
DATA ANALYTICS TO SUPPORT K-12 EDUCATION: 25 YEARS OF RESEARCH AND NATIONWIDE IMPLEMENTATION October 20, 2014 Robert H. Meyer, Research Professor, WCER
DATA ANALYTICS TO SUPPORT K-12 EDUCATION: 25 YEARS OF RESEARCH
AND NATIONWIDE IMPLEMENTATION October 20, 2014 Robert H. Meyer,
Research Professor, WCER and the La Follette School of Public
Affairs, Director of Value-Added Research Center
Slide 2
Plan for Today Mission of Value-Added Research Center Origins
and National Scale Up Project Example: How should Wisconsin
evaluate principals in 2015?
Slide 3
Mission Develop and implement data analytics and policy tools
that meet the twin standards of: Usefulness in the real world of
children and educators, and teaching and learning Scientific rigor
Embedded research: conduct research informed by the needs of
district and state partners Embody the Wisconsin Idea
Slide 4
Milwaukee Origins Milwaukee Embedded Research (1998)
Architects/engineers: Bill Clune & Norm Webb Leadership: Andy
Porter Funding: The Joyce Foundation, Warren Chapman & John
Luczak MPS: Deb Lindsey VAM Production (2001) WISCONSIN
Slide 5
Expansion in Milwaukee Milwaukee Embedded Researcher (2005)
Gary Cook, Brad Carl A Comprehensive Evaluation of Reading
Instructional Practices and Policies (2005) Evaluation of the NCLB
Supplemental Educational Services Program (2006) Evaluation of
Charter Schools and High Schools (2008) The Integrated Resource
Information System, IES (2008) Milwaukee Area Public and Private
Schools (2009) Performance Management and the Integrated Resource
Information System (2010) Milwaukee IDEAS Project (2012)
WISCONSIN
Slide 6
Wisconsin Design of Prototype Wisconsin State Value-Added
System (1989) Wisconsin SAGE Evaluation (2001) Madison (2008)
Racine(2009) WI State-wide VAM (2009) Production of value-added
estimates Consortia building with regional meetings & advisory
council Educator Effectiveness (2011) Principal Evaluation
WISCONSIN
Slide 7
Nationwide Partner Districts and States Chicago (2006)
Department of Education: Teacher Incentive Fund (TIF) (2006, 2010,
and 2012) New York City (2009) Minnesota, North Dakota & South
Dakota: Teacher Education Institutions and Districts (2009)
Hillsborough County/Tampa (2010) Atlanta (2010) Los Angeles (2010)
Tulsa (2010) Illinois (2012) Oklahoma Gear Up (2012)
Slide 8
Minneapolis Milwaukee Racine Chicago Madison Tulsa Atlanta New
York City Los Angeles Hillsborough County NORTH DAKOTA SOUTH DAKOTA
MINNESOTA WISCONSIN ILLINOIS Research & Data Analytics Projects
Collier County NEW YORK CALIFORNIA OKLAHOMA MICHIGAN
Slide 9
Principal Evaluation: The Wisconsin Policy Context Educator
Effectiveness Design Team (policy makers & stakeholders)
developed specific metrics and weights for teacher and principal
evaluation Separate scores for professional practice and student
outcomes For Principals with Value-Added: SLOs: 1.8 pts (45% of
score) School Value-Added (VA): 1.8 pts (45%) District Choice: (5%)
Schoolwide VA Reading or graduation rate (5%) Principals without
VA: SLOs are 90%
Slide 10
Criteria: Technical and Consequential Validity Am educator
effectiveness metric should be evaluated both with respect to:
Technical validity: the accuracy and fairness of the ratings
Consequential validity: the effects of the system on the behavior
of the principals, and possibly teachers, students, and parents The
two criteria reinforce each other in many instances. Example: Need
to design an evaluation metric so that it does not underestimate
the performance of principals in high poverty schools or in
turn-around schools (a technical validity concern) because this
would create a disincentive for effective principals to accept
assignments in such schools (a consequential validity
concern).
Slide 11
Why Value-Added? Growth in Student Achievement is the Goal
School-level and teacher-level value-added models provide
appropriate metrics for measuring the contributions of teachers and
schools to student achievement The models control for differences
across classrooms and schools that are external to classrooms and
schools, such as prior achievement, poverty, other factors What is
the principals contribution to VA?
Slide 12
Challenge: Principal Impact is Not Immediate Existing research
suggests that it may take several years for the full effects of a
principals effectiveness to kick in and be detected For both
continuing and new teachers School value-added during the first few
years of a principals assignment may be misleading School
value-added may be severely biased downward for highly effective
principals employed in turn-around schools This is undesirable from
the standpoints of both technical and consequential validity.
Slide 13
How Many New Principals?
Slide 14
Options for Measuring Principal Performance School VA
Inaccurate and unfair to principals assigned to turn- around
schools Change in VA Inaccurate and unfair to principals assigned
to high- performing school Principal value-added: Partial
adjustment model Potentially accurate and fair to all principals
Sustainability of high-performance is valued Improvement from low
to high VA is valued
Slide 15
Adjustment from Medium Starting Point
Slide 16
Adjustment from Low Starting Point
Slide 17
Adjustment from High Starting Point
Slide 18
Model of Principal/School VA Impact of new principal is
gradual, with adjustment rate True principal productivity =
Principal/school VA (for principal k at time t) = Principal/school
VA in year prior to start of new principal =
Slide 19
Principal Value-Added (PVA) Solve the equation for principal
productivity and indicate that all parameters are estimates
(denoted by the ^ symbol). As time (t) increases, the principal
rating approaches principal/school VA
Slide 20
Principal Value-Added (PVA) Principal value-added allows for an
adjustment transition from one principal to the next The speed of
the adjustment process is captured simply by a rate of adjustment
parameter An adjustment rate equal to 0.25, for example, implies
that in a single year 25% of the gap between past year and eventual
(true) performance is closed within a single year This rating
metric has the desirable feature of classifying the following as
excellent performance: increases in school value-added over time
and sustainability of high value- added over time (even with no
increase).
Slide 21
A Wrinkle: Is There a Disincentive to Transfer to New
Assignment? High and low productivity principals will typically
receive mid-level ratings during early years in new assignment Yes,
disincentive to transfer for high productivity principals No,
positive incentive to transfer for low productivity principals
Implication: Incorporate performance at prior schools. Bonus: Using
past and current information will substantially improve
reliability
Slide 22
Summary Principal VA controls for school VA in the period prior
to the assignment of a new principal and thus allows for an
adjustment transition from one principal to the next. The
indicators serve different purposes: School value-added measures
the contribution to schools to growth in student achievement,
controlling for factors external to schools (but including the
contributions of past and current principals) The PPR is designed
to measure the related, but different, dimension of the performance
of a particular person the principal.
Slide 23
Summary, continued PVA and school value-added differ only
during the early transition years from one principal to the next
The PVA is relatively noisy during the first year or two of a new
principal assignment due to the need to control for VA productivity
under the prior principal The precision of PVA and school VA are
both improved by measuring performance over many subjects, grades,
and years
Slide 24
Summary, continued Precision is further much improved by
combining performance from previous assignments, although this is
not an option for new principals This approach has the advantage of
neutralizing the incentive for a: High-performing principal to stay
at their current school (to retain the high PVA) Low-performing
principal to leave their current school (to escape the low
PVA)
Slide 25
Frontier: Channels of Principal Impact Changes in the
productivity of continuing teachers and staff Via being an
instructional leader, implementing effective professional
development, etc.) Changes in the productivity of new hires (versus
prior staff) A function of principal effectiveness in recruiting
and retaining effective teachers) Useful to provide separate
measures of school-level value-added for continuing and new
teachers (pending availability of student-teacher links)
Slide 26
Resources
Slide 27
Components of Value-Added
Slide 28
Average & Principal/School VA Average School VA
Principal/School VA