Searching for Effective TeachersSearching for Effective Teachers with Imperfect Information

Nonprofit Leadership Forum:Measuring and creating excellence in schoolsMeasuring and creating excellence in schools

May 6, 2010

D l O S iDouglas O. StaigerDartmouth College

MotivationMotivation

Huge literature about “teacher effects” on achievementDifficult to predict at hirePartially predictable after hire

Growing use of value added to identify effectiveGrowing use of value added to identify effective teachers for pay, promotion, and professional development

But concern that current value added estimates are too imprecise to be used in high-stakes decisions

How could such measures be used?How could such measures be used?

Use simple search model to illustrate how one could useUse simple search model to illustrate how one could use imperfect information on effectiveness to screen teachers

Use estimates of model parameters from NYC & LAUSD to simulate the potential gains from screening teachers

Evaluate potential gains from:Observing teacher performance for more yearsObtaining more reliable information on teacher performanceObtaining more reliable information at time of hire

Simple search modelSimple search model

Maintained assumptions across all simulationsMaintained assumptions across all simulationsSD of teacher effect = 0.15Turnover rate if not dismissed = 5%

Assumptions for simplest base case (will be varied later)No useful information at time of hireReliability of annual performance measure = 40%Cost of hiring new teacher = -.07 in 1st year, -.02 in 2nd yearDismissal only after first year (e g tenure decision after 1 year)Dismissal only after first year (e.g. tenure decision after 1 year)

Objective: Maximize student achievement by screening out ineffective teachers using imperfect performance measureineffective teachers using imperfect performance measure

Simple model: dismiss 80% of probationary teachers(!)p p y ( )

.808

.6e

.06

.0ed

.4po

rtion

nov

ice

.04

ge v

alue

add

e

.2P

rop

.02

Ave

rag

0

0

0 .2 .4 .6 .8 1Proportion dismissed

Average value added Proportion novice

Why dismiss so many probationary teachers?y y p y

Differences in teacher effects are large & persistent relativeDifferences in teacher effects are large & persistent, relative to short-lived costs of hiring a new teacher

E li bl f di t b t ti lEven unreliable performance measures predict substantial differences in teacher effects

Costs of retaining an ineffective teacher outweigh costs of di i i ff i hdismissing an effective teacher

Option value of new hiresFor every 5 new hires, one will be highly effectiveTrade off short-term cost of 4 dismissed vs. long-term benefit of 1 retained

Why not dismiss so many probationary t h ?teachers?

Smaller benefits than assumed in the model?Smaller benefits than assumed in the model?High turnover ratesTeacher differences that do not persist in future(including if PD can help ineffective teachers)High stakes distortion of performance measures

Larger costs than assumed in the model?Direct costs of recruiting/firing (little effect if added)Difficulty recruiting applicants (but LAUSD did)Higher pay required to offset job insecurity(particularly if require teacher-training up front)(particularly if require teacher training up front)

Requiring a 2nd or 3rd year to evaluate a b ti t h i b d idprobationary teacher is a bad idea.

T=1 T=2 T=3 T=4

A l dd d 0 080 0 075 0 068 0 061

Wait to dismiss until year:

Average value added 0.080 0.075 0.068 0.061

% dismissed 81% 75% 71% 68%

Allowing a 2nd or 3rd year to evaluate a b ti t h i d idprobationary teacher is a good idea.

T=1 T=2 T=3 T=4

A l dd d 0 080 0 095 0 099 0 101

Dismissal at any time through year:

Average value added 0.080 0.095 0.099 0.101

% dismissed (total) 81% 83% 84% 84%% dismissed (by year)% dismissed (by year)

In year 1 81% 67% 67% 67%In year 2 0% 16% 8% 8%In year 3 0% 0% 9% 4%In year 3 0% 0% 9% 4%In year 4 0% 0% 0% 5%

Obtaining more reliable information on teacher f i l bl littl ff t di i lperformance is valuable, little effect on dismissal

T=1T=2T=3T>3 86 T 1

.6.7

.8se

d.12

.14

.16

ded

3.4

.5or

tion

dism

iss

06.0

8.1

age

valu

e ad

d

.1.2

.P

ropo

.02

.04

.0A

vera

00

0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1Reliability of annual performance measure

Average value added Proportion dismissed

Obtaining more reliable information at time of hire i l bl d d di i l tis even more valuable, and reduces dismissal rate.

8025

608

ed2.2

ed

40tio

n di

smis

se

5.2

e va

lue

adde

20Pr

opor

t

.15

Ave

rag

0

.1

0 .2 .4 .6 .8 1Reliability of pre hire performance signalReliability of pre-hire performance signal

Average value added Proportion dismissed

SummarySummary

Potential gain is largePotential gain is largeCould raise average annual achievement gains by ≈0.08Similar magnitude to STAR class-size experiment and to recent

lt f h t h l l tt iresults from charter school lotteries

Gains could be doubled if had more reliable performance d i l d if b d hi himeasure, and tripled if observed this pre-hire

Select only the most effective teachers, and do it quicklyy q yBut may be practical reasons limiting success of this strategy

Focused on screening, but other uses may yield large gainsFocused on screening, but other uses may yield large gains