Implementing Merit Pay, Overcoming the Obstacles: Lessons from the Field Gary W. Ritter Endowed Chair in Education Policy Brent Riffel Nate Jensen Office

Implementing Merit Pay, Overcoming the Implementing Merit Pay, Overcoming the Obstacles: Lessons from the FieldObstacles: Lessons from the Field

Gary W. RitterEndowed Chair in Education Policy

Brent RiffelNate Jensen

Office for Education PolicyUniversity of Arkansas

Presented to:Oklahoma State Legislature

Oklahoma City, OKAugust 21, 2007

Implementing Merit Pay, Overcoming the Obstacles Slide 2

Presentation Overview

1. Why Modify Salary Structure2. Policy Alternatives3. Merit Pay Literature4. Evaluation of ACPP Merit Pay Program

in Little Rock5. Evaluation of Gilchrist County, Florida

Performance Pay Program6. Designing a Performance Pay Program7. Ten Pitfalls and How to Avoid Them


Policymakers Striving to Increase Student Performance

In an effort to increase student performance, where might policymakers look?

The research is clear and consistent in acknowledging the important role of teachers.

However, the research is not clear or consistent in identifying strategies for recruiting and retaining effective teachers.

Teacher salaries may be an appropriate place to exert policy influence.


Teacher Salaries: What’s the Status Quo?

Current Single Salary System• Based on tenure and degree• Lock-step

Arguments for single system• Fair• Simple

Arguments against single system

• Does not address teacher shortages – either by geographic area or subject area

• Counter-productive reward structure – good teachers encouraged to:

• Leave field (better salary)

• Transfer schools (better environment)

• Move to Administration (only real promotion)


Policy Strategy?

Policymakers may want to adjust the salary system to change the incentive structure.

To do that, where might policymakers look for change?


Entry Level Teacher Pay … Competitive

$25

$30

$0

$20

$40

$60

$80

Year 1 Year 5 Year 10

An

nu

al S

alar

y ($

000s

)

New Teacher

New

Business Graduate


Rewards for Teaching Excellence Decline Over Time

$75

$25

$40

$35

$40

$30

$0

$20

$40

$60

$80


An

nu

al S

ala

ry (

$000s)

Talented B-Grad Talented Teacher

Teacher


Rewards for Effective Teachers?

$40

$30$35

$0

$10

$20

$30

$40

$50

$60

$70

$80


An

nu

al S

ala

ry (

$000s)

Effective Teacher


Rewards for Effectiveness?

$35

$40

$30

$0

$10

$20

$30

$40

$50

$60

$70

$80


An

nu

al S

ala

ry (

$000s)

Effective Teacher Other Teacher


What Should School Leaders Consider?

Teachers affect student performance, however…

Status Quo does not necessarily encourage innovation or adopting more effective techniques.

Therefore, policymakers need a system to recruit and retain talented teachers and to reward high quality instruction.

What alternatives do we have?


Three Policy Alternatives to Recruit, Retain, and Reward Effective Teachers

• “Lump Sums”• Does not change incentive structure

• Legislature employed this alternative

• Differential Pay• Hard-to-staff schools

• Specific subjects

• Legislature employed this alternative

• Merit Pay • Teacher characteristics (e.g. National Board, Prof. Dev.)

• Teacher behavior

• Student performance


Merit Pay Debate

How might merit pay affect teacher performance?

Two types of potential effects:• Composition

• Motivation

Supporters believe performance pay leads to:• More innovation

• Increased work ethic

• Salary satisfaction

Opponents believe performance pay leads to:• Counter-productive competition

• Degraded work environment

• Decreased focus on high-performing students

What does the evidence say?


Merit Pay Background Performance pay is not a novel idea:

Such programs were quite common in the early twentieth century (Murnane and Cohen, 1986)

Present in some form in about 12% of public school districts in the early 1990’s (Ballou, 2001).

Some U.S. studies have found that programs providing bonuses to entire schools, rather than changing the pay of individual teachers, have a positive impact on student test scores (Clotfelter and Ladd, 1996; Ladd, 1999).


Merit Pay: Recent Studies Keys and Dee (2005):

• Evaluated a performance pay program in Tennessee.

• Students of teachers participating in the performance pay program made exceptional gains in math and reading.

• Limitation of the study was that the teachers were allowed to choose if they wanted to participate.

Figlio and Kenny (2006):• Surveyed schools that participated in the National Educational

Longitudinal Survey (NELS).

• They then supplemented the NELS dataset with information on whether schools compensated teachers for their performance.

• Found that test scores were higher in schools that individually rewarded teachers for their classroom performance.


Merit Pay Review

Podgursky and Springer (2006):• Conducted a comprehensive review of merit pay systems.

• Examined merit pay programs in Denver, Texas, Florida, Minnesota, and the Milken Family Foundation’s Teacher Advancement Program (TAP).

• Five of the seven existing studies had positive results.

In summary, merit pay programs have been shown to be effective ways of improving student test scores.


Merit Pay: What do we know?

Very few rigorous evaluations Many programs are short-lived Until recently, data limitations Existing research indicates:

• Teachers often displeased

• According to recent lit. review, student performance generally improves or stays the same

Let’s look at the evaluations of two programs …


Little Rock School District’s Achievement Challenge Pilot Project (ACPP)

Program Goals: • Increase student performance

• Reward effective teachers

• Improve school culture

5 elementary schools Financial rewards to teachers based on

annual gains in student performance


ACPP: Understandable, Non-Competitive for Teachers, Significant $, and Focus on Growth of Students

Table 1: Payouts for Wakefield for 2006-07

Employee Type / Position0-4%

Growth5-9%

Growth10-14% Growth

15%+ Growth

Maximum Payout

Principal $2,500 $5,000 $7,500 $10,000 $10,000

Teacher (Grades 4 -5) $50 $100 $200 $400 $11,200

Teacher (Grades 1-3) $50 $100 $200 $400 $10,000

Teacher (Kindergarten) $50 $100 $200 $400 $8,000

Coach $1,250 $2,500 $3,750 $5,000 $5,000

Specialist; Spec. Ed. $1,000 $2,000 $3,000 $4,000 $4,000

Music Teacher $1,000 $2,000 $3,000 $4,000 $4,000

Physical Examiner $500 $1,000 $1,500 $2,000 $2,000

Aide $250 $500 $750 $1,000 $1,000

Secretary & Custodian $125 $250 $375 $500 $500


Student Effects Question:

• What is the impact of the ACPP on the math performance of students?

Method:• Compare the difference in test scores for ACPP students to

the difference in test scores for comparison students

• Comparison based on math scores of 4th and 5th grade students due to data availability

• Tests used:

• Stanford Achievement Test-9

• Iowa Test of Basic Skills

• Reduces “gaming effect”


46.54

44.16

42.15

43.48

39

40

41

42

43

44

45

46

47

2004-05 Mean Score 2005-06 Mean Score

Avera

ge N

CE

Po

ints

, M

ath

Sco

res

ACPP Comparison

Student Effect Results


Teacher Survey Hypotheses TestedSupporters believe merit pay leads to…

1) More innovation2) Increased work ethic3) Salary satisfaction

Opponents believe merit pay leads to…4) Counter-productive competition5) Degraded work environment6) Decreased focus on high-performing

students

7) If merit pay is beneficial, then student achievement should improve. If merit pay is detrimental, then student performance should decrease.

58 ACPP teachers

74 comparison teachers


Teacher Effect Results

Constructs ACPP (% Agree)

Comparison (% Agree)

Result

1) Innovative 88% 90% Neutral

2) Work Harder 86% 99% + Comp

3) Satisfaction with Compensation

53% 35% + ACPP

4) Collaboration 95% 99% Neutral

5) Positive School Environment

86% 56% + ACPP

6) Openness to Challenges 86% 66% + ACPP

7) Effective Teacher 90% 78% + ACPP


Policy Implications & Conclusions ACPP improves student performance

• Student performance in math increased 3.5 NCE points (roughly 6 to 7 percentile points)

Teachers support the ACPP• Significantly more satisfied with ACPP than single

salary system

• The program did not lead to counterproductive competition

• The school environment is more positive with ACPP

• ACPP has positive impacts for students


Gilchrist County, Florida Performance Pay Plan

Enacted in the 1998-1999 school year by the Gilchrist County School Board; revised in 2001-2002.

Program Goals:• Reward & retain high performing teachers• Increase learning in the classroom

Teachers given monetary bonuses of 5% of their salary if they receive a performance rating of “Outstanding” • Improved student test scores• Principal evaluation

Limitation: Teachers compete in a “zero-sum” game• Only the top 25% of teachers can receive a bonus


Gilchrist County, Florida Performance Pay Plan

Question: Does performance pay increase student test scores?

Method:• Gilchrist County compared to a control group

comprised of demographically similar school districts

• FCAT test scores for both groups were used as the comparison tool


Gilchrist County, Florida Results

•In 1998, students in Gilchrist County scored an average of 11.3 points lower on standardized tests than the comparison group

• After 2001, students in Gilchrist County surpassed the control group.

• Since 2001, Gilchrist County students have increased their average FCAT test score by 22.2 points, compared to the 13.9 point gain made by students in the control group

Grades 3-10 FCAT Scores in Math and Reading

290.8

300.3

322.5

302.1

316.5

302.6

275

280

285

290

295

300

305

310

315

320

325

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007

Mea

n S

cale

Sco

re

Gilchrist

Control Group


Gilchrist County, Florida Limitations and Conclusions

Limitations of this evaluation:

• Imperfect control group • Gilchrist County has a significantly lower % of minority students than the

comparison group.

• Causality• Difficult to attribute increase in test scores directly to the performance pay

program.• However, the program clearly did not have a negative effect on test scores.

• The evaluation was much less comprehensive than the Little Rock ACPP.

However, our evaluation showed significant gains in student test scores since the implementation of the Performance Pay Program.


Implications for Program Development

As policymakers consider ways to recruit, retain, and reward effective public school teachers, they would do well to consider the results of this performance pay plan evaluation.

Why Have Programs “Failed” in the Past?

Characteristics of Merit Pay Programs with a Chance of Succeeding


Performance Pay: Possible Design

• Teachers rewarded for:• Classroom gains in student performance as measured by

state CRT and national NRT (30%)

• School-wide gains in student performance as measured by state CRT and national NRT(60%)

• Performance evaluation conducted by principal (10%)

• Ultimately, Performance Pay should increase student performance by:• Rewarding effective teachers

• Exerting a positive influence on school culture

• Recruiting, rewarding, and retaining effective teachers


Top Ten Obstacles to Merit Pay Programs

10. Only benefits teachers of top students

9. Based on a secret formula

8. Teaching to the test

7. Extra testing

6. All future raises will be based on test scores

5. What about art, PE, and music teachers?

4. Discourages collegiality

3. Test-taking automatons vs. fully-functional human beings

2. Teachers do not teach for money

1. Rewards aren’t worth extra work


Challenge Number 10

Complaint Only benefits teachers of

top students

Teachers with the easiest students will have an unfair advantage in a scheme based on test scores.

Performance Pay Teacher rewards should be

based on test score improvement of all students in a classroom for an entire year.

It may be easier for students at the low end to experience improvements!

Can produce the right incentives for school leaders to get great teachers to choose to work with needy kids.


Challenge Number 9

Complaint Based on a secret

formula

It is not at all clear to the teachers how the rewards will be decided and whether they are fair. If we don’t understand what will be rewarded, this cannot possibly work.

Performance Pay Teacher rewards should

be based on a simple calculation of test score improvement.

The gain measure is then straightforward: post-score minus the pre-score.


Challenge Number 8

Complaint Teaching to the test

A monetary bonus program will force teachers to teach only items on the test and ignore everything else.

Performance Pay The benchmark tests are

representative of curricular frameworks developed by teachers and leaders. Teachers will be rewarded by teaching skills that educators have decided are important.

Testing is not the only component of Performance Pay


Challenge Number 7

Complaint Extra Testing

Students will now have to spend more time testing and even less time in the classroom being instructed by effective teachers.

Performance Pay Teacher rewards

can be based on student scores from tests that are already administered.

No new testing would be required.


Challenge Number 6

Complaint All future raises will be

based on test scores

District leaders will now shift all new funds for salary increases into the bonus program and eliminate other raises.

Performance Pay A successful program

might focus only on bonuses (additions to standard salary).

Performance Pay would likely have no impact on future teacher salary increases.


Challenge Number 5

Complaint What about art, PE, and

music teachers? Bonus programs only

reward teachers of core subjects and ignore all other important subjects.

Other subjects will then be marginalized in the schools.

Performance Pay Teacher rewards based

on school wide student growth and principal evaluations.

Other subjects contribute to school-wide growth.

The job of the principal (whose rating is incorporated) is to ensure that teachers of all subjects be rewarded.


Challenge Number 4

Complaint Discourages collegiality

Teachers will no longer want to work together because they will begin to view their colleagues as their competitors in a race for bonus money.

Performance Pay Teacher rewards will not

be treated as a “zero-sum” game.

Teacher rewards are not decreased by rewards given to peers. Rewards are based on student improvement and all teachers may be rewarded.

Survey data reveals additional collaboration in schools with such programs.


Challenge Number 3

Complaint Test-taking automatons

vs. fully functional human beings

Teachers will be rewarded for producing students who can successfully fill in bubble sheets instead of producing “fully functioning” human beings.

Performance Pay Teacher rewards

based on exams that educators believe are important indicators of readiness for self-sufficiency.

The ability to be fully functioning is related to learning key skills that the tests measure.


Challenge Number 2

Complaint Teachers do not teach for

money.

Monetary rewards are not important to teachers. Teachers do not enter this profession to become rich.

Performance Pay Teachers are likely

similar to most people: they have many preferences, they prefer more money to less, and they prefer to be rewarded for good work.

Teachers, like other workers, do seek salary increases.


Challenge Number 1

Complaint Rewards aren’t worth it. A reward of one or two

thousand dollars cannot motivate great changes on a daily basis.

Many past similar teacher award programs have been viewed as unsuccessful, partially because of low levels of rewards.

Performance Pay Depending on

the model, effective teachers can earn a maximum reward of …

$10,000


Conclusions

Theoretically, there are reasonable arguments for and against performance pay

• Current system does have poor incentives that could be improved by performance pay

• There are also characteristics of teaching profession that complicate implementation of teacher incentive pay

Research evidence is not robust enough to point in either direction

We can conclude that it is worth trying this strategy as a policy alternative and then testing it rigorously


Contact Information:Gary Ritter, Associate Professor

Office for Education PolicyUniversity of Arkansas

http://www.uark.edu/ua/oepEmail: [email protected]

Phone: (479) 575-3773

Documents

Implementing Merit Pay, Overcoming the Obstacles: Lessons from the Field Gary W. Ritter Endowed Chair in Education Policy Brent Riffel Nate Jensen Office