Basics of Perf Based Comp

8/14/2019 Basics of Perf Based Comp

1/18

1Copyright American Federation of Teachers, AFL-CIO | July 09

THE BASICS OFDIFFERENTIATEDCOMPENSATION

QuEST CONFERENCEWASHINGTON, D.C.

JULY 2009

Rob Weil

Educational Issues

Jewell Gould

Research and Information Services

Matthew Di Carlo

Research and Information Services

2

One of the

biggest hurdles indeveloping a newcompensationsystem is theamount of changerequired toproperly supportto new program.As much changeis required of theschool district asthe teachers, ifnot more.


2/18


3

OBJECTIVES Describe differentiated pay in general and

how it should be approached Explain value-added methodology, how it

works, and how it should be used

Discuss the role of base salary in systems,and vice-versa

Summarize the existing research on howdifferentiated pay influences productivity

Provide some examples of differentiatedpay systems around the U.S.

List some of the funding sources available

Conclude with guidance as to how plansshould be designed and maintained

4

AN IMPORTANTPERSPECTIVE

Although Differentiated Compensation is apromising part of improving teacherquality, it is only one piece of the

puzzle. Differentiated Compensationsystems done in isolation have little

chance of success. To improve teachingquality, other aspects of teaching and its

development must be addressed.


3/18


5

RigorousPreparation with

Clear andEnforcedLicensureStandards

Peer Assistanceand Review,Induction &Mentoring

Evaluation Basedon P rofessional

Standards

Competitive

andProfessional

Compensation

Ongoing, Job-embedded

ProfessionalDevelopment

The Quality Teaching Puzzle

6

A WORD OF CAUTIONHow you communicate, bothinternally and externally, will play a

large role in the success of yourprogram. Take time to carefullyframe your message by using

appropriate language, controlling theflow of information to avoid

misunderstandings, and using factsand sound research.

By far, this is the number onemistake in the development of

differentiated pay plans


4/18


7

THE EASY PART

7

Pay for Skills and Knowledge (Input)Pay for Responsibility (Extra work)Pay for Student Learning* (Output)

*Pay for student learning will require alarge restructuring of most school

districts instructional support systems.

8

STRATEGIC & SYSTEMATIC

8

A B C

Where youare today

Where youwant to be

What youlearnhere

willimprove your

outcomes

Deve lop in g a St ra t eg ic Pro cess

Creates t he Bas is for a Systemat i c Approach

Differentiated pay is a process, not an event.Differentiated pay is a process, not an event.


5/18


9

THE IMPORTANCE OFFACE VALIDITY

Any plan must make certain that teachers influence themetrics in an understandable and observable way.

Teachers in variable pay plans must see how theirperformance directly effects the outcomes.

X Y

What I did This is the result

The importance of this validity goesbeyond individual need; it also is the key to

organizational growth.Value-added exemplifies this issue.

The importance of this validity goesbeyond individual need; it also is the key to

organizational growth.Value-added exemplifies this issue.

10

VALUE-ADDED METHODOLOGY

CURRENT TRENDS Widespread support for using growth

models to measure educational inputs Increased federal funding for data-driven

reform Using VAM as a measure of effectiveness The reckless use of concepts like value-

added or gain scores Combining VAM estimates with other

measures of performance to evaluate schoolsand teachers

Although its not reliable or valid, its betterthan the current system Devaluing of subjects and teachers that are

not tested


6/18


11

WHAT IS VALUE-ADDED?

FEATURES & ASSUMPTIONS

Unlike NCLB, focus is on growth, notachievement levels

Student performance as a function of pastperformance and/or student/schoolcharacteristics

Sometimes acknowledges that student andschool characteristics contribute to studentgrowth, as do non-educational factors (e.g.,poverty)

Requires extensive data collection and complexstatistical capabilities

Can be done at the teacher-, grade, school, or

district-level

12

WHAT IS VAM? (CONTINUED)

CAUSAL ISOLATION At its core, VAM uses complex statistical methods to produce

a quantitative measure of a teachers CAUSAL effect onstudent performance/growth Deals with confounding factors by controlling them in models Teacher effect estimates rely on uncertain assumptions

Most models dont actually produce teacher effects per se,but rather expected student performance All non-teacher effects are purged by comparing students to

others in similar schools with similar characteristics (e.g.,poverty, language), and/or by comparing students currentgrowth with their past growth

This produces an expected score for each student Any difference between that expectation and reality is attributed

to the teacher, and is considered the teacher effect

The two most common types of models are covariateadjustment models (Bob Mendro, Rob Meyer) andmultivariate or layered models (William Sanders, EVAAS)

The choice of models will influence results; both offer strengthsand weaknesses


7/18


8/18


15


STATE OF THE ART II VAM scores often conflict with results of other

types of evaluations This includes principal/peer evaluations, parental surveys This is in no small part due to the inability of test scores

to account for the whole of teaching ability For example, one study found that the adjusted

probability of VAM scores matching principal evaluationsof teachers ability to improve test scores was about 43%for reading and 63% for math*

Teacher effects are also unstable over time This is an under-researched area, but extant studies

clearly indicate that VAM estimates for the same teachervary greatly between years

Two analyses of San Diego and Florida districts foundconsistent results: only 25-33% of the best and worst

teachers remain there between years, while 20-30%move all the way from either top to bottom or bottom totop **

* Jacob, Brian A. and Lars Lefgren. 2005. Principals as Agents: Subjective Performance Measurement inEducation. NBER Working Paper 11463. Washington, D.C.: National B ureau of Economic Research.

** Koedel, Cory and Julian R. Betts. 2007. Re-examining the Role of Teacher Quality in the EducationalProduction Function. Working Paper #2007-03. Nashville, TN: National Center on PerformanceIncentives AND

McCaffrey, Daniel F., Tim R. Sass, and J.R. Lockwood. 2008. The Intertemporal Stability of TeacherEffect Estimates. Unpublished manuscript

16

STABILITY OF M.S. TEACHER EFFECTS

IN HILLSBOROUGH COUNTY

16.9%Move 3-4

23.1%Move 2

36.5%Move 1

23.1%Stable

McCaffrey, Daniel F.; Sass, Tim R. and J.R. Lockwood. 2008 TheIntertemporal Stability of Teacher Effect Estimates.Nashville: National Center for Performance Incentives.

1 2 3 4 5

1 4.2% 5.2% 5.2% 2.3% 2.9%

2 3.3% 4.2% 5.2% 4.9% 2.0%

3 2.3% 3.6% 5.2% 5.9% 3.3%

4 1.3% 2.6% 4.2% 6.5% 4.6%

5 2.3% 2.0% 2.9% 6.9% 6.9%

YEAR TWO QUINTILE

Y

EARO

NE

QUINTILE

Only 23% of teachersremained stablebetween years

40% of teachersmoved at least 2quintiles betweenyears


9/18


17


WHY ALL THE ERROR? SAMPLING ERROR: Small classes small samples huge margins of error

Without sufficient observations, accuracy suffers greatly, we cannotdraw conclusions about a single teacher from a small group of students

SCHOOL- or GRADE-LEVEL alleviates this problem (larger sample)

OMITTED VARIABLE BIAS: there are too manyfactors that influence student performance to isolateteachers roles Unmeasurable factors include: principal support/competence, support

from colleagues, facilities, textbooks, parental involvement, familyproblems, peer effects, curriculum

Unmeasured/unmeasurable factors bias estimates of measured variables

NON-RANDOM ASSIGNMENT: Students andteachers are not usually assigned to classes (or evenschools) randomly Often done for good education reasons, but huge problem for VAM May penalize teachers often assigned to difficult students

Forthcoming paper finds current teacher effects predict past effects * MISSING DATA: especially in large cities, students

move and drop out at high rates Even among stayers, attendance is also an issue Correcting for this may introduce additional bias

* Rothstein, Jesse. Forthcoming. Teacher Quality in Educational Production: Tracking,Decay, and Student Achievement. Quarterly Journal of Economics.

18


GUIDELINES FOR USE EXTREME CAUTION SHOULD BE USED IN CONSIDERINGINDIVIDUAL VAM SCORES AS THE PRIMARY COMPONENTOF A DIFFERENTIATED PAY SYSTEM Error and other problems are prohibitive at the individual-level Measures still have great diagnostic potential

VAM estimates are SENSITIVE TO MODEL SELECTION andDATA AVAILABILITY/ QUALITY Choice of models and variables to include will influence results Tests must be comparable across grades/subjects More years of data decrease error, 3 years helps mitigate sorting

bias*

SCHOOL-LEVEL VAM should be seriously considered as a partof any VAM-based differentiated pay system Severely reduces problems from sample size, non-random

assignment of students to classrooms far more accurate than

individual-level (teacher/classroom-level) estimates Encourages cooperation, not competition, and all can participate

DO NOT MISTAKE MARGIN OF ERROR FOR ACCURACY Scores reports should account for margins of error, but teachers

must understand that those margins are themselves error-prone Error estimates assume that the models are correctly specified For example, unmeasured or unmeasurable factors may be biasing

estimates and error margins* Coedel, Cory and Julian R. Betts. Forthcoming. Does Student Sorting

Invalidate Value-Added Models of Teacher Effectiveness? An ExtendedAnalysis of the Rothstein Critique. Education Finance and Policy.


10/18


19

ABOUT BASE SALARY

Base salary, along with benefits, are themost important parts of any compensation

system--differentiated or otherwise.The base pay structure must continue to

recognize the value of teaching experience.Similarly, pursuing extra education should

be rewarded.*

*Many criticize experience as a factor indetermining a teachers salary and then

express the concern that too manyinexperienced teachers are found in thelowest performing schools.

20

QUID PRO QUOFrom the beginning, it is important

to stress the point:

The District (or State) must notask teachers to look at their

financial interests differently ifthe District (or State) is not

w illing to look at its financialstructure differently.

In most cases, this will need to bestressed numerous times.

In most cases, this will need to bestressed numerous times.


11/18


21

lq~

a=`

A DEAL BREAKER

q~l

This type of system lowers ateachers standard of living.

This type of system lowers ateachers standard of living.

22

A DEAL MAKER

q~lTeachers

This type of system does notlower a teachers standard of

living.

This type of system does notlower a teachers standard of

living.

a=`

q~l


12/18


23

ABOUT BASE SALARY (CONTINUED)

STRUCTURAL EFFECTS All plans that provide flat-dollar performance

bonuses that not everyone receives carry concreteimplications for salary structures - they willfundamentally alter how and how much teachers arepaid

Differentiated pay partially severs the long-standingrelationship between degree/experience and earnings In many cases, BA teachers who receive bonuses will earn

more than non-recipient MA teachers at the same step Similarly, experienced teachers who do not receive bonuses

may have lower earnings than less-experienced teacherswho do

These effects are universal, and will only vary by extent

Teachers salaries have been determined by experienceand education for a long time; significant changes to thisrelationship are important

In general, it is better to give smaller bonuses to morepeople than to give larger bonuses to less people

24

DOES THIS STUFF WORK?

RESEARCH ON EFFECTS OFDIFFERENTIATED PAY

The research on the effects of differentiated pay plans forteachers is scarce, and there are no definitive findings . Results are mixed, and all suffer from limitations of interpretation A small group of international studies show generally positive

effects in Mexico, Israel, Kenya and India, but these studies cannoteasily be generalized to the U.S.

The jury is still out on the effects of these programs Several experimental studies in progress NYC, Chicago, and

two in Nashville, TN Definitive results will likely require additional studies and

considerable time

Success depends upon whether differentiated pay systemshelp improve (rather than just measure effects of) teaching

Too many supporters of my party have resisted the ideaof rewarding excellence in teaching w ith extra pay,even though we know it can make a difference in theclassroom. - President Barack Obama


13/18


25

EXAMPLES OF PROGRAMS

SCHOOLWIDE VAM

NEW YORK CITY Two-year pilot program collaboratively developed byUFT and NYCDOE

Random selection of schools from a non-randomselection of low-performing schools (schools chooseparticipation)

Criteria Performance (test scores 25%) Progress (test scores based on targets 60%) Attendance and Learning Environment (teacher, parent,

student surveys 15%) Bonus available

Schools that meet 100% of their performance target(an expected growth based on a composite of theschool ranking) receive $3000 per UFT member atthe school 75% of the performance target receive $1500

Schools have some discretion in how rewards aredistributed (including to non-instructional staff)

26


HYBRID SCHOOL/ INDIVIDUALHOUSTON ASPIRE

Program comprised of three core strands

STRAND I - VAM Campus-wide Improvement Bonuses to all instructional and non-instructional

staff based on school-level (i.e., campus-level)achievement growth

STRAND II - VAM Core Teacher Performance Pay based on individual-level VAM scores

Inclusiveness: non-core teachers rewarded basedon campus-wide progress

STRAND III Campus Improvement andAchievement Rewards core instructional staff based on how well

the school has improved compared with 40 similarschools in the state


14/18


27


CAREER LADDER SYSTEM

TOLEDO TRACS TRAC A Professional Development Required participation Research- and standards-based

TRAC B School Performance Annual goal determination for each school Rewards to schools teaching teams that meet or exceed

goals

TRAC C Teacher Performance Annual rewards to teachers who demonstrate multi-

criteria excellence, accept difficult-to-fill positions, takeon additional responsibilities, or show high levels ofstudent achievement

TRACS C is composed of three status levels; teachersmove up the ladder by demonstrating excellence

Successful placement on the TRACS program isrewarded with extra salary (as a percent of base)

Participation is voluntary, but participants must undergoregular evaluations to remain in the system

28


COMPREHENSIVEDENVER PRO-COMP

All staff are eligible to participate Base salary of participants varies by success in meeting

goals Program comprised offour parts Professional evaluation

Non-probationary teachers get a bonus every three years for asuccessful evaluation

Probationary teachers get a smaller bonus

Student growth incentives Payments for meeting one or two targets for student growth,

for working in schools receiving exemplary ratings, and forworking in schools classified as distinguished

Base salary increases with each target met

Market incentives Bonuses for moving to hard to staff or hard to serve

assignments/schools

Knowledge and skills Professional development, National Board certification, and

lifetime tuition accounts all provide separate bonuses


15/18


29

FUNDING SOURCES FORDIFFERENTIATED PAY

SYSTEMS TIF (Teacher Incentive Fund) - $487 million

in 2010 (effort to increase to $717 million)

Teacher Incentive Grants Competition

State Fiscal Stabilization Fund (ARRA)

Private foundations (e.g., Gates)

State funding (e.g., Q-Comp)

All external funding has two importantfeatures: it may run out at some point, and

it sometimes comes with hooks

All external funding has two importantfeatures: it may run out at some point, andit sometimes comes with hooks

30

CONCLUSION

GROUP PAYPROS CONS

Widely accepted More valid and reliable

metrics Positive public

relations Academically aligned

Tremendous amount ofwork at front end

Time-consumingassessment andreports

Some teachers notgroup workers

Groups adjustplans

Inter-rater reliability is easier

to manage and control.

Inter-rater reliability is easierto manage and control.


16/18


31

CONCLUSIONINDIVIDUAL PAY

PROS CONS Politically powerful Has high loyalty by

participants Relatively low cost Creates a group of

teacher leaders

Difficult to develop andmeasure

Creates expandingexpectations

Concerns regardingobjectivity

Creates falsepositives and truenegatives

Inter-rater reliability is hardto manage and control.

Inter-rater reliability is hardto manage and control.

32

CONCLUSION

COMBINATION SYSTEMSPROS CONS

Provide the mostflexibility

Allow teachers todevelop their ownprogram

Have high participantsatisfaction

Tremendous amount ofwork to maintain

Different parts of theplan can be incompetition

Time-consumingassessment andreports


17/18


33

SWING TO THE MIDDLE

33

The Teacher Com pensat io n Pendu lu m

Input

Ideal

Output

A Balance

Focusing onStudent Growth

Alone

Focusing onTeacher BehaviorAlone

34

CONCLUSION (CONT.) AFTS BASIC

REQUIREMENTSFOR A SUCCESSFUL PROGRAM

An Adequate Base Salary Sufficient and Stable Funding Credible, Agreed-upon Standards of Practice Support to Improve Professional Practice Labor/Management Collaboration (Trust) Incentives Available to All Teachers Easily Understood Standards for Rewards Necessary Support for Program

Not in any particular orderNot in any particular order


18/18


35

QUESTIONS OR

COMMENTS?ROB WEIL(202) [email protected]

JEWELL GOULD(202) [email protected]

MATT DI CARLO

(202) [email protected]

Documents

Basics of Perf Based Comp