Incentives and Program Evaluation – How to Get What You Want and Measure It Properly Jodi N. Beggs Economists Do It With Models Presented to Carlson Marketing

Incentives and Program Evaluation – How to Get What You Want and

Measure It ProperlyJodi N. Beggs

Economists Do It With Models

Presented to Carlson MarketingJuly 23, 2010

www.economistsdoitwithmodels.com

Two parts:

1.Consumer behavior and incentives

2.Data analysis and program evaluation

Part 1: Incentives (aka “Incentives Work…Except When They Don’t”)

• When do they work?• Routine effort, direct cause and effect

•More noise -> weaker incentives

• Incentives affect both effort and worker selection• Safelite Auto Glass:• Piece-rate compensation -> Productivity

increases by 44%• Half of increase due to increased

productivity• Other half due to employee self-selection

Source: Edward Lazear and Paul Oyer, “Personnel Economics,” Handbook of Organizational Economics, forthcoming.



• But, you get exactly what you pay for…• Non-linear incentives and timing

• Nonlinear incentives (eg. quotas) give workers an incentive to play a timing game• If a worker has made her quota, there is an incentive to push sales to the

future• Alternatively, there is an incentive to just not try for the rest of the

year• If a worker has no hope of making her quota, there is an incentive to push

sales future• Shirking is also a concern here

• If a worker is on the cusp of making the quota, there is an incentive to hurry sales• This often corresponds with discounts that may not be in the firm’s

best interest

• This behavior can even happen with linear incentives if bonuses are paid yearly, quarterly, etc.

Source: Paul Oyer, “Fiscal Year Ends and Nonlinear Incentive Contracts: The Effect on Business Seasonality,” The Quarterly Journal of Economics, February 1998.



• But, you get exactly what you pay for…• Non-linear incentives and timing• Distortion of multi-dimensional tasks• “Goldbricking” and quota restriction

• A sociologist’s summer research on a factory floor:• Compensation: greater of 85 cents per day or

piece rate for work turned in• “Gravy” jobs and “stinker” jobs• Piece-rate payout of 9 cents to $1.66 per day• Bimodal distribution of piece-rate earnings• One group at 45 to 54 cents (and getting the

day rate of 85 cents)• One group at $1.25 to $1.34•What is going on here?

Source: Donald Roy, “Quota Restriction and Goldbricking in a Machine Shop,” The American Journal of Sociology, March 1952.



• But, you get exactly what you pay for…• Non-linear incentives and timing• Distortion of multi-dimensional tasks• “Goldbricking” and quota restriction• Gaming the system in general

Source: http://www.filleritem.com

I Can’t Imagine This Is Profitable For Amazon…

Part 1: Incentives, continued

• Incentive strategy and framing also matters• Framing

“To increase transaction compliance, marketers sometimes temporally reframe the cost of a product from an aggregate one-time expense to a series of small ongoing expenses, often in spite of the fact that the physical payments remain aggregated.”

• “Pennies-a-Day” strategy• How does this affect incentives?

Source: John T. Gourville, “Pennies-a-Day: The Effect of Temporal Reframing on Transaction Evaluation,” The Journal of Consumer Research, March 1998.

• Observation: People dislike losses more than they like equivalent gains• Loss aversion and reference-dependent utility

• How can managers and markets use this observation?• Long-term effects


• Incentive strategy and framing also matters• Framing• Guaranteed versus probabilistic rewards

• In general, people are risk averse…but consider the following choices:

• $1 or a 1/2 chance of $2• $1 or a 1/3 chance of $3• $1 or a 1/5 chance of $5• $1 or a 1/10 chance of $10• $1 or a 1/100 chance of $100• $1 or a 1/1000 chance of $1000• $1 or a 1/1,000,000 chance of $1,000,000

• Not so clear, is it? • This can sometimes be leveraged when designing incentives• The Publisher’s Clearing House seems to have figured this out


• Incentive strategy and framing also matters• Framing• Guaranteed versus probabilistic rewards• Tournament incentives

• Tournament incentives are payouts based on rank order (relative performance)• Incentives get stronger as spread between

payouts for adjacent places increases• However, can have “underwater option” problem

for low and average performers


• Incentive strategy and framing also matters• Framing• Guaranteed versus probabilistic rewards• Tournament incentives

Follow up: In-kind incentives can sometimes be powerful, contrary to what traditional economic theory would predict. This is likely because people tend to artificially restrict their consumption as a means of self-control, so a gift that the person likes but wouldn’t buy herself can be more satisfying than the equivalent amount of cash. For more, see “Mental Accounting Matters” by Richard Thaler.


• When don’t incentives work?• Work involving cognition and/or creativity• Tunnel vision and suppression of intrinsic motivation

• Output-based incentives can inhibit “out-of-the-box” thinking• Eg. evidence shows that commissioned art is viewed

as less creative• Output-based incentives can also inhibit intrinsic or moral

incentives and have perverse effects• Day care center – fine for late parents actually

increased tardiness• NIS 10 per child for delays over 10 minutes• Fine added to monthly bill• Increase didn’t go away once fine was abolished

• “Pay enough or don’t pay at all”

Source: Daniel Pink, Drive; Uri Gneezy and Aldo Rustichini, “A Fine Is a Price,” The Journal of Legal Studies, January 2000; Uri Gneezy and Aldo Rustichini, “Pay Enough or Don’t Pay at All,” The Quarterly Journal of Economics, August 2000.


• When don’t incentives work?• Work involving cognition and/or creativity• Tunnel vision and suppression of intrinsic motivation

• Non-actionable output-based incentives

Roland Fryer: “In stark contrast to simple economic models, our results suggest that student incentives increase achievement when the rewards are given for inputs to the educational production function, but incentives tied to output are not as effective. Relative to achievement-increasing education reforms of the past few decades, student incentives based on inputs produce similar gains in achievement at lower costs. Qualitative data suggest that incentives for inputs may be more effective because students do not know the educational production function, and thus have little clue how to turn their excitement about rewards into achievement.

• Angrist and Lavy find that paying for passing has a greater impact on girls’ performance than on boys’ performance

Source: Roland Fryer, Jr., “Financial Incentives and Student Achievement: Evidence from Randomized Trials,” working paper, 2010; Joshua Angrist and Victor Lavy, “The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial,” The American Economic Review, 2009.


• What can a manager or marketer do instead?• Autonomy, mastery, purpose

Huffington Post: Foursquare:

Showing Mastery Via Badges:

Badges can be powerful in a community setting when the badges are public, noticeable and respected. It also doesn’t hurt if the badges are difficult (but not too difficult) to obtain.

• “Endowed progress effect”• People provided with artificial advancement toward a goal

exhibit greater persistence toward reaching the goal• Endow people with 2 completed steps out of 10 rather than 0

completed out of 8 and they are more likely to complete all steps• Reframing from “not undertaken” to “undertaken but

incomplete”• Also decreases completion time• Effect depends on perceptions of task completion rather

than misunderstanding of sunk costs• Attainability also matters

Source: Joseph C. Nunes and Xavier Dreze, “The Endowed Progress Effect: How Artificial Advancement Increases Effort,” The Journal of Consumer Research, March 2006.


• What can a manager or marketer do instead?• Autonomy, mastery, purpose• Status

• Status in loyalty programs:• Increasing the number of elites in the top tier dilutes

perceptions of status• Adding a subordinate tier enhances status• Tiers below the second tier do not affect those at the

top but can make those in the tier immediately above feel more elite• Those who do not qualify for status prefer hierarchies

with multiple tiers• Status-laden labels (eg. gold and silver) on their own

signal an increasingly selective hierarchy

Source: Xavier Dreze and Joseph C. Nunes , “Feeling Superior: The Impact of Loyalty Program Structure on Consumers’ Perceptions of Status,” The Journal of Consumer Research, April 2009.

Part 2: Program Evaluation (aka “Showing Causality Is Harder Thanit Looks”)

• The fundamental program evaluation problem

Source: http://xkcd.com/552/

The “Post Hoc, Ergo Propter Hoc” Fallacy in a Nutshell:

Source: http://gregmankiw.blogspot.com/2009/08/least-surprising-correlation-of-all.html

“Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids' SAT scores.”-- N. Gregory Mankiw

“The Least Surprising Correlation of All Time”

Part 2: Program Evaluation (aka “Showing Causality Is Harder Thanit Looks”)

• The fundamental program evaluation problem• Why life should be like a middle school science project or a

clinical trial• How do researchers get around this problem? Natural

experiments!

• The effect of babies on lifetime earnings• Why not just compare earnings of those who had kids at 24 to

those who had kids at 25?• Two more valid comparisons:• Women who had children at 24 to women who miscarried

at 24 and had children at 25• Women who had children at 24 by accident to women

who had children at 25

• Findings:• 1 year delayed childbirth for women in 20’s = 10% higher

lifetime earnings• Result goes away once women reach their 30’s

Source: Amalia R. Miller, “The Effects of Motherhood Timing on Career Path,” working paper, 2005.

• What is the effect of veteran status on earnings?• Researchers can control for what is observable- age, education,

gender, race, etc.• But, can’t just compare “identical” military and non-military people

• There is some difference that made some of them choose the military and others not, and this difference could be correlated with earning potential

• Vietnam draft lottery allowed researchers to circumvent this selection bias

• Findings:• Based on data from Vietnam War veterans, veterans earned 15% less

than equivalent non-veterans• Finding limited to white males and Vietnam War, so may not be

generalizable

Source: Joshua D. Angrist, “Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records,” The American Economic Review, June 1990.

Part 2: Program Evaluation, continued

• Other things to consider in program evaluation• Hawthorne Effect and Mere-Measurement Effect• Impact of feedback on subsequent behavior and loyalty• Salience and attention (pilot program effect)

• Participant selection• The Counterfactual

For more information, go tohttp://www.economistsdoitwithmodels.com/Carlson

Documents

Incentives and Program Evaluation – How to Get What You Want and Measure It Properly Jodi N. Beggs Economists Do It With Models Presented to Carlson Marketing