Quick Studies? An Investigation Into “Learning” Based on the

NFL Draft

By Christopher Ferraro

and Jacob Leibenluft

Introduction

• A recent analysis of NFL draft behavior (Massey and Thaler, 2005) finds that teams tend to exhibit “overconfidence” in their ability to assess unproven talent

• In particular, what Massey and Thaler find is that the NFL market tends to “overvalue” high draft picks relative to lower picks or players with known values

Teams value early picks far more than later picks…

…but some later picks have a greater surplus value

Does Learning Happen?

• We would expect teams to learn over time and exploit market inefficiencies

• Some have: the example of the New England Patriots– Only two top 10 picks in the last 10 drafts– Won three of the past four Super Bowls

• Loser’s curse: Bad teams get higher picks, but those picks do not produce as much surplus as lower choices

Experimental Setup

• List of players with estimated “PowerPoints”

• An “opt-out” value that declines with each successive pick

• Group 1: No information given after each round

• Group 2: Draft pick’s actual value, plus the names of the top 2 players revealed to each participant

Assumptions

• For participants 1-6, the optimal strategy will usually be to pick the player with the highest PowerPoints

• For participants 7-9, the optimal strategy will usually be to opt out

• Our hypothesis: Over time, participants will recognize their optimal strategy

Assigned Abilities

• Players 1-3: PowerPoints distributed around actual value, small spread (sd=5)

• Players 4-6: PowerPoints distributed around actual value, large spread (sd=20)

• Players 7-9: PowerPoints randomly assigned

0

10

20

30

40

50

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year10

Surplus Value – Normal Evaluation

• We looked at the surplus from each pick — what a participant earned over the comparable opt-out value.

• For subjects with PowerPoints distributed around the actual value, we expected to see steady oscillation above zero, meaning these subjects were generally making the correct picks.

* Data from Group 1, subject ID 6

Expected Results by Player

• For a subject with a random distribution (IDs 6-9), we expected to see some early oscillation around zero and then a convergence to zero as they choose to opt out more and more.

* Data from Group 2, subject ID 7

Surplus Value – Random Valuation

-25

-20-15

-10-5

0

510

15

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year10

ID 1

-10

0

10

20

30

40

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 2

-10

0

10

20

30

40

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 3

0

5

10

15

20

25

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 4

0

20

40

60

80

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 5

010203040506070

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 6

0

10

20

30

40

50

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

Normal Subjects

ID 7

-30

-20

-10

0

10

20

30

40

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 8

-40

-30

-20

-10

0

10

20

30

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 9

-20

-10

0

10

20

30

40

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

Random Subjects

Overall Results, Group 1

Normal Subjects

ID 1

-10

0

10

20

30

40

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 2

-10

0

10

20

30

40

50

60

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 3

-20

-10

0

10

20

30

40

1 2 3 4 5 6 7 8 9 10

ID 4

0

10

20

30

40

50

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 5

05

10152025

3035

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 6

0

10

20

30

40

50

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

Random Subjects

ID 7

-25-20-15-10-5

05

1015

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 8

-80

-60

-40

-20

0

20

40

60

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

ID 9

-80

-60

-40

-20

0

20

40

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

Overall Results, Group 2

• In general, Group 1 tended to opt out more frequently.

• With respect to the top 5 picks, 57 percent of our random subjects in Group 1 chose to opt out relative to 14 percent in Group 2

Summary of Opt-Out Strategies

Top 5, R. 6-10ID Number 1-6 7-9 1-6 7-9 1-6 7-9Total Opt-Outs 6 11 0 8 0 5Total Opt-Out Opportunities 60 30 36 14 19 6

Top 5, R. 6-10ID Number 1-6 7-9 1-6 7-9 1-6 7-9Total Opt-Outs 1 7 0 2 0 2Total Opt-Out Opportunities 60 30 36 14 19 6

Group 1Total Top 5 Picks

Total Top 5 PicksGroup 2

Opt-Out Analysis

-50

0

50

100

150

200

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

Group1 Group2

Gain for Top 5 Picks

Surplus from Top Five Picks

Note: In this graph, surplus for participants who opted out is calculated as the opt-out value minus what the participant would have earned if he had selected the highest rated player on his sheet

Regression Analysis

Group 1 Group 2 Group 1, after Round 4

Group 2, after Round 4

N = 90 N = 90 N = 54 N = 54 Predicted difference -0.0057311

(t = -2.45) -0.0027269 (t = -1.39)

-0.0078512 (t = -2.45)

-0.0040255 (t = -1.65)

Predicted difference * talent dummy

-0.006235 (t = -1.73)

0.0039241 (t = 1.70)

-0.0064276 (t = -1.29)

0.0086233 (t = 2.99)

Constant 0.448795 0.1434108 0.5169846 0.1800374 F-test (predictdiff + interact = 0)

19.28 ( p > F = 0.000)

0.95 (p > F = 0.3339)

14.17 (p >F = 0.0004)

8.66 ( p >F = 0.0044)

Learning appears to happen around Round 4 in the second group, but not in the first group - offering weak support for our hypothesis!

Analysis

• Why should we be cautious in interpreting these results?– Sample size (number of rounds and number

of players)– Some players chose alternate strategies we

didn’t expect (Group 2, Player 9)– Questionable design in some cases — three

groups instead of two

Applications

• The job market– Does a boss rely on his/her own assessment or on proven

performance? – How do different employers react to the behavior of other

employers?

• Auctions for antiques/art– Do buyers learn to follow their own assessments or the “herd”?– “Bargain hunting” vs. masterpieces