Expectation, Surprise and the Hindsight Bias: The Effect of Information and Outcome Congruency on Retrospective Judgments. Ivan K. Ash, Ph.D. Assistant

Expectation, Surprise and the Hindsight Bias: The Effect of Information and Outcome Congruency on

Retrospective Judgments.

Ivan K. Ash, Ph.D.Assistant Professor

Psychology Department

Old Dominion University

Outline

• Introduction to Retrospective Judgment Making and the Hindsight Bias

• Present the Theories of Retrospective Judgment Making and Hindsight Bias

• Discuss the Role of Expectation and Surprise in Retrospective Judgment Making

• Present Two Studies that Test the the Predictions of These Different Theories

Predictive Judgments

• Secretary of Defense Donald Rumsfeld, 9/19/02 Testimony to the Senate Armed Services Committee regarding Iraq

• “No terrorist state poses a greater or more immediate threat to the security of our people and the stability of the world than the regime of Saddam Hussein in Iraq.”

Retrospective Judgment

• 3/14/04, Face the Nation, CBS News • Asked: “why … did they [Hussein’s

regime] pose an immediate threat to us, to this country?

• Rumsfeld: “Well, you're the--you and a few other critics are the only people I've heard use the phrase ‘immediate threat.’ I didn't. The president didn't. And it's become kind of folklore that that's--that's what's happened.”

Predictive Judgments• Predictive Judgment – assessment of the

likelihood of different possible future outcomes to a situation or event

• Guide choices and decisions we make in life

• Determine what actions we will take to solve our problems

• Help us set goals and make plans• Serve as hypotheses that can be used to test

the validity of our current knowledge or beliefs

Retrospective Judgments• Retrospective Judgments- after-the-fact

assessments of how predictable you would have judged an outcome before it had occurred

• Judging the inevitability of an outcome

• Used as a post-hoc test of the validity of our beliefs and knowledge

• Source for belief revision, behavioral modification, or theory building

Hindsight Bias

• Tendency for people’s retrospective judgments influenced by the actual outcome of a situation or event

• In retrospective judgment making you are supposed to ignore the outcome and attempt to remember was your predictive opinion.

• Hindsight bias is the tendency for people behave as if they “knew it all along”

Hindsight Bias can be Harmful• Carli (1999) – HSB and Victim Degradation

– Gave participants a text describing a couple going out on their first date

– Rate the likelihood that the date ended in the woman getting sexually assaulted by the man

– Rate their approval of the women’s behavior on the date

– Rate their opinion on the how favorable attributes of the women’s character and personality were

• Prediction Group (story ended with woman agreeing to return to man’s apartment)

• Retrospection Group (told that the man assaulting woman)

Carli (1999) resultsLikelihood of Rape

0

2

4

6

8

10

P R

not a

t all

likel

y

extr

emel

y lik

ely

Approval of Behavoir

0

2

4

6

8

10

P R

extr

eme

disa

prov

al

e

xtre

me

apro

val

Ratings of Charater

0

2

4

6

8

10

P R

Unf

avor

able

Fav

orab

le

HSB is a Reliable Effect• Meta-analysis of 122 studies (N = 7359)

(Christensen-Szalanski & Willham, 1991)– They put all the measured hindsight effects on a

comparable scale by Calculating the effect size for each study: Cohen’s d (1988, 1992)

– d = (MR– MP)/SD

– Average d = 0.34, 95% CI > 0• Small effect 0.20

• Medium effect 0.50

• Large effect 0.80

Summary

• People are unable to ignore outcome information when making retrospective judgments

• HSB can be harmful

• HSB is a reliable effect

• Important to explain how people make retrospective judgments and why these processes lead to HSB

Information AvailabilityPredictive Judgments

Prior Knowledge& Beliefs

Prediction (P)

Retrospective Judgments

Prior Knowledge& Beliefs P

EventOut-come

Retro-spection (R)SR

SubjectiveReaction

Information AvailabilityPredictive Judgments:Likelihood of outcome A, given possible outcomes A or B


Pre-outcomeInformation

Support ASupport B

PredictiveJudgment

FavorA or B

P (A) = Support A / (Support A + Support B)

Information AvailabilityRetrospective Judgments: Likelihood of outcome A, given possible outcomes A or B


EventOut-come

R

Support ASupport B

OutcomeInformation

A or B

Pre-outcomeInformation

SubjectiveReaction

SR

Surprising,Ambivalent,

orExpected

PredictiveJudgment

FavorA or B

RetrospectiveJudgment

Theories of Retrospective Judgment and HSB

• Direct Recall Theories• Cognitive Reconstruction Theories

– Anchoring and Adjusting – Re-judging After Integration of Outcome – Estimation Based on Metacognitive Cues – Re-judging After Surprise Cued Sense-

making

Direct Recall Theories

• Predictions:– Those given an outcome may recall the outcome

instead of their predictive judgment will pull the mean judgments toward the “true outcome”

– That those who do not receive outcome information will be more accurate at remembering their predictive judgments


EventOut-come

R

Direct Recall Theories

• This can be tested in a Within Subjects HSB design

• By comparing the number of “hits” or correctly recalled predictive judgments between– Those that receive an outcome (retrospective)– Those that do not receive an outcome (recall)

• “No outcome” recall is not more accurate

• Likely to err in either direction(Dehn & Erdfelder, 1998; Erdefeld & Buchner, 1998; Stahlberg, Eller, Maass, & Frey, 1995; Schwarz and Stahlberg, 2003)

Theories of Retrospective Judgment and HSB

• Direct Recall Theories• Cognitive Reconstruction Theories

– Anchoring and Adjusting – Re-judging After Integration of Outcome – Estimation Based on Metacognitive Cues – Re-judging After Surprise Cued Sense-

making

Cognitive Reconstruction Theories

• At the point of the retrospective judgment people attempt to re-judge the situation in order to reconstruct what they would have thought

• The outcome information in some way biases this re-judgment process

• Theory of Re-judgment process:– How are people reconstructing their predictive

judgments?

– What cues and processes do people use?

– How do these processes lead to Hindsight Bias?

Anchoring and Adjusting Schwarz and Stahlberg (2003)

• People use the given outcome information as metric by which to reproduce their initial judgment

• Begin with the actual outcome and then adjust their estimate away from this value

• Use metacognitive beliefs about their level of knowledge in the content domain to make this adjustment

Anchoring and Adjusting Schwarz and Stahlberg (2003)

• Knowledgeable = small adjustment from the given value

• Not knowledgeable = larger adjustment to compensate

• People are often over optimistic about their expertise

• This leads to the hindsight bias

Anchoring and AdjustingSchwarz and Stahlberg (2003)

Expected Ambivalent Surprising

PredictiveRetrospective

FavorGiven

FavorAlternative

Type of Outcome in the Same Domain

Anchor

Adjustment

Anchoring and AdjustingSchwarz and Stahlberg (2003)



FavorGiven

FavorAlternative


Anchor

Adjustment

HSB Predictions for different types of outcomes

Other

Theory Expected Ambivalent Surprising

Anchor &

Adjust

Little, no, or reverse

Less HSB Most HSB Equal R judg.

• Outcome information is automatically integrated into one’s memory representation

• Information in memory associated with or supportive of this outcome is more available

• Both predictive and retrospective judgments are made based on the current accessibility information in memory that supports an potential outcome (availability heuristic)

Re-judging after outcome integrationFischhoff (1975) , Hawkins& Hastie (1990), Wasserman, Lempert, & Hastie(1991)

Re-judging after outcome integrationFischhoff (1975) , Hawkins& Hastie (1990), Wasserman,

Lempert, & Hastie(1991)



FavorGiven

FavorAlternative



Other


Anchor &

Adjust


Less HSB Most HSB Equal R judg.

Re-judge Integration

HSB HSB HSB

Estimation Based on Metacognitive Cues (Hoch and Loewenstein,1989; Ofir & Mazursky, 1990, 1997)

• People reconstruct there original judgment by using how “surprising” they found the outcome as a cue to their original opinion

• If you were not surprised by an outcome you think, “I would have known that!” – This leads to an overestimate of their predictive accuracy

(HBS)

• If you were surprised by an outcome you think “I would have never known that!” – This leads to either a more accurate (unbiased)

reconstruction or even a shift away from the given outcome

Estimation Based on Metacognitive Cues (Hoch and Loewenstein,1989; Ofir & Mazursky, 1990, 1997)



FavorGiven

FavorAlternative



Other


Anchor &

Adjust


Less HSB Most HSB Equal R jdgmts


HSB HSB HSB

Metacog.

Cue

Most HSB Less HSB No or reverse

- r

(sp, hsb)

Surprise cued “sense-making” Pezzo (2003)

• When a outcome is incongruent with prior expectations (surprising), individuals try to make sense of the outcome– Successful sense-making leads to new opinions and

beliefs about the situation– Or processing the outcome supporting information may

render that information more accessible in memory– A person may give more weight to the information that

supported the surprising outcome or think of new reasons that it occurred

– These new opinions and beliefs bias the reconstruction of one’s predictive judgment

Surprise cued “sense-making” Pezzo (2003)



FavorGiven

FavorAlternative



Other


Anchor &

Adjust




HSB HSB HSB

Metacog.

Cue


- r

(sp, hsb)

Sense-Making

No HSB Less HSB Most HSB + r

(sp, hsb)

Surprise and HSB

• Ofir & Mazursky (1997)

• Gave students a scenario of a man having a surgical procedure that had a 2% mortality rate

• Rate the probability of the man surviving the surgery

• Outcome group was told that man died

• Test for HSB on a surprising outcome

Ofir & Mazursky (1997)

80

82

84

86

88

90

92

94

96

98

100

No outcome Man Died

Pro

babi

lity

of L

ivin

g

Surprise and HSB

• Pezzo (2003)• Gave participant

descriptions of Psych studies with “surprising” outcomes

• “looks are equally important to men and women”

• Between subjects design

"Looks are equally important to men and women"

0

10

20

30

40

50

60

70

80

90

100

No Outcome Surprising Outcome

Pro

babi

lity

of S

urpr

isin

g O

utco

me

Problems with the DataConfounds and Incomplete Designs

• Unsystematic Manipulation– Do not look for HSB on the expected or ambivalent

outcomes– Vary materials and domain with their proposed surprise

manipulation

• Treatments Based on Intuition– “Surprise” manipulations simply tell people that one

outcome is extremely rare– Or depend on biases and beliefs that participants bring into

the lab

• Between Subjects Designs – do not actually have measures of expectation in the Retrospective Groups

Experiment 1:Effects of Pre-outcome Information on HSB

• All of the theories make different claims about effect that expectation and surprise should have on HSB

• Cannot manipulate expectation or surprise• Can manipulate the amount of pre-outcome

information that supports different outcomes

• Can manipulate whether this information is congruent or incongruent with the given outcome

Pre-outcome Information

• Text describing an upcoming Championship Tennis Match between two players or an ancient battle between two armies.

• Compare and contrast: where the supporting information for opponent was matched

• Completely fictional • “Bias” pre-outcome information by

manipulating the ratio of supporting information for each player

Pre-outcome Information Tennis Equated for Either Outcome

• Mark Krause– Experienced

– Trains hard

– Past Champion

– Top Ranked

– Great Serve

– Patient Style

– No Wind

– Over the hill

– Style Backfires

• Nathan Mitchell– Ambitious

– Natural Athlete

– Olympic Champ

– Top Ranked

– Great Backhand

– Aggressive Style

– Hot Day

– Inexperienced

– Style Backfires

Pre-outcome Information Tennis Supporting Mitchell Wins Outcome


– Trains hard

– Past Champion

– Top Ranked

– Great Serve

– Patient Style

– No Wind

– Over the hill

– Style Backfires


– Natural Athlete

– Olympic Champ

– Top Ranked

– Great Backhand


– Hot Day

– Inexperienced

– Style Backfires

Pre-outcome Information Tennis Supporting Krause Wins Outcome


– Trains hard

– Past Champion

– Top Ranked

– Great Serve

– Patient Style

– No Wind

– Over the hill

– Style Backfires


– Natural Athlete

– Olympic Champ

– Top Ranked

– Great Backhand


– Hot Day

– Inexperienced

– Style Backfires

Outcome Information

• Text that describes the “true” outcome of the match.

• It only tells who won the match

• Does not describe the event or justify the outcome in any way

• Allows for the manipulation of the congruency of the outcome with the pre-outcome information

Pre-outcome Information/Outcome Congruency

• Congruent Condition (Expected) n =31– Krause Supporting Text with Krause Wins outcome– Mitchell Supporting Text with Mitchell Wins outcome

• Equated Condition (Ambivalent) n = 29– Equal Text with Krause Wins outcome– Equal Text with Mitchell Wins outcome

• Incongruent Condition (Surprising) n =31– Krause Supporting Text with Mitchell Wins outcome– Mitchell Supporting Text with Krause Wins outcome

Experiment 1: General Procedure

• Read the pre-outcome text • Rated the likelihood of the possible outcomes

(Predictive Judgment)• Rated how relevant or supportive each sentence was

to the possible outcomes (Evidence Relevance Rating)• Read the outcome text• Rate the how surprising they found the outcome

(Surprise Rating)• One week interval• Attempted to remember their original ratings

(Retrospective Judgment)

Results: Surprise RatingsExperiment 1: Effect of Outcome Congruence on Surprise Ratings

0

10

20

30

40

50

60

70

80

Congruent Equated Incongruent

Text/Outcome Congruency Condition

Sur

pris

e R

atin

g

F (2, 88) = 26.37, p < 0.001, η2 = 0.38

HSB Predictions for Pre-outcome information/outcome congruency

Other

Theory Congruent(Expected)

Equated(Ambivalent)

Incongruent(Surprising)

Anchor &

Adjust




HSB HSB HSB

Metacog.

Cue


- r

(sp, hsb)

Sense-Making


(sp, hsb)

Results: Hindsight BiasExperiment 1: Hindsight Bias as a Function of Outcome Congrency

-40

-30

-20

-10

0

10

20

30

40

Congurent Equated Incongruent

Text/outcome condition

Jud

gm

en

t

Prediction Retrospection

d = -0.01 d = 0.54 d = 0.41

HSB main effect: F (1, 88) = 8.37, p < 0.01, η2 = 0.09HSB X condition: F (2, 88) = 2.36, p = 0.10, η2 = 0.05

HSB Predictions for Pre-outcome information/outcome congruency

Other

Theory Congruent(Expected)

Equated(Ambivalent)

Incongruent(Surprising)

Anchor &

Adjust




HSB HSB HSB

Metacog.

Cue


- r

(sp, hsb)

Sense-Making


(sp, hsb)

Experiment 1: Conclusions

• Congruent outcomes = very little HSB• Equated outcomes = usual HSB• Incongruent outcomes = usual HSB• Somewhat consistent with the surprise cued sense-

making account– Power < 50% detecting interaction– Are memory effects driving the HSB?– Is surprise mediating the HSB effect?

• Small sample for judging correlation predictions

Experiment 2

• Replicate the findings of Experiment 1

• Increased sample size for power to test interaction and relationship between surprise and HSB

• Added a memory test to see if memory effects are leading to the HSB

Pre-outcome Information/Outcome Congruency

• Congruent Condition (Expected) n =67

– Krause Supporting Text with Krause Wins outcome

– Mitchell Supporting Text with Mitchell Wins outcome

• Equated Condition (Ambivalent) n = 61

– Equal Text with Krause Wins outcome

– Equal Text with Mitchell Wins outcome

• Incongruent Condition (Surprising) n =74

– Krause Supporting Text with Mitchell Wins outcome

– Mitchell Supporting Text with Krause Wins outcome

Experiment 2: General Procedure• Read the pre-outcome text • Rated the likelihood of the possible outcomes

(Predictive Judgment)• Rated how relevant or supportive each sentence was to

the possible outcomes (Evidence Relevance Rating)• Read the outcome text• Rate the how surprising they found the outcome

(Surprise Rating)• One week interval• Attempted to remember their original ratings

(Retrospective Judgment) • Attempted to remember as much of the story as possible

(Neutral, Mitchell Supporting, Krause Supporting)

Results: SurpriseExperiment 2: Effect of Outcome Congruence on Surprise Ratings

0

10

20

30

40

50

60

70

80


Text/outcome congruence

Sur

pris

e R

atin

g

F (2, 199) = 45.11, p < 0.001, η2 = 0.31

Results: Hindsight BiasExperiment 2: Hindsight Bias as a Function of Outcome Congrency

-40

-30

-20

-10

0

10

20

30

40

Congurent Equated Incongruent

Text/outcome condition

Jud

gm

en

t

Prediction Retrospection

HSB main effect: F (1, 199) = 14.74, p < 0.001, η2 = 0.07HSB X condition: F (2, 199) = 5.05, p < 0.01, η2 = 0.05

d = -0.06 d = 0.38 d = 0.41

Memory Results

• Divided the story up into “information units”– 17 outcome neutral items– Equated Story = 21 for each player– Biased Stories = 21 for one player, 13 for other

• Two independent raters coded all recall sheets

• High inter-rater reliability ≈ 0.9

Results: Recall Neutral Items (17 possible)

Experiment 2: Memory for Nuetral Items

0

1

2

3

4

5

6


Text/Outcome Congruency

Nu

mb

er

of I

tem

s R

eca

lled

F (2, 199) = 1.40, p = 0.25, η2 = 0.01

Results: Recall in Equated Text Condition

Equated Group (21 possible items each)

0

1

2

3

4

5

6

Support Given Support Alternative

Type of Information

Nu

mb

er

of

Ite

ms

Re

calle

d

F (1, 66) = 43.40, p < 0.001, η2 = 0.40

d = 0.81

Experiment 2: Recall in Baised Text Conditions as a function of Outcome Congruence

0

1

2

3

4

5

6

21 Items Avalible 13 Items Avalible

Type of Information

Num

ber

of It

ems

Rec

alle

d

CongruentIncongruent

Results: Recall in Bias Story Conditions

d = 0.52

d = 0.54

Information main effect: F (1, 133) = 59.00, p < 0.001, η2 = 0.31Info X congruency: F (1, 133) = 32.67, p < 0.001, η2 = 0.20

Results: Surprise & HSBBoth Experiments (n = 293)

• No Correlation• Congruent

r = -0.16, p = 0.13• Equated

r = 0.11, p = 0.30• Incongruent

r = 0.03, p = 0.77

Summary of Experiment 2

• Congruent outcomes = very little HSB• Equated outcomes = usual HSB• Incongruent outcomes = usual HSB• Recalled more outcome supporting information in

all conditions• Surprise level did not predict the magnitude of the

hindsight bias

Surprise & Sense-Making

Info/OutcomeCongruency

Surprise SenseMaking

HSB

Pezzo (2003)

Info/OutcomeCongruency

Surprise

SenseMaking

HSB

Current Studies

Sense-Making & Retrospection

SenseMaking

Sense-making & Memory

ProcessOutcomeSupporting Information

StrongerMemoryOutcomeSupporting


SenseMaking

More Weight to Supporting

DiscountNon-supporting

New Representation

Sense-making & Beliefs


Take Home Message on HSB• “Those who forget the past are doomed to repeat it”• HSB was more prevalent when people were unsure

or just plain wrong in foresight!• This makes it an extremely maladaptive judgment

making bias. • Future directions

– Generalize these results to more “real world” judgment situations.

– Investigate whether the HSB may play a role in hindering learning in tasks that involve predictive and retrospective judgment making.

I bet you already knew all of this!!

Thanks UIC– Dr. Jennifer Wiley – Dr. Susan Goldman– Dr. Joe Magliano– Dr. Gary Raney– Dr. Keith Thiede– Mindy Jensen– Jimmy Wong

– University Fellowship

Thanks ODU– Clinton Comer

– Luke Lin

– Martin Smith-Rodden

– Jennifer McMaster

– Amanda Fletcher

– Jen Taylor

– Dave Finch

Contact: [email protected] Web: www.odu.edu/~iash

mailto:[email protected]

http://www.odu.edu/~iash

Documents

Expectation, Surprise and the Hindsight Bias: The Effect of Information and Outcome Congruency on Retrospective Judgments. Ivan K. Ash, Ph.D. Assistant