1

Psychological Impacts on Judgment in Cost Estimation

Jordan GarnerUC Davis (JPL Summer Hire)

Art B. ChmielewskiJet Propulsion Laboratory

California Institute of Technology

September 12, 2011

2

Special Thanks to

• Dr. David Ullman of Robust Decisions and Oregon State University for his assistance with the web experiment and continued support of this novel research.

• Prof. Don Forsyth of the University of Richmond for his expert consultation on socio-psychological effects in decision making.

Bad Cost EstimatesHappen

31400%

1500%

275% 1100%

2200%

258%

220%

100%

1400%600%

4

Overruns Start with Bad Initial Cost Estimates

• Bad cost estimates are in every sector of business world: construction projects, movie business, transportation projects, military programs, aerospace, etc.

• Bad cost estimates know no borders, race, sex or century.

5

Causes of Overruns

• Overruns start with flawed initial cost estimates and inadequate reserves.

• However, the post mortem analyses give less blame to the estimating than to failures in execution such as:– Changes in scope and requirements– Inadequate communication – Government and contractor intervention– Unforeseen technical issues– New technology– Acts of god

• Specific technical reasons for overruns seem to be more palatable than poor cost estimates.

6

Are Estimates Getting Better?

“For the past 70 years, for which data on cost estimation is observable, no significant improvements in forecasting, estimating or prediction a project’s cost have ever been made. This is despite the increase in awareness of past estimation inaccuracy, new strategies of estimation, the hiring of more experts to help the estimation process, inventions solving past technical and communication issues.”

– Prof. Bent Flyvbjerg, at Oxford University's Saïd Business School

7

Unaccounted Psychological Effects?

• Thesis: Could humans be prone to psychological factors that make them truly and honestly believe in poor estimates?

• We conducted a simple experiment to test and quantitatively measure the power of psychological fallacies on people’s ability to make estimates.

8

Overheard During Cost Estimating:

• “I have a bogey of $400k. Please give me your own estimate.” • “We will hold 30% reserve for you.”• “I sent you a WBS cost table. Can you fill it in?”• “We need your best estimate by Friday.”• You have an allocation of $1.3M, can you give me an

estimate?

Our simple experiment proved that the above common costing phrases guarantee overruns!

9

• Participants in the on-line experiment were asked in different ways to estimate the time needed to perform a simple task – washing the dishes shown on the next chart.

Dishwashing Experiment

10

11

Psychological Effects Tested• 5 psychological effects were tested :

1. Anchoring 2. Q&A Mismatch3. Decomposition4. Reserve Comfort5. Planning Fallacy

• Every respondent to the survey was randomly asked one of several questions testing different psychological heuristics or fallacies.

• 507 volunteers participated: 142 JPLers, 305 college students and 60 other adults. ~2300 data points were collected.

12

All answers were graphed and analyzed to establish conclusions

anchor best case nominal 0.5 decomposition 0.75 0.99 worst case0

10

20

30

40

50

60

70

80

90

lower Standard Deviation

Estim

ated

min

utes

Psychological category

13

Effect #1: Anchoring

The objective was to test how easily influenced people may be by a wrong answer – “the anchor.”The anchor set asked:Estimate how many minutes it will take you to clean the kitchen. One respondent estimated that it will take about 10 minutes to finish cleaning up. He may be wrong of course.

14

1. Anchoring Results

• The nominal value was 30 min, the anchored case 25 min.

• The “best case scenario” estimate (described later) was 27 min which was 2 min LONGER than the anchored result.

• The result points out that it is very easy to dramatically skew the estimates by asking anchored questions, such as: “We would like you to come in around $6M”, “I have a bogey of $400k for you”, “the last robot arm we built cost $7M”…

15

Effect #2: Q&A Mismatch

The purpose was to test if there is a mismatch between the type of estimate expected and provided.

Different participants were asked:• Estimate how many minutes it will take you to clean the whole

kitchen.• There is a 50% chance that you will finish this task within __ min• There is a 75% chance that you will finish this task within __ min• There is a 99% chance that you will finish this task within __ min

16

2. Q&A Mismatch Results

• The 50% confidence estimate was 31 min. The nominal estimate was 30 min. People unconsciously interpret the nominal as the 50% case, meaning that you will exceed your estimate in half the cases!

• However, when a manager asks for an estimate he/she expects a much more reliable result, possibly in the 80%-90% confidence range. This points out that there is mismatch between the expectation and the answer.

17

Effect #3: Decomposition

The objective was to test if decomposing the project into smaller pieces and deeper levels of a WBS improved accuracy of the estimate.Estimate decomposition was simulated by asking:1. How many minutes will it take to clean all the plates and the

sets of silver?2. How long will it take to clean the sets of coffee cups and

saucers?3. How long will it take to clean the bowls?4. Etc.

18

3. Decomposition Results

• Decomposition average was 31 minutes, just one minute longer than the nominal average (30 min). The attempt at becoming more accurate by cutting up the project was not accomplished.

• Decomposition, at least in this case, was more time consuming than helpful.

• Deep decompositions provide more detail but compound psychological effects.

19

Effect #4: Reserve Comfort

This question tested the realism of “a comfortable” reserve.

Respondents were asked:1. I am 90% sure that the time it will actually take to clean the

kitchen is within plus or minus __ minutes from my estimate.

20

4. Reserve Comfort Results

• The reserve for 90% confidence was 8 min or 28%. The 25-30% seems to be the magical intuitional comfort level that is used by many industries.

• When a manager asks for a reserve he/she means “I want to be very sure that I will not exceed this reserve. I want my reserve to cover almost the worst case.”

• However, that is not how it is interpreted by the employee. – The worst case estimate was 51 min and required 70% reserve.– The 99% confidence case averaged 45 min. and needed 50% reserve.

Both of these cases are significantly higher than the popular 30%.

21

Large Projects Reserve Comparison

10%

20%

30%

40%

50%

60%

70%

80%

20% 30% 40% 50% 60% 70% 80%10%

Nee

ded

Res

erve

Under

estim

ated

Reser

ve

Overe

stim

ated

Reser

ve

90%

100%

100%90%

Planned Reserve

Recent aerospace projects

The realistic amount of budget reserve required for 18 large projects studied is 52%.

22

Effect #5: Planning Fallacy

The planning fallacy, as defined by Daniel Kahneman and Amos Tversk is a tendency to be overly optimistic in planning. To asses the extent of optimism we asked:1. In the best case scenario (if everything went as well as

possible), how many minutes would it take you to clean the whole kitchen?

2. In the worst case scenario (if everything went as poorly as possible), how many minutes it would take you to clean the whole kitchen?

23

5. Planning Fallacy Results

• The following results were obtained:– 51 min worst case– 45 min 99% confidence– 30 min nominal– 27 min best case

• These results show how skewed people are toward optimism. The nominal estimate was 10% longer than the best case but 70% shorter than the worst case.

• People are so optimistic that it was easy to anchor them down but anchoring up failed.

24

Conclusions• To improve the quality of cost estimates it is recommended to

diminish the effects of psychological impact on judgment:

Train the managers not to anchor. Establish proper Estimation Language which makes the

questions compatible with common interpretation. Deep decompositions do not improve accuracy. Calculate the reserve based on risk. Account for optimism by including in the baseline likely,

historical and common risks.