1

Arthur B. ChmielewskiProject Manager

Jet Propulsion LaboratoryCalifornia Institute of Technology

Presentation at the 7th NASA Project Management ChallengeFebruary 9, 2010

Copyright 2010 California Institute of TechnologyUsed with permission

2

Many results presented here are based on the research conducted by Dr. David Ullman of Robust Decisions Inc. for the Space Technology 6 mission.

McRisk studies were conducted by Charles Garner, JPL

3

Famous business consultants have one thing in common. They develop an innovative premise, theory or a method which allows solving difficult business problems with sales, personnel, marketing or product development.

Consultants explain their premise in a book which validates it with dramatic examples from life of famous corporations.

5

We NASA managers, highly technically oriented, feel that the world of business consulting - marketing, sales or mergers -does not overlap with ours.

• Can the method of selection of TV commercials for American Idol help with vacuum testing of a deep space probe?• Could iPod sales be related to cost estimating for an asteroid lander?

Not only is the answer to these questions “yes” but the wisdom of top business consultants may be critical in the fight against the biggest aerospace plague:

61400%

1000%846%

1100%

600%

257%

230%

300%

1400%

7

Mission type

# of MissionsStudied

Reserve under-

estimated by

Flagship( > $800M)

4 164%

Medium size (< $300M)

5 19%

Large Instruments (< $150M)

2 34%

System Technology Experiments (< $150M)

3 131%

Technology Experiments(<$25M)

11 107%**

Small Technology Experiments (<$5M) 13 315%**

• Reserve estimation errors are very high with exception of medium size missions

• Reserve estimation accuracy has not improved in the last 20 years

• Reserve errors are not limited to one organization, center, type of mission

• Only 3/38* missions did not exceed the budget reserve allocation

• Only 1/38 missions used a strict process to calculate the amount of needed reserve

* 38 missions consisted of: 11 microgravity experiments, 13 In-Step experiments, 14 other NASA missions** No reserve was planned for most technology experiments

8

10%

20%

30%

40%

50%

60%

70%

80%

20% 30% 40% 50% 60% 70% 80%10%

Planned Reserve

Nee

ded

Res

erve

90%

100%

100%90%

Data from C. Garner, et al, 2002R. Metzger, et al, 2003R. Schmitz, 1991

9

NASA DOD DOT USA Germany Europe Asia Aeronautical Engineers Civil Engineers Campaign Managers Party Planners Pool builders

Engineers overran construction of Burj Al Arab in 2009Pharaohs needed more slaves to finish the pyramids in

3000 BC

Humans overrun!

10

BUT WHY?

THESE ARE THE USUAL POST-MORTEM ANSWERS: Poor communication Underestimated challenges Funding interruption Bad things happened Unknown unknowns popped up Requirements not well understood Reserves not adequate Initial cost estimates not accurate Strategic misrepresentation of cost*

* A real phrase from a DOD cost analysis report

11

We use data bases for cost estimates We have cost reviews We manage risks We use processes We use earned value We use parametric studies We try and try and try and…

…WE STILL OVERRUN!

Every overrun occurs at the end of the project but originates years earlier with the first entry into the Excel preliminary cost estimate spreadsheet.

This first entry is frequently wrong because its author is under influence of many mostly subconscious psychological factors.

If we could understand the psychology of cost estimates, then maybe we could improve their accuracy!

12

NASA scientists and engineers know well that psychology does not play any role in their precise, factual, highly technical cost estimating based on precise facts.

That is a great news because according to top consultants psychology plays a major role in business at every level. Let’s look at the psychological effects we are immune to.

13

14

In a simple test 215 people were asked how long it will take to clean up a stack of dirty dishes. They were given a detailed list of the dishes.

15

One group was asked:

The kitchen needs to be clean. Estimate how many minutes it will take you?

Another group was asked:

The kitchen needs to be clean in 15 min. Estimate how many minutes it will take you?

16

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 10 20 30 40 50 60

Time (min)

Prob

abili

ty

Base: mean = 32 minstd= 10 min

Anchored - 15 min: mean = 17 minstd= 5 min

17

The quality of the questions determines the quality of the answer. In cost estimating we allow ourselves very imprecise questions:

Manager: “How long will it take to do thermal vacuum testing?”

Engineer: “Oh, 5 weeks”

What does “5 weeks” convey? The average time is 5 weeks – 50% of the time it will

take longer, 50% shorter He is 90% sure can be done in 5 weeks Something else

18

In the dish washing experiment subjects were asked either:

• Estimate how many minutes it will take you to clean up the kitchen?

or• Estimate how many minutes so that you are 50%

sure you will be finished cleaning up the kitchen?or • Estimate how many minutes so that you are 90%

sure you will be finished cleaning up the kitchen?

19

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0 10 20 30 40 50 60

Time (min)

Prob

abili

ty

Base: mean = 32 minstd= 10 min

90%: mean = 29 minstd= 8 min

50%: mean = 18 minstd= 6 min

No one asked if a dishwasher was available

If someone gives you something you give him something back. If you do not follow this rule you suffer social consequences.

That is why you get: Free address labels 2 free nights before your timeshare presentation “Diet coke case study”

Cost estimating: I will give you a low estimate and you will give me a

job.

20

My second offer is so much better than the first that it cannot be declined. The zoo trip study

Cost estimating: After the initial cost estimate is too high “scrub” the

costs and, voila, get a much lower number Identify descopes to feel better about strained cost

estimate

21

There is a desire within human beings to be consistent with their actions, beliefs and attitudes. It is easier and quicker to make decisions based on our past conduct. Timeshare salesmen ask first if you like vacations Wii case study

In cost estimating the engineer is first asked to give a quick estimate. When asked later to revise it produces a similar result.

22

We conclude correctness by watching accepted practices. This especially applies under conditions of uncertainty and similarity. 4/5 dentist recommend this gum for patients who chew gum “I like football and I switched to ATT.”

In cost estimating we frequently hear: “Everybody in our section does cost estimating that way.” “We normally do not list any programmatic risks.” “The last mission had only 1 FTE for electronics.”

23

We like people who are similar to us. Liking greatly increases compliance. It is subconscious. Tiger Woods sells Buicks Student fund raisers Car ads with models

Cost estimating: Is the cost estimate affected by who requests it and

how the request is made? Is it affected by who will be your boss on the project

and if you like her?24

Subjects comply with requests from the perceived authority. An authority possesses a high level of knowledge and experience and may benefit the subject. Robert Young, MD Sanka commercials

Cost estimating: The manager asks: “Can you see if my $250k bogey

is a good estimate for this radiation test?” Would a cost estimate request from the Center

Director produce a different value than from a colleague?

25

People assign more value to anything when it is scarcer. This principle also applies to information. Limited, final release of a Disney movie Only 50 tickets remain…

In cost estimating: How does instrument development compare for a

competed award vs. internal development How does the cost estimate compare when the

engineer is swamped with jobs vs. worried about no new assignment on the horizon?

26

People get a lot of comfort from detailed plans. The more detail, the more comfort and feeling of accomplishment and safety.

Trouble is that plans rarely work and need to be updated so frequently that instead of project guidance they become project drag.

Costs: Elaborate plans can increase costs Guiding principles are better than plans Simple laws frequently are better than processes

27

Managers tend to be fascinated with how to do things, how to change things, how to develop processes. Frequently “hows” take precedence of “whats”. What is our business, what is our product, what is our uniqueness. Home Depot’s labor cutting initiative McDonalds – the largest leasing company

In costing: We focus on following a process We miss What is the meaning of each number and

What are the assumptions28

Companies trust their experienced personnel to develop similar products. Bad experiences are short term, good memories are

long term due to location in different parts of brain XBOX 360 vs. Zoom vs. iPod case study

In developing cost estimates experts assume that old experiences are applicable “Heritage”, “small modification”, “off-the-shelf” are

frequently words of doom

29

Complex human psychology is at play in cost estimating

The effect of psychology on cost is rarely acknowledged and thus pervasive

Psychological effects are subconscious and hard to correct

Psychological effects skew initial cost inputs which build up with the number of estimators and subsystems

30

31

1. The atoms of cost estimate must be free of bias, emotion and common cognitive errors caused by human psychology This is not even acknowledged by estimators

2. Use proper tools for reality check of grass roots We are becoming excellent at this

3. Calculate needed budget reserve based on risk This is almost never done

32

JPL and Robust Decisions Inc. are developing a software tool which calculates budget reserve based on risk.

The tool removes the effects of human psychology by asking precise cost questions from a menu of “worry generators”.

The tool has been validated by 6 NASA instrument developments.

We would welcome the opportunity to use it on your mission.

33

Making Robust Decisions, David G. UllmanWired to Care, Dev PatnaikWork Wanted, James W. Walker, Linda H. LewisInfluence: Science and Practice, Robert CialdiniThe Three Laws of Performance, Steve Zaffron, Dave LoganBuyology: Truth and Lies about Why We Buy, Martin Lindstrom, Paco UnderhillThe Innovator’s Prescription, Clayton M. Christensen, Jerome H. Grossman, J.

WongProve It before You Promote It, Steve Cuno, Michael ShermerHarvard Business Review, Jan-Dec 2009Rethink: A Business Manifesto for Cutting Costs and Boosting Innovation, Ric

Merrifield

34