4TH ANNUAL WORKSHOP C I C -S R A F A - ICEAA · DONALD RUMSFELD’S FAMOUS RIDDLE-LIKE EXPLANATION OF INCOMPLETE KNOWLEDGE*. . . there are known "knowns." ." . . . things we know

4TH ANNUAL WORKSHOP

CANADA

INTEGRATED COST-SCHEDULE RISK ANALYSIS

FOCUS ON ACTION

KEITH D. HORNBACHER, M.B.A.

SENIOR ASSOCIATE

DAVID T. HULETT, Ph.D.

Hulett & Associates, LLC Los Angeles, CA

+1.310.476.7699 www.projectrisk.com

© 2018 Hulett & Associates, LLC

OUTLINE

• Introduction – Context• Project Risk Management Frameworks, Methods and Practices

• Data quality and elicitation: roles of workshops, interviews

•Method Description• Starting a risk model: “healthy” cost-loaded CPM network (Best

Practices)

• Uncertainty (activity based impact ranges)

• Risk drivers (P, I – risk based impact ranges)

•Taking Action• Prioritized risks: treatment/response workshops

• Cost effective action (Risk Informed Decisions)

• Treatment → Response → Action

ICEAA Canada Workshop, 2018

CO

ST

TIME

CAN YOU ANSWER THIS QUESTION, “HOW CONFIDENT ARE

YOU OF YOUR PROJECT’S COST AND FINISH DATE?”

FRAMEWORKS AND CONTEXT

• ISO 31000

• PROJECT MANAGEMENT INSTITUTE


COMMUNICATE

AND CONSULT

MONITOR AND

REVIEW

ADAPTED FROM ISO 31000:2017

ESTABLISH

CONTEXT

ASSESSTREAT

RECORD

AND

REPORT

IDENTIFY RISKS

PERFORM

QUALITATIVE RISK

ANALYSIS

PERFORM

QUANTITATIVE RISK

ANALYSIS

PLAN RISK

RESPONSES

IMPLEMENT RISK

RESPONSES

MONITOR RISKS

PLAN RISK

MANAGEMENT

PROJECT RISK MANAGEMENT PROCESSES

RISKS: THREATS, OPPORTUNITIES

ELABORATED IN PMBOK® GUIDE, 6TH ED. SEPTEMBER 2017

WHEN APPROPRIATE

ALWAYS

QUANTITATIVE RISK ANALYSIS

•Evolving rapidly as simulation tools + integration with

scheduling software packages advance

•Many organizations require formal risk management

• Canadian Department of National Defence (DND), Canadian Armed

Forces (CAF), Treasury Board Secretariat (TBS)

• NASA Joint cost-schedule Confidence Levels (JCL) in missions,

programs, projects

• US Departments of Defense, Energy . . . Others

• Complex, Large and Mega projects world wide


DONALD RUMSFELD’S FAMOUS RIDDLE-LIKE

EXPLANATION OF INCOMPLETE KNOWLEDGE*

. . . there are known "knowns." . . . things we know that we

know.

There are known unknowns. That is to say there are things

that we now know we don't know.

But there are also unknown unknowns. . . . things we don't

know that we don't know . . .*Source: NATO (Brussels) 6 June 2002 http://www.nato.int/docu/speech/2002/s020606g.htm Emphasis added.


Known

Unknowns

Known

Knowns

Unknown

Unknowns

AW

AR

EN

ES

S

-(a

bsen

t)+

(pr

esen

t)

KNOWLEDGE- (absent) + (present)

RUMSFELD’S RIDDLE: AWARENESS VS. KNOWLEDGE

Unknown

Knowns


RUMSFELD LEFT OUT UNKNOWN KNOWNS

•What we do not like to know

• Unknown knowns

• Genuine ignorance of knowable information

• Willful ignorance - intentional refusal to acknowledge that we know

• http://journals.sagepub.com/doi/abs/10.1177/0967010607084994 Cited 03/20/2017

• See also: Stephen Colbert interview (January 26, 2016 – use of “Unknown-Knowns”) with Donald Rumsfeld:

https://www.youtube.com/watch?v=4Z3z7DvoA-M

Risk Registers are always incomplete. New risks are found in interviews. Often

these risks are “unknown knowns”. Sometimes anonymity is necessary.

ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC

ELICITATION OF RISK DATA

• Incomplete knowledge is a given condition

•Risk data elicitation strategies

• Gather credible, unbiased data

• Efficient - least amount of disturbance among key project members

•Risk workshops

• Set expectations, brainstorm, group dynamics

• Select treatment/response strategies; commit to action

•Risk interviews

• Confidential 1 on 1 with anonymity

• Elaborate Risk Register


PURPOSE OF QUANTITATIVE RISK ANALYSIS

Integrated cost-schedule risk analysis to answer 4 questions

1. How likely is the project to meet its schedule and cost goals on the

current plan?

2. How much schedule and cost contingency is needed to achieve the

desired confidence level of hitting both schedule and cost targets?

3. Which risks are causing potential overrun of schedule and/or cost,

including indirect schedule-driven cost, and are thus high priority for

risk mitigation?

4. Which mitigation (response) action plans have a favorable

cost:benefit ratio?


DESCRIPTION OF METHOD

• Get a good CPM plan/schedule consistent with GAO Best Practices

• Add costs as time-dependent and time-independent resources

• Interview for good Risk Data

• Model uncertainty

• Model project-specific and systemic risks using Risk Drivers

• Prioritize risks at percentile appropriate for your organization (typically P80)

• Workshop: Mitigate risks partially, recording mitigation costs

• Use Joint cost-schedule Confidence Level (JCL) as promise dates and costs

(NASA specifies JCL-70)

• Commit to the risk mitigations

• Take Risk Informed Action

Essential to Engage Your Team Members!!


TEAMS INPUT / FEEDBACK → CREDIBILITY

Critical Path Method (CPM)

Plan/Schedule Data Base

Teams Prepare, Review,

Verify and Validate

Well Planned

Plans/Schedules


Source: DYNM 683 Strategic Value in Quantitative Methods and

Tools ©2018 Keith D Hornbacher, University of Pennsylvania

INTEGRATION, AUGMENTATION, SIMULATION

Variation:1 Variation:45 Variation:11 Variation:3 Variation:5 Variation:4 Variation:3 Variation:50

Variation:134 Variation:49 Variation:32 Variation:13 Variation:7 Variation:8 Variation:6 Variation:108

01 Dec 19 20 Jan 20 10 Mar 20 29 Apr 20 18 Jun 20 07 Aug 20 26 Sep 20 15 Nov 20 04 Jan 21 23 Feb 21 14 Apr 21 03 Jun 21 23 Jul 21 11 Sep 21 31 Oct 21 20 Dec 21 08 Feb 22

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cu

mu

lativ

e P

ro

bab

ilit

y

Distribution Analyzer

Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date









Cost Estimate

Integrate

Source: DYNM 683 Strategic Value in Quantitative Methods and

Tools ©2018 Keith D Hornbacher, University of Pennsylvania

Critical Path Method Plan/Schedule

+Variables

Deliverables

Simulation


EXAMPLE CASE SCHEDULE

OFFSHORE GAS PRODUCTION PLATFORM PROJECT


TEST SCHEDULE

QUALITY AGAINST

GAO 10-POINT

SCHEDULING BEST

PRACTICES

http://www.gao.gov/assets/680/674404.pdf

http://www.gao.gov/assets/680/674404.pdf

GAO PRACTICES

FOR QUALITY

SCHEDULES

THIRD-PARTY SOFTWARE CAN TEST

SCHEDULE QUALITY

Oracle®

Primavera Risk

Analysis

Schedule

Check Report

Deltek

Acumen®

FUSE


ADDING UNCERTAINTY TO MODEL

• Uncertainty in schedule duration, resource usage

✓ Similar to “Common Cause” variation (six sigma process control

concepts) – Walter Shewhart and Edwards Deming

✓ Common Cause variability - source of variation caused by unknown

factors, results in steady but random distribution of output around the

average of the data

✓ Common Cause variation also called random variation, noise, non-

controllable variation

See: http://www.isixsigma.com/dictionary/common-cause-variation/


UNCERTAINTY

• Inherent Variability

• Estimating Error

• Estimating Bias (if it exists)

• Must be taken into account in the Risk Analysis

• Uncertainty - generally not reducible

Risk from uncertainty alone - best you can do . . .

Even with aggressive mitigation


IMPORT INTEGRATED COST-SCHEDULE XEROR MPP FILE INTO RISK ANALYSIS SOFTWARE

Case illustrations use Polaris™ by Booz Allen https://www.boozallen.com/s/product/polaris.html

39-month

$1.7 billion

Project


https://www.boozallen.com/s/product/polaris.html

UNCERTAINTY ASSIGNED TO ACTIVITY DURATIONS

BY REFERENCE RANGES (CATEGORIES)

Uncertainty parameters for entire activity class (engineering, procurement, fabrication . . .)

To use specified ranges on each activity within class, uncertainty values must be correlated 100%


RESOURCE USAGE UNCERTAINTY RANGES


TIME

SCATTERPLOT: EFFECT OF UNCERTAINTY

ON DURATIONS AND RESOURCES

Correlation Finish Date:Cost

calculated 62%. Upward

slope reflects effect of

uncertain durations on cost


CO

ST


EFFECT OF UNCERTAINTY ON FINISH DATE

A. Deterministic Finish

Date 04 April 2020

B. P-80 Finish Date

01 Sept 2020

Effect ≈ + 5 months

A.

B.

ICEAA Canada Workshop, 2018©2018 Hulett & Associates, LLC

EFFECT OF UNCERTAINTY ON COST

A. Baseline = $1.69

billion

B. P-80 cost = $2.12

billion

Over cost ≈ $427

million or 25%

A.

B.


PROJECT-SPECIFIC RISKS ― ADD TO UNCERTAINTY

• Risks may or may not happen

✓Risk - similar to “Special Cause” variation in six sigma

✓Special Cause variation caused by known factors that result in a non-random

distribution of output

✓Special Cause variation - a shift in output caused by a specific factor such as

environmental conditions or process input parameters.

✓Can be accounted for directly and potentially removed . . .”

http://www.isixsigma.com/dictionary/variation-special-cause/

• Pre-mitigated risks - subject of risk mitigation workshops


http://www.isixsigma.com/dictionary/variation-special-cause/

RISK DRIVERS – PROBABILITY OF OCCURRENCE

AND IMPACTS

•Each identified risk has a probability that it will occur with

some effect on time or cost

• If risk occurs, affects activities’ estimated durations, costs

• If time-dependent resources (labor, rented equipment) it will vary the

daily burn rate

• If time-independent resources (equipment to be installed, material) it

will affect the entire cost directly

• If risk does not occur, there are no affects (no change)


RISK DRIVERS – APPLICATION OF IMPACTS

•A risk may affect multiple activities

•Activities may be affected by multiple risks

• If risk driver occurs, it has multiplicative effect on

estimated durations, costs of activities it affects• Multiplier < 1.0 ➔ shorter duration, opportunity

• Multiplier > 1.0 ➔ longer duration, threat

•Multiplier is chosen at random from input distribution

(usually 3-point estimate, triangle)


INTRODUCING RISK DRIVERS THAT

CAUSE ADDITIONAL VARIATION IN THE SIMULATION

3 of 4 specified risk drivers are enabled; UID 1 (engineering productivity) will happen (P=100%)

Impact may be over or under estimated (0.9, 1.05, 1.3); UID 1 impacts two Design activities


100% LIKELY RISK DRIVER’S EFFECT ON 100 DAY DESIGN DURATION

Activity’s duration

looks like a triangle

Min = 90

Max =130

ML = 105


RISK DRIVER LIKELIHOOD < 100% (P = 40%)

UID 2: P = 40%, impact (if it occurs) is 0.9, 1.1 and 1.4. No impact

occurs in 60% of the trials. UID 2 applied to both Build activities


60% “SPIKE” HAS NO IMPACT (X 1.0)40% OF TRIALS EXPERIENCE RANGED IMPACTS


USING RISK DRIVERS METHOD

UID 8 impact distribution:

Triangle: 0.95, 1.05, 1.25

Risk Drivers Probabilities assigned

(UID 8, P 50%)

UID 8 Risk Driver

assigned to Activities


DILEMMA: WHAT END DATE AND TOTAL

COST SHOULD BE PUT FORWARD?

• P-80 finish date is 27-Jan-21, risks add 5 months to the project (above uncertainty)

• P-80 cost is $2.27 billion, risks add $150 million to the project (above uncertainty)

➢ Is this enough? Too much? What action should project executives take?


TIME – COST SCATTERPLOT CAN ESTIMATE

CONFIDENCE OF MEETING BOTH TARGET OBJECTIVES

• Histograms / cumulative distribution functions estimate

finish date and cost individually

• Combine results with scatterplots

• To meet both targets requires more conservative estimates

(later date, more cost)

• How much more time, cost depends on their correlation

• Joint cost-schedule Confidence Level (JCL)


PLAN TO MEET BOTH FINISH DATE AND

COST TARGETS FROM JCL SCATTERPLOT


JCL-80 COMPARED WITH P-80 RESULTS

Histogram/Cumulative Distributions (P-80) and Joint Confidence Level* (JCL-80)

Results with Project-Specific Risks and Uncertainty

Baseline

Finish Date 4-April-2020

Budgeted Cost $1.70 Billion

Risk Analysis Results

Schedule Date Months added

P-80 27-Jan-2021 9.8

JCL-80 14-Mar-2021 11.3

Difference 46 cal. days 1.5

Cost Billions Dollars Added (billions)

P-80 $2.27 $0.57

JCL-80 $2.31 $0.61

Difference $0.04 $0.04

*Several combinations of cost and time will satisfy JCL-80; need to select one or a range of those

likely to occur from most concentrated region on scatter plot


COMPARE WHAT RISK ANALYSIS TYPICALLY

PREDICTS VS. WHAT ACTUALLY HAPPENS

Source: John K. Hollmann, PE, “Reliable Risk Quantification for Project Cost and Schedule”,

AACE International webinar December 15, 2015


INCORPORATE SYSTEMIC RISKS

INTO THE MONTE CARLO SIMULATION (MCS)

• Systemic Risks may include

• Technical complexity, new technology challenging

• Scope not fully known

• Process definition not complete

• Megaproject complexity, size / duration, participants

• Project organization, e.g., joint venture, multiple EPCs

•Systemic factors can be measured and impacts on

project success estimated using parametric techniques

© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018

Projects often are• In tolerances (+ / - 20%)

• Some have serious problems

with overruns approaching 70%*

In our case example:• P-80 ➔28-Oct-21 ≈19 months

• The 80th percentile is approaching

the second mode in the histogram

*Hollmann, John, “Risk Analysis on the Edge of

Chaos,” Cost Engineering (© AACE

International), January/February 2015

COMPLEXITY & PRESSURE COMBINED IN OUR

MODEL PRODUCE A BI-MODAL DISTRIBUTION


SCATTERPLOT WITH SYSTEMIC RISKS ADDED


MOVING FROM ANALYSIS TO ACTION

•Focus on Action

• Prioritized risks: treatment/response workshops

•Cost effective action (Risk Informed Decisions)

•Conclusion: Treatment → Response → Action


ITERATIVE APPROACH TO PRIORITIZING RISK

Purpose, discover which risks contribute the most days at

the P-80 level

1. Compute the Baseline with All Risks In

2. Iteration # 1: Simulate with each risk disabled in turn,

recording the P-80 date

a. The risk with the earliest P-80 date is 1st priority

b. Take it out for Iteration # 2

3. Iteration # 2: Simulate the remaining risks, disabling each

in turn, recording P-80, choose earliest. Take it out for

Iteration # 3 . . . etc.


RISKS PRIORITIZED AT THE P-80 CONFIDENCE

LEVEL MEASURED IN “DAYS SAVED”

Systemic Risks are Important


RISKS PRIORITIZED TO P-80 AND DAYS SAVED

PLUS EFFECT OF UNCERTAINTYRisks Prioritized to P-80

NameDays

Saved

Megaproject may have excessive schedule pressure 210

The organization has other priority projects so personnel and funding may be unavailable 112

Megaproject may have interdependency problems 52

Fabrication yards may experience lower Productivity than planned 32

Engineering may be complicated by using offshore design firm 18

Megaproject may have coordination problems offshore sourcing 17

Suppliers of installed equipment may be busy 12

Fabrication and installation problems may be revealed during HUC 12

Installation may be delayed due to coordination problems 2

Bids may be Abusive leading to delayed approval 0

The subsea geological conditions may be different than expected 0

Contingency due to Project-Specific and Systemic Risks 467

Contingency due to Uncertainty 150

Total Contingency 617


RISK MITIGATION (TREATMENT, RESPONSE)

•Risks can be partially mitigated, perhaps substantially

•Mitigation actions

•New, not known to the interviewees, different from yesterday

•Discussed in Risk mitigation/treatment workshop

•Committed to by management so funded, staffed, monitored

and reported on

•When agreed, estimate cost and timing of mitigation

•Estimate improvement to risk parameters


COMPARE PRE AND POST MITIGATION SCHEDULE

Simple mitigation for Schedule improves

from 28-Oct-21 to 24-Jan-21 (≈ 9 months)

but still later than baseline 4-Apr-20


COMPARE PRE AND POST MITIGATION COST

Simple mitigation for Cost

• Reduces $2.48 billion to $2.36 billion ≈ $120

million

• Includes $70 million assumed for Mitigation

Costs

• Still $663 million above baseline cost


RECAP OF METHOD

• Get good CPM plan/schedule per GAO Best Practices

• Add costs as time-dependent and time-independent resources

• Interview for good Risk Data

• Model uncertainty

• Model project-specific and systemic risks using Risk Drivers

• Use JCL-80 as promise dates and costs

• Prioritize the risks at P-80 and days saved

• Mitigate risks partially, recording mitigation costs

• Commit to the risk mitigations, treatments, responses

• Take Risk Informed Action


4th Annual WorkshopCanada

Integrated Cost-Schedule Risk AnalysisFocus on Action

KEITH D. HORNBACHER, M.B.A.

SENIOR ASSOCIATE

DAVID T. HULETT, Ph.D.

Hulett & Associates, LLC Los Angeles, CA

+1.310.476.7699 www.projectrisk.com


Documents

4TH ANNUAL WORKSHOP C I C -S R A F A - ICEAA · DONALD RUMSFELD’S FAMOUS RIDDLE-LIKE EXPLANATION OF INCOMPLETE KNOWLEDGE*. . . there are known "knowns." ." . . . things we know