4TH ANNUAL WORKSHOP
CANADA
INTEGRATED COST-SCHEDULE RISK ANALYSIS
FOCUS ON ACTION
KEITH D. HORNBACHER, M.B.A.
SENIOR ASSOCIATE
DAVID T. HULETT, Ph.D.
Hulett & Associates, LLC Los Angeles, CA
+1.310.476.7699 www.projectrisk.com
© 2018 Hulett & Associates, LLC
OUTLINE
• Introduction – Context• Project Risk Management Frameworks, Methods and Practices
• Data quality and elicitation: roles of workshops, interviews
•Method Description• Starting a risk model: “healthy” cost-loaded CPM network (Best
Practices)
• Uncertainty (activity based impact ranges)
• Risk drivers (P, I – risk based impact ranges)
•Taking Action• Prioritized risks: treatment/response workshops
• Cost effective action (Risk Informed Decisions)
• Treatment → Response → Action
ICEAA Canada Workshop, 2018
CO
ST
TIME
CAN YOU ANSWER THIS QUESTION, “HOW CONFIDENT ARE
YOU OF YOUR PROJECT’S COST AND FINISH DATE?”
FRAMEWORKS AND CONTEXT
• ISO 31000
• PROJECT MANAGEMENT INSTITUTE
ICEAA Canada Workshop, 2018
COMMUNICATE
AND CONSULT
MONITOR AND
REVIEW
ADAPTED FROM ISO 31000:2017
ESTABLISH
CONTEXT
ASSESSTREAT
RECORD
AND
REPORT
IDENTIFY RISKS
PERFORM
QUALITATIVE RISK
ANALYSIS
PERFORM
QUANTITATIVE RISK
ANALYSIS
PLAN RISK
RESPONSES
IMPLEMENT RISK
RESPONSES
MONITOR RISKS
PLAN RISK
MANAGEMENT
PROJECT RISK MANAGEMENT PROCESSES
RISKS: THREATS, OPPORTUNITIES
ELABORATED IN PMBOK® GUIDE, 6TH ED. SEPTEMBER 2017
WHEN APPROPRIATE
ALWAYS
QUANTITATIVE RISK ANALYSIS
•Evolving rapidly as simulation tools + integration with
scheduling software packages advance
•Many organizations require formal risk management
• Canadian Department of National Defence (DND), Canadian Armed
Forces (CAF), Treasury Board Secretariat (TBS)
• NASA Joint cost-schedule Confidence Levels (JCL) in missions,
programs, projects
• US Departments of Defense, Energy . . . Others
• Complex, Large and Mega projects world wide
ICEAA Canada Workshop, 2018
DONALD RUMSFELD’S FAMOUS RIDDLE-LIKE
EXPLANATION OF INCOMPLETE KNOWLEDGE*
. . . there are known "knowns." . . . things we know that we
know.
There are known unknowns. That is to say there are things
that we now know we don't know.
But there are also unknown unknowns. . . . things we don't
know that we don't know . . .*Source: NATO (Brussels) 6 June 2002 http://www.nato.int/docu/speech/2002/s020606g.htm Emphasis added.
ICEAA Canada Workshop, 2018
Known
Unknowns
Known
Knowns
Unknown
Unknowns
AW
AR
EN
ES
S
-(a
bsen
t)+
(pr
esen
t)
KNOWLEDGE- (absent) + (present)
RUMSFELD’S RIDDLE: AWARENESS VS. KNOWLEDGE
Unknown
Knowns
ICEAA Canada Workshop, 2018
RUMSFELD LEFT OUT UNKNOWN KNOWNS
•What we do not like to know
• Unknown knowns
• Genuine ignorance of knowable information
• Willful ignorance - intentional refusal to acknowledge that we know
• http://journals.sagepub.com/doi/abs/10.1177/0967010607084994 Cited 03/20/2017
• See also: Stephen Colbert interview (January 26, 2016 – use of “Unknown-Knowns”) with Donald Rumsfeld:
https://www.youtube.com/watch?v=4Z3z7DvoA-M
Risk Registers are always incomplete. New risks are found in interviews. Often
these risks are “unknown knowns”. Sometimes anonymity is necessary.
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
ELICITATION OF RISK DATA
• Incomplete knowledge is a given condition
•Risk data elicitation strategies
• Gather credible, unbiased data
• Efficient - least amount of disturbance among key project members
•Risk workshops
• Set expectations, brainstorm, group dynamics
• Select treatment/response strategies; commit to action
•Risk interviews
• Confidential 1 on 1 with anonymity
• Elaborate Risk Register
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
PURPOSE OF QUANTITATIVE RISK ANALYSIS
Integrated cost-schedule risk analysis to answer 4 questions
1. How likely is the project to meet its schedule and cost goals on the
current plan?
2. How much schedule and cost contingency is needed to achieve the
desired confidence level of hitting both schedule and cost targets?
3. Which risks are causing potential overrun of schedule and/or cost,
including indirect schedule-driven cost, and are thus high priority for
risk mitigation?
4. Which mitigation (response) action plans have a favorable
cost:benefit ratio?
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
DESCRIPTION OF METHOD
• Get a good CPM plan/schedule consistent with GAO Best Practices
• Add costs as time-dependent and time-independent resources
• Interview for good Risk Data
• Model uncertainty
• Model project-specific and systemic risks using Risk Drivers
• Prioritize risks at percentile appropriate for your organization (typically P80)
• Workshop: Mitigate risks partially, recording mitigation costs
• Use Joint cost-schedule Confidence Level (JCL) as promise dates and costs
(NASA specifies JCL-70)
• Commit to the risk mitigations
• Take Risk Informed Action
Essential to Engage Your Team Members!!
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
TEAMS INPUT / FEEDBACK → CREDIBILITY
Critical Path Method (CPM)
Plan/Schedule Data Base
Teams Prepare, Review,
Verify and Validate
Well Planned
Plans/Schedules
ICEAA Canada Workshop, 2018
Source: DYNM 683 Strategic Value in Quantitative Methods and
Tools ©2018 Keith D Hornbacher, University of Pennsylvania
INTEGRATION, AUGMENTATION, SIMULATION
Variation:1 Variation:45 Variation:11 Variation:3 Variation:5 Variation:4 Variation:3 Variation:50
Variation:134 Variation:49 Variation:32 Variation:13 Variation:7 Variation:8 Variation:6 Variation:108
01 Dec 19 20 Jan 20 10 Mar 20 29 Apr 20 18 Jun 20 07 Aug 20 26 Sep 20 15 Nov 20 04 Jan 21 23 Feb 21 14 Apr 21 03 Jun 21 23 Jul 21 11 Sep 21 31 Oct 21 20 Dec 21 08 Feb 22
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Cu
mu
lativ
e P
ro
bab
ilit
y
Distribution Analyzer
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Spacecraft for PMChallenge 2009 - Entire Plan - Finish Date
Cost Estimate
Integrate
Source: DYNM 683 Strategic Value in Quantitative Methods and
Tools ©2018 Keith D Hornbacher, University of Pennsylvania
Critical Path Method Plan/Schedule
+Variables
Deliverables
Simulation
ICEAA Canada Workshop, 2018
EXAMPLE CASE SCHEDULE
OFFSHORE GAS PRODUCTION PLATFORM PROJECT
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
TEST SCHEDULE
QUALITY AGAINST
GAO 10-POINT
SCHEDULING BEST
PRACTICES
http://www.gao.gov/assets/680/674404.pdf
GAO PRACTICES
FOR QUALITY
SCHEDULES
THIRD-PARTY SOFTWARE CAN TEST
SCHEDULE QUALITY
Oracle®
Primavera Risk
Analysis
Schedule
Check Report
Deltek
Acumen®
FUSE
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
ADDING UNCERTAINTY TO MODEL
• Uncertainty in schedule duration, resource usage
✓ Similar to “Common Cause” variation (six sigma process control
concepts) – Walter Shewhart and Edwards Deming
✓ Common Cause variability - source of variation caused by unknown
factors, results in steady but random distribution of output around the
average of the data
✓ Common Cause variation also called random variation, noise, non-
controllable variation
See: http://www.isixsigma.com/dictionary/common-cause-variation/
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
UNCERTAINTY
• Inherent Variability
• Estimating Error
• Estimating Bias (if it exists)
• Must be taken into account in the Risk Analysis
• Uncertainty - generally not reducible
Risk from uncertainty alone - best you can do . . .
Even with aggressive mitigation
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
IMPORT INTEGRATED COST-SCHEDULE XEROR MPP FILE INTO RISK ANALYSIS SOFTWARE
Case illustrations use Polaris™ by Booz Allen https://www.boozallen.com/s/product/polaris.html
39-month
$1.7 billion
Project
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
UNCERTAINTY ASSIGNED TO ACTIVITY DURATIONS
BY REFERENCE RANGES (CATEGORIES)
Uncertainty parameters for entire activity class (engineering, procurement, fabrication . . .)
To use specified ranges on each activity within class, uncertainty values must be correlated 100%
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
RESOURCE USAGE UNCERTAINTY RANGES
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
TIME
SCATTERPLOT: EFFECT OF UNCERTAINTY
ON DURATIONS AND RESOURCES
Correlation Finish Date:Cost
calculated 62%. Upward
slope reflects effect of
uncertain durations on cost
ICEAA Canada Workshop, 2018
CO
ST
© 2018 Hulett & Associates, LLC
EFFECT OF UNCERTAINTY ON FINISH DATE
A. Deterministic Finish
Date 04 April 2020
B. P-80 Finish Date
01 Sept 2020
Effect ≈ + 5 months
A.
B.
ICEAA Canada Workshop, 2018©2018 Hulett & Associates, LLC
EFFECT OF UNCERTAINTY ON COST
A. Baseline = $1.69
billion
B. P-80 cost = $2.12
billion
Over cost ≈ $427
million or 25%
A.
B.
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
PROJECT-SPECIFIC RISKS ― ADD TO UNCERTAINTY
• Risks may or may not happen
✓Risk - similar to “Special Cause” variation in six sigma
✓Special Cause variation caused by known factors that result in a non-random
distribution of output
✓Special Cause variation - a shift in output caused by a specific factor such as
environmental conditions or process input parameters.
✓Can be accounted for directly and potentially removed . . .”
http://www.isixsigma.com/dictionary/variation-special-cause/
• Pre-mitigated risks - subject of risk mitigation workshops
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
RISK DRIVERS – PROBABILITY OF OCCURRENCE
AND IMPACTS
•Each identified risk has a probability that it will occur with
some effect on time or cost
• If risk occurs, affects activities’ estimated durations, costs
• If time-dependent resources (labor, rented equipment) it will vary the
daily burn rate
• If time-independent resources (equipment to be installed, material) it
will affect the entire cost directly
• If risk does not occur, there are no affects (no change)
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
RISK DRIVERS – APPLICATION OF IMPACTS
•A risk may affect multiple activities
•Activities may be affected by multiple risks
• If risk driver occurs, it has multiplicative effect on
estimated durations, costs of activities it affects• Multiplier < 1.0 ➔ shorter duration, opportunity
• Multiplier > 1.0 ➔ longer duration, threat
•Multiplier is chosen at random from input distribution
(usually 3-point estimate, triangle)
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
INTRODUCING RISK DRIVERS THAT
CAUSE ADDITIONAL VARIATION IN THE SIMULATION
3 of 4 specified risk drivers are enabled; UID 1 (engineering productivity) will happen (P=100%)
Impact may be over or under estimated (0.9, 1.05, 1.3); UID 1 impacts two Design activities
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
100% LIKELY RISK DRIVER’S EFFECT ON 100 DAY DESIGN DURATION
Activity’s duration
looks like a triangle
Min = 90
Max =130
ML = 105
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
RISK DRIVER LIKELIHOOD < 100% (P = 40%)
UID 2: P = 40%, impact (if it occurs) is 0.9, 1.1 and 1.4. No impact
occurs in 60% of the trials. UID 2 applied to both Build activities
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
60% “SPIKE” HAS NO IMPACT (X 1.0)40% OF TRIALS EXPERIENCE RANGED IMPACTS
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
USING RISK DRIVERS METHOD
UID 8 impact distribution:
Triangle: 0.95, 1.05, 1.25
Risk Drivers Probabilities assigned
(UID 8, P 50%)
UID 8 Risk Driver
assigned to Activities
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
DILEMMA: WHAT END DATE AND TOTAL
COST SHOULD BE PUT FORWARD?
• P-80 finish date is 27-Jan-21, risks add 5 months to the project (above uncertainty)
• P-80 cost is $2.27 billion, risks add $150 million to the project (above uncertainty)
➢ Is this enough? Too much? What action should project executives take?
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
TIME – COST SCATTERPLOT CAN ESTIMATE
CONFIDENCE OF MEETING BOTH TARGET OBJECTIVES
• Histograms / cumulative distribution functions estimate
finish date and cost individually
• Combine results with scatterplots
• To meet both targets requires more conservative estimates
(later date, more cost)
• How much more time, cost depends on their correlation
• Joint cost-schedule Confidence Level (JCL)
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
PLAN TO MEET BOTH FINISH DATE AND
COST TARGETS FROM JCL SCATTERPLOT
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
JCL-80 COMPARED WITH P-80 RESULTS
Histogram/Cumulative Distributions (P-80) and Joint Confidence Level* (JCL-80)
Results with Project-Specific Risks and Uncertainty
Baseline
Finish Date 4-April-2020
Budgeted Cost $1.70 Billion
Risk Analysis Results
Schedule Date Months added
P-80 27-Jan-2021 9.8
JCL-80 14-Mar-2021 11.3
Difference 46 cal. days 1.5
Cost Billions Dollars Added (billions)
P-80 $2.27 $0.57
JCL-80 $2.31 $0.61
Difference $0.04 $0.04
*Several combinations of cost and time will satisfy JCL-80; need to select one or a range of those
likely to occur from most concentrated region on scatter plot
ICEAA Canada Workshop, 2018© 2018 Hulett & Associates, LLC
COMPARE WHAT RISK ANALYSIS TYPICALLY
PREDICTS VS. WHAT ACTUALLY HAPPENS
Source: John K. Hollmann, PE, “Reliable Risk Quantification for Project Cost and Schedule”,
AACE International webinar December 15, 2015
ICEAA Canada Workshop, 2018
INCORPORATE SYSTEMIC RISKS
INTO THE MONTE CARLO SIMULATION (MCS)
• Systemic Risks may include
• Technical complexity, new technology challenging
• Scope not fully known
• Process definition not complete
• Megaproject complexity, size / duration, participants
• Project organization, e.g., joint venture, multiple EPCs
•Systemic factors can be measured and impacts on
project success estimated using parametric techniques
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
Projects often are• In tolerances (+ / - 20%)
• Some have serious problems
with overruns approaching 70%*
In our case example:• P-80 ➔28-Oct-21 ≈19 months
• The 80th percentile is approaching
the second mode in the histogram
*Hollmann, John, “Risk Analysis on the Edge of
Chaos,” Cost Engineering (© AACE
International), January/February 2015
COMPLEXITY & PRESSURE COMBINED IN OUR
MODEL PRODUCE A BI-MODAL DISTRIBUTION
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
SCATTERPLOT WITH SYSTEMIC RISKS ADDED
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
MOVING FROM ANALYSIS TO ACTION
•Focus on Action
• Prioritized risks: treatment/response workshops
•Cost effective action (Risk Informed Decisions)
•Conclusion: Treatment → Response → Action
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
ITERATIVE APPROACH TO PRIORITIZING RISK
Purpose, discover which risks contribute the most days at
the P-80 level
1. Compute the Baseline with All Risks In
2. Iteration # 1: Simulate with each risk disabled in turn,
recording the P-80 date
a. The risk with the earliest P-80 date is 1st priority
b. Take it out for Iteration # 2
3. Iteration # 2: Simulate the remaining risks, disabling each
in turn, recording P-80, choose earliest. Take it out for
Iteration # 3 . . . etc.
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
RISKS PRIORITIZED AT THE P-80 CONFIDENCE
LEVEL MEASURED IN “DAYS SAVED”
Systemic Risks are Important
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
RISKS PRIORITIZED TO P-80 AND DAYS SAVED
PLUS EFFECT OF UNCERTAINTYRisks Prioritized to P-80
NameDays
Saved
Megaproject may have excessive schedule pressure 210
The organization has other priority projects so personnel and funding may be unavailable 112
Megaproject may have interdependency problems 52
Fabrication yards may experience lower Productivity than planned 32
Engineering may be complicated by using offshore design firm 18
Megaproject may have coordination problems offshore sourcing 17
Suppliers of installed equipment may be busy 12
Fabrication and installation problems may be revealed during HUC 12
Installation may be delayed due to coordination problems 2
Bids may be Abusive leading to delayed approval 0
The subsea geological conditions may be different than expected 0
Contingency due to Project-Specific and Systemic Risks 467
Contingency due to Uncertainty 150
Total Contingency 617
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
RISK MITIGATION (TREATMENT, RESPONSE)
•Risks can be partially mitigated, perhaps substantially
•Mitigation actions
•New, not known to the interviewees, different from yesterday
•Discussed in Risk mitigation/treatment workshop
•Committed to by management so funded, staffed, monitored
and reported on
•When agreed, estimate cost and timing of mitigation
•Estimate improvement to risk parameters
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
COMPARE PRE AND POST MITIGATION SCHEDULE
Simple mitigation for Schedule improves
from 28-Oct-21 to 24-Jan-21 (≈ 9 months)
but still later than baseline 4-Apr-20
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
COMPARE PRE AND POST MITIGATION COST
Simple mitigation for Cost
• Reduces $2.48 billion to $2.36 billion ≈ $120
million
• Includes $70 million assumed for Mitigation
Costs
• Still $663 million above baseline cost
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
RECAP OF METHOD
• Get good CPM plan/schedule per GAO Best Practices
• Add costs as time-dependent and time-independent resources
• Interview for good Risk Data
• Model uncertainty
• Model project-specific and systemic risks using Risk Drivers
• Use JCL-80 as promise dates and costs
• Prioritize the risks at P-80 and days saved
• Mitigate risks partially, recording mitigation costs
• Commit to the risk mitigations, treatments, responses
• Take Risk Informed Action
© 2018 Hulett & Associates, LLC ICEAA Canada Workshop, 2018
4th Annual WorkshopCanada
Integrated Cost-Schedule Risk AnalysisFocus on Action
KEITH D. HORNBACHER, M.B.A.
SENIOR ASSOCIATE
DAVID T. HULETT, Ph.D.
Hulett & Associates, LLC Los Angeles, CA
+1.310.476.7699 www.projectrisk.com
© 2018 Hulett & Associates, LLC