Using an outside view of project risk for

realistic risk and capability assessments

Alexander Budzier, DPhil

Fellow in Management Practice

Saïd Business School, University of Oxford

Our Research Insights Helped Us to Understand the

Root Causes of Issues and Develop Long-Lasting Solutions

3

Symptoms

Root

causes

Issue Fix

Cost overruns,

schedule delays,

benefit shortfalls

Fixed price contact,

risk sharing, sticks &

carrots

Causes Unknown geology,

scope changes etc.

Reactive risk

management

• Optimism and

political bias

• Culture, mindsets

and behaviors

Predict & provide

Predict & prevent

▪ Long-lasting solutions need to

understand and address real

causes and their root causes

▪ Quick fixes only offer limited help

2 Strategies to De-Risk Projects

4

Accurate

planning

Outside view to

▪ Accurately estimate risk

(inside view leads to

optimism bias)

▪ Reference Class

Forecasting as one method

to systematically take the

outside view

Outside view to

▪ Identify potential

weaknesses in projects that

actually matter

▪ Identify early warning signs

and leading risk indicators

Challenged

capabilities

Inaccurate

planning

Improved

capabilities

Predict & provide

Predict &

prevent

The 3 Steps of RCF

1. Identify relevant reference class of past, similar

projects

2. Establish probability distribution for the selected

reference class

3. Compare specific project with distribution, in order to

establish most likely outcome

5© Bent Flyvbjerg

* The p-value of statistical tests indicates the strength of the evidence, if p < 0.05 the test is significant – here indicating that there is strong statistical evidence that these project types are different from HSR

Source: Oxford Database, August 2015 (Sample of n=361 projects)6

Statistical analysis of risk profile• Risk profile characterized by distribution of cost risk in reference classes

• Peak = P50 risk• Tail = Risks > P50

Building a Reference Class

7

• Bottom-up QRA estimates the

35% contingency to be a P90

• P90 = 9 out of 10 projects would

not exceed this envelope

• RCF shows that 35% contingency

is equivalent to P66

Building a Reference Class for HS2

8

• HS2: 35% contingency = P66

• P90 (conservative estimate) = 120%

• P95 (very conservative estimate) = 152%

Building a Reference Class for HS2

Comparing the Inside View OBC with

the Outside View RCF Estimate

9

2

1

Regress most likely estimate

toward median of reference class1

Expand estimate of tail risk to

to tail of the reference class 2

Increased Design Certainty Should (in Theory)

Reduce Cost Uncertainty

10

Project cycle

SBC OBC FBC Tender Construction Completion

Final cost

Increased design

certainty

Reduced cost

uncertainty

Different Reference Classes for Different

Points of the Estimate

11

Strategic Business Case

Outline Business Case

Full Business Case

Contract Control Totals(total value of contracts)

-10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

140%

20 25 30 35 40 45 50 55 60 65 70 75 80

Co

st R

isk

Level of Certainty of the Estimate (P Value)

• 35% contingency covers cost risk

exposure of

• SBC: P63

• OBC/FBC: P66

• Contract control totals: P77

35%

Key Concern is the Tail Risk in the

Reference Class Forecast

-70%

-60%

-50%

-40%

-30%

-20%

-10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

110%

120%

130%

140%

150%

0 10 20 30 40 50 60 70 80 90 100

Con

ting

en

cy n

eed

ed

P-value

12

35%

• Key concern is tail risk, when

risks start to grow

exponentially with increasing

uncertainty

• Tail of the RCF begins at P66

• 1 in 3 projects in reference

class had cost overruns >

35%

Example of HSR Project in the Tail

• NSB Nuremberg-Ingolstadt, Germany

• First proposed 1973, planning started 1991, decision to build 1994

• Construction start 2000

• Opening 2006

• Estimated cost: 2.3 bn EUR (YoE); estimated opening 2003

• Length 77.4 km incl. 15 km in tunnels, 82 viaducts

• Cost overrun of +133%

• Reasons for cost overrun• Geological problems in tunneling sections; hydrology problems along the alignment

• Changed security concept after the Eschede accident in 1998

• Increased environmental impact mitigation cost

• Underground archeological remains

• Unanticipated inflation

• Late design submission of pre-fab track bed

• Cost savings: not upgrading several station and using instead of upgrading existing track,

change of tunneling method

• Cost for interest payments

• Issues with integrating the HSR line into operational railway meant that travel time

savings were only achieved for end-to-end journeys, no clear concepts for tact

times, critique of abandoning same platform connections

• 2010 travel speed reduced from 300 km/h to 160 km/h due to quality issues with

the rubber mats in the track bed

13

Characteristics of Projects Exceeding

Contingency

14

Area Description HS2 Response

Design Late design changes due to external

demands

Hybrid Bill process controls scope

Funding Problems with funding causing

changes to profile. Including

excessive interest payments.

Agreed funding profile up front and

funding mechanisms including

necessary insurance strategy.

Amplification Problems on critical path escalate

rapidly without ‘damping’

Designing schedule to create higher

confidence at critical points eg end of

pre-construction. Early contractor

involvement in design

Integration Failing to appreciate and understand

scale of integration required to deliver

Structure of contracts. Assurance

approach. Technical specification and

assurance

Quality Problems with quality of work

completed

Assurance design. Hand over points

control.

Archaeology Significant volumes of archaeology

and geology issues

An allowance made and base is case

is towards the worst case

Inflation Inflation greater than expected Specific recognition and approach to

this issue.

Front-End Capability Maturity Model

15

5. Master BuilderLeadership

3. Reference Class Forecasting

1. Benchmarking

2. Due Diligence

4. Black Swan Management

▪ Projects planned with an inside view only

▪ Optimism bottom-up in estimates unchecked

▪ No internal capability to challenge cost and schedule forecast of contractors

▪ No capability to identify ‘low balled’ bids

▪ No understanding of the uncertainty of estimates

▪ Wide range of capabilities with few pockets of excellence

▪ Careful balance between individual and organisational capabilities

▪ High-powered experts lack understanding of way of working

▪ Lack of empowerment of experts (stifled by process)

▪ No culture of excellence but a culture of heroes

▪ No systematic approach to taking the outside view

▪ Variability between projects’ and sub-projects’ ability to reduce bias

▪ Ignorance of unknown-unknowns

▪ Incentives to de-risk projects and safeguard contingencies not aligned in supply chains

▪ Slow management information leads to predictable surprises

▪ Biased and narrow management reporting deaf to weak signals

▪ Overly complex projects (social, political and time complexity) with management of symptoms not causes (ie sources of uncertainty and complexity)

▪ Tight coupling and interactive complexity make project fragile

▪ Is the project conducting internal and external benchmarks?

▪ What level of realism does this show?

▪ Have lessons learned been incorporated?

▪ How is the procurement strategy and early contractor involvement structured?

▪ What are the incentives for forecasters?

▪ Is probabilistic forecasting used consistently?

▪ Has the project taken a systematic view to compare itself to other projects?

▪ How mature is the risk management (if any) and other PM disciplines to manage the front-end process?

▪ Has the project analysed its sources of complexity?

▪ Have complexities been actively managed or mitigated?

▪ How compressed is the schedule?

▪ How confident is the project in the capability of its leaders?

▪ How mature are the PM processes?

▪ Do the project and supply chain share understanding of success factors?

Risk Maturity Level

16

Maturity

Level

Area Problem factors Potential response (examples)

1 Bench

marking

Inside view only with optimism bias unchecked

Programme of learning from others.Benchmarking against other projects, industries and sectors

2 Due

diligence

No challenge to cost, schedule forecasts of contractors. Minimal understanding of uncertainty in estimate

Internal assessments of costs, schedules.Separate estimating from contractors

3 Reference

Class

Forecasting

No systematic outside view. Variability between projects and subprojects ability to align and reduce bias. Ignorance of ‘unknown unknowns’; lack of exploration.

Systematic and ongoing Reference Class Forecasting across different aspects of programme.Exploration of scenarios to understand ‘unknown unknowns’.Understand characteristics of tail

4 Black Swan

(predictable

surprises)

Slow management information communication. Management of symptoms not causes. Tight coupled system. Overly complex

Quick, clean, independent management information reportingAmplification of weak signals.Recognise complex scenarios and break down

5 Leadership High powered experts lack understanding of how to work effectively. No culture of excellence rather one of heroes. Lack of empowerment of experts

Identify necessary pockets of excellence and encourage culture. Continuous learning.Focus on ways of workingLeadership engage in scenario planningIncrease reliance on ‘qualitative’(needs more development)

Summary: Risk Management is about

Building Confidence not Certainty

17

Accurate

planning

Outside view taken to

▪ Establish de-biased view of

risk

▪ Engage with funder to

move away from P95

perceptions

Outside view taken to

▪ Learn systematically and

quickly from others

▪ Work on project capabilities

through maturity framework

thinking

▪ Stress importance of

improving planning and not

only construction phase

Challenged

capabilities

Inaccurate

planning

Improved

capabilities

Predict & provide

Predict &

prevent

Any Questions?

Alexander.Budzier@sbs.ox.ac.uk