Socio-economic analyses in perspective: Uncertainties and ... · PDF fileSocio-economic...

Socio-economic analyses in perspective: Uncertainties and bias in decision support

Associate Professor, PhD Kim Bang Salling DTU Transport Traffic days in Aalborg 2012 – Special session: “Uncertainties in Transport Project Evaluation (UNITE)”

DTU Transport, Technical University of Denmark 2

Project Plan of UNITE

Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages

(5) Evaluation methodologyWP5 project leader: Steen Leleur (DMG)

(4) Uncertainty calculation in transport modelsWP4 project leader: Otto Anker Nielsen (TMG)

(2) Organizational context of Modelling, an empirical study

WP2 project leader: Petter Næss (AAU)

(3) Uncertainty calculation of cost estimates

WP3 project leader: Bo Friis Nielsen (DTU Informatics)

(1) Systematic biases in transport models (recognized ignorance), an empirical studyWP1 project leader: Petter Næss (AAU)

DTU Transport, Technical University of Denmark 3

How do we evaluate transport projects?

• Various existing guideline report: –Denmark, Sweden, UK, European Union, ....

• Socio-economic analysis by the use of Cost-Benefit

Analysis (CBA)

• Produces single point estimates such as Net Present Values (NPV), Benefit Cost Ratios (BCR), etc

• However, no common rule have been set in order to acommodate the uncertainties in CBA!

–Recent conducted PhD dissertation proved this point

DTU Transport, Technical University of Denmark 4

Background & Motivation

• The Manual for socio-economic analysis in the transport sector (2003)

–Unique guidelines for evaluating transport infrastructure projects

–Lack of uncertainty handling –Expected revision 2012-2013

DTU Transport, Technical University of Denmark 5

How do we evaluate transport projects?

• However, no common rule have been set in order to acommodate the uncertainties in CBA!

–Recent conducted PhD dissertation proved this point

DTU Transport, Technical University of Denmark 6

The Case Study: HH-Connection • Connecting Denmark with Sweden: Scandinavian link

–Currently, close to the capacity limit on Oresund

HH-Connection (alternatives*)

Description (Alignment of connection)

Cost (million DKK)

Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700

Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500

Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500

Alternative 4 Bridge for road (2x2 lanes) 6,000

* Larsen, L.A. & Skougaard, B.Z. (2010). Vurdering af alternativer for en fast forbindelse Helsingør-Helsingborg, M.Sc. thesis, Department of Transport, Technical University of Denmark (in Danish)

DTU Transport, Technical University of Denmark 7

The UNITE-DSS Modelling Framework Todays Outline

DTU Transport, Technical University of Denmark 8

Results: Cost-Benefit Analysis

• Construction costs – by far the largest contributor of costs

• User Benefits – by far the largest contributor of benefits – Consists of Ticket revenue and time savings – Relies on the prognosis of future number of passengers i.e.

demand forecasts

HH-Connection (alternatives)

Cost (million DKK)

BCR NPV (million DKK)

Alternative 1 7,700 1.50 5,530

Alternative 2 5,500 0.16 -6,640

Alternative 3 11,500 2.71 28,240

Alternative 4 6,000 3.08 17,860

DTU Transport, Technical University of Denmark 9

Are we telling the truth?!?!

Construction cost overruns

0%

200%

400%

600%

800%

1000%

1200%

1400%

1600%

1800%

2000%

Suez

Can

al

Sydn

eyO

pera

Hou

se

Con

cord

eSu

pers

onic

Aero

plan

eBo

ston

'sAr

tery

/Tun

nel

Proj

ect,

USA

Hum

ber

Brid

ge, U

K

Bost

on-

Was

hing

ton-

New

Yor

kG

reat

Bel

tR

ail T

unne

l,D

KA6

Mot

orw

ayC

hape

l-en-

le-

Frith

/Wha

ley

Shin

kans

enJo

etsu

Rai

llin

e, J

apan

Was

hing

ton

met

ro, U

SA

Cha

nnel

Tunn

el, U

K &

Fran

ceKa

rlsru

he-

Bret

ten

light

rail,

Ger

man

yØ

resu

ndAc

cess

link

s,D

K &

Swed

enM

exic

o ci

tym

etro

line

,M

exic

oPa

ris-A

uber

-N

ante

rre

rail

line,

Fra

nce

Cos

t Ove

rrun

s (%

)

Q: Have we learned anything from history?

”Chunnel” in 1987 £2,600 million (’85 prices) Completion 1994 £4,650 million (’85 prices) Total cost overrun of approx. 80%

”Øresund access link” in 1991 3.2 billion DKK (’90 prices) Completion 1998 5.4 billion DKK (’90 prices) Total cost overrun of approx. 68%

DTU Transport, Technical University of Denmark 10

Theoretical anchoring The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004)

Reference Class Forecasting: Optimism Bias

Inside View Outside View

”Uniqueness” of Project

”The Planning Fallacy”

Reference Class Forecasting

Forecasting of particular projects

Forecasting from a group of projects

(1) Identification of relevant reference

classes

(2) Establishing probability distribution

(3) Placing and comparing the

project

Optimism Bias UpliftsCurrent Situation

DTU Transport, Technical University of Denmark 11

Optimism Bias and uplifts

• Deriving uplifts is highly dependet on large data-sets –Flyvbjerg from (AAU) has since 2003 developed a large

database –Unfortunately, it looks upon mega-projects

• The basis is Reference Class Forecasting i.e. statistical measurements on various project pools

Source: Flyvbjerg and COWI (2004)

DTU Transport, Technical University of Denmark 12

Results : Optimism Bias Uplifts

• The BCR are lower, however, still point estimates towards DM –Moreover an advanced form of sensitivity analysis

• Imply to introduce risk analysis and Monte Carlo simulation

HH-Connection (alternatives)

Cost (uplifted) (million DKK)

BCR (orig.) (from slide 8)

BCR (uplifts): 80% uplift

Alternative 1 12,090 1.50 0.97

Alternative 2 8,640 0.16 0.10

Alternative 3 15,180 2.71 1.75

Alternative 4 7,920 3.08 1.98

DTU Transport, Technical University of Denmark 13

The UNITE Project Database (UPD)

• The convention used is as follows: ( )( )

forecasted

forecastedactual

XXX

U100×−

=

Over estimation of Demand

DTU Transport, Technical University of Denmark 14

• Demand forecasts (user benefits) are derived: – U is percent inaccuracy, – Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build

• Combination of two database samples

0

5

10

15

20

25

30

(-12

0;-1

00)

(-10

0;-8

0)

(-80

;-60)

(-60

;-40)

(-40

;-20)

(-20

;0)

(0;2

0)

(20;

40)

(40;

60)

(60;

80)

(80;

100)

(100

;120

)

(120

;140

)

(140

;160

)

(160

;180

)

(180

;200

)

(200

;220

)

(220

;240

)

Freq

uenc

y of

occ

uren

ce (%

)

Inaccuracies in demand forecasts (%)

Inaccuracies in demand forecasts (road projects)

Salling et al. (2012)

Flyvbjerg et al. (2003)

Nicolaisen et al. (2012)

DTU Transport, Technical University of Denmark 15

The UNITE Project Database (UPD)

• The convention used is as follows: ( )( )

forecasted

forecastedactual

XXX

U100×−

=

Under estimation of costs

DTU Transport, Technical University of Denmark 16

• Construction costs bias derived similarly: – U is percent inaccuracy, – Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build

• Combination of two database samples

0

10

20

30

40

50

(-10

0;-8

0)

(-80

;-60)

(-60

;-40)

(-40

;-20)

(-20

;0)

(0;2

0)

(20;

40)

(40;

60)

(60;

80)

(80;

100)

(100

;120

)

(120

;140

)

(140

;160

)

(160

;180

)

(180

;200

)

(200

;220

)

(220

;240

)

Freq

uenc

y of

occ

uren

ce (%

)

Inaccuracies in construction costs (%)

Inaccuracies in construction cost (road projects)

Salling et al (2012)

Flyvbjerg et al. (2003)

Nicolaisen et al. (2012)

DTU Transport, Technical University of Denmark 17

Results (RCF): Monte Carlo simulation

DTU Transport, Technical University of Denmark 18

Conclusions

• Feasibility risk assessment can be carried out by using historical experience stemming from RCF in order to obtain interval results

• An important aspect in RCF and UNITE is to set and validate input parameters. Hence, empirical data enter the assessment.

• The RCF approach has been illustrated on a case example concerning the construction of a new fixed link, the HH-Connection, between Denmark and Sweden.

• Clearly vital to include uncertainties within socio-economic analyses in order to validate results

DTU Transport, Technical University of Denmark 19

Perspectives

• Recovering of further data (UPD) with regard to both the demand forecast uncertainty as well as the construction costs through large-scale research study

• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)

• More info on UNITE can be found: (www.transport.dtu.dk/unite)

• An international conference on the topic is scheduled in September 2013 – a specific call will be posted in the upcoming month.

DTU Transport, Technical University of Denmark

SIMSIGHT: Decision Conferencing (DC)

20

• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model

• Enables to include Stakeholders and Decision-makers in an early stage, i.e. to include experts opinion on MIN and MAX values as entries to the Monte Carlo simulation

DTU Transport, Technical University of Denmark 21

Results from DC and RSF

DTU Transport, Technical University of Denmark

SIMSIGHT: Overconfidence

22

DTU Transport, Technical University of Denmark 23

Perspectives

• Recovering of further data (UPD) with regard to both the demand forecast uncertainty as well as the construction costs through large-scale research study

• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)

• More info on UNITE can be found: (www.transport.dtu.dk/unite)

• An international conference on the topic is scheduled in September 2013 – a specific call will be posted in the upcoming month.

DTU Transport, Technical University of Denmark 24

Thank you for your attention!

Affiliation:

Associate Professor, PhD Kim Bang Salling

Department of Transport Technical University of Denmark

kbs@transport.dtu.dk