INDEPENDENT QUALITY REVIEW

(JASPERS IQR)

CUPT Conference on Traffic Modelling

Warsaw, 25 May 2017

2

I. Role of IQR

II. Scope of IQR assessment

III. Demand modelling and economic analysis

IV. Issues noted in appraisal of Polish projects

V. Conclusions

Contents

I. Role of Independent Quality Review

3

JASPERS in 2014-2020

European

Investment Bank

JASPERS

JASPERS

Advisory

JASPERS

Networking

Competence Centre

(NCC)

JASPERS

Independent Quality

Review

(IQR)

I. Role of Independent Quality Review (4)

4

In both routes:

Previous involvement of JASPERS Advisory is beneficial

for the quality of major project application

Identical scope of assessment for demand and economic

analysis for both routes (IQR and PSA)!

When is IQR used?

MANAGING

AUTHORITY

EUROPEAN

COMMISSION

Project Notification

JASPERS IQR

Post Submission

Appraisal (PSA)

I. Role of Independent Quality Review

How many projects have already been assessed by JASPERS IQR

(as of 03/05/2017)?

7 IQR reports issued so far, 6 of which were in transport sector, 2 of

which in turn were from Polish transport sector (1 road, 1 rail)

41 projects assessed in form of a post-submission appraisal,

including, 35 in transport sector, of which 30 from Polish transport

sector (26 roads, 4 rail/urban).

5

II. Scope of IQR assessment

Assessment by IQR team will cover virtually all aspects of the project,

including e.g.,

Technical, legal, financial and administrative capacity

Total eligible and non-eligible costs

Verification of state aid (for IQR projects only)

Option, demand, financial, economic and risk analyses

Climate change and environmental assessment (for IQR projects

only)

Appropriate measures to ensure optimal utilisation of infrastructure

Financing plan

Timetable

6

II. Scope of IQR assessment

With regard specifically to the assessment of the cost-benefit

analysis undertaken by IQR, this includes:

1. Verification of all the criteria for quality review of major projects

(Annex II to Regulation (EU) No 480/2014)

2. Verification of all the requirements of CBA Methodology

(Annex III to Regulation (EU) No 2015/207)

3. Compliance with EC Guide to Cost-Benefit Analysis of Investment

Projects

7

II. Scope of IQR assessment

1. Criteria for quality review of major projects outlined in Annex II to

Commission Delegated Regulation (EU) No 480/2014:

“…Reliability of the demand analysis … based on realistic estimates

and in line with main demographic trends and developments in the

respective sector, justifying the need for the project and the overall

capacity of the project facilities…

…Soundness of the conclusion that the project is economically and

financially viable and has positive socioeconomic effects justifying

the level of support to the extent envisaged under the ERDF or the

Cohesion Fund…”

8

II. Scope of IQR assessment

2. Methodology for carrying out the cost-benefit analysis outlined in

Annex III to Commission Implementing Regulation (EU) No 2015/207:

“…The objective of a cost benefit analysis (hereinafter referred to

as ‘CBA’) in the context of Cohesion Policy is to support the major

project assessment in order to: — assess whether the major project is

worth co-financing (from an economic point of view); — assess

whether the major project needs co-financing (from a financial point

of view)…

…A CBA for a major project shall include the elements set out

below: (1) Presentation of the context; (2) Definition of objectives; (3)

Identification of the project; (4) Results of feasibility studies with

demand and option analysis; (5) Financial analysis; (6) Economic

analysis; (7) Risk assessment…”

9

II. Scope of IQR assessment

2. Methodology for carrying out the cost-benefit analysis outlined in

Annex III to Commission Implementing Regulation (EU) No 2015/207

(cntd):

“…Demand analysis identifies and quantifies the social need for an

investment and considers as a minimum: — the current demand, by

the use of models and actual data; — the forecast demand, from

macroeconomic and sector forecasts and elasticity estimates of demand

to relevant prices, income, and other core determinants; — supply side

aspects including the analysis of existing supply and expected

(infrastructure) developments; — network effect (if any)…

Economic analysis is an analysis that is undertaken using economic

values, reflecting the social opportunity cost of goods and

services…The economic analysis should be carried out in constant

accounting (shadow) prices and shall be undertaken taking the financial

analysis cash flows as a starting point...”

10

II. Scope of IQR assessment

3. Guide to Cost –Benefit Analysis of Investment Projects:

“Demand Analysis. Demand analysis identifies the need for an investment by

assessing: current demand…; future demand (based on reliable demand

forecasting models that take into consideration macro- and socio-economic

forecasts, alternative sources of supply, elasticity of demand to relevant prices and

income, etc.) in both the scenarios with- and without-the-project.”

CBA Guide also includes:

(i) detailed principles underlying demand, economic, financial and risk analyses

(chapter 2)

(ii) discussion of sector specific issues (chapter 3 for transport)

(iii) worked case studies per sector (chapter 3, for road, railway and urban

transport)

11

Inputs Outputs

1. Costs (inv, O&M) -> Economic Analysis -> NPV

2. Parameter values -> Economic Analysis -> ERR

3. Demand -> Economic Analysis -> BCR

III. Demand modelling and economic analysis

12

III. Demand modelling and economic analysis

There are many different (valid) ways to calculate benefits for

transport projects, including:…

UK “COBA” approach – calculate link-by-link (for each link for each

year, time period, vehicle category…) benefits from time savings,

accidents, emissions, vehicle operating costs.

UK “TUBA” approach – calculate based on demand model skim

matrices (for each O-D pair for each year, time period…) benefits

from time savings and perceived vehicle operating costs. Need to

calculate external benefits (e.g., accidents) separately.

“Network indicators” approach – calculate all benefits based on

network indicators (e.g., passenger hours, vehicle kilometers)

derived from traffic model.

Polish approach seen so far more closely follows “network

indicators”. With each approach there are plusses and minuses….

13

IV. Issues noted in appraisal of Polish transport projects

Pros and cons of “Network indicators” approach…

Pros

Simple approach which is reasonably easy to implement

Re-use of standard spreadsheet (only need to check calculus

once)

Cons

Very reliant on traffic model outputs

Volatility of pattern of economic benefits if model outputs not

100% correct

How do you verify validity of calculated benefits?

14

IV. Issues noted in appraisal of Polish transport projects

Volatility: anonymised examples for recent road projects…

15

-100 000 000

-50 000 000

0

50 000 000

100 000 000

150 000 000

200 000 000

250 000 000

2019 2021 2023 2025 2027 2029 2031 2033 2035 2037 2039

Profile of undiscounted benefits

vehicle operating costs time savings accidents environment climate change noise

-150 000 000

-100 000 000

-50 000 000

0

50 000 000

100 000 000

150 000 000

200 000 000

250 000 000

300 000 000

20

18

20

19

20

20

20

21

20

22

20

23

20

24

20

25

20

26

20

27

20

28

20

29

20

30

20

31

20

32

20

33

20

34

20

35

20

36

20

37

20

38

20

39

Profile of undiscounted benefits

vehicle operating costs time savings

accidents environment

climate noise

-50000

0

50000

100000

150000

200000

250000

300000

Change in passenger Hours

Cars Light vans Heavy (w/out trailer) Heavy (with trailer) Bus

-500 000

-400 000

-300 000

-200 000

-100 000

0

100 000

200 000

300 000

400 000

500 000

20

15

20

17

20

19

20

21

20

23

20

25

20

27

20

29

20

31

20

33

20

35

20

37

20

39

20

41

20

43

20

45

Change in PKM per day

Cars Light vans Heavy (w/o trailer) Heavy (with trailer) Bus

IV. Issues noted in appraisal of Polish transport projects

Checking CBA outputs when using “Network indicators”

approach…

Q1. Are calculated benefits stable? Are there any years for which

there is a sharp increase / fall in benefits? If so, why? Do passenger

hours and kms follow a sensible stable trend?

Q2. Does the overall level of benefits in year one make sense? If

you calculated e.g., savings in passenger hours per vehicle, does

the level make sense considering the length of the project and

speeds involved?

Q3. Are estimated speeds (WP, WOP) reasonable?

Q4. Does the overall magnitude of each calculated benefits

makes sense given the nature of the project? E.G., if we were

replacing one single carriageway (1x1 road) with another we would

probably not expect to see much accident benefits.

16

IV. Issues noted in appraisal of Polish transport projects

Some issues noted on recent projects…

• High ERRs. For several road projects, very high ERRs (>>20/30%)

noted. Unusual for a transport project and ‘flags up’ potential issues

with CBA. IQR has had to recalculate ERRs. Better to be

conservative? […no prizes awarded for highest ERR on a project!]

• Low speeds on existing road (to be replaced) in without project

scenario. In some cases speeds of 50-55km/hour on relatively

uncongested rural roads (in seemingly good condition)

• High growth rates or transfer of traffic predicted by model (again,

better to be conservative?).

• Additional detail on model build, calibration and validation.

17

IV. Issues noted in appraisal of Polish transport projects

Other infrequent issues…

Roads sector… justification of cross section (i.e., carriageway

type). Roads where modelled traffic volume either too low or too high

for provided capacity.

Benefits being dependent on other projects on corridor which

may / may not happen.

Incompleteness of project (“half a bypass”) with remaining

sections to be built at a much later date (and therefore with

overcapacity in the intervening period).

Multimodal issues. Difference in approach to competing modes (for

roads, rail excluded; for rail, road included and transfer traffic high).

Need for consistency between modes along same corridor.

18

V. Conclusions

Polish transport sector and roads in particular viewed as the success of the

current programming period.

For road projects some issues noted with stability of demand model

outputs but we understand that following interaction with JASPERS

Advisory colleagues that these are being sorted and for future projects they

should be addressed.

Speed surveys to verify assumed speed on existing roads? [journey time

surveys for rail / PT].

(For roads sector) consider cross section in options analyses?

Some basic checking of demand model outputs / CBA results might be of

use (e.g. hours saved per passenger, stability of profile over time)?....

To minimise delays JASPERS Advisory support could be sought for

difficult projects.

19

Thank you!

For info or further questions on this presentation, please contact:

Fergal Trace

JASPERS IQR

Tel.: +32 2 712 41 84

f.trace@eib.org

www.jaspersnetwork.org

jaspersnetwork@eib.org

20

20