Driving better Project Evaluation in Queensland - the role of the Revised

Project Evaluation Guidelines

Presentation to the Australasian Evaluation Societyby Scott Hook and Mark Rice

May 2006

What is today about?• Understand the reasons for having the Project

Evaluation Guidelines (PEG).• The PEG framework and key concepts.• Importance of ex-ante and ex-post evaluation.• Assess the changes in the guidelines since the

previous 1997 version.• Using the PEG in your analysis.• Key lessons.

Objective – section 1• Key concepts.• Rationale for evaluation at the beginning

of a project.• Why it is important to use the Project

Evaluation Guidelines.• When to use the PEG?

Key Concepts – what to understand

• Cost-benefit analysis• Cost-effectiveness analysis• Discount rates• Externality• Multiplier• Opportunity cost• Shadow pricing

Mistakes happen!

However, planning can help!

Things can go wrong!• Projects can become more expensive, take

longer to complete and do not achieve the initial objective.

• Consider– Victoria’s Federation square– Commonwealth’s Seasprite acquisition

• UK government experience with planning for infrastructure projects suggests that:– Timing for non-standard engineering projects be

revised in the range of 3-25 per cent;– Costs be revised in the range of 6-66 per cent.

Fail to plan, plan to fail• Consistent, transparent and accurate

information is needed to– Align agencies’ policies, projects, programs

and activities to the Government’s priorities;– Prioritise individual projects within programs;– Ensure that their project procurement and

resource allocation decisions achieve maximum value and benefit for the community.

What are the Project Evaluation Guidelines?• The development of the PEG reflected a

concern that proposals for new projects, services or policy require guidelines.

• The PEG provides the rationale, process and requirements for project evaluation in Queensland Government Agencies.

• There is also an increasing need to use cost-benefit analysis in the evaluation process.

PEG Background• Previous Project Evaluation Guidelines –

February 1997.• Requirement for more detailed guidelines,

with inclusion of templates. Inclusion of additional material on risk management, governance arrangements and project management.

• The revised Project Evaluation Guidelines have been circulated to all agencies for comment and use.

PEG background (cont)• Charter of Social and Fiscal

Responsibility (April 2004) requires– The direction of resources and activity

towards the Government’s highest priority areas; and

– The provision of strategies to achieve value for money in delivering infrastructure and services to the community.

• PEG seeks to assist rigorous and robust analysis across Queensland Government Agencies.

Why do we need the PEG?• To provide a standard methodology and whole-

of-Government approach to the preparation of project evaluations and the analyses they contain.

• Increased recognition of the need to improve rigour and clarity for decision making purposes by increasing the use of cost-benefit analysis.

• Establishment of a whole-of-Government framework to clarify uses and linkages between other guidelines, such as PPP.

Part of Good Governance

• Thinking about options and then ranking them can improve the quality of Government decision making.

• Cabinet seeks the maximum value benefit for the whole community.

What PEG is not• A one-stop project management tool. • A method to

– micro-manage all projects;– slow agency new policy development; and– only evaluate physical infrastructure.

• Instead, PEG is– Part of the policy development toolkit for

agencies. – PEG can used for infrastructure, service delivery

and new policy initiatives.

Ex-what?• Traditional evaluation has been

conducted at the end of a project.• Increasingly, risks of a project in terms of

time, cost and quality require evaluation upfront.

• PEG and business cases are an example of ex-ante evaluation.

• Ex-post evaluation still has an important role.

Changes in 2005 PEG• 2005 PEG applies to all projects (involving

capital or recurrent expenditure or regulation), rather than capital projects.

• Increased detail on how to undertake each of the analyses.

• Social and environmental analyses integrated into CBA.

• Budget analysis on accrual basis.• New section on regulatory analysis.• Stronger statement discouraging use of

multipliers.

Objectives – Part 2• PEG contents.• Project Evaluation framework.• Understanding the cost-benefit analysis

methodology.• Role of agencies.

PEG Contents• Introduction.• Framework for Evaluating Projects.• Project Evaluation Pro forma.• Common Project Evaluation Techniques.

What is Cost-benefit analysis?• Identifies values and compares the costs

and benefits derived from investing scarce resources, such as land, labour and capital, in project options to achieve a particular objective. Costs and benefits are valued in dollar terms and adjusted for market distortions or imputed where the market does not exist.

Project Life Cycle

PEG applicationPEG application

Set objective

Generate options

Evaluate Options

Select preferred option

Project ImplementationProject Implementation

Building the case for a preferred option

Transitioning to the preferred option

Framework for Evaluating ProjectsFive Steps• Objective, priority and justification of project.• Project options identified.• Preliminary assessment.• Evaluation of project options (explained further

shortly).• Selection of preferred option.

Evaluation of Project Options

• For each of the project options, agencies prepare a:– Risk Analysis– Financial Analysis– Cost Benefit or Cost Effectiveness Analysis– Budget Analysis– Regulatory Analysis (where appropriate)

Risk Analysis• Risk Identification

– Determining what, why, where, when and how events could prevent, degrade, delay or enhance the project.

• Risk Assessment– Sources of project risk, their positive and negative consequences and

probabilities that they will occur.

• Allocation and Mitigation– Can it be prevented and/or what to do with it once it has materialised.

Financial Analysis

• Measures costs and revenues for the sponsoring agency or agencies.

• Involves cash expenditure and revenue, including capital.

• Measures the ability of the agency to cover expenses of the project option through a revenue stream.

Cost Benefit Analysis• Adopts economy wide perspective (usually for the

State as a whole). A financial analysis is the starting point.

• Includes a wider range of social and environmental costs and benefits than the financial and budget analyses, eg pollution, land degradation, traffic congestion, health outcomes, etc.

• Measures true economic worth of resources. • Project option with the greatest net present value –

most economic use of resources.

Identify and estimate the expected economic benefits and costs

We are measuring ‘social’ costs and benefits• Not calculating the distribution of those costs

and benefits.• Valued in dollar terms at their ‘true’ economic

worth.• Not confined to market transactions

– Non-market costs and benefits can be valued by imputation or by appropriate adjustment of the market price.

Calculate the net present economic value• Discounting allows benefits and costs in different

options occurring at different times to be compared on the same basis, and recognises that a dollar now is valued more highly than a dollar in the future.

• The difference between the discounted streams of benefits and costs of each project option is the Net Present Economic Value of the project option.

• A positive or negative value helps determine the economic viability of a project.

• Choice of the discount rate will be an important determinant.

Expected economic benefits (continued)

Consideration should be given to estimating:• Avoided costs;• Cost savings;• Revenues;• Benefits to consumers and to the community

as a whole; and• Residual value of assets.

Key Points – the NPV analysis• Capital expenditure estimates are

reasonable and accurate• All legitimate costs and benefits have been

included. • No invalid costs and benefits. • All costs and benefits have been valued at

their market value or economic value.• A reasonable timeframe for the project and

costs and benefits have been forecast reasonably and with transparency.

Budget Analysis• Covers the budget impact for the relevant

agency or agencies.• Involves accrual expenses and revenue

and includes non cash items, such as depreciation.

• Measures impact on Operating Statement and Statement of Financial Position.

• Measures impact on the State Government’s fiscal position.

Regulatory Analysis

• Projects are assessed to ensure they:– Do not restrict competition under the TPA;– Produce a net benefit for State where

regulations are changed; and– A Regulatory Impact Statement and /or

Public Benefit Test may be required for changes to legislation and/or subordinate legislation.

Don’t forget we need to…• Discount the value of money to account for

different flows occurring over different time periods.

• Use sensitivity analysis for the Financial Analysis and the Cost Benefit Analysis (eg, changes in prices, construction costs, usage of services or discount rates).

• Include project implementation and post-implementation review.

Garbage in – Garbage out

The analysis should contain• Textually and numerically explicit

explanations of the assumptions underlying all capital and recurrent estimates regarding labour costs, energy costs, demand growth, charges etc.

• Clear and referenced data sources for validation purposes.

When to use the PEG• The PEG should guide the assessment of

any project under consideration by an agency. Can be used for infrastructure, service delivery and new policy initiatives

• However, judgement is needed. A detailed financial and economic analysis may be impractical for small projects or expensive and complex.

• Assessment should be made according to scope, cost, complexity, riskiness and sensitivity of the project.

What are some alternatives• Economic impact analysis, which estimates flow-on economic

impacts of a project, can supplement cost-benefit analysis.– Input-output modelling measures a gross flow-on from project

expenditure (a multiplier). Applying input-output multipliers usually overstates project impacts.

• General equilibrium modelling uses a mathematical model of the economy to estimate the whole of economy impact of changes in policies or new projects.

• Why are they different to CBA?– Focuses on economic impacts (not valuing social impacts)– Includes flow-on impacts to all sectors (possibility of

overestimating benefits).

What is the role of Agencies?• The agency must make a decision about the

level of analysis and the allocation of suitable resources.

• Identify and quantify expenses, costs, revenues and benefits and project affordability.

• Undertake a rigorous cost-benefit analysis.• Liaise with the relevant business branch and

OESR for guidance on specific technical issues.

What is Treasury’s role?

• Provide training, advice and guidance to an agency to support the preparation of a cost-benefit analysis.

• Advise the Treasurer and CBRC on the information quality of project evaluations and the merits of output/investment proposals put forward by Government departments and agencies.

• Establishment of a data base that agencies can access when preparing project evaluations

The process• Several stages

– Consultation on the PEG.– Refinement and finalisation of the PEG.– Integrate PEG within the Project

Assurance Framework.

Relationship with other frameworks• Other guidelines that are related include the

Public-Private Partnership Policy Guidelines, Public Benefits Test Guidelines and Regulatory Impact Statement Guidelines.

• A Project Assurance Framework is being developed by FMB– The PAF will include guidance on a gateway process to

ensure success management of project processes.

• The PEG is being integrated into the PAF.

25 October 2005

Business Case Development

Supply Strategy Development

Source Supplier/s

Establish Service Capability

Deliver ServiceManagement of Project Agreements

Preliminary Evaluation

Strategic Assessment of Service Requirement

PPP Business Case Development

Expressions of Interest

Bidding Process

Progress as PPPProgress as traditional delivery

Key Learnings• That any project evaluation should be using the

PEG as a template and all data and assumptions are documented, and costs and benefits are explained.

• Cost-benefit analysis allows choices to be recommended to Government.

• Include the status quo as an option but exclude estimates of costs and benefits do not follow logically from assumptions.

• Never include invalid costs and benefits and uses a multiplier.

Questions?

• Views?

• Opinions?

• Comments?