Approach of the Office of the C&AG to Examining Value for Money

Seamus McCarthyDeputy Director of Audit

Office of the Comptroller and Auditor General

Irish Evaluation NetworkFriday 14 December 2007

Objectives

Place VFM examination work by OCAG in the context of accountability and evaluation structures in Ireland

Outline how evaluation challenges are dealt with in VFM examination work

Briefly survey some possible future developments for OCAG assessment of VFM

C&AG’s VFM reporting mandate

C&AG may examine and report on

– the economy and efficiency achieved by audited bodies in the acquisition, use and disposal of resources

– the adequacy of the systems, procedures and practices used by audited bodies to evaluate the effectiveness of their operations

C&AG may make such comparisons as he considers approporate (including comparisons of systems, procedures and practices)

Evolution of the mandate

C&AG role is part of the Westminster model of government

Original focus on financial audit

– Appropriation accounts system

– State agencies’ accounts

Progressively more reports on cases of loss, waste or extravagance e.g.

– Project cost overruns

– Fishery harbours

– Prison planning

Project audits: 1990-1993

Comptroller and Auditor General (Amendment) Act, 1993

The accountability cycle

Comptroller and Auditor General

Comptroller and Auditor General

Committee of Public Accounts

Committee of Public Accounts

Departments/ agencies

Departments/ agencies

Report

Dáil ÉireannDáil Éireann

Report

Information

Public hearing

Minister for FinanceMinister for Finance

InformationMinute

Public auditing principles

Independence

Objectivity/freedom from bias

Wide scope of mandate

Freedom to report publicly

Authority

Certification audit vs Reporting

Certification auditStatutory/obligatory

Output is a certificate/pro forma

Accounts submitted for audit

Numbers based

Standards set

Set audit procedures

Answers are clear

Ongoing relationship with auditees’ staff

ReportingDiscretionary

Output is a report

Nothing formal to ‘audit’

Narrative/analysis based

Standards must be developed

No fixed procedures

No ‘right’ answers

New relationships with auditees’ staff

Organisation of OCAG

C&AGC&AG

Director of AuditDirector of Audit Director of AuditDirector of Audit Secretary and Director of Audit

Secretary and Director of Audit

Vote audits

Vote audits

Semi State audits

Semi State audits

VFM exams

VFM exams

Corporate Services

Corporate Services

Certification audit

Certification audit

Financial management

reporting

Financial management

reporting

Value for money reporting

Value for money reporting

Strategic reporting issues

Reporting governed by ethical/professional standards

Quality assurance

– Expert input

– Internal review/audit of working papers

– Challenge/clearance process

– ‘Cold’ review

Resources

– Internal staffing

• Developing capacity

– Consultants

• Ensuring independence/objectivity

• Understanding of requirements for VFM reporting

Stages in the examination process

Selection of examinations

Selection of examinations

PlanningPlanning

InvestigationInvestigation

ReportingReporting

Follow-upFollow-up

Selection of examinations

C&AG exercises discretion

Strategic themes/priorities e.g.

– Public sector management reforms

– PPPs

– Health sector

Matters arising (from audits, controversies, etc)

Consultation with PAC

Review of potential reporting issues based on

– Risks to value for money in the body under review (performance risks)

– Potential added value of a C&AG report

– Feasibility

Some recent reports

Schools PPP

Waste management in hospitals

Rent supplement

The Financial Regulator

Follow up review of examinations carried out 1997-2000

eGovernment

Drug addiction treatment and rehabilitation

Examination planning

Scope setting - bound by statutory mandate

– Issues to be examined – may include financial management issues

– Time frame

– Financial scope

Analytic framework development

Liaison with auditee(s)

Tests of data availability

Time and budget planning

Formal review of plan by C&AG

Socio/economic statistics

Business targets

Output measures

Activity management

Activity measures

Capacity management

Measures of resource use

Budgeting Financial results

Public sector business model

Financial outturnFinancial outturn

Inputs (resources)Inputs (resources)

Activities/processesActivities/processes

OutputsOutputs

Identify social needsIdentify social needs

Set objectivesSet objectives

BudgetBudget

Planned inputsPlanned inputs

Planned activitiesPlanned activities

Planned outputsPlanned outputs

Planned impactsPlanned impacts

Planned resultsPlanned results

ImpactsImpacts

ResultsResults

Analytic framework

Core VFM concepts (economy, efficiency, effectiveness) are very high level

Risk-based prioritisation of key performance areas

For each performance area, identify

– Performance standard in terms of 3 Es (+ equity)

– Performance measure(s)

– Target(s)

Focus on performance of relevant bodies – things they control or influence, rather than context indicators

Framework set at outset, but may evolve in light of information

Engagement with auditees

Objectives

– Brief management about plans, and get their views

– Identify data sources/data holders

– Agree working procedures

– Resolve difficulties

– Inform management about progress

– Share findings (no surprises principle)

– Get the management perspective

Investigation

Statutory access rights

Data sources

– Official documents

– Literature search

– Administrative data e.g. databases

– Semi-structured interviews (with managers, staff, clients)

– Questionnaires/surveys

– Sampling of cases (e.g. for process review)

Data handling procedures

Analysis

Data checking/validation incl. system review

Forms of analysis used

– Exploratory data analysis

– Descriptive statistics

– Trend analysis

– Regression

– DCF-based financial models

– Activity-based costing

Review/audit of results

Analysis of professional judgements results

Reporting

Target audiences: members of the Dáil; the public; public service managers

Style: non-specialist; concise but comprehensive; clear

Conclude on examination questions

Include

– recommendations for improvement

– examples of transferrable good practice

Formal clearance of drafts

– two (or more) rounds of clearance

Follow-up

Publication of report

Briefing/update for C&AG for appearance at PAC meeting

PAC hearing

Follow-up review or re-audit

Potential impacts (during or post examination)

– Expenditure savings

– Greater efficiency

– Improved service

– Better systems for evaluating effectiveness

Possible future developments

Review of C&AG’s legislative mandate

Reporting of transferable good practice

Ad hoc reference groups for specific evaluations

Development of performance reporting (e.g. Output Statements, VFM reports)

Changing structure of public sector – new agencies, inspectors, regulators, PPPs, community/voluntary sector, etc.

– Revised performance models/accountability regimes

– New issues e.g. gaps in service provision; boundary disconnection; contract/service level management

Discussion