Canadian Best Practices in Measuring

Efficiency, Effectiveness and Performance of the Public

SectorD. Brian Marson

APO International AdvisorColombo, June 2015

The Importance of Performance Measurement and Accountability

• Governments around the world are under pressure from citizens to demonstrate higher levels of performance in:• the management of government funds, in • the delivery of programs and services, and in• the achievement of stated policy goals.

• In this context many governments, including Canada, have introduced performance measurement and accountability systems- both for internal management purposes, and for external reporting purposes.

The Top Ten Canadian Public Management IssuesAccording to an Institute of Public Administration of Canada survey of federal and provincial Deputy Ministers across Canada, their top ten management issues over the next few years are:

1. Recruitment, retention, retirement and succession planning

2. Improved accountability and transparency 3. Performance and results management, measurement and reporting

4. Improving the quality of service delivery, E-service 5. Citizen-centred service, single window, service partnerships 6. Training and leadership development 7. Responding to cost pressures and budget reductions 8. Knowledge transfer and management, and corporate memory 9. Rethinking intergovernmental relationships and fiscal arrangements

10. Employee well-being and employee commitment

The Top Ten Management Challenges for Public Sector Executives in Canada

(Survey of Secretaries and CEOs by Marson and Ross for IPAC)

Canada’s Goal: A Professional Public Service Achieving Excellent Results for Citizens

World Bank- Government Effectiveness RankingsSource: Institute for Government: The State of the Service, July 2009Data sources: Bertelsmann Sustainable Governance Indicators 2009; World Bank Worldwide Governance Indicators 2008

International Government Rankings

2

3

4

5

6

7

8

9

0 0.5 1 1.5 2 2.5

World Bank Government Effectiveness

Ber

tels

man

n M

anag

emen

t In

dex

Mexico

TurkeyItaly

Poland

Greece

HungarySlovakia

Portugal

Czech Republic

South Korea

Belgium

FranceJapan

IcelandIreland

NorwayDenmark

Sweden

Switzerland

Spain

United StatesGermanyLuxembourg

Austria Canada

United KingdomAustralia

Netherlands

Finland

Canada is Recognized as a Leader in Good Public Management

6

Single-Window Service

Digital DeliveryMeasurement of

Service and Benchmarking

Municipal Service Delivery

Training of Service Managers

Strong Service Culture

PART ONE:

CANADA’s RESULTS-BASED PERFORMANCE

MANAGEMENT AND AND REPORTING SYSTEM

Canada’s system of performance measurement, performance management, and performance reporting is perhaps the most comprehensive of the OECD countries:

1. Budgeting: Canada has an advanced results-based budgeting and results reporting system (RPP/DPR)

2. Management Performance: Canada uses its Management Accountability Framework (MAF) to hold heads of departments accountable for performance in ten management results areas.

3. Program Efficiency and Effectiveness: Canada has a comprehensive system of systematic program effectiveness and efficiency evaluation, linked back to the resource allocation process

4. Policy and Program Performance Reporting: Canada has developed a system for reporting back to citizens on economic and social outcomes.

http://www.tbs-sct.gc.ca/reports-rapports/cp-rc/2010-2011/cp-rctb-eng.asp

Comprehensive Government Performance Management and Accountability in Canada

Annual Departmental

Report on Plans and Priorities

(RPP)

Annual DepartmentalPerformance

Report(DPR)

Evaluation of Program Results

on aFour-Year Cycle

Management Accountability

Framework- Priority Areas of Improvement

Management Accountability

Framework- Annual Report

Card

Annual Performance Management

Agreement with Agreed Results

Up-Front Review of Proposed New Programs by

Treasury Board

Year-end Performance Review and

Performance Pay

BUDGET-BASED PERFORMANCE ACCOUNTABILITY

PROGRAM PERFORMANCE ACCOUNTABILITY

DEPARTMENTAL MANAGEMENT PERFORMANCE ACCOUNTABILITY

INDIVIDUAL MANAGERS’ PERFORMANCE ACCOUNTABILITY

The Canadian Government`s Results-Based Performance Accountability Systems

PLANNED RESULTS ACTUAL RESULTS

The Government of Canada ensures that its program expenditures are efficient, effective, coordinated, and tied to Government priorities through a two level review process:

1. Rigorous up-front reviews by Treasury Board of all new spending programs for efficiency, effectiveness and links to Government priorities;2. A cyclical Strategic Review of all ongoing government programs over a four year cycle, to ensure efficiency, effectiveness, and relevance of departmental programs to Government objectives and priorities. Each year (linked to the Budget Process), departments must identify 5% of expenditures from the programs under review that could be re-allocated to other Government or departmental priorities.

Treasury Board supports the program review processes through both a Program Evaluation Policy, and a Centre of Excellence in Evaluation.

Efficiency and Effectiveness Evaluations are Linked to Annual Budget Decisions

The Canadian Government has adopted a disciplined approach up-front in the planning stages of new program spending by:

1. Anchoring new spending in the priorities of Government for Canadians, 2. Developing rigorous spending proposals, and 3. Linking new spending with existing spending.

This process ensures that the government can make informed decisions on the use of funds, such as ensuring that funding goes to those programs that are high priority for the Canadian Government and produce the intended results.

Accountability for Program Effectiveness:

1. the Upfront Review for New Programs

The Strategic Review Process: organizations must review 100 per cent of their direct spending and the operating costs of major statutory programs on a four-year cycle to assess how and whether these programs:

1. Are effective and efficient;2. Meet the Government’s priorities for Canadians; and3. Are aligned with federal responsibilities.

All organizations are required to identify reallocation options totaling 5 per cent of the department budget from their lowest-priority and lowest-performing programs.

Reporting on Government Performance to Citizens: The overall Expenditure Management System provides the framework that ensures Canadians have important information to hold the government to account for the management of their tax dollars.

Accountability for Program Effectiveness:

2. Regular Evaluation of Existing Programs

PART TWO:

CANADA’s MANAGEMENT

ACCOUNTABILITY FRAMEWORK (MAF)

In 2003 The MAF Integrated Several Existing Government Management Frameworks

Management Accountability

Framework

PS Excellence Framework (Baldrige +)

ModernComptrollership

Framework

Risk Management

Framework

Strategic HR Framework

TBS Study of High Performing

Public Organizations

Canada`s Management Accountability Framework Overview

The Purpose of MAF► Introduced in 2003, MAF is a performance management framework used

across the federal government to support management accountability of Department Heads and improve management practices

Objectives of MAF► Clarifies management expectations for Department Heads and identifies areas for

improvement► Provides a comprehensive perspective on government management performance► Promotes continuous improvement

Evolution of MAF► MAF has evolved over 12 years into TB’s key management oversight

instrument, assessing management capacity and performance of all departments and small agencies

► Has a direct impact on Department Heads’ performance commitments and pay ► Now used in resource allocation decisions and to risk-manage departmental

submissions to Treasury Board.

Canada’s Management Accountability Framework

Canada’s Management Accountability Framework - Performance Criteria

Leadership and Strategic Direction

Articulates and embodies the vision, mandate and strategic priorities that guide the organization while supporting Ministers and Parliament in serving the public interest.

Results and Accountability

Uses performance results to ensure accountability and drive ongoing improvements and efficiencies to policies, programs, and services to Canadians.

Public Sector Values

Exemplifies the core values of the public sector by having respect for people and democracy, serving with integrity and demonstrating stewardship and excellence.

Continuous Learning and Innovation

Manages through continuous innovation and transformation, to promote organizational learning and improve performance.

Governance and Strategic ManagementMaintains effective governance that integrates and aligns priorities, plans, accountabilities and risk management to ensure that internal management functions support and enable high performing policies, programs and services.

.People Management

Optimizes the workforce and the work environment to enable high productivity and performance, effective use of human resources and increased employee engagement.

Financial and Asset Management

Provides an effective financial management function founded on sound internal controls, timely and reliable reporting, and fairness and transparency in the management of assets and acquired services.

Information Management

Safeguards and manages information and systems as a public trust and a strategic asset that supports effective decision-making and efficient operations to maximize value in the service of Canadians.

Management of Policy and Programs

Designs and manages policies and programs to ensure value for money in achieving results.

Management of Service Delivery

Delivers client-centred services while optimizing partnerships and technology to meet the needs of stakeholders

MAF Annually Assesses Departmental Performance and Capacity in 10 Key Management Areas

1. Values and Ethics2. Managing for Results3. Governance and Planning4. Citizen-focussed Service5. Internal Audit6. Evaluation7. Financial Management and

Control8. Management of Security9. Risk Management10. People Management11. Procurement12. Information Management13. Information Technology14. Asset Management15. Investment Planning and

Management of Projects

Rating Scale

Strong

Acceptable

Opportunity for Improvement

Attention Required

Areas of Management

TEN AREAS OF PERFORMANCE

ACCOUNTABILITY

MAF is a rigorous assessment process► Assessments performed annually by the

Treasury Board of Canada Secretariat (TBS) and is based on evidence submitted by federal organizations

► All major federal departments and a third of small agencies are assessed on a rotational basis, which represents 55 to 60 organizations each year

► Each organization is assessed against expectations outlined under specific areas of management

► Rigorous assessments are prepared by TBS experts and drafts are discussed with departments and agencies before they are finalized

► Results are used as an input for annual assessments of Deputy Ministers

► Summaries of final assessments are made available to the public

Treasury Board supports departments and agencies throughout the MAF process

by providing tools, guidance and advice, as well as by

promoting the exchange of best practices.

A MAF Performance Assessment Example:Values and Ethics

Values and EthicsLines of Evidence (LoEs) 1.1 The organisation demonstrates a culture of mutual respect, integrity and

professionalism. ◦ For example: Activities leading to the development, implementation, and

communication of an organizational code of conduct. 1.2 Leaders demonstrate and promote V&E behaviours.

◦ For example: Senior management develops and implements a comprehensive V&E strategy/plan and communicates it to the organization.

1.3 The organization practices continuous improvement in the area of V&E. ◦ For example: The organization seeks to identify common V&E issues across the

public service or other jurisdictions and tailors solutions to its organizational needs.

Key Changes from Round VII to Round VIII: Public Service Employee Survey (PSES) results will be used every 3 years;

qualitative and process-based measures will be used the 2 years in between.

MAF’s Secure Assessment Portal

MAF Section 13. Effectiveness of Information Technology Management- Rating: Strong

• 3.1 Leadership: Acceptable: CIO has responsibility and accountability for virtually the full scope of information technology responsibilities.

• Web accessibility is partially integrated into the span of control. • Adequate participation in setting government-wide directions for information technology is evident. 13.2 Planning: Strong• Acceptable information technology plan is in place that aligns with the government-wide directions for information

and has an integrated planning process.• Organization is making efforts to institutionalize web accessibility into planning and governance processes. 13.3 Value: Strong• Organization has well established processes and performance reporting on progress for all aspects of information

technology management (including on Common Look and Feel implementation plans). • Organization has integrated performance measurement tools and metrics including an established costing model for

information technology services and a service costing baseline that guide information technology investment decisions

• Organization has processes that demonstrate sharing, re-using or leveraging across the government for ideas, best practices, assets and implementations.

Commended for their progress and encouraged to share their IT plan and integrated set of processes and practices for planning and progress reporting in order to monitor and oversee the delivery of business value from IT investments.

• Governance model for effective management of the organization's web presence (i.e., citizen-facing web content and applications).

• Participation in GC-wide working groups and GC-wide collaborative work spaces to improve opportunities for sharing and re-use in order to reduce complexity and duplication, promote alignment and interoperability and optimize service delivery

Example of the Assessment for the IT Component of MAF for Revenue Canada (Taxation Department) http://www.tbs-sct.gc.ca/maf-crg/assessments-evaluations/2009/nar/nar-eng.asp

The Departments` MAF Assessments are published on a Public Government Website

Key success factors to effectively implement a performance management approach

► Based on a 7-year experience of implementing the MAF, some lessons learned include:

Leadership at the top is critical to improve management practices Recognize at the outset that managing with a focus on results

requires a culture shift and that progress will take time and sustained focus

Performance pay of Department Heads should be linked to management performance

Performance management assessments should be constructive and encourage continuous improvement, not be a means to penalize organizations

Assessment tools need to be kept evergreen and room needs to be left for good judgment and contextualization

► MAF provides an excellent platform for cooperative sharing of best and leading practices, benefiting all federal departments

“Internationally, MAF is considered to be one of the more sophisticated management practices systems.”

(Independent Five-year MAF Evaluation, based on OECD study)

PART THREE:

CANADA’s MANAGEMENT AND SERVICE RESULTS

ARE STEADILY IMPROVING

Analysis indicates that MAF is helping the federal government to manage better.

► MAF is changing departmental management behaviour - organizations are making concerted efforts to improve their MAF ratings

► Deputy Ministers are using MAF to support their management accountabilities, and to enhance management performance with their executive team

► Measurable progress is being made and the bar is being raised

55%

63%

0%

51%

31%

18%

7%

50.5%

37%

5%

14%

27%

4%

14%

62%

21%

3%

21%

16%

1%

18%

68%

14%

0.5%0%

10%

20%

30%

40%

50%

60%

70%

80%

Attention RequiredOpportunity for ImprovementAcceptable Strong

2004-05 2005-06 2006-07 2007-08 2008-09 2009-10

… and improvements are being made across

most Areas of ManagementMAF 3-year Rating Comparison by Area of Management(Round VII ratings include carry-over ratings from Round VI; Large Departments & Agencies only; •• Core AoMs)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

IT M

anag

emen

t

Gov

ernm

ent-wid

e Pr

iorities

Perfor

man

ce R

epor

ting

Man

agin

g Ch

ange

Proc

urem

ent

Citize

n-fo

cuss

ed S

ervi

ce

Corp

orat

e M

anag

emen

t St

ruct

ure

Corp

orat

e Pe

rfor

man

ce Fra

mew

ork

Proj

ect M

anag

emen

t

Qua

lity

of A

naly

sis

Asset

Man

agem

ent

Info

rmat

ion

Man

agem

ent

Secu

rity

and

Bus

ines

s Co

ntin

uity

% o

f LD

As R

ate

d a

s "

Accep

tab

le"

or

"Str

on

g"

2007 2008 2009

•• P

eopl

e M

anag

emen

t

•• V

alue

s & E

thics

•• R

isk

Man

agem

ent

•• In

tern

al A

udit

•• F

inan

cial

Man

agem

ent a

nd C

ontr

ol

•• E

valu

atio

n

1.

Building Leadership Excellence Leadership Competencies, Management Training and Leadership Development

Building Workforce Excellence

◦ Recruitment & Promotion: Competition and Merit Principle

◦ Accountability: Performance Management Agreements

Building Workplace Excellence

◦ Employee Surveys and Employee Engagement

Institutional Arrangements

◦ Public Service Commission- Parliamentary Agency

◦ Treasury Board (Chief Human Resource Officer)

◦ Privy Council (Cabinet) Office

◦ Canada School of Public Service

Implementing Strategic Human Resources Management in the Government of Canada

Excellent Leadership +Excellent Workforce + Excellent Workplace = Excellent Results

Human Resource Management component of MAF- Data from the Employee Survey is one measure

People Management: Every Government Department has an Employee Engagement Score from the

Public Service Employee Survey

“Over 201,000 public servants in 89 organizations took part in the Public Service Employee Survey (PSES). Employees shared their thoughts on leadership, culture, the workforce, the workplace, and their own personal level of engagement.

The participation rate of 72.2% was the highest in our history. The overall results demonstrated that the Public Service of Canada continues to offer a workplace of choice. The results also demonstrate our strong commitment to public service. The majority (89%) of employees are proud of the work they do and 94% are willing to put in the extra effort to get the job done.”

-Wayne Wouters,

Former Head of the Canadian Canadian Public Service

HR Results: The Regular Canadian Government Employee Surveys Show a

Dedicated Workforce and a Positive Workplace

Employee Engagement Score

66 out of 100 73 out of 100

Level of Engagement

37% high 51% high

21% low 13% low

YEAR ZERO YEAR TWO

ONTARIO PROVINCE’S IMPROVING EMPLOYEE ENGAGEMENT SCORES

+10%

Improving Business Sector Satisfaction with Government Services in Canada

Promoting Continuous Improvement in Citizen and Business Satisfaction with

Canadian Government Service

…Long-term trend for 26 services

7273

676464

50

60

70

80

1998 2000 2003 2005 2008

Average service quality rating(0-100)

Year

2014

74

Source: ICCS CF5 & CF6

AverageCitizen

Satisfaction Score – for 21 Government

Services

Results Improvement for the Royal Canadian Mounted Police

Improving Royal Canadian Mounted Police Performance on the “Drivers”

81-84% OverallCitizen Satisfaction

The Royal Canadian Mounted Police Surveys over 7,000 Citizens Annually Using the Common Measurements

Tool Survey Tool, and Uses the Results to Improve Service

Some Best Practice Results(Institute for Citizen Centred Service, Toronto,

Benchmarking Service (www.iccs-isac.org)

PART FOUR:

CANADA’s POLICY AND PROGRAM RESULTS REPORTING SYSTEM

CANADA’S ANNUAL PERFORMANCE REPORTS- THE FRAMEWORK

The purpose of this framework is to map the financial and non-financial contributions of federal organizations receiving appropriations by aligning their program activities to a set of high level outcome areas defined for the government as a

whole. http://www.tbs-sct.gc.ca/reports-rapports/cp-rc/2010-2011/cp-rctb-eng.asp

PART 5

Some Concluding Thoughts

Improving Management Accountability- Canada’s Implementation Experience

1. Measure the right things

2. Establish a few key measures

3. Build management accountability into the performance agreements of every manager

4. Focus on continuous improvement in results, not on the measurement process

EXAMPLE: Building a Comprehensive Performance Management System - The Philippine Government’s

Adaptation of the Canadian Performance Management Model

Annual Departmental

Report on Plans and Priorities

(RPP)

Annual DepartmentalPerformance

Report(DPR)

Evaluation of Program Results

on aFour-Year Cycle

Management Accountability

Framework- Priority Areas of Improvement

Management Accountability

Framework- Annual Report

Card

Annual Performance Management

Agreement with Agreed Results

Up-Front Review of Proposed New Programs by

Treasury Board

Year-end Performance Review and

Performance Pay

BUDGET-BASED PERFORMANCE ACCOUNTABILITY

PROGRAM PERFORMANCE ACCOUNTABILITY

DEPARTMENTAL MANAGEMENT PERFORMANCE ACCOUNTABILITY

INDIVIDUAL MANAGERS’ PERFORMANCE ACCOUNTABILITY

The Canadian Government`s Results-Based Performance Accountability Systems

PLANNED RESULTS ACTUAL RESULTS

Strategic Performance Management System (Individual)

Organizational Outcomes

Good Governance

and Anti-Corruption

Human Development & Poverty Reduction

Economic Developmen

t

Security, Justice, and

Peace

Climate Change

Adaptation

Societal Goals/Outcomes

Sectoral Goals/Outcomes

Financial Stewardship

Internal Process Client Satisfaction

Leadership, Learning and

Growth

Major Final Outputs (customer-oriented results)

STR/SERSocio-Economic Report

PPARC Priority Program Accountability Report Card

MARC- IMFO Accountability Report Card

MARC- IIManagement Accountability Report Card

Government of the Philippines Harmonized Results-Based Performance Management System (RBPMS))

To quality for PBB, agencies must meet at least 90% of PPARC and MARC-I, and satisfy MARC-II (good governance conditions)

1. THE WHY?◦ What is the purpose of the performance management system: to measure and

report performance; to improve management and policy performance, or both?

2. THE WHAT?◦ Should we measure Management performance, or program and policy

performance, or both? What management factors should be measured? ◦ What level is the measurement- at Government level, Departmental level or

both?

3. THE HOW?◦ What measures will be used for determining performance for each

factor? Will measures be process measures, results measures, or both? How will results be reported and to whom- President, Congress, Citizens? ……..How?

4. THE WHO?◦ Which central agency will conduct the reviews and provide

oversight? What is the role of departments and what is the role of central agencies?

Conclusion- Considerations in Building a Performance Management Framework

Canada’s Results: A Professional Public Service Achieving Excellent Results for Citizens

ANNEX

CANADA’s PROGRAM EFFICIENCY AND EFFECTIVENESS

EVALUATION SYSTEM

Treasury Board Policy Sets Out the Requirements for Departmental Program Evaluation capacity and process:

http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024&section=text#cha5 6.1 Deputy heads are responsible for establishing a robust, neutral

evaluation function in their department and for ensuring that their department adheres to this policy and to its supporting directive and standard. Deputy heads:

◦ 6.1.1 designate a head of evaluation at an appropriate level as the lead for the evaluation function in the department;6.1.2 ensure that the head of evaluation has direct and unencumbered access to the deputy head, as required;

◦ 6.1.3 ensure that a committee of senior departmental officials (referred to as the Departmental Evaluation Committee) is assigned the responsibility for advising the deputy head on all evaluation and evaluation-related activities of the department and is chaired by the deputy head or a senior level designate; supported by the head of evaluation on evaluation matters; and structured with specific roles and responsibilities.

◦ 6.1.4 approve evaluation reports, management responses and action plans in a timely manner; ◦ 6.1.5 use evaluation findings to inform program, policy, resource allocation and reallocation decisions;◦ 6.1.6 ensure that complete, approved evaluation reports along with management responses and action plans

are made easily available to Canadians in a timely manner while ensuring that the sharing of reports respects the Access to Information Act, Privacy Act, and the Government Security Policy;

6.1.7 approve for annual submission to the Treasury Board of Canada Secretariat, a rolling five-year departmental evaluation plan

Treasury Board Program Evaluation Policy

Strategic reviews are done at the same time every year in order to put recommendations forward for consideration as part of the Government of Canada's annual budget planning process.

Each organization is required to seek advice from arm's-length external advisors to get a third party perspective on the process and its results. Once organizations have completed a comprehensive review of all of their programs, they are required to identify a total of five per cent of their program spending from their lowest-priority and lowest-performing programs. These funds are proposed for reallocation to higher priorities.

Savings achieved through strategic reviews are redirected to budget priorities to better meet Government priorities.

The Departmental Strategic Review Process

A more comprehensive (100%) 2012 Strategic Review process required organizations to review all spending from the following perspectives:

Operating efficiency—To what extent are results being achieved efficiently? Can this activity be delivered at a lower cost or in a more effective way?

Internal services—Are internal services (e.g. human resources management, financial management, communications) as efficient as possible? Can improvements be made to reduce any overlap and duplication?

Effectiveness—To what extent is this program, activity or service achieving the expected results for which it was designed?

Affordability—Is the program, activity or service a priority, and is it affordable during a period of fiscal restraint?

Relevance and need—To what extent is there still a need for this program, activity or service?

Federal role—To what extent is this program, activity or service consistent with the federal government’s roles and responsibilities?

Organizational role—Would greater efficiencies be achieved if another department or agency, a government service provider, or the private sector delivered the program, activity or service?

The Strategic Review Questions (2012)

Between 2007–08 and 2010–11, over $2.8 billion in ongoing savings were generated from strategic reviews, making these funds available to reduce the government deficit or to allocate to higher-priority programs. 98% of Government expenditures were reviewed over the first 4 years.

Guided by experts from outside government, in 2012, organizations were asked to focus on achieving efficiencies in their operations, as well as to refocus business processes and service delivery platforms. As a result, further Strategic Review savings of $5.2 billion were identified, representing 6.9 per cent of an aggregate review base of $75.3 billion.

The Strategic Review Results

The final results of the annual Strategic Review are announced in the federal budget of the following year.

The results of the Strategic Reviews are published in the annual Budget Documents each year, and are available on-line for the public to examine.

Reporting the Strategic Review and Evaluation Results to the Public

The Treasury Board`s Centre of Excellence for Evaluation provides functional leadership, including guidance in the conduct evaluation practices across the federal government. The CEE website offers information, tools and resources for evaluation professionals in the federal government.

Evaluation Support for Departments: The Centre of Excellence for Evaluation