Workshop: Development and Use of Outcome-based Measures in Government Planning & Reporting

1

Workshop:Development and Use of

Outcome-based Measures in Government Planning & Reporting

May 24th, 2006, Union Centre, Winnipeg, Manitoba

Presented by:

Manitoba Treasury Board Secretariat ,Office of the Provincial Comptroller of Manitoba, and

The International Institute for Sustainable Development (IISD)

2

AgendaI. The Context in Manitoba [30 minutes]

• Introductions and overview• Performance Measurement in Manitoba• Role of performance measures in annual reporting

II. Trends in Planning, Reporting and Performance measures [45 minutes]• Why are outcome-based measures important?• Innovative examples and trends in reporting

III. Sharing of Manitoba Examples [45 minutes]• Plenary discussions based on Pre-work assignment

IV. Developing and Using Outcome-based Measures [105 minutes]• Key steps in development, use and analysis of outcome-based

measures• Group work on key steps

V. Reporting Outcome-based Measures [75 minutes]• Completing a narrative table for performance indicators

Closing Remarks

http://www.clipsahoy.com/webgraphics/as0306.htm

http://www.clipsahoy.com/webgraphics3/aw4736.htm

http://www.clipsahoy.com/webgraphics2/as3803.htm

http://www.clipsahoy.com/webgraphics3/aw4736.htm

3

Objectives of Workshop

• Intro to developing performance measures linked to organizational goals

• Overview of current practice elsewhere

• Understanding/discussion of issues related to government performance measurement reporting

• Context for new Annual Report requirements (looking back) and new PSO requirements (looking ahead)

4

Part I. The Big Picture

• Performance Measurement in Manitoba Drew Perry (Treasury Board Secretariat)

• Role of performance measures in annual reporting

Betty-Anne Pratt (Provincial Comptroller)

I. The Big Picture

5

Trends and context

Trend toward reporting on changes in socio-economic and environmental conditions that matter to Manitobans

I. The Big Picture

http://www.gov.mb.ca/finance/mbperformance/perf01.html

6

2005 Discussion Document examples: Economy

I. The Big Picture

7

2005 Discussion Document examples: People

I. The Big Picture

8

2005 Discussion Document examples: Community

I. The Big Picture

9

2005 Discussion Document examples: Environment

I. The Big Picture

10

The Planning and Reporting context

• Performance Measurement is not done in isolation

• Identified as part of Priorities & Strategies Overview (PSO)

• reported upon in Annual reports

• Performance Measures show whether our plans are working

I. The Big Picture

11

Full cycle view

PSO (plan) for following

fiscal

New initiatives

for following

fiscal

Estimates for

following fiscal Budget

announcedEstimates

supplementExecution of plans during fiscal year

Annual Report

Spring Yr1 Fall Yr1 Spring Yr2 Fall Yr3Spring Yr3

I. The Big Picture

12

Full cycle view


fiscal

New initiatives

for following

fiscal

Estimates for


announcedEstimates


Annual Report


I. The Big Picture

13

Multiple cycles in play

BudgetEstimates

supplement

Annual Report

Spring 04 Fall 04 Spring 05 Fall 06Spring 06Fall 05

PSO (plan)

New initiatives

Estimates

Budget

Estimates supplement

Annual Report

PSO (plan)

New initiatives

Estimates

Budget

Estimates supplement

PSO (plan)

New initiatives

Estimates

04/

05

07/

08

05/

06

06/

07

I. The Big Picture

14

Issues/Opportunities re Annual Reports

• Previous guidelines did not request measures of progress or performance

• No specific requirement to link annual reports to larger process (Estimates Supplement or plans)

• No specific requirement for Department annual reports to be placed online

• New direction set in 2005 through Reporting to Manitobans on Performance

I. The Big Picture

15

Trends and issuesin government annual reporting

• Accountability• Who uses annual reports? (internet)• Public expectations (of results, of reporting)• Limited resources, demographic trends

(retirements)… • …governments need to do “more with less”• Trend to shared, horizontal efforts• Important to agree on how to assess, report and

use results

16

Recap: Full cycle view


fiscal

New initiatives

for following

fiscal

Estimates for


announcedEstimates


Annual Report


Annual Reports, including measures, are connected back

to plans…

I. The Big Picture

17

Part II. Trends in Government Planning and Reporting

A. What are outcome measures and why are they important

Concepts and vocabulary

B. Who is doing this really well? Oregon: Results-oriented Strategic Planning

C. What are Alberta and Saskatchewan doing? Alberta: Goal-based Budgeting Sask. Government Accountability Framework

II. Trends in Reporting

18

A. What are outcome measures and why are they important?

Our People

Our Economy

Our Environment

de•vel•op (di•vel′əp) v.t. 1. To expand or bring out the potentialities, capabilities, etc.


mailto:%[email protected]

19

A. What are outcome measures and why are they important?

Our People

Our Economy

Our Environment

de•vel•op (di•vel′əp) v.t. 1. To expand or bring out the potentialities, capabilities, etc.

Experience has shown that a pathway to sustainability cannot be charted in advance. Rather, the pathway must be navigated through processes of learning and adaptation.

National Academy of Science 1999. Our Common Journey: A Transition Toward Sustainability.


mailto:%[email protected]

20

IISD International and National-Level Perspective

• Nations starting to develop information systems to gauge societal wellbeing and sustainability

The UK– “Securing the Future” Strategy– UK National Framework Indicators– Departmental strategies– Independent review by sustainable

development commission

• Norway– National strategy and indicators– Led by Prime Minister’s office and

chaired by Finance Ministry


21

• Provinces and states using societal goal and outcome-based planning and budgeting systems

Oregon– Oregon Shines state-wide vision – Oregon Benchmarks to gauge progress – State departments to link performance

measures to Oregon Benchmarks– Independent review

• Alberta– 12 provincial goals and 60+ indicators– Departmental business plans and annual

reporting linking to provincial indicators

IISD Provincial and State Level Perspective


22

IISD Community-level Perspective

• Communities creating new forms of social infrastructure to navigate quality of life and sustainability

Orlando– Healthy Community Indicators Initiative

of Greater Orlando

• Winnipeg– Quality of Life Indicators System concept

being proposed by range of community stakeholders


23

Output The result of an activity

IntermediateOutcome

Short to medium-term consequence of an output

High-levelOutcome

ActivityProcesses and Inputs of a policy, program, or project

Hybrid from three different logic models (Canadian International Development Agency, Province of Alberta, State of Oregon)

Contributing to society’s wellbeing and

sustainability objectives

External influences

Feedback

Change in wellbeing conditions (economic, social and environmental)

accountability

Key Priority Areas

Logic Model

High-level Objectives

Actions

24

OutputThe result of an activity

IntermediateOutcome


High-levelObjectives

Change in well-being conditions (economic, social and environmental)

ActionsProcesses and Inputs of a policy, program or project

Stream water quality

(turbidity)

Rate of soil erosion by

water

Kilometers of river bank with

vegetation

% of re-vegetation programs completed

Environmental Social Economic

Key Priority Areas Prosperous Economy

Healthy, sustainable

surroundings

Quality Jobs

Example MeasuresUnderstanding Linkages

Gross Domestic Product

Share of GDP in manufacturing, business and

commercial services

Implement Economic Development and

Innovation Initiative and participate in other cross-

ministry initiatives that influence the province's

prosperity

Economic Development and

Innovation Initiative completed

Children entering school ready to learn

% children enrolled in Pre-K

program

Demographic surveys for Head-

start program completed

Head-start program

implemented

25

What are the benefits?

• What are the benefits?

• What are the risks?

• What are the constraints?


26

Climbing the Steps toward Performance Management

Mission/GoalsObjectives

PerformanceMeasures

Analysis for Continuous

Improvement

Mission statements declare the agency’s long-range intent; its purpose. Although the goals expressed in a mission statement may help shape the agency’s values and its organizational culture, they often are imprecise and sometimes even a bit vague.

Objectives are unambiguous statements of the agency’s performance intentions, expressed in measurable terms, usually with an implied or explicit timeframe.

Performance measures indicate how much or how well the agency is doing. Ideally, they track the agency’s progress toward achieving its objectives.

Many agencies compare this month’s or this year’s performance measures to those of the past. Some are beginning to make comparisons with other agencies and to begin the process of benchmarking.

(From Gov. of Alberta 1996)


27

Who does this really well?The Oregon Shines Case Study

Oregon Shines Case Study

28

Vision – “Oregon Shines II”

Economy: Quality jobs for all Oregonians

People: Safe, caring andengaged communities

Environment: Healthy, sustainable surroundings

Oregon Shines Case StudyFrom Conrad (2005)

29

Oregon Shines

Oregon's Strategic Plan- Oregon Shines (1989)- Updated every eight years

- Encompasses the entire stateOregon Progress Board

- independent agency created to be the steward of Oregon Shines

- law mandates Board to report biennially

- chaired by governor


30

Oregon BenchmarksMeasures for how Oregon as a whole is doing.

• Quality Jobs for All Oregonians– Economy (#1-17)– Education (#18-29)

• Engaged, Safe & Caring Communities– Civic Engagement (#29-38)– Social Support (#39-60)– Public Safety (#60-67)

• Healthy, Sustainable Surroundings– Community Development (#68-74)– Environment (#75-90)


High-levelObjectives

Key Priority Areas

31

Quality JobsEconomy Benchmarks


Indicator

Data Target

Source

Comparison

32

Engaged, Safe and Caring CommunitiesSocial Support Benchmarks


33

Healthy, Sustainable SurroundingsEnvironment Benchmarks


34

High Level Outcomes

(Benchmarks)

Is work happening? Outputs

Is society benefiting?

IntermediateOutcomes

Are strategies working?

Linking Government to the Benchmarks


35

Linking Government to the Benchmarks

Benchmarks

Organization’s Progress

Performance Measures

Oregon’s

Progress

External Influences


36

ODA addresses soil erosion from agriculture through a partnership that

includes the USDA Natural Resources Conservation Service,

USDA Farm Service Agency, OSU Extension Service, and Soil and

Water Conservation Districts (SWCDs). ODA coordinates with

these agencies and is specifically responsible for implementing a

regulatory program for nonpoint source


IntermediateOutcome

Output

Oregon Shines Case Study

Healthy, Sustainable Surroundings

Key Priority Areas

37 Oregon Shines Case Study

Aligning Departmental Performance Measures with Oregon Benchmarks

38 Sask. Case StudyI. The Big Picture

Saskatchewan Government

39

Goal Objective Performance Measure 1) A sustainable transportation infrastructure

Preserved principal highway network to meet the future economic needs of the Province

Per cent of the principal highway network in “good” condition

Amount of principal pavements beyond their service life

Transformed regional transportation network to meet the future needs of rural Saskatchewan

Per cent of regional highway network in “good” condition by surface type: Pavement; Thin Membrane Surface (TMS); Gravel

Reduced damage on the highway system caused by overweight vehicles

Per cent of overweight trucks on the highway system

Increased funding from additional sources

Additional funding from non-provincial government sources

Ratio of road operations to overhead 2) The transportation system strengthens economic development and serves social needs

Reduced cost of moving goods and people by road, rail and air

Value of economic development generated by the Department’s trucking programs

Per cent of principal highway network available at primary weights on an annual basis

Targeted infrastructure investment for economic growth and social utility

Cumulative per cent of twinned highway opened to traffic

Improved connections in the North

Cumulative per cent of improved northern community access roads

3) Safe movement of people and goods

Reduced collisions on the road Per cent of collisions involving an injury or fatality Ratio of partnership trucking fleet collision rate

compared to Canadian commercial trucking fleet collision rate Per cent of commercial vehicles inspected that are not

mechanically fit and placed out of service Number of Commercial Vehicle Safety Alliance

(CVSA) inspections conducted per year Per cent of provincial railway operators with approved

safety management plans Increased workplace safety Number and severity of at-work injuries

Key Priority

Area


Which of these are outcome based?

Which are more output based?Example –

Sask. Dept. of Highways and

Transportation

40

Example Indicator

Sask. Case Study

Source: http://www.highways.gov.sk.ca/docs/reports_manuals/reports/annual_05.pdf

41

Towards Goal-based Budgeting in Alberta

Alberta Case Study

http://www.finance.gov.ab.ca/publications/annual_repts/govt/ganrep05/execsumm.html

42

43

44 Alberta Case Study


46


Government Goal 7: Alberta will have a prosperous economy

Core Government Measures (Measuring Up):

Goal 7 Budget and Actuals (2004-05)

$2,519 million $3,206 million

Goal 7 Government Strategies/Actions• Securing Tomorrow’s Prosperity value-added strategy• Agri-Food Discovery Place (AFDP) construction• Agrivalue Processing Business Incubator• Cottage Winery Policy• Promoting tourism

Ministry of Economic Development

• 3-year Business plan

• goals and performance measures

Alberta’s 20-year Strategic Plan: Today's Opportunities, Tomorrow's Promise: A Strategic Plan for the Government of Alberta

• Tourism industry revenues• Barriers to trade • Business connectiveness• Sponsored research at University of

Alberta• Industrial disposition planning• Biodiversity monitoring system• Oil Sands production

* Gross Domestic Product (GDP)• Job growth• Labour force participation rate• Labour productivity• Personal disposable income• Value-added industries• Export trade


Ministry of Economic Development Goal(s) (that supports Government Goal 7):

Goal 1: Economic growth and diversification through collaborative

strategic planning and policy development

Ministry Measures• Share of GDP in manufacturing, business and

commercial services

Goal 7 Budget and Actuals (2004-05)

Ministry Strategies/Actions1.1. Lead the implementation of the Economic Development and Innovation Initiative and participate in

other cross-ministry initiatives that influence the province's prosperity.1.2 Lead the implementation of Securing Tomorrow's Prosperity with other participating ministries to

support the development of competitive value-added goods and services industries.1.3 Provide longer-term strategic input to the development of policy to support a sustainable and

diversified economy in Alberta.1.4 Collaborate with the Alberta Economic Development Authority to set out the key directions where

targeted actions and strategies can produce the best return for Albertans.1.5 Provide the skills, processes, and technology to enhance information collection and analysis,

management and sharing of knowledge within the Ministry, and to support strategic consultation with our clients.

Opportunities and Challenges

• importance of innovation

• skilled labour shortage

• benchmarking Alberta’s performance

49

Alberta Goal-based Budgeting: Recent advances and lessons

1. Business planning and performance measurement system in place since 1990s

2. Early focus on a few core indicators (23 in Measuring Up)

3. The Government Accountability Act to ensure ministries maintained their business planning and performance reporting system over time.

4. Several improvements in Alberta’s accountability system has occurred in the past 3 years including:

• 20-year Government Strategic Business Plan• Goal-based budgeting and performance reporting• Strategic priorities and cross-ministerial initiatives (involving more

than one ministry) budgeting and performance reporting.• High-level Societal Indicators reintroduced

50

Part III. Manitoba Examples

• Sharing of Manitoba examples in the use of outcome-based measures

III. Manitoba Examples

51

Manitoba Examples:Pre-Work Discussion

• Review your pre-workshop assignment

• Brief presentations[5 minutes + 5 min questions]

• General discussion of issues

III. Manitoba Examples

52

Part IV: Development and Use of Outcome-based Measures

A. Overview of key steps in the development and use of outcome-based measures

B. Working through an example• Groups of at least two (per

department if possible)

IV. Outcome-based Measures

http://www.clipsahoy.com/webgraphics2/as3598.htm

53

Output The result of an activity

IntermediateOutcome


High-levelOutcome

ActivityProcesses and Inputs of a policy, program, or project

Hybrid from three different logic models (Canadian International Development Agency, Province of Alberta, State of Oregon)

Contributing to society’s wellbeing and

sustainability objectives

External influences

Feedback

Change in wellbeing conditions (economic, social and environmental)

accountability

Key Priority Areas

Logic Model


Actions

54

Key Steps

1. Frame the logic model for your issue

2. Identify SMART measures for the logic model

3. Understand and articulate key external influences

4. Analyze feedback based on the SMART measures and external influences


55

Output:

________________

Intermediate Outcome:

_________________

High-level Objective:

_______________

Actions

________________

2. Measures and Targets

Indicator:

Indicator:

Indicator:

1. Logic Model3. External Influences

Influences:

Influences:

Influences:

4. Analysis & Feedback

Indicator:

Analysis:

Analysis:

Analysis:

Key Priority Areas


56 Hybrid from three different logic models (Canadian International Development Agency, Province of Alberta, State of Oregon)

Output:

River bank re-vegetation


Reduced soil erosion


Improved stream Water quality

Actions:River bank re-vegetation

program

1. Logic ModelKey Priority Area

Safe drinking water, enjoyable recreation

57

2. Identify SMART Measures

• Specific

• Measurable

• Aggressive, yet achievable targets

• Relevant

• Time-bound


58

Anatomy of an measure

From Conrad (2005)

Indicator

Data Target

Source


59

Types of Targets

Type ExamplePolicy-specific targets

Determined in a political and/or technical process taking past performance and desirable outcomes into account.

Example: official development assistance shall be 0.4 percent of national GNP.

Standards Nationally and/or internationally accepted properties for procedures or environmental qualities.

Example: water quality standards for a variety of uses.

Thresholds The value of a key variable that will elicit a fundamental and irreversible change in the behaviour of the system.

Example: maximum sustainable yield of a fishery.

Benchmark Comparison with a documented best-case performance related to the same variable within another entity or jurisdiction.

Example: highest percentage of households connected to sewage system in a comparable jurisdiction.

Principle A broadly defined and often formally accepted rule.

Example: the policy should contribute to the increase of environmental literacy.

60

Output:







program


River water quality (turbidity, nitrogen

concentration)

Rate of soil and rill erosion on farmland

Kilometers of river bank

with vegetation

1. Logic Model

% of re-vegetationPrograms complete

Key Priority Area



61

STEP 3: External Influences

• Your policies, programs and projects are not the only influence on the desired outcomes and outputs

• Direct influences• Other department activities, businesses, NGOs, civil society• Other jurisdictions (provinces, countries)• Nature (e.g., weather)

• Indirect influences• Broader societal driving forces (e.g., demographics, markets,

consumption patterns)


62

Output:







program



concentration)

Rate of soil and rill Erosion on farmland


with vegetation

1. Logic Model


3. External Influences

• Quality of water flowing into MB• Treatment plant effectiveness

Zero till practices,weather, crop type

• Independent actionsfarmers• Natural growth

Key Priority Area



63

4. Analysis and Feedback

• What do the measures tell you about your activities?– Output level analysis– Intermediate outcome level analysis– High-level outcome analysis

• How does your understanding of external influences help your analysis?


64

Output:




High-level Objectives:



program



concentration)

Rate of soil and rill Erosion on farmland


with vegetation

1. Logic Model


3. External Influences

Zero till practices,weather, crop type

• Independent actionsof farmers• Natural growth/disease


Analysis: Water quality improving. Quality of water flowing into province has improved significantly.

Analysis: Could be due to less bank vegetation, change in crop or extreme series of extreme rainfall events

Analysis: Cropland expansion could be reason or natural disease. Need to research further

95%

Feedback: Water quality improving but not due to program. Other factors counteracted program such as expanded cropland and reduced zero tillage

• Quality of water flowing into MB• Treatment plant effectiveness

Key Priority Area


65

Exercise 1: Development of an Outcome-based Measure

• Forming your work group [5 min]– Find your working partner (your departmental colleague if

possible)• Select indicator to focus on [10 min]

– With your partner select a measure from your department to focus on for the exercise (from your workshop pre-assignment, or something else of mutual interest and value for your reporting cycle)

• Working with your Colleague [45 min]– Use the attached template as a guideline to carry out the 4

analysis steps for developing and/or using outcome-based measures

• Summarize your analysis on an overhead sheet• Plenary [30 minutes]

– Two groups will be asked to share their results, each followed by a plenary discussion

66

Output:

________________


_________________


_______________

Action:

________________


Indicator:

Indicator:

Indicator:

1. Logic Model3. External Influences

Influences:

Influences:

Influences:


Indicator:

Analysis:

Analysis:

Analysis:

Key Priority area

67

Part V: Requirement for Departmental Annual Reporting

• Overview of instructions for current round of departmental annual reporting

Dave Hill, Policy, Communication & Training

Office of the Provincial Comptroller

Preparing the narrative• Groups of at least two (per department if

possible)• Fill out a narrative table

V. Prepare Narrative

68

Expectations for this round of Department Annual Reports (2005-06)

New: Section featuring department performance reporting

New: Central review of the new section

New: Annual Reports to be consistently available online

I. The Big Picture

69

What is required for the new section of the Report?

• Five progress or performance measures• The few critical indicators that illustrate

progress against desired outcomes• Ideally, those that support key Department

priorities• Can be drawn from previous PSOs,

Reporting to Manitobans on Performance document, or other sources

I. The Big Picture

70

Next steps

• Instructions for new section were issued week of March 27th by Provincial Comptroller

• New performance reporting section due back for review at Treasury Board Secretariat June 1st

• Review by TBS and Comptroller’s Office

• Response back to Departments within 3 weeks

• Remainder of annual report can be completed as per usual process

• Support from TBS and Comptroller’s office

I. The Big Picture

71

Overview of Key Questions That Need to Be Addressed


What is Being Measured and

How? (A)

Why is it Important to Measure? (B)

What is the most Recent Available

Value for this Indicator? (C)

What is the Trend over

time for this Indicator? (D)

Comments/Recent

Actions/Report Links (E)

1.

2.

3.

4.

5.

72

Illustrative ExampleWhat is being

Measured and How?

Why is it Important

to Measure this?

What is the Most Recent Available

Value for this Indicator?

What is the Trend over time

for this Indicator?

Comments / recent actions /

report links

We are measuring access to health services by looking at the percentage of Manitobans reporting “no difficulty” in accessing health services.

One key determinant of population health is its access to quality health services. This measure assesses access only.

For 2003 (most recent survey data), % of Manitobans who reported “no difficulty” accessing: Health information or advice: 82% Immediate care: 75% Routine care: 81%

ImprovingWait times for key procedures have reduced significantly over the last five years. See page X of the report for details.The source of these data is the Statistics Canada Health Access Survey. There is no directly comparable study prior to 2003. The study will be repeated in 20xx.

See page X of the report for a discussion of recent actions addressing access to health services.

NOTE: All information below is adapted from Reporting to Manitobans on Performance 2005 Discussion Document, and is meant for illustrative purposes only.

73

Exercise 2: Prepare Narrative

• Work with partner(s) from previous exercise• Fill out narrative table

– For the high-level objectives measure you worked with in previous exercise complete the narrative table provided

– Use the paper copy of the table to draft your narrative. – Transfer your key points on to an overhead transparency

• Plenary Presentation and Discussion– Two or three groups will be asked to present their work– Plenary discussion on narratives and questions regarding

current annual reporting requirements


74

Narrative Worksheet

A. What are we measuring and how?

B. Why is it important to measure this?

C. What is the most recent available data value for this indicator?

D. What is the trend over time for this indicator?

E. Comments/ recent actions / report links


75

Closing Statements and Next Steps

• Workshop evaluation process

• First thoughts from the group on further capacity building?

76

ReferencesAnielski, M. (2006). Alberta examples and observations are based on reflections on

Alberta’s Government Performance Measurement and Accountability system (Measuring Up) based on conversations with Murray Lyle, head of Alberta Finance’s Performance Measurement team on March 2, 2006.

Government of Alberta 2004-05 Annual Report – Measuring Up. [http://www.finance.gov.ab.ca/publications/measuring/measup05/index.html]

Government of Saskatchewan. Government Accountability Framework. [http://www.gov.sk.ca/finance/accountability/default.htm]

Manitoba Provincial Sustainability Report. Government of Manitoba [http://www.gov.mb.ca/conservation/sustainabilityreport/]

Oregon Progress Board: www.oregon.gov/DAS/OPB

Rita Conrad 2005. Results Oriented Strategic Planning: A Framework for Developing Effective Performance Measurement. Presentation to the Botswana Delegation, Salem, Oregon, September 29, 2005.

Rita Conrad 2005. Oregon’s Experience with Performance Reporting. AGA’s First National Performance Management Conference. Oregon Progress Board, November 14 2005.

Reporting to Manitoba on Performance: 2005 Discussion Report. [http://www.gov.mb.ca/finance/mbperformance/index.html]

http://www.oregon.gov/DAS/OPB

http://www.oregon.gov/DAS/OPB

Documents

Workshop: Development and Use of Outcome-based Measures in Government Planning & Reporting