Measuring S&T Performance atEnvironment Canada

Eric GagnéA/ Director, Science Policy DivisionEnvironment Canada

PIPSC Science Policy SymposiumMay 13, 2010

Page 2

Outline

• Context– Why measure S&T performance?– Overview of S&T at Environment Canada

(EC)• Measuring R&D performance:

Methods and results from the first report (2009)

– Available on-line at http://www.ec.gc.ca/scitech(under “Managing S&T”)

• Measuring Related Scientific Activities (RSA) performance: A work in progress

Measuring the Performanceof Related Scientific

Activities

(2009)

(2011)

Page 3

Why measure S&T performance at EC?

• EC Science Plan commitments (2007):“…performance measures will provide information to monitor progress, inform future editions of the Plan, and ensure a process of continuous improvement within Environment Canada’s science.”

• Federal S&T Strategy (2007): “Canada’s federal government will increase its accountability to Canadians by improving its ability to measure and report on the impacts of its S&T expenditures.”

• Better management of horizontal S&T issues

• Strategic positioning of S&T to inform large-scale reviews / audits and to communicate EC S&T activities.

Page 4

R&D / RSA Overview

associated with “public good” science –monitoring, risk

assessments, scientific service delivery, surveys, standards development,

etc.

Research & Development:“Creative work undertaken on a systematic basis in order to increase the stock of scientific and technical knowledge and to use this knowledge in new applications.”

Related Scientific Activities:“Those activities which complement and extend R&D by contributing to the generation, dissemination and application of scientific and technological knowledge.”

• R&D and RSA have distinct functions but are highly interconnected and inform one another.

• As defined by the OECD / Statistics Canada:

associated with the innovation chain and

new knowledge creation

Page 5

Overview of R&D / RSA at EC

RSA ~45% of EC expenditures ($450M) and ~40% of personnel (2500 full-time equivalents).

R&D ~25% of EC expenditures ($250M) and ~15% personnel (1000 full-time equivalents)

EC is the largest performer of natural science RSA in the federal government.

EC is one of the top 10 environmental R&D institutions in the world, and the top producer of environmental research in Canada (2006).

Directly allows EC to deliver on its mandate, through monitoring, testing, prediction, etc

Provides the basic knowledge, credibility to support EC’s regulations, policies, programs, etc.

Related Scientific Activities (RSA)R&D

Science-Metrix (2006)

Page 6

Challenges for measuring S&T at EC

• All Strategic Outcomes, and Program Activities rely on S&T

• With current reporting structures, the challenge is to measure S&T as a horizontal activity

Strategic Outcomes

Program Activities

Sub-activities

Sub-sub-activities

LEGEND

Strategic Outcomes

Program Activities

Sub-activities

Sub-sub-activities

LEGEND

Environment Canada’s Program activity architecture (PAA)

Page 7

Phased approach to measuring S&T

• Phase 1: Measurement of R&D performance, using readily accessible quantitative and qualitative indicators

EC developed a logic model and performance measurement framework for R&D, using four principles articulated in the EC Science PlanThe 2009 report is the first of its kind at EC.

• Phase 2: Measurement of RSA performance Lead by EC, the Interdepartmental Working Group is developing a new logic model, framework and preliminary set of indicators to measure federal RSA.Report on RSA at EC to be published in 2011.

• Phase 3: Measurement of EC’s S&T management and governance (S&T Policy)

• Phase 4: Integration of S&T and policy performance measures

COMPLETED

IN PROGRESS

PLANNED

Page 8

Phase 1: Measuring R&D performance

• Reviewing current best practices and in collaboration with scientists, managers, performance measurement experts across EC, we developed a framework.

• Each principle implied the development of several key qualitative and quantitative indicators:

–Bibliometrics (collaborations, citation rates, etc.)

–Surveys–Quantitative HR information–Expenditures and personnel

(StatCan)EC Framework for R&D Performance Measurement

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 9

A “dashboard” performance measurement of R&D

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 10

Risks assessment

Aquatic ecosystems protection / conservation

Environmental monitoring

Strategic approaches

Governance and policy coordination of ecosystem approachWildlife protection / conservation

Risk management

Risk mitigation and implementation

Weather and environmental prediction research

Information on pollutants and GHGs

Alignment: Linking R&D with users

1A11A2

1B1

1C1

1C21C3

2A1

2A22A3

2B1

2B2 2B3

2B5

3A23A4

3A53B1

3B2

3C1

3C2 3C3

6B4

7A2

7A4

7B2

7B3

7B4

• Strong horizontal links between R&D users and producers at EC

• R&D “hubs” often correspond to government-wide priorities

• Internal functions (policy, regulation, etc.) as the main users of EC’s R&D, found it both timely and responsive

Conservation of the environment

Environmental protection

Internal services

Weather and climate services

R&D links across EC’s PAA showing the connections between producers and users of R&D (2009)

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 11

Linkages: Increasing collaboration rates• We revealed collaborations

with key sectors, geographic areas and scientific specialties

• We found strong connections with universities in particular (over half of all papers published)

• Collaboration greatly increases the number of citations a paper receives

EC’s collaboration rates (2003-2007)

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 12Collaborations between EC branches / directorates and with partners (Figure produced by Science-Metrix, 2009)

Linkages: National collaboration networks

Note: Thickness of lines and size of circles (EC Directorates / Branches) indicate the number of collaborations

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 13

Excellence: Benchmarking productivity and impact• EC environmental

research publications area are cited 40% more often than the world average

• EC benefits from particularly high levels of specialization in certain environmental research sub-fields

• Many high-impact publications in first-rate interdisciplinary journals

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 14

Enabling Environment: cross-cutting inputs that make a difference

EC’s R&D expenditures (current dollars) EC’s R&D Capital Expenditures (current dollars)

R&D personnel at EC

• Constant intramural expenditures• Increasing number of R&D

personnel• Recent increases in capital

spending

http://www.ec.gc.ca/scitech: “Managing S&T”

Page 15

Next steps: moving the “yardstick”forward• Our tools for R&D

measurement are largely transferable across science-based departments and agencies

• What about the rest of federal S&T?

• How can we adapt and extend existing tools and approaches to measure related scientific activities ?

0

1000

2000

3000

4000

5000

6000

7000

8000

1999-2000

2000-2001

2001-2002

2002-2003

2003-2004

2004-2005

2005-2006

2006-2007

2007-2008

2008-2009

*2009

-2010*

mill

ions

of d

olla

rs

R&D

RSA

Intramural, naturalsciences R&D

Intramural, naturalsciences RSA

Federal RSA / R&D expenditures (current dollars)

Towards RSA performance measurement…

Page 16

Environment Canada

Natural ResourcesCanadaHealth Canada

Fisheries and OceansCanadaOther

National ResearchCouncilIndustry Canada

AAFC

Canadian Museum ofNatureCFIA

24%

Environment Canada

24%Environment Canada

3%

4%

5%

8%

15%

17%

19%

24%

Environment Canada

Natural ResourcesCanadaHealth Canada

Fisheries and OceansCanadaOther

National ResearchCouncilIndustry Canada

AAFC

Canadian Museum ofNatureCFIA

Phase 2: Why measure RSA?

• To better account for $1.5B of expenditures on natural science RSA performed across the federal government

• To better understand “public good”S&T – RSA is almost exclusively performed by the governments and often directly supports science-based regulations / legislation, services to Canadians, etc

• To better situate RSA in the federal S&T and innovation system

• To better manage RSA as a cross-cutting horizontal function, addressing potential gaps/challenges and taking advantage of opportunities

Top federal spenders on intramural, natural sciences and engineering RSA (2006-07)

http://www.gcpedia.gc.ca:“Related Scientific Activities”

Page 17

A collaborative approach to develop a functional toolbox

• Measuring RSA is a government-wide challenge

• Our goal is to develop tools to measure RSA that will benefit the entire federal S&T community

• EC created a working groupto consult with evaluators, scientists, policy analysts, managers and corporate planners across SBDAs

• An open space for collaboration is provided on GCPedia

• Identify RSA performance measurement drivers(Management, Horizontality and Accountability), applications and intended audience.

• Provide background information on RSA (overall, within the federal government and within EC)

• Lay out the RSA Measurement Framework and Logic Model

• Introduce possible RSA Indicators, their range of applicability and how to use them

• Indicate a path forward for developing and using tailored performance measures

Process Expected results

http://www.gcpedia.gc.ca:“Related Scientific Activities”

Page 18

Summary

Expected result:• A 2011 report on EC’s RSA

performance that will contribute to better accounting for the “other half” of federal S&T

• Developing an RSA measurement toolbox, in collaboration with other Departments

Result:• Demonstration of EC’s

performance based on four framework elements

• Set of transferable tools for measuring RD&D developed

• The 2009 report: Measuring Environment Canada’s R&D Performance

Page 19

How can these and other approaches to performance measurement help us take stock of federal S&T activities and their impacts?

Discussion

For more information, visit www.ec.gc.ca/scitech (click on “Managing S&T”)Contribute to our development of our RSA performance measurement “toolbox”on www.gcpedia.gc.ca (search for “Related Scientific Activities”)Contact Eric Gagné at eric.gagne@ec.gc.ca