1

Improving Measurement and Reporting on the Impacts of Federal Science and Technology

Presentation to the FPTT 2008 National MeetingJune 3, 2008

André DownsDirector General

About the Policy Research Initiative• Horizontal policy research that supports strategic policy

development and medium-term planning

• Overseen by a Deputy Ministers Steering Committee

• Support CCDM policy committees and PCO planning

• Research portfolio: economic policy social policy climate change environmental scanning (e.g., Canada@150)

2

Summary DiagnosticConducting policy research with federal departments and agencies on measuring and reporting on the impacts of federal S&T is essential

– a key commitment under the federal S&T Strategy announced in May 2007

– federally-performed S&T is considerable (about $5 billion annually)

– a significant recent policy literature to draw from

3

The Project in 1 Slide• Draw implications from recent policy research, the

experience of practitioners, and from original analysis, to suggest how the impact of the S&T performed by the federal government can be better measured and reported, and thereby potentially enhanced over time.

• A forum or “neutral-space” for federal departments and agencies to identify and explore: concepts that can be applied to draw a more complete

picture of the diverse impacts of S&T activities performed by the federal government;

options for improving reporting on those activities; and approaches for how to maximize the benefits to

Canadians of those activities in relation to costs.

4

Governance• Oversight by an inter-departmental

steering committee

• Consensus-driven

• Preliminary findings tested inside and outside the federal government

• Public report by February 2009

5

Project Deliverables1. Research report on how publicly-funded research organizations

transmit technology and knowledge to innovators. (StatCan lead)2. Identify and assess approaches for additional reporting. (PRI lead)3. A “think piece” on identifying, describing, and measuring the

impacts of federally-performed S&T. (IC lead)4. Synthesis reports on the market and non-market impacts on

Canadians of federal S&T investments. (PRI lead)5. Four events to present and discuss results.6. Lessons learned and best practices for assessing and reporting

impacts. (NRC lead)7. Additional work on non-market impacts. (EC lead)

6

1. Public labs and innovators• Research report by Frances Anderson from

Statistics Canada

• Applies international definitions and data from the 2005 Canadian Survey of Innovation in Manufacturing to examine how public labs (universities, federal labs, and provincial labs) transmit knowledge and technology to innovative firms

7

1. Public labs and innovators

Source Some importance

High importance

Clients or customers 91.2%* 51.9%*

Suppliers 89.7%* 33.4%*

Conferences, trade fairs and exhibitions 81.7%* 14.3%*

Universities 45.3% 3.8%

Federal research labs 35.2% 1.3%

Provincial research labs 34.0% 1.0%

Percent of Manufacturing Innovators Indicatingthat the Source was Important for Innovation

Source: Statistics Canada, Survey of Innovation 20058

1. Public labs and innovators• Sectors more likely to identify public labs as important:

natural resource processing machinery and equipment aerospace

• The innovators most strongly linked to public labs tended to:

be larger; have employees devoted to R&D; be engaged in more innovation activities; collaborate more with public and private organizations;

and receive funding from more public and non-conventional

private sources.

9

1. Public labs and innovators

Percent of Manufacturing Innovators Indicating thattheir Innovations had an Important Impact, by Impact Area

Impact Strongly linked

Not strongly linked

Improving job quality 43.1%* 33.2%Reducing environmental impacts 29.4%* 18.2%Improving health and safety 42.1%* 31.5%Meeting regulatory requirements 40.7%* 26.6%Reducing materials or energy per unit output

42.2%* 26.7%

Source: Statistics Canada, Survey of Innovation 2005

10

2. ReportingDefinitions

Resources are allocated to …

… Activities that lead directly to …

… Outputs that are intended to achieve …

… Outcomes (or Results) that benefit the country.

Impacts encompass Outputs and Outcomes

11

2. ReportingProportion of Federal Technology Commercialization Programs

by Type of Indicator in Estimates and Related Documents

All programs Larger programs

None 51% 35%

Activity* 19% 27%

Output 17% 19%

Outcome 13% 19 %

Total 100% 100%

* Funded activity described, but no indication provided of its impact

Source: PRI research, based on a database prepared by Industry Canada reported in The Impact Group (July 2005), Commercialization Activities of the Federal Government: Program Synopsis.

12

2. ReportingThe Relationship among Planning,

Reporting, and Impact Type

Kinds of Impact

Outcome Activity Output

PlanningPlanned outcomes

≈ Objectives ≈ Strategic outcomes

Planned activities

Planned outputs

Reporting

Results of evaluations, audits and

reviews

Actual activities

Actualoutputs

13

2. ReportingPossible Approaches

1. Explicitly incorporate S&T in the Program Activity Architectures (PAAs) of departments and agencies by:A. Treating S&T as an internal service;B. Portraying S&T activities as a single program within each

department and agency;C. Describing S&T in terms of activities that in some cases fall

under broader programs; orD. Describing S&T in terms of projects (where possible).

2. Expand the annual Statistics Canada survey to collect data in terms of the higher-order PAA categories of departments and agencies.

3. Prepare one government-wide S&T report.

4. Prepare annual S&T reports for each department and agency.

14

3. Think piece on identifying, describing and measuring the impacts of federally-performed S&T

• Federal S&T Strategy committed to increasing accountability by improving ability to measure impacts of federal S&T investments

• Considers lessons learned based on past efforts by federal S&T performers to measure impacts

• Paper designed to help better capture full range of impacts for federally-performed S&T

– Analytical framework linking impacts to Strategy’s Entrepreneurial, Knowledge and People Advantages

– Currently being finalized

15

4. Synthesis reportsThe Market Impacts of Federal S&T Investments

• No recently prepared synthesis has fully reflected the sizeable empirical literature

• Report will identify: Policy implications based on the synthesis; The methodologies that have been most

successful in deriving results; Outstanding policy-relevant research questions;

and Suggestions for further research and data

development.

16

4. Synthesis reportsThe Non-Market Impacts of Federal S&T Investments

• Report will provide: An overview of federal S&T; Conceptual discussion of the non-market impacts of

S&T investments; Literature review with a focus on health, safety and

security, and the quality of the environment; Empirical evidence on the linkages between market

and non-market impacts; Outstanding research issues and knowledge gaps; and Policy implications.

17

5. Project events• Preliminary findings in July 2008• Public Science and the Marketplace: Connecting,

Measuring and Reporting in September 2008 How do governments support researchers and

innovators to develop new ideas and bring them to the marketplace, and how can they measure and report on these efforts?

Presentations by Howard Alper, Chair of the Science, Technology and Innovation Council, Ken Carlaw, Richard Lipsey's co-author and collaborator, and Paul Johnston, President and CEO of Precarn, among others

• Synthesis reports in October 2008• Draft project report in early 2009

18

Next steps

• Further work on deliverables

• Events from July 2008 to early 2009

• Final report by February 2009

19