Copyright © 2012

American Institutes

for Research.

All rights reserved.

Building a Scientific Basis for Research Evaluation

Rebecca F. Rosen, PhDSenior Researcher

Research Trends Seminar

October 17, 2012

2

Outline

• Science of science policy• A proposed conceptual framework• Empirical approaches:

NSF Engineering Dashboard ASTRA – Australia HELIOS – France

• Final thoughts

3

Outline

• Science of science policy• A proposed conceptual framework• Empirical approaches:

NSF Engineering Dashboard ASTRA – Australia HELIOS – France

• Final thoughts

4

The emergence of a science of science policy

• Jack Marburger’s challenge (2005)

• Science of Science & Innovation Policy Program at the National Science Foundation (2007) An emerging, highly interdisciplinary research field

• Science of Science Policy Interagency Task Group publishes a “Federal Research Roadmap” (2008): The data infrastructure is inadequate for decision-

making

• STAR METRICS (2010)

5

Why a science of science policy?

• Evidence-based investments• Good metrics = good incentives• Science is networked and global

• Build a bridge between researchers and policymakers• Researchers ask the right questions

• The adjacent possible: leverage existing and new research and expertise • New tools to describe & measure

communication

6

The timing is right:

7

A conceptual framework for a science of science policy

8

Getting the right framework matters

• What you measure is what you get Poor incentives Falsification

• Usefulness• Effectiveness

9

A proposed conceptual framework

Adapted from Ian Foster, University of Chicago

10

A framework to drive person-centric data collection

WHO is doing the research

WHAT is the topic of their research

HOW are the researchers funded

WHERE do they work

With WHOM do they work

What are their PRODUCTS

11

Challenge – The data infrastructure didn’t exist

However, some of the data do exist

12

Empirical Approaches

Leveraging existing data to begin describing results of the scientific enterprise

13

An empirical approach

• Enhance the utility of enterprise data• Identify authoritative “core” data elements• Develop an Application Programming

Interface (API) Data platform that provides programmatic

access to public (or private) agency information

• Develop a tool to demonstrate value of API

14

Topic modeling: Enhancing the value of existing data

David Newman - UC Irvine

NSF proposals

NSF proposals Topic Model:

- Use words from(all) text

- Learn T topics

t49t18t114t305

t49t18t114t305

Topic tags for each and every proposal

Automatically learned topics (e.g.):

…

t6. conflict violence war international military …

t7. model method data estimation variables …

t8. parameter method point local estimates …

t9. optimization uncertainty optimal stochastic …

t10. surface surfaces interfaces interface …

t11. speech sound acoustic recognition human …

t12. museum public exhibit center informal outreach

t13. particles particle colloidal granular material …

t14. ocean marine scientist oceanography …

…

15

Stepwise empirical approach

• Enhance the utility of enterprise data• Identify authoritative “core” data elements• Develop an Application Programming

Interface (API) Data platform that provides flexible,

programmatic access to public (or private) agency information

• Develop a tool to demonstrate value of API

16

17

Stepwise empirical approach

• Enhance the utility of enterprise data• Identify authoritative “core” data elements• Develop an Application Programming

Interface (API) Data platform that provides programmatic

access to public (or private) agency information

• Develop a tool to demonstrate value of API

18

Outline

• Science of science policy• A proposed conceptual framework• Empirical approaches:

NSF Engineering Dashboard ASTRA – Australia HELIOS – France

• Final thoughts

19

20

21

22

23

24

Outline

• Science of science policy• A proposed conceptual framework• Empirical approaches:

NSF Engineering Dashboard ASTRA – Australia HELIOS – France

• Final thoughts

25

Linking administrative and grant funding data in Australia

26

Outline

• Science of science policy• A proposed conceptual framework• Empirical approaches:

NSF Engineering Dashboard ASTRA – Australia HELIOS – France

• Final thoughts

27

People People

Describing public-private partnerships in France

28

What does getting it right mean?

• A community driven empirical data framework should be: Timely Generalizable and replicable Low cost, high quality

• The utility of “Big Data”: Disambiguated data on individuals

- Comparison groups

New text mining approaches to describe and measure communication

??

29

Final thoughts

30

Policy makers can engage SciSIP communities:

• Patent Network Dataverse; Fleming at Harvard and Berkeley

• Medline-Patent Disambiguation; Torvik & Smalheiser at U Illinois)

• COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science); Zucker & Darby at UCLA

31

The power of open research communities

• Internet and data technology can transform effectiveness of science: Informing policy Communicating science to the public Enabling scientific collaborations

• Interoperability is key

• Publishers are an important part of the community

32

Rebecca F. Rosen, PhD

E-Mail: rrosen@air.org

1000 Thomas Jefferson Street NWWashington, DC 20007

General Information: 202-403-5000TTY: 887-334-3499

Website: www.air.org

THANK YOU!

33