Building a Scientific Basis for Research Evaluation

  • Published on
    08-Feb-2016

  • View
    20

  • Download
    0

Embed Size (px)

DESCRIPTION

Building a Scientific Basis for Research Evaluation. Rebecca F. Rosen, PhD. Senior Researcher. Research Trends Seminar October 17, 2012. Outline. Science of science policy A proposed conceptual framework Empirical approaches: NSF Engineering Dashboard ASTRA Australia HELIOS France - PowerPoint PPT Presentation

Transcript

Tips for Using This Template

Building a Scientific Basis for Research EvaluationRebecca F. Rosen, PhDSenior ResearcherResearch Trends SeminarOctober 17, 2012Copyright 2012 American Institutes for Research.All rights reserved.Good morning I am RR, a SR at AIR, where we are building international capacity for evidence-based science policy

The session title is approaches to the economic impact of science investments. However, I am going to take a few steps back for my talk today.

In order for researchers and policymakers to begin to understand the economic impacts of science investments, we first need to do a better job of characterizing those investment portfolios and then describing the results of scientific work over time,

since science and its outcomes can take a very long time

1OutlineScience of science policyA proposed conceptual frameworkEmpirical approaches:NSF Engineering DashboardASTRA AustraliaHELIOS FranceFinal thoughts#I will discuss some work from the science of science policy research field to enhance our ability to describe scientific results, and these emerging methodologies might eventually be harnessed to describe economic impact of science investments.

2OutlineScience of science policyA proposed conceptual frameworkEmpirical approaches:NSF Engineering DashboardASTRA AustraliaHELIOS FranceFinal thoughts#I will discuss some work from the science of science policy research field to enhance our ability to describe scientific results, and these emerging methodologies might eventually be harnessed to describe longer term societal impacts of science investments.

3The emergence of a science of science policyJack Marburgers challenge (2005)

Science of Science & Innovation Policy Program at the National Science Foundation (2007)An emerging, highly interdisciplinary research field

Science of Science Policy Interagency Task Group publishes a Federal Research Roadmap (2008):The data infrastructure is inadequate for decision-making

STAR METRICS (2010)

#A very quick history of the development of this field from a US perspective:

Dr. Marburger called for the creation of a community of practice that would create the data sets, tools, and methodologies needed to assist science policy decision makers as they invest in Federalresearch and development and make science policy decisions.

In 2005 Jack Marburger, Science Adviser to President George W. Bush, gave a keynote address to the American Association for the Advancement of Science in which he described the relationship between basic and applied research as much more complex than is usually shown by the linear model of science and technology policy. In this model knowledge from basic science research goes through a linear process of application and market innovation, but the model does a poor job describing how science works. In his speech Marburger called for "better benchmarks" to understand and predict the relationship between science and societal benefit.

The direct result of this speech, and a subsequent editorial by Marburger in the journal Science, was the institution of a new, highly interdisciplinary research program at the National Science Foundation called the Science of Science and Innovation Policy (SciSIP). Goals of SciSIP:

To understand the contexts, structures and processes of S&E research, to evaluate reliably the tangible and intangible returns from investments in research and development (R&D), and to predict the likely returns from future R&D investments within tolerable margins of error and with attention to the full spectrum of potential consequences.

"Specifically, the research and community development components of SciSIPs activities will: (1) develop usable knowledge and theories of creative processes and their transformation into social and economic outcomes; (2) develop, improve and expand models and analytical tools that can be applied in the science policy decision making process; and (3) develop a community of experts across academic institutions focused on SciSIP.

At the same time, they joined the SOSP ITG in response to the Marburger challenge and developed a federal roadmap

2 Key Roadmap findings, the second spurred the development of SMDifferent data models, data, and tools in use at the agencies to understand their investments

SM was founded in 2010, catalyzed by the ARRA reporting requirements, NIH NSF and OSTP, etc, as an effort to collect better, richer data on the science & engineering enterprise and develop new tools to demonstrate the value of high quality data infrastructureWhile the SciSIP academic community moved forward to crystallize the framework, tools and methodologies of the new research field, the agencies looked inward to understand, enhance, and eventually interlink their grant and HR data systemsDevelop an empirical basis of science policy

4Why a science of science policy?Evidence-based investmentsGood metrics = good incentivesScience is networked and global

Build a bridge between researchers and policymakersResearchers ask the right questions

The adjacent possible: leverage existing and new research and expertise New tools to describe & measure communication

#Why do we need a science of science policy? Fewer resources are available while science costs are increasing. We need a scientific approach to spend the money wisely. This will help to inform new incentive structures for research

There are decades of excellent work in the science and tech studies but now we need to pull in the policymakers to guide the researchers to be collecting the data and building the tools to ask the relevant policy questions. Policymakers are part of the community, finally bridging the gap between social science research and policy

Finally, we are at a time when we are building significant expertise in new research fields that are deriving meaning out of massive data sets.The interdisciplinary community of researchers and policymakers will leverage years of research and expertise in social sciences and computer and information science, among other fields5The timing is right:

#Scientific advances (graph theory; RCT)New application (social networks)New ways of communicating knowledgeNew application (graph oriented databases)New data (natural language processing; computational linguistics)=> Potential for new science, new scientific field and theoretically grounded, metrics

6A conceptual framework for a science of science policy#An emerging research field needs to agree upon a common conceptual framework to describe how the system of study works, at its most basic unit of analysis

The overarching goal of SoSP is to create a community of practice of social scientists, policy makers, researchers, and technical experts, so that we can make better decisions about science, from a policy view and from a researcher view.Efforts like CASRAI will significantly contribute to the SoSP community of practice, reducing burden and increasing performance capabilities for community members

7Getting the right framework mattersWhat you measure is what you getPoor incentivesFalsificationUsefulnessEffectiveness

#8A proposed conceptual framework

Adapted from Ian Foster, University of Chicago#The emerging, interdisciplinary field of science of science policy is beginning to shine a light on the black box between science research and the broader impacts of scientific knowledge on society. A conceptual framework for this research should describe the people, or groups of people, who create, transmit, and adopt scientific knowledge. If we can begin to describe these phases of knowledge transfer, eventually we might be able to examine the effects of exogenous interventions (such as funding) on the outcomes of scientific research.

What is most important about this framework here is that the person is the unit of analysis. By developing a science of science policy based on people and their knowledge production transmission and adoption, we can encourage the appropriate incentive structures for the scientific enterprise

Here is this framework, funding is the intervention that affects how people, or teams of people, create and transmit knowledge.

Efforts like CASRAI will significantly contribute to the SoSP community of practice, reducing burden and increasing performance capabilities for community members

9A framework to drive person-centric data collectionWHO is doing the researchWHAT is the topic of their researchHOW are the researchers fundedWHERE do they workWith WHOM do they workWhat are their PRODUCTS

#This framework informs the type of data collection we will need to describe and then analyze the scientific ecosystem.

Most important, we want to have information on who does the research. When we identify data sources to answer the other questions listed here, we can start to link these data sources to create a richer data infrastructure that will enable the application of new analytical tools

The CASRAI standards will ensure that these types of data infrastructures efforts will not exist in silos around the world.

10Challenge The data infrastructure didnt exist

However, some of the data do exist#That said, as evidenced by the findings in this report that I mentioned early, such a data infrastructure didnt exist.

But lots of data to answer those questions do exist. And the SOSP community is leveraging these data sources to begin describing the scientific ecosystem, how scientific ideas are created, transmitted and adopted, and how various interventions influence the various outputsIT WILL TAKE A MASSIVE EFFORTS TO BRING THE DATA TOGETHER

11Empirical Approaches Leveraging existing data to begin describing results of the scientific enterprise#With numerous collaborators, our team has demonstrated the value of a person-centric data infrastructure in describing scientific research portfolios and some of their results. I will describe our collaborative efforts in the U.S., France, and Australia to integrate existing administrative, programmatic, and results databases into data platforms that feed novel portfolio visualization tools. In building such data infrastructures, science of science policy researchers have shown how new technologies can be used to leverage and add value to existing data sources. Further, when data platforms are developed by an open community with open technologies, they can be highly flexible and extensible for use by various researcher and policy stakeholders. 12An empirical approachEnhance the utility of enterprise dataIdentify authoritative core data elementsDevelop an Application Programming Interface (API)Data platform that provides programmatic access to public (or private) agency informationDevelop a tool to demonstrate value of API#With numerous collaborators, our team has demonstrated the value of a person-centric data infrastructure in describing scientific research portfolios and some of their results. I will describe our collaborative efforts in the U.S., France, and Australia to integrate existing administrative, programmatic, and results databases into data platforms that feed novel portfolio visualization tools. 13Topic modeling: Enhancing the value of existing data David Newman - UC IrvineNSF proposalsTopic Model: Use words from(all) text Learn T topicst49t18t114t305

Topic tags for each and every proposalAutomatically learned topics (e.g.):t6. conflict violence war international military t7. model method data estimation variables t8. parameter method point local estimates t9. optimization uncertainty optimal stochastic t10. surface surfaces interfaces interface t11. speech sound acoustic recognition human t12. museum public exhibit center informal outreacht13. particles particle colloidal granular material t14. ocean marine scientist oceanography #14Stepwise empirical approachEnhance the utility of enterprise dataIdentify authoritative core data elementsDevelop an Application Programming Interface (API)Data platform that provides flexible, programmatic access to public (or private) agency informationDevelop a tool to demonstrate value of API#

#16Stepwise empirical approachEnhance the utility of enterprise dataIdentify authoritative core data elementsDevelop an Application Programming Interface (API)Data platform that provides programmatic access to public (or private) agency informationDevelop a tool to demonstrate value of API#OutlineScience of science policyA proposed conceptual frameworkEmpirical approaches:NSF Engineering DashboardASTRA AustraliaHELIOS FranceFinal thoughts#First I will talk about work in the united states to link administrative data with research funding data, in order to describe the local and national workforce directly supported by federal investments in science18

#

#

#

#

#OutlineScience of science policyA proposed conceptual frameworkEmpirical approaches:NSF Engineering DashboardASTRA AustraliaHELIOS FranceFinal thoughts#First I will talk about work in the united states to link administrative data with research funding data, in order to describe the local and national workforce directly supported by federal investments in science24Linking administrative and grant funding data in Australia

#OutlineScience of science policyA proposed conceptual frameworkEmpirical approaches:NSF Engineering DashboardASTRA AustraliaHELIOS FranceFinal thoughts#First I will talk about work in the united states to link administrative data with research funding data, in order to describe the local and national workforce directly supported by federal investments in science26People People Describing public-private partnerships in France

#What does getting it right mean?A community driven empirical data framework should be:TimelyGeneralizable and replicableLow cost, high qualityThe utility of Big Data:Disambiguated data on individualsComparison groupsNew text mining approaches to describe and measure communication??

#If we have a community-driven, international empirical data infrastructure for science policy analysis, we have the opportunity to get up-to-date information on who is doing what and with whom. It is reliable data for analyses by many groups, and data can be linked in a way that saves money and leverages local curation efforts

Nexus of technology, open communities, big data, etc - When the community and the system work.

End the slide discussing text mining approaches that enable us to measure communication, the currency of science28Final thoughts#Policy makers can engage SciSIP communities:Patent Network Dataverse; Fleming at Harvard and Berkeley

Medline-Patent Disambiguation; Torvik & Smalheiser at U Illinois)

COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science); Zucker & Darby at UCLA

#Just a sample of some of the communities who are working to build better data...

Recommended

View more >