Mapping Research Impact Ian Viney

Medical Research Council

• Analysis/learning

• To better understand what leads to impact –

maximise return

• Accountability

• Efficient and effective use of funds -

Government, the public etc.

• Advocacy

• Explaining delivery, making the case for

continued funding

• Allocation

• Prioritising research areas for support

Why Measure Research Impact?

Developing a research evaluation framework http://www.rand.org/content/dam/rand/pubs/research_briefs/RB9700/RB9716/RAND_RB9716.pdf

Input Outputs/Outcomes Impacts (Academic/Economic/Social)

Improvements to health (living longer and with better quality of life)

Academic impact (effects on further research including other disciplines)

Improving the performance of existing businesses

Creating new businesses (that contribute to economic growth and further R&D)

Delivering highly skilled people to the labour market

Attracting R&D investment (from global business and non-UK funding sources)

Improving public policy and public services

Engaging public support for medical research

Pathways to Impact

Funding for Research and Training

Generation of new Knowledge/publication

Trained people

Intellectual property/ licensing

Influences on policy & practice

Development of new products/processes

Dissemination of research

Development of collaborative networks

Research materials/ technologies

Measurement Challenges

• Causality – link between inputs and outputs/impact non-linear and complex (“chain-link”/”payback” models)

• Attribution – how much of the benefit results from research input, as opposed to non-research inputs (e.g. marketing, “spillovers” etc.)

• Cross country effects – collaboration, co-funding, mobility of researchers

• Timescales – research might take decades to lead to impact, premature measurement will over-emphasise policies encouraging research bringing short-term benefit

The Benefits from Publicly Funded Research (Martin and Tang, 2006) https://www.sussex.ac.uk/webteam/gateway/file.php?name=Fac-BRM-UMIP&site=25

Macro economic approaches – whole economy studies

1989

1987

1988

19901991

1992

20001999

2001

1998 1997

2002

1993

19951996

1994

2003

2004

2005 20072006

-.01

0

.01

.02

.03

.002 .0025 .003 .0035RCoun_RD_y_n_L1

Smoothed TFPG Fitted values

Smoothed TFPG and RCouncil R & D spend • Total factor productivity growth is the extent to which growth occurs over and above that due to additional investment in capital, private sector research and development etc.

• There is a good correlation

between Research Council expenditure and TFPG, but weak or no correlation between TFPG and civil or defence R&D

• “Spillover” benefit is high from

Research Council research

Public support for innovation, intangible investment and productivity growth in the UK market sector (Haskell and Wallis, 2010) http://spiral.imperial.ac.uk/bitstream/10044/1/5280/1/Haskel%202010-01.pdf

“A reduction in Research Council spend of £1bn may reduce UK GDP by £10bn”

Micro-economic approaches Time, Attribution and Impact (Alan Hughes, 2011)

Defined, structured and prospectively collected output information

• Trace interactions between the producers and users of research, establishing evidenced links across as many steps as possible

• Publications may have a minor impact in transferring knowledge to industry, people exchanges and problem solving (collaboration) are more important

• Similarly focussing on hard commercialisation metrics (spin outs, patents etc.) miss a large range of activities (e.g. policy influences)

• Challenge is to capture robust details of collaborations and other processes effectively (using unambiguous categories, clear guidance etc.)

• Failures also have to be factored in (e.g. lack of productivity, absence of output), and so effort has to be made to capture feedback from all eligible researchers

• Presents an opportunity to signal to the research community the outcomes that funding agencies value

• Researchfish™ is the system developed used to collect feedback

from MRC-funded researchers about the outputs, outcomes and

impact of their research.

• Researchers can update their record of output in Researchfish™ at

any time, but are asked to confirm that it is complete during the

annual submission period (usually October – November).

• Researchfish™ allows researchers with multiple funders to enter

an output just once and attribute it to more than one award

from more than one funder

• Researchfish™ has proved to be an invaluable tool in enhancing

our understanding of how MRC research improves the health and

wellbeing of society through delivering economic, academic and

social impact

• As at August 2013, there are approximately 80 research

organisations and funders using Researchfish™, including 54

medical research charities and 11 universities. Approximately

£3bn of new research projects and programmes are added to

the dataset to be tracked each year

• Use of the Researchfish approach is being considered by other

organisations outside of the UK

Researchfish™ is becoming an established UK approach for funding agencies to capture research output

• Consistent time series for medical research funding in cardiovascular disease and mental health research from 1975-1992

• Clear conceptual framework relating to GDP gain from “spillovers”

• Estimation from literature of the magnitude of these effects

• Development and application to CVD and MH, of a ‘bottom-up’ approach to estimate health gain in terms of QALYs

• Analysis of UK guidelines to provide indicators of elapsed time (lags) and proportion of benefits attributable to UK

• Suggestions for developing research agenda Strong quantitative argument for investment in medical research

Medical Research: What’s it worth? (2008) - A hybrid macro/micro economic study

Case Study - DNA Technologies UK academic research has created a multi-billion dollar global market

The structure of DNA

Famously described by Watson and Crick working in MRC laboratories

DNA Sequencing

MRC scientist Fred Sanger invents the first viable DNA sequencing technique

The Southern Blot

MRC scientist Ed Southern invents a technique to analyse DNA fragments that still underpins molecular biology today

1995 Oxford Gene Technologies (OGT) established to commercialise Ed Southern’s microarray technology

1998 Nematode Genome Sequence UK and US researchers including the MRC Laboratory of Molecular Biology’s Dr Sydney Brenner and Sir John Sulston finished sequencing the C. elegans genome – the first complete sequence of a multicellular organism lays the foundation for the human genome project

2003 the Human Genome Project completes the human genome sequence

2005 Oxford Nanopore Technologies (ONT) a spin out based on MRC and EPSRC funded research is bringing to market a transformative next generation sequencing approach

2007 Solexa a Cambridge spin out based on BBSRC funded work on SBS sequencing is acquired by US company Illumina

Imperial College spin out DNAe licenses technology to Life Technologies – they develop and market the sequencing approach as Ion Torrent

2005, the UK Forensic Science Service used DNA fingerprinting to match 40,000 samples from crime scenes to people’s records on the National DNA Database, including 165 homicides, 100 attempted murders and 570 rapes.

Worldwide market based on genomics and gene-based services is estimated at £3.5bn

2012 OGT employs 60 staff near Oxford and opens a New York office

2011 An economic analysis suggests that the US $3bn investment in the HGP has provided almost $800bn of benefit to the US alone

2013 ONT employs 90 staff in Oxford and Cambridge, and has raised £105m since formation in 2005

2012 Roche launches an unsuccessful hostile bid for Illumina of $6bn

2013 Life Technologies is acquired by Thermo Fisher for $14bn

1953

2000

1970

1990

1980

1984 DNA fingerprinting technology invented by MRC-funded scientist Sir

Alec Jeffreys at the University of Leicester. Led

to research into genetic markers of human diseases

2010

DNA Fingerprinting has revolutionised forensic science, matched donors to patients in

need of lifesaving organ transplants, and provided an infallible method of paternity

testing

Strategic Evaluation Research Studies

• The MRC is encouraging academic researchers to take an interest in designing new approaches to evaluating impact – two rounds of funding since 2012

• This research should improve the current estimates of the economic return and more fully exploit the growing body of high quality data about output.

• In 2013 this new “science of science policy” programme grew to four small studies, which we intend to add to each year.

• MRC Call for proposals to understand better the link between research and impact http://www.mrc.ac.uk/Fundingopportunities/Highlightnotices/MRCEconomicImpact/MRC009122

Discussion points

• NIH plans following its scientific management review board consideration of the issue

• Encouraging greater exchange of ideas between evaluation staff across international research organisations

• Potential for collaboration on cross-organisation analyses/international comparisons

• Replication of studies in different countries

• Need for common approaches – data models/definitions

• Ideas to strengthen the embedding of evaluation analyses in the funding/strategy development life-cycle