Mechanisms and pathways to impact in public health ... · health research and outcomes. The pathways to impact observed in our in-depth case studies include contributing to debates

RESEARCH ARTICLE Open Access

Mechanisms and pathways to impact inpublic health research: a preliminaryanalysis of research funded by the NationalInstitute for Health Research (NIHR)Harriet Boulding1 , Adam Kamenetzky1 , Ioana Ghiga2, Becky Ioppolo2, Facundo Herrera2, Sarah Parks2,Catriona Manville2, Susan Guthrie2 and Saba Hinrichs-Krapels1*

Abstract

Background: The mechanisms and pathways to impacts from public health research in the UK have not beenwidely studied. Through the lens of one funder (NIHR), our aims are to map the diversity of public health research,in terms of funding mechanisms, disciplinary contributions, and public health impacts, identify examples of impacts,and pathways to impact that existing reporting mechanisms may not otherwise have captured, and provideillustrations of how public health researchers perceive the generation of non-academic impact from their work.

Methods: A total of 1386 projects were identified as ‘public health research’ by the NIHR and listed in the NIHRPublic Health Overview database (2000–2016). From these, a subset of 857 projects were matched as potentiallyhaving begun reporting impacts via an external data-gathering platform (Researchfish). Data on the 857 projectswere analyzed quantitatively, and nine projects were selected to investigate further through semi-structuredinterviews with principal investigators. Two workshops took place to validate emerging and final findings andfacilitate analysis.

Results: In addition to the NIHR School for Public Health Research and the NIHR Public Health ResearchProgramme, 89% of projects contained in the NIHR Public Health Overview portfolio as ‘public health research’ arefunded via other NIHR research programmes, suggesting significant diversity in disciplines contributing to publichealth research and outcomes. The pathways to impact observed in our in-depth case studies include contributingto debates on what constitutes appropriate evidence for national policy change, acknowledging local ‘unintended’impacts, building trusted relationships with stakeholders across health and non-health sectors and actors,collaborating with local authorities, and using non-academic dissemination channels.

Conclusions: Public health as a discipline contributes substantially to impact beyond academia. To support thediversity of these impacts, we need to recognise localized smaller-scale impacts, and the difference in types ofevidence required for community and local authority-based impacts. This will also require building capacity andresources to enable impact to take place from public health research. Finally, support is required for engagementwith local authorities and working with non-health sectors that contribute to health outcomes.

Keywords: Research impact, Public health, Impact pathways, Research impact assessment

© The Author(s). 2020 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

* Correspondence: [email protected] Policy Institute, King’s College London, 22 Kingsway, London WC2B 6LE,UKFull list of author information is available at the end of the article

Boulding et al. BMC Medical Research Methodology (2020) 20:34 https://doi.org/10.1186/s12874-020-0905-7

http://crossmark.crossref.org/dialog/?doi=10.1186/s12874-020-0905-7&domain=pdf

http://orcid.org/0000-0003-3144-0108

http://orcid.org/0000-0002-1214-3338

http://orcid.org/0000-0001-9043-8847

http://creativecommons.org/licenses/by/4.0/

http://creativecommons.org/publicdomain/zero/1.0/

mailto:[email protected]

Background‘Impact’ from academic research can be defined differ-ently by policy and practice organisations, academia,commissioners, and funders, but it broadly refers to anychange or benefit to society beyond producing academicpublications. For example, UK Research and Innovation(UKRI) refers to impact as “the demonstrable contribu-tion that excellent research makes to society and theeconomy”.1 For the purposes of the UK 2014 and 2021Research Excellence Framework (REF), impact is definedas “an effect on, change or benefit to the economy, soci-ety, culture, public policy or services, health, the envir-onment or quality of life, beyond academia”. Analysisand assessment of these wider impacts from research al-lows for better allocation of research funding, creates ac-countability for research, and supports advocacyinitiatives in policy and practice [1]. In the case of publichealth research, few studies have been conducted thatspecifically demonstrate the contribution that the fieldhas made to society. This may be partly due to the scar-city of research impact analyses generally, or, specificallyfor public health, due to the diversity of ways in whichthe field is described. Although there is no global,organization-wide definition of ‘public health’, a com-mon point for scoping is the population health approachand the production of generalizable knowledge for futureinterventions [2]. For example, one of the most widelyused definitions is that of the WHO based on the workof Acheson [3], where public health is defined as “thescience and art of promoting and protecting health andwellbeing, preventing ill health and prolonging lifethrough the organised efforts of society”. As such, thereare many disciplines that could potentially contribute towhat may be considered public health benefits to society,or ‘public health impacts’.This diversity provides an opportunity to extend the

scope of ways that public health research contributes tosociety from a variety of disciplines and in variety ofpathways. Capturing and communicating these mecha-nisms can enable future researchers learn how to focustheir activities for impact to take place, and can helpfunders of health research to support and promote suchactivities to their award holders. Our motivation in thispaper is to contribute to the scarce evidence base onpublic health research impact analyses, to demonstratethe value of public health research in its diverse forms,and bring to light the mechanisms and pathwaysthrough which public health research contributes to so-ciety. Through this we hope to provide accountabilityand advocacy for the field and inform both researchersand funders on the types of mechanisms, pathways and

activities that could be supported to further encouragewider impacts from public health research.At the same time, we are mindful of the challenges

posed by conducting impact analysis for such a diversefield. Generally, the challenges in conducting impactanalyses relate to data access, time-lags and non-linearity of impact pathways, and attribution of impactsto research [4–6]. Firstly, the data challenge is that, un-like academic publications, impact data are not currentlycaptured in a systematic way [7]. There are a growingnumber of tools to facilitate the collection of evidencefor impact within research organisations, such asResearchfish® and Symplectic, through which researcherscan report their impact related activities, such as mediaengagements, conferences, non-academic publications,and such. However, there is no standardized mechanismfor evaluating or even reporting on impact more broadly(i.e. ‘how did you actually make a difference?’) and manyresearchers are not yet used to reporting impact regu-larly. For the purposes of the UK 2014 REF, informationwas collected in the form of “impact case studies” con-sisting of roughly four-page narratives now available toread in an online searchable database,2 which allowedfor a more detailed description of the impact journeyand the impacts themselves. Secondly, the timing of cap-turing impact information is challenging as it can take inthe range of 15–25 years on average for research to betranslated into public impacts [8]. Furthermore, recentinvestigations into research impact mechanisms havehighlighted that while both researchers and policymakers often assume a linear relationship between aca-demic outputs and corresponding benefits to wider soci-ety, the majority of impacts are indirect andconsequently very challenging to capture [5, 6, 9]. Thisliterature suggests that many impacts develop over timethrough formal and informal professional networks andprocesses. These processes include ‘knowledge mobilisa-tion’ [10], the use of an embedded researcher [11], and‘co-production’ methods between researchers and practi-tioners [12]. There is also evidence suggesting that someimpacts are achieved during the research process itself,especially in cases where the research was co-createdwith beneficiaries. Studies such as the Retrosight studies[13, 14], the analysis of the REF 2014 impact case studies[15], and a recent evaluation of the NIHR HTAprogramme [16], have all highlighted examples of re-search where study participants had already received thebenefits of the research. Analyses of impact thereforeoften include the intermediary steps, or ‘proxies’ to theultimate impact. Finally, it is also evident that researchprojects build on each other in order to reach impacts,

1https://www.ukri.org/innovation/excellence-with-impact/ 2http://impact.ref.ac.uk/CaseStudies/

Boulding et al. BMC Medical Research Methodology (2020) 20:34 Page 2 of 20

https://www.ukri.org/innovation/excellence-with-impact/

http://impact.ref.ac.uk/CaseStudies/

which makes it challenging to attribute single impacts tosingle researchers or projects.

MethodsApproach to our studyAgainst this background, it is important to consider thebroad scope of public health research to provide more com-prehensive analyses on what occurs in the field. To simplyrely on specific types of impact, such as a national change inpublic health policy, may neglect other impacts on benefi-ciaries which occur during the research itself. Furthermore,many public health research projects may not follow a linear‘evidence to public health impact’ pathway; studies havehighlighted that research evidence is the least frequently usedform of information in public health policy and programmedecision-making [17, 18]. These highlight that the context inwhich public health studies are undertaken, and the informa-tion needs of the different roles within organisations, are acrucial part of any examination of impact.With this in mind, we have approached our study by

following the ‘payback model’ by Buxton and Hanney[19], a conceptual framework that has been identified asthe dominant one (though not specific to public health)for describing how research leads to impact according toa review by Raftery et al. [20]. This model articulates thestages of research, from conceptualisation to impact,leading to five possible kinds of payback: knowledge pro-duction (e.g., academic publications), research targetingand capacity building (e.g., training new researchers),informing policy and product development (e.g., infor-mation base for clinical policies), health and health sec-tor benefits (e.g. cost savings and greater equity), andbroader economic benefits (e.g., commercial spin-outs).Designed as a method to evaluate the payback for par-ticular pieces of research, the model requires in-depthcase studies and an evaluation of both quantitative andqualitative information, including detailed interviewswith those who conducted the research and created theimpact. We adopt these approaches while mindful of theopportunity to discover other impacts that may occurduring any part of the research process.Given the lack of standardized databases on ‘impact’ al-

luded to earlier, we have also had to select a source of datafor impact based on projects funded by a single funder,the National Institute of Health Research (NIHR). TheNIHR funds a variety of health and care research and isone of the main sources of funding in the UK for publichealth research [21]. The NIHR has two funding streamsthat explicitly name public health within their titles: theSchool for Public Health Research (a partnership betweeneight academic centres with excellence in applied publichealth research in England) and the NIHR Public HealthResearch Programme (that provides funding for researchon non NHS interventions to improve public health), and

we sought to identify projects that included these twofunding mechanisms and beyond.

Aim of this projectOur specific aims were to (a) map the diversity of publichealth research funded by the NIHR, in terms of fundingmechanisms, disciplinary contributions, and publichealth impacts, (b) through interviews identify examplesof impacts, and pathways to impact that existing report-ing mechanisms (such as Researchfish)3 may not other-wise have captured, and (c) provide illustrations of howpublic health researchers perceive the generation ofnon-academic impact from their work.

Data sourcesNIHR established a Public Health Overview (PHO) team,based out of the NIHR Evaluation, Trials and Studies Co-ordinating Centre, University of Southampton (one of thefive NIHR managing centers responsible for deliveringNIHR’s operations). The PHO team had already taggedstudies funded by the NIHR research programmes andSchools to create a subset of studies pertaining to publichealth research. This mapping exercise sought to analysethe portfolio since the NIHR’s formation in 2006, andidentify gaps in evidence. The outcomes of the mappingexercise have been published for studies funded up to2013 [21]. The database provided to us for our analysis bythe PHO team in early 2017 covers the period 2000-March 2016, and we are aware that the database and itsworking definitions are continually updated. Table 1shows the inclusion and criteria used by the team foridentifying these studies at the time:We note that since the database was made available to

us, the PHO team has published their approach to iden-tify projects that could be tagged as public health re-search, in which they focused on investigations and/orstudies that are anticipated to have “an effect on healthor on health inequity at a population level.” [22] A totalof n = 1386 studies were identified from projects fundedbetween 2000 and 2016 as being public health focussedwithin the PHO dataset. The PHO studies were subse-quently categorised according to the Public Health Out-comes Framework [23] to identify the types of outcomesaddressed in each. To give us an indication of studiesdeemed likely to have begun achieving impact, the PHOteam identified a subset of projects eligible to go into aResearchfish submission period with PIs whom NIHRwas asking to report data. A total of 857 such studieswere identified, using a mix of automated and manual

3Researchfish is an online platform that enables research fundingorganisations to capture and track the impact of their investments, andresearchers to log the outputs and outcomes of their work. Availableat: https://www.researchfish.net/home


https://www.researchfish.net/home

matching. All visualisations of project outputs shown inthe graphs in this paper are based on either the originaln = 1386 studies identified or this smaller set of 857 pro-jects provided to us at the time.

Identifying interviewees and preparing for interviewsFrom the final dataset received, we selected a range ofprojects to ensure a mix of sizes and research topics. Todo this, we classified each project according to:NIHR source of funding: One project could have been

supported from more than one source of funding(Researchfish database allows this information to becaptured).Size of funding award: We grouped projects into three

sets: £0–£350,000; £350,000–£1million, and; over £1million;Domains of improvement set out in the Public Health

Outcomes Framework: Healthcare public health and pre-venting premature mortality (which in the dataset wassplit into ‘healthcare public health’ and ‘preventing pre-mature mortality’ as two separate domains), Health Im-provement, Health protection; and Improving the widerdeterminants of health.Every project was assigned a funding diversity score

depending on how many funding streams supported it(minimum 1, maximum 2 in our database). For eachoutcome domain we created three sets of projects ac-cording to funding size, and within each set we selectedthe projects with the highest funding diversity score.This meant all scores of 2 were selected and comple-mented with a random selection of projects with a

funding diversity score of 1. This yielded a total of 75projects. This was our new sample from which to thenrandomly select projects for interview – during whichwe checked our list to ensure there was diversity withrespect to institutions. In addition, we contacted theNIHR School for Public Health Research directly to askfor suggestions of studies, since they do not report viathe Researchfish platform. We then manually identifiedthe different kinds of research designs and methods usedin each of our interview sample projects to observe howdifferent approaches influenced the pathways for re-search impact. Our interview sample spanned bothquantitative and qualitative methodologies, includingrandomised controlled trials, natural experiments, math-ematical modelling, systematic reviews, ethnographies,and mixed-methods studies.We invited a total of twenty PIs to participate in a

semi-structured telephone interview, of which ten ac-cepted. One PI was unavailable for the relevant period,and therefore nine case studies were included in ourfinal sample. Table 2 provides a synthesis of each projectwe investigated. Where possible we conducted follow upinterviews with other members of the sampled studyteams in order to enrich our understanding of the re-search process and impact mechanisms. This happenedon five occasions, therefore a total of 14 interviews wereconducted. All interviews were conducted using a singletopic guide (see Additional file 1: Annex A), which ad-dressed the nature of the research project and any col-laborators, the findings and mechanisms for sharingthese findings, any perceived impact from the project,the perceived mechanisms via which any impact wasachieved, and any perceived barriers to achieving impactfrom research. The categories for impact in this topicguide were inspired by those in the ‘payback model’ byBuxton and Hanney [19]. Interviews lasted approxi-mately 45 mins and were audio recorded with partici-pants’ consent. Interviews were conducted by fivemembers of the project team, with the majority of inter-views conducted by HB and AK.

Analysis and synthesisUsing insights from the quantitative analysis and inter-views, the project team conducted two workshops to ex-plore the emerging findings: one with four invitedparticipants including Directors of various NIHR re-search programmes and representation from the NIHRPublic Health Overview function, and an internal ana-lysis workshop during which the project team discussedthe case studies in depth, addressing the nature of im-pact in public health, impact mechanisms and pathways,and the way in which impact was understood by PIs.Our analysis was conducted in the tradition of groundedtheory, establishing core themes through an open coding

Table 1 Inclusion and exclusion criteria for classification ofprojects as ‘public health research projects’ within the NIHRpublic health overview (PHO) dataset used for this analysis

Inclusion criteria:

• Preventative interventions at the population level

• Early identification and screening programmes

• Identification of clinical thresholds and care pathways of commonconditions

• Health inequalities

• Improving services – health needs assessment and health planning

• Health protection including patient safety, infection control

Exclusion criteria:

• Treatment interventions with the exception of treatments for perinatalmental disorders and where increased physical activity is used as theintervention

• Basic clinical studies

• Diagnostic test

• Workforce issues, including leadership and training issues

• Projects investigating study design

• Secondary prevention studies (e.g. prevention of relapse of depression)


Table 2 Case study projects with impact summaries

Project Title Institution NIHR SupportStream

Period of support Fundingamount(£000 s)

Summary of impact

How effective is theForestry CommissionScotland’s woodlandimprovement programme -‘Woods In and AroundTowns’ (WIAT) - atimproving psychologicalwellbeing in deprivedcommunities?

University ofEdinburgh

Public HealthResearch (PHR)Programme

2012–16 350–1000 This study set out to explore linksbetween environmentalinterventions – in this case, effortsto regenerate and enhanceaccess to local woodland – andlocal residents’ perceived stressand mental wellbeing. Theprincipal investigator, a professorof landscape architecture, workedcollaboratively to generate ideasfor the research from an initialconcept developed with theForestry Commission. Themultidisciplinary research teamincluded epidemiologists, healthgeographers, psychologists, andhealth economists. The team hadnot yet published findings, butnoted a number of practitionergroups, as well as membership ofa NICE guideline-writing groupon physical activity, as receptiveaudiences with whom they haddeveloped functional links.

Improving employmentoutcomes for youngpeople with first episodepsychosis

South London andMaudsley NHSFoundation Trust

Research forPatient Benefit(RfPB) Programme

2008–12 0–350 This was the first UK randomisedcontrolled trial exploring how toovercome barriers to helpingyoung people with psychosisreturn to work via the delivery ofindividual placement and support(IPS) services – an intervention forwhich there was already goodevidence of patient benefit. Itexamined the effect of trainingclinically-based vocational staff inmotivational interviewingtechniques, to enhance theirprovision of IPS services. Theteam partnered with ImROC, anon-profit consultancy withexperience of working with NHSTrusts to implement recovery-based models of service delivery.Key factors in the success of thestudy appeared to be theprovision of a dedicated memberof the ImROC team working withclinical staff to ensure fidelity tothe recovery-based model of care.The team also reflected thatclinical teams respondedpositively to seeing the results ofpatients’ return to work (i.e. socialoutcomes), helping to overcomea perceived risk that patientswere more likely to relapse. Thestudy showed the benefits ofplacing and training specialistvocational support staff withinthe clinical team, with signifi-cantly more patients remainingemployed following their receiptof the enhanced IPS model. Theresearchers noted a short-termview of the financial burdenplaced on NHS trusts from


Table 2 Case study projects with impact summaries (Continued)



Summary of impact

employing these staff as a barrierto achieving impact.

An evaluation of a waterfluoridation scheme inCumbria

St George’s, Universityof London

Public HealthResearch (PHR)Programme

2013–20 1000+ This study was set up in responseto a systematic review and MRCworking group report calling forimproved evidence of thebenefits of water fluoridisation. Itseeks to evaluate the effects ofreintroducing fluoridisation to apopulation of 5–11 year-olds inCumbria, following its removal fora period of 5 years while thewater system was updated, andthe effect of this on children’sdental caries. The team workedwith local schools to recruitparticipants to the study,reflecting that they held back onusing social media as an adjunctto recruitment, due to thenegative reaction of groupsadvocating against fluoridisation.They noted the complexity ofconvening multiple groups –researchers, commissioners,policymakers – as a challenge forpublic health research, madeeasier by having Public HealthEngland as a central conveningbody. They also highlighted acapacity issue arising from limitedclinical staff choosing publichealth as a career, and the needfor advocacy from NIHR to makeresearchers more aware of thedifferent pathways available tothem. The team published theirstudy protocol in 2016, and wereawaiting results of the studybefore publishing furtherinformation.

Diabetes Prevention Model University of Sheffield School for PublicHealth Research(SPHR)

2014 Not available The central aim of this projectwas to develop a new model toevaluate the cost-effectiveness ofpublic health activities to preventdiabetes. The team had earlyengagement with representativesfrom NHS England, who in 2015had expressed an interest inunderstanding the economicimpacts of activities (such aseducation on healthy eating andlifestyle, help to lose weight andbespoke physical exerciseprogrammes) planned under thenational Diabetes PreventionProgramme. The team developedthe model into an online toolthat allowed local NHS commis-sioners to assess the potential re-turn on investments fromprevention activities planned intheir local areas, and on differentlocal populations. In parallel theyalso developed a similar tool for





Summary of impact

local authorities, following a callfrom Public Health England. Infor-mation from the modelling workinformed rollout of the DiabetesPrevention Programme in 2016and work commissioned by NICEon who to prioritise when plan-ning diabetes preventionprogrammes.

TOMMY trial: A comparisonof TOMosynthesis withdigital MammographY inthe UK NHS BreastScreening Programme

University ofCambridge

HealthTechnologyAssessment (HTA)Programme

2010–13 1000+ Breast tomosynthesis is arecently-developed techniquethat uses a series of low-doseX-rays to build up a 3D picture ofthe breast tissue. The team’smotivation for trialing thistechnique was to detect cancerswhen they were smaller, andreduce the ‘recall’ (false positive)rate of patients called back forfurther testing. The team workedwith INVOLVE and a patientadvocacy group – IndependentCancer Patients’ Voice – on thebest way to design an initial trial,which recruited to its target ofover 7000 women and waspublished in 2015. Discussionswith Public Health England andthe National Screening Commit-tee made it clear to the studyteam that this initial data wasinsufficient to effect a policychange on national screening forbreast cancer. They highlightedboth the conservative nature ofthese organisations, and also thecontroversy of breast screeningwith the public. The team aretherefore using the data from theinitial trial to design a larger trialinvolving 100,000 patients. Theyare considering ways to involvepolicymakers upfront, and alsoworking with a social scientist toexplore public perceptions ofscreening in the context of theirresearch.

A 1-year follow-on studyfrom a randomised, head-to-head, multicentre, open-label study of two pan-demic influenza vaccines inchildren

University of Oxford HealthTechnologyAssessment (HTA)Programme

2010–11 350–1000 This study responded to the UKGovernment’s purchase in 2009of two pandemic influenza (H1N1,or ‘swine flu’) vaccines, Celvapanand Pandemrix, and an urgentneed to test their safety andefficacy in children. Public HealthEngland had contacted theOxford group given their ‘trackrecord in delivering trials’, thoughthe study team involved threeother centres with establishedclinical trials groups. The team fedemerging results through toPublic Health England and theJoint Committee on Vaccinationand Immunisation, as data wasgathered. They reflected that the





Summary of impact

UK was unusual in usingunpublished data in this fashionto inform policy decisions, andnoted that the UK regulatoryenvironment and exitingrelationships with theDepartment of Health helpedensure that the research was ableto move swiftly, given theemergency nature of thepandemic. Impacts of the trialincluded the children who werevaccinated and the Departmentof Health, who benefitted fromguidance on how to conducttrials in pandemic situations. Onenegative impact was anassociation with childrendeveloping narcolepsy, whichwas noted as a rare side effect ofthe Pandemrix vaccine.

Evaluation of a NationalSurveillance System formortality alerts

Imperial CollegeLondon

Health Services &Delivery Research(HS&DR)Programme

2014–16 350–1000 Following their work on theShipman Inquiry, the Dr. Fosterunit at Imperial College Londondeveloped a national system togenerate monthly alerts fromroutinely collected hospitaladministrative data on 122diagnoses and procedures. Theirevaluation was designed toaddress concerns of bias andestablish the extent to whichvariations in quality of care was afactor in triggering alerts. Theteam had already been sendingmortality data freely to NHStrusts, copying the Care QualityCommission (CQC) if it exceededa certain threshold. They ran aworkshop with CQC to help withrecommendations based on theirresearch, in particular the serviceand mode of delivery of thealerts. The team were taking amixed-methods approach tounderstand institutional responsesto mortality alerts. While theynoted that the research was at anearly stage, and that death is acrude indicator of underlyingissues, the team hoped that theirapproach was supporting bestpractice in CQC investigations,and how hospitals can respondwell. They flagged that socialmedia engagement around theirfindings had ‘turned nasty’ due toits timings coinciding with thejunior doctors’ strike.

ComprehensiveLongitudinal Assessment ofSalford Integrated Care(CLASSIC): a study of theimplementation andeffectiveness of a new

Salford Royal NHSFoundation Trust

Health Services &Delivery Research(HS&DR)Programme

2014–2016 1000+ The aim of this project was toevaluate the changes that hadbeen made in Salford to theprovision of care for long-termconditions reflecting a movetowards integrated care. The





Summary of impact

model of care for long-term conditions

study found that integrated caredid not have a significant impactin terms of reducing hospitaladmissions as intended. Thechallenge in disseminating thesefindings since they are incontradiction to the currentdirection of travel in terms of theorganisation of care within theNHS and the wider policylandscape. Significant investmenthas been made in integrated carein some locations and as suchthis is a challenging message toshare that will not necessarily beopenly received by policy makersand practitioners.

Prenatal screening andtreatment strategies toprevent group Bstreptococcal (GBS) andother bacterial infections inearly infancy: costeffectiveness and expectedvalue of informationanalyses

University CollegeLondon

HealthTechnologyAssessment (HTA)Programme

2005–6 0–350,000 HTA commissioned this study inthe context of wider decision-making processes around levelsof evidence required to adoptscreening strategies for commoninfections in early infancy. A largerrandomised controlled trialcosting £12million had beenplanned, involving 600,000women, to evaluate the effect ofscreening for group Bstreptococcal (GBS) infections.The 2005 study brought togethermethods experts in techniques tomodel the economic value ofgenerating further evidence – inthis case, applied to existingtreatment and screening optionsfor GBS infections. The mainimpact was that the study teamconcluded that running the largercontrolled trial would not becost-effective: the trial wastherefore called off. Instead,resources would be better spenton treating patients at a higherrisk of infection, and investing indeveloping a vaccine. The teamnoted that their involvement infora that brought togetherdifferent groups throughout theresearch – their case, a patientadvocacy group, members of thelarger planned trial team, and theUK National Screening Committee– provided the means to allowthem to apply relevant evidencefrom their study. They noted abarrier to further impacts in thisarea as the challenge of makingroutinely captured infectionsurveillance (i.e. audit) dataavailable for research purposes,and that trials funding mightusefully be redirected into waysof improving access to and use ofthese data.


process and populating these with corresponding inter-view data. Analysis was conducted by HB, SHK, SG, AK,IG and BI. Any differences in interpretation were re-solved through discussion among the project team.

CaveatsOne major limitation of our study is that we have focusedon one UK-based funder only. This could potentially limitour understanding of impact within public health, as weare not drawing on the full extent of research across thecountry (or indeed globally). As noted earlier, however,the NIHR is one of the main sources of funding in the UKfor public health research. Furthermore, the NIHR doesprovide a varied sample of research projects relating topublic health because of the breadth of programmes,methods and health areas that it funds. While our casestudies are drawn from one funder, our findings should beinteresting to those examining research impact morebroadly, both in the field of public health and beyond. Wedo not claim to showcase a representative sample of pub-lic health research projects in the UK, but a sample that il-lustrates researchers’ experiences and perceptions of thevaried mechanisms that demonstrate how impact is cre-ated from public health research.We are also mindful that this article is written from

the perspective of how we define public health in theUK. Although the UK is found to be one of the maincontributors to publications classed as public healthwithin Europe [2], we are mindful that other countriesoutside of Europe may have different scope and defini-tions they employ. As noted in the Background section,however, most definitions imply a focus on health of thepopulation, which is what constituted the focus of thePHO criteria for identifying the projects tagged as publichealth made available to us. We have relied on the NIHRPHO coding of projects to identify our sample, fromwhich we selected our interviewees. We trust thisprocess was completed rigorously although caveat thatwe may have missed some examples of projects thatcould also have been considered public health. However,we feel that because we are not making general claimsabout public health research as a whole, nor of the total-ity of research in the UK, this limitation does not have asignificant impact on our conclusions.

ResultsOur focus in this paper is to illustrate the perceptionsand experiences of public health researchers in demon-strating and delivering on non-academic impact fromtheir work. This is captured in section (ii) of our results.First, to provide some background and overview of thepublic health research portfolio at the NIHR, in section(i) we describe the general trends observed from the fulldataset we received of projects that were reported on

Researchfish and were tagged as public health projectsby the NIHR team.

Mapping the funding mechanisms and impacts for publichealth researchIn this section we describe the general trends observedby mapping the data from the projects we received(those that had data available within the Researchfishreporting dataset).

A variety of funding mechanisms support public healthresearchAs noted above, we identified 1386 projects fundedbetween 2000 and 2016 as being public health fo-cussed for our analysis. We mapped the fundingstreams for each of these projects and show these inFig. 1 and note that there are many more fundingmechanisms within NIHR that fund public health re-search activity (Fig. 1). In addition to the NIHRSchool for Public Health Research and the NIHRPublic Health Research Programme, 89% of the 1386projects are funded via other funding programmes,demonstrating the diversity of funding streams thatsupport public health-related research. A large pro-portion of these are funded through the Health Ser-vices and Delivery Programme (HS&DR). It is worthnoting that several of the large funding streams (e.g.the NIHR HTA programme) support a significantnumber of projects that are classified as public health(Fig. 1). Although this may be due, in part, to thelength of time that some of the programmes havebeen running, this chart captures the diversity of‘public health’ and the range of types of research thatcan contribute to public health outcomes.

A variety of impacts arise from public health researchWe also mapped the types of impacts reported by re-searchers in the dataset received. Within the Research-fish online interface, all individual entries are labelled as‘outputs’ of research, including academic and non-academic outputs and any wider outcomes that may beconsidered ‘impact’, entered by researchers themselves.Looking at this data self-reported by researchers via theResearchfish platform – a subset of 857 studies matchedwithin the NIHR Public Health Overview portfolio – wesee a diverse range of different types of research activitiesand outputs (Fig. 2). In line with previous analyses of datacaptured via the Researchfish platform, we found that in-vestigators reported academic publications more fre-quently than any other output category. Followingpublications (not included in the table), ‘engagement ac-tivities’ is the most commonly reported item in Research-fish (3383 instances), followed by ‘collaborations’ (1692).However, we also note a smaller but still substantial


number of impacts on patients (724), and policy and prac-tice (658). We note that these graphs rely on data self-reported by researchers, and therefore the emphasis givento particular types of research activity may determine whatwas reported within Researchfish. Each researcher mayalso have interpreted their activity different (putting anentry under ‘engagement activity’ which someone elsemay have considered to be ‘policy and practice’ or ‘collab-orations’). Through our qualitative interviews, the findingsof which are described in section (ii), we were keen toidentify what these activities, such as ‘engagement activ-ities’ or ‘collaborations’ entailed.

Researchers’ perspectives on pathways and mechanismsto impactIn this section we report on the findings from the qualita-tive interviews, focussing on researchers’ experiences inproducing and articulating the impact from their work.

Different interpretations of the meaning of impact frompublic health researchThe diversity we observed in the quantitative data wasreflected in our in-depth case studies. We found that re-searchers had different interpretations of what publichealth could include as a research discipline, and several

Fig. 2 Counts of research outputs self-reported by principal investigators using Researchfish, by Researchfish output category, excludingacademic publications (n=9428 reported activities in the 857 projects included)

Fig. 1 Counts of studies included in the NIHR Public Health Overview (PHO) dataset received from our analysis, by their respective NIHR fundingstream (total n = 1386)*. * Note: The NIHR has different managing agents for coordinating funding and research delivery. The funding streamsincluded in this diagram run across different managing agents. The exception is the Trainees Coordinating Centre, for which we have grouped alltheir projects into one as they are training schemes and are broken down by stage of career, and this would not indicate the type of diversity weare illustrating here in terms the types of funding provided


of those we contacted questioned whether their researchshould be classified as public health at all.One of the main themes arising from our interviews was

diverse interpretations of how evidence created from pub-lic health research can lead to impacts in national policy.Interviewees felt that the diverse forms of evidence pro-duced by public health research do not always correspondwith those required by policy makers to effect change. Forlarger, drug-based interventions, randomized controlledtrial methodologies may still generate the most appropri-ate form of evidence, but this is not the case for many in-terventions in public health, especially those related tolifestyle factors. Our participants reported lack of clarity asto what constitutes appropriate evidence to achieve im-pact. As one researcher noted, “there are different expecta-tions for different fields about what counts as robustevidence. You could argue that you are unlikely to do harmin a community by making their park easier to use, but weare held to same standard as clinical drug trials. You can’teasily do RCTs in this field, but that’s the standard towhich health evidence is held.” Interviewee 1.We also found that researchers struggled to ascertain

clarity on the level of evidence required to influence publichealth policy. Having worked on a study that was deemedby a national committee not to have produced sufficientevidence to bring about a policy change, one intervieweetold us that they had subsequently asked what would beneeded to achieve this, but were not given a tangible an-swer. Reflecting on this, the researcher commented “[pol-icymakers] are understandably cautious … so it makes itdifficult for an academic to know how much evidence youneed in order to actually effect the change” Interviewee 2.For this researcher, the uncertainty surrounding the levelof evidence required left them feeling that it was a struggleto make changes in a timely way.The uncertainty regarding the type of evidence re-

quired for change was a recurring theme emerging fromour discussions with researchers, one of whom had spe-cifically developed techniques to model the economicvalue of generating further evidence regarding screeningand treatment strategies to prevent infections in early in-fancy. Reflecting on how to tease out whether and howfurther evidence was needed, the researcher explained:

“‘What do you want to do now, given the evidence suchas it is?’ and, ‘Do you need more evidence to inform thatchoice in the future?’ By separating those two questionsout, you can have a sensible answer to both, to completelymove away from hypothesis testing.” Interviewee 3

Reflecting on the way researchers discussed national pol-icy impact, we noted that their responses often suggestedthat their main interpretation of what ‘impact’ means is evi-dence that produces change in national policy. When

interviewed, a few of the researchers were cautious in de-scribing the impact of their projects, emphasising that theyhad not quite produced all the evidence to reach policy. Yetwhen probing further we found that small, unintended ben-efits from their work was directly observed with the non-academic organisations with whom they worked, such ashealth service delivery or local government. When describ-ing the relationships developed during research projects(described further in the next section), they noted that theserelationships themselves had the capacity to influencechange directly within an organisation. For example, oneresearcher commented that while they considered the po-tential primary impact of their research to occur at a na-tional level, the more immediate benefits occurring at alocal level were the “unintended impact.” Interviewee 4.The challenges in getting evidence into policy and practice

apply to both national and local contexts, as pointed out byseveral of the researchers we interviewed (whose perspectivesare from the UK contexts). Reflecting on the devolved natureof public health in the UK, one participant told us:

“The problem is that the research has been ratherdissociated from the practitioners … Cost effective-ness, timing, relevance and generalisability havebeen rather lost … We’re learning how to work withlocal government, making relationships … We’d doneall that over 50 years with the NHS, and now we’vegot to do it with local government.” Interviewee 5

In spite of enthusiasm for interdisciplinary collabora-tions, uptake in practice is not always possible due tothe silos in the way health, and other services that sup-port health, are delivered throughout the country. Oneinterviewee referred to the lack of cross-departmentalcollaboration on public health issues:

“We have multidisciplinary/interdisciplinary researchfindings, so to achieve practice based on them youneed intersectoral budget management to reflectdelivery demands. I don’t see any sign of thathappening.” Interviewee 1

The responses in this section suggest that there aredifferent interpretations of what constitutes evidence re-quired for public health research impact, especially if theimpact is intended to occur at national policy level.Smaller, localised benefits from public health researchare also acknowledged, although not always interpretedas what counts as ‘impact’ by the researchers.

Engaging external stakeholders to facilitate impactFollowing academic publications, engagement activities werethe most commonly reported item in the Researchfish out-comes data we received in our dataset (Fig. 2). Our


interviews enabled us to explore the nature of these activities.One of the most significant impact mechanisms reported byour interviewees was the relationships they developed with arange of external stakeholders, including hospital trusts, theDepartment of Health and the medical technology industry.Several of our participants suggested that these relationshipswere a way to navigate the complexities of the public healthlandscape. We noted that these relationships appeared to bemost effective if they were in place from the start of the re-search, and several of our interviewees reported calling onrelationships with external stakeholders that they had knownprofessionally for many years.One researcher informed us that their team had been

chosen by the Department of Health to respond to a publichealth emergency on the basis that the researchers and theirwork were already known and trusted by those in a positionto implement their findings. This researcher also noted thatgood relations between the Joint Committee on Vaccinationand Immunisation (JCVI) and researchers meant that re-search findings were able to inform practice much sooner:

“There are good links between academia and JCVIwhich makes the UK well-positioned in being able toaccess the necessary data quickly in order to informcritical decisions without having to wait for things tobe published.” Interviewee 6

As a counterpoint to navigating through the intersec-toral silos identified in the previous section, a selectionof interviews demonstrated initiative in reaching out toorganisations outside the health sector, pointing out thatthey could be the people to help facilitate impacts overthe lifetime of the research study and beyond:

“If you want something done about the environment,you have to be working with parks managers in localauthorities and organisations like the ForestryCommission or the National Trust - people who givegrants and who are going to make physical differenceto real environment - if you want your research toinform policy and practice.” Interviewee 1

Similarly, another one of our participants emphasizedthe importance of generating the necessary relationshipswith a broad range of external stakeholders to garner thenecessary support for a new health initiative:

“You have to set it up with the policy makers first,and then the funders, do the work and gather theevidence to make the change. I wouldn’t go directlyto NIHR unless I had support from the screeningcommittee of England. You need to lay the ground-work. That’s one way to impact, getting opinionleaders on board, getting the community behind you

and the professional bodies and policy makers”.Interviewee 2

Several of the researchers we interviewed pointed outthe importance of these relationships in facilitating hit-ting the right policy or impact ‘window’, sharing exam-ples of where timing had been a principle factor for bothpolicy-makers and practitioners. One researcherexpressed concerns that findings were not informingpractice in time to be useful to practitioners:

“Practitioners are completely uninterested in re-search findings in 5 years’ time.” Interviewee 3

Although they emphasized the need to reduce the timeit takes for research findings to reach practitioners, wealso found that research that is ‘ahead of the curve’ maynot attract interest from policy makers until years later,particularly if these relationships are not already in place.Another researcher told us that although their project ini-tially failed to gain the necessary support in PHE to effectthe desired change in national screening guidelines, theywere approached years later and asked to contribute theirexpertise on the screening technique their team had advo-cated, at a time when the need was felt to be greater andthe benefits of the technology better understood. Here wesee a different scenario, in which researchers had to waitfor a ‘policy window’ years down the line:

“Interestingly, PHE have now realized they have areal manpower problem and so have come back tome nine years after I published evidence on readingscreening mammograms. Sometimes it’s about tim-ing. Maybe we were ahead of the game at that point… it’s about timeliness.” Interviewee 2

While generating relationships with external stake-holders was often rewarding, researchers also commen-ted on how this can take time and is “resource intensive.”Interviewee 7. It can therefore be frustrating when thereis high turnover either in policy or hospital practice sothat personal relationships with relevant organisationsare lost or have to be rebuilt over time. The challenge inbuilding relationships or indeed in engagement activitiesover and above writing and disseminating an academicpaper, is that these usually need to continue after afunded project is closed, and so finding the resources tosustain them is tricky.

Dissemination mechanisms to facilitate impactFor each of the in-depth case studies, we found that re-searchers employed a variety of different dissemination ac-tivities beyond academic publications. Our participants


provided detailed perspectives on these activities, andthese discussions suggested that many of the presentationswere made to non-academic audiences using infographics,animation and web-based media to communicate mainheadlines clearly, or engaging with mainstream media asappropriate. Messages and mechanisms were often tai-lored to those in a position to drive implementationforward:

“It is not science sitting in isolation; it’s sciencesitting in a complex group of stakeholders. The purescience piece is the submission to NIHR. But how itis disseminated and spread will have to be verycarefully undertaken.” Interviewee 8

Several of our researchers highlighted mainstreammedia as a means to facilitate impact, offering examplesof where a news story had helped drive change. One re-searcher told us:

“I’ve had experience of high profile bits of work whichhave effected policy change, and that has involved alot of exposure on the media, you know, almostputting pressure on the policy makers to dosomething about it.” Interviewee 9

Another researcher told us that they had given aninterview about a high-profile research project for apopular monthly magazine, and subsequently received atelephone call late at night from a patient overseas whowanted to discuss the findings after having read aboutthe study in the magazine.Being mindful of the right communication channels also

meant not investing in activities that were not appropriatefor a particular project, for example, not soliciting main-stream media if the individual conversations and meetingswith stakeholders were more important and would helpdrive the adoption in practice. One researcher told us:

“We were directly addressing those to the audiences thatneeded to hear them, either through presentations or thereports. So we stopped there, and actually I think that’sappropriate. I think those messages needed to be agreedwith and then owned by others in order to take themforwards. I think it would have been inappropriate for usto be pushing [media engagement].” Interviewee 10

Several researchers also noted that engaging with so-cial media could be challenging, with some having hadnegative experiences which had made them more cau-tious about this kind of dissemination mechanism:

“It’s a jungle out there. It all got very nasty veryquickly, and actually my poor junior researcher who

happened to be corresponding author and the firstauthor on the paper, he had the most awful timewith Facebook campaigns against him, and it was areally nasty business.” Interviewee 9

Overall we observed a balanced approach on the partof most researchers to selecting and engaging with dif-ferent dissemination mechanisms. However, whenreflecting on their experiences with media engagement,several researchers noted that bad experiences with bothmainstream media and social media would make themmore cautious about using these mechanisms in future.

Acknowledging “negative” findingsSeveral of our case studies revealed the difficulties asso-ciated with achieving impact with a negative researchfinding, especially if findings ran contrary to currentthinking and practice. As one researcher explained:

“Positive results can hold attention and enthusiasmbecause they can be scaled up [ …] so that’s in yourinterest as an academic. Negative results are harder[ …] This is a particularly difficult space for researchfor evaluating someone’s innovation—the air can goout of the room.” Interviewee 7

The same researcher discussed the wider culture surround-ing positive and negative findings in terms of research impact,emphasizing the danger of incentivizing positive results:

“If you want impact, you need positive results, andthat’s dangerous for research [...]. Having these in-centive structures puts academics in a difficult situ-ation: you need something new and exciting thatworks, and that can’t or doesn’t always happen.”

The potential value of impact derived from negative find-ings was illustrated by another of our case studies, whereresearchers were able to implement their negative findingssuccessfully. This project used a multi-parameter evidencesynthesis to examine the value of screening as one of anumber of interventions to prevent infection in early in-fancy. The study findings indicated that conducting a largercluster randomized controlled trial to evaluate the efficacyof screening, planned at a cost of £12 million, would not beworthwhile. The researchers commented that:

“The single most specific impact of the study was tostop the larger cluster randomised controlled trialfrom going ahead.” Interviewee 10

This saved a significant amount of money, which couldthen be invested in other public health research projects.


Researchers as drivers of impactOne of the most striking elements to emerge from our casestudies was the role of researchers’ own perceptions andskills in determining research impact. We found a range ofdifferent opinions among our participants of when andhow impact is achieved, with some emphasizing the role ofpresentations and collaborations, and others suggesting thatthe publication of research findings in academic journalswas the main springboard for impact. One researcher notedthat impact occurs once findings have been written up:

“Once the organizational, institutional stuff has beenproperly written up we can then put in some recom-mendations or guidelines…That’s the plan, but wehaven’t got there yet.” Interviewee 9

Other interviewees highlighted the impacts that occurearlier in the research process, with one noting that par-ticipants in their trial benefitted directly from the re-search in addition to the longer term impacts that theywere aiming to have at a national level:

“The main beneficiaries at the time were the individ-uals who were vaccinated in the trial, and then morebroadly it was the availability of the data to guidethe department of health on how to move forwards.”Interviewee 6

We also observed a number of different views on thedrivers of impact from the perspective of researchers.One researcher highlighted passion as a key factor mo-tivating public health researchers:

“Ultimately, we are not just curious. I come from adiscipline where we plan to make a difference inpeople’s lives in practical ways.” Interviewee 1

Another commented that the most effective means toachieve research impact was through conducting researchthat was addressing important public health challengesthat are of particular interest to policy makers, the mediaand the public in addition to the academic community:

“I think [effecting change is about] just having someinteresting research. You know, something that otherpeople can relate to and it’s important.” Interviewee 9

Some researchers felt duty-bound to ensure that theyfacilitated impact from their research. As one researchercommented:

“Retreat if you did a bad study, but if you did a goodstudy, then it’s your responsibility to push that out.”Interviewee 8

However, while researchers felt that it was importantthat their work be disseminated in ways that would ef-fect positive change, some raised questions as to how farresearchers should be responsible for impact. One of ourworkshop participants noted the potential for conflict ofinterest if researchers felt the need to advocate impactsfrom their own research, and suggested that a neutralthird party could take on the responsibility for advocacy.Similarly, one of our interviewees commented that al-

though researchers should make their findings clear tothose with the capacity to implement change, they feltthat researchers should stay removed from decision-making, and did not necessarily have the skills to engagein impact activities:

“I wouldn’t expect impact to be straightforward orsimple. I’m not entirely sure I’m skilled enough forthat or that it’s my job. Not to say it’s not an import-ant role or responsibility. I feel our role there is to beavailable, accessible and clear about what we found.In terms of decision making, that is several stepsaway from me and that’s the way it should be Ithink. They are responsible. They have to go to theirlocal elected representatives.” Interviewee 7

We noted that there was also an acknowledgementthat the skills required for engagement beyond academicpeers, be it with social media, mainstream media, or in-deed other forms of communication, are not alwaysreadily available to researchers. As one researchercommented:

“We have capacity issues in public health researchers,particularly those with clinical qualifications.” Inter-viewee 8

Our conversations with researchers suggested thatthey would benefit from support for impact and engage-ment, both in terms of building skills, and also throughbuilding impact elements more explicitly into the re-search process. One interviewee explained that re-searchers are not always taught about impact and howto effect change, and suggested that one way to supportthem in doing that would be to make impact elements afeature of funding applications:

“We’ve very good at teaching people research meth-odology, but we’re not very good at teaching themhow to influence…In the funding applications… itcould be not only have we discussed this with pa-tients and the public but we have discussed this withpolicy holders, and we have checked that if this issuccessful then this is what would need to be done. Idon’t want to put another barrier in to people


getting research money, but it could be useful in get-ting people to think…” Interviewee 2

DiscussionThis analysis of the 857 completed or ongoing NIHR-funded public health research projects, which includednine in-depth case studies, suggests that public healthresearch as a discipline can contribute substantially toimpact beyond academia. The funding mechanisms thatsupport this broad range of public health research (forthe NIHR) extend beyond those that include publichealth in their title. The pathways to impact observed inour in-depth case studies include contributing to debateson what constitutes appropriate evidence for nationalpolicy change, acknowledging local ‘unintended’ impacts,building trusted relationships with stakeholders acrosshealth and non-health sectors and actors, collaboratingwith local authorities, and using non-academic dissemin-ation channels. While this study was by no means ex-haustive, there are lessons regarding impact mechanismsand pathways that are useful to draw out for further dis-cussion, that could benefit funders of public health re-search, researchers, policymakers and practitioners.

Considerations for public health research institutions andfundersThe mapping of the NIHR projects with a public healthfocus showed that these impacts are derived from a var-iety of funding programmes offered by NIHR, beyondthe Public Health Research Programme and School forPublic Health Research (see Fig. 1). This breadth ofNIHR public health research is one of the drivers forPHO activity and serves as strong advocacy for the field.While we have not tested this for other funders, it islikely that a broader consideration of the many fundingstreams that contribute to public health research woulddemonstrate for other funders the substantial investmentand contributions made within the discipline.During our analysis we reflected that supporting re-

searchers’ initiatives in implementing their pathways toimpact should be a principal consideration for fundersand supporters of public health research. The re-searchers we interviewed showed great enthusiasm andmotivation for responding to public health challenges.The motivation for having societal benefit (or ‘impact’)from their research is therefore implicit, but we identi-fied difficulties in having the time, financial resources,and skills to take the necessary actions to facilitate it. AsRedman et al. [24] have pointed out, while there arecourses, tools and funding modules to assist researchersin making an impact, there is little evidence aboutwhether these approaches work in practice, and morecould be done to support researchers in making the con-nections they require to facilitate impact. It is not to say

investigators should play no part in enabling impactfrom their work; indeed, some of the researchers weinterviewed showed initiative in engaging stakeholdersoutside of their field and taking part in non-academicdissemination of their work. For others, however, thereneeds to be an acknowledgement of the skills and re-sources required for engagement – or more nuancedimpact-facilitating – activities, and these may have to beshared with other colleagues or support staff.Our qualitative findings suggest that relationships with

those in a position to facilitate change were developedand strengthened over time, providing short, mediumand long-term impacts that helped facilitate publichealth goals. Greenhalgh and Fahy [6] observed exam-ples of this in their discussion of 162 REF case studies,commenting that “In the policy setting, [impact] occurswhen researchers and policymakers, through repeatedinteraction over time, come to better understand eachother’s worlds and develop goals that are compatible ifnot fully aligned.” In their work on ‘productive interac-tions’ Spaapen and van Drooge focus on capturing infor-mation about knowledge exchange interactions betweenresearchers and stakeholders as a prerequisite to achiev-ing impact [25]. Our case studies yielded some exampleswhere relationships with policy-making bodies had beendeveloped over years, and others where impacts hademerged from more fledgling relationships with stake-holders, for example through researcher interactionswith hospital staff. Indeed the devolved nature of publichealth in the UK (following reforms in 2013 in shiftingpublic health services and planning to local authorities)requires an understanding of local public health chal-lenges in addition to wider national policy requirements.A systematic scoping review of the use of evidence inlocal English public health decision-making identifiedthe primacy of local evidence, the important role of localexperts in providing evidence and knowledge, and thehigh value placed on local evaluation evidence despitevarying methodological rigour [26]. This suggests theneed for supporting relationship building and engage-ment between public health researchers/experts andlocal decision-makers to create localised impacts. In an-other study data drawn from an NIHR-funded projectexamined how evidence is used in commissioning andplanning, and also concluded the role of localised know-ledge. They found that published evidence is “made fit”for local commission and planning by relating the evi-dence to local context and needs and tailored into localactionable messages [27].How support for such long-term engagement is put

into practice can vary for funders or research institutions– some resources may be available within universities inthe form of ‘impact officers’ or research administrators.There is also much to be learnt from ‘translational


research’ or guidance for public health organisations onhow to develop ‘knowledge to action’ strategies such asfostering connections among stakeholders, and workingbeyond their academic institutions to understand con-textual factors [28]. Furthermore, there are other modelsfor facilitating uptake of research. Ward [10] provides auseful framework for identifying and making use of‘knowledge mobilisers’ who can facilitate getting evi-dence into practice or policy. The use of an ‘embeddedresearcher’ has been shown to be productive also inmobilising knowledge in general quality improvement[11] but also specifically in exchanging knowledge be-tween public health research and a local authority in theUK [29]. Another mechanism is the use of the re-searcher in residence model [30]. All of these do requireeither shifting of funding resources on existing projectsor funding specific for these types of activities.Deriving maximum benefit from such relationships re-

quires not only institutional frameworks that supportstakeholder engagement, but also building the skills andconfidence of researchers in their engagement activitiesand building trusted relationships. Our findings suggestwhile it is important that stakeholder relationships areestablished through research, these relationships are farmore effective if they are maintained and strengthenedover time. Upton et al. studied individual and institutionalperspectives on research impact in nine universities andfound that qualities to enable impact include a high degreeof motivation with regards to knowledge transfer mecha-nisms and other skills that support translation [31]. Bayleyand Phipps [32] argue for building ‘impact literacy’ - de-veloping staff capacity to judge, articulate and optimiseimpact across contexts. They highlight evidence that inte-grated methods of generating impact are more effectivethan end-of-grant methods. It is also important to provideacademics with opportunities to develop their networking,media-management and advocacy skills in order to ensurethat they can take best advantage of opportunities forstakeholder engagement.Finally, our study suggests that funders can also work

collaboratively with researchers to facilitate impact,through incorporating consideration of impact more fullyinto research planning and upstream assessment of appli-cations. Funders could also support early and frequentcollaboration with stakeholders, targeted engagement ac-tivities, and dialogue with award holders on appropriatedownstream evaluation of impact, throughout the re-search process. Though evidence of the causal nature ofthis is not conclusive, Guthrie et al. [13] showed that col-laborations between different groups (industry, patientsand practitioners, regulators) was associated with impactacross a number of domains. Researchers can also be sup-ported in maintaining their impact activities through theirfunder’s existing mechanisms, such as NIHR’s existing

research funding ecosystem, which provides different op-portunities for dissemination and encourages participatoryresearch and patient and public involvement in research.

Considerations for public health policy and practiceOur analysis suggests that the pathways and activitiesthat lead to wider impact are diverse, and can occurboth during the research process and on a longer-termbasis. We observed more traditional forms of impact, in-cluding research that had influenced policies or guide-lines (especially those reported by researchers within theResearchfish database), but also less tangible impactssuch as research that empowered project participants toeffect change in the hospital trusts in which theyworked. These findings have implications for how publichealth research can influence public policy and practicein three ways: the nature of evidence and evaluation, theneed to engage with local authorities, and importance ofcross-sectoral working.One of our most striking findings is that, while impact

is often conceptualised in terms of having impact at apolicy level, there are also many forms of impact thatcan emerge in localized settings and numerous differentenvironments. It is also the case that where researchmight be best utilized to improve public health policiesand guidelines, the evidence produced is by no meansguaranteed to reach those who have the power to effectchange such as this. As Hunter [33] points out, whilemost public health research is government funded, theextent to which research findings are used to informpolicy are extremely variable. Macintyre [34] argues thatboth policy makers and researchers need to developtheir understanding of impact mechanisms further, toensure that public health research supports improve-ments in public health:

“Policy makers certainly need to be more sophisticatedin understanding and commissioning different types ofresearch and acting on it. However, researchers alsoneed to be much more sophisticated and less naive inunderstanding how research does and does not influ-ence policy, and how to go about helping policy makersto interpret the jigsaw of evidence, and its relevanceand usability”.

Our study also highlights the importance of acknow-ledging the context in which evidence in public health isdeployed. Case studies touched on the need to thinkabout timing (hitting the right ‘policy window’ as sug-gested by Kingdon [35] and discussed by Cairney [36]),negotiating the presentation of negative findings, andunderstanding the different types of evidence requiredby those responsible for implementing public health ini-tiatives. It may also be the case that the type of evidence


required for change goes beyond that traditionally pro-duced by researchers, which would also require a greaterfocus on how results might translate in specific socialand political contexts. For example, in her evaluation ofthe kinds of evidence used in understanding the socialdeterminants of health, O’Campo argues that “researchshould not stop at demonstrating whether and how aprogram or policy improves well-being. More detailedinformation is also needed to facilitate adaptation, tailor-ing, and implementation of those programs and policiesto local settings and target populations” [37]. Further,while evidence drawn from large controlled interventiontrials meets higher standards of rigour, this evidence isnot necessarily the most applicable to ‘real world’ prac-tice [38] and may hinder the implementation of cost-effective interventions. Fischer et al. have argued that inthe case of many public health interventions it is imprac-tical to demonstrate individual effects through RCTs,but such interventions may be cost-effective even with-out evidence demonstrating efficacy [39].It is also important to consider the points of view of

public health professionals, which we did not include inthis study. However, a useful qualitative study by van derGraaf which explored how public health professionalsview and engage with research, identified three mainbarriers when trying to engage with researchers: 1) dif-ferences in timescales; 2) limited budgets; and 3) difficul-ties in identifying appropriate researchers [40]. Some ofthe challenges associated with these barriers could beovercome with better communication and relationshipsbetween researchers and practitioners. In addition toproviding the appropriate form of evidence, stakeholderengagement also plays a major part of facilitating impactas observed in our case studies. Researchers we inter-viewed we were able to ascertain the importance of highquality engagement with stakeholders such as local au-thorities and public health practitioners. They also ac-knowledged that given the interdisciplinary nature ofcomplex public health problems, more inter-sectoral col-laboration is needed. This may imply continuous shiftsin the way financial resources are allocated at the localauthority level to support public health activity, andworking with initiatives in transport, social care, andeducation to tackle public health issues.

Lessons for future research impact analysesSupplementing our analysis of NIHR projects with thesein-depth case studies revealed the extent to which im-pact pathways can diverge from the more linear modelsof impact that have informed understandings of impactprocesses in the past. Although population health im-provement may be considered the endpoint for publichealth research impact, short and medium-term changesthat facilitate this goal can also be considered impacts

arising from this field [9]. The question of how theseshort and medium-term impacts should be accountedfor in research impact analyses is a prominent feature ofthe literature on this subject [25, 41]. Further, while par-ticipants tended to focus on more direct, quantifiableforms of impact such as incorporation into policies orguidelines, our case studies also drew out several exam-ples where participants had achieved less tangible formsof impact. This included impacts achieved through in-formal networks, non-academic publications and thosecoproduced with research participants. This also carriesimplications for how impact is understood and mea-sured, especially in cases where research has a qualitativecomponent which often produces less direct pathways toimpact. As Greenhalgh and Fahy [6] have noted, the em-phasis of the REF impact case study format on measur-able impacts are such that these examples of researchinvolving more complex and indirect pathways to impactare rare. Our analysis suggests that improving the re-cording and understanding of less tangible pathways toimpact would render these more visible, and increasethe likelihood of public health research being translatedinto practice in ways that benefit patients and the public.

ConclusionsIn addition to the School of Public Health and PHR, theNIHR supports public health research through a variety ofother mechanisms, especially the HTA and HS&DR fund-ing streams. Impacts from public health research were ob-served in a wide range of disciplinary areas. Our in-depthcase studies highlighted a range of impact mechanisms in-cluding relationships with external stakeholders, targeteddissemination methods, getting the timing and type of evi-dence right and perseverance when disseminating negativefindings. Interviews with PIs and other researchers associ-ated with the projects also pointed to the pivotal roleplayed by researchers’ own assumptions and initiatives indetermining research impact. To support impacts frompublic health research, we encourage the acknowledge-ment and measuring of impact at different stages of theimpact pathway, including localised impacts, and the dif-ference in types of evidence required for community andlocal authority-based impacts. Finally, we note the import-ance of building capacity and resources to support im-pacts from public health research, and would like to seegreater intersectoral cooperation in order to ensure thatpublic health research fulfils its potential to effect positivechanges at both national and local levels.

Supplementary informationSupplementary information accompanies this paper at https://doi.org/10.1186/s12874-020-0905-7.

Additional file 1. Topic guide for interviews.


https://doi.org/10.1186/s12874-020-0905-7

https://doi.org/10.1186/s12874-020-0905-7

AbbreviationsDH: Department of Health; EME: Efficacy and Mechanisms Evaluation;HS&DR: Health Services and Delivery Research; HTA: Health technologyAssessments; I4I: Invention for Innovation; NIHR: National Institute of HealthResearch; PGfAR: Programme Grants for Applied Research; PHO: Public HealthOverview; PHR: Public Health Research; PI: Principal Investigator;REF: Research Excellence Framework; RfPB: Research for Patient Benefit;SRP: Systematic Reviews Programme; TCC: Trainees Coordinating Centre;WHO: World Health Organisation

AcknowledgementsWe are grateful for the input by the team at National Institute for HealthResearch (NIHR) who provided the data used in this analysis, namelyGenevieve Baker, Kay Lakin, Adam Lockwood and Sarah Thomas at NIHREvaluation Trials and Studies Coordinating Centre, and Manpreet Matharu atthe NIHR Centre for Business Intelligence.

Authors’ contributionsHB conducted interviews and qualitative analyses, and co-led writing of the finalmanuscript with SHK. AK conducted interviews and qualitative analyses and con-tributed to writing of the manuscript. IG conducted interviews and contributed tothe internal and external workshops and qualitative analyses. BI conducted inter-views and contributed to the internal and external workshops and qualitative ana-lyses. FH contributed to the quantitative analyses. SP conducted the finalquantitative analyses. CM co-led the design of the study and contributed to theoverall analyses. SG co-led the design of the study and contributed to the overallanalyses and final writing of the manuscript. SHK co-led the design of the study,contributed to the qualitative analyses, and co-led writing of the final manuscriptwith HB. All authors have read and approved the final manuscript.

FundingThis article was written as part of the Policy Research in Science andMedicine (PRiSM) unit, which was commissioned and funded by the PolicyResearch Programme in the Department of Health. This is an independentarticle by the PRiSM unit; the views expressed are not necessarily those ofthe Department of Health. The funder was not involved in the design of thestudy and collection, analysis, and interpretation of data and in writing themanuscript.

Availability of data and materialsData was provided by the National Institute of Health Research (NIHR) forthe purposes of this research. Requests to view will be referred to NIHR.

Ethics approval and consent to participateApproval from King’s Research Ethics Office, Reference: MR/16/17–365.Consent was obtained from all participants included in the study.Participants were sent a letter by email which outlined the nature of theproject and what participation involved, and made clear that participationwas entirely voluntary. It was not deemed practical to obtain written consentas participants were all interviewed over the phone and located in differentparts of the country. This procedure was approved by the ethics committee,who note that where written consent is not practical, verbal consent may beobtained. Verbal consent was given to take part in interviews, andparticipants approved the use of quotes included in the study.

Consent for publicationNot applicable.

Competing interestsThe authors declare that they have no competing interests.

Author details1The Policy Institute, King’s College London, 22 Kingsway, London WC2B 6LE,UK. 2RAND Europe, Westbrook Centre, Cambridge CB4 1YG, UK.

Received: 19 November 2018 Accepted: 20 January 2020

References1. Morgan MM, Grant J. Making the grade: Methodologies for assessing and

evidencing research impacts. Dean, A., Wykes, M. and Stevens, H.(eds). 7Essays on Impact 2013;7:25–43.

2. Clarke A, Gatineau M, Grimaud O, Royer-Devaux S, Wyn-Roberts N, Le Bis I,Lewison G. A bibliometric overview of public health research in Europe. EurJ Public Health. 2007;17(suppl_1):43–9.

3. Acheson D. Public health in England: the report of the Committee ofInquiry into the future development of the public health function; 1988.

4. Bornmann L. What is societal impact of research and how can it beassessed? A literature survey. J Am Soc Inf Sci Technol. 2013;64(2):217–33.

5. Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, anddefinitions of research impact: a review. Research Evaluation. 2014;23(1):21–32.

6. Greenhalgh T, Fahy N. Research impact in the community-based healthsciences: an analysis of 162 case studies from the 2014 UK researchexcellence framework. BMC Med. 2015;13(1):232.

7. Hinrichs-Krapels S, Grant J. Exploring the effectiveness, efficiency and equity(3e’s) of research and research impact assessment. PalgraveCommunications. 2016;2:16090.

8. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question:understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20.

9. Meagher L, Lyall C, Nutley S. Flows of knowledge, expertise and influence: amethod for assessing policy and practice impacts from social scienceresearch. Research Evaluation. 2008;17(3):163–73.

10. Ward V. Why, whose, what and how? A framework for knowledgemobilisers. Evidence Policy. 2017;13(3):477–97. https://doi.org/10.1332/174426416X14634763278725.

11. Vindrola-Padros C, Pape T, Utley M, Fulop NJ. The role of embedded researchin quality improvement: a narrative review. BMJ Qual Saf. 2017;26(1):70–80.

12. Heaton J, Day J, Britten N. Collaborative research and the co-productionof knowledge for practice: an illustrative case study. Implement Sci.2015;11(1):20.

13. Guthrie S, Kirtley A, Garrod B, et al. A 'DECISIVE' Approach to ResearchFunding: Lessons from Three Retrosight Studies. Rand Health Quarterly.2016;6(1):6. https://europepmc.org/article/med/28083434.

14. Wooding S. Mental health Retrosight: understanding the returns fromresearch:(lessons from schizophrenia): policy report. RAND Europe; 2013.

15. King’s College London and Digital Science. The nature, scale andbeneficiaries of research impact: an initial analysis of Research ExcellenceFramework (REF) 2014 impact case studies. Report to the Higher EducationFunding Council for England. Bristol: HEFCE; 2015.

16. Guthrie S, Bienkowska-Gibbs T, Manville C, Pollitt A, Kirtley A, Wooding S.The impact of the National Institute for Health Research Health TechnologyAssessment programme, 2003-13: a multimethod evaluation. HealthTechnol Assess. 2015;19(67):1-291. https://doi.org/10.3310/hta19670.

17. Zardo P, Collie A. Type, frequency and purpose of information used to inform publichealth policy and program decision-making. BMC Public Health. 2015;15(1):381.

18. Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S. Theuse of research evidence in public health decision making processes:systematic review. PLoS One. 2011;6(7):e21704.

19. Buxton M, Hanney S. How can payback from health services research beassessed? J Health Services Res Policy. 1996;1(1):35–43.

20. Raftery J, Hanney S, Greenhalgh T, Glover M, Blatch-Jones AJ. Models andapplications for measuring the impact of health research: update of asystematic review for the health technology assessment programme. HealthTechnol Assess. 2016;20(76):1–254.

21. Guegan EW, Dorling H, Ollerhead L, Westmore M. Mapping public healthresearch across the National Institute for Health Research 2006–2013. BMCPublic Health. 2016;16(1):911.

22. Lockwood A, Walters H. Making the most of public health research. J PublicHealth. 2018;40(4):673–4.

23. Department of Health. The Public Health Outcomes Framework for England,2013-2016. Crown copyright, 2012.

24. Redman S, Haynes A, Williamson A. Research impact: neither quick nor easy.BMC Med. 2015;13(1):265.

25. Spaapen J, van Drooge L. Introducing ‘productive interactions’ in socialimpact assessment. Research Evaluation 2011;20(3):211–18.


https://doi.org/10.1332/174426416X14634763278725

https://doi.org/10.1332/174426416X14634763278725

https://europepmc.org/article/med/28083434

https://doi.org/10.3310/hta19670

26. Kneale D, Rojas-García A, Raine R, Thomas J. The use of evidence in Englishlocal public health decision-making: a systematic scoping review.Implement Sci. 2017;12(1):53.

27. van der Graaf P, Cheetham M, McCabe K, Rushmer R. Localising andtailoring research evidence helps public health decision making. HealthInformation Libraries J. 2018;35(3):202–12.

28. Willis C, Riley B, Lewis M, Stockton L, Yessis J. Guidance for organisationalstrategy on knowledge to action from conceptual frameworks and practice.Evidence Policy: J Res Debate Practice. 2017;13(2):317–41.

29. Cheetham M, Wiseman A, Khazaeli B, Gibson E, Gray P, Van der Graaf P,Rushmer R. Embedded research: a promising way to create evidence-informed impact in public health? J Public Health. 2018;40(suppl_1):i64–70.

30. Vindrola-Padros C, Eyre L, Baxter H, Cramer H, George B, Wye L, Fulop NJ, UtleyM, Phillips N, Brindle P, Marshall M. Addressing the challenges of knowledgeco-production in quality improvement: learning from the implementation ofthe researcher-in-residence model. BMJ Qual Saf. 2019;28(1):67–73.

31. Upton S, Vallance P, Goddard J. From outcomes to process: evidence for a newapproach to research impact assessment. Research Evaluation. 2014;23:352–65.

32. Bayley JE, Phipps D. Building the concept of research impact literacy.Evidence & Policy: A Journal of Research, Debate and Practice. 2019 Nov 1;15(4):597-606.

33. Hunter DJ. Relationship between evidence and policy: a case of evidence-based policy or policy-based evidence? Public Health. 2009;123(9):583–6.

34. Macintyre S. Evidence in the development of health policy. Public Health.2012;126(3):217–9.

35. Kingdon JW, Thurber JA. Agendas, alternatives, and public policies. Boston:Little, Brown; 1984.

36. Cairney P. Three habits of successful policy entrepreneurs. Policy Polit. 2018;46(2):199–215.

37. O’Campo P. Are we producing the right kind of actionable evidence for thesocial determinants of health? J Urban Health. 2012;89(6):881–93.

38. Pettman TL, Armstrong R, Doyle J, Burford B, Anderson LM, Hillgrove T,Honey N, Waters E. Strengthening evaluation to capture the breadth ofpublic health practice: ideal vs. real. J Public Health. 2012;34(1):151–5.

39. Fischer AJ, Threlfall A, Meah S, Cookson R, Rutter H, Kelly MP. The appraisal ofpublic health interventions: an overview. J Public Health. 2013;35(4):488–94.

40. van der Graaf P, Forrest LF, Adams J, Shucksmith J, White M. How do publichealth professionals view and engage with research? A qualitative interviewstudy and stakeholder workshop engaging public health professionals andresearchers. BMC Public Health. 2017;17(1):892.

41. Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrativereview. BMC Med. 2016;14(1):78.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.


Documents

Mechanisms and pathways to impact in public health ... · health research and outcomes. The pathways to impact observed in our in-depth case studies include contributing to debates