Assessing the Impact of Health Research and Innovation: A Systems Approach
CASRAI Canada ReConnect14 November 20, 2014
Dr. Kathryn Graham Director, Performance Management and Evaluation
Alberta Innovates Health Solutions
Overview
1. Understanding health research and innovation impact: A Canadian approach
2. Promising tools for addressing system needs
3. Building capacity to optimize the Canadian environment in assessing impact
Impact Assessment Defined
Assesses the changes that can be linked to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. …...
(Source: ISRIA 2014; World Bank; GAO 2012)
“Making a Difference”
CAHS Impact Framework (con’t)
Five Impact Categories: 1. Advancing Knowledge2. Capacity Building3. Informing Decision Making4. Health Benefits5. Broad Economic and Social Benefits
CAHS Impact Framework (con’t)
Library of 66 Impact Indicators
CAPACITY BUILDING Indicator Description Level of Application Category
PERSONNEL
Graduatedresearch students in health‐ relatedsubjects
* Number of graduated PhD/ MSc/MD, annual, year over year* Should be able todisaggregate to subjects, gender, etc.
* Not recommended atthe individual level* Can be used atinstitutional level* Most usefulprovincially or nationally
* As an aspiration, we would like to track the success of training programs in producing outstanding scientists and the progress that all research graduates make* Could be done in part using the StatisticsCanada National Graduate Survey
System Needs and Challenges
• “Large-scale social change requires broad cross-sector coordination, yet the social sector remains focused on the isolated intervention of individual organizations….
Source: FSG website – Collective Impact
• The funding ecosystem consists of multiple funders investing in multiple research projects with many contributions. As funders we are challenged to tell policy and decision makers about the overall impact of our research and innovation investments
Source: Trochim, W., Kane, C., Graham, M., and Pincus, H. (2011). Evaluating Translational Research: A Process Marker Model. Clinical and Translational Sciences 4, 153-162.
Systems Approach: Horizontal
1. Understand the context
2. Identify the assessment purpose
3. Measure: Define Indicators of success
4. Develop the design, methods and data collection
5. Communicate and use findings
6. Manage the assessment
Common Approach: Alberta Protocol for Research Impact Assessment
INFORMING DECISION MAKING COMMERCIALIZATION
$309 Leveraged
$51 Invested
LEVERAGED FUNDING (Millions) CAPACITY BUILDING
AIHS: Measuring Impacts Quantitatively (ROI Dashboard)
AIHS: Measuring Impacts Qualitatively
AIHS was part of a three- year international study to improve the translation of mental health research from bench to bedside using 25 years worth of data.
In 2011-12, AIHS invested over $11 million and supported projects in mental health…
95
“Research that had researchers span boundaries had higher impact ”
Alberta Innovates Health Solutions
• Across 4 Research Pillars
• 12 Areas of Strategic Focus
Research Capacity
• $20m invested
• $28m in attracted funding
•209 research investigators
• 37 international recruitments
Res
ea
rch
Res
ult
s Knowledge Pool
1196
C
oll
ab
ora
tio
ns
wit
h o
the
r R
es
ear
ch
ers
Health System Collaborations
• Task forces• Primary care networks
24 partnerships with Industries
•17 Reports of invention/invention disclosures
Collaborations with Government
• Change in policies
collaborations with Non-profits
• Guidelines
Health Care•Improvements to health care effectiveness• Improvements to health care efficiency & accessibility
Improvements in:• Therapeutics • Diagnostic tools• Clinical practice guidelines
Determinants of Health• Personal behaviour• Social/cultural determinants• Environmental
Improvements in health and wellbeing • Morbidity• Mortality• Quality of life
Economic & Social Impact
Impacts feed back into inputs for future research
National/ Global Research
Research activity
That produces results
That influences decision making in….
That affects healthcare and health risk factors
That contribute to changing health, social and economic wellbeing
Health Status and function, well-being, economic conditions
Research Diffusion
AdvancingKnowledge
Capacity Building
InformingDecisionMaking
HealthImpact
SocioeconomicImpact
• 750 publications
• 49% peer review
• 32% abstracts
•552 public outreach activities
• 107 at an international level
• Invited talks
•11 Patent applications • 6 Spin-off companies
The Public• Media events
AIHS: Measuring Across the CAHS Framework
Systems Common Tool: Measuring Impact
• A research portal to capture and analyze common research outputs so impacts can be tracked over time and across funders
• Can reduce administrative burden and be cost effective
• Provides an opportunity to capture “true” impact and contributions
3. Building Capacity to Optimize the Canadian Environment in
Assessing Impact
AIHS website for work on health research impact and the international school http://www.aihealthsolutions.ca/performance.
Key Messages• Share frameworks and tools to support the
optimization of research and innovation impact in Canada
• Common approach - CAHS Impact Framework- Alberta Protocol
• Common language - CASRAI
• Common tools and measurement systems
• Continue collaborative efforts to measure the impacts of the Canadian health research and innovation system
Thank You
Dr. Kathryn GrahamDirector of Performance Management and EvaluationAlberta Innovates - Health [email protected]