Iscrr Impact Assessment Report

of 32

Return on Investment (ROI) Project

Final Report 26th October 2012

Authors: Dr Alex Collie, Chief Research Officer

Ms Pauline Zardo, Research Translation Officer Dr Sue Gargett, Health Economist

Ms Jane Monk, Consultant

For further information regarding this report please contact: Dr Alex Collie

Chief Research Officer Institute for Safety Compensation and Recovery Research

Level 11, 499 St Kilda Road, Melbourne 3004 VIC, Australia Email: [email protected]

Telephone: +61 (0) 3 9097 0610
mailto:[email protected]

of 32

EXECUTIVE SUMMARY A primary objective of the Institute for Safety Compensation and Recovery Research (ISCRR) is to conduct excellent health and safety, compensation and recovery research that positively impacts the performance of WorkSafe Victoria and the Transport Accident Commission. Assessment of its impact, including return on investment is one of ISCRRs key performance indicators. In late 2011, the ISCRR Board initiatied a Return on Investment (ROI) project to assess the impact of ISCRRs research on WorkSafe and the TAC, and to establish a method for assessing the impact of ISCRRs research that can be embedded in ongoing operations of the institute. The ROI project has three components, including:

1. A series of nine detailed case studies of ISCRR research projects to assess in depth the qualitative and quantitative impact of the projects on WorkSafe, the TAC and their stakeholders.

2. A content analysis of information captured in ISCRRs project management system to (a) assess the extent of adoption of ISCRR research by WorkSafe, the TAC and their stakeholders and (b) identify factors that lead to adoption of ISCRR research.

3. A content analysis of ISCRR project level information and the findings of the case studies to determine the number of projects for which impact could be demonstrated as well as the types of impact.

The total financial investment in the nine case studies to date is $725,000. The financial return on investment has included a $1.5 million actuarial release from the Noise Induced Hearing Loss project (a five fold ROI), a $1.8 million cost avoided from the Implantable Pain Therapies evidence review, and costs avoided of up to $6.8 million for the Body Weight Supported Treadmill Training evidence review. The financial impact of three TAC focussed projects is yet to be determined. These estimates will be known in coming months as the impact of the research on claims costs at TAC become clearer. In addition, the non-financial value of the projects has been substantial, and includes positive changes in the culture of evidence-based decision making at both WorkSafe and the TAC, increasing the understanding within the TAC of the impact of claims management processes on TAC clients, improvements in research capability amongst leading injury research group at Monash, enabling the implementation of Recovery model at the TAC, establishing a basis for important health policy positions at WorkSafe and the TAC, and enabling enhanced interactions between rehabilitation coordinators and TAC clients. The research has also enabled the continuance of an important workplace safety debate in the community (i.e., quad bikes safety devices). The impact of the case studies is summarised in the table below. Analysis of data captured in ISCRRs project management system identified evidence of use of ISCRRs research in 27 of 30 individual projects that had produced output by end August 2012. In 17 of these projects qualitative information demonstrating the impact of the projects on WorkSafe, the TAC or their stakeholders has been collected. Content analysis identified five major types of impact, being financial impacts (such as reductions in actuarial liabilities and costs savings), claims processing and claims decision making impacts, impacts on employers and the broader community. In addition, one ISCRR project has demonstrated an impact on the health and wellbeing of compensable clients with severe injury.

of 32

Table: Summary of ROI from nine case studies

CASE STUDY INVESTMENT RETURN

Noise Induced Hearing Loss

$312,000 $1.5 million actuarial release

Allowed WorkSafe to focus its strategic response to growing hearing loss costs

Quad bike Safety Devices $10,500 Re-ignited public debate around important workplace safety issue

Contributed to a 'case for change' in safety standards

Contributed to community behavioural change

Client Perceptions of Recovery

$37,700 Enhanced understanding of TAC role in improving client experience

Provided information for the education of claims officers

Provided opportunity for key researcher to engage with TAC

Contributed to capacity building in research team at Monash

Implantable Pain Therapies

$73,400 $1.8 million in costs avoided consisting of $0.65m in costs saved over 3 years to June 2012 and $1.16 million in future costs avoided.

Contributed to change in culture towards evidence based decision making

Helped WorkSafe to contain a potential "cost-blowout"

Leant credibility of WorkSafe policy of reviewing IPT requests

Body Weight Supported Treadmill Training

$65,600 Contributed to a shift towards evidence-based decision making in TAC.

Provided a foundation for an important health services policy position.

$17k to $342k in current costs avoided, depending on scenario modelled.

Potential liability saving of up to $6.8 million in costs avoided.

Musculoskeletal Disorders Toolkit

$71,600 In one participating organisation, involvement in the research led to:

increased knowledge and skills re work-related MSK disorders,

increased communication on OHS in the workplace,

increased involvement of staff in OHS issues, and

the introduction of initiatives to enhance skill development.

Pelvic Ring Fractures $36,000 Improved physical and mental health outcomes for severely injured TAC clients where there was a surgical intervention.

Financial impact yet to be determined.

of 32

Evaluation of the Client Conversational Tool

$106,300 Provides TAC Rehabilitation Coordinators with a greater insight into client needs.

The CCT-revised is more usable, more likely to be administered in specified timeframes and more likely to be completed by rehabilitation coordinators than the CCT.


Return to Work Predictive Model

$10,000 plus $500 in ongoing operational costs p.a.

Enabled Recovery model implementation

Led to earlier interactions with TAC clients

Facilitated enhanced client / rehabilitation coordinator relationships.


Content analyses identified eight factors that influence the adoption of ISCRRs research including engagement and interactions, alignment with partner strategic priorities, timeliness of research, ease of implementation, partner organisations structure and processes, risk and issue prioritisation, the presence (or absence) of an internal champion for the research, and the credibility of the research method and source. This analysis provide substantial value to ISCRR, as it informs ongoing development of our project development and translation system to ensure that research findings are more likely to be adopted. This analysis identified a substantial return on investment from ISCRR projects to date. While there is some inherent uncertainty in the financial estimates, the financial ROI calculated for three of the case studies demonstrates a return on investment that is many times the investment in the research. Other projects have demonstrated the substantial non-financial value for WorkSafe and the TAC from ISCRR research. The analysis has been undertaken on a small proportion of completed projects from the ISCRR research portfolio. As ISCRRs research program matures the financial and non-financial benefits to WorkSafe and the TAC may grow.

of 32

CONTENTS Section Page

1. Title Page 1

2. Executive Summary 2

3. Contents 5

4. Acknowledgments 6

5. Background and Context 7

6. Objectives 10

7. Project Governance 11

8. Methods 12

9. Impact Assessment Framework 13

10. Case studies 16

11. Adoption of ISCRR research 18

12. Impact of ISCRR research 24

13. Summary and conclusions 26

14. References 29

15. List of Appendices 30

of 32

ACKNOWLEDGEMENTS Collation of a body of work such as this is only possible with the active involvement of a large number of people. The authors would like to acknowledge the substantial number of individuals within ISCRRs three partner organisations who have contributed to this report. Names and position titles of individuals who have contributed information to the case studies are included in the appendices. In addition we would like to acknowledge Ms Khic Huoy-Prang from ISCRR who assisted with data analysis for some case studies, and Mss May Cheng and Dora Qian from Corporate Finance, Monash who assisted with financial statements data. Of particular note we would like to thank the members of the ISCRR Return on Investment Committee:

Mr Chris Latham (ISCRR) and Professor Niki Ellis (ISCRR) for providing leadership throughout the ROI project including chairing the committee meetings and ensuring the project met the diverse needs of ISCRRs partners.

Dr Duncan Buckeridge (Monash) for critical input into the development of the Impact Assessment Framework and methodology for assessing quantitative impact throughout the project.

Mr David Gifford (TAC) and Ms Julie Evans (WorkSafe Victoria) for substantial contribution to methods development and for overseeing the quantative evaluation on a number of case studies.

Mr Tehn Chin (WorkSafe Victoria) and Mr David Attwood (TAC) for contributions to case studies including undertaking a number of quantative analysis and regular engagement and interaction with the project team.

Mr Tony Tenaglia (ISCRR) for contributions to individual case studies and strategic input throughout the project.

Ms Gillian Syres (ISCRR) for early leadership of the ROI project including contributions to developing the Impact Assessment Framework.

The project would not have been possible without the substantial contributions of the ROI group.

of 32

BACKGROUND AND CONTEXT

The Institute for Safety Compensation and Recovery Research ISCRR was established in April 2009 as a joint initiative of WorkSafe Victoria (WorkSafe), the Transport Accident Commission (TAC) and Monash University (Monash) to provide a sound research and evidence base to improve compensation scheme policy, practice and performance. The collaboration provides an opportunity to achieve a best practice model in multi-disciplinary industry-led research and to provide support for TAC and WorkSafe strategic directions and future operations. ISCRRs goals, as identified in its Strategic Plan to 2015, are to:

1. Have high impact on compensation scheme performance. Set, deliver and translate a research agenda that will positively impact on WorkSafe and TAC scheme operations and performance, both in the short and long-term.

2. Create a model of excellence for industry led research. Create a best practice model for multi-disciplinary, industry-driven research based on strong stakeholder engagement and supported by a broad funding base.

3. Be acknowledged leaders in compensation scheme research. Maximise the value of research undertaken by driving local and overseas partnerships and leading the national research effort in our areas of focus.

Measuring research impact There is a growing focus amongst academic research organisations and research funding agencies on conducting research with real world impact and on the measurement of the return on research investment. This is embodied in part by the Monash Universitys research strategy which has the goals of conducting both excellent research and research with impact. Increasingly, Australian and international research organisations are developing methods of measuring return on investment (ROI) in research (Clay et al, 2006; Collie 2010, Kingwell et al, 2006). For example the Murdoch Childrens Research Institute (MCRI) has developed a method that includes evaluation of eight indicators across three categories: knowledge creation; inputs to research; and commercial, clinical and public health outcomes (Schapper et al, 2012). With a few notable exceptions (Insight Economics, 2006) most efforts focus on assessing ROI in areas relevant to academic research institutions, such as knowledge dissemination, research income and capacity building. ISCRR is uniquely placed to assess the impact of its research on both conventional academic indicators and on the policies and practices of its partner funding agencies. ISCRR conducts applied research and has established strong collaborative relationships with WorkSafe and the TAC that should facilitate the use of its research. This final report describes the outcomes of the institutes ROI project to 18th October 2012.

of 32

The ISCRR research pipeline At the time of writing, ISCRR had been operating for 3 years and 6 months (April 2009 to October 2012). As per the Strategic Plan published in 2010, ISCRR is coming towards the end of its development phase.1 A feature of this stage was delivering short term value to the institutes partner organisations (Figure 1, taken from the ISCRR Strategic Plan 2010 to 2015). This project seeks to assess the short term value delivered to WorkSafe and the TAC. A second purpose for the project is to develop an impact assessment methodology that can be integrated into ISCRRs ongoing program management systems beyond 2012.

Figure 1. ISCRR phases of development

Evidence from previous Australian research impact evaluations suggests that on average, evidence of economic impact requires nine years to realise from the point of research initiation (Insight Economics, 2006). During its 3 years of operations, ISCRR has established a substantial research portfolio encompassing over 100 individual projects. This ROI project has focussed on those that have reached the adoption and impact stage of the research pipeline (Figure 2)2. These projects represent less than a quarter of all of ISCRRs current research projects.

1 ISCRR Strategic Plan, 2010 to 2015 2 Data taken from ISCRR June 2012 quarterly research status report. Note that projects can be in multiple stages at any

point in time (eg, active and producing output).

of 32

Figure 2. The ISCRR research pipeline as at 30 June 2012

As ISCRR has developed, there has also been a substantial growth in research output (Figure 3), measured both in terms of reports to WorkSafe and the TAC and peer reviewed academic research publications. Many of the instances of research use (or adoption) and impact cited in this report have arisen from projects that produced output during the 9/10 and 10/11 financial years. As ISCRRs research outputs grows, it is reasonable to expect a growth in instances of research adoption and impact, as the outputs are used by WorkSafe, the TAC and their stakeholders.

Figure 3. ISCRR research output 2009/10 to 2011/12.

0

5

10

15

20

25

30

35

2009/10 2010/11 2011/12

Reports to Partners

Peer-reviewed Publications

of 32

OBJECTIVES The ROI project was initiated by the Board of ISCRR in late 2011 to develop a method for assessing the impact of ISCRRs research on WorkSafe and the TAC. The output from the project is one input to the ISCRR review being conducted in late 2012. It is anticipated that the outcomes of the ROI project will be considered by the review team. The ROI project has four objectives: 1. To develop a framework for assessing research impact; 2. To determine the impact of ISCRR research on WorkSafe and the TAC; 3. To determine the factors that have contributed to the adoption of ISCRR research by

WorkSafe and the TAC; and 4. To develop a model for measuring return on investment that can be embedded in

ISCRRs ongoing operations. This document is the Final Report of the project. It addresses all of the above objectives.

of 32

PROJECT GOVERNANCE The ROI project is an initiative of the ISCRR Board and is governed by a steering committee (the ROI committee) that reports to the ISCRR Board. Formation of the ROI committee was approved at the February 2012 Board meeting and noted at the March 2012 Relationship Management Committee (RMC) meeting. The committee includes representation from WorkSafe Victoria, the TAC, Monash and ISCRR, and is chaired by the Acting Chair of the ISCRR Board. Members include:

Mr Chris Latham (Chair)

Mr David Gifford, Head Business Intelligence and Risk, TAC

Mr David Attwood, Senior Manager Claims Research, TAC

Ms Julie Evans, Manager Actuarial Services, WorkSafe Victoria

Mr Tehn Chin, Actuarial Consultant, WorkSafe Victoria

Mr Duncan Buckeridge, Research Strategy Advisor, Monash University

Prof Niki Ellis, CEO, ISCRR

A/Prof Alex Collie, Chief Research Officer, ISCRR

Mr Tony Tenaglia, Corporate Services Manager, ISCRR A project team based at ISCRR reports to the ROI committee. The project team is responsible for conducting the ROI project and includes the following members:

A/Prof Alex Collie, Chief Research Officer, ISCRR

Dr Susan Gargett, Health Economist, ISCRR

Ms Jane Monk, Consultant, ISCRR

Ms Pauline Zardo, Research Translation Officer, ISCRR The ROI committee has met monthly from February 2012 to guide the project.

of 32

METHODS The methods developed for the ROI project are described in a separate document attached in Appendix A. In brief there are three separate components to the project:

1. A series of nine case studies of ISCRR research projects to assess in depth the qualitative and quantitative impact of the projects on WorkSafe, the TAC and their stakeholders.

2. A content analysis of information routinely captured in ISCRRs project management system to (a) assess the extent of adoption of ISCRR research by WorkSafe, the TAC and their stakeholders and (b) identify factors that lead to adoption of ISCRR research.

3. A content analysis of ISCRR project level information and the findings of the case

studies to identify types of impact. The methods document was endorsed by the ROI Committee at their August 2012 meeting and is provided at Appendix A.

of 32

IMPACT ASSESSMENT FRAMEWORK The first component of the ROI project entailed the development of an Impact Assessment Framework. The Framework was adapted from models developed by Duncan Buckeridge and further developed by the ROI project team in collaboration with the ROI committee over the period December 2011 to March 2012. The framework was updated and finalised in September 2012. The Framework was used to guide the case studies and the adoption and impact content analyses. In essence, the Framework describes five stages of the research continuum and the methods that may be used to assess research projects at each stage of the continuum. The five stages are: 1. Planning and development of research. 2. Conduct of research. 3. Outputs of research. 4. Adoption of research outputs. 5. Impact of research.

At any point in time, individual research projects may be located at one or more points along this continuum. It is anticipated that all research projects will produce outputs, and that most, ideally all, of these will lead to adoption and impact.

Each of the five phases has multiple, varied components. For example, a variety of research methods may be used in the conduct of research. Similarly, adoption of research may be indirect (e.g., influencing the thinking of a person or group at WorkSafe or the TAC) or direct (e.g., used to establish or support a policy position). The impact of research may be measured qualitatively (e.g., influencing a strategic discussion within the schemes) or quantitatively (e.g., as a financial impact, reduction in claim lodgement).

The Framework establishes a foundation for the ROI evaluation methods, and is now the basis for ISCRRs routine research reporting. The Framework document is provided at Appendix B, and was endorsed by the ROI committee at their September 2012 meeting.

of 32

CASE STUDIES A series of nine case studies were identified by the ROI committee for detailed analysis of return on investment. Cases represent a sample of the portfolio of ISCRR research projects, selected using a set of criterion described in the methods document. Cases could be either stand-alone research projects or sub-studies arising from larger research projects.The nine case studies are listed in Table 1. Table 1. List of ROI cases agreed for inclusion

CASE STUDY TARGET ORGANISATION

RESEARCH PROGRAM

ROI METHOD

Return to Work Predictive model TAC RTW/Recovery Qual & Quant* Noise induced hearing loss WS OHS Qual & Quant Implantable pain therapies review WS/TAC HDSD Qual & Quant Body weight supported treadmill training

TAC HDSD Qual & Quant

Quad bike safety devices WS OHS Qual only Client conversational tool TAC Comp Systems Qual & Quant* Client perceptions of recovery TAC Comp Systems Qual only Pelvic ring fractures TAC RTW/Recovery Qual & Quant* Toolkit for MSK disorders WS OHS Qual only Note: Qual = qualitative assessment of impact; Quant = quantitative assessment of financial impact except for asterisked case studies (*) where quantitative measurements included non financial metrics.

Nine case study documents are appended to this report (Appendices C to K). For three of the case studies the financial impact is yet to be evaluated, including:

Pelvic Ring Fractures

Client Conversational Tool

Return to Work Predictive Model These three case studies will be completed in coming months as the required TAC claims cost information becomes available for analysis. Each case study contains the background to the research project and an assessment of the qualitative and/or quantitative impact of the research on WorkSafe and/or the TAC and/or their stakeholders. The case studies represent a cross section of ISCRRs research portfolio, including projects from the Occupational Health and Safety, Health and Disability Services Delivery and Return to Work research programs. The case studies also include evidence reviews, a development grant and a stand-alone project. The case studies are included as appendices to this document.

of 32

Summary of case study outcomes Table 2 summarises the output of the case studies as at 27th October.

Table 2. Summary of case study outcomes

CASE STUDY INVESTMENT RETURN

Noise Induced Hearing Loss

$312,000 $1.5 million actuarial release

Allowed WorkSafe to focus its strategic response to growing hearing loss costs

Quad bike Safety Devices $10,500 Re-ignited public debate around important workplace safety issue

Contributed to a 'case for change' in safety standards

Contributed to community behavioural change

Client Perceptions of Recovery

$37,700 Enhanced understanding of TAC role in improving client experience

Provided information for the education of claims officers

Provided opportunity for key researcher to engage with TAC

Contributed to capacity building in research team at Monash

Implantable Pain Therapies

$73,400 $1.81m in costs avoided consisting of $0.65m in costs saved over 3 years to June 2012 and $1.16m in future costs avoided.

Contributed to change in culture towards evidence based decision making

Helped WorkSafe to contain a potential "cost-blowout"

Leant credibility of WorkSafe policy of reviewing IPT requests

Body Weight Supported Treadmill Training

$65,600 Contributed to a shift towards evidence-based decision making in TAC.

Provided a foundation for an important health services policy position.

$17k to $342k in current costs avoided, depending on scenario modelled.

Potential liability saving of up to $6.8 million in costs avoided.

Musculoskeletal Disorders Toolkit

$71,600 In one participating organisation, involvement in the research led to:

increased knowledge and skills re work-related MSK disorders,

increased communication on OHS in the workplace,

increased involvement of staff in OHS issues, and

the introduction of initiatives to enhance skill

of 32

development.

Pelvic Ring Fractures $36,000 Improved physical and mental health outcomes for severely injured TAC clients.


Evaluation of the Client Conversational Tool

$106,300 Provides TAC Rehabilitation Coordinators with a greater insight into client needs.

The CCT-revised is more usable, more likely to be administered in specified timeframes and more likely to be completed by rehabilitation coordinators than the CCT.


Return to Work Predictive Model

$10,000 plus $500 in ongoing operational costs p.a.

Enabled Recovery model implementation

Led to earlier interactions with TAC clients

Facilitated enhanced client / rehabilitation coordinator relationships.


A diverse range of impacts have emerged from the projects studied, including both qualitative and financial or other quantitative impacts. As at 3rd October 2012, it has been possible to demonstrate a financial ROI in three of the case studies. These include a $1.5 million actuarial release realised for WorkSafe from the Noise Induced Hearing Loss project, $1.8 million cost avoided from the Implantable Pain Therapies evidence review, and a potential in costs avoided of up to $6.8 million for the Body Weight Supported Treadmill Training evidence review. The total financial investment in the nine case studies to date has been approximately $725,000. Qualitative feedback received has demonstrated the substantial non-financial value of ISCRRs research. For example, in relation to the Implantable Pain Therapies project (and other evidence reviews completed by ISCRR), it has been observed by senior clinical convenor of the HSG that ISCRR research has contributed to a culture of evidence based decision making at WorkSafe and the TAC:

Evidence Service Reviews are the most applied aspect of ISCRRs work. Evidence Service Reviews are designed to give WorkSafe and the TAC the upper ground

in discussion on clinical decisions and sharpens the focus of practitioners in their clinical decision making.

Dr Peter Harcourt, Clinical Convenor, Health Services Group

Other qualitative feedback indicated that ISCRR research had sparked a public debate on an important workplace safety issue. In relation to the evidence review on crush protection devices on quad bikes, it was noted that the report

reignited tension in the debate regarding the use of Crush Protection Devices in preventing Quad bike fatalities

of 32

Ross Pilkington, Director, Health and Safety Operations, WorkSafe Victoria Still other senior executives noted the potential of ISCRRs research to influence future decision making at WorkSafe and the TAC. For example, in relation to the Client Perceptions of Recovery project: In my view, this is one of the most important external pieces of research TAC has received,

from the client perspective. David Gifford, Executive Manager, Business Intelligence, TAC

ISCRRs research program has also delivered value for Monash. In relation to the Client

Perceptions of Recovery project, the lead researcher Associate Professor Belinda Gabbe

noted that:

this research was the first qualitative research undertaken within the context of the Victorian

State Trauma Registry and the Victorian Orthopaedic Trauma Outcomes Registry. The

findings of this study have been hugely beneficial for improving our understanding of the

trauma patients experience with the trauma system and the impacts of injury. In addition,

the project provided a great opportunity to build capacity within the research team to

undertake Qualitative Research. In addition to building Qualitative Research capacity, strong

international links were established with experienced social scientists in the field from the

University of Surrey and University College London. The collaborations established will

extend into future projects."

The case studies provide a snapshot of the return delivered via a relatively small series of ISCRR projects. It is anticipated that in future, similar analyses of projects currently in the development, activity or output stage of the research cycle will reveal a further diversity of impact and demonstrate the value of ISCRRs research to its partner organisations.

of 32

ADOPTION OF ISCRR RESEARCH Another component of the ROI project has been analysis of all ISCRR research projects to determine the factors that have contributed to the adoption of research by WorkSafe and/or the TAC. This component seeks to address two questions, being:

What attributes of a project lead to research findings being adopted and used to inform policy, programs and practice?

What attributes of a project lead to research findings NOT being adopted and used to inform policy, programs and practice?

The term project is used in its broadest sense to include the research, project management, project governance and communications. The detailed methods for this analysis are described in Appendix A.

Factors influencing adoption of ISCRR research Thirty (30) projects met the criteria for inclusion in the analysis. Note that for the purposes of this analysis, each evidence review produced under the auspices of the Evidence Service project were considered to be individual projects. Of these, ISCRR had recorded evidence of adoption for 27 projects. The remaining 3 projects did not have any evidence of adoption recorded on ISCRRs project management system. Of those projects with evidence of adoption:

18 arose from a specific request from WorkSafe/ TAC for the research.

4 arose from an investigator initiated processes, including development grants.

5 arose from collaborative development of the research scope and methods. The type of adoption was categorised according to the 3 typologies of research utilisation described by Hanney et al (2003) following Weiss (1979). These included:

Instrumental use of research (N=17 projects). This indicates research that was used directly in policy formation or decision making. For example, an evidence review on intrathecal infusions was used to review and revise a treatment payment policy by the TAC/WorkSafe health services group.

Symbolic use of research (N=7 projects). This refers to use of research to support a position already taken, or to support debate for particular potential policy positions or strategic directions, which may be to continue with existing policies or courses of action. For example, the client outcomes evidence review confirmed that the TACs approach to measuring client outcomes as an organisational key performance indicator was unique within Australia and internationally.

Conceptual use of research (N=3 projects). This refers to use of research to provide insight into an issue or problem rather than directly used in policy formation or

of 32

decision making. For example, an evidence review on mediation and return to work has been used to inform the WorkSafe RTW division about the issue, without any specific action being taken to date.

Content analysis identified eight factors as barriers or facilitators of adoption. These are described in the table below.

Table 2. Factors leading to adoption of ISCRR research

ADOPTION FACTORS

DESCRIPTION EXAMPLE

Engagement and Interaction

Sponsor/Contact Engagement

Researcher Engagement

Project Manager engagement

Stakeholder Engagement

Quality of Interactions

The degree and quality of engagement between the researcher/research team and the business sponsor/contact.

Research projects with high levels of engagement between researcher and the project business sponsor/contact were associated with successful adoption. Low sponsor/researcher engagement associated with adoption failure or adoption lags.

High ISCRR Project Management engagement was also associated with effective adoption. Stakeholder engagement (other relevant people involved other than sponsor/contact) was also associated with adoption success. The right stakeholders need to be engaged to enable successful adoption.

Poor quality/limited interaction between parties was associated with lack of adoption; poor quality communication between parties and during interactions was associated with lags or lack of adoption.

The Quad Bike Evidence review was developed following significant interaction and communication between the project key contact and the researcher. The project sponsor also ensured other key stakeholders were engaged. The researcher was responsive to the project key contact throughout the research process, even maintaining contact and positive communications when delivery dates were pushed back.

Both the researcher and the project key contacts engagement and the high quality of thier interactions was critical to the success of this piece of research, which has been experienced as highly relevant, timely and actionable by WorkSafe and the broader injury prevention community.

Alignment with Partner Strategic Priorities

WorkSafe / TACs perception of the relevance of the research with regard to their current priorities.

Research that is perceived as directly

The WorkHealth Evaluation is directly linked to deliverables of the WorkHealth program

of 32

ADOPTION FACTORS

DESCRIPTION EXAMPLE

relevant to current work being undertaken and directly related to user needs was adopted. Research questions that were very specific and resulted in very specific findings were experienced as relevant and were adopted.

Research that was not considered to be well aligned with partner strategic priorities failed to be adopted.

undertaken by WorkSafe.

The researchers measure the uptake and impact of the WorkHealth initiative following its implementation. The on-going provision of results have been used by the project sponsor and contact to communicate the progress and success of the project to WorkSafe and the Victorian community throughout the life of the WorkHealth program.

Ease of Implementation

The extent to which the research findings were actionable, or able to contribute to a decision. Research that provided clear, actionable recommendations were adopted. In other words, research that enabled a decision to be made, whether this was at the conceptual or practical level, was more likely to be adopted. Research where implementation was low cost or required relatively little effort were adopted.

The Evidence Service Review on NENET was adopted quickly after completion of the report. Implementation involved informing clinicians and claims managers about the evidence around NENET, and led to the development and immediate use of a NENET database to capture information about NENET claims management.

Timeliness

Delivery of research while the issue is still being addressed or considered by WorkSafe / the TAC.

Research findings that are delivered within policy timeframes, or while the issue is already being addressed internally, enables adoption. Research findings provided while an issue is considered a hot topic are more likely to be adopted, if this window is missed adoption is less likely. Alignment in timeframes between when research findings are delivered and when policy decisions must be made is important for

The NIHL project was undertaken while there was also an active internal WorkSafe Steering committee addressing the issue. The research was fed directly into this steering committee to inform decision making by the key contact.

This research findings were delivered while this issue was considered a hot topic, and when solutions had not

of 32

ADOPTION FACTORS

DESCRIPTION EXAMPLE

adoption. been identified/finalised or implemented.

Partner Organisational Structures and Processes

WorkSafe and TAC organisational and structural process can affect adoption.

Having a structure or process in place to receive the research and process the findings facilitated adoption. For example, evidence reviews arising from the Evidence Service project were delivered directly to the HSG Health Policy Governance Group.

Substantial changes in organisational process and infrastructure affecting project sponsors and contacts during the period within which research was undertaken were predominantly experienced as a barrier to adoption.

The Work Related Fatalities project, one that was not adopted, experienced difficulties due to a significant organisational restructure occurring during the project. This resulted in the WorkSafe project sponsor and contact changing several times throughout the life of the project. The new business sponsors had a different perspective of the relevance of the research to those who initiated it, contributing to lack of adoption.

Internal champion for research

Research projects that have a strong champion or sponsor for the research within WorkSafe / the TAC have been adopted. Research related to government directives are also likely to have a strong internal champion and to be adopted. Where such ownership has been lacking or where there is internal organisational disagreement about adoption there have been barriers to adoption.

The TAC 2015 Evaluation has had consistent and strong leadership, interest and involvement by senior managers in the TAC. This has been critical to the swift adoption of research arising from this project, for example the implementation of a revised version of the TAC Client Converstional Tool.

Risk and Issue Prioritisation

Research related to areas with high/rising claims costs are more likely to be adopted in a timely fashion.

Research focussed on low priority issues and/or competing priorities were less likely to be adopted or to have a lag in adoption.

Often low priority was described as not a large cost issue or not a rising cost issue.

The Intrathecal Evidence Service Review was adopted quickly as this issue was linked with rising claims costs. Adoption involved the development of a policy and claims management procedure regarding Intrathecal Infusions.

Other evidence service

of 32

ADOPTION FACTORS

DESCRIPTION EXAMPLE

Negative financial impact related to implementing research was associated with non-adoption.

reviews delivered at similar times, e.g. Benzodiapines have been partially adopted (claims managers and clinicians have been using the informatino to inform discussion with clients) but the actual policy document has not been updated to reflect and reference the evidence review because the issue is not linked rising or high claims costs, so not viewed as priority work.

Credibility of Research Method and Source

Credibility of research methods/scientific analysis process supports adoption. Where the researcher is considered a trusted source that enables adoption. That research is understood as unbiased/objective allows for decisions to be made where there is lack of clarity or difference of opinions.

In the case of both the Art Therapy and BWSTT Evidence reviews the credibility and the objectivity of the research was critical to its adoption. The objective evidence provided by the review allowed for a decision to be made where there was prior disagreement about the efficacy of these treatments and whether or not they should be compensated.

Comparison with prior research The factors identified in this content analysis are aligned with factors identified in research and systematic reviews of barriers and enablers to use of research evidence. Engagement and Interaction, Alignment with Partner Strategic Priorities, Timeliness and Risk and Issue Prioritisation are clearly linked with the three factors most frequently identified in systematic reviews: timeliness; interaction between researchers and policy makers; and accordance between researchers and policy maker strategies, priorities, values and interests (Lavis, 2009). Research on use of evidence has also identified the critical role of research champions and leaders. Internal Champion for research reflects these findings, which has found that research that is informed, supported and disseminated by leaders and influential individuals

of 32

is more likely to be perceived as valuable and relevant by key stakeholders and therefore adopted into practice (Mitton, 2007; Greenhalgh, 2004). Similarly, research has also shown that choice of researcher and research communicator is an important influencing factor affecting use of research.Studies have shown that where there is poor choice of researcher or research communicator (i.e. researcher not considered expert or experienced, or the communicator has poor speaking or presentation skills etc) research is unlikely to be adopted into practice (Mitton 2007; Orton et al 2011).The factor Credibility of Research Method and Source is consistent with these findings. In a systematic review by Mitton et al (2007) barriers and enablers to use of evidence were identified as related to either: the individual, the organisation, communication or time. The factors identified in this analysis can also be associated with these categories. Factors exisiting at the individual level and related to communication include: Engagement and Interaction; Internal Champion for Research; and Credibility of Research Method and Source. The organisational level factors identified here were: Alignment with Partner Strategic Priorities; Ease of Implementation; Partner Organisational Structures and Processes; and Risk and Issue Prioritisation. Timeliness is related to time.

Implications for ISCRR Information about the factors affecting use of evidence in specific decision making contexts, such as that provided by this content analysis, can be used to target and tailor the design and implementation of tools, strategies and interventions seeking to increase use of evidence and increase the likelihood of their effectiveness (Greenhalgh, 2004; Pawson 2005; Dobrow 2004; Scott 2012). This analysis has substantial implications for ISCRRs future research activities. It provides a useful framework of the factors that influence the likelihood that research will be adopted. The challenge for ISCRR is to integrate this knowledge into research development and project management processes. The evolution of ISCRRs recently developed programmatic approach to research management provides an opportunity to embed these principles into business as usual processes at ISCRR. For example, one can foresee a checklist of critical factors being established as a set of go/no-go criterion for new projects under consideration by the institutes Program Advisory Groups (PAGs) or Board.

of 32

IMPACT OF ISCRR RESEARCH The final component of the ROI project is a content analysis of those ISCRR research projects that have led to impact. This component seeks to address two questions:

How many ISCRR projects have had a demonstrable impact on WorkSafe, the TAC and/or their stakeholders?

What are the types of impact that ISCRR research has had on WorkSafe, the TAC and their stakeholders?

Once again the term project is used in its broadest sense to include the research, project management, project governance and communications. The detailed methods for this analysis are described in Appendix A.

Number of projects having impact Of the twenty seven projects for which ISCRR had documented evidence of adoption:

17 had demonstrated impact

10 had no documented evidence of impact (yet). Note that for the purposes of this analysis, each evidence review produced under the auspices of the Evidence Service project was considered to be an individual project. Of those projects identified as having impact:

11 arose from a specific request from WorkSafe/ TAC for the research. These were all evidence reviews.

3 arose from an investigator initiated processes. These included 2 development grants and a research fellowship.

3 arose from collaborative development of the research scope and methods. These included 2 evaluations of major WorkSafe and TAC policy initiatives, and a mixed-methods study of noise induced hearing loss.

Of those projects with evidence of impact, all had qualitative evidence of impact in the form of spoken feedback obtained from WorkSafe or TAC staff or documented feedback. The case studies undertaken (described above) also provided quantitative evidence of impact.

Types of impact Impacts ranged from increasing community awareness of health and safety issues to changes in TAC claims processes to improvements in WorkSafe scheme viability via actuarial releases. Content analysis identified five major types of impact. These are described in the table below with examples to illustrate each type.

of 32

Table 3. Types of impact

TYPE OF IMPACT

DESCRIPTION EXAMPLE

Community/public perception impact

Driving improvements in community perceptions, understanding and awareness of workplace safety issues and solutions.

The Quad bike evidence review re-ignited a public debate regarding the use of Rollover Protection Devices on quad bikes used in agricultural settings.

Claims processing impacts

Impact related to changes in operational or decision making processes, particularly claims handling processes.

The evaluation of the Client Conversational Tool (CCT), led to a revision of the tool and a revised process for its administration that was implemented by the TAC in April 2012. Similarly, the RTW predictive model contributed to a new process for early identification of TAC clients at-risk of poor return work outcomes.

Claims decision making impacts

Claims managers and clinical panel members are able to make evidence informed decisions about provision of appropriate treatment and services, including benefits and risks to the client.

The Body Weight Supported Treadmill Training evidence review enabled the TAC to make a decision not to support funding of the Lokomat service for spinal cord injured clients.

Financial impacts

Impacts related to changes in claims costs or liabilities, or costs avoided.

The Noise Induced Hearing Loss project was considered to have led to a $1.5 million reduction in claims liabilities for WorkSafe Victoria.

Employer impacts Impacts around employer awareness and perception of workplace health, safety and compensation issues.

The WorkHealth evaluation has informed employer awareness campaigns that have raised the profile of workplace health promotion in the state.

Client impacts Impacts on TAC client or injured worker health and well-being.

The Pelvic Ring Fractures project demonstrated that TAC clients receiving a certain surgical procedure had superior physical and mental health outcomes than those managed conservatively.

of 32

SUMMARY AND CONCLUSIONS This report of the ROI project sought to determine the impact of ISCRR research on WorkSafe and the TAC and their stakeholders. A second aim was to examine factors that led projects to be adopted and to develop a typology of impact to enable a more nuanced understanding of what things lead to adoption of ISCRR research by WorkSafe and the TAC and how ISCRRs research is being used by the two schemes. Two primary research methods have been used, being a series of detailed case studies and a content analysis of project level information held in ISCRRs program management systems.

What has been the return on investment? The case studies provide evidence of the different ways in which ISCRRs research has delivered value for WorkSafe and the TAC. In some cases it has been possible to measure this value using quantitative methods, including estimation of the economic impact of the project. The total financial investment in the nine case studies to date has been $725,000. The financial return on investment has included a $1.5 million actuarial release from the Noise Induced Hearing Loss project (a five fold ROI), a $1.8 million cost avoided from the Implantable Pain Therapies evidence review, and potential costs avoided of up to $6.8 million for the Body Weight Supported Treadmill Training evidence review. While there is some inherent uncertainty in the financial estimates, the financial ROI calculated for three of the case studies demonstrates a return on investment that is many times the investment in the research. The financial impact of three TAC focussed projects is yet to be determined. These estimates will be known in coming months as the impact of the research on claims costs at TAC become clearer. In addition, the non-financial value of the projects has been substantial, and includes positive changes in the culture of evidence-based decision making at both WorkSafe and the TAC, increasing the understanding within the TAC of the impact of claims management processes on TAC clients, improvements in research capability amongst leading injury research group at Monash, enabling the implementation of Recovery model at the TAC, establishing a basis for important health policy positions at WorkSafe and the TAC, and enabling enhanced interactions between rehabilitation coordinators and TAC clients. The research has also enabled the continuance of an important workplace safety debate in the community (i.e., quad bikes safety devices). Analysis of data captured in ISCRRs project management system identified evidence of use of ISCRRs research in 27 of 30 individual projects that had produced output by end August 2012. In 17 of these projects qualitative information demonstrating the impact of the projects on WorkSafe, the TAC or their stakeholders has been collected. Content analysis identified five major types of impact, being financial impacts (such as reductions in actuarial liabilities and costs savings), claims processing and claims decision making impacts, impacts on employers and the broader community. A single project also demonstrated an impact on client health and wellbeing. Content analyses identified eight factors that influence the adoption of ISCRRs research including engagement and interactions, alignment with partner strategic priorities, timeliness

of 32

of research, ease of implementation, partner organisations structure and processes, risk and issue prioritisation, the presence (or absence) of an internal champion for the research, and the credibility of the research method and source. This analysis provide substantial value to ISCRR, as it informs ongoing development of our project development and translation system to ensure that research findings are more likely to be adopted. The analysis also identified some features of projects that are less likely to be used by WorkSafe, the TAC or their stakeholders. The findings are also highly relevant for the partner organisations activities as they demonstrate that effective use or translation of research findings requires active engagement from the research end-user. The analysis has been undertaken on a small proportion of completed projects from the ISCRR research portfolio. As ISCRRs research program matures the financial and non-financial benefits to WorkSafe and the TAC may grow.

Next Steps Apart from achieving its primary purpose of measuring ISCRRs impact, including return on investment, as required by its key performance indicators, the findings of this ROI project will be applied in a number of ways. First the final report will be provided to the ISCRR review team. It is expected that it will receive an informed critical appraisal in this process. Secondly, the ROI methodology developed will be incorporated into ISCRRs ongoing project management system to continue to improve the way ISCRR routinely captures information on adoption and impact. The results of the analysis will be presented to industry partners to enable a discussion about the ways in which those organisations can facilitate translation of research. It is anticipated that this may lead to process changes within the partner organisations. Finally, the academic value of this exercise will be exploited via research publications and presentations. Having reviewed the published research literature in this field, the authors are of the opinion that ISCRRs impact assessment methodology is a unique example of how to assess return on research investment. Dissemination of the methodology via conventional research avenues will help to raise the profile of ISCRR in the research translation field.

of 32

REFERENCES Clay MA, Donovan C, Butler L, Oldenburg BF. The returns from cardiovascular research: the impact of the National Heart Foundation of Australias investment. Med J Aust 2006; 185: 209-212.

Collie A. Gains in neurotrauma research activity and output associated with a Victorian state government funding program. Medical Journal of Australia 2010; 712-714. Dobrow, M.J., Goel, V., Upshur R.E.G. Evidence Based Health Policy: Context and Utilisation. Social Science and Medicine 2004; 58: 207-217. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O. Diffusions of Innovations in Service Organisations: Systematic Review and Recommendations. Milbank Quarterly 2004; 82(4): 581-629.

Insight Economics. Economic impact study of the CRC programme. Canberra: Insight Economics, 2006. http://www.universitiesaustralia.edu.au/documents/news/media_releases/2006/CRC_Economic_Impact_Study-Oct06.pdf (accessed Apr 2010)

Kingwell BA, Anderson GP, Duckett SJ, et al. Evaluation of NHMRC funded research completed in 1992, 1997 and 2003: gains in knowledge, health and wealth. Med J Aust 2006; 184: 282-286.

Lavis, J. How Can We Support the Use of Systematic Reviews in Policymaking? Plos Medicine 2009; 6(11).

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. Journal of Health Services Research and Policy 2005: 10(1); 21-34.

Mitton, C., Adair, C.E., McKenzie, E., Patten, S.B., Waye Perry, B. Knowledge Transfer and Exchange: Review and Synthesis of the Literature. Milbank Quarterly 2007; 85(4) 729-768.

Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One 2011; 6(7).

Schapper CC, Dwyer T, Tregear GW, Aitken M, Clay MA. Research performance evaluation: the experience of an independent medical research institute. Aust Health Rev. 2012 May;36(2):218-23. Scott SD, Albrecht L, O'Leary K, Ball GD, Hartling L, Hofmeyer A, Jones CA, Klassen TP, Kovacs Burns K, Newton AS, Thompson D, and Dryden DM. Systematic review of knowledge translation strategies in the allied health professions. Implementation Science. 2012; 7(1):70.
http://www.ncbi.nlm.nih.gov/pubmed?term=Pawson%20R%5BAuthor%5D&cauthor=true&cauthor_uid=16053581http://www.ncbi.nlm.nih.gov/pubmed?term=Greenhalgh%20T%5BAuthor%5D&cauthor=true&cauthor_uid=16053581http://www.ncbi.nlm.nih.gov/pubmed?term=Harvey%20G%5BAuthor%5D&cauthor=true&cauthor_uid=16053581http://www.ncbi.nlm.nih.gov/pubmed?term=Walshe%20K%5BAuthor%5D&cauthor=true&cauthor_uid=16053581http://www.ncbi.nlm.nih.gov/pubmed?term=Orton%20L%5BAuthor%5D&cauthor=true&cauthor_uid=21818262http://www.ncbi.nlm.nih.gov/pubmed?term=Lloyd-Williams%20F%5BAuthor%5D&cauthor=true&cauthor_uid=21818262http://www.ncbi.nlm.nih.gov/pubmed?term=Taylor-Robinson%20D%5BAuthor%5D&cauthor=true&cauthor_uid=21818262http://www.ncbi.nlm.nih.gov/pubmed?term=O'Flaherty%20M%5BAuthor%5D&cauthor=true&cauthor_uid=21818262http://www.ncbi.nlm.nih.gov/pubmed?term=Capewell%20S%5BAuthor%5D&cauthor=true&cauthor_uid=21818262http://www.ncbi.nlm.nih.gov/pubmed/22624645http://www.ncbi.nlm.nih.gov/pubmed/22624645

of 32

Wells R, Whitworth JA. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy 2007; 4: 14-17. http:/ / www.ncbi.nlm.nih.gov/pmc/articles/PMC1929109/ (accessed Apr 2010). Zardo P, Collie A. Use of evidence in public health policy: An injury compensation policy content analysis. Accident Analysis and Prevention (submitted).

of 32

LIST OF APPENDICES

a. Methods document

b. Impact Assessment Framework document

c. Noise Induced Hearing Loss case study

d. Client Perceptions of Recovery case study

e. Quad Bike Safety Devices case study

f. Implantable Pain Therapies case study

g. Body Weight Supported Treadmill Training case study

h. Musculoskeletal Disorders Toolkit case study

i. Pelvic Ring Fractures case study

j. Client Conversational Tool case study

k. Return to Work Predictive Model case study

x. Calculation of overhead costs

Documents

Iscrr Impact Assessment Report