View
61
Download
0
Embed Size (px)
Citation preview
Evaluation of new digital health innovations
Andy Scally, Bradford Evaluation and Trials Unit (BETU)
DHEZ festival, June 2016
3 May 2023 Evaluation of new digital health innovations1
Triple Aim
3 May 2023 Research and Evaluation in Healthcare2
Population Health
Patient experience Reduce cost
Adapted from Institute for Healthcare Improvement (USA)
An exercise in balance.
Stages of development• For pharmaceutical products
– Discovery – many compounds investigated– Development – pharmacokinetics, mode of delivery, manufacture, testing (in vitro, in vivo)– Phase I Trials – safety in humans– Phase II Trials – dose finding and safety– Phase III Trials – efficacy/effectiveness and safety– Phase IV Trials – effectiveness and safety post-marketing
• Average time from preclinical testing to approval is approximately 12 years and less than 10% of products make it to the end of this journey• For other (e.g. digital) interventions/products – similar number of phases, differing in precise characteristics
– but SME’s can’t afford this failure rate and can’t wait 12 years!
3 May 2023 Research and Evaluation in Healthcare3
What input can BETU offer?• You (small, medium and large scale enterprises) have the
ideas regarding innovation, application and the technical expertise for initial development
• BETU can offer advice and input to the development of all stages of evaluation, from initial evaluation through product refinement to large scale testing– Research design– Qualitative and quantitative evaluation– Process evaluation– Seeking ethical and governance approval (can link with
expertise on MHRA approval)– Data analysis– Writing for publication
3 May 2023 Research and Evaluation in Healthcare4
Research v. Evaluation• Whether we call it research or evaluation, the aim is to
generate evidence that informs us of the effectiveness/usefulness of an intervention, which could be a product, a process or a combination of both.
• Research – a rigorous process for assessing the effectiveness of an intervention before ‘rolling it out’.
• Evaluation - a rigorous (one hopes!) process for assessing the effectiveness of an intervention once it’s in place.
• The methods involved are conceptually broadly the same and some types of research, especially those involving complex interventions, require some form of ‘evaluation’, in its generic sense, of the internal components of the intervention – ‘process’ evaluation.3 May 2023 Evaluation of new digital health innovations5
Challenges in the digital world• The regulatory and approval systems for health care
innovations are largely based upon the pharmaceutical model– This can present many challenges for digital innovations
• We need to identify and develop time- and cost-efficient mechanisms for evaluating digital innovations before they are likely to be widely embedded in practice– Digital health technology companies lack the resources
of big pharma– Research funding is very competitive and can take
much effort, and a long time, to obtain– Technology rapidly evolves over time
• We can’t always rationally evaluate the ‘digital’ component independently – it is often embedded within a larger ‘process’3 May 2023 Evaluation of new digital health innovations6
Challenges in the digital world• Increasing shift in the nature of healthcare towards more
patient autonomy and pro-active engagement with their health management
• Exponential growth in ‘health apps’, with very little evidence base, that the public can try for themselves
• Health practitioners are increasingly ‘trying them out’ in their practice– By the time they have been ‘proven’, using the current
paradigm, they may have become obsolete• Can we ‘try them out’ and evaluate as we do so?
– Yes, in many circumstances, but the way in which we do this can significantly impact our ability to evaluate them rigorously
3 May 2023 Evaluation of new digital health innovations7
Addressing the challenges• Early user testing to establish a prima facie case for early
adoption and further development• Think clearly about the nature of the ‘benefit’ of the
product/innovation and how this can be measured• Be aware of the principles of good research design and threats
to validity• Design the evaluation at the time of implementation. If it is an
afterthought, it may be too late!– ‘To consult the statistician after an experiment is finished is
often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.’ - Ronald Fisher
• Make smarter use of routinely collected electronic data– Huge potential savings in time and cost, but there are data
governance, and research governance, challenges to address.
3 May 2023 Evaluation of new digital health innovations8
3 May 2023 Evaluation of new digital health innovations9
Thank you