Developing appropriate evaluation pathways for complex invasive

therapies: A surgeon’s perspective

Peter McCulloch

Professor of Surgical Science & Practice

Oxford University

Chair, IDEAL Collaboration

What are medical devices?

• Term covers a huge range of artefacts from dressings and condoms to diagnostic kits to operating robots

• Focus of regulation rightly on more invasive technology with greater potential for harm

• Focus of this talk on FDA Class 2b and 3 therapeutic devices

All Medical Devices

More invasive devices

FDA Class 2b and 3 therapeutic devices

Who evaluates these devices and why?

1. Government regulatory bodies tasked with evaluating new therapies (eg FDA, MHRA) require and supervise evaluation mainly for SAFETY and TECHNICAL EFFECTIVENESS (but also for clinical effectiveness)

2. Purchasers of healthcare for large populations (e.g. CDC, Blue Cross, NHS Commissioners) evaluate mainly for RELATIVE EFFECTIVENESS and COST-EFFECTIVENESS (but also for safety) – resulting in the “HTA industry”

3. Where devices represent a new treatment concept, clinical & academic researchers evaluate a range of outcomes, of which EFFICACY and RELATIVE EFFECTIVENESS are usually most important (but also safety and cost-effectiveness)

Why is evaluation of these devices a problem? Ciani O et al, International Journal of Technology Assessment in Health Care, 31:3 (2015), 1-12. Tarricone et al Expert Rev. Pharmacoecon. Outcomes Res. Early online, 1–12 (2014)

• Relatively poor quality of evidence presented by developers

• Repeated modifications of devices

• Problems in generalizing the evidence obtained in a specific setting – very variable results

• The “learning curve” effect of outcomes improving with experience

• Difficulties in defining the correct scope of the evaluation • Technical performance • Reliability • Safety • Ease of use • Market demand, • Likely cost-effectiveness, profitability • Rare and late problems

Why is it so?

• Regulatory answer – because there is no mandatory requirement for specific evidence types or standards

• Environmental Answer – small companies with limited resources and knowledge

• Ontological Answer – the nature of devices makes evaluation more difficult

R

E

O

Developing Evidence for Surgical Therapy

PHARMA PARADIGM

• Theory

• Lab demonstration

• (animal studies)

• First-in-man study

• Toxicity study (Phase I)

• Efficacy Study (Phase II)

• RCT (Phase III)

• Post-Marketing Surveillance (Phase IV)

SURGICAL PARADIGM

• Idea

• (animal/bench trial)

• First-in-man (CS)

• Outcome experience (CS)

• Confirmation/dispute from others (more CS)

• RCT??

2016: Pharma versus Surgery

ANALYSIS: BARRIERS TO SURGICAL RCTs

REAL

• Need for refinement of technique

• Need for definition of technique

• Delivery Variation (learning curves, quality control)

• “equipoise” Clinician • (surgical personality?)

• “equipoise” Patient • Tough choices

• No way back

NOT SO REAL

• Rare conditions

• Heterogeneous patients

• Blinding

• Time and Resources

With enough information, we could surmount all of these

Shared problems of evaluation: Devices & Surgery

• Weak or non-existent legal and regulatory evidence requirements

• Small underfunded groups with weak scientific skills & massive conflict of interest evaluating themselves

• Strong preferences based on heuristic biases*

• Need to modify initial design*

• Need for learning curves*

• Variability in delivery*

Q. “What should an evaluation process for X look like?” A. “What is X?”

“The first step in analysis is description”

IDEAL Framework* A 5 stage description of the journey of surgical innovation

• Stage 1 - IDEA

• Stage 2a - DEVELOPMENT

• Stage 2b - EXPLORATION

• Stage 3 - ASSESSMENT

• Stage 4 – LONG TERM MONITORING Barkun JS, Aronson JK, Feldman LS, Maddern GJ, Strasberg SM, McCulloch P; Balliol Collaboration: Evaluation and stages of surgical innovations. Lancet 2009 Sep 26;374(9695):1089-96. Ergina PL, Cook JA, Blazeby JM, Boutron I, Clavien PA, McCulloch P, Balliol Collaboration, Challenges in evaluating surgical innovation. Lancet. 2009 Sep 26;374(9695):1097-104. McCulloch, P., Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, Nicholl J., No surgical innovation without evaluation: the IDEAL recommendations. Lancet, 2009. 374(9695): p. 1105-12.

Key Questions at each IDEAL Stage

Each stage is defined by one key issue:

STAGE 1: What is the new treatment concept?

STAGE 2a: Has it been optimised to a state of stability?

STAGE 2b: Can we reach agreement that the prerequistes for a randomised trial versus current standard of care have been satisfied?

STAGE 3: Is the new treatment better than current practice?

STAGE 4: Are there any surprises?

Implications of the Key Questions

• Studies at each stage should be designed to answer the key question for that stage

• IDEAL Recommendations describe study formats designed to do this

IDEAL: An integrated evaluation pathway

REGISTRATION OF 1st in MAN

PROSPECTIVE DEVELOPMENT STUDY

PROSPECTIVE EXPLORATION STUDY

RCT

REGISTRY

The IDEAL Recommendations • Idea (First in Man) 1.

• Development 2a.

• Exploration 2b.

• Assessment 3

• Long Term Study 4

• Complete technical description • Explanation of patient selection • Registration of report

• Prospective Cohort Study (PDS) • Transparent Consecutive Reporting of Cases • Explanation of Changes in Technique, Indication

• Prospective collaborative cohort study (PES) • Evaluation of learning curves • Definition of QC parameters • Estimation of power calculations • Early joint analysis leading to RCT • Feasibility/Pilot RCT • Definitive RCT • Removal of investigator bias from recruitment

• Registry to detect late/rare events • Monitoring of indication and performance

creep

Legal Firewall for Stage 1 database

• UK Access to Medical Treatments (Innovation) Act 2016

• Requires innovative treatments to be recorded in database

• Offers opportunity to protect innovators from legal challenge on the basis of what is recorded in the database

Stage 2a: Development

• Prospective Development Studies:

- Detailed description of selection criteria

- Detailed technical description

- Prospective account of ALL cases consecutively

- Clear STANDARDISED definitions of outcomes reported

- Description of ALL modifications and when made in series

- Registration of PROTOCOL before study starts

- Use of STATISTICAL PROCESS CONTROL methods to evaluate progress

PDS Example: Development of Robotic Oesophagectomy

Why do Development studies?

• Techniques in DEVELOPMENT stage are not yet stable

• Reporting changes and their reasons allows others to learn faster and not repeat mistakes

• Ethically superior to current practice

• Should increase speed of development process

Stage 2b: Exploration

Prospective Exploration Study (collaborative non-randomised

prospective study)

• To evaluate technique prospectively and co-operatively

• To agree definition of the procedure, quality standards & patient selection

• To accumulate data for power calculations

• To evaluate learning curves

• To evaluate preferences and values amongst patients and clinicians

• To achieve consensus on future trial question and comparator group

• To develop a multi-centre randomised trial OR demonstrate this is not required

PES example: HIFU for fibroids • Previous papers track progress of

technology through stages (0), 1 and 2a

• Large prospective cohort comparing surgery with HIFU (patient choice)

• Defined parameters of treatment

• Measured learning curves

• Tight quality control

• Used results to plan RCT protocol

20 centre study: checking learning curves

• LC-CUSUM technique shows when improvement stops

• All operators showed satisfactory learning BEFORE trial entry

• Learning curve checked during trial as well

HIFU Exploration study results

Significant Complications Patients treated

0

5

10

15

% MajorComplicatio…

Hospital Stay (days) Return to Work (days)

HIFU 3.63 4.07

Myomectomy 8.96 24.01

Hysterectomy 10.53 29.49

Quality of Life (6 & 12 months) HIFU clearly superior but margin small

Why do Exploration (2b) studies?

• Organising surgical RCTs is difficult: it requires TRUST and UNDERSTANDING between surgeons

• 2b studies improve trust and understanding by improving joint ownership and belief in data

• 2b studies allow questions which worry surgeons to be answered, e.g. • Which variations of the procedure are acceptable? • Are some colleagues still learning? • Which patients are suitable? What is the comparator? • How many patients will we need? • What outcome questions are important to patients – and surgeons?

• In this way 2b studies improve FEASIBILITY of RCTs

Is IDEAL applicable to device evaluation? A Delphi Process to develop IDEAL-D *

• Need Stage 0 with minimum declared dataset • Recommendations ??

• Need flexible approach to mixes of Development (2a) and Exploration (2b) stages • 2a often redundant • 2b only partly relevant

• Controversy over RCTs for “me too” and successor devices

• Need Registries from an early stage, developing and changing with needs • Potential of UDI and Big Data • TWICS approach

IDEAL-D: a rational framework for evaluating and regulating the use of medical devices. Sedrakyan A, Campbell B, Merino JG, Kuntz R, Hirst A, McCulloch P. BMJ. 2016 Jun 9;353:i2372. doi: 10.1136/bmj.i2372. *

Outline of IDEAL-D

• Greater emphasis on pre-clinical testing => Stage 0, Less Stage 2a

• UDI and TPLC concepts => Registries from Stage 1

• Number of relevant issues in Stage 2b reduced

• Potential fusion of Stages 2a and 2b

iBRA Network

• Focus on reconstruction after breast cancer treatment (including implants and devices)

• Plans to collaborate on large prospective cohort studies (2b studies)

• Has adopted the IDEAL recommendations as standard operating procedure

• Aims to use these to define where RCTs are feasible and useful

• One study completed, two in pipeline

Will IDEAL help? What is the potential value?

• Having accepted, logical, standardised study formats for evidence provision would greatly enhance transparency and reduce redundancy

• The provision of standards for early clinical evidence (Development and Exploration studies) would facilitate appropriate timely discussions with purchasers and regulators

• The study formats involved are cheaper and faster than RCTs and would represent a much lower barrier to (provisional) market entry

• Much current evaluation activity by purchasers and regulators, looking at reviews of deeply flawed evidence, could be abandoned.

What did we just say?

• Establish standard study formats

• Have an integrated pathway

• Link graded access to progress

• Rely more on this prospective evidence than on reviews

Radical step or logical progression?

• UK - Commissioning through Evaluation • US – Coverage with Evidence

Development • Germany – new §137 Social Code

ALL LINK COVERAGE TO SPECIFIC EVIDENCE DEVELOPMENT UK AND GERMAN MODELS HAVE IDEAL D AND E-LIKE STAGES ALREADY

• WHY NOT SPECIFY AND REQUIRE

FORMATS? • WHY NOT LINK COVERAGE TO EVIDENCE

PATHWAY PROGRESS?

Current IDEAL Adoption in Rx evaluation

• NHS England specialist commissioning – Commissioning with evidence development programme – pilot studies under discussion

• Research Collaboratives e.g.iBRA

• NHS England NIHR research HTA Programme calls for IDEAL 2b (Exploration) studies for: • Incontinence surgery devices

• Fenestrated aneurysm stenting

• Pilonidal sinus treatments

• Surgery for Crohns’ disease

IDEAL as the core of a broader evaluation framework: • EpiCOR programme in Canada

(Martin J et al) using IDEAL to make purchasing decisions

• Radboud University Medical Centre, Nijmigen, Netherlands (Rovers et al) – using IDEAL to advise on Health Technology Assessment reports.

• EXCITE International – an new international body offering end-end expert 3rd party evaluation for therapeutic devices

• All three seek to add “pre-IDEAL” evaluations to provide a spectrum of evidence to meet the demands of both regulators and purchasers • User forums – qualitative work on

value • HE modelling to predict cost-

effectiveness • Ergonomic evaluation to optimise

usability

Integrating evidence for regulation & purchasing

• Goals and evidence types presented likely to differ – but overlap +++

• All new entrants need to confront both issues – eliminating redundancy would reduce costs and delays, simplify process

• Both regulators and payers face the same types of weakness in the evidence they are given

• Common language and understanding of the evidence development process would be hugely helpful for inter-agency work

Using IDEAL in Regulation and Purchasing

• Where is this procedure on the evaluation pathway?

• What questions need to be answered NOW?

• What studies are appropriate?

• What will happen if we don’t insist on proper evaluation?

• Regulator or payer takes responsibility for decision on RCT

Potential Models for using IDEAL: “Commissioning” & Regulation

COMMISSIONING

• Funding only in context of appropriate (IDEAL 2a or 2b) study

• Mandatory data review

• Decision: Clearance or Coverage within RCT only – or decision one is not required.

• Discussion on funding & curation of Registry activities

REGULATION

• Develop and require common minimum Stage 0 dataset

• Require IDEAL stage 2 studies for Class 2b and 3 devices

• Monitor performance of notified bodies in delivering these

• Insist on greater transparency with public on realities of safety assessement.

Contrast with Claxton framework Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development K Claxton, Palmer, L Longworth, L Bojke,

• Claxton assumes current NICE decision framework will persist and defines criteria for decisions to impose restrictions on use related to further study

• Also assumes that current binary approach to market approval will persist, hampering good data collection post-market.

1 Is it cost-effective?

2 Are there significant irrecoverable costs?

3 Does more research seem worthwhile?

4 Is the research possible with approval?

5 Will other sources of uncertainty resolve over time?

6 Are the benefits of research greater than the costs?

7 Are the benefits of approval greater than the costs?

• IDEAL assumes the unification of regulatory and purchasing evidence requirements

• Also assumes a system in which marketing is strictly limited to cases entered into a study of the appropriate type for the IDEAL stage i.e graduated approval dependent on evidence collection.

Thorny Issues for an IDEAL-based system • First-in-man Registry: legal firewall needed

• Commissioning research: who pays?

• Oversight & curation of Registries: responsibilities • Defining dataset with clinical community • Ensuring quality control of data • Supervising data protection • Ensuring registry fulfils intended functions

• “Me too” devices • Should we follow Pharma model? • How to deal with “minor” changes – beware the 510K route

• Boats we missed • Accepted treatments with weak evidence: Time-limited IDEAL 2b study?

• Rare & expensive treatments: research only? • Proton beam therapy e.g. • How prescriptive should we be? • Will we create “orphan” treatments?

The IDEAL proposition for devices

• Establish a set of standards for clinical studies related to the stage of development of the technology

• Establish an integrated pathway for devices to pass from one stage to the next through evidence development

• Link market access to stage of progression – in principle restrict access to patients in ongoing studies

• Use IDEAL-D Stage 2 evidence for critical decision: RCT required or not?

• Establish registries from an early stage

• If you’re worried it might slow innovation – pilot it properly.