1

Generating appropriate and reliable evidence for the value assessment of medical devices: An ISPOR 1 Medical Devices and Diagnostics Special Interest Group Report, Part 2 2

Introduction 3

Economic evaluations have played a pivotal role in assessing and communicating the value of pharmaceuticals 4

(Drummond 1997, Fry 2003). The incorporation of good-quality health economic evidence has led to value-based 5

assessment and funding of several pharmaceuticals - a process considered to influence value-based pricing (VBP) 6

(McGuire 2008) as well as inform coverage policies and reimbursement decisions (Sullivan 2009). The increasing 7

reliance on economic evidence in health care decision making in many developed countries has enabled rational 8

decisions about pharmaceutical product adoption and patient access (Massetti 2015). 9

However, there is a shortage of economic evaluations conducted and reported for medical devices (MDs). With the 10

need to determine the value for money spent on MDs, understanding the trade-offs in resource costs and the benefit 11

of incorporating a new medical device (MD) into an existing health service delivery system has led to a growing 12

interest in evidence-based health technology assessment (HTA) (Huot 2012, Schreyögg 2009, Tarricone 2011). 13

Emerging new technologies have the potential to positively impact patient care; but also hold implications for 14

population health, healthcare systems, organizations and professionals as well as posing challenges for decision-15

makers in assessing their value for money. One key question relates to whether existing HTA tools and approaches 16

fully reflect the value of new technologies, or is there is a need for improvement in the clinical and economic 17

evaluation of MDs? Physicians, hospitals, industries, insurers, regulators and patients are increasingly requiring that 18

clinical and economic information be linked to pricing, such that the benefits of the health technology justify its 19

payments (Eisenberg 1989, Buxton 2006). This increased interest in value-based assessment of MDs for coverage, 20

reimbursement and pricing needs to formally rely on economic evaluation methods in order to balance access to 21

effective new medical technologies and resource allocation decisions. Yet, the practical use of economic 22

evaluations in decision making for the device industry presents several challenges, both in the classification of a 23

product as a medical device and in the assessment of its cost-effectiveness (Robinson 2008, Kirisits 2013, Ventola 24

2008). To overcome some of these challenges, a clear framework for assessing the value of MDs should describe 25

the appropriate design, methods, analysis and reporting of economic evaluations. This article will make the case that 26

2

the quantity and quality of evidence is important in building a framework aimed at developing competence in the 27

HTA of MDs. 28

In part 1 of this series, we began by addressing the need for defining value across the full range of MDs, while 29

taking into account the influence of various financing systems and different stakeholders ‘perception of value. The 30

following article addresses the key differences between MDs and pharmaceuticals which impact the choice of study 31

design, addressing the issue of hierarchy of evidence; subsequently lending towards establishing best practices for 32

generating appropriate and reliable evidence to assist stakeholders involved in decision-making and resource 33

allocation. 34

Key differences between pharmaceuticals and medical devices impacting on research methods 35

Like pharmaceuticals, medical devices are intended for use in the diagnosis of disease or other conditions, or in the 36

cure, mitigation, treatment, or prevention of disease ; however, they are very peculiar products in terms of their 37

classification, size, durability, complexity (from simple tongue depressors to programmable pacemakers and x ray 38

machines), packaging and user involvement. In addition to some important differences between pharmaceuticals and 39

devices (Table 1), there are many factors that play a role in choosing the ‘optimal study approach’ and study design, 40

one of the most important being establishing “medical necessity” (Craig 2015). Other important considerations 41

include: phase of the device’s life cycle; type of device (i.e., whether a device is therapeutic or information 42

generating, invasive or non-invasive, or requiring user interaction/interpretation); working mechanism through 43

which a device leads to risks, benefits or less burdensome care; intended medical context(s), intended indication; 44

targeted patient population(s); intended users and prevailing care in the intended context(s) (KNAW 2014). 45

3

Table 1. Key differences between pharmaceuticals and medical devices, the need for an evaluation framework for evidence generation and device evaluation

Pharmaceuticals Devices

Target population Well defined and larger Can be small in size

Product

lifecycle(PLC)

Typically there are three different life phases: an extensive

early development period; a highly competitive period; and

a significant late post-patent period

Medical devices development is characterized by a constant flow of

incremental product improvements. Accordingly, the life cycle of a

specific type or variation of a device is often as short as 18 – 24

months

Choice of, or relevant,

comparator(s)

Generally existing standard of care, best available, usual

care or best supportive care

The standard technology which is sometimes a similar or equivalent

technology used as part of current management, but assessment

may be problematic due to differing device features

Pricing Static or near static pricing Pricing changes due to market dynamics

Outcome measures Assessment of clinical events, mortality, health-related

quality of life, PROs, and QALYs

Medical devices can be therapeutic or diagnostic ; and appropriate,

meaningful health outcomes depend on device type and utilization.

For example, measures of diagnostic accuracy includes the

sensitivity, specificity, positive predictive value (PPV) etc.; and

measures of therapeutic effect include clinical events, performance

and PROs. Important outcomes (e.g., ease of use) are usually

reported by users (e.g., health providers, patients).

4

Table 1 (continued)

Pharmaceuticals Devices

Clinical safety Safety and toxicity studies Technical reliability and feasibility studies. For instance, the CE

mark certifies that a product has met EU consumer safety, health or

environmental requirements

Efficacy and

effectiveness

Adequate evidence of effectiveness (e.g., endpoints, study

design, patient populations) to support efficacy claim

Evidence of clinical performance for regulatory purposes—medical

devices achieves its intended purpose as claimed by the

manufacturer

Generation of new

evidence

Throughout the product lifecycle Throughout the product lifecycle, although because medical devices

are “incremental Innovations” (during product life there is an

incremental sometimes step-wise innovation), evidence generation

is often limited

Time horizon A longer time horizon to capture meaningful differences in

costs and outcomes between the intervention and

comparators, typically evaluated at 1-year, 3-year, 5-year

and over the course of the disease

Debate on when to assess a product innovation—early in the

product lifecycle? Medical devices are considered fast-changing

technologies and as a result, the timing of assessments are usually

short

Measuring long term

outcomes

Product lifecycle allows for identification and measuring of

relevant short-term and long-term outcome measures

The short product lifecycle makes it difficult for the identification

and measuring of relevant short-term and long-term outcome

measures

Study design Double-blind randomized controlled trials Double-blind controlled randomized trials are generally preferable

but may not be feasible for many medical devices. Patients and

providers cannot be blinded, and due to differing device

characteristics randomization without fully informing the patient

may not be ethical. More use of observational study designs.

5

Table 1 (continued)

Pharmaceuticals Devices

Learning Curve Depends on the rate of adoption and physician preference

for treatment. The learning curve is generally short.

Efficacy/effectiveness outcomes are less user-dependent and

adverse events may increase with use.

Depends to a large degree on the user’s experience with similar

devices and procedure in question, the "Device-Operator

Interaction"; HCPs may require training and show a longer learning

curve. Efficacy/effectiveness outcomes are hence user-dependent

and adverse events decrease with use.

Organizational aspects Low organizational impact. Formulary committees are

responsible for managing the formulary system in health

care organizations which establish policies regarding on

what therapy is medically appropriate and cost-effective,

amongst other things

Introducing a new device may require not only training

requirements, but also possible changes in organizational policies.

Such organizational adjustments may introduce additional costs,

some of which may be one-time (e.g., physical plant renovation),

and others which may be ongoing (e.g., training). Formulary

committee review for medical devices might not be as

straightforward, and in some instances, may be not necessary or

even non-existent.

Regulatory landscape Regulatory approval process is typically through Phase 1,

Phase II, Phase III, to Phase IV

The responsible regulatory body and the approval process may

differ from that for pharmaceuticals depending on location. For

instance, in the US, medical devices are approved through the

Premarket Approval (PMA) application process vs. Premarket

Notification 510(k) regulatory process

Perspective The societal perspective is recommended for primary

analysis, with optional perspectives (e.g., payer)

The payer and/or health service/provider perspective may be

preferred for the primary analysis vs. the societal perspective,

depending on country/region/local setting

6

Table 1 (continued)

Pharmaceuticals Devices

Evaluation method

Cost effectiveness analysis (and CUA if it is clinically

appropriate and possible) and Budget Impact Models (BIM)

for value-based decision-making

Choice of medical device is usually based on safety, physician

preference, financing (e.g., either costs or savings)

Modelling When direct primary or secondary empirical evaluation of

effectiveness is not available or is limited

Modelling from intermediate outcomes or post-marketing

observational data

Analysis of cost data All resources used that are relevant to the analysis. All costs

should be valued at opportunity costs

All resources used that are relevant to the analysis. The facility

computer system may lack the ability to track product specific

costs

Sources of Data Generally, at least two adequate and well-controlled

confirmatory studies; however, effectiveness of a new use

may be extrapolated entirely from existing efficacy studies

(e.g., pediatric uses), from a single adequate and well-

controlled study supported by information from other

related adequate and well-controlled studies (e.g., different

doses, regimens, or dosage forms) and a single multicenter

study without supporting information from other adequate

and well-controlled studies

Data used from a combination of sources: devices can be evaluated

using clinical (e.g., randomized clinical trials, partially controlled

studies, well-documented case series conducted by qualified

experts, etc.) and non-clinical (e.g., performance testing for product

safety and reliability, human factors and usability engineering

testing, computer simulations, etc.). A single confirmatory study

may be sufficient for regulatory approval.

Evidence base Evidence available is more extensive and higher quality -

RCT is the gold standard

Evidence available is typically less extensive with increased use of

observational studies, the best available evidence may come from

controlled cohort or before-after studies

7

Generating appropriate evidence to support device performance characteristics and coverage 46

Transforming data into evidence gathering throughout a device’s life-cycle requires a clear strategy that supports the 47

evidence generation process. The starting point for coordinating evidence generation activities should be early—48

preferably within the product design phase. A robust evidence-generation strategy will allow data integration from 49

heterogeneous sources (e.g., patient registries, RCT, etc.) and promote the development of better evidence that 50

aligns with reimbursement policy frameworks. In the US, the Centers for Medicare & Medicaid Services (CMS) 51

provides an evidence-based framework for the Medicare coverage of an item or service in the context of a clinical 52

study through its coverage with evidence development (CED) policy. Although the vast majority of Medicare 53

coverage decisions are provided through local coverage determinations (LCDS), the CED represents a paradigm 54

shift and opportunity for device manufactures to collect additional data in the post-market setting. The impact of 55

CED and other pay-for-performance and risk-based reimbursement models (e.g. risk sharing arrangements) that are 56

connected to data collection or performance assessment will likely link reimbursement to the value delivered by 57

showing cost savings and quality improvements. However, as new reimbursement models continue to emerge, 58

device manufactures may face resource challenges associated with the costs of conducting such studies for evidence 59

generation and building a data and research infrastructure (Daniel 2013, Garrison 2013, Walker 2012, Neumann 60

2011). For example, the investment cost in conducting additional research in order to satisfy both statutory 61

requirements - FDA requirements for approval of drugs and devices based on safety and efficacy data and CMS 62

requirements for coverage determinations based on items and services that are considered reasonable and necessary. 63

With the new payment models, reimbursement will not only reflect device performance characteristics on safety (i.e. 64

risk associated with use) and effectiveness (i.e. does it work); but also encompass other quality metrics or measures 65

on health outcomes that determine its usability, user satisfaction, relative effectiveness and cost-effectiveness. It will 66

also reflect on how effectively a device delivers on non-health outcomes that focus on cost such as the financial 67

consequence of adoption (e.g., budget impact analysis) and productivity (e.g., time-and-motion analysis). 68

In assessing the value of MDs, the question will emerge as to what constitutes appropriate evidence and to what 69

extent evidence of effectiveness – primarily rooted in pharmaceutical efficacy studies – can help. Evidence in a 70

conventional sense generally refers to proof of efficacy in a controlled setting, with randomized controlled trials 71

(RCTs) often considered as the highest level of evidence (Oxford Centre for Evidence-Based Medicine 2011, 72

8

NHMRC 2009) even though they may not always be feasible or practical for medical devices. In the next section, 73

we explain why the traditional notion of evidence and existing evidence generation methodologies are not always 74

appropriate for assessing the value of many MDs. 75

Capturing device effects as a demonstration of value 76

Most MDs used in in-patient settings, whether therapeutic or diagnostic, are incorporated into a medical procedure 77

or treatment. This principle is operationalized in many health care systems in an integral structure of hospital 78

financing based on DRG (like) groups. DRGs finance a treatment pathway as a whole, irrespective of the devices 79

being used. This financing system has implications for assessing the value of MDs. A MD is of value when its use 80

‘improves’ the performance of the existing procedure/treatment pathway. Here improvement means providing the 81

end-user with a better health outcome or the same outcomes at a reduced cost, or both. For instance, the value of a 82

new surgical device for resecting a tumor depends on the device’s ability to provide a better surgical outcome (e.g. 83

reduced tumor burdens) or to reduce the total cost of that surgical care episode. Ideally for MDs, solutions for 84

realizing better outcomes or reducing costs as a result of device use can be regarded as ‘evidence of value’ and can 85

be demonstrated in the product’s business model or implementation plan. Nevertheless, this is currently a fairly 86

uncommon practice. 87

Classical and traditional assessment methods, notably RCTs, are primarily oriented towards testing a mechanism of 88

action rather than exploring benefit/risk principles (KNAW 2014). For many MDs, however, the technical 89

performance of the device is just the bottom-level of effect. Effect largely depends on how well, how often, and for 90

whom the device is used. Therefore, efficacy, even if demonstrated across a range of conditions does not necessarily 91

reliably predict an improvement in patient and/or population health outcomes and whether it is worth the costs 92

involved at a given level of benefit. Table 2 provides a list of questions to help identify the appropriate methods for 93

MD value assessment. 94

9

Table 2. Relevant questions to address in order to demonstrate the value of a medical devices for pricing and

reimbursement decisions

Subset of value assessment Questions for generating relevant evidence

Identification of target

indications

Which patient population could benefit the most?

Added value (clinical) How does the new device improve the performance/ outcome of the whole

treatment pathway?

How does the new device fit into the whole treatment pathway?

Does it replace another treatment? Which part of the treatment pathway will

become obsolete?

Added value (economic) How does the new device affect the costs of the whole treatment pathway?

What are the average costs of the treatment pathway including the new

device?

Prospective risk inventory What are the potential risks (safety, financial, legal)?

Time frame of the

assessment

What is the reasonable time frame for outcome improvement and/or cost

reduction given the product life cycle and the state of medical practice?

Protocol for evidence

generation and data

collection

Which research design and instruments are in place for generating relevant

evidence?

Is conducting RCT necessary or feasible? If not, why?

What are relevant outcome measures?

What is the relevant comparator?

(Pre)requirements for

delivering value

Under which circumstances could the new device deliver value? How feasible

and realistic are the preconditions on the return on investment, infrastructural

adjustments, operator training, logistics, liability, etc.

Price assessment To what extent is the price elastic to volume and scale?

Observational comparative effectiveness research (CER) has been employed to demonstrate the value of MDs. 95

These methods offer a tailored way of assessing the effectiveness of MDs in clinically realistic settings (Fischer 96

2012). CER results are, however, not helpful in identifying the appropriate target population, e.g. which subgroup of 97

patients benefit the most. Besides, there remains a methodological challenge in how to compare benefits from 98

alternative treatment pathways (with and without a new device) as they are setting-specific and multivariate. The 99

choice of comparator, the duration of follow-up and sample size are also challenging decisions (Jalbert 2014; Price 100

2015). Moreover, the unit of comparison is often the device itself (device selection may be an independent outcome 101

variable) rather than the performance of the treatment pathway as a whole. 102

Decisions on timing of assessment can also pose a challenge. Proof of clinical benefit is often lacking or contested 103

during the early phases of medical device use. This is unavoidable primarily because a new device must be used in 104

10

order for its effect to be proven. Secondly, clinical evidence is generally not required before regulatory approval in 105

many markets since regulatory review often focuses more on safety with differing standards on “effectiveness”. 106

Generating clinical evidence can take months to years. Once cleared by regulators, physicians, hospitals and 107

patients may have strong incentives to promptly adopt new medical devices. Related to this is the often short 108

lifecycle of devices, which confines the time span of assessment. Existing evidence could thus be of limited 109

applicability when a device is modified. Accordingly, results of CER may not be directly interpreted as evidence of 110

value to guide whether and how to use a MD (Fischer 2012; Price 2015). 111

It should also be noted that stakeholder dialogue on value in advance of market entry can help clarify what counts as 112

‘relevant’ outcome measures, ‘added benefit’ or ‘meaningful cost’. This is not yet incorporated into common 113

practice. There are, however, initiatives for a collaborative assessment of MDs at an early stage in order to provide 114

initial insights into the device’s safety and performance to inform subsequent value assessment, and to guide device 115

modification before finalising the design. Examples include early dialogue (ED) between stakeholders (Backhouse 116

2011) or early feasibility medical device clinical studies (EFS) (FDA 2013). 117

Device effects versus impact in the real-world 118

Unlike pharmaceuticals, the impact of a MD is often dependent on how it is used in real-world practice. Regulatory 119

agencies in the EU and Class II devices in the US typically require data on a device`s technical performance, such as 120

the delivery of radiation or generation of electrical signals. However, additional evidence is needed for assessing 121

value in real-world settings at a post-approval phase. Using the previous examples, in addition to device 122

performance data, evidence is needed to show that the radiation delivered leads to improved disease outcomes or the 123

electrical signals result in decreased pain. 124

Regardless of whether a control or comparator group is used, MDs should ideally be assessed in the actual settings 125

in which they are used, for example, inpatient, outpatient and/or the home environment. In addition, focusing on 126

proving the effectiveness of a device in a test setting inevitably requires excluding other elements of the treatment 127

pathway – these are deemed as confounding factors even though they are important indicators of the actual value of 128

11

the treatment (e.g., user’s experience, training, teamwork and coordination, a hospital’s volume profile, digital and 129

technical infrastructure, organizational readiness) (Abrishami 2015). From a methodological point of view, 130

controlling for these confounding factors increases confidence in the internal validity of the experiment. 131

Accordingly, the findings of a RCT setting lack real-world relevance (KNAW 2014). 132

Real-world data as part of post-market surveillance can provide a way to demonstrate actual improvement in 133

outcomes during the course of implementation (Reynolds 2014). With the collection of data on treatment costs 134

alongside clinical outcomes, one can examine actual cost profile and cost-saving potential of the new device relative 135

to alternatives. This can be done by collecting data on the resources used and outcomes gained as part of outcomes 136

registries and other observational studies. Such pragmatic approaches need to be better established and 137

mainstreamed for MD value demonstration and assessment. As investment in real-world data becomes more 138

apparent in influencing the value of a MD, we must increasingly recognize that leveraging real-world data sources 139

such as setting up ‘value registries’ will demand commitment, collaboration, know-how and incentives. 140

Building a shared framework for evidence generation 141

Economic evaluations support rational decision-making by evaluating and reporting information on the costs and 142

consequences of health technologies. However, methodologies to fully understand the long-term clinical and 143

economic value of MDs are not well-established. The assessment of safety and performance of MDs by regulatory 144

agencies is insufficient to quantify the entire product value proposition and to allow for successful market access and 145

adoption in a value-driven health care system (Pham 2014). For a value-based access and adoption process, device 146

manufacturers need to demonstrate the value of their devices to other stakeholders (e.g., physicians, payers, patients) 147

whose needs are important and valued differently (World Health Organization 2010). Despite greater attention by 148

medical device manufacturers to the value proposition in the early stages of the product life cycle, significant 149

barriers remain (Bergsland 2014), particularly in how product value is measured and assessed. In generating the 150

required evidence to demonstrate product value to stakeholders, device manufacturers should consider (Sorenson 151

2008) the data collection process, study design, relevant endpoints, real-world performance, timing of assessment 152

and pricing when selecting a framework intended to contribute to MD evaluation. Other considerations are 153

12

explained in Table 3. 154

MD value assessment and existing methodological guidelines 155

The increasing use of MDs has led some HTA bodies to develop methodology guidelines specific to MDs. For 156

instance, some of the challenges in assessing the value of MDs are acknowledged and addressed in a few guidelines 157

(HAS 2009, NICE 2011). Several medical device assessment agencies have also been established (Ciani 2015). The 158

fact that MDs are often offered to patients as part of a medical procedure or treatment is acknowledged in the UK 159

National Institute for Health and Care Excellence (NICE) guidelines (NICE 2011) and the terminology “system 160

outcomes” is introduced to refer to outcomes of a treatment episode or procedure. The same is mentioned in HAS 161

guidelines (HAS 2009). 162

HAS reimbursement recommendations for a product or service are based on two criteria, ‘actual benefit’ and ‘added 163

clinical value’. RCTs are considered as the optimal study design by HAS but they acknowledge that it might not be 164

possible for all MDs. NICE accepts contributions from experts, patient and carer organizations. 165

The inherent uncertainty surrounding the performance of MDs in the real world are suggested to be addressed with 166

further targeted research by NICE under conditional reimbursement decisions; and by additional clinical follow up 167

studies after reimbursement decision by HAS when deemed necessary. HAS also highlights the importance of 168

conducting multicentre studies. 169

13

Table 3. Differences in pharmaceuticals and medical devices from different stakeholders` perspective and proposed solutions to differences in

frameworks

Difference to pharmaceutical value

assessment

Pro to Differences Con to Differences Proposed Solution

Diverse nature of devices

(decentralised reimbursement

pathways, relatively easier regulatory

approval, KOL knowledge levels are

generally low but increasing)

Forces creative negotiation

among payer / manufacturer

Creates uncertainty and ambiguity

among stakeholders, this slows

patient access

Acknowledgement from payers that

diversity in landscape means acceptance

of diversity of methodologies and

processes

Patients have potential access to

solutions that perpetually

improve their quality of life

Creates dis-incentive for risk on

R&D/ideas (not device use) given

uncertain outcomes

Acceptance from manufacturers that

evidence must be provided to prove

value

Stakeholder usage (asymmetry in

device appreciation)

Stronger bond between

HCP/patient and manufacturer

given that there is a close

partnership for device usage

Stakeholder engagement

(marketing) is far more costly as

multiple stakeholders need to be

considered

Acceptance from all stakeholders that

certain patient access strategies simply

mean a different business model to

Pharma.

Appropriate evidence generation

(RWE is better suited for MDs

compared to RCTs)

Effectiveness is promoted as

opposed to efficacy which is in

an ideal ‘lab’ situation

Observational studies give more

opportunity for bias in study

design

More transparency on risk sharing

agreements and what it means for

stakeholders

Ethics are placed above academic

rigour

Much more costly for system to

operate RWE studies. Who pays

for engagement of study protocol?

Acceptance that the RCT is not

necessarily the gold standard – it does

not promote effectiveness which is

instrumental to devices

RCTs can be long, expensive and

cumbersome. Observational

studies can increase patient

access

Device industry is dominated by

SME’s therefore large, expensive

trials are prohibitively expensive

Manufacturers must still strive to prove

value through clinical rigour. This

should be considered a core cost in

marketing value proposition

development

HCP skill plays important role in

success

Much more HCP engagement in

the value chain, therefore more

appropriate solutions for patients

Payers must have deep knowledge

of the care pathways and how the

device can change this

Manufacturers become much more

articulate about partnering with HCP’s

and the value this brings to their

products

Patients are also engaged in the

value chain as there is a direct

interaction with the patient.

‘Beyond the product’ solutions are

neither valued nor considered

necessary as it is assumed HCP

should be able to utilise their

clinical expertise

Payers must appreciate that

pharmaceuticals are an embodied

technology. Devices require significant

investment in training and HR, and this

should be acknowledged in value

propositions and models

Table 3 (continued)

14

Difference to pharmaceutical value

assessment

Pro to Differences Con to Differences Proposed Solution

HCP skill plays important role in

success (continued)

Manufacturers must provide

‘beyond the product’ as a core

component of value proposition

HEOR models are often not

designed to reflect these

costs/benefits

Value Models don’t account for the

learning curve

From a payer`s perspective this

means processes and

methodologies are simple and

replicable

Manufacturers are trying to fit a

square peg into a round hole.

Devices just do not fit into pharma

value models

More industry sponsored ad-boards to

identify how these parameters can be

included in modelling techniques.

There are no advantages to this

from a manufacturer perspective,

as this immediately detracts from

core value proposition

Organizations like the NHS can

materially improve its efficiency

by factoring non-drug/device

spending improvements

Shorter product life cycle: Forces more incremental product

design that adds to material

patient benefit over time. Eg.

Pen Needles, stents

Less opportunity for R&D costs to

be realised

Consider education and training more as

a core component of device offering.

No education = no ‘sale’ or patient

access. Responsibility should be on

manufacturers to provide training and

educations to HCP / patients. This acts

as an ‘educational patent’ to ensure

product pushers cannot mitigate

innovation

Dis-incentivises R&D as `me too`

products quickly arrive. Little

advantage to being first mover

No exclusivity period – unique to

devices

More competition is allowed to

enter the market therefore

allowing for greater diversity for

patients

Gives less appreciation of small

innovation which don’t have the

same stepwise benefit to patients

Device companies have ‘exclusivity’

while engaging in a post-approval trial.

This incentivises ‘first to market’

manufacturers not the ‘immediate

responders’

15

Implementation of Real World Evidence 170

Real-world evidence (RWE) is an umbrella term for data regarding the effects of health interventions collected outside of conventional controlled trials. RWD is 171

collected either prospectively or retrospectively during observations of routine clinical practice. Data collected include, but are not limited to, safety and 172

effectiveness outcomes, patient reported outcomes (PRO) such as patient satisfaction and health-related quality of life, and economic outcomes. RWD can be 173

obtained from many sources including patient registries, electronic medical records, billing records and through direct interaction with end-users (Adapted from 174

Garrisson 2007 in GetReal 2015, p.30). Real-world evidence (RWE) is the evidence derived from the analysis and/or synthesis of RWD (GetReal 2015). 175

RWE to demonstrate medical device value is increasingly considered by payers, HTA agencies, regulators and policy makers, such as NICE in the UK and the 176

FDA in the US; however, the methods applied for data collection are cautiously evaluated and data availability prior to decisions on device uptake is extremely 177

limited. Critics often argue about the uncertainty and low reliability of RWD and consider it to provide low level evidence. The quality of research ultimately 178

depends on its methodology. RWD cannot be used for proving efficacy. However, with precise implementation of an a priori research protocol, RWD can 179

provide a vast amount of information on value; for example, how and what extent of device use leads to improved outcomes, given that data are collected and 180

analyzed by valid and reliable methods. Key considerations in implementing observational studies and for improving validity and reliability are listed in Table 4. 181

RWE enables the assessment of longer term user experience and costs. The impact of a device, whatever its classification, is often best realized through patient 182

and/or family interaction and this is becoming more and more important to obtain reimbursement. Disseminating study results can help various stakeholders gain 183

insight into actual device use and benefit in a broad group of patients. Patient registries provide a source for monitoring the safety profile and actual resource 184

utilization pattern of a medical device. Health care providers (HCPs) can also use this information to determine which treatments are better suited for individual 185

patients and to further optimize medical device outcomes according to patient characteristics in one’s own setting. 186

16

Table 4. Methods for implementing and improving the quality of observational

studies

Methods Usefulness

Study planning Studies can be initiated by healthcare providers who

approach the industry for provision of hardware/software; or

by the industry who approaches providers to coordinate

research using their devices.

Compared to pharmaceuticals, devices can be more costly and need

to be delivered in accordance with indication criteria for the

specified target population. Cooperation between providers and

industry can reduce the burden, improve the correct utilization of

new devices and ensure that patients derive a benefit.

A research protocol should be created whether data

collection is retro- or prospective.

Developing a research protocol is particularly important when there

are many outcomes of interest and resource use is limited; and when

different providers are collecting and reporting data, such as in the

creation of patient registries.

The clinical outcomes identified should be appropriate for

measuring device performance and patient benefit. It could

be useful to correlate device performance with several PRO

measures to assess which measures are more sensitive.

Demonstrating patient benefit next to device performance is

particularly important for obtaining reimbursement when many

devices are available for the same indications (Dawisha 2011).

For conducting an economic evaluation measure of device

use, valid and reliable functional status questionnaires,

generic and disease-specific health-related QOL measures

should be considered. It is recommended to use both a

disease-specific and a generic instrument that are appropriate

and sensitive to the health condition of interest (EUnetHTA

2015).

While a disease-specific instrument gives more information on the

impact of the specific health condition; generic instruments can

illustrate how the specific health condition can impact other aspects

of health that would otherwise remain unknown. Also, the latter

allows a comparison of different interventions intended for similar

target populations and/or indications of use (EUnetHTA 2015)

Study design Consider including a comparison group in the study, even

when the study aim is to describe device outcomes. This

cohort could include patients who have not yet been exposed

to the medical device or those representing a different

pathway or indication of use.

A direct comparison makes it possible to assess whether patients

really benefit from a new device or technological feature. These

study designs provide higher level of evidence and are more useful

for demonstrating medical device value, particularly when a RCT is

not possible.

Blinding participants to the allocated treatment may be

possible in CER of diagnostic and therapeutic devices, but is

not always feasible in practice, particularly for implantable

devices.

This helps to reduce bias and uncertainty in study results.

Consider including an independent statistician who is blinded

to the data.

17

Ethical approval and

voluntary consent

Ethics approval and voluntary consent should be sought by

providers even if research is of a retrospective nature.

As data protection laws are becoming more stringent on the use of

patient-level data (Miani 2014) there is a growing importance of

obtaining ethical approval and voluntary consent particularly for

retrospective analysis.

Table 4. (continued)

Methods Usefulness

Data collection and

analysis

At study initiation the characteristics of study participants

should be collected and recorded to allow covariate analysis.

Covariate analysis assesses the sensitivity of study outcomes, and

helps quantify and reduce the uncertainty in RWE.

Study monitoring by a third-party is crucial for regulatory

studies and should also be considered for post-market

studies. Third parties can also collect data and supply the

anonymized information to the respective stakeholders.

Monitoring by regulators and/or other third parties can reduce bias

and confirm the research protocol is being correctly implemented,

which would carry particular importance in early stages of

developing registries. Data collection by third parties can also

remove barriers to patient-level data (Miani 2014).

Study reproting Any kind of industry involvement should be disclosed. Industry involvement does not necessarily imply a conflict of

interest. Most industry partners are subject to government

regulations that impose stringent criteria on medical device research

studies to secure clinical, economic and quality of life evidence. This

would depend on how the industry is involved in research and to

what degree.

Decisions on ethical appraisal and voluntary consent should

be disclosed.

The lack of such information particularly in retrospective studies can

degrade the reliability of the study and the quality of the publication.

If data is collected at several intervals the results from not

just the last follow-up visit, but also from all intervals should

be reported.

Long-term studies help illustrate the stability and reliability of

device benefit. This kind of data supports decisions on

reimbursement.

Individual-level outcome data should always be provided

either in the report or as supplementary material.

This increase transparency and allows further statistical analysis as

part of technology appraisals.

Dissemination of

results

Upon completing the research, results should be published

and disseminated to all involved parties as soon as possible.

Studies showing a large effect of an intervention may be crucial for

making decisions on device utilization, and may even be requested

by payers and decision makers early on in the data collection and

analysis process (Polly 2011).

18

Such data also help improve the design and conduct of CER by offering a better understanding of which outcome 187

measures really matter and which subgroup of patients benefit the most (Sullivan 2011). These become more 188

apparent as data accumulates over time. Big databases can provide information for a period longer than a product 189

lifetime and give insight into comparative effectiveness of incremental product development. 190

Conducting randomized and quasi-randomized controlled trials 191

Designing trials that are fit for purpose is key in determining the type of evidence needed as part of a pay-for-192

performance and risk-based reimbursement models and supporting the expected product value story. Clinical 193

evaluations of innovative high-risk medical devices are necessary to obtain premarket approval (PMA). Establishing 194

high-level evidence for demonstrating the safety and efficacy of MDs requires manufactures to conduct randomized 195

clinical trials. While beneficial for satisfying statutory requirements, the results of an RCT, however, may not 196

necessarily demonstrate device performance. In addition, they are labor intensive, time consuming, expensive, and 197

methodologically challenging to conduct for MDs. Furthermore, the specific characteristics of patients enrolled in a 198

RCT may also limit the generalizability of the results to a broader population. As such, more pragmatic designs may 199

be reasonable options for demonstrating the value of MDs. 200

Bernard et al. (2014) explored various methodologies to support the clinical development of MDs when faced with 201

the specific challenges of timing of assessment, eligible patient population and recruitment, acceptability, blinding, 202

choice of comparator group, and the learning curve. The authors discussed several quasi-randomized experimental 203

designs, also noting their limitations: 204

a. Zelen’s design trials and randomized consent design trials: Individuals are randomized to treatments before 205

consenting to participate, and are asked if they accept the new treatment during the consent stage. Those 206

who decline are then enrolled in the standard treatment arm (Zelen 1979). This study design facilitates 207

patient recruitment; however, a high number of participants may be needed to reach good statistical power. 208

Such a design can cause selection bias and pose ethical considerations. 209

b. Expertise-based RCTs: Individuals are randomized to different HCPs that only provide the intervention 210

they are specialized in. This study design presents uncertainty about whether observed differences are 211

19

primarily due to the expertise of clinicians. This could be reduced by establishing similar levels of clinician 212

expertise and similar number of interventions provided by each clinician (Devereaux 2005). 213

c. Tracker trials: A new device can be compared to standard practice early on in its development using a 214

flexible protocol that is sensitive to tracking progress in device outcomes over time. With fewer 215

prerequisites this study design allows changes to the study protocol during the trial and requires interim 216

analysis (Lilford 2000). Such studies could provide rich information; however, there are practical 217

difficulties in study organization, methodology and costs. 218

d. Cluster randomized trials: Instead of individuals, clusters of patients are randomized to treatments, and 219

each center is randomized to provide one treatment (Campbell 2000). Differences in cluster size and 220

participant characteristics can lead to selection bias and lack of statistical power. 221

e. Cross-over trials: Individuals are exposed to different interventions delivered in a random order; hence, 222

everyone receives all treatments. This reduces the number of patients needed. Such studies are difficult to 223

execute for devices with an associated learning curve, and are not appropriate for evaluating rapidly 224

changing health conditions or device outcomes. Special considerations include the length of device 225

use/exposure, time between device use (the wash-out period), the participant`s learning curve and 226

accounting for the carry-over effect in statistical analysis (Hills 1979). 227

f. Sequential trials: These are trials implementing interim analysis where results obtained with patients 228

already included are analyzed before the randomization of new patients. This allows smaller sample sizes 229

and early termination based on initial results; however, only short-term outcomes can be measured. The 230

study design requires oversight by an independent committee and may lack power for secondary endpoints 231

or adverse events. 232

g. Bayesian methods combine a priori information from the literature or expert opinion to inform 233

development of more efficient trial designs, but run the risk of including erroneous prior information. 234

There continues to be a paucity of high-quality clinical evidence even with alternative designs – a concern that could 235

very well create additional barriers to device procurement and reimbursements in a value-seeking global payer 236

environment. A study by Boudard (2013), reviewing hospital-based HTAs for innovative MDs, found that only 47 237

(22%) of 215 clinical studies included in the assessments provided high-quality clinical evidence on the Sackett 238

20

scale (levels 1-2), with only 33 (15%) of those being RCTs. A majority (52.1%) of studies included fewer than 30 239

patients, and only 14 of the 47 high-quality studies reported the amount of missing data. A follow-up period was 240

mentioned in only 84 (71.8%) studies of implantable MDs, averaging 18.9 months. Interestingly, the methodological 241

quality did not increase with the risk level of the medical device. 242

If a device impacts a relatively small patient population or if the effects are observed only in the long-term, a 243

traditional trial setting may not be appropriate to demonstrate value. An assessment of surrogate endpoints, the 244

extrapolation of long-term outcome by modelling, or both may be considered. These are also currently accepted 245

methods for providing supportive data for RCTs where the benefits extend beyond the duration of the trial. 246

Practical solutions to address challenges in medical device value assessment 247

While it is recognized that there may not be a perfect method for evaluating all MDs, effort needs to be made to 248

pursue the most feasible and appropriate assessment. Along with determining clinical outcomes, evidence should be 249

able to adequately inform procurement and reimbursement decisions based on economic evaluations demonstrating 250

the impact of technology adoption. 251

The level of evidence required for value assessment and the pathway followed for reimbursement differ between 252

devices and should reflect the specific aspects of the device under evaluation. For MDs in higher risk categories, 253

more data should be provided on effectiveness and safety at the time of value assessment compared to those in lower 254

risk categories. Moreover, different inputs and outcomes should be considered according to the type of product 255

under review such as benefit-risk determinations, patient education, end-user preferences, user's perspective on the 256

device's ease of use and compatibility, need for organizational changes, pricing information, performance data, etc. 257

i. Think of the real world when generating evidence 258

The ability to assess the value of medical devices depends primarily on the availability of reliable evidence. Small 259

numbers of early adopters and high research cost often make it difficult to generate clinical evidence prior to market 260

21

approval. The collection of effectiveness and safety data post approval is fundamental in order to adequately assess 261

the performance of the new technology in the real-world setting. The generation of RWE is becoming a common 262

practice, particularly through the development of registries. An a-priori agreed upon protocol for generating 263

evidence will then be necessary to support pooling data from different centres. 264

The number of patients using a specific medical device in each health care center might be lower than desired to 265

gather enough evidence for evaluating the added value of the new device. Because of this, it is recommended that 266

RWE be generated by pooling together data from as many centers as possible. 267

Due to the difficulty in obtaining clinical evidence before market approval for some MDs, decisions made by the 268

payer will contain a high level of uncertainty in terms of the value added by the new device. It is recommended that 269

the interested parties agree on reimbursement strategies which will allow sharing the risks and/or costs of 270

introducing the new technology in the market. These types of arrangements are especially recommended for those 271

health care centers introducing new technology at an early stage after market approval. Regulators and 272

reimbursement agencies in the US and in some European markets already offer such channels for market access, and 273

these should be considered for broader adoption. Known as “coverage with evidence development”, these strategies 274

are only rarely employed. 275

ii. Go “beyond the product” 276

As described in previous sections, a medical device's safety and effectiveness often depends on indirect factors such 277

as the learning curve of the user's skill. These factors are not measured in economic evaluation techniques 278

commonly used in the assessment of drugs. Specific methodologies to overcome the technical challenges in HTA 279

inherent to MDs have already been proposed. For example, the learning curve can be introduced in a cost-utility 280

analysis through a decreasing rate of short-term failure or technical failure which will have a direct impact on the 281

utility outcome (Suter 2013). In light of limited safety outcomes, information on how much to decrease the short-282

term rate can be obtained through communication with clinicians or providers. When sufficient evidence exists to 283

carry out a systematic literature review, the contribution of effect-modifying factors such as the learning curve 284

22

should be considered and quantified (EUnetHTA 2015). 285

iii. Focus value assessments on strong indications for use 286

The intended use of MDs is generally broader than the indications given for pharmaceuticals. For example, the CE 287

mark given for European marketing approval comes with a general description in terms of target population or place 288

in the therapeutic pathway. This broad applicability of devices makes the work of HTA bodies and decision makers 289

more burdensome; but has the advantage of potentially increasing the applicable study population. Moreover, the 290

broader the scope the more difficult it is for manufacturers to demonstrate the added benefit of their device 291

compared to existing alternatives (every additional study represents an increased cost to the manufacturer which is 292

burdensome if a product’s commercial potential is small, even when medical impact may be large). Given the 293

increasing number of MDs available in the market and the increasing constraint of resources available to payers, it is 294

recommended that manufacturers narrow the scope of the intended use of the technology at the time of value 295

assessment. Fewer numbers of indications are less confusing and burdensome to the supply chain. If a product is 296

used for a broad range of indications (i.e., hypodermic needles, syringes, vascular stents), manufacturers are advised 297

to try to demonstrate the benefit of the product for only those applications which they believe are easiest to prove, or 298

where the product is likely to be more widely used or accepted. Additionally, subgroup analyses may be useful for 299

many MDs in order to perform value assessment from a more realistic perspective. 300

iv. Get creative if you can`t randomize 301

Due to the invasive nature of most high-risk medical devices, it is rarely possible to blindly randomize patients to 302

different treatments (Sedrakyan 2010). Another challenge is identifying a similar comparator device, which is also 303

the case for innovative MDs targeting a population that is otherwise left untreated (KNAW 2014). Well-designed 304

quasi-randomized controlled trials may be a solution, particularly where patients are allocated to intervention arms 305

depending on their own or physician`s preference. Direct observational CER may resolve some issues for regulatory 306

purposes with a control group of individuals waiting to receive treatment. An indirect comparison of alternatives by 307

decision-modelling or a network meta-analysis may be a more amenable randomized design. In such situations, it is 308

23

advisable to contact regulators, and sometimes HTA bodies, early on in the process to discuss data requirements and 309

alternative approaches. 310

v. Ensure value-based healthcare is at the core of decision making 311

Particularly for medical devices, procurement and value assessment can be detached, creating uncertainty in patient 312

access. As indicated in our manuscripts, this is driven by misalignment of the processes of medical device value 313

assessment. All stakeholders including manufacturers, HTA organisations, procurement councils, CCG’s, health 314

funds and other ‘payers’ need to ensure an unmet need is aligned on, from the same perspective, in order for 315

practical methods of value assessment to be implemented. Otherwise, what is suitable for one stakeholder may not 316

be so for another, creating delays and inefficiencies, and short-term decision making. The only way around this is 317

cross-silo collaboration, and clear communication. This requires all parties to sit around the same table. Thus, a core 318

recommendation is to put value based healthcare at the center of the equation to clearly define unmet needs. 319

Conclusions 320

While comparison with pharmaceutical evaluations may be useful, limiting assessment methodology options to 321

RCTs is inappropriate for medical devices. Instead, payers and providers may consider observational studies using 322

sound, reliable designs. Manufacturers and healthcare researchers could then be freed from the burden of expensive 323

RCTs (reducing validation costs may appear to jeopardize product safety) and may be encouraged to systematically 324

develop and invest in continuous evidence generation that could offer useful data for health systems, organizations, 325

payers and patients and help develop evidence-based standards of care. The level and volume of evidence should be 326

a function of medical device safety risk and anticipated extent of product adoption. 327

Payers and manufacturers should cooperate to further de-risk device development, and facilitate performance based 328

payment and incentive contracts which ultimately benefit patients and stakeholders by encouraging the use of 329

relatively better performing devices. Developing classification systems of devices that could help with such a 330

process will therefore be a common interest. 331

24

Reference List 332

333 1. Drummond M, Jönsson B, Rutten F. The role of economic evaluation in the pricing and reimbursement of 334

medicines. Health Policy 1997;40(3):199-215. 335 2. Fry RN, Avey SG, Sullivan, SD. The Academy of Managed Care Pharmacy format for formulary submissions: 336

An evolving standard—A foundation for Managed Care Pharmacy Task Force Report. Value Health 337 2003;6:505–521. 338

3. McGuire A, Raikou M, Kanavos P. Pricing pharmaceuticals: Value based pricing in what sense. Eurohealth 339 2008;14(2):3-5. 340

4. Sullivan SD, Watkins J, Sweet B. et al. 2009. Health Technology Assessment in Health‐Care Decisions in the 341 United States. Value in Health. 2009; 12(s2): S39-S44. 342

5. Massetti M, Aballéa S, Videau Y, Rémuzat C, Roïz J, Toumi M. A comparison of HAS & NICE guidelines for 343 the economic evaluation of health technologies in the context of their respective national health care systems 344 and cultural environments. J Mark Access Health Policy 2015;3:24966. 345

6. Huot L, Decullier E, Maes-Beny K, Chapuis FR. Medical device assessment: scientific evidence examined by 346 the French national agency for health – a descriptive study. BMC Public Health 2012;12:585. 347

7. Schreyögg J, Bäumler M, Busse R. Balancing adoption and affordability of medical devices in Europe. Health 348 Policy 2009;92(2):218-24. 349

8. Tarricone, R., Drummond, M. Challenges in the clinical and economic evaluation of medical devices: The case 350 of transcatheter aortic valve implantation. J Med Market 2011;11(3):221-9. 351

9. Eisenberg JM. Clinical Economics: A guide to the economic analysis of clinical practices. JAMA 352 1989;262(20):2879-86. 353

10. Buxton MJ. Economic evaluation and decision making in the UK. Pharmacoeconomics 2006;24(11):1133-42. 354 11. Robinson JC. Value-based purchasing for medical devices. Health Affairs 2008;27(6):1523-31. 355 12. Kirisits A, Redekop WK. The economic evaluation of medical devices: Challenges ahead. Appl Health Econ 356

Health Policy 2013;11(1):15-26. 357 13. Ventola CL. Challenges in evaluating and standardizing Medical Devices in Health Care facilities. Pharm 358

Therapeutics 2008;33(6):348-359. 359 14. Craig JA, Carr L, Hutton J, Glanville J, Iglesias CP, Sims AJ. A review of the economic tools for assessing new 360

Medical Devices. Appl Health Econ Health Policy 2015;13:15–27. 361 362

15. KNAW (Royal Netherlands Academy of Arts and Sciences). Evaluation of new technology in health care: In 363 need of guidance for relevant evidence. Amsterdam: KNAW, 2014. Available from: 364 https://www.knaw.nl/en/news/publications/evaluation-of-new-technology-in-health-care-365 1/@@download/pdf_file/verkenning-new-technology-health-care.pdf [Accessed September 9, 2015] 366

16. Daniel GW, Rubens EK, McClellan M. Coverage with evidence development for Medicare beneficiaries: 367 challenges and next steps. JAMA internal medicine 2013; 173(14):1281-1282. 368

369 17. Garrison L, Towse A, Briggs A, de Pouvourville G, Grueger J, Mohr P, Siviero P, Miguel Sleeper AC. 370

Performance-Based Risk-Sharing Arrangements-Good Practices for Design, Implementation and Evaluation: 371 An ISPOR Task Force Report Value Health 2013; 16: 703-719 372

18. Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, risk 373 sharing, or patient access scheme? A framework for coverage decisions. Value Health 2012; 15(3):570-579. 374

19. Neumann PJ, Chambers JD, Simon F, Meckley LM. Risk-sharing arrangements that link payment for drugs to 375 health outcomes are proving hard to implement. Health Affairs 2011; 30(12):2329-2337. 376

377 20. OCEBM Levels of Evidence Working Group. “The oxford 2011 Levels of Evidence” Oxford Centre for 378

Evidence-Based Evidence Available from http://www.cebm.net/ocebm-levels-of-evidence/. [Accessed March 379 20, 2013]. 380

25

21. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Available 381 from 382 http://faculty.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf 383 [Accessed May 6, 2014]. 384

22. Fischer MA, Avorn J. Academic detailing can play a key role in assessing and implementing comparative 385 effectiveness research findings. Health Aff (Millwood) 2012;31:2206-12. 386

23. Jalbert JJ, Ritchey ME, Mi X, et al. Methodological considerations in observational comparative effectiveness 387 research for implantable medical devices: an epidemiologic perspective. Am J Epidemiol 2014;180:949-58. 388

24. Price D, Graham C, Parkin CG, et al. Are systematic reviews and meta-analyses appropriate tools for assessing 389 evolving Medical Device Technologies? J Diabetes Sci Technol 2015; pii: 1932296815607863. [Epub ahead of 390 print]. 391

25. Backhouse ME, Wonder M, Hornby E, et al. Early dialogue between the developers of new technologies and 392 pricing and reimbursement agencies: A pilot study. Value Health 2011;14:608-15. 393

26. U.S. Food and Drug Association (FDA). Investigational Device Exemptions (IDEs) for Early Feasibility 394 Medical Device Clinical Studies, Including Certain First in Human (FIH) Studies (2013). Available from: 395 http://www.fda.gov/downloads/MedicalDevices/DeviceRegulati%20onandGuidance/GuidanceDocuments/UC396 M279103.pdf2 [Accessed April 22, 2016]. 397

27. Abrishami P, Boer A, Horstman K. How can we assess the value of complex medical innovations in practice? 398 Expert Rev Pharmacoecon Outcomes Res 2015;15:369-71. 399

28. Reynolds IS, Rising JP, Coukell AJ, et al. Assessing the safety and effectiveness of devices after US Food and 400 Drug Administration approval: FDA-mandated postapproval studies. JAMA Intern Med 2014;174:1773-9. 401

29. Pham B, Tu HAT, Han D et al. Early economic evaluation of emerging health technologies: protocol of a 402 systematic review. Syst Rev 2014;3:81. 403

30. World Health Organization (WHO). Clinical evidence for medical devices: Regulatory processes focussing on 404 Europe and the United States of America: background paper 3, August 2010. Available from: 405 http://apps.who.int/iris/bitstream/10665/70454/1/WHO_HSS_EHT_DIM_10.3_eng.pdf [Accessed April 22, 406 2016]. 407

31. Bergsland J, Elle OJ, Fosse E. Barriers to medical device innovation. Med Devices 2014;7:205–209. 408 32. Sorenson C, Drummond M, Kanavos P. Ensuring value for money in health care: the role of health technology 409

assessment in the European Union. WHO Regional Office Europe; 2008. 410 33. Dawisha SM, Kline Leidy N. Patient reported outcomes in decision making and communication. In: Ackerman 411

SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic Device Outcomes 412 Research. USA: International Society for Pharmacoeconomics and Outcomes Research, 2011. 413

34. European Network for Health Technology Assesment (EUnetHTA).Guidelines for Therapeutic medical devices, 414 2015. Available from http://www.eunethta.eu/eunethta-guidelines [Accessed February 1, 2016]. 415

35. Miani C, Robin E, Horvath V et al. Health and healthcare: Assessing the real-world data policy landscape in 416 Europe. RAND Europe, 2014. 417

36. Polly DW. Therapeutic and diagnostic device clinical outcomes research practical considerations. In: 418 Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E., eds., Therapeutic and Diagnostic Device 419 Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes Research, 2011. 420

37. Haute Autorite de Sante (HAS). Medical Device assessment in France. 2009. Available from: http://www.has-421 sante.fr/portail/upload/docs/application/pdf/2010-03/guide_dm_gb_050310.pdf [Accessed October 03, 2015]. 422

38. National Institute for Health and Care Excellence (NICE). Medical Technologies Evaluation Programme 423 Methods guide. 2011. Available from http://www.nice.org.uk/Media/Default/About/what-we-do/NICE-424 guidance/NICE-medical-technologies/Medical-technologies-evaluation-programme-methods-guide.pdf 425 [Accessed April 22, 2016]. 426

39. Ciani O, Wilcher B, Blankart CR et al. Health technology assessment of medical devices: A survey of non-427 European union agencies. Int J Technol Assess Health Care 2015;31:154-65. 428

40. GetReal. Glossary of definitions of common terms. Innovative Medicines Initiative (IMI) GetReal. Available 429

26

from: http://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary.pdf 430 [Accessed February 22, 2016]. 431

41. Sullivan E, Mathias SD. Patient reported outcomes in therapeutic and diagnostic device research. In: Ackerman 432 SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic Device Outcomes 433 Research. USA: International Society for Pharmacoeconomics and Outcomes Research, 2011. 434

42. Bernard A, Vaneau M, Fournel I. et al. Methodological choices for the clinical development of medical devices. 435 Med Devices (Auckl). 2014;7:325-34. 436

43. Zelen M. A new design for randomized clinical trials. N Engl J Med. 1979;300:1242-45. 437 44. Devereaux PJ, Bhandari M, Clarke M et al. Need for expertise based randomised controlled trials. BMJ 438

2005;330(7482):88. 439 45. Lilford RJ, Braunholtz DA, Greenhalgh R, Edwards SJ. Trials and fast changing technologies: the case for 440

tracker studies. BMJ 2000; 320(7226):43-6. 441 46. Campbell MJ. Cluster randomized trials in general (family) practice research. Stat Methods Med Res 442

2000;9(2):81-94. 443 47. Hills M., Armitage P. (1979). The two-period cross-over clinical trial. Br J Clin Pharmac;8:7-20. 444 48. Boudard A, Martelli N, Prognon P, Pineau J. Clinical studies of innovative medical devices: What level of 445

evidence for hospital-based health technology assessment? J Eval Clin Pract 2013;19(4):697-702. 446 49. Suter LG, Paltiel AD, Rome BN, Solomon DH, Thornhill TS, Abrams SK, Katz JN, Losina E. Placing a price 447

on medical device innovation: The example of total knee Arthroplasty. PLoS One 2013;8(5):e62709. 448 50. Sedrakyan A, Marinac-Dabic D, Normand SL, Mushlin A, Gross T. A framework for evidence evaluation and 449

methodological issues in implantable device studies. Med Care 2010;48(6 Suppl):S121-8. 450