[Type text] DRAFT – DO NOT CIRCULATE [Type text]

www.cmtpnet.org

Building a Strategic Framework For Comparative Effectiveness Research in Oncology

NATIONAL LEADERSHIP SUMMIT ON CER PRIORITIES, METHODS AND POLICY

Final Report

Center for Medical Technology Policy

September 2011

NATIONAL LEADERSHIP SUMMIT ON CER PRIORITIES, METHODS AND POLICY

2

Table of Contents

Executive Summary ....................................................................................................................................... 5

About CMTP ................................................................................................................................................ 10

Project Staff................................................................................................................................................. 10

Acknowledgements ..................................................................................................................................... 10

Introduction ................................................................................................................................................ 11

Session 1: Setting the Stage ....................................................................................................................... 12

Keynote 1: Oncology and CER: Essential Elements and Priorities, Dr. Mark McClellan ............................ 15

Session 2: Engaging Patients in CER ........................................................................................................... 15

The Role of the Patient Advocacy Community in Advancing CER........................................................... 16

New Registries Supported by Advocacy Community .............................................................................. 17

Improving Knowledge and Training among Patient Advocates .............................................................. 17

Examples of Successful Patient Engagement in Research ...................................................................... 18

Engaging Patient Advocates in Research Design .................................................................................... 19

The Role of Patients in CER ..................................................................................................................... 19

Keynote 2: Transforming America’s Healthcare, Dr. Otis Brawley ............................................................ 21

Session 3: Improving Clinical Evidence for Oncology ................................................................................. 21

Case Example: Off-Label Prescribing in Oncology .................................................................................. 22

Improving the Quality and Utility of Evidence ........................................................................................ 23

Oxaliplatin for Stage III Colon Cancer: Example of a Registry Study ....................................................... 25

Development of Efficient Data Systems to Support CER ........................................................................ 27

Session 4: The Critical Challenge of Genomics and Personalized Medicine .............................................. 30

An Evolving Framework for Drug Development ..................................................................................... 30

New Research Strategies to Advance Drug Development for Genomic Medicine ................................. 31

Incorporation of Patient Values into Genomic CER ................................................................................ 31

Critical Role of Genetic Diagnostics in Genomic Medicine and CER ....................................................... 32

Collaboration to Advance Genomic and Personalized Medicine through CER ....................................... 34

Session 5: Translating Evidence into Practice ............................................................................................. 37

Clinical Practice Guidelines ..................................................................................................................... 37

Quality Initiatives .................................................................................................................................... 38

3

Learning Healthcare Systems .................................................................................................................. 39

Engaging Community Practice ................................................................................................................ 40

The Economic Imperative ....................................................................................................................... 40

Keynote 3: Reimbursement Reform, Dr. Peter Bach ................................................................................. 43

Session 6: Value and Costs in Oncology ..................................................................................................... 43

Patients’ Role in Value Decisions ............................................................................................................ 44

The Role of Cost ...................................................................................................................................... 45

Developing Costs and Value Information ............................................................................................... 46

CER and Value-Based Pricing .................................................................................................................. 46

References .................................................................................................................................................. 49

Appendix A: Advisory Committee .............................................................................................................. 51

Appendix B: Summit Attendees ................................................................................................................. 52

Appendix C: Agenda ................................................................................................................................... 60

Appendix D: Acronyms ............................................................................................................................... 63

4

List of Tables and Figures

Table 1: Comparison of strength of evidence for radiation treatment options in clinically localized

prostate cancer (adapted from the AHRQ Technology Assessment, August 2010). .................................. 12

Table 2: Results of a survey of research advocates (n=48) by the Research Advocacy Network ............... 18

Table 3: Presence of 14 agent and cancer combinations across five compendia ...................................... 23

Table 4: Genetic tests with established medical necessity according to Regence Blue Cross and Blue Shield

.................................................................................................................................................................... 33

Figure 1: Increase in number of clinical trials performed between 1975 and 2005 ................................... 22

Figure 2: Criteria defined by GRADE working group to score the quality of evidence ................................... 25

Figure 3: The current model of cancer genomic test development, including the relevant stakeholders,

outcome at each stage, and limitations ........................................................................................................ 33

Figure 4: Test developers, research groups and the health care systems must all play specific roles in order

to improve the quality of tests in development. ........................................................................................... 35

Figure 5: Evidence generation in a learning healthcare system ................................................................. 39

Figure 6: Some of the many considerations that factor into patients’ definitions of value ....................... 44

5

Executive Summary

The delivery of medical care in the United States (US) is undergoing radical change, made possible

through improved and more systematic methods of clinical data collection and application in clinical

practice. Initiatives and funding under the American Recovery and Reinvestment Act (ARRA) in 2009 and

the Patient Protection and Affordable Care Act (PPACA) in 2010 have injected new energy and urgency

into this transformation. Much of the activity focuses on comparative effectiveness research (CER),

defined by the Federal Coordinating Council (FCC) as “the conduct and synthesis of research comparing

the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor

health conditions in “real world” settings. 1 Despite substantial interest in CER, uncertainty persists

about the CER enterprise that will emerge over the next few years and the role of the various

stakeholders such as the Patient Centered Outcomes Research Institute (PCORI), AHRQ, National

Institutes of Health (NIH), patients, payers, and industry.

There is wide agreement on the goal of CER —to create new and better evidence on the relative

effectiveness of available treatments. Advancement from this goal to action and results will require a

systematic approach. First, we must agree on which priorities to focus in order to optimize the impact

of CER. Priority-setting in CER is unique. CER defines the patient as the central player, so studies must

be designed to address the needs of patients and clinical providers as they choose among treatment

options. Accustomed to investigator- and hypothesis-driven research that is designed to advance the

state of the science (which may or may not be aligned with patients’ interests), the research community

has limited experience and tools for engaging patients and providers in the research process.

Second, we must develop strategies for advancing CER in ways that yield answers that will maximize

clinical utility. Within the broad framework of CER, the diversity of relevant questions necessitates the

utilization of a range of methods and study designs. Randomized controlled trials (RCTs) represent the

gold standard for ensuring the validity of results, but they have significant shortcomings. For instance,

they do not necessarily address the questions of greatest interest to patients, they are resource

intensive, and they can be difficult to generalize. Some CER questions may require other methods,

though reliance on non-randomized studies generally implies a tradeoff between the higher internal

validity offered by well-designed RCTs and the greater external validity of other CER designs.

Understanding how to weigh these tradeoffs is a central challenge, and potential achievement, of CER.

In November 2010, the Center for Medical Technology and Policy (CMTP) convened the National

Leadership Summit on CER Priorities, Methods, and Policy, focusing this year on oncology research. In

this report, we present proceedings from the Summit, whose goal was to convene leading experts in CER

and oncology research in order to share groundbreaking work and discuss priorities, methods, and

policies to advance CER in the coming decade. A diverse group of stakeholders attended, including

representatives from government, academia, patient organizations, payers, and philanthropic

organizations. The information and ideas that emerged from the presentations, panel discussions, and

breakout sessions represent an important body of knowledge that provides a foundation for CER in

6

oncology. As PCORI and others develop strategies to address critical evidence needs, this summary—

representing the best thinking of leaders and stakeholders in oncology and CER - may provide helpful

guidance.

Oncology was selected as the focus of the 2010 inaugural National Leadership Summit on CER Priorities,

Methods and Policy for compelling reasons. Cancer is a particularly challenging field for CER because

new technologies are rapidly evolving, numerous new treatment regimens and targeted agents are

available, and personalized medicine is becoming a reality. Because of this rapid change, there is an

urgent and escalating need for evidence to guide treatment and payment policies. Six principles

emerged from discussions at the Summit:

1. CER studies in oncology are urgently needed.

In terms of mortality and decreased quality of life, cancer exacts a high toll. The human and

economic costs to individuals and to society are immense. Dramatic advances in treatment have

been achieved, but important gaps remain in our understanding of which treatments work and which

benefit patients most. An explosion in the number of available agents to treat patients and an

exponential rise in the number of trials assessing different treatment regimens has created decision-

making chaos; some patients receive treatments that may be ineffective, harmful, and even deadly

while other patients may not receive therapies with high potential benefit. There is an urgent need to

create better evidence to support these decisions.

2. CER should fully engage patients and the general public.

Patient involvement in clinical research design helps illuminate the questions that are most

meaningful and relevant to treatment decisions. CER is often envisioned as “real-world” research;

its results are meant to be applicable to actual patients in order to help them make scientifically

sound decisions about their healthcare. With a central role in clinical decision-making, patients

must have a mechanism through which to communicate their needs and this information must

contribute to the process of determining research questions, outcomes of interest, and study

designs.

Although challenging, resources are available to facilitate this aspect of CER. Patient advocacy

groups represent a critical link to patients by providing education, encouraging enrollment in trials,

providing guidance to researchers and helping to disseminate new clinical findings. Furthermore,

advocacy groups have developed novel approaches to engaging patients in research such as the

Love/Avon Army of Women, which now comprises over 340,000 volunteers. At this juncture, we

must cultivate engagement with patient advocates and educate them as well as patients about CER

so that they understand its potential to generate knowledge.

7

In addition to patients and patient advocates, we must engage the public and other decision-

makers directly in discussions of CER, as individuals from diverse domains can inform and guide

research and ensure that it is applicable to real-world scenarios and concerns. Education of the

public about CER can help dispel potential suspicions about its goals (e.g., perception that cost-

saving is the primary motivation), thus increasing the willingness of patients to participate in

studies. Transparency throughout the research process can help to align the objectives of the

medical community with those of the public. Encouraging mutual respect and understanding

among stakeholders will facilitate the exchange of information about objectives, values, and

priorities.

3. CER requires the development of new research methods.

The conduct of CER in oncology raises new challenges. The traditional RCT can be an inefficient and

costly way to address research questions, but other study designs require us to navigate complex

tradeoffs between the level of certainty and the usefulness of the resulting evidence for real-world

decision-making. Registry studies, pragmatic clinical trials and simpler variations of randomized trials

offer alternatives that, when enhanced with innovative methodologies, can yield reasonably robust,

clinically applicable results. A thoughtful and intentional balance is necessary.

Development of a national CER enterprise will require new evidentiary standards, new data sources,

and improved methods for observational research. Systems are being developed to evaluate the

quality of observational research. These approaches include the Grading of Recommendations

Assessment, Development and Evaluation (GRADE) and The Good Research for Comparative

Effectiveness Initiative (GRACE). Equally important, new data standards are being developed so that

clinical information can be reliably collected, shared and analyzed. With clinical data from many

diverse sources, agreement on comparators and outcomes of interest, as well as on methods of

synthesis, is necessary in order to facilitate the exchange of information on a national scale.

Overarching the issue of data standards is that of research infrastructure. More efficiency is needed

to prevent development, de novo, of methodology and procedures for each new study – in essence

repeatedly “recreating the wheel” and then jettisoning it on study completion. The analogy

suggested at the conference was of a “reusable rocket.” Among large registries, for example, a

structural overlay could allow investigators to query multiple databases simultaneously, as they

explore clinical questions or preliminary hypotheses. The long-term investment in CER required for

infrastructure development will yield scientific and clinical benefits.

4. Genomics and personalized medicine must be central to all CER studies in oncology.

Personalized medicine relies upon individual genomic profiles and markers to anticipate disease,

predict treatment response and choose between treatment options. Following the mantra of

8

“getting the right care to the right patient at the right time,” personalized medicine aims to use

genomic data to clearly understand which treatment(s) will optimize outcomes for the individual

patient. Better evidence is needed to support this goal. For example, despite tremendous progress over

the past decade in developing drug therapies for kidney cancer, we have limited understanding of the

appropriate utilization of these agents.

The role of CER in personalized medicine has yet to be clearly defined. Oncology, more than any

other field, utilizes therapies based on differences in disease and treatment response predicted by

molecular biomarkers. Data necessary to support these sorts of clinical decisions are currently

insufficient. Given the proliferation of treatment options in oncology, however, development of a

robust evidence base using traditional research and analytic methodologies could be prohibitively

expensive. Pooled data analyses, decision modeling, and other approaches provide avenues for

addressing the extensive evidence gaps. Additionally, CER studies examining clinically relevant issues,

such as the variations in treatment efficacy among different patient populations, will be necessary

to maximize the effect of treatments. In the public arena, architects of the CER enterprise will need

to allay concern that CER will be used to ration access to genetic testing and novel targeted agents.

Pharmacogenomic studies illustrate the importance of a partnership between stakeholders in CER.

Pharmacogenomic tests do not have a defined approval process and evidence of clinical utility is

often lacking. The CER enterprise will promote co-development of agents and genetic tests and help

to ensure incorporation of patients’ goals, values, beliefs, and expectations into study design and

analyses. Incentives motivating test developers may not be aligned with patients’ values, requiring the

research community to play a proactive role in identifying promising candidate tests for CER and in

enlisting biomedical partners in generating evidence.

5. CER results must be translated into clinical practice.

In evidence-based healthcare, findings from up-to-date, high-quality research inform clinical care.

The research effort will be wasted unless obstacles to evidence translation are addressed. Improved

methods with which to promote translation will increase the ability of clinicians to implement

change. Clinical practice guidelines and quality initiatives provide potential mechanisms to integrate

emerging data into practice, yet their development requires additional funding and coordination.

Attention to various aspects of evidence development will facilitate translation of CER results. First,

the generation of reliable data requires the enrollment of patients from community settings who are

representative of real-world clinical populations, as the vast majority of patients receive care there.

Gathering data in community settings is challenging due to wide variations in practice patterns,

limited systematic data collection, and a lack of financial incentives for community-based clinicians

to participate in research. Despite these issues, initiatives such as the NCI Community Cancer

Centers Program (NCCCP) have made progress.

9

Second, CER translation depends upon infrastructure development. Re-alignment of payment

incentives would support the development of the information technology (IT) infrastructure.

Similarly, the development of methodological and data standards would provide a coordinated

framework for data acquisition. The IT infrastructure must accommodate patient-reported

outcomes (PROs), as these data are increasingly recognized as a central feature of CER. Feedback

loops that connect researchers, guideline panels, and practitioners must be “hard-wired” through IT

pathways to ensure that information flows bi-directionally to support formulation of clinically

important research questions, hypothesis generation, study conduct, translation of results, and

evaluation of impact. This is the starting point for the development of rapid learning systems in

oncology.

6. The CER enterprise must address cost and value.

While Congress limited consideration of cost and value in ARRA and PPACA, they will eventually

demand attention. Defining cost and value is not straightforward. Even if the effect of a given

treatment on survival is known, its true value to a patient is often much more difficult to define. At

the same time, direct and indirect costs are not captured in a systematic way, leading to difficulties

in analyzing cost drivers. It is important that physicians, researchers, payers, and policy makers avoid

substituting their own value judgments for those of patients and that costs not be narrowly defined.

Learning how to capture patients’ values and to incorporate this aspect of treatment evaluation into

research requires improved methods and a continuing public conversation. Ultimately, CER will need to

incorporate nuanced metrics of patient value that include measures of patient experiences, out-of-

pocket costs and other cost and value dimensions. New systems will need to be developed to

consistently and reliably capture these data.

The Summit succeeded in initiating a conversation surrounding the direction of CER in oncology, and

included a broad range of stakeholders with diverse visions in the conversation. Convergence was

seen on a strategic framework that has the potential to accelerate CER in oncology and, thereby, to

improve the quality of cancer care.

This conversation has just begun and the stakeholders involved in the process must continue the

dialogue. CMTP will do its part to foster this conversation. The first strategy is to appeal for

reactions to this report and request suggestions for further enhancing CER in oncology. As the

discussion evolves, we will make it accessible to the public through our website (www.cmtpnet.org).

10

About CMTP

The Center for Medical Technology Policy (CMTP) is a private, non-profit organization that provides a

neutral forum in which patients, clinicians, payers, manufacturers, and researchers can work together to

design and implement prospective, real-world studies to inform health care decisions. CMTP focuses on

a range of methods for evaluating comparative effectiveness, including pragmatic trials, adaptive

designs, clinical registries, and other study designs that generate evidence to provide patients, clinicians

and payers with a reasonable level of confidence in their decision-making. The primary goal of CMTP is

to improve the process for generating reliable and credible information about the real world risks,

benefits, and costs of promising new medical technologies.

Project Staff

Sean Tunis, Director

Robert B. Giffin, Senior Research Director

Laura Esmail, Senior Project Manager

Sharon Murphy, Senior Advisor

Daniel Mullins, Research Fellow

Bradford Hirsch, Research Fellow

Russ Montgomery, Research Associate

Swapna Karkare, Research Associate

Merianne Spencer, Research Associate

Julie Simmons, Event Planner

Acknowledgements

Through their dedication and hard work, many individuals helped to make this meeting and the resulting

report a success. We would like to first acknowledge the many individuals who participated in the

advisory committee that guided the program before the meeting and helped to shape this final report

(see Appendix A). We would also like to acknowledge the federal agencies and private organizations that

played an important role in providing advice and expertise as the project developed. A huge thanks goes

to all of our speakers, moderators, response panelists and workgroup leaders whose willingness to share

their ideas led to an incredibly rich discussion at the summit (See Appendix C). Finally, we extend a very

special thanks to the sponsors of this program, without whose commitment and support, the Summit

would not have been possible. They include Amgen, the Blue Cross and BlueShield Association, Eli Lilly,

Genentech, GlaxoSmithKline, Humana, Pfizer, and Sanofi-Aventis.

11

Introduction

In 2011, instead of being based on solid evidence regarding the harms and benefits of interventions, the

practice of medicine in America is still, to an alarming degree, based on beliefs and traditions that have

evolved as a result of clinical experience and expert consensus,. While there has been undeniable

progress, there are examples of the negative impact of failing to base patient care on robust evidence,

such as the use of high-dose chemotherapy and bone marrow transplant for metastatic breast cancer.

Clinicians utilized this aggressive and costly intervention until randomized controlled trials (RCTs)

demonstrated a lack of superiority to conventional chemotherapy. This and similar examples clearly

underscore the importance of basing treatment decisions on sound evidence. Unfortunately, there is

limited evidence to guide physicians and patients for many clinical scenarios. For example, despite a

number of treatment options for early-stage prostate cancer—such as surgery, external beam radiation,

and watchful waiting -- little is known about which is most effective and the impact of each on a

patient’s quality of life.2

Advanced as a paradigm for effective healthcare since the 1990’s, evidence-based medicine (EBM)

requires the availability of sufficient, clinically relevant data to inform healthcare decisions.3 With the

United States (US) healthcare system in crisis, more and possibly different types of evidence are needed

to enable truly evidence-based care. The 2010 Patient Protection and Affordable Care Act (PPACA)

included the goal of increasing the systematic creation and use of evidence in the delivery of care.4 Its

approach builds on the already well-established fields of health technology assessment and EBM. PPACA

provided for the creation of the Patient Centered Outcomes Research Instituted (PCORI), which

commissions research meant to help patients and their healthcare providers make informed decisions.

PPACA and PCORI advance the federal investment in evidence development beyond that of the

American Recovery and Reinvestment Act (ARRA) of 2009, which appropriated $1.1 billion for

comparative effectiveness research (CER).5

The concept of CER was born out of the desire for robust evidence that is directly relevant to real-world

clinical decision-making. The Federal Coordinating Council has provided the following definition of CER,

which has been endorsed with minor variations by the Institute of Medicine (IOM) and other national

agencies:

CER is the conduct and synthesis of research comparing the benefits and harms of

different interventions and strategies to prevent, diagnose, treat and monitor

health conditions in “real world” settings. The purpose of this research is to

improve health outcomes by developing and disseminating evidence-based

information to patients, clinicians, and other decision-makers, responding to their

expressed needs, about which interventions are most effective for which patients

under specific circumstances.1

12

CMTP held a two-day conference on November 1-2, 2010, to address opportunities, challenges and

approaches to CER. Participants represented academic medicine, professional/medical associations,

federal agencies, industry, and the non-profit sector. The Summit was organized into six sessions

focused on the following topics: (1) Introduction/Setting the Stage, (2) Engaging Patients in CER, (3)

Improving Clinical Evidence for Oncology, (4) The Critical Challenge of Genomics and Personalized

Medicine, (5) Translating Evidence into Practice, and (6) Value and Cost in Oncology. The proceedings

presented herein adhere to this structure and present the main themes of each session.

Session 1: Setting the Stage

Sean Tunis, Founder and Director of CMTP, opened the Summit by discussing the importance of CER and

the need for concrete, actionable goals that can move the field forward and guide the PCORI. He

described its transformative potential, highlighting the dire need for actionable evidence today. As an

example of the lack of reliable evidence with which to make clinical decisions, he highlighted the recent

systematic review of treatment modalities for early-stage prostate cancer performed by the Agency for

Healthcare Research and Quality (AHRQ).6 (Table 1) Despite the availability of hundreds of clinical trials

on this topic, the investigators conducting the review could not find sufficient evidence to recommend

specific modalities.

Table 1: Comparison of strength of evidence for radiation treatment options in clinically localized prostate cancer (adapted from the AHRQ Technology Assessment, August 2010).

6

Comparisons Disease specific

survival

Freedom from

biochemical failure

GU/GI toxicity

No treatment Insufficient Insufficient Insufficient

Cyberknife/EBRT Insufficient Insufficient Insufficient

SBR/Brachy (HD) Insufficient Insufficient Insufficient

SBR/Brachy (LD) Insufficient Insufficient Insufficient

EBRT/Brachy (HD) Insufficient Insufficient Insufficient

EBRT/Brachy (LD) Insufficient Insufficient Insufficient

Brachy HD/LD Insufficient Insufficient Insufficient

Combined modality Insufficient Insufficient Insufficient

SBR (variable) Insufficient Insufficient Insufficient

EBRT (proton, IMRT) Insufficient Moderate Moderate

Brachy (variable) Insufficient Insufficient Insufficient

Abbreviations: EBRT, external beam radiation therapy; Brachy, brachytherapy; HD, high dose; LD, low dose;

SBRT, stereotactic body radiation therapy; IMRT, intensity modulated radiation therapy.

Unfortunately, the systematic review of early-stage prostate cancer yielded results that are far from

unusual. Despite the publication of more than 18,000 RCTs each year, systematic reviews routinely

13

conclude that there is inadequate evidence to guide clinical decisions. Dr. Tunis described the

phenomenon as an “evidence paradox,” a term often used to describe the situation in which the

available data is growing exponentially, but clinicians have limited useful information. Dr. Tunis

attributed this paradox to insufficient engagement of decision-makers, patients, clinicians, and payers in

(a) selecting pertinent questions to ask, (b) designing studies that are appropriate to answer the given

questions, and (c) establishing a balance between internal and external validity.

Internal validity refers to the statistical precision and reliability of a study. A study has high internal

validity if one would expect similar results if it were repeated. A properly designed and executed RCT

represents the gold standard for internal validity. Yet, it can take so long to complete that its results may

no longer be relevant and it can have limited external validity or “generalizability” to populations

outside of the study context.

Dr. Tunis illustrated the tension between internal validity and external validity with an example from

radiation oncology. In this field, a great deal of effort is expended to maintain rigorous trial controls,

ensuring that each patient is treated nearly identically in terms of radiation dose, timing, and other

variables. While this is theoretically ideal and is unquestioned as the best way to determine efficacy, in

the real world there is great variability in how radiation is delivered. RCTs may not reflect the experience

of patients in real-world settings such as community clinics. Furthermore, the cost of achieving this

degree of rigor is high, limiting the number of trials that can be conducted, and the number of questions

that can be answered. RCTs are typically implemented sequentially and the time from trial conception to

ultimate publication of results is long, delaying the timing of evidence generation. By relying on study

designs that relax certain internal controls, the shortcomings of RCTs can be mitigated. However, some

worry that less rigor in study design might lead to results that are difficult to interpret because of

confounding factors.

Today, most NIH-funded studies are RCTs, reflecting unwillingness among scientific review panels to

forfeit a degree of internal validity in order to gain external validity. The “correct” balance is not

inherently clear; as Dr. Tunis explained, this balance is ultimately a value question and cannot be

empirically determined. The desires and values of diverse decision-makers will need to be considered in

the study design process in order to strike the correct balance.

Another challenge to CER is the divide between research and practice, a schism intrinsic to the current

research infrastructure. The research community is largely driven by intellectual curiosity to conduct

studies and generate new evidence; the research questions which investigators formulate may or may

not align with the interests and values of patients. Because of a lack of communication between

stakeholders, and specifically between those in the research and clinical realms, the questions that are

most pressing to patients and physicians often go unasked. Likewise payers may have specific questions

in mind, but such groups often lack the opportunity to discuss these issues with researchers. The divide

shows up again in the translation and dissemination of results. Health technology assessments attempt

to actively transfer knowledge of trial results to practitioners, but this process is arduous. Clinical

practice guidelines seek to translate research findings into practice recommendations, but due to

14

communication gaps there is a little feedback about the frequency with which guidelines are

implemented or the associated outcomes. By joining the evidence needs of patients, clinicians, and

payers with the energy and resources of the research enterprise, CER is meant to address these issues.

PPACA legislation, by establishing the PCORI, embeds the principles of EBM in a statutory framework

and provides a new impetus for the development and implementation of a research system designed to

support EBM. While the broad goals of CER are clear, clarity has yet to be achieved regarding the

priorities and strategies that will accomplish CER goals. CER studies will have to differ in crucial ways

from many of the traditional research methods.

Oncology represents a particularly challenging field for CER because new technologies are rapidly

evolving and multiple treatment regimens with targeted agents are available to clinicians and patients.

With this quick evolution, there is limited evidence to guide treatment choices and policy decisions.

Because of the importance and urgency of generating high-quality evidence to support clinicians making

daily treatment decisions in oncology, cancer was selected as the focus of the 2010 inaugural National

Leadership Summit on CER Priorities, Methods, and Policy.

15

Keynote 1: Oncology and CER: Essential Elements and Priorities, Dr.

Mark McClellan

Session 2: Engaging Patients in CER

The role of the patient in CER was a key topic for discussion as they are not adequately engaged in the

research enterprise. According to Susan Love, the lack of patient engagement results in clinical trials that

often fail to address the issues of most importance to patients. The research enterprise is wholly

dependent on the medical community for trial design, recruitment and data collection, resulting in

myopic studies that emphasize scientific concerns over those of patients.

Patient involvement in research confers many benefits: it can aid fundraising, increase patients’

willingness to participate in research, enhance the “reach” of dissemination and contribute important

Keynote 1 Highlights

The biggest challenge for the healthcare system is its unsustainable trajectory, but the reform

law provides a window of opportunity to address some of the underlying issues. Within five

years, the law builds in savings from significant tightening of reimbursement rates. A

controversial aspect creates the Independent Payment Advisory Board, which is tasked with

decreasing spending growth to levels three to four percent below the historic growth rate of

Medicare spending. However, tightening reimbursement alone will not lead to higher quality

care and it misses an opportunity to improve the health of the population.

It is also critical to produce evidence that is relevant to policy reform. Changes are underway

in how providers are paid by Medicare, which will create momentum and impact private

insurance markets. In the vast majority of these cases, we don’t know if activities like bundled

payments, pay-for-performance, and accountable care organizations will lead to higher value

care and it is very hard to implement reform without evidence of success.

Payment reform will provide an opportunity to gather high quality evidence about various

treatments. When reimbursement rates change, treatment choices and utilization are altered

for patients. In a sense, it is what researchers attempt to accomplish through randomization,

as payment reform is a factor unrelated to the health of the patient that influences how they

are treated.

Building infrastructure to support better decisions must involve data systems. Meaningful use

and high-tech investments will enable a more consistent method of data collection in

individual patient care. Typically, systems start with a modest, practice level IT system meant

to facilitate billing and care delivery however comprehensive, fully interchangeable electronic

records must be the goal with standardized data and interoperability.

16

insights into trial design. No longer a secondary benefit, patient engagement is now considered a

necessary component of evidence development. Full adoption of patient engagement introduces many

challenges and a limited amount is known about the most effective and acceptable methods of engaging

patients, so new research will be needed. The session focused on the strategies and methods that hold

the most promise as we move forward.

The Role of the Patient Advocacy Community in Advancing CER

Several speakers noted that significant capacity already exists to actively engage patients in the

emerging CER enterprise in order to form a bridge between the patient and research communities. The

primary infrastructure for patient engagement consists of advocacy groups that educate the public and

generate support for critical research. Strategic alliances and collaborations between advocates and

researchers could improve patient accrual, patient education and the dissemination of results, while

providing insights that help elucidate the most pressing research questions.

Mary Lou Smith discussed the role of research advocates. While patients are rightly focused on

individual health issues and are often riding an emotional rollercoaster, advocates possess in-depth

knowledge about the diseases on which they focus. They strive to use their position within the patient

community to facilitate research and to improve the delivery of care and often view their role as one of

“giving back,” which is sometimes the last step in the healing process for former patients and provides

them with considerable motivation, insight and dedication. Ms. Smith mentioned four main areas in

which advocates are often included: (1) facilitating direct patient services such as trial enrollment and

education, (2) political advocacy, (3) fundraising, and (4) research design.

Advocates themselves can aid the research community in various direct ways. They already participate

in NCI study sections, review research concepts for cooperative groups, and sit on Institutional Review

Boards (IRBs). However their involvement is not being maximized at present, according to Ms. Smith.

Advocates could exert greater influence by sitting on a greater number of relevant committees (e.g.,

IRBs), by being asked to sponsor research directly through their fundraising efforts, and by being

engaged in policy discussions. These sorts of collaborations can lead to fruitful research synergies.

In ongoing trials, advocates are critical to patient education. Ms. Brown described the role of Susan G.

Komen For the Cure in teaching patients about the importance of CER and the potential benefits of

research participation. She has found that busy researchers and clinicians often lack the time to explain

the risks and benefits of trial participation to patients and that this limits patients’ willingness to enroll.

Advocates can help to overcome this stumbling block as they have personal experiences to share and

time to address concerns. Komen for the Cure also educates providers how to better communicate with

patients regarding clinical trials and how to use advocates to bridge the gap.

The role that the advocacy community envisions for itself is broad and critically important. Ms. Bonoff

discussed the activities of the National Breast Cancer Coalition (NBCC) which helps craft informed

17

consent documents, sits on data safety monitoring committees and helps disseminate results. Their

advocates draft materials used to educate patients, to ensure that messages are presented in plain

language and speak to the intended audience. Advocates can provide information in differing degrees of

detail and via various media to help reach patients where they live and work, making efforts more

successful. Often there is resistance from researchers to the involvement of advocates, assuming that

they have little to add; however all of the discussants had numerous examples of personal experiences

to the contrary.

New Registries Supported by Advocacy Community

Dr. Love described several approaches to involving the public, including the use of open registries to

circumvent the medical community. Traditional registries recruit specific populations such as individuals

with a given risk factor or disease. These registries are often maintained in academic medical centers

and utilize employees to manually enter all relevant information. Consequently, they are expensive to

maintain, are labor-intensive, and limit research to questions that have been asked directly of a patient

or that rely on data that can be pulled from the medical record.

Open registries, in contrast, rely on the recruitment of individuals through marketing and word of

mouth, often via the Internet and email. They include both healthy volunteers and individuals who have

a condition of interest. Email lists and websites are generated and then used to conduct surveys,

approach potential study participants and allow the public to select trials of interest in which to

participate. Open studies can be circulated well beyond the mailing lists’ direct participants, relying on

viral dissemination. This model is less expensive than traditional registries and can accommodate a

wider range of questions. It is being utilized with increasing success in a time of decreasing budgets and

poor accrual in traditional settings.

Improving Knowledge and Training among Patient Advocates

If advocates are to play a meaningful role in the CER enterprise, it is important to convince them of its

importance. Ms. Smith presented results from a survey of more than 200 research advocates conducted

by her organization, the Research Advocacy Network, prior to the Summit. (Table 2) The 51

respondents represented a variety of cancer groups and geographic locations. While 28% of the

respondents had been in advocacy for less than five years, 48% had been involved for ten or more.

When asked about their current knowledge of CER, only 4% ranked it as “excellent” and 25% “very

good.”

Some expressed positive perceptions about the role of CER based on the belief that CER would help to

enhance our knowledge about the effectiveness of medical care, would encourage the use of evidence-

based practices in decision-making, and would encourage research to compare approved therapies. But

many also expressed concern that CER may be at odds with the concept of personalized medicine

18

because it may promote what is best for the average patient and neglect the needs of those who require

alternative treatment options. There were similar concerns about its impact on orphan diseases where it

could be used to ration care by eliminating reimbursement for expensive, yet effective, treatments.

Table 2: Results of a survey of research advocates (n=48) by the Research Advocacy Network

Do you feel that CER will:

Advantages Agreement

Improve the quality of care for the population of the US? 30 (62.5%)

Advance scientific discoveries into clinical practice? 27 (56.3%)

Improve an individual patient’s quality of care? 23 (47.9%)

Increase access to care for more patients? 19 (39.6%)

Disadvantages

Limit the treatment choices the provider can offer? 21 (43.8%)

Appear to ration care based on cost 21 (43.8%)

Stand in the way of personalized medicine? 13 (27.1%)

Is a barrier to scientific discovery? 12 (25.0%)

Inappropriately interfere with physician’s recommendations

for treatment?

11 (22.9%)

Limit who patients can choose as a provider? 9 (18.8%)

SOURCE: Mary Lou Smith presentation (November 1, 2010). Reprinted with permission from Mary Lou Smith.

A number of organizations have instituted educational initiatives to improve the knowledge and skills of

patient advocates. Ms. Bonoff discussed the training provided by the NBCC to its advocates. Project

LEAD is the NBCC’s “premier science training program for advocates.” It is a five-day scientific course

covering the basics of cancer biology, genetics, epidemiology, research design, and advocacy. Project

LEAD is intended to provide advocates with the knowledge and skills necessary to interact with the

research community in helpful ways.

As patient advocates learn more about research, and CER in particular, recognition is growing that it can

improve our understanding of which treatments are effective. The extreme care that has been taken to

ensure that CER addresses differences in treatment effects across population subgroups, as enshrined in

the law that established PCORI itself, tends to alleviate the concern of patient groups. The challenge

remains, however, to deliver on the promise of adequately accounting for differences across subgroups

of patients.

Examples of Successful Patient Engagement in Research

The presenters offered compelling examples to illustrate the feasibility and positive impact of engaging

patients. Dr. Love discussed the Love/Avon Army of Women which helps streamline research by

providing investigators with access to a ready pool of healthy volunteers and patients. Individuals

19

interested in helping in the fight against breast cancer join the Army of Women online. Researchers

meanwhile submit studies, including both surveys and clinical trials, for approval by the Love/Avon

Army’s Scientific Advisory Committee. If the Committee favorably reviews the study, it will be forwarded

to the entire membership. Women can then opt to complete the online screening process and, if

appropriate, their contact information is passed along to the researchers.

As of November 2010, the Army of Women included 342,000 women, 80% of whom were healthy

volunteers. To date, 38 studies have been conducted, involving 44,000 of the enrolled women. Six of

these studies increased their enrollment goals because of the ease and success of accrual. This success

demonstrates the potential benefit of a coordinated, direct approach to patient engagement.

Advantages of patient engagement through the Army of Women are bidirectional, benefiting patients as

well as research scientists. In exchange for access to the Love/Avon Army’s database, researchers are

required to educate participants about their research, for example, by holding webinars. This helps to

educate the public and allay reservations of patients about the goals of research. Thus far, studies have

addressed a broad range of topics, from diet and exercise to the utility of different biomarkers. For

many of the studies, advocacy groups and academic researchers are designing the questions in

collaboration; this public involvement helps to ensure that the questions are relevant and encourages a

culture of engagement.

Engaging Patient Advocates in Research Design

Several presenters discussed the role of advocates in improving research design by bringing attention to

patient concerns and inserting the patient’s perspective into the process of study design. For example,

because advocates were concerned about the suffering of multiple myeloma patients while on high

doses of dexamethasone, a trial was conducted by the cooperative groups to determine whether or not

lower doses would result in similar outcomes. The trial was closed early as the low-dose dexamethasone

arm was found to be superior, changing treatment for the better and answering a critical question for

patients. Examples were cited of other trials in which advocates helped to formulate the research

questions or to improve the study design, resulting in similar outcomes.

The Role of Patients in CER

This session yielded clear agreement that (a) patients must be involved in the CER enterprise in order to

reach its goal of identifying and efficiently answering patient-centered, clinically relevant questions; and

(b) the advocacy community can play a pivotal role in facilitating the involvement of patients in

research. Advocates, and patients themselves, can generate awareness and raise funds, assist with trial

design to ensure that the correct questions are asked, help build infrastructure that facilitates accrual,

educate patients about the importance of research participation and help with dissemination of

research results.

20

Discussion and Key Themes – Patient Engagement in CER

The following key themes were distilled from the presentations, panel discussions, and

breakout sessions, with input from the multi-stakeholder planning committee:

1. The central principle of CER is that it is “patient-centered.” The intention of CER is to

provide evidence needed by patients and their physicians in order to make informed

choices among treatment options. Inclusion of patients in the research process will

ensure that it adequately addresses their concerns and provides the information which

they need. Patient involvement can change the research questions posed, the outcomes

measured, and the overall design of clinical trials.

2. Patient advocacy groups should be utilized to enhance the development of CER in

oncology. Patient advocates can fill numerous roles. They can:

Educate patients on complex medical issues, resulting in better decision-making

Facilitate patient enrollment in trials

Bring patient perspectives to advisory groups, guideline committees, study teams,

and policy-making entities

Rapidly disseminate new clinical evidence

Develop novel approaches to engaging patients in research, such as the use of

internet registries

3. The research community must more actively engage the public at large. Some within the

general public harbor suspicions about the true goals of CER and fear that the “C” stands

for “cost.” This skepticism reduces patient participation in research and limits the ability

to answer critical questions. Making research more transparent and fully engaging

patients throughout the research process will help to align the objectives of the medical

community with those of the public and will dramatically enhance the opportunities for

conducting insightful CER studies.

21

Keynote 2: Transforming America’s Healthcare, Dr. Otis Brawley

Session 3: Improving Clinical Evidence for Oncology

For clinicians, important evidence gaps hinder their ability to practice EBM. As a result, many patients

receive treatments that may be ineffective, harmful, and even deadly. These negative outcomes are rarely

systematically captured and evaluated, preventing results from informing care patterns. Because of the

enormous impact of cancer on mortality and morbidity, there is an urgent need to generate better

evidence to support decision-making.

Keynote 2 Highlights

To put things in perspective, Dr. Brawley asked the crowd to pretend they were all 50 year-

old males and a preventative pill existed that would double their risk (from 10% to 20%) of

being diagnosed with prostate cancer during their lifetime, but would decrease the risk of

death related to prostate cancer by 20%, from an absolute risk of 3% to 2.4%. When asked

who would take it, no hand was raised, yet these are the statistics often quoted as a best-

case scenario in favor of screening.

The goal is to move to the rational use of health care resources, not the rationing of it. It is

important to define what we know and what we don’t know, as doctors often have trouble

differentiating between what they know and what they believe.

Screening has been shown to be effective in other types of cancer, but its utility is unclear in

prostate cancer. While PSA testing is frequently performed in men over 50 years of age, the

test may miss as much cancer as it finds and often fails to differentiate disease that requires

treatment from that which does not.

Even after prostate cancer is diagnosed, we do not know how to best treat localized

disease. Is surgery better than radiation? Within surgery, are conventional open

procedures better than robotic and laparoscopic procedures? How does one compare the

different radiation modalities such as conventional, intensity-modulated, proton beam, and

brachytherapy?

It is also important to look at drivers of treatment. As many as of a third of prostate cancer

patients receive GnRH antagonists, yet they are not risk free. Until 2004, clinicians could

procure them at a wholesale rate and then mark up the cost of the drugs when seeking

reimbursement from Medicare. In 2004, Medicare changed its reimbursement policy and

the profit margin associated with GnRH antagonists decreased, resulting in a sharp decline

in utilization. According to Dr. Brawley, there was no change in the appropriate use of

androgen deprivation therapy, only a decline in situations with less evidence to justify

treatment.

22

Conducting systematic comparisons of treatments is particularly challenging. For many types of cancer,

there are a wide variety of treatments available, there is substantial genetic variation among patients

within disease categories, and the field has seen a rapid introduction of new drugs and technologies.

CER is also a “moving target.” Often while trials examine existing treatment options, new agents are

already being introduced. The goal of the Summit’s third session was to identify some of the most

important knowledge gaps in oncology and to discuss strategies for addressing these deficiencies, such as

novel trial designs, new analytical techniques and improvements in bioinformatics.

Case Example: Off-Label Prescribing in Oncology

To explain some of the complexities confronting CER in oncology, Amy Abernethy described the

extensive use of oncology drugs for off-label indications. As shown in Figure 1, there has been a large

increase in the number of trials across all areas of medicine since the 1980s; however, the increase has

been especially rapid in oncology.

Figure 1: Increase in number of clinical trials performed between 1975 and 2005

SOURCE: Amy Abernethy presentation (November 2, 2010). Reprinted with permission from Amy Abernethy.

The off-label use of drugs currently accounts for between 50% and 75% of all prescriptions in oncology,

up from 30% in the 1990s.7 The evidence supporting its use is of variable quality, yet off label prescribing

has been gaining increasing prevalence since the Omnibus Budget Reconciliation Act of 1993 stipulated

that Medicare must cover treatments that are recommended by one of a handful of compendia.8 The

statute identified certain compendia as authoritative sources for determining a “medically-accepted

indication.” The intention of this legislation was to expedite patient access to potentially life-saving

cancer treatments. While the statute pertained specifically to coverage by the Centers for Medicare and

Medicaid Services (CMS), most third-party payers and state legislatures follow the guidance of CMS.

23

To better understand the state of the evidence, Dr. Abernethy and colleagues completed a technology

assessment in 2008.7 The study was sponsored by the Agency for Healthcare Research and Quality

(AHRQ). In comparing the evidence used by, and recommendations of, five compendia, the

investigators found limited agreement and considerable divergence in update cycles, methods for

handing equivocal evidence and choice of citations. The compendia exhibited difficulty in keeping up to

date with the rapidly evolving literature. Table 3 outlines inconsistencies found between compendia

across 14 agent and cancer combinations. In light of the complexities of keeping abreast of constantly

evolving regimens, the study raised the question of whether a compendia-based approach to evidence

review and dissemination is valid or even feasible (see additional discussion of compendia in Session 5:

Translating Evidence into Practice).

Table 3: Presence of 14 agent and cancer combinations across five compendia

SOURCE: Reprinted with permission from Annals of Internal Medicine: Abernethy AP, et al. Systematic review:

reliability of compendia methods for off-label oncology indications. Ann Intern Med. 2009 Mar 3;150(5):336-343.

The technology assessment raised concern about the evidence being used to support off-label

prescribing. Trials used by the compendia to make recommendations were predominantly Phase I or II

studies which enrolled small numbers of patients and used inconsistent outcome measures. They were

therefore difficult to interpret and of questionable reliability. As an example, the number of studies of

bortezimib increased from 75 in 2005 to 550 by 2009 and the interpretation of the results across

compendia was inconsistent. The lack of post-regulatory incentives, the limited funding for continued

study and the narrow scope of many RCTs raise questions of both internal and external validity.

Meanwhile, the context of clinical urgency pushes clinicians and patients to make decisions despite the

debatable applicability of the available data to real-world situations.

Improving the Quality and Utility of Evidence

Given the limitations of the RCT, new approaches are needed to ensure that future oncology research

will generate reliable, consistent and useful results in a timely manner. The optimal design for a study

cannot be predetermined and depends on the question being asked. The first step is to carefully assess

24

the type of information that patients and their physicians need in the context of our limited resources.

Richard Gliklich first discussed some of the strengths and weaknesses of RCTs:

Realism - RCTs do not always reflect real-world experience as they are conducted in tightly

controlled environments. In a classic example, large differences in mortality at individual

institutions have been found when a procedure is performed as a part of a clinical trial as

opposed to routine practice.

Generalizability – The results of RCTs, which are generally conducted in academic centers,

can be difficult to apply to clinical scenarios in the community. Eligibility criteria are often

narrow and exclude patients with multiple comorbidities.

Applicability – The questions asked by researchers and answered through RCTs are not

necessarily pertinent to practitioners and patients.

Availability – The number of RCTs that can be conducted is limited by the expense of these

complex studies, their resource requirements and the length of time to study completion.

As a result, many important clinical questions go unanswered.

Observational studies can address some of these issues, but many researchers express skepticism about

their dependability. In a few high-profile examples, observational CER (OCER) led to erroneous

conclusions. Dr. Gliklich used the Nurses’ Health Study as an example. The initial results indicated that

hormone replacement therapy led to fewer heart attacks among women; however, this was later

refuted in RCTs. Such mistakes have created a perceived “glass ceiling” for OCER in which its

dependability is capped at a certain level and is never given the weight afforded to RCTs. In the

evidence hierarchy for the National Institute for Clinical Excellence (NICE) in the United Kingdom,

observational studies are limited to a lower level of confidence in guideline development (level two or

below).9

Observational studies are prone to two main types of bias—selection and confounding. Selection bias

occurs because, without randomization, patients are likely to receive particular treatments based on

factors such as the availability of different treatments at a given institution or the preference of the

physician, all of which are unrelated to a patient’s presentation and consequently skew the results.

Confounding occurs when particular characteristics (apart from the defined study variables) affect the

outcome of a study and researchers cannot account for them in the statistical analysis. Randomization

helps to decrease these biases. There are effective tools to minimize their role in OCER, such as

propensity scoring and instrumental variable analysis, and confounding is less likely in certain settings.

Dr. Gliklich used the example of prostate cancer treatment. Because proton beam therapy is only

offered in select medical centers, treatment choice is more closely related to geography than to patient

self-selection; outcomes with this therapy can therefore be more easily assessed using OCER with a

lower likelihood of bias.

RCTs are not always superior to OCER as both display a range of quality. The ability to judge the quality

and reliability of OCER is critical and multiple methods to do so are under development. The GRADE

(Grading of Recommendations Assessment, Development and Evaluation) Working Group is formulating

25

criteria with which to evaluate the design and quality parameters of various study types.10 Studies are

scored according to factors outlined in Figure 2, with poorly conducted RCTs receiving less weight and

well-executed registry studies gaining prominence.10

Figure 2: Criteria defined by GRADE working group to score the quality of evidence While RCTs start with high evidence and observational research with low, the grade can be recalculated according to the defined criteria.

SOURCE: Reprinted with permission from the BMJ Publishing

Group Ltd: Atkins D, et al. Grading quality of evidence and

strength of recommendations. Grade working group. BMJ

2004; 328:1490.

Several initiatives are also attempting to strengthen the design of OCER studies. In 2007, AHRQ developed a

guidance document entitled Registry for Evaluating Patient Outcomes: A User’s Guide.11 It was meant to

establish quality standards among registries and to provide a checklist of good and bad practices. While not

specific to CER, it effectively addressed many of the important issues. The Good Research for Comparative

Effectiveness (GRACE) Initiative, also started in 2007, seeks to enhance the quality of OCER by outlining

good practices.12 In addition to these activities, the statute that created PCORI charges its Methodology

Committee with developing a translation table to guide the selection of the most appropriate methods to

address different CER questions.

Oxaliplatin for Stage III Colon Cancer: Example of a Registry Study

Registries are being used effectively for OCER, but various challenges must be overcome. To illustrate

the issue, Deborah Schrag discussed her experience conducting a registry-based CER trial. She sought to

evaluate the comparative effectiveness of adding oxaliplatin to adjuvant chemotherapy for stage III

colon cancer. She defined for the audience the steps that she found critical to the success of her trial: 1)

identifying a priority question; 2) reviewing the efficacy data; 3) framing the question; 4) assessing if the

intervention is in use; 5) examining the trends in outcomes; 6) framing the analysis (PICOTS); 7)

assembling the data sources and obtaining approvals; 8) defining the cohorts; 9) ascertaining the

26

treatment groups; and 10) evaluating the similarity of the cohorts. She then described each in the

context of the trial.

The first three steps involved developing the research question. In Dr. Schrag’s case example, review of

the available data from earlier trials showed that adding oxaliplatin to a 5-fluorouracil (5-FU) backbone

resulted in superior efficacy as compared to 5-FU alone. There were, however, substantial tradeoffs in

terms of toxicities13 and it was unclear if the prior study results were representative of real world

practice. After reviewing the evidence, the research team engaged stakeholders to help frame the

analysis and to develop the research question. The goal of this inclusive process was to ensure that the

concerns and values of decision-makers would be adequately addressed by the resulting study.

After framing the question, an assessment of the study feasibility was needed to determine whether the

data contained in available registries were sufficient to answer the question. Dr. Schrag used data from

the Medicare- Surveillance, Epidemiology and End Results (SEER) database to evaluate patterns of care

and to ensure that oxaliplatin-based regimens were used widely enough to make conclusions. She also

examined trends in outcomes and identified possible covariates to include in the analysis. A trend was

seen toward better survival, providing positive initial feedback on the hypothesis. Following this

background work, Dr. Schrag framed the analysis using the AHRQ-specified PICOTS framework and the

GRACE principles.

Assembling data sources and obtaining approvals proved to be the most difficult and time-consuming

step in the study. These steps accounted for approximately 85% of the overall project time. A high

degree of political skill was required in dealing with the many relevant stakeholders, numerous IRB, and

many other individuals whose consent was required. Ultimately, the project included trial data from the

Accent Database; efficacy data from Medicare, the Cancer Care Outcomes Research and Surveillance

(CanCORS) Consortium, and the National Comprehensive Cancer Network (NCCN); and state-level data

from California and New York.

With data sources and approvals secured, the next step involved defining the cohorts and ensuring the

application of consistent criteria to each dataset, in order to maximize comparability and transparency.

This step required engagement with the curator of each dataset to understand the unique attributes of

each. The previously defined covariates were used to assess the similarity of cohorts as well as

heterogeneity across cohorts, using techniques such as propensity score matching.

The development of a statistical analysis plan appropriate to the specific question at hand is a critical

step in the design of any high-quality research study. Dr. Schrag used an array of methods to analyze the

data, including univariate and multivariate analyses, propensity score weighting, and inclusion/exclusion

of subgroups. Selection of analytic methods was followed by a priori specification of a sensitivity

analysis, considering factors such as time horizons, treatment assignments and outcomes.

Upon completion of the analysis, the final steps were to share the results with the stakeholders, gather

their responses, and then reanalyze the assumptions and results. In the end, the study revealed a

27

survival advantage that became apparent approximately two years after the initial treatment. The

benefit was small but statistically significant. Dr. Schrag admitted that “the stakeholders knew what they

were talking about when they wanted us to ask this question.” Nonetheless, the study accomplished an

important purpose in helping to distill the steps needed to complete a registry-based CER study in

oncology.

Reflecting on the project, Dr. Schrag stressed the importance of being able to leverage existing data. She

cautioned that the conduct of such research is difficult and time-consuming, leading some to question

the utility of large-scale, registry-based studies unless the process can be streamlined. Systems to

facilitate efficient data collection and analysis and to ensure a high level of quality while maintaining the

privacy of patients, could enhance the role of registry-based CER. Despite concerns, the promise of

OCER to increase the quantity and quality of available evidence is very high.

Development of Efficient Data Systems to Support CER

Electronic data systems provide an unprecedented research opportunity. If data quality, accessibility,

interoperability and governance issues are addressed, progress could be made toward implementing a

rapid learning health system in the US. Peter Yu outlined the steps required to allow the use of

electronic databases for CER. He described the electronic medical record as “an opportunity to redesign

how we approach the collection of data.” In an effort to make physicians comfortable with the transition

from paper-based to electronic capture of clinical data, vendors have tried to replicate the paper chart.

Data captured this way are idiosyncratic, consisting of personal notations, abbreviations and

inconsistent context which limit its usefulness and reliability.

Data interoperability (i.e., the ability to coordinate data from disparate and heterogeneous datasets) is a

key challenge in developing systems that can support CER. Interoperability requires changes in existing

information systems, such as standardization of data formats so that it can be easily pooled and

interpreted. The research community must also agree on the common data elements to be collected,

with standardized terminology and values for each element such as disease stage, patient comorbidities,

chemotherapy details and response criteria.

This work has already begun. The American Society of Clinical Oncology (ASCO) and the NCI have agreed

on many of the data elements that are important to oncology. The Certification Committee for Health

Information Technology (CCHIT), created by the Bush Administration with the mission of accelerating

the adoption of health IT, is currently conducting an assessment of the oncology certification process,

with a report expected in 2011. The IOM and the Office of the National Coordinator (ONC) also

identified three critical elements needed to allow progress in the field, two of which were addressed

through health reform legislation. These include requirements for the adoption of electronic health

records nationally and programs to enable semantic interoperability between systems. The final step

will be to define a way to aggregate and analyze data. Addressing issues of privacy, data ownership and

consent will be critical to success.

28

Dr. Yu concluded by outlining two approaches to issues of privacy and data ownership arising with

interoperable electronic data systems. In a cooperative group model, local data is extracted and

maintained in a centralized data repository. While this approach avoids many of the pitfalls of data

sharing, it is resource intensive, can result in poor data quality due to inconsistent methods of

extraction, and may lead to complexities with the release and sharing of information. The alternative

approach, a distributed data network, avoids redundant systems as the data is held, curated and

controlled through a local provider network. Security and privacy are maintained by provider

organizations but queries can be sent to a provider’s database. Dr. Yu advocated the latter, but

acknowledged that there is a lack of universal agreement on the best approach.

As electronic health records mature and interoperable, standardized data systems evolve, the

opportunity to conduct efficient and reliable observational studies will increase. Progress toward this

future state will require significant effort, funding and collaboration.

29

Discussion and Key Themes – Improving Clinical Evidence for Oncology

The following key themes were distilled from the presentations, panel discussions, and breakout

sessions, with input from the multi-stakeholder planning committee:

1. In oncology, there is an urgent need for an evidence development system which focuses

on the comparators of importance, outcomes of interest and the types of trials that

should be conducted. There has been an explosion in the number of available agents to

treat patients as well as an exponential rise in the number of trials assessing treatment

regimens. The research community must prioritize studies and methods that will lead to

improved clinical care.

2. Before beginning a CER study, its relevance and ability to answer the question at hand

must be established. Sponsors and research teams should be required to address two

questions: What is the probability that the study will definitively answer questions that

matter to patients? Will the study fill an evidence gap, such as those highlighted through

systematic reviews? This must be a stakeholder-driven process.

3. RCTs alone are not adequate to address the enormous evidence gaps in oncology. RCTs do

not always reflect real-world experience, as they are conducted in tightly controlled

environments. Because of their expense, complexity, resource-intensity, and timelines, they

can only address a limited number of questions. Observational studies can potentially fill

the gaps left by RCTs, but stakeholder skepticism regarding this less rigorous design must be

overcome. Defining research and evidentiary quality standards so as to ensure good practices

and reliability could help allay concerns.

4. Electronic health records should be designed to facilitate CER. Currently, the format and

content of electronic health records do not simplify or expedite the tasks entailed in CER.

The data captured in the current electronic health records are idiosyncratic, consisting of

personal notations, abbreviations and inconsistent organizational structures. To move

forward, researchers must define the data elements that need to be captured and

standardize the relevant terminology and values.

5. Conduct of large, registry-based studies will require a culture of collaboration. Oftentimes,

the rate-limiting step in registry-based CER revolved around assembling the data sources

and obtaining approvals. Greater efficiency and an overlying system to facilitate analysis is

needed.

30

Session 4: The Critical Challenge of Genomics and Personalized Medicine

Genomics could transform the delivery of medical care by allowing patients to receive only those agents

likely to provide benefit according to their unique profile. This vision is consistent with the goals of CER.

The potential benefits of genomically-based approaches to clinical care are widely acknowledged, but

important questions remain unanswered. For example, how will genomics be integrated into clinical trial

design without driving the costs of research prohibitively high? How should tissue be banked for future

study? When are biopsies appropriate in clinical trials? This session addressed these topics.

An Evolving Framework for Drug Development

The current approach to drug development involves identifying a molecule, demonstrating that it works,

conducting trials in humans and after establishing clinical activity, trying to identify which sorts of patients

are most likely to benefit. According to Neal Meropol, this approach is no longer viable and should be

replaced by a system in which drug development and diagnostic development proceed hand in hand. Dr.

Meropol also noted that the IOM identified the role of biomarkers in oncology as one of the top quartile

areas in need of CER. Stephen Eck reinforced this point with the observation that many treatments

delivered today do not benefit patients to the extent one might hope because patient populations are not

uniform and a “one size fits all” approach fails to account for these differences. Based on genetic variations,

patients are likely to respond differently to the same drug. 14

In addition, it has taken decades to improve treatment options for many types of cancer and a better

understanding of genetics may expedite this process. Dr. Eck described how treatments evolve. When a

new disease is discovered, it is initially treated with “non-technology” in which patients receive only

supportive care and symptom control. At the outset of his fellowship, treatment for multiple myeloma was

largely supportive. Next, “half-way technologies” are introduced. These new and exciting interventions tend

to be overused, and entail uncertainty about their best use. Though they postpone death, their mechanism

of action is poorly understood and their benefit is limited. Eventually “high technology,” the ultimate

objective, is developed. With these treatments, the effect size can be large and patients may be cured. In

this stage, we know which treatments should be provided to a given individual. Different cancers rest at

different positions along this developmental trajectory. In kidney cancer, tremendous progress has been

seen over the past decade, yet we have a limited understanding of the order in which to use the available

treatments and it is difficult to differentiate between indolent and aggressive disease. This disease remains

at the halfway point in the developmental framework. Hopefully, better molecular approaches will

accelerate this change.

31

New Research Strategies to Advance Drug Development for Genomic Medicine

Genomic research will require new methods that are capable of handling the intrinsic complexity of the

field. Dr. Meropol noted that it is often difficult to answer CER questions in oncology with only phase III

trials. It is also challenging to find financial support for the development of tests that identify which patients

will benefit from a specific drug, as industry does not want to limit the potential pool of candidates for a

given treatment.

Interpreting the results of studies is also difficult. Scott Ramsey provided an example of the complexity

clinicians face when applying the results of genomic studies to practice. In a recent study, researchers

claimed that they had identified a group of genetic sequences which predicted an increased risk for prostate

cancer with an odds ratio of 9.46.15 While the claim sounded impressive, after accounting for age and family

history, the predictive power of the test only identified a 0.02% increase in the risk of cancer. Hence the test

is not useful. This illustrates the importance of sophisticated interpretation of genomic studies.

The use of pooled analyses, decision modeling and other emerging approaches can help make sense of this

complexity and will move treatments in various diseases toward the “high technology” state. Two relevant

observations are that (1) it is often difficult to gather enough information in a single study to reach a sound

conclusion, and (2) there is increasing availability of tissue with excellent clinical annotation with which to

study various patient populations. Dr. Meropol used the example of Rat Sarcoma (RAS) and Rapidly

Accelerated Fibrosarcoma (RAF) mutations to elucidate the benefits of pooling studies and leveraging tissue

samples across repositories. RAS mutations predict resistance to treatment with Epidermal Growth Factor

Receptor (EGFR) inhibitors. But a downstream gene, RAF, also plays a role. The question was asked, should

patients with a wild-type RAS but a mutated RAF gene still get EGFR inhibitor therapy? RAS is mutated in

40% of colon cancer patients, while RAF is only abnormal in 8% of patients. To look at RAF is difficult enough

in a prospective RCT; an even larger study is needed to look at RAF in combination with RAS. Yet, one can

take tumor samples and outcome data from multiple trials, study genetic sequences across populations and

then pool the results. Such a study has been conducted and the results demonstrated that a RAF gene

mutation does not confer complete resistance to EGFR inhibitors.16 A more recent study further shows that

the location of the mutation on a RAS gene might affect outcomes with an EGFR inhibitor.17

Incorporation of Patient Values into Genomic CER

Patient decision-making is a critical consideration in genomically-based personalized medicine and thus, in

CER, meant to inform it. Dr. Meropol raised several real-world questions, such as “How do we develop

evidence and build decision models that take into account a patient’s goals, values, beliefs and

expectations?” A continuum exists of how patients value different factors such as survival and quality of

life. According to Dr. Meropol, patients “weigh decisions based on side effects, cost, inconvenience of

therapy and each patient may have a different end point that matters to them. Is it living two months to

see my daughter get married?” The traditional focus purely on survival needs to be modified to factor in the

relevant considerations.

32

Dr. Meropol discussed stage II colon cancer as an example. While many stage II colon cancer patients can

be cured with surgery, about 20% will relapse. Adjuvant chemotherapy may reduce the risk of relapse, but

only for 3% to 4% of patients. This relatively small benefit does not justify universal treatment. 18 Who

should receive the six-month course of chemotherapy? Dr. Meropol and his collaborators are developing a

genomic-based recurrence score to help identify who will benefit and who will not. A decision model can

also be built that takes into account more than just overall survival, helping to account for a patient’s

perceived gains and losses. CER will play an important role in supporting this type of complex analysis,

modeling, and personalized clinical approach.

Critical Role of Genetic Diagnostics in Genomic Medicine and CER

While genetic tools for widespread screening are not yet a reality, prognostic and predictive tests are

being introduced with increasing frequency. Dr. Ramsey provided an example in colorectal cancer in

which the sub-population of patients with a specific genetic marker (KRAS wild-type) are more likely to

benefit from cetuximab, an EGFR inhibitor, than those with a mutation in the gene.19 This finding allows

targeted treatment of wild-type patients, increasing the drug’s efficacy and the associated clinical

outcomes. Despite isolated areas of progress such as this, the field is in a state of flux. Thousands of

tests will come to the market over the next decade and it is unclear who will drive their uptake, how

clinicians will know when to use them and how insurers will decide when to pay.

In genetic testing, an established framework is used to help interpret the quality of available evidence

for a given test. Dr. Ramsey summarized the key attributes of the framework:

Analytic validity refers to how accurately and reliably a test reveals the presence of a specific

gene or trait. It is the “low-hanging fruit” as it is the minimum that should be expected of any

test.

Clinical validity refers to how often a test estimates a given outcome. For instance, if a test is

intended to provide prognostic information about prostate cancer, how often does it do so

accurately?

Clinical utility denotes how likely a test is to significantly change or improve a patient’s

outcomes compared to no testing. If a genetic test predicts response to chemotherapy, how

often does using it to guide therapy improve overall survival?

Value is the ultimate goal. This does not refer only to financial savings, but also to how often a

test provides value to a patient according to his/her priorities by improving quality of life or

other outcomes of interest.

33

Currently, diagnostic tests are often deficient in one or more of these criteria and there is limited

understanding of the quality of evidence underlying the tests used in practice. In a review conducted by

Regence Blue Cross and Blue Shield of 96,500 submissions for reimbursement of genetic tests, the total

cost to the insurer was $85,000,000 – nearly $1,000 per test.20 Despite these enormous costs, the

review found evidence of medical necessity for only five of the tests as shown in Table 4. "Why,” he

asked, “is it that we have so many tests that are developed and ultimately reimbursed without any real

understanding of their value?”

Table 4: Genetic tests with established medical necessity according to Regence Blue Cross and Blue Shield21

Test Indication

KRAS Predict response to cetuximab (Erbitux) & panitumumab (Vectibix) in

patients with metastatic colorectal cancer

BRCA1 & BRCA2 Detects inherited susceptibility for breast/ovarian cancer

Oncotype DX Determines breast cancer recurrence risk

APC, MYH, MMR Detects inherited susceptibility for FAP, Lynch Syndrome

MEN 2A, MEN 2B Detects inherited susceptibility for medullary thyroid cancer

The answer lies, in part, in the regulatory framework and its lack of rigor. Tests are often developed in ways

that do not answer questions that are relevant to patients and physicians. Once a promising genetic test is

discovered in an academic laboratory, it is likely to be licensed to biotechnology companies for clinical

validation. In the absence of a regulatory requirement to demonstrate analytic or clinical validity prior to

marketing, shortcuts are often taken to minimize expenses. Published results are likely to deal only with the

discovery phase, with marketing following soon after publication, as shown by the blue arrow in Figure 3.

Tests often bypass the FDA and are marketed under the Clinical Laboratory Improvement Amendments

(CLIA), which imposes minimal requirements. As a result, the clinical validity and utility of genetic tests

often remain unclear.

Figure 3: The current model of cancer genomic test development, including the relevant stakeholders, outcome at each stage, and limitations

SOURCE: Scott Ramsey presentation (November 1, 2010).

Reprinted with permission from Scott Ramsey.

34

Co-development of drugs and diagnostic tests is essential to the advancement of our understanding of

novel therapeutics. Dr. Shak discussed the co-development of HER2 testing with trastuzumab as an

example. Trastuzumab’s development involved investigators in fifteen countries, many of whom were

skeptical about the feasibility of drug and test co-development. Advocates helped design the trials and, in a

subsequent evaluation by statisticians, it was seen that if HER2 testing had not been conducted to better

understand response, the phase III trials would have been entirely negative. In Dr. Shak’s opinion, “this very

effective drug would never have been seen to have any activity…in the absence of the selection of

appropriate patients for testing.” This, he feels, is an important lesson for investigators. The results of the

correct test can greatly enhance drug effectiveness and utility. The encouraging example of trastuzumab,

however, does not yet represent the norm.

Dr. Shak also highlighted the Oncotype DX recurrence score, marketed by his employer, Genomic Health.

Unlike HER2, it was developed independent of a specific drug but is intended to help guide therapy. A

landmark study in the 1990s established the benefit of chemotherapy plus hormonal therapy, compared to

hormonal therapy alone, in early stage breast cancer.22 Because 85% of women were cured with surgery

and hormonal therapy alone, it was unclear whether it was necessary to expose all early-stage patients to

the toxicity of chemotherapy to benefit a few. Researchers heard from physicians, advocates, and patients

that guidance was needed and they set out to develop a test to fit this very specific purpose. Oncotype DX

helped to answer the question. Dr. Shak pointed out that one must be prepared to do multiple studies in

order to be convinced that a test is doing what it claims. Despite thirteen prior studies, OncoType DX is still

being evaluated in the TAILORx trial to define the benefit of chemotherapy in patients who fall in the

moderate risk category. 8

According to Dr. Eck, in the context of new drug development in a competitive market, CER could be used to

distinguish between true game changers and modest or incremental improvements and to clearly

distinguish economic value. Increased market pressures, with higher standards for market survival, will

encourage drug developers to shelve those agents that are marginally important and return to the lab

earlier in order to produce something superior. Dr. Eck also predicted an increase in head-to-head

comparisons because, with so many different mechanisms and side effects, the risks and benefits will need

to be truly understood. He cautioned, however, that there is a real potential for CER to require many more

studies and substantial additional investment in order to get drugs to market, thus adversely impacting

innovation and limiting the therapeutic options available to patients.

Collaboration to Advance Genomic and Personalized Medicine through CER

Advancement of genomic and personalized medicine through CER will require partnerships between test

developers, patients, regulators and clinicians in order to ensure that the right questions are being asked

and that the development proceeds in a coordinated way. Test developers will continue to be involved in

discovery, verification and standardization of assays. The research community must play a proactive role in

identifying promising candidate tests and in acquiring biomedical partners. Researchers must guide the

35

design and implementation of studies since the incentives of test developers are not always aligned with

those of patients. As shown in Figure 4, the health care system is a critical component as the source of both

patients and outcomes data. With cooperation among test developers, researcher groups and the health

care system, drug and test development will be far more likely to succeed.

Figure 4: Test developers, research groups and the health care systems must all play specific roles in order to improve the quality of tests in development.

SOURCE: Scott Ramsey presentation (November 1, 2010). Reprinted with permission from Scott Ramsey.

36

Discussion and Key Themes – The Critical Role of Genomics and Personalized Medicine

The following key themes were distilled from the presentations, panel discussions, and

breakout sessions, with input from the multi-stakeholder planning committee:

1. In oncology, CER must develop robust methods that address patient- and population-level

variations in treatment effects, values and access. Oncology, more than any other field of

medicine, relies on targeted therapies. The appropriate application of these agents depends

upon understanding differences in treatment effects based on patient characteristics. Fully

addressing these variations is expensive and difficult using the research methods currently

available to us. There remains concern that the focus will be on average benefits across large

patient populations instead of on patient-level differences. Creative approaches and vigilance

will be needed in order to ensure success.

2. Methods must be developed and implemented to efficiently deal with complex genetic

variation and the breadth of treatment options. Many questions cannot be addressed through

RCTs and large-scale surveillance studies. New approaches, such as pooled data analyses and

decision modeling, must be developed to address the extensive evidence gaps in oncology with

scientific and analytic rigor.

3. Evidence on the clinical utility of genomic tests is lacking despite its critical role in CER. Most

pharmacogenomic tests are not subject to the FDA approval process, making evidence of

clinical utility, though crucial to the appropriate use of these tests, generally lacking. Co-

development of drugs and diagnostic tests is essential, as the development of better drugs will be

facilitated by a better understanding of their effects on patients with specific genetic variations.

Co-development will necessitate incentives or requirements for the conduct of such research.

37

Session 5: Translating Evidence into Practice

Clinical Guidelines

Bottlenecks in the translation of evidence into clinical practice remain one of the greatest impediments

to evidence-based medicine. While the principal focus of CER is on the development of new evidence to

inform treatment decisions, it is critical that the CER enterprise also address translation so that findings

can impact clinical choices. Session 5 outlined a range of initiatives for developing this field including

activities in the areas of clinical practice guidelines, quality initiatives, learning healthcare systems and

engagement of community practices.

Clinical Practice Guidelines

Clinical practice guidelines serve as a vehicle for translating evidence into practice. Ethan Basch

discussed ways CER could enhance the impact of guidelines. Guidelines are often based on expert

opinion and the best available evidence: however, the evidence is often inconclusive. Over time, CER

could be used to build a more complete and robust evidence base through iterative and mutually

informative processes. Those who conduct reviews and develop guidelines can identify gaps in

knowledge and prioritize research questions, while researchers provide new evidence to inform reviews

and guidelines. By directly engaging clinical practices in this iterative process, the real-world nuances of

clinical practice can guide the prioritization of research topics. The vision of a continuous loop of

evidence generation, guideline update, practice improvement and research prioritization will lead to

improved patient care.

A number of examples of guideline development processes were discussed, with particular focus placed

on the speakers’ experiences with ASCO and NCCN. ASCO initially developed guidelines in 1994, focusing

on questions such as the use of chemotherapy in colon cancer, the use of anti-emetics during treatment

and the role of biomarkers. ASCO disseminates its findings through several outlets including The Journal

of Clinical Oncology, The Journal of Oncology Practice and web-based tools. Its goals are to provide

reliable recommendations and to disseminate knowledge in order to influence clinical practice,

formularies, and coverage. More than half of the top 20 downloads from The Journal of Clinical

Oncology website are ASCO guidelines, demonstrating their popularity.

Dr. Basch stressed the importance of establishing and adhering to evidentiary standards when

developing guidelines, stating that this rigor is equal in importance to the generation of evidence itself.

To him who devotes his life to science, nothing can give more happiness

than increasing the number of discoveries, but his cup of joy is full when

the results of his studies immediately find practical implications.

Louis Pasteur

38

Consistent standards allow reliability across evidence development groups and increased confidence in

their recommendations. ASCO’s process for guideline development relies on a systematic review of the

literature followed by a structured consensus process as outlined in its methodology manual—this

minimizes the sole reliance on expert opinion and makes potential conflicts of interest transparent. The

final step is an extensive peer review process.

Guidelines play a pivotal role in encouraging the appropriate use of tests and treatments, thereby

optimizing the use of scarce resources. By allowing only high quality evidence to influence guidelines, it

puts pressure on academia and industry to improve the processes of evidence generation (following the

Summit, the Institute of Medicine issued a report with recommendations for ensuring the quality and

integrity of Guidelines).23

William McGivney discussed the NCCN’s use of its compendia and guidelines to promote the

dissemination of clinical evidence. The NCCN draws from 21 cancer centers across the US to form 44

multi-disciplinary panels that develop treatment guidelines. Between 100,000 and 140,000 people visit

the NCCN website every month to access its recommendations and over 200,000 people do so per year.

A Genentech report showed that 84% of US oncologists use NCCN guidelines in their practice, while 60%

utilize ASCO guidelines, 27% internal guidelines, and 8% payer guidance. 24 In addition, payers such as

United Healthcare, Aetna, and CMS use them in making coverage determinations. These statistics

demonstrate the importance that the clinical community places on guidelines.

Quality Initiatives

Quality initiatives provide a mechanism for improving care while also identifying research questions

through their practice evaluations. As an example, Dr. Basch discussed the Quality Oncology Practice

Initiative (QOPI), which consists of a network of 600 oncology practices serving 900 sites. QOPI

participants conduct chart reviews to determine provider and organizational adherence to as many as

90 quality measures. Data are compiled and the results are used to understand practice patterns,

identify areas of concern and assess guideline adherence, with the ultimate goal of improving care and

addressing gaps in knowledge. Prior identification of deficiencies in three areas— infertility,

chemotherapy administration within two weeks of death and end of life discussions—has led to

education and research efforts to address the gaps. As another example, after ASCO released its

guidelines for the treatment of non-small cell lung cancer, QOPI helped to identify sites that deviated

from the updated guidelines and worked to understand the underlying reasons for non-compliance. Dr.

McGivney also discussed the NCCN collaboration with Ingenix, a large health informatics company, to

translate NCCN guidelines into quality rules that can be applied across tumor types. These systems

encourage adherence to evidence-based practices and help to ensure the delivery of high-quality care.

They must become more robust.

39

Learning Healthcare Systems

The idea of a learning healthcare system is garnering attention and widespread acceptance as a model

for continuous, iterative evidence generation and clinical practice improvement.25,26 The concept of the

learning healthcare system is motivated by three fundamental goals: (1) to generate high-quality

evidence and apply it to individual patients (i.e., to support personalized medicine); (2) to make

scientific discovery “a natural outgrowth of patient care” by using clinical experience to assess

effectiveness and generate hypotheses; and (3) to improve the value and efficiency of healthcare.25,26 In

practical terms, rapid learning healthcare describes a system in which data that are collected during

routine patient care are fed into an ever-growing databank and the system “learns” by frequent analysis

of the captured data. The new insights are used to inform subsequent care.27 CER, as research that

evaluates the relative benefits of available treatments under real-world conditions, is by definition

integral to learning healthcare systems.

A learning healthcare system and the CER enterprise that supports it requires a continuous feedback

loop (Figure 5). New evidence is used to develop or update guidelines which, in turn, improve and

update clinical practice. Multiple feedback loops at each step return relevant information to clinicians

and researchers, fueling further inquiry, discovery and translation, ultimately improving both clinical

outcomes and the functioning of the research enterprise. This continuous process defines rapid learning

healthcare. Dr. Basch highlighted the central importance of patient-reported outcomes (PROs) in rapid

learning systems. PROs add the patient’s perspective to the processes of question formulation,

prioritization and evaluation. Likewise, electronic health records that directly collect data about quality

from individual practices can be used to iteratively assess practice quality, identify research targets and

drive evidence generation.

Figure 5: Evidence generation in a learning healthcare system

Abbreviations: PRO, patient reported outcome; EHR, electronic

health record; CER,

comparative effectiveness research

SOURCE: Ethan Basch presentation (November 2 , 2010).

Reprinted with permission from Ethan Basch.

40

Engaging Community Practice

Eighty-five percent of cancer care is delivered at community-based clinics and hospitals; therefore, CER

that is applicable must include community settings. However, these institutions often have limited

research infrastructure and experience. Research in community settings can be complicated as care is

often fragmented, adherence to evidence-based guidelines is challenging and data on quality is lacking.

Steven Clauser described an effort to improve research capacity in community-based oncology through

the NCI Community Cancer Centers Program (NCCCP). This program engages community oncology

practices that are not affiliated with academic medical centers in order to facilitate the integration of

evidence and guidelines into practice, while also encouraging research participation. As Dr. Clauser

explained, the purpose of the NCCCP is to “try to build a culture of a learning environment, to create a

network among these centers and also to begin to develop research platforms that could be used by our

investigator community to try to have better access to community research subjects for studies in a much

more efficient and cost-effective manner.”

The NCCCP program has grown quickly, increasing from 14 hospitals at the beginning of 2010 to 30 by

November of 2010. The affiliated institutions care for 58,000 new cancer patients each year and cover a

population of 23 million people across 22 states. The program focuses on building research

infrastructure (clinical trial capacity, information technology, and biospecimen repositories) while also

improving the availability and quality of care (advocacy, survivorship, quality initiatives and health

disparities). It leverages NCI resources by requiring participating sites to match any resources provided

by the NCI. To date, member institutions have provided $2.65 for every NCI dollar invested. Prior to

enrolling, prospective members must ensure the direct involvement of senior hospital leadership and

agree to undergo a continuous, rigorous evaluation to ensure that goals are being met and that quality

is improving.

Community practices will face challenges as they prepare for greater research involvement,. One of the

biggest is the need for data from their practices as their current information systems are typically

inconsistent and fragmented. Incentives to develop data capacities and to engage in CER may be limited

as private practitioners often lack financial resources and time to participate. In light of the challenges,

continuing to build community-level involvement will depend on (1) provision of incentives for

community physicians to participate, (2) creation of better alignment between the needs and interests

of community practices and the NCI research goals, and (3) definition of community practice as a central

focus of CER development in oncology. The NCCCP is a model for this work.

The Economic Imperative

Economic pressures make issues of timeliness in evidence generation and generalizability of research

results more relevant today than ever before. Spending in oncology lags behind only cardiology and

orthopedics and is rising dramatically – from $27 billion in 1990 to $90 billion in 2008.28 Robert

41

McDonough presented data on the expected increase in the costs of oncology drugs and noted that the

fee-for-service system in the US does not encourage adherence to evidence-based guidelines.29 He

envisions a system in which pathways guide patients along the entire spectrum of oncology care, moving

away from a revenue stream based on the number of agents given or procedures performed to a value-

based contractual revenue stream. The traditional role of the insurer—enforcing pre-certification and

conducting utilization review—will not lead to such a system. It could instead be achieved through a

closer association with practices, specifically tracking pathway compliance and altering drug

reimbursement in order to drive the selection of the most cost-effective medications. This model would

rely on accountability, total patient care to avoid unnecessary complications, decision support and care

coordination to encourage evidence-based delivery. Robust evidence upon which to base guidelines is

necessary, with CER at the core of such a system. Dr. McDonough described a collaboration between

Aetna and US Oncology as an example of this type of pathway. He stated that implementing pathways in

US Oncology practices led to a cost savings of 35% over the course of a year while maintaining equivalent

patient outcomes.

42

Discussion and Key Themes – Translating Evidence into Practice

The following key themes were distilled from the presentations, panel discussions, and breakout

sessions, with input from the multi-stakeholder planning committee:

1. Clinical practice guidelines and quality initiatives are extremely valuable mechanisms

for translating evidence into practice. At present, guidelines rely as much on expert

opinion as empirical evidence. As new CER studies emerge, guidelines will become more

robust and will represent a key platform by which emerging evidence can be integrated

into practice. Likewise, quality initiatives will gain increasing traction.

2. In order to effectively translate the results of CER into improvements in clinical care,

the gulf between research and practice must be bridged. Eighty-five percent of cancer

patients are treated in community settings, thus engaging community practices in

research is critically important. The conduct of CER in community-based settings entails

several challenges, such as wide variations in practices, limited systematic data collection

and a lack of financial resources and incentives to participate.

3. Developing effective methods of translation will require a substantial investment in

infrastructure. Funding for this infrastructure development could come from increased

federal investment, re-alignment of payment incentives to support electronic health

records and support from private payers. Because of its limited funding, PCORI should

focus on developing standards, while CMS and other stakeholders could leverage their

resources to enhance IT systems that will function as the needed infrastructure for CER.

4. In order to promote better translation of evidence, private payers must move beyond

their traditional functions of pre-certification and utilization reviews. New roles for

third-party payers and new pathways for advancing research evidence into practice,

include new payment systems and new approaches to coordination with patients and

clinicians that guide the selection of cost-effective treatment choices.

43

Keynote 3: Reimbursement Reform, Dr. Peter Bach

[1] Abelson R. Insurers test new cancer pay systems. New York Times 2010 October 19, 2010.

[2] Bach PB, Mirkin JN, Luke JJ. Episode-based payment for cancer care: a proposed pilot for Medicare. Health affairs

2011;30:500-9.Session 6: Value and Costs in Oncology.

[3] Jacobson M, Earle CC, Price M, Newhouse JP. How Medicare’s payment cuts for cancer chemotherapy drugs changed

patterns of treatment. Health affairs 2010; 29:1391-9.

Keynote 3 Highlights

The introductory price of every new drug approved by the Food and Drug Administration (FDA)

since 1965, the beginning of Medicare, has been substantially higher than the one preceding it in

the same therapeutic category. This helps to explain the continued rise in the cost of cancer care.

The fee for service system, which ties profitability to utilization, leads to overuse of the newest

agents, allowing profitability to influence treatment choice.

A New York Times article listed the average cost per month of eight lung cancer treatment

regimens [1]. Per NCCN guidelines and available research on the topic, all of the drugs are

considered to have equivalent efficacy, making them largely interchangeable in practice. Despite

these facts, cost does not play a role in choosing between the options.

An ideal solution is an episode-based payment system or global fee bundling [2]. Patient costs

would be reimbursed at a set rate, depending on a patient’s diagnosis. By providing a global fee for

a treatment course, oncologists transition from discretionary spenders to prudent purchasers who

care about how the dollars they receive are marshaled out.

The flexibility within the system would allow clinicians to choose the right treatment for a given

clinical scenario and would require them to meet certain quality measurements to avoid

decreasing care outright.

Some may not believe the underlying argument that doctors respond to financial incentives, but

the data clearly shows otherwise. A Health Affairs article by Jacobson et al assessed the effect of

payment changes that occurred under the Medicare Modernization Act on treatment choices and

found that there was a significant change in how non-small cell lung cancer patients were cared

for, almost to the day that the reimbursement changed.

44

In establishing the framework for PCORI, Congress placed some limits on the Institute’s ability to

consider the cost and value of care in its work. For example, PCORI cannot use cost per quality-adjusted

life year (QALY) as the basis for treatment evaluations. Nonetheless, a large driver of the national

interest in CER comes in response to the rising costs of care so both must ultimately be a part of the

discussion. The session’s moderator, Sarah Garner, opened the session by discussing the range of

considerations inherent in defining value, which she depicted in Figure 6.

Figure 6: Some of the many considerations that factor into patients’ definitions of value

SOURCE: Sarah Garner presentation (November 2, 2010). Reprinted with permission from Sarah Garner.

Patients’ Role in Value Decisions

Envisioned as part of a national endeavor to “get the right care to the right patient at the right time,”

CER needs to incorporate patients into the discussions of value. CER is not meant to determine if a

treatment is appropriate based on cost alone nor is it intended to generate evidence to support cut-offs

for what will or will not be reimbursed. Its purpose, instead, is to help improve the value of care

delivered. Louis Jacques gave a number of examples of how his patients described their treatments,

which provided him with insights into the diverse ways in which they valued their care. Some are willing

to accept significant risks for almost any degree of benefit, while others state that “if I had known it was

going to turn out like this, I would have just stayed home.” Without engaging patients, one cannot truly

understand their perspective. Dr. Jacques noted that physicians, researchers and policy makers must not

attempt to guess patients’ values, in effect substituting their own judgments for those of patients. Instead,

learning how to capture and incorporate value requires more research into novel methodologies and a

continuing public conversation.

45

The Role of Cost

Despite the many values considered, cost cannot be ignored. The literature on the cost of care is quickly

growing and physicians realize its importance. Peter Neumann presented data from surveys assessing

the role of cost in the decisions of practicing oncologists.30 These surveys showed that 77.5% of

oncologists believe that “every patient should have access to effective cancer treatments regardless of

costs.” However, individuals’ responses were inconsistent. For example, respondents indicated that a

“reasonable definition of good value for money” equated to between $50,000 and $100,000 per QALY

gained, but they also stated that “a new cancer drug which costs $70,000 should provide an average

minimum increase in life expectancy” of two to four months, equivalent to $280,000 per QALY gained.

Similar findings were observed in a more recent survey of 1,400 oncologists.31 67% agreed that patients

should have unlimited access to effective care, yet 56% state that the “costs of new cancer drugs

currently influence my treatment recommendations” and 84% agreed that a “patient’s out-of-pocket

costs influence my treatment recommendations.”31 Regardless of their acknowledgment that costs

influence treatment decisions, when asked “how often do you discuss the costs of new cancer

treatment with your patients,” only 35.6% answered “frequently” and 6.6% “always.” Less than half

reported feeling prepared to “interpret and use cost-effectiveness information in treatment decisions.”

When asked directly if there should be more use of cost effectiveness data in coverage and payments,

80% said “yes.”

Over the last few years, the number of published cost analyses has increased significantly, with 12% to

15% pertaining to oncology. Peter Neumann maintains a registry of 2,500 such analyses which can be

accessed at www.cearegistry.org. The treatments included in the registry appear to have a wide range

of “value.” Thirty-five percent of studies demonstrated an incremental cost-effectiveness ratio of less

than $20,000, while one in ten increase costs and worsen outcomes.32

Despite the increase in research on the topic of cost-effectiveness and interest expressed by oncologists,

translating information in such studies into improvements in clinical care is difficult. Physicians are not

trained to communicate the results of health analyses and most are not prepared to integrate them into

their practice. The information has not been widely used to make decisions about insurance coverage

or resource allocation either. While private payers may make internal decisions based on cost analyses,

the PPACA pushes decision-making in the opposite direction, prohibiting PCORI from using certain types

of cost information. Dr. Neumann outlined possible mechanisms to address the rising costs of

healthcare, including cuts in entitlements, delivery reform and payment reform. However, cost alone

should not drive care as value is an integral part of the evaluation.

46

Developing Costs and Value Information

Researchers face several challenges in quantifying costs and effectiveness. Steven Pearson outlined a

range of questions on the topic. Is effectiveness measured in terms of survival, surrogate outcome

measures, quality of life or events averted? How do we assess the meaning of time and relative health

to those who know they are nearing the end of life? Dr. Pearson stressed the need for explicit and

transparent elucidation of these values and suggested that a possible role for PCORI is to help define the

methods by which clinicians can elicit and incorporate preferences.

Unit costs and utilization rates represent the basic building blocks of CER analyses, however, they can be

difficult to quantify. The first challenge is to find reliable cost data. The second is to use these numbers

to express value in ways that make sense to patients and other stakeholders. While the cost literature

often refers to cost utility metrics, such as cost per QALY, these metrics may not be easily understood or

appropriately interpreted by patients and physicians. It may be easier for diverse stakeholders to

understand costs if they are reported using cost consequences such as “cost per adverse event

avoided.” However, because these sorts of costs are not always related to a final effectiveness measure,

they do not fully address the payers’ need to quantify the budget impact of a treatment. Dr. Pearson

illustrated this point using cholesterol lowering agents which, while cost-effective, become very

expensive when distributed to a large proportion of the population.

Even for a given test, cost-effectiveness evaluations by different researchers or in various settings can

lead to widely divergent conclusions. Dr. Pearson illustrated this using the example of colorectal cancer

screening. The Institute for Clinical and Economic Review (ICER) conducted an evaluation for the

Washington State Healthcare Authority and found that colonoscopies and CT colonography (CTC) were

equally effective. Since the unit cost for the CTC procedure was twice as high as that of colonoscopy in

Washington, coverage of CTC was denied in that state. By contrast, in Wisconsin, the same CTC

procedure was offered at one third the cost of colonoscopies, making it a high-value service that was

ultimately covered. Clearly, the definition of value is not inherent to a procedure, but emerges as a

product of numerous factors including costs, outcomes, patient preferences and other stakeholder

perspectives.

CER and Value-Based Pricing

Given concern among federal officials and the general public that CER will be used only to save money,

in parallel with the recognition that research is needed to help determine where costs can be trimmed

while still optimizing outcomes, the relationship between cost and value in CER must be more clearly

defined. Dr. Pearson suggested that CER be viewed, not as a method for determining whether or not to

cover a treatment, but rather as a method of generating evidence that can be used to guide pricing

policy. Decision-making related to coverage and payment rates for Medicare have historically been

47

distinct processes. In general, coverage is first decided upon, and then reimbursement rates are set.

This approach can result in high prices for newly covered drugs, despite a lack of evidence of increased

efficacy over the prior standard of care. The US currently lacks a method for relating price and

reimbursement to evidence demonstrating higher value. Dr. Pearson highlighted a number of recent

interventions such as proton beam therapy, erythropoiesis-stimulating drugs and off-label prescribing

that illustrate this point.

Several approaches could be used to embed calculations of value into determinations of

reimbursement, including global payments, episode-based payments, and least cost alternatives. Dr.

Pearson discussed a novel process in which available evidence drives pricing and coverage decisions.33

At the time when reimbursement is approved for a given treatment, the comparative effectiveness of

that intervention would be a primary factor in determining pricing. Lower pricing would be assigned to

a treatment with limited, equivocal or lower-quality evidence until new evidence is generated and

pricing is re-evaluated.

Assessments of value, though always difficult to conduct, are particularly challenging in a complex

interventional area such as oncology. Conducting research that evaluates value, however, is essential in

order to develop systems of care that will ultimately provide the greatest possible healthcare value to

patients and to the public.

48

Discussion and Key Themes – Value and Costs in Oncology

The following key themes were distilled from the presentations, panel discussions, and

breakout sessions, with input from the multi-stakeholder planning committee:

1. It is important for researchers to gain a better understanding of patients’ and

oncologists’ perspectives on cost and value. Congress limited consideration under

PCORI. General consensus holds that the value of a treatment depends on more than

its cost, though costs cannot be ignored. When attempting to assess or define value,

physicians, researchers, payers and policy-makers must be careful not to substitute their

own judgments for those of patients.

2. Methods for understanding cost and value must be further developed. Learning how

to capture and incorporate patient values, given the complexity of clinical practice,

requires the development of new methods and a continuing public conversation. Research

should incorporate nuanced metrics of patient value that include patient-reported

outcomes and out-of-pocket costs. Better systems must be developed to consistently

capture reliable data on treatment costs.

49

References 1. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President

and Congress. Washington, DC: US Department of Health and Human Services; 2009 June 30. 2. Barry MJ. The prostate cancer treatment bazaar: Comment on "Physician visits prior to

treatment for clinically localized prostate cancer." Arch Intern Med. 2010;170:450-2. 3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine:

What it is and what it isn't. 1996. Brit Med J. 2007;455:3-5. 4. Patient Protection and Affordable Care Act, H.R. 3590, 111th Cong., 1st Sess. (2010). 5. American Recovery and Reinvestment Act In, H.R. 1, 111th Cong., 1st Sess. (2009). 6. Ip S, Patel K, Obadan N, Chung M, Bannuru R, Lau J. Comparative evaluation of radiation

treatments for clinically localized prostate cancer: An update. Rockville (MD): US Department of Health and Human Services, Agency for Healthcare Research and Quality. 2010. Technology Assessment.

7. Abernethy AP, Raman G, Balk EM, et al. Systematic review: Reliability of compendia methods for off-label oncology indications. Ann Int Med. 2009;150:336-43.

8. Sparano JA. TAILORx: Trial assigning individualized options for treatment (Rx). Clin Breast Cancer. 2006;7:347-50.

9. Nice: Guideline development methods. Chapter 7: Reviewing and grading the evidence. London (UK): National Institute for Health and Clinical Excellence. 2004.

10. Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. Brit Med J. 2004;328:1490.

11. Gliklich RE, Dreyer NA, eds. Registries for evaluating patient outcomes: A user’s guide. Cambridge (MA): Outcomes Sciences, Inc. (d/b/a Outcome); 2007. Contract No.: HHSA29020050035I_TO1. Sponsored by the Agency for Healthcare Research and Quality.

12. Good ReseArch for Comparative Effectiveness. http://www.graceprinciples.org/. 13. Andre T, Boni C, Mounedji-Boudiaf L, et al. Oxaliplatin, Fluorouracil, and Leucovorin as adjuvant

treatment for colon cancer. New Engl J Med. 2004;350:2343-51. 14. Bardelli A, Siena S. Molecular mechanisms of resistance to cetuximab and panitumumab in

colorectal cancer. J Clin Oncol. 2010;28:1254-61. 15. Zheng SL, Sun J, Wiklund F, et al. Cumulative association of five genetic variants with prostate

cancer. New Engl J Med. 2008;358:910-9. 16. Laurent-Puig P, Cayre A, Manceau G, et al. Analysis of PTEN, BRAF, and EGFR status in

determining benefit from cetuximab therapy in wild-type KRAS metastatic colon cancer. J Clin Oncol. 2009;27:5924-30.

17. De Roock W, Jonker DJ, Di Nicolantonio F, et al. Association of KRAS p.G13D mutation with outcome in patients with chemotherapy-refractory metastatic colorectal cancer treated with cetuximab. JAMA. 2010;304:1812-20.

18. Figueredo A, Coombes M, Mukherjee S. Adjuvant therapy for completely resected stage II colon cancer. Cochrane Database Syst Rev. 2008;3:CD005390.

19. Lievre A, Bachet JB, Le Corre D, et al. KRAS mutation status is predictive of response to cetuximab therapy in colorectal cancer. Cancer Res. 2006;66:3992-5.

20. Medical Policy: Genetic and diagnostic testing. 2010. http://blue.regence.com/trgmedpol/geneticTesting/gt20.html. (Last accessed March 1st, 2011).

21. Regence Blue Cross and Blue Shield. Medical Policy: Genetic and diagnostic testing. 2010. http://blue.regence.com/trgmedpol/lab/lab20.html. (Last accessed March 1st, 2011).

50

22. Wood WC, Budman DR, Korzun AH, et al. Dose and dose intensity of adjuvant chemotherapy for stage II, node-positive breast carcinoma. The New Engl J Med. 1994;330:1253-9.

23. Clinical practice guidelines we can trust. Washington, DC: Institute of Medicine; 2011. 24. The Genentech oncology trend report: Perspectives from managed care, specialty pharmacy

providers, oncologists, and practice managers. 2008-2009 Oncology Trend Report. South San Francisco (CA): Genentech; 2009.

25. Etheredge LM. A rapid-learning health system. Health Aff. 2007;26:107-18. 26. Slutsky JR. Moving closer to a rapid-learning health care system. Health Aff. 2007;26:122-4. 27. Abernethy AP, Ahmad A, Zafar SY, Wheeler JL, Reese JB, Lyerly HK. Electronic patient-reported

data capture as a foundation of rapid learning cancer care. Med Care. 2010;48:S32-8. 28. Elkin EB, Bach PB. Cancer's next frontier: Addressing high and increasing costs. JAMA.

2010;303:1086-7. 29. Neuss MN, Desch CE, McNiff KK, et al. A process for measuring the quality of cancer care: The

quality oncology practice initiative. J Clin Oncol. 2005;23:6233-9. 30. Nadler E, Eckert B, Neumann PJ. Do oncologists believe new cancer drugs offer good value? The

Oncologist. 2006;11:90-5. 31. Neumann PJ, Palmer JA, Nadler E, Fang C, Ubel P. Cancer therapy costs influence treatment: A

national survey of oncologists. Health Aff. 2010;29:196-202. 32. Greenberg D, Earle C, Fang CH, Eldar-Lissai A, Neumann PJ. When is cancer care cost-effective? A

systematic overview of cost-utility analyses in oncology. J Nat Canc Inst. 2010;102:82-8. 33. Pearson SD, Bach PB. How Medicare could use comparative effectiveness research in deciding

on new coverage and reimbursement. Health Aff. 2010;29:1796-804.

51

Appendix A: Advisory Committee

Amy Abernethy, MD, Program Director, Cancer Care Research Program, Duke University Medical Center

Michael del Aguila, PhD, Senior Director, Biometrics, Health Outcomes and Payer Support, Genentech

Jeff Allen, PhD, Executive Director, Friends of Cancer Research

Naomi Aronson, PhD, Executive Director, Technology Evaluation Center, Blue Cross Blue Shield Association

Ethan M. Basch, MD, MSc, Oncologist, Outcomes Researcher, Memorial Sloan-Kettering Cancer Center, Chair,

ASCO Guidelines Committee

Roy Baynes, MD, VP, Oncology, Global Clinical Development, Amgen

Josh Benner, PharmD, ScD, Research Director, Brookings Institution, Engelberg Center for Health Care Reform

Timothy D. Birner, PharmD, MBA, Director, Global Evidence and Value Development, Medical Affairs, Global

R&D, Sanofi-Aventis

Amy Bonoff, MBA, Advocate, National Breast Cancer Coalition

Kim Caldwell, RPh, Director, Competitive Health Analytics, Humana

Stephen Eck, MD, PhD, Vice President, Translational Medicine and Pharmacogenomics, Eli Lilly

Erin Karnes, MPH, MBA, Research Associate, Brookings Institution, Engelberg Center for Health Care Reform

Muin Khoury, PhD, Director, National Office of Public Health Genomics, Centers for Disease Control and Prevention

William Li, MD, President and Medical Director, Angiogenesis Foundation

Robert McDonough, MD, Head of Clinical Policy Research & Development, Aetna

Sharon Murphy, MD, Scholar-in-Residence, Institute of Medicine

Joshua Ofman, MD, MSHS, VP, Global Coverage & Reimbursement, Global Government Affairs; VP, Global Health Economics and Global Development, Amgen

Andrew Perry, Senior Director, Oncology Payer Marketing, GlaxoSmithKline

BW Ruffner, MD, President, Tennessee Medical Association

Mary Lou Smith, JD, MBA, Co-Founder, Research Advocacy Network

Josephine Sollano, PhD, Head, Oncology Medical Communications and Global Health Economics and Outcomes Research at Pfizer Pharmaceuticals

Tamas Suto, Vice President and Head of Medical Affairs, Oncology, Sanofi-Aventis

52

Appendix B: Summit Attendees

Name Title Organization

Amy Abernethy Associate Professor, Duke University

Medical Center, Director Duke Cancer Care

Research Program

Duke University Medical Center

Gerry Adler Social Science Research Analyst Office of Strategic Planning, Centers for Medicare

and Medicaid Services

Raiz Ali Director Avalere Health

Jeff Allen Executive Director Friends of Cancer Research

Kirsten

Anderson

Chief of Staff, Office of the Chief Medical

Officer Aetna

Marietta

Anthony Director, Women's Health Critical Path Institute

Naomi Aronson Executive Director Blue Cross Blue Shield Association

Leonard Arzt Executive Director National Association for Proton Therapy

Tom Ault Principle Health Policy Alternatives

Dan Ball Research Advisor Eli Lilly and Company

Tracy Baroni

Allmon Executive Director, Health Policy Novarits Oncology

Charlie Barr Director of Patient Registries Genentech

Andrea

Baruchin Director, NIH Relations Foundation for NIH

Roy Baynes Vice President, Global Development Amgen

Robert Beltran Physician Executive Registry of Physician Executives

Marc Berger Vice President, Global Health Outcomes Eli Lilly

Nathan Blake Director of Legislative Affairs Jenkins Hill Consulting

Giselle Bleecker President Medical Market Strategists

Diane Bodurka Professor University of Texas M.D. Anderson Cancer Center

Amy Bonoff Advocate National Breast Cancer Coalition

Jonathan Bor Senior Editor Health Affairs

Pamela Bradley Associate Director of Science Policy American Association for Cancer Research

Otis Brawley Chief Medical Officer American Cancer Society

53

Name Title Organization

Steven

Brotman Senior Vice President Advanced Medical Technology Association

Paul Brown Research Associate Professor University of North Carolina

Susan Brown Director, Health Education Susan G. Komen for the Cure

Suanna Steeby

Bruinooge Director, Research Policy American Society of Clinical Oncology

Sarah Butler Senior Associate Avalere Health

Tanisha Carino Senior Vice President Avalere Health

Kenneth Carson Assistant Professor of Medicine Washington University School of Medicine

Steven Clauser Chief, Outcomes Research National Cancer Institute

Vivian Coates Vice President ECRI Institute

Perry Cohen Project Director Parkinson Pipeline Project

Deborah

Collyar President Patient Advocates In Research

Mitra Corral Associate Director Bristol-Myers Squibb

Martin Corry Director of Federal Health Policy Buchanan Ingersoll & Rooney PC

Eva Culakova Biostatistician III Duke University

George

Dahlman Senior Vice President, Public Policy The Leukemia & Lymphoma Society

Nancy

Davenport-

Ennis

Chief Executive Officer National Patient Advocate Foundation

Chris Dawe Health Staff Senate Finance Committee

Michael Del

Aguila Senior Director Genentech

Jessica

DeMartino Manager, Health Policy Programs National Comprehensive Cancer Center

Claude

Desjardins

Professor & Director of Clinical Scholars

Project University of Illinois Medical Center

Anthony Dias Managing Director Blue Cross Blue Shield Association

Yvonne Dragan Associate Director, Safety Assessment-US Astrazeneca

Robert Dubois Chief Science Officer National Pharmaceutical Council

Subash

Duggirala Senior Medical Advisor Centers for Medicare and Medicaid Services

54

Name Title Organization

Sydney Dy Associate Professor Johns Hopkins University

Peggy Eastman Writer Oncology Times

Stephen Eck Vice President, Translational Medicine and

Pharmacogenomics Eli Lilly

Stephen Edge Chief, Breast Division Roswell Park Cancer Institute

Lynn Etheredge Director, Rapid Learning Project George Washington University

Doug Faries Senior Research Advisor Eli Lilly

Rosemarie

Filart Medical Officer National Institutes of Health

Andrew

Freedman

Branch Chief National Cancer Institute

Sarah Garner Harkness Fellow in Healthcare Policy and

Practice 2010-2011 Tufts Medical Centre

Lou Garrison Professor University of Washington

Ron Geigle Partner Polidais

Mary Giffin Senior Health Policy Analyst General Accountability Office

Richard Gliklich Chief Executive Officer Outcome

Cliff Goodman Vice President The Lewin Group

Mark Gorman Director of Survivorship Policy National Coalition for Cancer Survivorship

Jennifer Graff Research Director, Methods, Evidence &

Coverage National Pharmaceutical Council

Melvyn

Greberman President Public Health Resources

Sharon Green Patient Advisory Committee Chair LifeBridge Health

Amy Guo Senior Director Novartis Oncology

Jim Hahn Health Economist Congressional Research Service

Michael

Halpern Senior Fellow RTI International

David Harrison Director of Global Health Amgen

Thomas Hassall Senior Director Regulatory Policy & Intelligence

Bradford Hirsch Fellow Duke University

Andrew

Hoffman

PhD Candidate, Department of Social

Studies of Medicine

McGill University

Marco Huesch Assistant Professor Duke University

55

Name Title Organization

Louis Jacques Director, Coverage & Analysis Group Centers for Medicare and Medicaid Services

Lee James Medical Director Pfizer

Janet Jenkins-

Showalter

Director, Food and Drug Administration

Liaison Roche

Patti Jewell Senior Director Pfizer

Kevin Johnson Senior Vice President, Public Policy ZERO - The Project to End Prostate Cancer

Francis Kalush Diagnostics and Personalized Medicine

Network Leader

Office Center Director, Center for Devices and

Radiological Health, Food and Drug Administration

Arif Kamal Oncology Fellow Duke University Medical Center

Erin Karnes Research Associate The Brookings Institution, Engelberg Center for

Health Care Reform

Keith

Kerneklian Legislative Manager Academy of Radiology Research

Deb Kirkland Nurse Navigator LifeBridge Health

Roger Klein Medical Director, Molecular Oncology BloodCenter of Wisconsin, Medical College of

Wisconsin

Catherine

Knowles Policy Director Colorectal Cancer Coalition

James Knutson Risk Manager Aircraft Gear Corp

Hon-Sum Ko Medical Officer Food and Drug Administration

Linda Kohn Director General Accountability Office

Michael

Kolodziej Medical Director US Oncology

Beth Kosiak Associate Executive Director of Health

Policy American Urological Association

Anne Laffoon Assistant Director General Accountability Office

Janet Leigh Robert Wood Johnson Foundation Health

Policy Fellow Louisiana State University Health Sciences Center

Drew Levy Medical Science Director Genentech

Astra Liepa Health Outcomes Scientist Eli Lilly

Dan Longo Deputy Editor New England Journal of Medicine

Susan Love President Dr. Susan Love Research Foundation

Simon Lowry Vice President US Medical Affairs Pfizer

Bryan Loy Market Vice President for Kentucky Humana

56

Name Title Organization

Ann Luggen Professor Emeritus Northern Kentucky University

Gary Lyman Professor of Medicine & Director,

Comparative Effectiveness & Outcomes

Research

Duke University

Xiaomei Ma Associate Professor Yale University

Martin

Marciniak

Senior Director, Global Health Outcomes

Oncology Eli Lilly

Katie Martinez PhD Student Johns Hopkins Bloomberg School of Public Health

Viraj Master Associate Professor Emory University

Michael

McCaughan Editor The RPM Report

Kristin

McDonald Division of Advocacy and Health Policy American College of Surgeons

Robert S.

McDonough

Head of Clinical Policy Research &

Development Aetna

Bill McGivney Chief Executive Officer National Comprehensive Cancer Network

Jason McKitrick Principal Liberty Partners Group

Jay McKnight Manager, Pharmacy Clinical Strategies Humana

Richard

Mercier

Chairperson, Department of Clinical

Oncology Marshfield Clinic

Neal Meropol Chief, Hematology and Oncology University Hospitals Case Medical Center

John Metz Chief Executive Officer & Chairman JustHealth

Lesley-Ann

Miller Research Scientist Eli Lilly

Steve Miller Executive Vice President of Regulatory

Affairs National Patient Advocate Foundation

Wilhelmine

Miller Senior Fellow National Opinion Research Center at The University

of Chicago

Sandra Mitchell Research Scientist National Cancer Institute

Padmaja

Mummaneni Consumer Safety officer

Center for Devices and Radiological Health, Office of

Translational Sciences, Office of Clinical

Pharmacology, Food and Drug Administration

Anthony J.

Murgo

Associate Director, Office of Oncology Drug

Products Food and Drug Administration

Sharon Murphy Scholar-in-Residence Institute of Medicine

57

Name Title Organization

Sharyl Nass Director, National Cancer Policy Forum Institute of Medicine

Peter Neumann Director Center for the Evaluation of Value and Risk in Health

Kathryn Nix Researcher The Heritage Foundation

Ginger Norris Program Specialist American Society of Clinical Oncology

Virginia Pappas Chief Executive Officer Society of Nuclear Medicine

Lori A.

Paserchia Medical Officer Centers for Medicare and Medicaid Services

Steven Pearson President Institute for Clinical and Economic Review

John Penrod Health Economist Bristol-Myers Squibb

Eleanor

Perfetto Senior Director Pfizer

Arnold Potosky Director, Health Services Research Georgetown University, Lombardi Comprehensive

Cancer Center

Irene Prabhu

Das Health Scientist National Cancer Institute, Division of Cancer Control

and Population Sciences

Bruce Quinn Senior Health Policy Specialist Foley Hoag

Elda Railey Co-Founder Research Advocacy Network

Scott Ramsey Cancer Prevention Faculty Fred Hutchinson Cancer Research Center

Vanessa Reddy Health Economist Genentech

Philip Renner Principal Consultant, Metrics Development Kaiser Permanente - Care Management Institute

Jeff Roche Medical Officer Centers for Medicare and Medicaid Services

Alan Rosenberg Vice President of Medical Policy,

Technology Assessment & Credentialing

Programs

Wellpoint, Inc.

Gary Rosner Professor Johns Hopkins University

Greg Rossi Vice President, Global Health Economics

and Pricing Genentech

B W. Ruffner President Tennessee Medical Association

Peter Rumm Deputy Director Division of Surgical, Orthopedic, and Restorative

Devices

Deborah

Runkle Senior Program Associate American Association for the Advancement of

Science

Mark Rutstein Senior Medical Director ImClone Systems/Eli Lilly

58

Name Title Organization

Leah Sansbury Epidemiologist National Cancer Institute

Cheri Sanzi Director of Outreach American Urological Association Foundation

Christy Schmidt Director, Quality Division American Society of Clinical Oncology

Deborah Schrag Associate Professor Dana-Farber Cancer Institute

Jessica

Schumacher Assistant Professor University of Florida

Gail Schwartz Advocacy Manager SHARE: Self-help for Women with Breast or Ovarian

Cancer

Mekré

Senbetta Director, Outcomes Research Johnson & Johnson

Marilyn Serafini Robin Toner Fellow and Special

Correspondent Kaiser Health News

Gregory Smith Leader, US Outcomes Research Pfizer

Mary Lou Smith Co-founder Research Advocacy Network

Stacey Smith Principle Jenkins Hill Consulting

Claire Snyder Assistant Professor Johns Hopkins School of Medicine

Josephine

Sollano Head, Global Outcomes Research Pfizer

Melony

Sorbero Policy Researcher RAND Corporation

Kendall

Stevinson

Manager, Global Health Outcomes,

Oncology

Merck

Saira Sultan Director Sanofi-Aventis

Ralph Swindle Research Fellow Eli Lilly

Lisa Taylor Founder Survivor Diver

Tamar

Thompson

Vice President, Reimbursement & Health

Policy Kimbell & Associates

Sibyl Tilson Specialist in Health Care Financing Congressional Research Service

Edward Trimble Head, Gynecologic Cancer Therapeutics and

Quality of Cancer Care Therapeutics

Clinical Investigations Branch, Cancer Therapy

Evaluation Program, Division of Cancer Treatment

and Diagnosis, National Cancer Institute

Claudia Uribe Clinical Research Manager Humana

Robert Valdez Executive Director and RWJF Professor Robert Wood Johnson Foundation Center for Health

Policy

Bhadrasian

Vikram Branch Chief National Cancer Institute

59

Name Title Organization

Joel Wallace Director, Global Health Economics & Pricing Genentech

Taylor Walsh Consultant Integrative Health and Wellness Strategies

Myrtle

Washington Patient Navigator Smith Farm Center for Healing and the Arts

Tasmeen Weik Public Health Analyst Health Resources and Services Administration

Agnes

Westelinck Head of Global Regulatory Affairs, Oncology GlaxoSmithKline

Stephanie

Wheeler Assistant Professor Health Policy and Management, University of North

Carolina at Chapel Hill

Christina

Williams Project Manager Durham Veterans Affairs Medical Center

Rhys Williams Executive Director Amgen

Elaine Yu Associate Director Genentech

Peter Yu Director, Cancer Research Palo Alto Medical Foundation

Joan Zlotnik Director Social Work Policy Institute, National Association of

Social Workers

60

Appendix C: Agenda

Monday, November 1st

, 2010

SESSION I: SETTING THE STAGE

SEAN TUNIS, MD, MSC

President and Chief Executive Officer, Center for Medical Technology Policy

ENGAGING PATIENTS IN CER

Moderator

ETHAN M. BASCH, MD, MSc

Oncologist, Outcomes Researcher, Memorial Sloan-Kettering Cancer Center

Chair, ASCO Guidelines Committee

Presenters

SUSAN M. LOVE, MD, MBA

President and Medical Director, Dr. Susan Love Research Foundation

MARY LOU SMITH, JD, MBA

Co-Founder, Research Advocacy Network

SUSAN BROWN, MS, RN

Director, Health Education, Susan G. Komen for the Cure

AMY BONOFF, MBA

Advocate, National Breast Cancer Coalition

Response Panel

SCOTT RAMSEY, MD, PHD

Director, Cancer Outcomes Research, Fred Hutchinson Cancer Research Center

DEBORAH COLLYAR

President, PAIR: Patient Advocates in Research

MICHAEL DEL AGUILA, PHD

Senior Director, Biometrics, Health Outcomes & Payer Support, Genentech

CRITICAL CHALLENGE OF GENOMICS AND PERSONALIZED MEDICINE

Moderator

GARY LYMAN, MD, MPH, FRCP (EDIN)

Professor of Medicine and Director, Comparative Effectiveness & Outcomes Research, Duke University

Presenters

Scott Ramsey, MD, PhD

Director, Cancer Outcomes Research, Fred Hutchinson Cancer Research Center

NEAL MEROPOL, MD

Chief, Division of Hematology & Oncology, University Hospitals Case Medical

Chair, ASCO Cancer Research Committee and the ASCO CER Task Force

STEVE SHAK, MD

Chief Medical Officer, Genomic Health

STEPHEN ECK, MD, PHD

Vice President, Translational Medicine and Pharmacogenomics, Eli Lilly

61

Response Panel

JEFF ALLEN, PHD

Executive Director, Friends of Cancer Research

SHARON MURPHY, MD

Scholar-in-Residence, Institute of Medicine

GREGORY ROSSI, PHD

Vice President, Global Health Economics and Pricing, Genentech, Inc.

Tuesday, November 2nd

, 2010

IMPROVING CLINICAL EVIDENCE FOR ONCOLOGY

Moderator

CLIFFORD GOODMAN, PHD

Vice President, Lewin Group, Center for Comparative Effectiveness Research

Chair, Medicare Evidence Development & Coverage Advisory Committee

Presenters

RICHARD GLIKLICH, MD

President and CEO, Outcome Sciences, Inc.

PETER YU, MD

Palo Alto Medical Foundation

DEBORAH SCHRAG, MD, MPH

Attending Oncologist and Associate Professor, Dana-Farber/Harvard Cancer Center

AMY ABERNETHY, MD

Program Director, Cancer Care Research Program, Duke Univ. Medical Center

Associate Professor of Medicine, Duke University School of Medicine

Response Panel

MARK GORMAN, JD

Director of Survivorship Policy, National Coalition for Cancer Survivorship

B.W. RUFFNER, MD

President, Tennessee Medical Association

JOSEPHINE A. SOLLANO, DR. PH

Head, Global Outcomes Research, Pfizer Oncology Global Medical Affairs

TRANSLATING EVIDENCE TO PRACTICE

Moderator

LYNN M. ETHEREDGE

Director, Rapid Learning Project, George Washington University

Presenters

ETHAN M. BASCH, MD, MSc

Oncologist, Outcomes Researcher, Memorial Sloan-Kettering Cancer Center

Chair, ASCO Guidelines Committee

STEVE CLAUSER, PhD

Chief, Outcomes Research Branch of the Applied Research Program, NCI

ROBERT S. MCDONOUGH, MD

Head of Clinical Policy Research & Development, Aetna

62

WILLIAM T. MCGIVNEY, PHD

Chief Executive Officer, National Comprehensive Cancer Network (NCCN)

Response Panel

MICHAEL KOLODZIEJ, MD

Medical Director, US Oncology

MARY LOU SMITH, JD, MBA

Co-Founder, Research Advocacy Network

JOHN W. MCKNIGHT, PHARM.D.

Manager, Pharmacy Clinical Strategies, Humana Pharmacy Solutions

VALUE AND COSTS IN ONCOLOGY

Moderator

SARAH GARNER, PhD

Associate Director for R&D, National Institute for Health and Clinical Excellence (UK)

Harkness Fellow in Health Care Policy and Practice (2010-11), Tufts Medical Center

Presenters

LOUIS JACQUES, MD

Director, Coverage and Analysis Group, Centers for Medicare and Medicaid Services

STEVEN PEARSON, MD, MSC

President, Institute for Clinical and Economic Review, Harvard Medical School

PETER NEUMANN, SCD

Director, Center for the Evaluation of Value and Risk in Health

Institute for Clinical Research and Health Policy Studies

Response Panel

SUSAN BROWN, MS, RN

Director, Health Education, Susan G. Komen for the Cure

NAOMI ARONSON, PHD

Executive Director, Technology Evaluation Center, Blue Cross Blue Shield Association

TIMOTHY D. BIRNER, PHARMD, MBA

Director, Global Evidence & Value Development, Medical Affairs, Global R&D,

Sanofi-Aventis

KEYNOTE ADDRESSES

MARK MCCLELLAN, MD, PHD

Director, Engelberg Center for Health Care Reform, The Brookings Institution

PETER BACH, MD, MAPP

Attending Physician, Memorial Sloan-Kettering Cancer Center

OTIS BRAWLEY, MD

Chief Medical Officer, American Cancer Society

Professor of Hematology and Oncology, Emory University School of Medicine

63

Appendix D: Acronyms

AHRQ Agency for Healthcare Research and Quality

APC Adenosis Polyposis Coli gene

ARRA American Recovery and Reinvestment Act

ASCO American Society of Clinical Oncology

BRAF Serine/threonine-protein kinase B-Raf

BRCA1 Breast Cancer Type 1 Susceptibility Protein

BRCA2 Breast Cancer Type 2 Susceptibility Protein

CanCORS Cancer Care Outcomes Research and Surveillance Consortium

CCHIT Certification Committee for Health Information Technology

CER Comparative Effectiveness Research

CMS Centers for Medicare and Medicaid Services

CMTP Center for Medical Technology Policy

CTC CT (virtual) Colonography

EBM Evidence Based Medicine

EBRT External Beam Radiation Therapy

EGFR Epidermal Growth Factor Receptor

EHR Electronic Health Record

FAP Familial Adenomatous Polyposis

FDA Food and Drug Administration

5-FU 5-Fluorouracil

GnRH Gonadotropin Releasing Hormone

GRACE Good Research for Comparative Effectiveness Initiative

GRADE Grading of Recommendations, Assessment, Development and Evaluation

HD High Dose

HER2 Human Epidermal Growth Factor Receptor 2

IOM Institute of Medicine

IMRT Intensity Modulated Radiation Therapy

IRB Institutional Review Boards

IT Information Technology

KRAS Kirsten Rat Sarcoma Viral Oncogene

LD Low Dose

Medicare - SEER Medicare – Surveillance, Epidemiology and End Results

MEN 2A Multiple Endocrine Neoplasia, type 2A

MEN 2B Multiple Endocrine Neoplasia, type 2B

MMR Mismatched Repair Mechanisms

MYH MYH-associated polyposis

NBCC National Breast Cancer Coalition

NCCCP NCI Community Cancer Centers Program

NCI National Cancer Institute

NCCN National Comprehensive Cancer Network

NICE National Institute for Clinical Excellence

NIH National Institutes of Health

NSABP - 20 National Surgical Adjuvant Breast and Bowel Project

OCER Observational Comparative Effectiveness Research

64

ONC Office of the National Coordinator

PCORI Patient Centered Outcomes Research Institute

PICOTS Population Intervention Comparators Outcomes Timing Setting study design

PIK3CA Phosphoinositide-3-Kinase Catalytic Alpha Polypeptide

PPACA Patient Protection and Affordable Care Act

PROs Patient Reported Outcomes

PSA Prostate-Specific Antigen

PTEN Phosphatase and Tensin Homolog

QALY Quality-Adjusted Life Year

QOPI Quality Oncology Practice Initiative

RAF Rapidly Accelerated Fibrosarcoma

RAS Rat Sarcoma

RCT Randomized Controlled Trials

SBRT Stereotactic Body Radiation Therapy

TAILORx Trial Assigned Individualized Options for Treatment (Rx)

US United States