Understanding Comparative Effectiveness

© Cerner Corporation All Rights Reserved 1

Robert W. Dubois, MD, PhDChief Medical OfficerCerner LifeSciences

Demystifying Comparative Effectiveness Research: A Case Study Learning Guide


Topics

Opening thoughtsFramework for reviewing a new CER report Case studies and lessons learned

Randomized trial-incomplete answerMeta-analysis-misleading conclusionObservational study-incorrect change in clinical practice

Final thoughts


Institute of Medicine Definition*

“…The purpose of CER is to assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve health care at both the individual and population levels.”

*IOM 2009 Initial National Priorities for Comparative Effectiveness Research

© Cerner Corporation All Rights Reserved

Not All Studies Are Created (or interpreted) Equally

Good study

Well informed

Not so good study

Not so well

informed

Good study that doesn’t answer your

question

Uninformed


Proceed With Caution…


When the reports arrive…


It may feel like this…


Categorizing is good…


Understanding is even better…


Topics

Opening thoughtsHow to approach CER


How to Approach CER

Step 1 Consider for

whom the findings are applicable

Step 2 Consider whether

aspects of the study design might affect the results

Step 3Consider

whether the findings may change with new research


Step 1: Consider for Whom the Findings are Applicable

An H1N1 study in school age kids may (or may not) help an internist with his/her adult population.A study of patients with severe rheumatoid arthritis may (or may not) apply to a population of milder patients.A study of diabetic control after one year may (or may not) apply to the longer term implications of a policy change.

Will the study results generalize to your environment?


Questions for Step 1

Does the study resemble your population of interest (e.g., age, gender, SES, disease profile)?Does the clinical setting resemble your own (e.g., primary care, inner city)?Is the study conducted in the “real world” or in a highly controlled environment?Are the outcomes appropriate to your needs?


How to Approach CER

Step 1 Consider for



aspects of the study design

might affect the results


There are 3 Common Types of CER

Randomized trialsMeta-analysisObservational studies

Was the right design chosen?


StudyQuestion

Study Design

Study Results


Does the design match the study question?

RCT: may not meet your “real world” needs.Observational study: may have too many uncontrollable factors to be valid.Meta-analysis: just combining studies because they are “there” may not be appropriate.


Step 2: Consider Whether Aspects of the StudyDesign Might Affect the Results

Does the study design match the question being asked?Was it carried out with adequate rigor?

Each study design has its own issues to consider.One size does not fit all.


One size does not fit all…


One size does not fit all…


How to Approach CER

Step 1 Consider for




Step 3Consider



“How sure are we… really”

http://www.hasyudeen.com/2008/02/network-centric-competition-in-flat.html




Topics

Opening thoughtsFramework for reviewing a new CER report (a.k.a. “Readers’ Guide”Case studies and lessons learned

Randomized trial-incomplete answer• Step 3-new information• Step 1-revisit the relevant population


Can it Work?

Randomized Controlled Trials (RCTs)Can an intervention work under certain controlled conditions?Characteristics:

Planned experimental frameworkDefined treatment optionsSpecified outcomesMay be single or double blinded


RCT Design


RCTs Have Strengths and Weaknesses

Strengths:Substantial internal validityReduced likelihood of bias or confoundingGold standard for clinical researchPathway to FDA approval

Weaknesses:May not generalize Modest size reduces ability to observe rare events.Placebo may not be the best “real world” comparator.Some patients leave the trial or “cross over” to the other therapy.


RCT Case Study: Cetuximab

Clinical SituationColorectal cancer is the 3rd most common cancer and cause of cancer death.Only an 11% survival if metastaticTreatment is not curative.Understanding of cancer genetics led to biomarker testing and targeted therapies.Cetuximab is an anti-epidermal growth factor receptor agent (EGFR).


Cetuximab RCT*: Was This the Full Story?

Cetuximab Comparison0

1

2

3

4

5

6

7

*Jonker N Engl J Med 2007

Survival in Months

572 patients with mCRCNon-blinded RCTCetuximab + supportive care vs. supportive care alone


Not the Full Cetuximab Story

Basic science research suggested that the KRAS gene and its mutations could influence efficacy.Subgroup analysis performed based upon testing tissue samples

Those with non-mutated gene had much greater benefit from cetuximab (5 months greater overall survival).


Not the Full Cetuximab Story

FDA narrowed the drug’s indications to patients with a non-mutated gene.Recommendations to test patients for the genetic marker

Therapy targeted to patients most likely to respond.Patients unlikely to respond avoid ill effects of the drug.


RCT Tips for the Consumer

RCTs assess whether an intervention can work in a controlled environment.Although RCTs are viewed as the “gold standard,” the initial impression of the cetuximab study was incomplete.

Multiple studies designed similarly would have likely yielded similar incomplete conclusions.A subgroup analysis based upon genetic markers yielded different results and conclusions.

No results are permanent.


Topics


Randomized trialMeta-analysis-misleading conclusion

Step 2: methods used (and not used)


Meta-Analysis

“If one RCT is good, then more RCTs must be better…”

Quantitative and statistical combination of study resultsHighest level of evidenceUseful when different studies have different results


Bagshaw SM, Ghali WA. Acetylcysteine for prevention of contrast-induced nephropathy after intravascular angiography: a systematic review and meta-analysis. BMC Med. 2004;2(1):38.

Metanalysis Clarifies the Benefit


Wilcken, MJA. 2007;186 (7):368-370.

Metanalysis Clarifies the Benefits


Meta-Analysis

StrengthsIncreases the effective sample sizeProvides statistically stronger conclusionsDetects lower frequency events and more subtle distinctions

WeaknessesCreates an impression of “truth”Easy to do wrong…If care is not taken, results may be invalid.


Meta-Analysis Case Study: Avandia

Clinical SituationBlood sugar control reduces certain diabetic complications.But, oral drugs have been associated with an increased risk of heart disease (tolbutamide).TZDs seemed safer than sulfonylureas.Avandia approved by the FDA in 1999.


Original Article Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes

Steven E. Nissen, M.D., and Kathy Wolski, M.P.H.

N Engl J MedVolume 356(24):2457-2471

June 14, 2007


What was Done and Found?

A meta-analysis examined the impact of Avandia on cardiac events.Out of 116 studies, 42 met the authors’ inclusion criteria.

15,565 Avandia patients12,282 comparison patients


Avandia Adversely Impacts Heart Disease

Compared with the placebo, estrogen plus progestin resulted in:

Increased risk of heart attack

Increased risk of stroke

Increased risk of blood clots

Increased risk of breast cancer

Reduced risk of colorectal cancer

Fewer fractures

No protection against mild cognitive impairment and increased risk of dementia (study included only women 65 and older)

Nissen SE, Wolski K. N Engl J Med. 2007;356:2457-2471.



Was This the Right Answer?

“Thou shalt not combine heterogeneous studies…”Some studies had placebo and others active comparators.Some studies observed one arm longer than the other.Excluded “zero” event trials



The choice of statistical methods impacts the results.

Re-analysis with an alternative approach showed no increased risk.

Inclusion of the “zero event” studies also eliminated any statistical differences.New trial results (RECORD) showed no difference in deaths but an increase in heart failure.


Meta-analysis Tips for the Consumer

Results are highly dependent on the studies included and excluded.Statistical methodology can impact study results.Nothing is permanent - emerging data may change the conclusions.


Topics


Randomized trialMeta-analysisObservational study-incorrect change in clinical practice

Step 1: population differed in subtle ways from a more typical oneStep 2: intervention and comparison groups were different


Observational Studies

Answers the question: will it (likely) work (not can it work)?Examines the effects of treatment without formal randomizationPerformed prospectively or retrospectively



Framingham Study identified heart disease risk factors.Commonly uses existing “real world” databases

Administrative or billing dataElectronic health records



StrengthsAssess health care in the “real world”Lower cost and faster to performLarge sample sizes feasibleHypothesis generating

WeaknessesAdministrative databases have minimal clinical detail and may contain errors.Uncontrolled design leads to potential bias or confounding.Subject to “data dredging”Cannot prove cause and effect


Observational Case Study: Hormone Treatment

Clinical SituationHeart disease is the leading cause of death in women above age 50Estrogen falls after menopause

May account for the postmenopausal heat disease risk


What was Done and Found?

Nurses’ Health Study (NHS) collected data from 48,470 women aged 30-55.Focused on the impact of hormones on outcomesOngoing surveys assessed risk factors and health outcomes.

Estrogen use associated with lower heart disease risk


Implications of the Nurses’ Health Study

Estrogen use associated with

lower heart disease risk



Estrogen use associated with lower

heart disease risk

Menopausal women

should take hormone

treatment




heart disease risk

Menopausal women

should take hormone

treatment?


The clinical trials were designed to test the effects of postmenopausal hormone therapy, diet modification, and calcium and vitamin D

supplements on heart disease, fractures, and breast and colorectal cancer.


Women’s Health Initiative

RCT with 27,347 post-menopausal womenMean age was 63.6 years.


Women’s Health Initiative Muddied the Waters…

RCT with 27,347 post-menopausal womenMean age was 63.6 years.Study halted early.

Increased risk of clots, breast cancer and stroke.Estrogen-progestin users: increased risk of heart disease Estrogen only users: no CV benefits.




heart disease risk

Menopausal women

should take hormone

treatment?


Why the Different Results?

Theory 1: WHI population was older than the NHS cohort (started hormones much later)Theory 2: NHS women who took hormones were healthier

Higher educationHigher SESLeanerLacked prior cardiac disease

Theory 3: The NHS women who took hormones differed from those who did not


The Differences Were Explainable (Partly)

1. NHS data re-analyzed…Less benefit for older women starting HRT later.

2. WHI re-analyzed…Some benefit for younger women starting HRT early.

Steps 1 and 2

With observational studies, make sure…

• The population resembles yours

• The intervention and comparison groups are similar


Observational Study Tips for the Consumer

Useful to understand “real world” benefits and harmsLook for confounding factors (Step 2)Nothing is permanent - emerging data may change the conclusions


Topics

Opening thoughtsFramework for reviewing a new CER report (a.k.a. “Readers’ Guide”Case studies and lessons learnedFinal thoughts


Final Thoughts

1. Different study types can offer different understandings.

Question

Study Design

RCTMetanalysis

Observational study

Results


Final Thoughts


2. Results matter.Avandia sales fell after meta-analysis published.Hormone treatment plummeted after results of the WHI.Cetuximab use is growing.


Final Thoughts


2. Results matter.3. Consider the potential impact upon policy making.

Influence guidelines?Impact reimbursement?Stable clinical area or advancing rapidly?


Demystifying Comparative Effectiveness Research


Demystifying Comparative Effectiveness Research


Demystifying Comparative Effectiveness Research: A Case Study Learning Guide

Reader’s Guide Check Lists

Robert W. Dubois, MD, PhDChief Medical OfficerCerner LifeSciences


How to Approach CER

Step 1 Consider for




Step 3Consider



Use Check Lists With Caution

The lists are not all inclusive.A “yes” to all is not all that you need…All items are not created equally.


Step 1: Are the Findings Applicable?

Assessing External Validity1. Does the study population reflect your population of

interest? ___yes ___no

2. Does the type of clinical environment reflect your own? ___yes ___no

3. Is the study conducted in a “real-world” setting? ___yes ___no

4. Are the outcomes being measured appropriate to your needs? ___yes ___no

5. Is the evidence clinically significant, and the effect size large enough to be relevant? ___yes ___no

6. Does the duration of the study allow you to draw any long-term conclusions? ___yes ___no

7. Have appropriate subgroups been considered? ___yes ___no


Step 2: Design Affect the Results?

Randomized Controlled Trials

1. Is there a clear hypothesis? ___yes ___no

2. Were the inclusion criteria logical? ___yes ___no

3. Was the comparison group appropriate? ___yes ___no

4. Were study methods clearly described and valid? ___yes ___no

5. Are all relevant outcomes reported? ___yes ___no

6. Is there a strong exposure/outcome link? ___yes ___no

7. Are study limitations discussed? ___yes ___no


Step 2: Design Affect the Results?Meta-Analyses

1. Is there a clear hypothesis? ___yes ___no

2. Were the outcomes clearly defined and clinically relevant? ___yes ___no

3. Were the inclusion criteria logical? ___yes ___no

4. Were all relevant studies included? ___yes ___no

5. Do the authors assess the potential for publication bias? ___yes ___no

6. Was the search for relevant studies thorough? ___yes ___no

7. Were multiple searchable databases employed? ___yes ___no

8. Were the included studies of high methodologic quality? ___yes ___no

9. Did the authors test whether their studies were similar (homogeneous) or different (heterogeneous)? ___yes ___no

10. Were the combined studies sufficiently similar? ___yes ___no

11. Did the authors adequately describe and defend the statistical methods they used for analysis? ___yes ___no


Step 2: Design Affect the Results?

Observational Studies1. Does the population seem properly defined for the

stated outcomes of interest? ___yes ___no

2. Are the baseline characteristics similar for the groups being studied? ___yes ___no

3. Were study methods clearly described and valid? ___yes ___no

4. Are all relevant outcomes reported? ___yes ___no

5. Is there a strong exposure/outcome link? ___yes ___no

6. Did the authors adequately describe and defend the statistical methods they used for analysis? ___yes ___no

7. Have potentials for bias and confounding been discussed and accounted for? ___yes ___no


Step 3: Findings Change With New Research?

Innovation and Change*

1. Is this research in an area that is changing rapidly, (such as cancer research)? ___yes ___no

2. Are advances in scientific knowledge and understanding likely to influence how we view the effectiveness of this intervention?

___yes ___no

3. Will these results change clinical guidelines or impact policy? ___yes ___no

4. Is future work in this arena likely to take place from existing databases? ___yes ___no

5. Could subgroup analysis clarify these results? ___yes ___no

*In contrast to the prior tables, a “yes” indicates that the reader should view the study results with caution.

Documents

Understanding Comparative Effectiveness