Questioning the Quality and Nature of Evidence

• Context given current Health Reform• Concato et al• Benson et al• Implications for the future

• Evidence-Based Medicine as a Basis for Comparative Effectiveness • Stimulus (Feb 2009) includes $1.1 B for Comparative Effectiveness•Peter Orzag1 (Director of the Office of Management and Budget):

• We are "woefully underinvested in research”• Evidence-based medicine will hold down US healthcare

costs• A guide for reimbursement (CMS) decisions2

• Medical device industry

Sources: (1) Commonwealth Fund, May 2007 (2) Dhruva et al, New England Journal of Medicine, 2009

Concato et al, New England Journal of Medicine, 2000

• Sacks et al, 1982• Landmark study comparing published RCTs with observational

studies• 6 different therapies evaluated:

• 50 RCTs• 56 trials with historical controls

• Treatment groups had similar rates of outcome • Control subjects in observational studies had worse outcomes

than RCT controls• Efficacy of treatment intervention differed:

• Observational studies: 79% • RCTs: 20%

• Sacks et al concluded that selection bias skewed results to favor new interventions

• Meta-analyses from five major journals: Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, The Lancet, New England Journal of Medicine• Time period: 1991-1995• Studies classified into (1) RCT only (2) Observational only (3)

Both• DerSimonian and Laird analytic method used (Fleiss, 1993)• Exclusion Criteria:

(1) Cohort studies with historical controls(2) Clinical trials with non-random assignment(3) Results not reported as point estimates (RR or OR) or with CI’s

• 102 meta-analyses:• 72 RCT only• 24 Observational only• 6 Both

• 9 clinical topics:• 5 topics met eligibility criteria (99 articles, 1,871,681 study

subjects)• 4 topics were excluded

• Clinical topics selected: • BCG Vaccine and Tuberculosis• Screening Mammography and Breast Cancer Mortality• Cholesterol Levels and Death due to Trauma• Treatment of Hypertension and Stroke• Treatment of Hypertension and Coronary Heart Disease

Benson & Hartz, New England Journal of Medicine, 2000

• Observational studies are thought to be (Feinstein, 1989):

• Less Expensive• More timely• Cover a broader range of patients

•Several studies done in the 1970-1980s suggest that observational studies exaggerate treatment effects

• 56% of observational studies had favorable treatment effects vs. 30% in RCT’s (Chalmers et al, 1983)

• Observational study search: • Medline • Cochrane Database of Systematic Reviews

• Years included: 1985-1998• Journals limited to those cited in the Abridged Index Medicus• 3868 articles identified• Inclusion criteria for Observational Studies:

(1) Non-experimental study(2) Assessed difference between two treatments or one treatment and non-

intervention(3) Treatments administered by physicians(4) Study included a control group

• Articles that met all criteria were searched through Medline (1966-1998)• Mantel-Haenszel analytic method (Fidalgo et al, 2008)• 19 treatment comparisons (53 observational, 83 RCTs, 2 pseudo-randomized):

• 7 Cardiologic Treatments• 11 Non-Cardiologic Treatments• 2 Pseudo-randomized

• Authors interpreted their findings as different from previous published studies• Observational studies not as susceptible to confounding

bias as previously thought• Despite subjectivity inherent to clinical condition and

study selection, in sum data supports the conclusion that observational data can provide high quality data to use as a benchmark

"Half of what you'll learn in medical school will be shown to be either dead wrong or out of date within five years of your graduation; the trouble is that nobody can tell you which half -- so the most important thing to learn is how to learn on your own"

-Dr. David Sackett, a pioneer of evidence-based medicine

• RCTs: •Patient selection and cherry-picking•Subgroup-analyses

• Personalized medicine

• Bhattacharyya et al, Health Affairs, March/April 2009•Medicare pay-for-performance data, hip and knee replacement by orthopedic surgeons • 260 hospitals in 38 states•Finding: Conforming to or deviating from expert quality metrics had no relationship to actual complications or clinical outcomes

• Glickman et al, JAMA, 2007•5,000 heart failure patients at 91 hospitals •Finding: Application of most federal quality process measures did not change mortality from heart failure

“Human beings are not uniform in their biology. A disease with many effects on multiple organs, like diabetes, acts differently in different people. Information evolves and changes. Rather than rigidity, flexibility is appropriate in applying evidence from clinical trials. To that end, a good doctor exercises sound clinical judgment by consulting expert guidelines and assessing ongoing research, but then decides what is quality care for the individual patient. And what is best sometimes deviates from the norms.”

-Jerome Groopman & Pamela Hartzband, Wall Street Journal, 2009

• Commonwealth Fund, May 22nd 2007 Retrieved from: http://www.commonwealthfund.org/Content/Newsletters/Washington-Health-Policy-in-Review/2007/May/Washington-Health-Policy-Week-in-Review---May-29--2007/Orszag-Discusses-New-Ways-of-Alleviating-Soaring-Health-Care-Costs.aspx •JL Fleiss Review papers : The statistical basis of meta-analysis Statistical Methods in Medical Research, August 1993; 2: 121 - 145. •Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000 Jun 22;342(25):1887-92. •Benson K, Hartz AJ A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000 Jun 22;342(25):1878-86. •Feinstein AR. Current problems and future challenges in randomized clinical trials. Circulation 1984;70:767-774. •Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983;309:1358-1361•Fidalgo, Angel M., Madeira, Jaqueline M. Generalized Mantel-Haenszel Methods for Differential Item Functioning Detection Educational and Psychological Measurement 2008•Timothy Bhattacharyya, Andrew A. Freiberg, Priyesh Mehta, Jeffrey Neil Katz, and Timothy Ferris Measuring The Report Card: The Validity Of Pay-For-Performance Metrics In Orthopedic Surgery Health Affairs March/April 2009 - Volume 28, Number 2 •Seth W. Glickman; Fang-Shu Ou; Elizabeth R. DeLong; Matthew T. Roe; Barbara L. Lytle; Jyotsna Mulgund; John S. Rumsfeld; W. Brian Gibler; E. Magnus Ohman; Kevin A. Schulman; Eric D. Peterson Pay for Performance, Quality of Care, and Outcomes in Acute Myocardial Infarction JAMA. 2007;297(21):2373-2380. •Groopman J and Harzband P. Apr 2008. Why 'Quality' Care Is Dangerous. Wall Street Journal.