How to Evaluate New Medical Treatments

Mark Perloe, M.D.www.ivf.com

Mass-Media Influence

Discoveries and innovations characterized by– unjustifiable degree of certainty– Immediate promise of reducing

the burden of illness or increasing life expectancy

Technical detail and scientific jargon are boringExaggerates the risks posed by putative health hazardsPharmaceutical company advertising

Internet Medicine

Where to find information– National Library of Medicine

Medline, Grateful Med, PubMed – Expert Chats– Organization Websites– Mailing Lists

Limitations– Credentials not evident– Financial bias– Self promotion

Rationale for Evidence-based Medicine

Cost of medical care rising sharplyWide variations in outcomes of medical carePatients are better informed and more vocal seeking to make informed choicesHealth-care decision making should involve the patient and be based on best evidenceStatistical tools (meta-analysis) are better understoodEvaluation of research papers for validity and relevance

Evidence-based Medicine

Incorporation of relevant scientific information into the clinical decision making processProvides the impetus to gather, review and summarize evidence on effective and cost of health-care interventions (meta-analysis)Optimizes the utilization of health-care resources by avoiding ineffective and medical & surgical therapies

Clinical Study Types

Experimental Studies– Randomized Control Trials (RCT)– Randomized Cross-Over Trial

Observational Studies– Cohort (Incidence, Longitudinal)– Case-Control – Cross-Sectional (Prevalence)– Case Series– Case Report

Statistics

ANOVA, Student’s t-test, Chi-squareP < 0.05Power CalculationsClinical vs. biological significance

Evaluating Medical Studies

Validity: Truth– External Validity: Can the study be generalized to the

population of the reader– Internal Validity: Study is well designed. Results not

due to chance, bias or confounding factors– Symmetry Principle: Groups are similar

Evaluating Medical Studies

Confounding: distortion of the effect of one risk factor by the presence of anotherBias: Any effect from design, execution, & interpretation that shifts or influences results– Confounding bias: failure to account for the effect of one or more

variables that are not distributed equally– Measurement bias: measurement methods differ between groups,

lack of blinding– Sampling (selection) bias: design and execution errors in sampling– Reader/Investigator bias: human tendency to accept information

that supports pre-conceived opinions and reject studies that don’t– Sponsorship bias: studies designed to support sponsors views

What’s a Meta-analysis?

Met-analysis provides an overview of clinical trialsMeta-analysis is a set of statistical procedures designed to accumulate experimental and correlational results across independent studies that address a related set of research questions.

Meta-Analysis

Variability in populationsVariability in study design– Study quality– Endpoint reportage– Availability of data

Variability in interventions

Limitations of EBM

Impact of prevailing political, ideological, economic and technological forces– NIH research often politically directed (stem cell)– Pharmaceutical industry financing limits comparative studies

of alternative therapies

Homogenous population required for studiesLimited evaluation of co-therapiesLack of evidence does not equal lack of effectiveness

Clinical Decision-making

What is my patient’s RISK ?– of the event the treatment strives to prevent?– of the side-effect of treatment?

What is my patient’s RESPONSIVENESS?What is the treatment’s FEASIBILITY in my practice/setting?What are my patient’s VALUES ?