A Skeptic’s Guide to Reading the Medical Literature

A Skeptic’s Guide A Skeptic’s Guide to Reading the to Reading the

Medical LiteratureMedical Literature

Joe Lex, MD, FACEP, MAAEMJoe Lex, MD, FACEP, MAAEMClinical Professor of Emergency MedicineClinical Professor of Emergency Medicine

Temple University School of MedicineTemple University School of MedicinePhiladelphia, PA USAPhiladelphia, PA USA

[email protected] @JoeLex5 @JoeLex5

mailto:[email protected]

Peer ReviewPeer Review

Peer Review = Gold StandardPeer Review = Gold Standard

SlowSlow ExpensiveExpensive IneffectiveIneffective ““Somewhat of a lottery”Somewhat of a lottery” Prone to bias and abuseProne to bias and abuse Hopeless at spotting error and Hopeless at spotting error and

fraudfraud


“…“…peer review is impossible to peer review is impossible to define in operational terms…Peer define in operational terms…Peer review is thus like poetry, love, or review is thus like poetry, love, or justice.”justice.”

Smith R. J R Soc Med. 2006 Smith R. J R Soc Med. 2006 Apr;99(4):178-82. Apr;99(4):178-82.


““When I was editor of the BMJ I was When I was editor of the BMJ I was challenged by two of the cleverest challenged by two of the cleverest researchers in Britain to publish an researchers in Britain to publish an issue of the journal comprised only issue of the journal comprised only of papers that had failed peer review of papers that had failed peer review and see if anybody noticed. I wrote and see if anybody noticed. I wrote back ‘How do you know I haven’t back ‘How do you know I haven’t already done it?’”already done it?’”

Richard Smith – Editor, BMJRichard Smith – Editor, BMJ


Two reviewers, same paperTwo reviewers, same paper

Reviewer A:Reviewer A: I found this paper an I found this paper an extremely muddled paper with a extremely muddled paper with a large number of deficits large number of deficits

Reviewer B:Reviewer B: It is written in a clear It is written in a clear style and would be understood by style and would be understood by any readerany reader

Richard Smith – Editor, BMJRichard Smith – Editor, BMJ


Benefits: ???Benefits: ??? Editors are convinced that it is Editors are convinced that it is

invaluableinvaluable ““If (peer review) was a drug it If (peer review) was a drug it

would never get onto the market.”would never get onto the market.”

Drummond Rennie, Drummond Rennie, Deputy Editor of JAMADeputy Editor of JAMA

Positive-Outcome BiasPositive-Outcome Bias

Dummy article, 210 reviewersDummy article, 210 reviewers Same except for conclusionSame except for conclusion

– Half got “positive conclusion”Half got “positive conclusion”

– Half got “no difference”Half got “no difference” Intentional errors includedIntentional errors included Methods section sameMethods section same

Emerson GB et al. Arch Int Med 2010, 170: 1934 Emerson GB et al. Arch Int Med 2010, 170: 1934



Positive Positive outcomeoutcome

Neutral Neutral outcomeoutcome

Recommend Recommend publicationpublication 97.3%97.3% 80.0%80.0%

Errors Errors identifiedidentified 0.410.41 0.850.85

Methods Methods validity scorevalidity score 8.248.24 7.537.53


““Compromise(s) integrity of the Compromise(s) integrity of the literature”literature”– Inflates treatment effects when Inflates treatment effects when

included in meta-analysisincluded in meta-analysis


Peer Review Misses ThingsPeer Review Misses Things

Fake manuscriptFake manuscript Contained 10 major and 13 minor Contained 10 major and 13 minor

errorserrors Sent to 262 reviewersSent to 262 reviewers 203 returned203 returned

Baxt WB et al. Ann Emerg Med 1998:32, 310 Baxt WB et al. Ann Emerg Med 1998:32, 310


15 ‘accept’15 ‘accept’– 17% major, 12% minor17% major, 12% minor

117 ‘reject’117 ‘reject’– 39% major, 21% minor39% major, 21% minor

67 ‘revise’67 ‘revise’– 30% major, 22% minor30% major, 22% minor



68% did not recognize that 68% did not recognize that conclusion was not supported by conclusion was not supported by datadata

Failed to identify 68% of major Failed to identify 68% of major errorserrors


Cochrane AnalysisCochrane Analysis

““We could not identify any We could not identify any methodologically convincing methodologically convincing studies assessing the core effects studies assessing the core effects of peer review”of peer review”

““Little empirical evidence to support Little empirical evidence to support use of editorial peer review to use of editorial peer review to ensure quality.”ensure quality.”

Jefferson T, et al. Cochrane DatabaseJefferson T, et al. Cochrane Database

Watch out for…Watch out for…

……strong strong associationsassociations

……independent independent predictorspredictors

……citation biascitation bias……amplificationamplification……inventioninvention……composite composite

endpointsendpoints

……workup biasworkup bias……spectrum biasspectrum bias……referral biasreferral bias……surrogate markerssurrogate markers……publication biaspublication bias……absolute vs absolute vs

relative riskrelative risk

Strong AssociationStrong Association

There is a “strong association” There is a “strong association” between hanging out in bars and between hanging out in bars and developing lung cancerdeveloping lung cancer

People who hang out in bars tend People who hang out in bars tend to smoke and drink a lotto smoke and drink a lot

Does hanging out in bars cause Does hanging out in bars cause lung cancer?lung cancer?


There is a “strong association” There is a “strong association” between the rooster crowing at between the rooster crowing at 06:15 and the sun rising at 06:2006:15 and the sun rising at 06:20

Does the rooster’s crow cause the Does the rooster’s crow cause the sun to rise?sun to rise?


No correlation with…No correlation with…

……sensitivitysensitivity

……specificityspecificity

……accuracyaccuracy

……usefulnessusefulness

Independent PredictorIndependent Predictor

If patients are matched in all other If patients are matched in all other parameters, the group with parameters, the group with abnormal test results will have abnormal test results will have more outcome events than the more outcome events than the group with normal test resultsgroup with normal test results

Independent PredictorIndependent Predictor

In the ICU, three organ failure at In the ICU, three organ failure at three days carries a 97% mortality three days carries a 97% mortality raterate

So who don’t I have to treat, since So who don’t I have to treat, since it’s so futile?it’s so futile?

Predictors & AssociationsPredictors & Associations

Does not mean:Does not mean:““Your test is negative, you can go Your test is negative, you can go

home now.”home now.”““Your test is positive, you need to Your test is positive, you need to

stay and have more studies.”stay and have more studies.” NoNo valuevalue in making diagnostic or in making diagnostic or

therapeutic decisionstherapeutic decisions

Negative Predictive ValueNegative Predictive Value

Probability of no disease among Probability of no disease among patients with a negative testpatients with a negative test

NPV = TN / (TN + FN)NPV = TN / (TN + FN) Denominator is Denominator is number of number of

patients who test negativepatients who test negative So if disease prevalence low…So if disease prevalence low…

Negative Predictive ValueNegative Predictive Value

NPV does NPV does notnot tell you test value tell you test value If disease prevalence 5%, coin flip If disease prevalence 5%, coin flip

has NPV = 95%has NPV = 95% If disease prevalence 85%, coin If disease prevalence 85%, coin

flip has PPV = 85%flip has PPV = 85%

Citation BiasCitation Bias

Citation biasCitation bias: not citing papers : not citing papers that refute or weaken beliefthat refute or weaken belief

AmplificationAmplification: expand belief : expand belief system by citing papers that system by citing papers that present no data addressing itpresent no data addressing it

InventionInvention: hypothesis converted : hypothesis converted into fact through citation aloneinto fact through citation alone

Redundant PublicationRedundant Publication

Ondansetron in post-op vomitingOndansetron in post-op vomiting 84 trials with 11,980 patients84 trials with 11,980 patients On closer inspection: 70 trials with On closer inspection: 70 trials with

8645 patients8645 patients 17% duplicate publications17% duplicate publications 28% inflation of patients28% inflation of patients

Tramer MR et al. BMJ 1997:315;635-640.Tramer MR et al. BMJ 1997:315;635-640.

Redundant PublicationRedundant Publication

Four identical trials published with Four identical trials published with different authorsdifferent authors

Duplicated publications most likely Duplicated publications most likely to have positive resultsto have positive results

NNT in nonduplicated trials: 9.5NNT in nonduplicated trials: 9.5 NNT in duplicated trials: 3.9NNT in duplicated trials: 3.9

Tramer MR et al. BMJ 1997:315;635-640.Tramer MR et al. BMJ 1997:315;635-640.

Multiple HypothesesMultiple Hypotheses

Multiple HypothesesMultiple Hypotheses

10,674,945 Ontario residents 10,674,945 Ontario residents aged between 18 and 100 years aged between 18 and 100 years – Derivation cohort 5,337,472 Derivation cohort 5,337,472

– Validation cohort 5,337,473Validation cohort 5,337,473

895 diagnoses for which patients 895 diagnoses for which patients had emergent / urgent had emergent / urgent hospitalizationhospitalization

Born Under a Bad SignBorn Under a Bad Sign

Goal: identify two diagnoses for Goal: identify two diagnoses for each astrological sign for which each astrological sign for which the probability of hospitalization the probability of hospitalization was statistically significantly was statistically significantly greater compared to residents greater compared to residents born under the remaining 11 signsborn under the remaining 11 signs


Found 72 such associationsFound 72 such associations Number of diagnoses ranged from Number of diagnoses ranged from

2 (Scorpio) to 10 (Taurus), with a 2 (Scorpio) to 10 (Taurus), with a mean of 6 diagnoses for each mean of 6 diagnoses for each astrological signastrological sign

The P-values for 72 significant The P-values for 72 significant associations: 0.0003 to 0.0488associations: 0.0003 to 0.0488


Validation cohort: residents born Validation cohort: residents born under sign of Leo had significantly under sign of Leo had significantly higher probability of higher probability of hospitalization due to GI hospitalization due to GI hemorrhage compared to other hemorrhage compared to other residents of Ontarioresidents of Ontario– Relative risk = 1.15 (p = 0.0483)Relative risk = 1.15 (p = 0.0483)


Validation cohort: residents born Validation cohort: residents born under sign of Sagittarius had under sign of Sagittarius had significantly higher probability of significantly higher probability of hospitalization for fractures of hospitalization for fractures of humerus compared to residents humerus compared to residents born under the remaining 11 born under the remaining 11 astrological signsastrological signs– Relative risk = 1.38 (p = 0.0125)Relative risk = 1.38 (p = 0.0125)

If an ineffective therapy is tested If an ineffective therapy is tested for 20 indications, by chance it is for 20 indications, by chance it is likely to demonstrate a significant likely to demonstrate a significant effect for at least one of theseeffect for at least one of these

Do the Math!!Do the Math!!

Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.Ioannidis JP, et al. BMJ. 2010 Sep 13;341:c4875.

If 10 outcomes are tested for each If 10 outcomes are tested for each indication, a statistically significant indication, a statistically significant effect will be observed for at least effect will be observed for at least one outcome for nearly half of the one outcome for nearly half of the indicationsindications



In 91 trials discontinued early, the In 91 trials discontinued early, the true effect was, on average, only true effect was, on average, only 70% of that suggested by trials 70% of that suggested by trials

It was <50% of perceived effect It was <50% of perceived effect when trials were discontinued when trials were discontinued after occurrence of <200 eventsafter occurrence of <200 events



Workup BiasWorkup Bias

In routine care the diagnostic In routine care the diagnostic workup of a patient is by definition workup of a patient is by definition determined by previous test determined by previous test resultsresults

Positive screening test is referred Positive screening test is referred to receive verification of diagnosis to receive verification of diagnosis by “gold standard”by “gold standard”

Workup BiasWorkup Bias

Reference test is always Reference test is always interpreted with knowledge of interpreted with knowledge of preceding test information preceding test information

Example: treadmill stress testExample: treadmill stress test– Family practitioner’s office: Family practitioner’s office:

sensitivity 40%, specificity 85%sensitivity 40%, specificity 85%

– Cardiologist’s office: sensitivity Cardiologist’s office: sensitivity 70%, specificity 70%70%, specificity 70%

Spectrum BiasSpectrum Bias

Diagnostic test has different Diagnostic test has different sensitivities or specificities in sensitivities or specificities in patients with different clinical patients with different clinical manifestations of the disease for manifestations of the disease for which the test is intended which the test is intended


Patient obviously has disease: Patient obviously has disease: test is positivetest is positive

Patient obviously does not have Patient obviously does not have disease: test is negativedisease: test is negative

““I don’t know if the patient has the I don’t know if the patient has the disease” – test in betweendisease” – test in between– Example: BNPExample: BNP


If pretest probability <5%, you’re If pretest probability <5%, you’re more accurate than BNPmore accurate than BNP

If pretest probability >95%, you’re If pretest probability >95%, you’re more accurate than BNPmore accurate than BNP

The “In Between” GroupThe “In Between” Group

Sensitivity: 88% Sensitivity: 88%

Specificity: 55%Specificity: 55%


If the serum BNP is…If the serum BNP is…

……>500 pg/mL >500 pg/mL rule in CHF rule in CHF

……<100 pg/mL <100 pg/mL rule out CHF if low rule out CHF if low pretest probabilitypretest probability

……100 – 500 pg/mL: not sensitive, 100 – 500 pg/mL: not sensitive, not specific, and not very helpfulnot specific, and not very helpful

Referral BiasReferral Bias

Sanford GuideSanford Guide says: “80% of cat says: “80% of cat bites get infected”bites get infected”

Dog bite: self-referral because of Dog bite: self-referral because of wound size, rabies worry, etc.wound size, rabies worry, etc.

Cat bite: “It’s infected”Cat bite: “It’s infected” Headache in oncologist’s officeHeadache in oncologist’s office Step on a nailStep on a nail

Other BiasesOther Biases

Length biasLength bias: indolent disease is : indolent disease is more likely to be detected in a more likely to be detected in a screening program than screening program than aggressive disease, leading to aggressive disease, leading to apparent improved outcome apparent improved outcome

Other BiasesOther Biases

Lead-time biasLead-time bias: survival of a : survival of a screened population is measured screened population is measured from the date of screening, from the date of screening, whereas survival of an whereas survival of an unscreened population is unscreened population is measured from detection of measured from detection of symptomatic diseasesymptomatic disease

Absolute vs relative Absolute vs relative

Can statins prevent VTE?Can statins prevent VTE? JUPITER trial says “yes!!”JUPITER trial says “yes!!” ““Patients taking rosuvastatin Patients taking rosuvastatin

((CrestorCrestor) 20 mg/day have ) 20 mg/day have 43% 43% lower risklower risk of DVT or PE” of DVT or PE”

Absolute vs relative Absolute vs relative

These patients relatively healthy These patients relatively healthy ABSOLUTEABSOLUTE risk drops from about risk drops from about

0.7% to 0.4%0.7% to 0.4% Treat 333 patients for one year to Treat 333 patients for one year to

prevent one DVT or PE prevent one DVT or PE

Surrogate EndpointsSurrogate Endpoints

Some surrogate end points may Some surrogate end points may be part of disease or risk factor be part of disease or risk factor

Sometimes surrogate endpoint Sometimes surrogate endpoint confused with diseaseconfused with disease

Example: sitagliptin enhances Example: sitagliptin enhances incretins and enhances incretins and enhances physiological glucose controlphysiological glucose control

DrugDrug SurrogateSurrogate EndpointEndpoint

EncainideEncainide ↓ ↓ premature premature depolarizationdepolarization ↑ ↑ mortalitymortality

DoxazosinDoxazosin ↓ ↓ BPBP ↑ ↑ mortalitymortality

EstrogenEstrogen ↓ ↓ cholesterolcholesterol ↑ ↑ cancer, stroke, cancer, stroke, dementiadementia

TorcetrapidTorcetrapid ↓ ↓ cholesterolcholesterol ↑ ↑ mortalitymortality

NesiritideNesiritide ↓ ↓ wedge wedge pressurepressure ↑ ↑ mortalitymortality

FluorideFluoride ↑ ↑ bone mineral bone mineral densitydensity

↑ ↑ nonvertebral nonvertebral fracturesfractures


RosiglitazoneRosiglitazone ↓ ↓ S-HbAS-HbA1c1c↑ ↑ myocardial myocardial infarctioninfarction

Intensive Intensive treatment treatment Type II DMType II DM

↓ ↓ S-HbAS-HbA1c1c

<6%<6%↑ ↑ mortalitymortality

MilrinoneMilrinone ↑ ↑ cardiac cardiac contractilitycontractility ↑ ↑ mortalitymortality

IbopamineIbopamine ↑ ↑ cardiac cardiac contractilitycontractility ↑ ↑ mortalitymortality

EpoprostenolEpoprostenol ↑ ↑ cardiac cardiac contractilitycontractility ↑ ↑ mortalitymortality


TobutamideTobutamide ↓ ↓ blood sugarblood sugar ↑ ↑ mortalitymortality

PhenforrminPhenforrmin ↓ ↓ blood sugarblood sugar ↑ ↑ mortalitymortality

ErythropoeitinErythropoeitin ↑ ↑ hemoglobin in hemoglobin in renal failurerenal failure ↑ ↑ mortalitymortality

AprotininAprotinin ↓ ↓ blood loss in blood loss in cardiac surgerycardiac surgery ↑ ↑ mortalitymortality

MetoprololMetoprolol ↓ ↓ perioperative perioperative ischaemiaischaemia ↑ ↑ mortalitymortality

Need POEM, Not DOENeed POEM, Not DOE

ED Patients

Experimental: New Test

Control: Standard Tests

Randomize Randomize

Outcomes: Morbidity, Mortality, Admissions, Length of Stay, Cost

CURE Trial: ClopidogrelCURE Trial: Clopidogrel

>12,500 patients for 3 – 12 >12,500 patients for 3 – 12 monthsmonths

Inclusion criteria: USA + Inclusion criteria: USA + cardiac cardiac biomarker or dynamic ECG biomarker or dynamic ECG changechange


Primary end point: composite of Primary end point: composite of cardiovascular death cardiovascular death oror MI MI oror nonfatal strokenonfatal stroke

Placebo + aspirin: 11.4%Placebo + aspirin: 11.4%

Clopidogrel + aspirin: 9.3% Clopidogrel + aspirin: 9.3%


Ads says “19.2% reduction in Ads says “19.2% reduction in death or cardiovascular event”death or cardiovascular event”

This is RELATIVE reductionThis is RELATIVE reduction Absolute reduction: 2.1%Absolute reduction: 2.1% NNT to prevent one death NNT to prevent one death oror MI MI

oror stroke: 48 stroke: 48

ClopidogrelClopidogrel

n = 6259n = 6259

PlaceboPlacebo

n = 6303n = 6303

P P valuevalue

Any 1Any 1oo or 2 or 2oo outcomeoutcome 16.5%16.5% 18.8%18.8% <0.001<0.001

CV deathCV death 5.1%5.1% 5.5%5.5% NSNS

MIMI 5.2%5.2% 6.7%6.7% <0.001<0.001

StrokeStroke

IschemiaIschemia

1.2%1.2%

8.7%8.7%

1.4%1.4%

9.3%9.3%

NSNS

NSNS



NNT to prevent one:NNT to prevent one:

……Cardiovascular death Cardiovascular death 250250

……Myocardial infarction Myocardial infarction 67 67

……StrokeStroke500500

……Refractory ischemiaRefractory ischemia 167167


Adverse effects: major bleedAdverse effects: major bleed Aspirin only: 2.7%Aspirin only: 2.7% Aspirin + clopidogrel: 3.6%Aspirin + clopidogrel: 3.6% Relative increase: 33%Relative increase: 33% Absolute increase: 0.9%Absolute increase: 0.9% NNH: 110NNH: 110


NNT to cause*NNT to cause*

Major bleedMajor bleed 100100

Life-threatening bleedLife-threatening bleed 250250

DiarrheaDiarrhea 91 91

PurpuraPurpura 62 62

ItchingItching 59 59

*compared to *compared to aspirinaspirin

Fibrinolysis and StrokeFibrinolysis and Stroke

Manufacturer-sponsored study of Manufacturer-sponsored study of IV alteplase in 821 patientsIV alteplase in 821 patients

Given 3 – 4.5 hours after onset of Given 3 – 4.5 hours after onset of ischemic stroke symptomsischemic stroke symptoms

Exclude from treatment: minor or Exclude from treatment: minor or rapidly improving, severe stroke, rapidly improving, severe stroke, prior stroke, diabetes, age >80prior stroke, diabetes, age >80

Hacke W, et al, NEJM 2008 Sep 25, 359(13):1317Hacke W, et al, NEJM 2008 Sep 25, 359(13):1317


AlteplaseAlteplase Controls Controls

Modified Rankin 0 – 1Modified Rankin 0 – 1 52.4%52.4% 45.2%45.2%

Intracerebral Intracerebral hemorrhage (ICH)hemorrhage (ICH) 27.0%27.0% 17.6%17.6%

Symptomatic ICHSymptomatic ICH 2.4%2.4% 0.2%0.2%

90-day mortality90-day mortality 7.7%7.7% 8.4%8.4%

Hacke W. et al, NEJM 2008 Sep 25, 359(13):1317Hacke W. et al, NEJM 2008 Sep 25, 359(13):1317



ECASS-III dichotomized the ECASS-III dichotomized the modified Rankin Scalemodified Rankin Scale

Instead of using 0-6Instead of using 0-6

0 – 1 was favorable0 – 1 was favorable

2 – 6 was unfavorable2 – 6 was unfavorable But 2 is slight disability…But 2 is slight disability…

……and 6 is deathand 6 is death


00 No symptoms at allNo symptoms at all11 Able to carry out all usual activitiesAble to carry out all usual activities22 Slight disability, but able to look after Slight disability, but able to look after

own affairs without assistanceown affairs without assistance33 Moderate disability; requires some help, Moderate disability; requires some help,

but walks without assistancebut walks without assistance44 Moderately severe disability; unable to Moderately severe disability; unable to

walk without assistance walk without assistance 55 Severe disability; bedridden, incontinentSevere disability; bedridden, incontinent66 DeadDead


00 No symptoms at allNo symptoms at all11 Able to carry out all usual activitiesAble to carry out all usual activities22 Slight disability, but able to look after Slight disability, but able to look after

own affairs without assistanceown affairs without assistance33 Moderate disability; requires some help, Moderate disability; requires some help,

but walks without assistancebut walks without assistance44 Moderately severe disability; unable to Moderately severe disability; unable to

walk without assistance walk without assistance 55 Severe disability; bedridden, incontinentSevere disability; bedridden, incontinent66 DeadDead


Now adjust the scaleNow adjust the scale

Make 0 – 2 favorableMake 0 – 2 favorable

Make 3 – 6 unfavorableMake 3 – 6 unfavorable

……and the advantage of IV rt-PA and the advantage of IV rt-PA over placebo goes away in the over placebo goes away in the intention-to-treat groupintention-to-treat group

The Australasian Journal of The Australasian Journal of Bone and Joint Medicine Bone and Joint Medicine

““Merck paid an undisclosed sum to Merck paid an undisclosed sum to Elsevier to produce several volumes of a Elsevier to produce several volumes of a publication that had the look of a peer-publication that had the look of a peer-reviewed medical journal, but contained reviewed medical journal, but contained only reprinted or summarized articles –only reprinted or summarized articles –most of which presented data favorable most of which presented data favorable to Merck products – that appeared to act to Merck products – that appeared to act solely as marketing tools with no solely as marketing tools with no disclosure of company sponsorship.”disclosure of company sponsorship.”

ElsevierElsevier

Elsevier also publishes...Elsevier also publishes...

……pseudoscientific pseudoscientific HomeopathyHomeopathy

……several “integrative medicine” several “integrative medicine” journals journals

……highly dubious highly dubious ExploreExplore, with , with articles on “distant healing” and articles on “distant healing” and food imbued with “intent”food imbued with “intent”

What Gets Published??What Gets Published??

Why Articles Are FalseWhy Articles Are False

1.1. The smaller the studies, the less The smaller the studies, the less likely the research findings are to likely the research findings are to be truebe true

2.2. The smaller the effect, the less The smaller the effect, the less likely the research findings are to likely the research findings are to be truebe true

Ioannidis JPA. PLoS Med. 2005;2(8):e124Ioannidis JPA. PLoS Med. 2005;2(8):e124


3.3. The greater the number and the The greater the number and the lesser the selection of tested lesser the selection of tested relationships, the less likely the relationships, the less likely the research findings are to be trueresearch findings are to be true



4.4. The greater the flexibility in The greater the flexibility in designs, definitions, outcomes, designs, definitions, outcomes, and analytical modes, the less and analytical modes, the less likely the research findings are to likely the research findings are to be truebe true



5.5. The greater the financial and The greater the financial and other interests, the less likely the other interests, the less likely the research findings are to be trueresearch findings are to be true



6.6. The hotter a scientific field (with The hotter a scientific field (with more scientific teams involved), more scientific teams involved), the less likely the research the less likely the research findings are to be truefindings are to be true


Publication BiasPublication Bias

Negative study unlikely to be first Negative study unlikely to be first publishedpublished

Positive study published: many Positive study published: many people are early adopterspeople are early adopters

Once accepted as “standard,” Once accepted as “standard,” company has no incentive to company has no incentive to sponsor further studiessponsor further studies


Turner EH, Matthews AM, Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Linardatos E, Tell RA, Rosenthal R.

Selective publication of Selective publication of antidepressant trials and its antidepressant trials and its influence on apparent efficacy.influence on apparent efficacy.

N Engl J Med. 2008 Jan N Engl J Med. 2008 Jan 17;358(3):252-60. 17;358(3):252-60.

Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60. Turner EH, et al. N Engl J Med. 2008 Jan 17;358(3):252-60.


74 FDA-registered studies on 74 FDA-registered studies on antidepressantsantidepressants

Represented 3449 participantsRepresented 3449 participants 37 studies viewed as positive and 37 studies viewed as positive and

were publishedwere published 1 study viewed as positive and 1 study viewed as positive and

was not publishedwas not published



74 FDA-registered studies on 74 FDA-registered studies on antidepressantsantidepressants

22 studies viewed as negative or 22 studies viewed as negative or questionable: not publishedquestionable: not published

11 studies viewed as negative or 11 studies viewed as negative or questionable: published in way questionable: published in way that conveyed positive outcomethat conveyed positive outcome



According to published literature: According to published literature: 94% of trials on antidepressant 94% of trials on antidepressant drugs are positivedrugs are positive

Actual data: 51% of trials on Actual data: 51% of trials on antidepressant drugs are positiveantidepressant drugs are positive


Who Pays the Piper?Who Pays the Piper?


Second-generation antipsychotics Second-generation antipsychotics for schizophreniafor schizophrenia

Head-to-head comparisonsHead-to-head comparisons Contradictory evidence…Contradictory evidence…

……depending on who paid for the depending on who paid for the studystudy


Comparison DrugsComparison Drugs Favorable reportsFavorable reports

SponsorSponsor Sponsor’sSponsor’s ComparisonComparison

OlanzapineOlanzapine vs vs risperidonerisperidone

LillyLilly 55 00

JanssenJanssen 33 11

OlanzapineOlanzapine vs vs clozapineclozapine

LillyLilly 22 11

NovartisNovartis 11 00




ClozapineClozapine vs vs risperidonerisperidone

NovartisNovartis 11 00

JanssenJanssen 11 00

ZiprasidoneZiprasidone vs vs olanzapineolanzapine

PfizerPfizer 11 11

LillyLilly 22 00




Amisulpride vs Amisulpride vs olanzapineolanzapine

LillyLilly 11 00

Sanofi-SyntholabSanofi-Syntholab 11 00

Of 30 studies, 27 (90%) showed the Of 30 studies, 27 (90%) showed the drug made by the company that drug made by the company that sponsored the study was bettersponsored the study was better

McClatchy NewspapersMcClatchy Newspapers2 February 20092 February 2009

““In the waning days of the Bush In the waning days of the Bush administration, the FDA issued new administration, the FDA issued new guidelines to make it easier for drug guidelines to make it easier for drug manufacturers to promote ‘off-label’ manufacturers to promote ‘off-label’ prescription drug use…”prescription drug use…”

“…“…doctors can find good information doctors can find good information on new drug uses in medical on new drug uses in medical journals…”journals…”

Quote of the YearQuote of the Year

““It is simply no longer possible to It is simply no longer possible to believe much of the clinical research believe much of the clinical research that is published, or to rely on the that is published, or to rely on the judgment of trusted physicians or judgment of trusted physicians or authoritative medical guidelines.”authoritative medical guidelines.”

Marcia Angell, MD Marcia Angell, MD Past Editor, New England Journal of Medicine Past Editor, New England Journal of Medicine

New York Review of Books; 15 December 2008New York Review of Books; 15 December 2008

Quote from ‘09Quote from ‘09

““Far too large a section of the treatment Far too large a section of the treatment of disease is today controlled by the big of disease is today controlled by the big manufacturing pharmacists, who have manufacturing pharmacists, who have enslaved us in a plausible pseudo-enslaved us in a plausible pseudo-science...The blind faith which some science...The blind faith which some men have in medicines illustrates too men have in medicines illustrates too often the greatest of all human often the greatest of all human capacities - the capacity for self capacities - the capacity for self deception...”deception...”

William Osler – June 1909William Osler – June 1909

““Far too large a section of the treatment Far too large a section of the treatment of disease is today controlled by the big of disease is today controlled by the big manufacturing pharmacists, who have manufacturing pharmacists, who have enslaved us in a plausible pseudo-enslaved us in a plausible pseudo-science...The blind faith which some science...The blind faith which some men have in medicines illustrates too men have in medicines illustrates too often the greatest of all human often the greatest of all human capacities - the capacity for self capacities - the capacity for self deception...”deception...”

And Finally…And Finally…

When you see the phrase, “This When you see the phrase, “This drug (or test) may be useful…”drug (or test) may be useful…”

……it is an incomplete sentenceit is an incomplete sentence It should say, “This drug (or test) It should say, “This drug (or test)

may may or may notor may not be useful.” be useful.”

Three Takeaway PointsThree Takeaway Points

1.1. About 50% of what you are About 50% of what you are learning is wrong. Nobody learning is wrong. Nobody knows which 50%.knows which 50%.

2.2. 90% of medical literature is crud90% of medical literature is crud““90% of everything is crud.”90% of everything is crud.”

Sturgeon’s Law (1958)Sturgeon’s Law (1958)

Three Takeaway PointsThree Takeaway Points

3.3. Love your teachers as you love Love your teachers as you love your parents, but never forget your parents, but never forget that your teachers are just as full that your teachers are just as full of BS as your parents.of BS as your parents.

Documents

A Skeptic’s Guide to Reading the Medical Literature