Evidence-Based Medicine Critical Appraisal of Harm Department of Medicine - Residency Training Program Tuesdays, 9:30 a.m. - 12:00 p.m., UW Health Sciences

Evidence-Based MedicineCritical Appraisal of Harm

Department of Medicine - Residency Training ProgramTuesdays, 9:30 a.m. - 12:00 p.m., UW Health Sciences

Library

Steps in Practicing EBM1. Convert the need for information into

an answerable question.2. Track down the best evidence with

which to answer that question.3. Critically appraise the evidence for its

validity, impact, and applicability.4. Integrate the evidence with our clinical

expertise and our patient’s characteristics and values.

Review Last Week’s Session






The Answerable Question

Good questions are the backbone of practicing

EBM. It takes practice to ask the well-formulated

question.

Well-Built Clinical ?’s• Directly relevant to the care of the

patient and our knowledge deficit.• Contains the following elements:

– the patient or problem being addressed

– the intervention or exposure being considered

– the comparison intervention or exposure, when relevant

– the clinical outcomes of interest.

Well Formulated ?’s• Focus scarce learning time on evidence

directly relevant to patient’s needs and our particular knowledge needs.

• Suggest high-yield search strategies.• Suggest forms that useful answers might take.• Help us to model life-long learning techniques

for our colleagues and students.• Are answerable and, thus, reinforce the

satisfaction of finding evidence that makes us better, faster clinicians.

Harm Questions






General ResourcesMETA-SEARCH ENGINESPrimeAnswers TRIP+ SUMSearch

SYSTEMATIC REVIEWS/META-ANALYSESCochrane Library PubMed Clinical Queries

EVIDENCE GUIDELINES/SUMMARIESAHRQ Evidence Reports Clinical Evidence AHRQ Preventive Services

CLINICAL RESEARCH CRITIQUESACP Journal Club 1996- Bandolier 1994- BestBETs

CASE REPORTS/SERIES, PRACTICE GUIDELINES, ETCNational Guideline ClearinghousePubMedMeSH Headings






Strategies for Critical Appraisal of Studies of Harm

Clinical ImportanceValidity

Applicability


Validity

Judging validity with just 4 questions!

1. Did investigators assemble clearly defined groups of patients similar in all important ways other than exposure?

2. Were exposures and outcomes measured in the same ways in both groups (objective/blinded)?

3. Was follow-up sufficiently long and complete (5% and 20% rule)?

4. Do the results of the harm study fulfill some of the tests for “causation”?

Types of Studies(in order of decreasing likelihood of being valid)

• Systematic reviews are ideal because individual RCTs seldom large enough to detect rare adverse events with precision - unfortunately, SR are uncommon.

• RCTs are difficult to conduct for most studies of harm.

• Cohort studies - exposed and unexposed followed for development of outcome of interest.

• Case-control studies - cases with outcome of interest compared with controls for “exposure”.

• Cross-sectional studies.• Case reports.

*

*

Criteria for Inferring Causality

• Is it clear that the exposure preceded the onset of the outcome?

• Is there a dose-response relationship?• Any positive evidence from a dechallenge-

rechallenge study?• Is the association consistent across

studies?• Does the association have biological

plausability?


Clinical Importance

Judging clinical importance with just 2 questions!

1. What is the magnitude of the treatment effect?

RR = (Exposed ER - Unexposed ER)/Unexposed ERAR difference = Exposed ER - Unexposed ERNNH = 1/AR difference

2. How precise is this estimate of the treatment effect?

95% CI - range of values within which we can be 95% sure that the population value lies.

Calculating NNT/NNH1. A randomized trial of new drug “Ligatite” reveals that 25% of World Cup skiers who take the drug for one year have ACL tears whereas 50% of World Cup skiers who take the placebo for the year have ACL tears. What is the NNT?

NNT = 1/AR reduction = 1/(0.50-0.25) = 4

3. An advertisement for a new drug fails to mention that it increases the relative risk of myocardial infarction by 50 % over 5 years. You read a valid study describing this finding. What is the NNH?

Unknown without knowing the event rate in the control population.

2. The study of the drug “Ligatite” also notes that 20% of athletes taking the drug develop clinical depression whereas 10% of athletes taking the placebo develop depression. What is the NNH?

NNH = 1/AR increase = 1/(0.20-0.10) = 10

The Odds Ratio• Used as an estimate of the risk ratio if the risk of

the disease in a population is low.• Is the principle measure of effect from case-control

studies (cannot calculate event rates). Also used to report effect size in meta-analysis.

• Odds of exposure in the disease group divided by odds of exposure in non-diseased group.

Disease

Present Absent

Risk Factor

Present a b

Absent c d

OR = (a/c)/(b/d)

= ad/cb

Converting OR to NNH

Calculator available at:http://www.cebm.utoronto.ca/practise/ca/statscal/orToNnt.htm

For OR greater than 11.1 1.25 1.5 1.75 2 2.25 2.5

PEER

0.05 212 86 44 30 23 18 16

0.1 113 46 24 16 13 10 9

0.2 64 27 14 10 8 7 6

0.3 50 21 11 8 7 6 5

0.4 44 19 10 8 6 5 5

0.5 42 18 10 8 6 6 5

0.7 51 23 13 10 9 8 7

0.9 121 55 33 25 22 19 18

Avoiding TIV(table induced vertigo)

• OR’s greater than 1.5 produce NNH < 50 across most PEER’s

• Patient needs to be at risk (non-trivial PEER) in order to be concerned.

• for any OR, NNH greatest when PEER=0.5

• Consider carefully nature of harm (are your patient’s values disrupted by the intervention and its sequelae)

Estimating Our Patient’s Expected Event Rates (PEER)

1. Assign our patient the overall control event rate from the study.

2. If there is a subgroup of patients in the study with similar characteristics assign the event rate for that subgroup.

3. If a validated clinical predication guide is available use it to assign an event rate.

4. Look for a different paper that describes the prognosis of untreated patients more similar to our patient and use its results to assign an event rate.

Clinical Tools for Estimating PEER

Available at: http://hin.nhlbi.nih.gov/atpiii/calculator.asp?usertype=prof%09


Applicability

Applicable to Our Patient?

1. Is our patient so different from those in the study that its results cannot apply?

2. What is our patient’s risk of benefit and harm from agent?

3. What are our patient’s preferences, concerns, and expectations from this treatment?

4. What alternative treatments are available?

Returning to “Ligatite”The trial of the drug revealed that 25% of World Cup skiers who take the drug for 1 year have ACL tears whereas 50% of skiers who take the placebo have ACL tears. It also revealed that 20% of exposed skiers developed depression whereas 10% of unexposed skiers developed depression.

Your patient reads about this in Ski Magazine and asks you to write a prescription. In discussing the medication with her you want to provide her with an estimate of the magnitude of risk reduction she would realize. Is her NNT = 4 and should she take this medication?

Probably Not:1. Her risk of an ACL tear is substantially less so you have to re-estimate her expected event rate.2. She is unlikely to be skiing year round so NNT is at least 2 to 3 times as high.3. There is risk of developing depression (NNH = 10 over 1 year).4. Whether or not to take the drug should take into account the relative value to the patient of preventing an ACL tear faced with the probability of developing depression.

Harm Questions

Documents

Evidence-Based Medicine Critical Appraisal of Harm Department of Medicine - Residency Training Program Tuesdays, 9:30 a.m. - 12:00 p.m., UW Health Sciences