Critical Appraisal of Articles on Therapy

Learning ObjectivesAre the results likely to be valid?Are the valid results important? (Magnitude)Applying the valid, important results to a particular patient

The best study design for Therapy studies is RCT

Are the Results Likely to be Valid?Was the assignment of patients to treatment randomized?

What was the method of randomization?

Were patients and clinicians kept blind to treatment?

Key ConceptsRandomizationBlindingControl/ComparisonInternal ValidityExternal Validity = Generalizability

Study of Any InterventionRandom Error = Chance deviation from the underlying truthBias = Systematic deviation from underlying truth

Limitations for ParticipationExclusion CriteriaBy restricting the heterogeneity of the group, we reduce the opportunity for differences in outcome that arent due to the treatment itselfImproves INTERNAL VALIDITYMakes generalization of the results more precise but limits EXTERNAL VALIDITY to a smaller portion of the population

Population sampledLocal populationGeneral populationExternal Validity (Generalizability)Conclusions

Need a ComparisonControl group Minimizes the Hawthorne effectBy virtue of being in a study, the patients behavior changes and has a better prognosisObservation group Still may have a placebo effect unless placebo given to control groupBeyond the Hawthorne effectGiving a pill with an expected/potential result can provide effect even if the pill is inert

Selection of Control GroupAttempts to minimize selection biasRestriction: limits the range of characteristics of the patients in the studyMatching: for each patient in the study group, select one or more patients with the same characteristics for a comparison groupAdjustment: mathematical corrections to create an equal weight for dissimilar characteristicsStratification: compare outcomes from subgroups of each group with similar characteristics (i.e. age by decades)Randomization of the study populationClinical Epidemiology The Essentials. 3rd Ed. Fletcher et al. 1996, p 129.

Are the 2 Groups Comparable?Selection bias/Confounding biasKnown and unknown confounding variables

Want equal prognostic characteristicsRandomize > matchingStratified randomization schemes Can verify effectiveness of allocation

Potential BiasStrategy AgainstSampling BiasSelection BiasPlacebo Effect

Cointerventions

Assessment BiasFollow UpTarget populationRandomizationPlacebos and blinding participantsBlinding providers, treatment protocolsBlinding providersEnsuring completeness

Study of Any InterventionBottom Line:The 2 groups should be the same at the startConcealed randomizationBaseline characteristicsThe 2 groups should be treated exactly the same during the study except for the intervention being studiedTestsNon-study treatmentsBlinding

Variable Regression AnalysisIf the baseline prognostic characteristics are not equal in the groups, you can estimate the impact that each variable had on the results through statistical analysisUnivariate vs Multivariate regression analysisYou may have different prognostic characteristic that had no impact

Guyatt, et al Users Guides to the Medical Literature. AMA Press. 2002: 526-7

Evaluating Studies of TherapyAre the results valid?Was the assignment of patients to treatment randomized and was randomization concealed?Was follow-up of patients sufficiently long and complete?Were patients and clinicians kept blind?Were the groups treated equally, except for the experimental therapy?Were the groups similar at the start of the trial?Were all patients analyzed in the groups to which they were randomized?Intention to Treat preserves the value of randomization

How to Use an Article About TherapyIntention to Treat: The principle of attributing all patients to the group to which they were randomized

Preserves the value of randomization

Prognostic factors that we do and dont know about will be, on average, equally distributed between the two groups, and the effect we see will be just that due to the treatment assigned

How to Use an Article About TherapyIntention to Treat: Excluding non-compliant patients from the analysis leaves behind those who may be destined to have a better outcome, and destroys the unbiased comparison provided by randomizationIn many randomized trials, non-compliant patients from both treatment and placebo groups have fared worse than compliant patientsPatients too sick to receive a tx shouldnt only count as control cases

Two Ways to Report the ResultsPer-ProtocolResults analyzed according to the treatment received

Intention-to-TreatResults analyzed according to the treatment assigned whether or not they actually received it

1152 ptsDrug XDrug A1136 ptsPopulation of patients with a condition52 Dropped out (stopped the study drug)36 Dropped out (stopped the study drug)

150501152 52 150 + 50=1000

1136 36 50 + 150=1200

Per-ProtocolUnintentional CROSS-OVER

1152 ptsDrug XDrug A1136 ptsPopulation of patients with a condition52 Dropped out36 Dropped out150501152 1136 Intention-to-Treat

Cross-OverAt the end of a RCT of a medication versus a placebo, the groups switch treatments (still blinded) and the trial is repeatedIf the initial results also occur in the new treatment group (which previously was the placebo group), it lends credibility to the study

1152 ptsDrug XPlaceboResults

Benefit

No BenefitDrug XPlaceboResults

No Benefit

Benefit1136 ptsIntentional CROSS-OVER

Drop outs change prognosis of original groupExperimental GroupControl GroupAve Age = 58Ave Age = 57

Drop outs change prognosis of original groupControl GroupExperimental GroupAve Age = 57Ave Age = 51Ave Age = 72

Intention-to-TreatProsReal life effectivenessWhat the results will be on a population of patientsNeeded for economic analysisIntended to account for possible bias from not including subjects that would have affected the resultsIf the results still show benefit of treatment despite including patients who never received the treatment, the results may even be an underestimate

Intention-to-TreatConsAssumes the treatment cant be made more tolerableResults dont apply to the individualIf the nonadherence is not equal between the two groups, the results may be biasedIf the treatment effect is harmful, the inclusion of patients that never received the treatment might make it look less harmful

Intention-to-TreatBottom LinePer protocol analysis is biased if enough subjects arent countedIntention to treat likely to create a false negative result of an effect

Biased effect vs unbiased lack of effect

The Importance of Follow Up (Not the same as Drop Out) An Example

Trial ATrial B Treatment ControlTreatment ControlWorst case scenario (sensitivity analysis): assume the worst outcome in the treatment group for those that were lost to follow upGuyatt, et al Users Guides to the Medical Literature. AMA Press. 2002: 63-64

1000100030 (3%)30 (3%)200(20%)400(40%)(.40 .20)/.40= .2/.4 = .5 (50%)

1000100030 (3%)30 (3%)30 (3%)60(6%)(.06 - .03)/.06 = .03/.06 = .5 (50%)

# Patients# Lost to follow up# Deaths in eachgroupRRR not accounting for lost to follow up

Trial ATrial B Treatment ControlTreatment Control

1000100030 (3%)30 (3%)230(23%)400(40%)0.2/0.4 = 0.5 (50%)

1000100030 (3%)30 (3%)60 (6%)60(6%)0.03/0.06 = 0.5 (50%)

# Patients# Lost to follow up# Deaths

RRR not accounting for lost to follow up

RRR including lost to follow up

0.17/0.4 = 0.43(43%)

0.0/0.06 = 0No Reduction

Evaluating Studies of TherapyAre the valid results important?What is the magnitude of the treatment effect?How precise is this estimate of this treatment effect?

Are the results Clinically Significant?Control Event Rate CER= c / c + d

Experimental Event Rate EER = a / a + b

Absolute Risk ReductionARR = CER EER

Number Needed to TreatNNT = 1 / ARR

OutcomePresentOutcomeAbsentTotalsDrugABA+BPlaceboCDC+DTotalsA + CB + DA+B+C+D

Magnitude of EffectRR = Relative Risk or Risk Ratio:ER in the experimental group divided by the ER in the control groupRR = EER / CERRRR = Relative Risk Reduction:ARR / AR of the ControlAn estimate of the proportion of baseline risk (CER) that is removed by the therapy

RRR = ARR / CER= [CER EER] / CER

So How Does it Work??? Therapy Decisions Real-TimeMeasures of Treatment Effect:Shortcut Calculation: Downloadable to PDAEBM calculatorEBM Calculator - WebsitePre-appraised evidence: ACP Journal ClubNNT NomogramNNT tables

EBM Calculator

DeceptionRCT: drug 284 v. placebo for sepsis mortalityDrug 284 Placebo800/2000 1200/2000= 40% = 60%

RR = 40% / 60% = 0.67= 67%RRR= (60 40) / 60= 33% RCT: Panaceanex v. placebo for prostate cancer mortalityPanaceanex Placebo4/20006/2000= 0.2%= 0.3%

RR = 0.2% / 0.3% = 0.67= 67%RRR = (0.3 0.2) / 0.3= 33%

DeceptionRCT: drug 284 v. placebo for sepsis mortalityDrug 284 Placebo800/20001200/2000

400 lives savedRCT: Panaceanex v. placebo for prostate cancer mortalityPanaceanex Placebo4/20006/2000

2 lives saved

Number Needed to Treat (NNT)Drug 284 Placebo800/20001200/2000= 40%= 60%RRR= 33%

ARR = 60% - 40% = 20% = 0.20NNT = 1/ARRHow many people to treat so that 1 event will be preventedIf you reduce the numbers to a common denominator of 100, you can see it more easily

Number Needed to Treat (NNT)PlaceboDrug 2841200/2000800/2000120/20080/20060/10040/100

ARR = 60% - 40% = 20%So, if you say that you can prevent death for 20 people in every 100 you treat:(= 1 in 5)To save one life you need to treat 5 people

NNTInstead of saying if I treat X number of patients, I will prevent Y events, you say in order to prevent 1 event, I have to treat Z number of patients

If the ARR is 25%, the NNT is?

NNTInstead of saying if I treat X number of patients, I will prevent Y events, you say in order to prevent 1 event, I have to treat Z number of patients

If the ARR is 25%, the NNT is? 4

What about the Panaceanex?NNT = 1 / ARR= 1 / 0.1%= 1 / 0.001To prevent one event, must treat 1000 patients

RCT: Panaceanex v. placebo for prostate cancer mortalityPanaceanex Placebo4/20006/2000= 0.2%= 0.3%ARR = 0.3% - 0.2%= 0.1%

How to use an Article About TherapyAre the results applicable to my patient?Is our patient so different from those in the study that its results cannot apply?Is the treatment feasible in our setting?What are our patients potential benefits and harms from the therapy?What are our patients values and expectations for both the outcome we are trying to prevent, and the treatment we are offering?

Analyzing the ResultsEfficacyIf taken, does the treatment have an effect?AKA: Explanatory (explains the action of the treatment)

Effectiveness Will the treatment have good results if offered?Takes into account tolerability and harmful effectsBetter external validity

The End(s)!!!!!

**In assessing the benefit of treatment, assume the people who dropped out or crossed over actually stayed in the assigned group. In this scenario, those who never received the treatment appear as treated without any effect and dilute the findings. If the study still shows a treatment benefit despite dilution, it is a reliable result. However, if the treatment is actually harmful, the dilution principle could create a false sense of security by diluting the harmful effect. *RRR = A reduction in the Relative Risk