Download pdf - Critical Appraisal - UKMi...Critical Appraisal Steve Haigh Senior Medicines Information and Formularyormulary Pharmacist Sherwood Forest Hospitals, Nottinghamshire. [email protected]

Critical AppraisalCritical Appraisal

Steve HaighSteve HaighSenior Medicines Information and Senior Medicines Information and FFormularyormulary PharmacistPharmacistSherwood Forest Hospitals, Nottinghamshire.Sherwood Forest Hospitals, [email protected]@sfh--tr.nhs.uktr.nhs.uk

Critical (adj)1. Inclined

to

judge

severely

and

find

fault.2. Characterized

by

careful,

exact

evaluation

and

judgment:

a critical reading.

Appraisal (noun)An assessment

or

estimation

of

the

worth,

value,

or

quality

of

a

person

or

thing.

This lecture and other resources at This lecture and other resources at http://nww.nottstpct.nhs.uk/formulary/teaching/http://nww.nottstpct.nhs.uk/formulary/teaching/

Version 19.6.2015Version 19.6.2015

Presenter

Presentation Notes

The second one is the meant definition, but the first one is possibly better. Which seems harsh, but when you realise you are up against people trying to mislead you, it becomes more appropriate. Other useful words are skeptic (like the second definition) and cynic (the first definition).

mailto:[email protected]



http://nww.nottstpct.nhs.uk/formulary/teaching/

Identify key components of clinical trial design and apply theseIdentify key components of clinical trial design and apply these to a critical to a critical appraisal of the literatureappraisal of the literatureBe able to work out measures of effectivenessBe able to work out measures of effectiveness

AimsAims

FromFrom……In this In this noninferioritynoninferiority

trial, we randomly assigned 18,113 patients who had atrial fibrtrial, we randomly assigned 18,113 patients who had atrial fibrillationillationand a risk of stroke to receive, in a blinded fashion, fixed dosand a risk of stroke to receive, in a blinded fashion, fixed doses of dabigatran es of dabigatran --

110 mg or 150 mg 110 mg or 150 mg twice daily twice daily --

or, in an unblinded fashion, adjustedor, in an unblinded fashion, adjusted--dose warfarin. The median duration of the followdose warfarin. The median duration of the follow--up up period was 2.0 years. The primary outcome was stroke or systemicperiod was 2.0 years. The primary outcome was stroke or systemic

embolism.embolism.

Rates of the primary outcome were 1.69% per year in the warfarinRates of the primary outcome were 1.69% per year in the warfarin

group, as comparedgroup, as comparedwith 1.53% per year in the group that received 110 mg of dabigatwith 1.53% per year in the group that received 110 mg of dabigatran (relative risk withran (relative risk withdabigatran, 0.91; 95% confidence interval [CI], 0.74 to 1.11; P<dabigatran, 0.91; 95% confidence interval [CI], 0.74 to 1.11; P<0.001 for 0.001 for noninferioritynoninferiority))and 1.11% per year in the group that received 150 mg of dabigatrand 1.11% per year in the group that received 150 mg of dabigatran (relative risk, 0.66;an (relative risk, 0.66;95% CI, 0.53 to 0.82; P<0.001 for superiority). The rate of majo95% CI, 0.53 to 0.82; P<0.001 for superiority). The rate of major bleeding was 3.36% perr bleeding was 3.36% peryear in the warfarin group, as compared with 2.71% per year in tyear in the warfarin group, as compared with 2.71% per year in the group receivinghe group receiving110 mg of dabigatran (P = 0.003) and 3.11% per year in the group110 mg of dabigatran (P = 0.003) and 3.11% per year in the group

receiving 150 mg ofreceiving 150 mg ofdabigatran (P = 0.31). The rate of hemorrhagic stroke was 0.38% dabigatran (P = 0.31). The rate of hemorrhagic stroke was 0.38% per year in the warfarinper year in the warfaringroup, as compared with 0.12% per year with 110 mg of dabigatrangroup, as compared with 0.12% per year with 110 mg of dabigatran

(P<0.001) and(P<0.001) and0.10% per year with 150 mg of dabigatran (P<0.001). The mortalit0.10% per year with 150 mg of dabigatran (P<0.001). The mortality rate was 4.13% pery rate was 4.13% peryear in the warfarin group, as compared with 3.75% per year withyear in the warfarin group, as compared with 3.75% per year with

110 mg of dabigatran110 mg of dabigatran(P = 0.13) and 3.64% per year with 150 mg of dabigatran (P = 0.0(P = 0.13) and 3.64% per year with 150 mg of dabigatran (P = 0.051).51).

……To EnglishTo English

Or alternativelyOr alternatively……

Presenter

Presentation Notes

This slide is the abstract from the RELY study that will be looked at in the workshop session. Hopefully by the end of the session, everyone will be able to extract the relevant information from trials like this and assess its validity

If we use warfarin in 1000 people, If we use warfarin in 1000 people, 15 will come 15 will come a cropper but if we use dabigatran only 10 will.a cropper but if we use dabigatran only 10 will.

So this makes a difference for 1 in every 172So this makes a difference for 1 in every 172 patients.patients.

For exampleFor example……

Smiley face chartSmiley face chart

Cates plot: Cates plot: www.nntonline.net/visualrxwww.nntonline.net/visualrx//

Presenter

Presentation Notes

Or even better distil it down to something even a patient can understand. But before you get excited about dabigatran we need to check the quality of the data. We are not the Daily Mail who trot out any old nonsense without fact checking.

http://www.nntonline.net/visualrx/


Systematic examination of evidence to assess its validity and reSystematic examination of evidence to assess its validity and relevancelevance

What is Critical Appraisal?What is Critical Appraisal?

‘‘The conscientious, explicit and judicious use of current The conscientious, explicit and judicious use of current best evidence in making decisions about the care of best evidence in making decisions about the care of individual patientsindividual patients’’

SackettSackett

DL, Richardson WS, Rosenberg W et al. EvidenceDL, Richardson WS, Rosenberg W et al. Evidence--based medicine: how to practice and teach based medicine: how to practice and teach EBM. Churchill Livingstone, London. 1997EBM. Churchill Livingstone, London. 1997

What is Evidence Based Medicine?What is Evidence Based Medicine?

Presenter

Presentation Notes

Critical appraisal is part of the overall master plan - Evidence Based Medicine. Which is about using the best available evidence to make decisions about patients. To do this, you need to be able to analyse the information that is provided. Cf science based medicine – where the whole of scientific knowledge is taken into account. Eg trial data has to be weighed against the laws of physics.

Just because something is published doesnJust because something is published doesn’’t make it validt make it validIf the data is valid within the confines of the trial protocol, If the data is valid within the confines of the trial protocol, is it applicable to is it applicable to your patients?your patients?If the data is applicable to your patients, is it in a readilyIf the data is applicable to your patients, is it in a readily--understandable understandable form for your audience?form for your audience?

Why do Critical Appraisal?Why do Critical Appraisal?

CASP tools very usefulCASP tools very usefulDifferent tools for different types of trialsDifferent tools for different types of trials

What isWhat is……? Series from Bandolier? Series from Bandolier

BMJ publishing BMJ publishing –– How to Read a Paper How to Read a Paper (Trisha (Trisha GreenhalghGreenhalgh))

Bandoliers little bookBandoliers little book

How to critically appraise the evidenceHow to critically appraise the evidence

Presenter

Presentation Notes

The tools available help to give you a system to appraise trials and make sure you don’t miss anything. You can have a look at various tools after the course, but we’ll run through most of the key aspects to consider in this session. We’re going to run through a few aspects of trial design, and thinking about what we’d ideally want in a trial (if you know what you’d ideally like then you can compare what you have with it and see whether its good enough) Another barrier to understanding and appraising a trial is the use of specialist terminology. The use of ‘jargon’ isn’t bad in itself, as it allows very precise meanings to be communicated, but it can appear intimidating if you don’t know what it’s about. A lot of this session will be about working out what the terminology means, so you can pull the relevant information out

MetaMeta--analysis analysis Randomised Controlled Trial (RCT)Randomised Controlled Trial (RCT)CohortCohortCaseCase--controlcontrolCrossCross--overoverCase studies / case seriesCase studies / case seriesProspective / retrospectiveProspective / retrospective

Discussion:Discussion:

What do the above types of study actually mean? What type do yoWhat do the above types of study actually mean? What type do you think is u think is ideal? Is this true / feasible for all situations?ideal? Is this true / feasible for all situations?

Types of studyTypes of study

““In my experienceIn my experience””--Dr Mark Dr Mark CrislipCrislip ((QuackcastQuackcast podcast)podcast)

What about anecdotes?What about anecdotes?

What are the three most dangerous words in medicineWhat are the three most dangerous words in medicine……??

Presenter

Presentation Notes

Going to discuss the things you want to think about when critically appraising a RCT, but there are other types of study Meta-analysis – combines information from a number of trials RCT – usually considered gold standard Cohort – follow the same cohort of patients through time until the event happens Case control – event already happened, find similar people without event and compare histories Cross-over – every participant gets both trial and comparator treatments – act as own control Case studies – ‘we did this in a patient and this happened’

Headache stack graphHeadache stack graph

Placebo

.

Branded

placebo

Aspirin

Branded

aspirin

Nothing

Patients with headacheP

ain

relie

f

Signal

Noise!

Presenter

Presentation Notes

This trial shed some light on the problem of anecdotes. Discerning the signal from the noise. From the Book Meaning, Medicine and the 'Placebo Effect' by Daniel Moerman. Referencing: Analgesic effects of branding in treatment of headaches. A BRANTHWAITE, P COOPER. BMJ VOL 282 16 May 1981.

Overwhelming observable evidence?Overwhelming observable evidence?

Presenter

Presentation Notes

The actual highest level of evidence! Everyone can observe it for themselves and it’s indisputable. Doesn’t apply to many aspects of using drugs as the effects are often too subtle. Applies to things like “Stemming the blood flow from a severed artery is a good idea” and “Does propofol make you sleep”. But not “is propofol better than sevoflurane”

Aspects that need critical appraisalAspects that need critical appraisal••

Treatment selection and comparatorsTreatment selection and comparators••

Patient selectionPatient selection••

Treatment allocationTreatment allocation••

Treatment protocolTreatment protocol••

Data collectionData collection••

Data analysisData analysis••

ConclusionsConclusions

RCTsRCTs

Presenter

Presentation Notes

Simple Mnemonic: PICO – Population / Intervention / Control / Outcomes Lets have a look at each and some principles…

BiasBias

Discussion:Discussion:••

Where can bias come from?Where can bias come from?••

How can unintentional bias occur?How can unintentional bias occur?••

How can bias be minimised?How can bias be minimised?

RCTs RCTs ––

appraisal of trial designappraisal of trial design

Presenter

Presentation Notes

Whilst reviewing the various aspects of trial design and conduct, we’re going to be thinking about bias and how it can happen / how it can be minimised. Its important to remember that bias is often not the result of someone deliberately trying to skew the results, but is more likely to be due to trial design. Another important aspect to consider is that often bias cannot be eliminated, but instead measures should be taken to minimise it. The job of appraising the evidence is to assess whether bias is likely and to interpret your findings in light of this potential. For example, most new drug trials are sponsored by the company that sells the drug, introducing the potential bias of their commercial considerations impinging on the interpretation, but if we eliminated all company-sponsored trials from our evidence base we’d be left with nothing (as covered in the previous session, drug discovery is expensive, so who else is going to undertake research on an as-yet unproven therapy?)

What is the study treatment being compared against?What is the study treatment being compared against?

Atorvastatin 80mg vs Atorvastatin 80mg vs ……Targinact (oxycodone + naloxone) vs Targinact (oxycodone + naloxone) vs ……

Comparing against gold standard, or against something that will Comparing against gold standard, or against something that will make the make the new treatment look good?new treatment look good?

Treatment SelectionTreatment Selection

Presenter

Presentation Notes

Atorvastatin 80mg vs simvastatin 20mg – atorvastatin 80mg is more effective (!) Targinact vs oxycodone (no laxatives) – standard treatment would be oxycodone plus laxatives

Are the group of patients recruited particularly unwell / healtAre the group of patients recruited particularly unwell / healthy?hy?

Do they match your population?Do they match your population?Exclusion criteriaExclusion criteria

Are they allocated to treatment / control randomly?Are they allocated to treatment / control randomly?Stratified (random Stratified (random ≠≠

equal)equal)

Is the allocation truly random?Is the allocation truly random?Concealed allocationConcealed allocation

Are there enough patients?Are there enough patients?Power calculationPower calculation

Patient selection and treatment allocationPatient selection and treatment allocation

Presenter

Presentation Notes

If you were choosing patients for your study, how would you choose them? Need to be representative of the population you’d want to treat How would you randomise patients to control and treatment? What would you not do? Possibly think of simple ways of randomising that could accidentally (or subconsciously) introduce bias into your system Exclusion criteria example: pregabalin trials excluded patients who did not respond to gabapentin

Is the study openIs the study open--label / blind / doublelabel / blind / double--blind? Is this appropriate?blind? Is this appropriate?••

Balance between ideal and feasibleBalance between ideal and feasible

Is the study length appropriate?Is the study length appropriate?••

Balance between ideal and feasibleBalance between ideal and feasible

How are patients followed up?How are patients followed up?

How many patients make it to the final analysis? Is this reasonHow many patients make it to the final analysis? Is this reasonable?able?

Treatment protocol / data collectionTreatment protocol / data collection

Presenter

Presentation Notes

Question: what trials couldn’t be double-blind, single-blind etc? How could you minimise problems with this? (answers: surgical interventions can’t be double-blind, capsaicin patches (patients getting capsaicin patches needed oxycodone to manage the discomfort of having the patch) RELY: prospective, randomized, open-label, blinded endpoint evaluation (PROBE) – because control was warfarin, need INR monitoring which is difficult to blind. Q: can you think of a way that blinding could have happened? Consider how long you would like a trial follow-up period to be, and is this feasible? i.e. for acute pain, its valid to have a trial length of a few hours, but for stroke prevention? Parkinsons? How long would the patient be on the medicine in ‘real life’ (safety), and how long would you expect it to take before benefits were seen (effectiveness)

Where did the patients who donWhere did the patients who don’’t make it to the final analysis go and why?t make it to the final analysis go and why?

Moved house?Moved house?Adverse effects?Adverse effects?

Ineffective?Ineffective?

How is this factored into the final analysis?How is this factored into the final analysis?

Patient followPatient follow--upup

IntentionIntention--toto--treat (ITT) / modified ITTtreat (ITT) / modified ITT

Include everyone recruitedInclude everyone recruitedFairly true to use in the wildFairly true to use in the wild

Per protocol (PP)Per protocol (PP)

Include everyone who completes the trial as plannedInclude everyone who completes the trial as plannedPossibility of bias: only assessing highly motivated pillPossibility of bias: only assessing highly motivated pill--takers / treatment takers / treatment

successes (need to know why people left the trial early)successes (need to know why people left the trial early)

Different ways of managing loss to followDifferent ways of managing loss to follow--upup

Last observation carried forward (LOCF)Last observation carried forward (LOCF)

Useful for static conditionsUseful for static conditionsIf response is changeable over time (response to antiIf response is changeable over time (response to anti--ParkinsonParkinson’’s s

treatments?) may overtreatments?) may over--estimate effectsestimate effects

Different ways of managing loss to followDifferent ways of managing loss to follow--up (continued)up (continued)

Presenter

Presentation Notes

This problem should also be borne in mind if a trial is discontinued early.

What are we proving?What are we proving?

••Superiority?Superiority?Null hypothesis Null hypothesis –– no difference no difference –– if disprove, new treatment is if disprove, new treatment is ‘‘betterbetter’’

••NonNon--inferiority?inferiority?If new treatment is no worse than a specified margin (If new treatment is no worse than a specified margin (ΔΔ) then new ) then new treatment is nontreatment is non--inferiorinferior

••Equivalence?Equivalence?

Data analysisData analysis

Patient orientated outcomesPatient orientated outcomesSomething the patient will notice (death, stroke etc)Something the patient will notice (death, stroke etc)

Disease orientated outcomesDisease orientated outcomesMeasurement of the disease (PSA etc)Measurement of the disease (PSA etc)

What are we measuring?What are we measuring?

Hard / soft?Hard / soft?Proxy?Proxy?Subjective?Subjective?Recognised measure?Recognised measure?

StrokeStroke

Cholesterol levelCholesterol level

QoLQoL

Blood pressureBlood pressure

PSAPSA

MIMI

What outcomes are we measuring?What outcomes are we measuring?

Primary outcomesPrimary outcomesThe main focus of the trialThe main focus of the trial

What the power calculation tends to be focussed onWhat the power calculation tends to be focussed on

Secondary outcomesSecondary outcomesSubsidiary measureSubsidiary measure

Trial may not be powered to detect differencesTrial may not be powered to detect differences

Primary / secondary outcomesPrimary / secondary outcomes

Take care with subgroup analysesTake care with subgroup analysesTrials often not designed to reliably investigate subTrials often not designed to reliably investigate sub--groups, particularly if groups, particularly if

not specified in advance (postnot specified in advance (post--hoc analysis)hoc analysis)

Spurious correlations websiteSpurious correlations website……

www.tylervigen.comwww.tylervigen.com//

SubgroupsSubgroups

Presenter

Presentation Notes

Mention the trial showing gemini star sign patients did well. Spurious correlations website. Data mining: Metformin MR: • Chart review between October 2001 and May 2002, n=471 • Incidence of GI side effects in the first year of treatment with metformin IR and metformin XR • Subgroup analysis of patients switched from metformin IR to metformin XR – GI side effects pre and post- switch (n= 205) • Frequency of any GI adverse event in first year of treatment not significantly different between IR and XR groups (11.39% vs 11.94% respectively) • In subgroup switched from metformin IR to metformin XR, incidence of GI side effects in first year was 26.34% when on IR metformin, compared to 11.71% when switched to XR metformin (p = 0.0006, NNT = 6.8)

http://www.tylervigen.com/

http://www.tylervigen.com/

There are many different ways to represent data. All have partiThere are many different ways to represent data. All have particular cular advantages / uses and disadvantagesadvantages / uses and disadvantages

OddsOddsAbsolute RiskAbsolute RiskNumber needed to treatNumber needed to treatRelative RiskRelative Risk

Data presentationData presentation

OddsOdds““Odds of something happeningOdds of something happening””

Number of people something happened toNumber of people something happened toNumber of people something didnNumber of people something didn’’t happen tot happen to

Odds ratioOdds ratioOdds in treatment group / odds in control groupOdds in treatment group / odds in control group

OddsOdds

Presenter

Presentation Notes

Lets get odds out of the way first. Quite meaningless to interpret, only useful for meta analysis? The smaller the odds, the less likely an event is likely to occur (odds = 1 means event happens 50% of the time) An odds ratio of 1 means that the odds in the treatment group and control group are the same. Greater than 1 means that the event is more likely to happen in the treatment group, less than 1 means that the event happens more often in the control group

Treatment group: Treatment group: 24/100 had a DVT24/100 had a DVTControl group:Control group:

31/100 had a DVT31/100 had a DVT

Odds of event in treatment group: Odds of event in treatment group: 24/(10024/(100--24) = 0.3224) = 0.32Odds of event in control group:Odds of event in control group:

31/(10031/(100--31) = 0.4531) = 0.45

Odds ratio = 0.32/0.45 = 0.71Odds ratio = 0.32/0.45 = 0.71

1 means events are equally likely in both groups.1 means events are equally likely in both groups.scale goes from 0 (no events in treatment group) to infinity (noscale goes from 0 (no events in treatment group) to infinity (no

events in control group)events in control group)

Usually presented with: Usually presented with: 95% CI. 95% CI. eg eg 0.71 (0.61 0.71 (0.61 ––

0.94)0.94)or a p valueor a p value

eg eg 0.71 (p=0.04)0.71 (p=0.04)A 95% confidence interval that doesnA 95% confidence interval that doesn’’t include 1 or a p value <0.05 indicates t include 1 or a p value <0.05 indicates ‘‘statisticalstatistical significancesignificance’’

When events are rare, odds are similar to riskWhen events are rare, odds are similar to risk

Odds exampleOdds example

Presenter

Presentation Notes

What are the odds of a boy? 50:50. 50 it did happen to, 50 it didn’t. OR=1 Odd are used because statistical nerds like them for meta analysis type things but are fairly meaningless to us. We are now going to forget abut them.

Absolute riskAbsolute risk““Percentage of people where something happensPercentage of people where something happens””

Absolute RiskAbsolute Risk

Number of people something happened toNumber of people something happened toTotal number of people you looked atTotal number of people you looked at

Absolute risk reductionAbsolute risk reductionDifference in absolute risk between treatment and control groupsDifference in absolute risk between treatment and control groups



Absolute risk in treatment group: Absolute risk in treatment group: 24/100 = 0.24 (24%)24/100 = 0.24 (24%)Absolute risk in control group:Absolute risk in control group:

31/100 = 0.31 (31%)31/100 = 0.31 (31%)

Absolute risk reduction = 0.31 Absolute risk reduction = 0.31 ––

0.24 = 0.07 (7%)0.24 = 0.07 (7%)

Absolute risk exampleAbsolute risk example

Number needed to treat:Number needed to treat:

““Number of people you need to treat for one person to avoid an evNumber of people you need to treat for one person to avoid an eventent””

11

OrOr……same thingsame thing……

100100ARR ARR Differences in percentagesDifferences in percentages

Need to include the time periodNeed to include the time period

Number Needed to TreatNumber Needed to Treat



Absolute risk in treatment group: Absolute risk in treatment group: 24/100 = 24% 24/100 = 24% Absolute risk in control group:Absolute risk in control group: 31/100 = 31%31/100 = 31%

Absolute risk reduction = 31% Absolute risk reduction = 31% –– 24% = 7%24% = 7%

NNT = 100% / 7% ~ 15NNT = 100% / 7% ~ 15

You would need to treat 15 people with drug (x) instead of drug You would need to treat 15 people with drug (x) instead of drug (y) for (z) (y) for (z) years for 1 person to not have a DVTyears for 1 person to not have a DVT

NNT exampleNNT example

If we treated 100 people with drug (y) instead of drug (x), whatIf we treated 100 people with drug (y) instead of drug (x), what

is likely to is likely to happen?happen?

69 people who wouldn69 people who wouldn’’t have had a DVT on drug (x) anyway will not have a t have had a DVT on drug (x) anyway will not have a DVT on drug (y)DVT on drug (y)24 people who would have had a DVT on drug (x) will have a DVT o24 people who would have had a DVT on drug (x) will have a DVT on drug (y) n drug (y) as wellas well7 people who would have had a DVT on drug (x) will avoid that DV7 people who would have had a DVT on drug (x) will avoid that DVT on drug (y)T on drug (y)

Whole picture?Whole picture?

Smiley face chartSmiley face chartCates plot. http://www.nntonline.net/visualrx/


Cates plot: Cates plot: www.nntonline.net/visualrx/www.nntonline.net/visualrx/

Calculator: http://nww.nottstpct.nhs.uk/formulary/MI/Calculator-OddsRatio.xls


http://nww.nottstpct.nhs.uk/formulary/MI/Calculator-OddsRatio.xls

Relative risk: absolute risk in treatment group expressed relatiRelative risk: absolute risk in treatment group expressed relative to control ve to control group riskgroup risk

AR in treatment groupAR in treatment groupAR in control groupAR in control group

(Gives an unhelpful number similar to odds ratio)(Gives an unhelpful number similar to odds ratio)

Relative risk reduction: ARR expressed relative to control groupRelative risk reduction: ARR expressed relative to control group

riskrisk

AR in control group AR in control group ––

AR in treatment groupAR in treatment groupAR in control groupAR in control group

Relative risk (Boo, hiss!)Relative risk (Boo, hiss!)



Absolute risk in treatment group: Absolute risk in treatment group: 24/100 = 0.24 24/100 = 0.24 Absolute risk in control group:Absolute risk in control group:

31/100 = 0.3131/100 = 0.31

Relative risk reduction = (0.31 Relative risk reduction = (0.31 ––

0.24)/0.31 = 0.22 (22%)0.24)/0.31 = 0.22 (22%)

Relative risk reduction exampleRelative risk reduction example

Bang goes the theory…

Presenter

Presentation Notes

Relative risk reduction is a useful measure of treatment effectiveness, but it takes away any appreciation of the background event rate, so when explaining to a lay audience it can appear to over-represent treatment effectiveness for conditions with low event rates. As an example, most people would probably assume that a relative risk reduction of 22% would mean that about 1 in 4 people taking the drug will benefit, because we’ve lost the perspective of the number of people who would be fine anyway (22% of people who would have had a DVT don’t have one, but it glosses over the 69 people who wouldn’t have had a DVT anyway)

DropashedloadDropashedload©© halves death rate!halves death rate!

Treatmentdeaths pa

Placebodeaths pa

RRR ARR NNT Cost(£100pp pa)

2/100 4/100 50% 2% 50 £5000

2/10000 4/10000 50% 0.02% 5000 £500,000

20/100 40/100 50% 20% 5 £500

Presenter

Presentation Notes

Yes, it may halve death rate, but what does that mean.

A new anticoagulant, A new anticoagulant, shinyboxagatranshinyboxagatran, has been , has been brought to market. In a trial (2 years) of high risk brought to market. In a trial (2 years) of high risk patients, 97 / 2,432 on patients, 97 / 2,432 on shinyboxagatranshinyboxagatran

had a stroke had a stroke

or thrombotic event, compared to 131 / 2,629 on or thrombotic event, compared to 131 / 2,629 on cheaparincheaparin

(current gold standard treatment)(current gold standard treatment)

In groups, derive a (true) stat that fits your world view:In groups, derive a (true) stat that fits your world view:Group 1: Lead Pharmacist at Big PharmaGroup 1: Lead Pharmacist at Big PharmaGroup 2: Lead Pharmacist for cynicismGroup 2: Lead Pharmacist for cynicism

Group 3: Lead Pharmacist for patient understandingGroup 3: Lead Pharmacist for patient understanding

CynicismCynicism: : An attitude of scornful or jaded negativity, especially a generaAn attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.l distrust of the integrity or professed motives of others.

c.f. Skepticism: A questioning attitude towards knowledge.c.f. Skepticism: A questioning attitude towards knowledge.

Quick test:Quick test:

Presenter

Presentation Notes

S: absolute risk: 0.04 C: absolute risk: 0.05 S: odds: 0.04 C: odds: 0.052 OR: 0.78 ARR: 0.01 (1%) RRR: 0.2 (20%) NNT: 100 For every 100 patients treated with shinyboxagatran, 95 will not have a thrombotic event (and wouldn’t have if they’d been on cheaparin anyway), 4 will have a thrombotic event despite being on shiny, and one who would have had a thrombotic event on cheaparin will avoid it on shiny

The optionsThe options……S: absolute risk: 0.04 (4%)S: absolute risk: 0.04 (4%)C: absolute risk: 0.05 (5%)C: absolute risk: 0.05 (5%)S: odds: 0.041S: odds: 0.041C: odds: 0.052C: odds: 0.052

OR: 0.78OR: 0.78ARR: 0.01 (1%)ARR: 0.01 (1%)RRR: 0.2 (20%)RRR: 0.2 (20%)NNT: 100NNT: 100For every 100 patients treated with For every 100 patients treated with shinyboxagatranshinyboxagatran, 95 , 95 will not have a thrombotic event (and wouldnwill not have a thrombotic event (and wouldn’’t have if t have if theythey’’d been on d been on cheaparincheaparin anyway), 4 will have a anyway), 4 will have a thrombotic event despite being on shiny, and one who thrombotic event despite being on shiny, and one who would have had a thrombotic event on would have had a thrombotic event on cheaparincheaparin will will avoid it on shinyavoid it on shiny

Presenter

Presentation Notes

Print this slide at end for handouts

GraphsGraphs

Number of people suffering a stroke

53

53.5

54

54.5

55

55.5

56

56.5

Control Active

Group

% Number of people suffering a stroke

Presenter

Presentation Notes

Check your axes! Both graphs are deliberately small to replicate fine print in journals.

GraphsGraphs

Number of people suffering a stroke

0

10

20

30

40

50

60

70

80

90

100

Control Active

Group

% Number of people suffering a stroke

Presenter

Presentation Notes

Check your axes! Both graphs are deliberately small to replicate fine print in journals.

Efficacy and safety of tapentadol prolonged release for chronic osteoarthritis pain and low back painAdvances in Therapy

June 2010, Volume 27, Issue 6, pp 381-399 http://link.springer.com/article/10.1007%2Fs12325-010-0036-3

Presenter

Presentation Notes

Heres another cropped graph example to exaggerate an effect. There is a difference

http://link.springer.com/journal/12325

http://link.springer.com/journal/12325/27/6/page/1

http://link.springer.com/article/10.1007%2Fs12325-010-0036-3

4

5

6

7

8

9

10

11

Presenter

Presentation Notes

That’s the full scale! But ive mislead you too now

11

10

9

8

7

6

5

4

3

2

1

0No pain

Pain as bad as you can imagine

Presenter

Presentation Notes

This is the most honest representation as everyone started with pain on average around 8-9

Any trial only looks at a Any trial only looks at a samplesample of the of the populationpopulation, and we use this to estimate , and we use this to estimate what would happen in the whole populationwhat would happen in the whole population……

There is therefore a chance that the sample of the population weThere is therefore a chance that the sample of the population we

looked at looked at were not representativewere not representative……

The p value is the probability that any difference seen between The p value is the probability that any difference seen between treatment and treatment and control groups was just by chance.control groups was just by chance.

The 95% confidence interval is the range of values that you are The 95% confidence interval is the range of values that you are 95% sure the 95% sure the population value lies between.population value lies between.

SignificanceSignificance

Presenter

Presentation Notes

Significance tends to be a concept that is misunderstood. In an ideal world, we would want to know the benefits of the drug in the whole population, but this is obviously not possible. We therefore take a sample of the population and use them to estimate the effects in the population. We try to make sure that our sample is a fair representation of the population, but, because events are random, we can never guarantee that what happened in our sample is truly representative of what would happen in the population – the significance measures are a representation of how confident we can be in this As an example of how a sample may not be representative of the wider population (just by chance), try number of birthdays each month in the group (be prepared for this to go horribly wrong!) – national statistics show generally an even spread through the year (slightly more in summer), but this is unlikely to happen in a group of 20-odd (impossible to happen unless you have a group of 12 or 24)

Significance Significance ––

two ways to measure ittwo ways to measure it……

Treatments are the same

New treatment is better

Old treatment is better

What you found in your sampleThe range you

are 95% sure the population’s value lies in

P value: the probability that the treatments are actually the same, and what you saw in

your sample arose just from chance

Confidence IntervalsConfidence Intervals

Treatments are the same

New treatment is better

Old treatment is better

a.

b.

c.

d.

Presenter

Presentation Notes

So, based on what we know, what do the following graphical representations of confidence intervals mean? Discussion about the width of the confidence interval (the smaller, the more accurate the trial), and whether the confidence interval crosses the ‘no effect’ line. Which show new treatment is better? Which show its worse? Which don’t show anything with significance? On click, the end bars disappear and the trial size boxes appear to convert the diagram into a pseudo Forest plot – size of the boxes represent trial size - the larger the trial, the narrower the confidence interval, so the more likely the result you get is reflective of the population On the next click, it adds a diamond at the bottom for a pseudo meta-analysis for a quick mention.

Odds Ratio 0.7 (95% CI 0.64 Odds Ratio 0.7 (95% CI 0.64 ––

0.76)0.76)

ThThis is means the OR for your sample is 0.7, and that you are 95% sure tmeans the OR for your sample is 0.7, and that you are 95% sure that the hat the populationpopulation’’s OR is between 0.64 and 0.76. s OR is between 0.64 and 0.76. ThThis difference is probably not due to is difference is probably not due to chance as the CI doesnchance as the CI doesn’’t overlap 1.t overlap 1.

Deaths Deaths on drug = 2.4%, on placebo 2.7%. (on drug = 2.4%, on placebo 2.7%. (p=0.p=0.1515))

The p value of 0.The p value of 0.1515 means that there 15% chance that the difference between means that there 15% chance that the difference between treatment and control groups occurred by chance (the smaller thetreatment and control groups occurred by chance (the smaller the value, the value, the more confident you are in the result). This is more than 5% so imore confident you are in the result). This is more than 5% so is not s not ‘‘statistically statistically significantsignificant’’..

Quick test:Quick test:

New antihypertensive drug, reduces blood pressure by 2mmHg, p<0.New antihypertensive drug, reduces blood pressure by 2mmHg, p<0.001001

Statistically significant, but will the patient notice?Statistically significant, but will the patient notice?Relate the findings to your patient cohort (comes back to patienRelate the findings to your patient cohort (comes back to patient orientated t orientated

outcomes vs disease orientated outcomes)outcomes vs disease orientated outcomes)

Statistical significance vs clinical significanceStatistical significance vs clinical significance

Misuse of significanceMisuse of significance

N Results ARR RRR NNT P value

20 5/10 dead vs4/10 dead

10% 20% 10 P=0.7

200 50/100 dead vs40/100 dead

10% 20% 10 P=0.2

2000 500 dead vs400 dead

10% 20% 10 P=0.001

6,000,0

00,000500,000 vs 499,900

0.00002% 0.02% 60,000,000 P=0.05

P values are dependent on n (and the value/spread of the two results)

Bad PharmaBad Pharma

Presenter

Presentation Notes

Start with the background of the people… here we know there is a bias

Presenter

Presentation Notes

Why should we be cautious? http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Scott Reuben Scott Reuben scumbagscumbag

CLASS trial. Celecoxib

Presenter

Presentation Notes

More reason to be cautious… Big Pharma are THIS evil. Peter Jüni. Are selective COX 2 inhibitors superior to traditional non steroidal antiinflammatory drugs? BMJ 2002;324:1287–8

CLASS trial. Celecoxib scam

6 months 12 months

% Ulcer complications

Presenter

Presentation Notes

Peter Jüni. Are selective COX 2 inhibitors superior to traditional non steroidal antiinflammatory drugs? BMJ 2002;324:1287–8

How did this paper get picked for publication?How did this paper get picked for publication?

If you were a drug company, would you push for publication if a If you were a drug company, would you push for publication if a trial trial showed your drug was worse?showed your drug was worse?

If you were an editor of a journal, would you prefer to publish If you were an editor of a journal, would you prefer to publish a paper that a paper that showed an amazing leap forward, or one that showed that the new showed an amazing leap forward, or one that showed that the new product product

is about the same / worse than the old one?is about the same / worse than the old one?

Why did Why did youyou pick this paper?pick this paper?

Is it the only one you could get access to, or did you do a systIs it the only one you could get access to, or did you do a systematic ematic literature review?literature review?

A final thing to think aboutA final thing to think about……

Essential reading!Essential reading!