1
Critical AppraisalTherapy
J.EKO WAHONO. R.dr.SpS.M.KesSMF I.P.Saraf
RSUD Dr.Soetomo Surabaya
2
Principal of critical appraisal-primary research
• Involves 3 overall question
– What is the PICO of the study, and is it closed enough to your PICO ?
– How well was the study done ?– What do the results mean and could they
have been due to chance ?
3
Critical appraisal Tx
Study
Your question(PICO)
What do the result mean ?
How well wasThe study done ?
Is the study questionthe same as your question ?
4
Wheter wearing elastic stockings on long-haul flights helps to prevent deep vein trombosis
(DVT)
• Population/problem = passengers on long-haul flights
• Intervention = wearing elastic compression stockings
• Comparator/control= no elastic stockings
• Outcome = symptomless DVT
5
Critical appraisal Tx DVT
• Clinical question :– In passengers on long-haul flights, does
wearing elastic compression stockings, compared to not wearing elastic stockings, prevent DVT ?
• Search terms :– Based on the clinical question (PICO) – (flight*OR travel*) AND stockings*AND (DVT
OR trombosis)
6
Critical appraisal Tx DVT
• Search result :– PubMed Clinical Queries therapy,broad), 20
hits(referring to 5 studies and several reviews, including 1 recent Cochrane review)
– For exercise :• Scurr et al (2001). Frequency and prevention of symptomless DVT
in long-haul flights:a randomised trial
• The Lancet 357:1485-1489
• Authors’conclusion :– Wearing of elastic compression stockings during long-haul air
travel is associated with a reduction in symptomless DVT
7
Analyse
• How do we know that the results are valid and real ?
• Wearing elastic stockings is an intervention --- RCT ok
• In the real life – straight to the Cochrane systematic review
8
Question 1 : Is the PICO of the study close enough to your PICO ?
• If you find a study that will answer your clinical question – Study PICO match your PICO or not !
• Example your PICO– P = In patients with rheumatoid arthritis– I = Does taking anti-inflammatory drugs– C = no treatment or simple analgetics
– O = Increase or reduce fatique
9
If study PIC = your PICbut study O # your O
• Study O = reduce joint pain
• Your O = fatique
• However :– The study may report on some measure of
fatique as secondary outcome– Please decide quickly wheter to critize or not
10
Question 2 :How well was the study done ?
• The quality of an epidemiological study– Internal validity
• Free from bias & confounding factors
– Bias• The degree to which the result is skewed away
from the truth– Selection bias– Treament bias– Measurement bias
• To overcome bias RCT & Blinding
11
Question 2 :How well was the study done ?
• The quality of an epidemiological study– Internal validity
• Free from bias & confounding factors
– Confounding factors• Patients features & causal factors• To overcome CF
– Both Group are closely matched/similar– The management of the group is the same
12
How well was bias & confounding factors were avoided ?
• Check each stage of the study How fairly were : – the subjects recruitted (the “P”)– the subjects allocated to groups ( the I and C )– the study group maintained through equal
management and follow up of subjects ( the I and C )
– the outcome measured ( the O )
13
Stucture of a comparative health care research study
Study
question
Aim Study methods Critical appraisal
P Fair recruitment
Subject representative of the target pop
Large enaugh sample + randomly
R
I Fair allocation Randomly allocated
Adjust confounding (statitical adjustment/matching)
A
C Fair maintenance Manage grup equally
Follow up all subjects
M
O Fair measurement
Valid & unbiased outcome measure
Measure outcome
Blinded
Objective measure
M
B
o
14
Steps in critical appraisal of primary research - RAMMBO
• Recruitment :– Were the subjects representative of the target pop ?
• Allocation or adjustment– Was the tx allocation concealed before randomisation and were the
groups comparable at the start of the trial
• Maintenance :– Was the comparable status of the study groups maintained through
equal management and adequate follow up ?
• Measurement– Were the outcomes measured with
– Blinded subjects and assessors, and/or
– Objective outcomes ?
15
RecruitmentWere the subjects representative of target population ?
• If the subject are not representative – difficult to know to which pop may be applicable
• The best way to ensure – study group representative is to :– Recruit potential subjects
• Sequentially• At random from population
– Only apply exclusion criteria – relevant for study methods
• Excluding deaf people from study requiring subjects to listen the music
16
RecruitmentWere the subjects representative of target population ?
• Prefer large study, – because small study group
• imprecise estimate of the effects !
• Continuous outcomes 50 – 100 pt
• Binary outcomes :– Common event hundreds pt– Rare event thousands pt
17
RecruitmentWere the DVT trial subjects representative of target population ?
• Inclusion/exclusion criteria– For RCT
• difficult random sampling due to inform consent
– Clear idea who they do represent– Describe
• the severity,• duration and/or • risk level of the patients recruited
18
RecruitmentWere the DVT trial subjects representative of target population ?
• Volunteers– Were recruited by placing advertisements in paper– Passengers
• > 50 yo • Economy class • At least 8 hours flight within 6 weeks
– Various exclusions
• Size of study groups– 231 116 received stockings & 116 no– This seem small
• As a 10% DVT rate – 12 events
19
AllocationWere the study groups comparable ?
• It is vital the groups are matched – except for the interventions ( or
exposure/other indicator)
• Ways in which groups could differ – Age– Sex– Smoker/nonsmoker– Disease severity
• Random allocation
20
Allocationwere the DVT study groups comparable ?
• The paper states –– “Volunteers were randomised by sealed
envelopre to one of two groups”
21
Allocation Characteristic of DVT study groups
No stockings StockingsNumber 116 115
Pre-study
Age 62(56-68) 61(56-66)
Females 61(53%) 81(70%)
Varicose veins 41 45
Hb 142 140
During study
Hours flying 22 24
Day of stay 17 16
P < 0.01
22
Maintenancewas the comparable status of the study groups
maintained through equal management and adequate follow-up ?
• Once comparable groups have been set up stay a that way
• Equal management• Unequal tx invalidates result !
– In a trial of vit E in preterm infants– Vit E appeared to prevent retrolental fibroplasia ?– It was not !– Control groups 100% O2
– Tx groups not 100% O2
• because the babies were removed from O2 for freq dose of vit E
23
MaintenanceAdequate follow-up
• Inevitably, some subjects drop out, change groups or variously lost to follow up during study uncomparable groups !
• Check :– Subject at start = at the end– Subject are analysed in the groups that they
stated out in ( Intention-to-treat principle )
24
MeasurementWere the outcomes measured with Blinded subjects
and accessorrs and/or Objective measures ?
• Measurement bias– Human tendency to unfairly “nudge” results– Can be overcome by
• Blinding• Objective measurement
25
….MeasurementBlinding
• Best Double-blind trial
• Moderate Single blind
• Worst Not blinded
• Placebo effect
26
Question 3:What do the result mean ?
• Outcome measures– Binary– Continuous
• Are the results real and relevent ?
• Assesing change :– P-values (hypothesting)– Confidence interval (estimation)
27
Outcome measures for binary outcomes
Measure Meaning Example
Relative risk Risk of outcome in the Tx group/risk in the control group
RR – how many times more likely event in the tx group relative to control group
RR = 1 – no diff between 2 groups
RR < 1 – the tx reduces the risk of event
RR > 1 – the Tx increass the risk of the event
RR = 0.1/0.15= 0.67
RR < 1
The Tx – decrease the risk of death
28
Outcome measures for binary outcomes
Measure Meaning Example
ARR
Risk of event in the control group – risk event in the tx group
ARR = 0 – no diff
ARR + the Tx is beneficial
ARR - harmfull
ARR = 0.15 – 0.10 = 0.05 (5%)
The absolute benefit of Tx is a 5% reduction in the death rate
29
Outcome measures for binary outcomesRRR
Measure Meaning Example
RRR
ARR/risk of event in the control group
RRR Reduction in the rate of event in the Tx group relative to control group
RRR =ARR / Risk of event control group
30
Outcome measures for binary outcomesNNT
Measure Meaning Example
NNT
= 1 / ARR
NNT =The number of pts we need to treat in order to prevent bad event
31
P-values
• Are a measure of the probability that the result is purely due to chance
32
Confidence interval (CIs)• More informative > P – value• An estimate of the range of value that likely to
include the real value• 95% means :
– The range of values that have a 95% chance of including the real value
• If the 95% CI for the diff between Tx & Control group– Small– No overlap the “no effect “ point
The result is real ( that is, with a P-value < 0.05)
33
……Confidence interval (CIs)
Diff betweenTx vs Control
Null hypothesis(no effect)
A B C D
= point estimate
= CI
A : statistically sig result (p < 0.05 ) but low precisionB : statistically sig result (p < 0.05 ) with high precisionC : not statistically sig result (p > 0.05 ) with low precisionD : not statistically sig result (no effect ) with high precision
34
Intervention = usesful if …
• The 95% CI includes clinically important tx effects
• Statistically sig– Relates to the size of the effect and the 95%
CI in relation to the Null hypothesis
• Clinical importance– Relates to the size of effect and the 95% CI in
relation to a minimum effect that would be considered to be clinically importantce
35
……Clinical importance
Minimum clinicalImportant diff
Null hypothesis(no effect)
A B C D
= point estimate
= CI
A : Diff is statistically sig and clinical importanceB : Diff is not statistically sig but is clinical importanceC : Diff is statistically sig but not clinical importanceD : Diff is not statistically sig and not clinical importance
Recommended
