1
Indirect Comparisons: heroism or heresy
Neil HawkinsWith acknowledgements to Sarah DeWilde
2
Outline
• Brief introduction to indirect comparisons• Two practical examples• Heroism or Heresy?
3
A Taxonomy of Comparisons
A
B
A C
B C
A B
A C
B C
A B
Direct Comparison
Naive or Unadjusted Indirect Comparison: Absolute effect estimates from individual trial arms.
‘Adjusted’ indirect comparison: Difference between relative treatment
Network Meta-Analysis:Adjusted Indirect comparison extended to more complex networks of trial evidence
4
Adjusted Indirect Comparisons
Based on an assumption of transitivity
• HRAB = HRAC/HRBC
• Log Transformation: LN HRAB = LN HRAC – LN HRBC
5
Assumptions
• Transitivity on the chosen scale:A-B = (A-C)-(B-C)• Requires exchangeability of relative treatment effects:
– Between Subjects Within trials (randomisation)– Between trials including the same comparators
(pairwise meta-analysis)– Between trials comparing different comparators– Ultimately, between different treatment comparisons
6
Indirect comparisons: heroism or heresy?
7
Stuart Pocock: Heresy
“… their statistical methods are so complex, “an extension of multivariable Bayesian hierarchical random effects models for mixed multiple treatment comparisons” that many are mystified by whether the conclusions make sense.” from Safety of drug-eluting stents: demystifying network meta-analysis
www.thelancet.com Vol 370 December 22/29, 2007
8
Lu & Ades: Heroism
“. . . to ignore indirect evidence either makes the unwarranted claim that it is irrelevant, or breaks the established precept of systematic review that synthesis should embrace all available evidence”
. Stat Med 2004;23:3105–24.
9
Cochrane: Not Sure
“Indirect comparisons are not randomized comparisons, and cannot be interpreted as such. They are essentially observational findings across trials, and may suffer the biases of observational studies, for example due to confounding. “
Reference: Cochrane Handbook 2.4.6.
10
Ref: NEJM 359;12
11
Could the (-ve) results of PRoFESS have been predicted?
AS
P
AS
P +
ER
DP
CLO
P
CAPRIE
ESPRIT
ESPS2
12
Network Graph (Stroke Endpoint)
ASP
CLOP ASP + ER DP
0.92(0.8:1.07) 0.79(0.67:0.92)
13
ASP + ERDP vs. CLOP
• Indirect Comparison: ESPS2, ESPRIT & CAPRIE Trials– Odds Ratio 0.85 ( 0.66:1.06 )
• Direct Comparison: PRoFESS Trial– Odds Ratio 1.02 (0.93 to 1.11)
• What happened?
Indirect Comparison ASP + ERDP vs CLOP Odds Ratio 0.85 (0.69:1.06)
14
Kent & Thaler (2008)“the results of the PRoFESS trial show us once again that
the compelling logic of the transitive property, so reliable in mathematics, has little authority in the often illogical world of clinical trials”
“Although the inconsistency among trial results should make us examine the trials for differences in design or populations that might support explanatory hand-waving, it is also reasonable to conclude from these comparisons that efficacy should not be the sole, or perhaps even the major, determinant of treatment decisions for antiplatelettherapy after stroke.”
15
Extended Trial Network
AS
P +
CLO
P
AS
P +
ER
DP
AS
P h
igh
dose
AS
P lo
w d
ose
AS
P m
ed d
ose
CLO
P
ATC2002
CAPRIE
CHARISMA
ESPRIT
ESPS2
MATCH
16
Extended Network Graph
ASP med dose
ASP + CLOP
CLOP
ASP + ER DP
ASP high dose
ASP low dose
0.83(0.66:1.05)
1.08(0.93:1.25)0.84(0.64:1.08)
1.06(0.95:1.18)
1.08(0.94:1.25)
1.59(1.39:1.83)
1.31(1.08:1.6)
1.5(1.34:1.68)
17
ASP + ERDP vs CLOP
• Extended Network Meta-Analysis: – Odds Ratio 1.11 ( 0.87 to1.4)
• Direct Comparison: PRoFESS Trial– Odds Ratio 1.02 (0.93 to 1.11)
• Extended Network Meta-Analysis + PRoFESS– Odds Ratio 1.03 ( 0.94 to 1.13 )
18
Kent and Thaler (2008)“In the era of comparative effectiveness, when
multiple agents are pitted against one another, randomized trials often cannot be understood in isolation. Rather, they need to be interpreted in the context of a sometimes complex network of other similar or relevant evidence.
The reduction of such complex networks to treatment recommendations is not always straightforward, since different paths within the network may give inconsistent results, and the network may be incoherent”
19Ref: Statist. Med. 2007; 26:1237–1254
20
NMA of Anti-TNFs in RA
Adalimunab
0.0 0.5 1.0 1.5 2.0 2.5
Log Odds Ratio of ACR50
Anakinra
Etanercept
Infliximab
MTX
21Ref: Statist. Med. 2007; 26:1237–1254
22
Adjusted for Disease Duration
0.0 0.5 1.0 1.5 2.0 2.5
Log Odds Ratio of ACR50
Anakinra
Etanercept
Infliximab
Adalimunab
MTX
23
Final thought: exchangeability is implicit in clinical decision-making
RCT: A vs Placebo: Exchangeability
RCT: B vs Placebo:
Future Patient:
24
Heroism, Heresy, or Pragmatism
– Formal methods of indirect comparisons are invaluable for analysing more complex networks
– Assumptions required for indirect comparisons are related to the assumptions of generalisability
– Identifying and seeking to explain the heterogeneity and incoherence identified in networks of trial evidence should improve our understanding of comparative effectiveness