THE INCORPORATION OF META-ANALYSIS RESULTS

INTO EVIDENCE-BASED DECISION

MODELLINGNicola Cooper, Alex Sutton, Keith Abrams,

Paul LambertDepartment of Epidemiology & Public Health, University

of Leicester.PSI Meeting “Statistical Advances in Health

Technology Assessment ”10th June 2003

• Increasingly decision models are being developed to inform complex clinical/economic decisions

• Parameters can include: –clinical effectiveness, –costs, –disease progression rates, and –utilities

• Evidence based - use systematic methods for evidence synthesis to estimate model parameters with appropriate levels of uncertainty

                                     BACKGROUND

                                     

• Statistical error

• Systematic error

• Evidence relating to parameters indirectly

• Data quality, publication bias, etc.

SOURCES OF UNCERTAINTY IN

DECISION MODELS

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

81

0

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

81

0

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

81

0

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

81

0

RCT1 RCT2 RCT3 OBS1 OBS2 ROUTINE EXPERTDATA SOURCES

Gen. synthesisMeta-analysisEVIDENCESYNTHESIS

COMPREHENSIVE DECISION MODEL FRAMEWORK

DECISIONMODEL Stroke

No strokeTreating patients with atrial fibrillation?

Warfarin

No warfarin

Stroke

No stroke

Bleed

No bleed

Bleed

No bleed

Bleed

No bleed

Bleed

No bleed

….. …..….. …..

….. …..….. …..….. …..….. …..

….. …..….. …..

Clinical Effect

MODEL INPUTS

Adverse Events

Utility Cost

Opinion pooling

Bayes theorem In combination

                                     EXAMPLES

1) Net Clinical Benefit Approach

• Warfarin use for atrial fibrillation

2) Simple Economic Decision Model

• Prophylactic antibiotic use in caesarean section

3) Markov Economic Decision Model

• Taxane use in advanced breast cancer

1) Net Clinical Benefit Approach

• Warfarin use for atrial fibrillation

2) Simple Economic Decision Model

• Prophylactic antibiotic use in caesarean section

3) Markov Economic Decision Model

• Taxane use in advanced breast cancer

1. Meta-analyse available evidence to obtain a distribution for each model parameter using random effect models

2. Transformation of the pooled results, if necessary, and input into the model directly as a distribution and evaluate the model

3. All analyses (decision model and subsidiary analyses) implemented in one cohesive statistical model/program

4. Implemented in a fully Bayesian way using Markov chain Monte Carlo simulation within WinBUGS software

5. All prior distributions intended to be ‘vague’. Where uncertainty exists in the value of parameters (i.e. most of them!) they are treated as random variables

                                     GENERAL APPROACH

Warfarin for Non-Rheumatic Atrial Fibrillation

• Evidence that post MI, the risk of a stroke is reduced in patients with atrial fibrillation by taking warfarin

• However, there is a risk of a fatal hemorrhage as a result of taking warfarin

• Do the benefits outweigh the risks?

                                     EXAMPLE 1: NET CLINICAL

BENEFIT

EVALUATION OF NET BENEFIT

BENEFITS minus HARMS = NET CLINICAL BENEFIT

(if NCB >0 benefits outweigh harms)

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

EVALUATION OF NET BENEFIT

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

Multivariate riskequations

Meta analysisof RCTs

Metaanalysis of RCTs obs studies QoL study

EVALUATION OF NET BENEFIT

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

0.002 0.004 0.006 0.008 0.010 0.012 0.014

050

100

150

200

250

300

risk of bleed per year

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

810

-1.5 -1.0 -0.5 0.0 0.5 1.0

02

46

reduction in relative risk

0 20 40 60 80 100

0.0

0.1

0.2

0.3

0.4

Outcome ratio

Multivariate riskequations

Meta analysisof RCTs

Metaanalysis of RCTs obs studies QoL study

EVALUATION OF NET BENEFIT

Multivariate Risk Equation Data Net Benefit (measured in stroke equivalents)

% of

cohort

T hrombo -

embolism

rate (%

per year

(95% CI))

Mean

(s.e.)

Median

(95%

CrI)

Probability of

Benefit > 0

Simulated PDF

12

17.6 (10.5

to 29.9)

- 0.0004

(0.15)

0.06

( - 0.29 to

0.20)

54.2 %

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4

01

23

45

6

2 or 3 Clinical factors

EVALUATION OF NET BENEFIT

(Risk of stroke Relative reduction in risk of stroke)

- (Risk of fatal bleed Outcome ratio)

=

Net Benefit

0.002 0.004 0.006 0.008 0.010 0.012 0.014

050

100

150

200

250

300

risk of bleed per year

-2.95 -2.90 -2.85 -2.80 -2.75 -2.70 -2.65

02

46

810

-1.5 -1.0 -0.5 0.0 0.5 1.0

02

46

reduction in relative risk

0 20 40 60 80 100

0.0

0.1

0.2

0.3

0.4

Outcome ratio

Multivariate riskequations

Meta analysisof RCTs

Metaanalysis of RCTs obs studies QoL study

EVALUATION OF NET BENEFIT

                                     “TAKE-HOME”

POINTS 1

Net-benefit provides a transparent quantitative framework to weigh up benefits and harms of an intervention

Utilises results from two meta-analyses and allows for correlation induced where studies included in both benefit and harm meta-analyses

Credible interval for net benefit can be constructed allowing for uncertainty in all model parameters

Use of Prophylactic Antibiotics to Prevent Wound Infection following Caesarean Section

                                     EXAMPLE 2: SIMPLE DECISION TREE

No infection (1-p2) Cost with antibiotics

Yes

Infection (p2) Cost with antibiotics + Cost of treatment

Prophylactic antibiotics?

No infection (1-p1) Cost with no antibiotics

No

Infection (p1) Cost of treatment

1) Cochrane review of 61 RCTs evaluating prophylactic antibiotics use for caesarean section

2) Event data rare: use “Exact” model for RR 3) Meta-regression: Does treatment effect vary with patients’

underlying risk (pc)?

ln(RRadjusted ) = ln(RRaverage)+ [ln(pc) - mean(ln(pc))]4) Risk of infection without treatment from ‘local’ hospital

data (p1)5) Derive relative risk of treatment effect for ‘local’ hospital

(using regression equation with pc=p1)6) Derive risk of infection if antibiotics introduced to ‘local’

hospital (p2)

p2 = p1 * RRadjusted

                                     METHOD OUTLINE

                                     UNDERLYING BASELINE RISK

ln(R

ela

tive

Ris

k)

ln(control group risk) centred on mean)

ln(relative risk) fit

-2.5 -2 -1.5 -1 -.5 0 .5 1 1.5

-3

-2.5

-2

-1.5

-1

-.5

0

.5

1

1.5

2=0.24

(-0.28 to 0.81)

No treatment effect

Local hospital event rate

                                      RESULTSrr[1] sample: 20000

0.1 0.2 0.3 0.4 0.5

0.0 2.5 5.0 7.5 10.0

RR

diff[1] sample: 20000

-150.0 -100.0 -50.0

0.0 0.01 0.02 0.03 0.04

cost using antibiotics

-£49.53 (-£77.09 to -£26.79)

p1 sample: 20000

0.025 0.075 0.1 0.125

0.0

20.0

40.0

60.0p1

No infection (1-p2) Cost with antibiotics

Yes

Infection (p2) Cost with antibiotics + Cost of treatment

Prophylactic antibiotics?

No infection (1-p1) Cost with no antibiotics

No

Infection (p1) Cost of treatment

p2[1] sample: 20000

0.0 0.02 0.04

0.0 25.0 50.0 75.0 100.0

p20.02

(0.02 to 0.03)

p1 sample: 20000

0.025 0.075 0.1 0.125

0.0

20.0

40.0

60.0 0.08 (0.06 to 0.10)

p1

                                     SENSITIVTY OF PRIOR DISTRIBUTIONS

[1]

[2]

[3]

caterpillar plot

Cost difference -80.0 -60.0 -40.0 -20.0

[1] Gamma(0.001,0.001) on 1/variance

[2] Normal(0,1.0-6) truncated at zero on 1/sd

[3] Uniform(0,20) on 1/sd

[1]

[3]

[2]

Caterpillar plot

-80 -60 -40 -20Cost

“TAKE-HOME” POINTS 2

Incorporates M-A into a decision model adjusting for a differential treatment effect with changes in baseline risk

Meta-regression model takes into account the fact that covariate is part of the definition of outcome

Rare event data modelled ‘exactly’ (i.e. removes the need for continuity corrections) & asymmetry in posterior distribution propogated

Sensitivity of overall results to prior distribution placed on the random effect term in a M-A

                                     

Stable

 Progressive

Death

                           

   EXAMPLE 3: MARKOV MODEL

Response

Cycle length 3 weeks

QR , CR QS , CS

QP , CP

QD = 0

Quality of Life (Q)Cost (C)

PSP

PPD

PRP

PR PS

PP

Probability (P)

PSR

Taxanes - 2nd line treatment of advanced breast cancer

                                     MODEL PARAMETER

ESTIMATION PSR, TAX – The probability of moving from

stable to response in a 3 week period

3) Transformation of ln(odds)distrn to transition probability

)3/52/(1

/1

]42.01[1

)],(1[1

jjo ttP

mu.rsprtD sample: 12001

-5.0 0.0 5.0

0.0 0.5 1.0 1.5 2.0

2) Pooled ln(odds) distribution1) M-A of RCTs: Annual ln(odds) of responding

Odds - log scale.1 .25 1 5

Combined

Bonneterre

Sjostrom

Nabholtz

Chan

-0.3 (-0.9 to 0.3)

4) Apply to model

Respond

Stable

Progressive

Death

PSR

THE REMAINING PARAMETERS

• The Transition Probabilities need estimating for each intervention being compared

• Costs & Utilities can be extracted from the literature and synthesised using a similar approach within the same framework

                                     META-ANALYSES OF

LITERATURE(where required) No. of

studies Time in weeks

(95% Credible Interval) Progression-free time 3 25 (15 to 24)

Time to response from stable 1 12 (6 to 18) Time to progressive from response 1 35 (29 to 41)

Overall survival time 3 53 (35 to 74) Probabilities

Response rate 4 0.43 (0.29 to 0.58) % moving directly to progressive at stage 2. 1 0.13 (0.08 to 0.18)

% with infections / febrile neutropenia 3 0.18 (0.04 to 0.56) % hospitalised with infection / febrile neutropenia 1 0.08 (0.05 to 0.11)

% dying from infections / febrile neutropenia 1 0.01 (0.00 to 0.02) % discontinue treatment due to adverse event 3 0.16 (0.03 to 0.49)

% with Neutropenia grades 3 & 4 2 0.94 (0.82 to 0.98) % with Anaemia grades 3 & 4 2 0.03 (0.00 to 0.28) % with Diarrhoea grades 3 & 4 3 0.09 (0.06 to 0.14) % with Stomatis grades 3 & 4 3 0.08 (0.04 to 0.14) % with vomiting grades 3 & 4 2 0.03 (0.00 to 0.12)

% with fluid retention grades 3 & 4 3 0.05 (0.02 to 0.12) % with cardiac toxicity grades 3 & 4 1 0.00 (0.00 to 0.02)

                                     TRANSITION PROBABILITIES

FOR MODEL (Derived from M-As)

Transition Probabilities

(95% Credible Interval)

Infection/FN 0.09 (0.02 to 0.32)

Hospitalised due to infection/FN 0.04 (0.03 to 0.05)

Dying from infection/FN after hospitalisation 0.00 (0.00 to 0.01)

Discontinuation due to major adverse events 0.04 (0.04 to 0.16)

Adverse events – Neutropenia 0.50 (0.34 to 0.63)

Adverse events – Anaemia 0.01 (0.00 to 0.07)

Adverse events – Diarrhoea 0.02 (0.01 to 0.37)

Adverse events – Stomatis 0.02 (0.01 to 0.04)

Adverse events – Vomiting 0.01 (0.00 to 0.03)

Adverse events – Fluid retention 0.01 (0.00 to 0.03)

Adverse events – Cardiac toxicity 0.00 (0.00 to 0.01)

Transition directly to ‘progressive’ state 0.12 (0.08 to 0.18)

Transition ‘stable’ to ‘stable’ 0.65 (0.44 to 0.75)

Transition ‘stable’ to ‘response’ 0.16 (0.11 to 0.28)

Transition ‘stable’ to ‘progressive’ 0.18 (0.11 to 0.37)

Transition ‘response’ to ‘response’ 0.94 (0.93 to 0.95)

Transition ‘response’ to ‘progressive’ 0.06 (0.05 to 0.07)

Transition ‘progressive’ to ‘progressive’ 0.93 (0.79 to 0.96)

Transition ‘progressive’ to ‘death’ 0.07 (0.04 to 0.21)

                                     EVALUATION OF THE MODEL

• A cohort of 1,000 persons is run through the model over 35 3-weekly cycles (until the majority of people are dead) for each treatment option

• Costs and utilities are calculated at the end of each cycle and the average cost and utilities for an individual across all 35 cycles for each treatment option are calculated

• This process is repeated 4,000 times (each time different values from each parameter distribution are sampled)

Bayesian (MCMC) Simulations

-£4,000

-£2,000

£0

£2,000

£4,000

£6,000

£8,000

£10,000

-0.50 -0.40 -0.30 -0.20 -0.10 0.00 0.10 0.20 0.30 0.40 0.50

Incremental utility

Inc

rem

en

tal

co

st

Standard dominates

Taxane more effective but more costly

Taxane less costly but less

effective

Taxane

dominates

                                     COST-EFFECTIVENESS PLANE

NW NE

SW SE

                                     C-E ACCEPTABILITY CURVE

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

£0 £50,000 £100,000 £150,000 £200,000 £250,000

Value of ceiling ratio, Rc (£)

Pro

babi

lity

cost

-effe

ctiv

e

'vague' priors

                                     ELICITATION OF PRIORS

e.g. Response RateTaxane

x

x

x x x

x x x x x

x x x x x

x x

x x

x

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Standard

x

x

x x x x

x x x x x

x x x x x

x x

x x

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

                                     SENSITIVITY ANALYSIS

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

£0 £50,000 £100,000 £150,000 £200,000 £250,000

Value of ceiling ratio, Rc (£)

Pro

babi

lity

cost

-effe

ctiv

e

Expert opinion

'vague' priors

“TAKE-HOME” POINTS 3

Synthesis of evidence, transformation of variables & evaluation of a complex Markov model carried out in one unified framework (facilitating sensitivity analysis)

Provides a framework to incorporate prior beliefs of experts

ADVANTAGES OF APPROACH

Synthesis of evidence, transformation of variables & evaluation of a complex decision model carried out in a unified framework

Facilitates sensitivity analysis

Provides a framework to incorporate prior beliefs of experts

Allows for correlation induced where studies included in the estimation of more than one parameter

Uncertainty in all model parameters automatically taken into account

Rare event data modelled ‘exactly’ (i.e. removes the need for continuity corrections) & asymmetry in posterior distribution propagated

FURTHER ISSUES

1. Handling indirect comparisons correctly• E.g. Want to compare A vs. C but evidence

only available on A vs. B & B vs. C etc.• Avoid breaking randomisation

2. Necessary complexity of model?• When to use the different approaches outlined

above?

3. Incorporation of Expected Value of (Perfect/Sample) Information

4. Incorporation of all uncertainties

1. Handling indirect comparisons correctly• E.g. Want to compare A vs. C but evidence

only available on A vs. B & B vs. C etc.• Avoid breaking randomisation

2. Necessary complexity of model?• When to use the different approaches outlined

above?

3. Incorporation of Expected Value of (Perfect/Sample) Information

4. Incorporation of all uncertainties

REFERENCES

1. Cooper NJ, Abrams KR, Sutton AJ, Turner D, Lambert P. Use of Bayesian methods for Markov modelling in cost-effectiveness analysis: An application to taxane use in advanced breast cancer. Journal of the Royal Statistical Society Series A 2003; 166(3).

2. Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A. Comprehensive decision analytical modelling in economic evaluation: A Bayesian approach. Health Economics 2003 (In press)

3. Cooper NJ, Sutton AJ, Abrams KR. Decision analytical economic modeling within a Bayesian framework: Application to prophylactic antibiotics use for caesarean section. Statistical Methods in Medical Research 2002;11: 491-512.

4. Sutton AJ, Cooper NJ, Abrams KR, Lambert PC, Jones DR. Synthesising both benefit and harm: A Bayesian approach to evaluating clinical net benefit. (Submitted to Journal of Clinical Epidemiology).

1. Cooper NJ, Abrams KR, Sutton AJ, Turner D, Lambert P. Use of Bayesian methods for Markov modelling in cost-effectiveness analysis: An application to taxane use in advanced breast cancer. Journal of the Royal Statistical Society Series A 2003; 166(3).

2. Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A. Comprehensive decision analytical modelling in economic evaluation: A Bayesian approach. Health Economics 2003 (In press)

3. Cooper NJ, Sutton AJ, Abrams KR. Decision analytical economic modeling within a Bayesian framework: Application to prophylactic antibiotics use for caesarean section. Statistical Methods in Medical Research 2002;11: 491-512.

4. Sutton AJ, Cooper NJ, Abrams KR, Lambert PC, Jones DR. Synthesising both benefit and harm: A Bayesian approach to evaluating clinical net benefit. (Submitted to Journal of Clinical Epidemiology).