Escher 3.2: towards e ective, transparent and …drugis.org/files/valkenhoef-pres-cluster2010.pdfEscher 3.2: towards e ective, transparent and accountable assessment of bene t-risk

Introduction Meta-analysis Network meta-analysis Benefit-risk analysis Discussion

Escher 3.2: towards effective, transparent andaccountable assessment of benefit-risk

using information technology and evidence synthesis

Gert van Valkenhoef

Department of Epidemiology, University Medical Center Groningen (NL),Faculty of Economics and Business, University of Groningen (NL)

Escher 3.x Cluster Meeting, 8 Dec 2010Utrecht, The Netherlands


Outline

1 Introduction2 Meta-analysis3 Network meta-analysis4 Benefit-risk analysis5 Discussion

After every part, there will bean opportunity to askquestions.


Outline

1 Introduction2 Meta-analysis3 Network meta-analysis4 Benefit-risk analysis5 Discussion

After every part, there will bean opportunity to askquestions.


Escher 3.2 Goals

Develop a drug information system:

Effective knowledge access and management

Answer drug efficacy and safety questions

in an efficient, transparent and accountable waywithin and across compoundsfor a broad audience (including regulators)

Improve consistency in regulatory decision making

Based on systematic review and meta-analysis


Effective knowledge access: problems

Review of existing systems:

Evidence-based decision making time-consuming/error-prone

No comprehensive source of trial information existsTrial information is insufficiently structured

Missed opportunities to introduce more structure

Trial registration, regulatory submission and systematic review

It is unclear how the information should be structured

Prototypes should be developed now, to discover thisRelated manuscripts:1) G. van Valkenhoef, T. Tervonen, B. de Brock, H. Hillege, Deficiencies in the transfer and availability of clinicalevidence in drug development and regulation. Manuscript under review.2) T. Tervonen, E.O. de Brock, P.A. de Graeff and H.L. Hillege (2010). Current status and future perspectives onDrug Information Systems.Proceedings of the 18th European Conference on Information Systems (ECIS2010),June 6-9, 2010, Pretoria, South Africa.


Prototype: global requirements

Interviews with major stakeholders

To develop the overall vision for the prototype

Database of clinical trialsAnswer efficacy/safety questionsStreamline benefit-risk decision makingFor regulatory authoritiesUsing aggregated data


Aggregate data

Regulatory submissions in Europe contain aggregate data

EMA, SmPC, Galvus (EMEA/H/C/000771 -II/0007), updated 2010-04-27.


Aggregate data

Journal articles report aggregate data

Chouinard G, Saxena B, Belanger MC, Ravindran A, Bakish D, Beauclair L, et al. A Canadian multicenter,double-blind study of paroxetine and fluoxetine in major depressive disorder. J Affect Disord. 1999;54:39-48.


Aggregate data

Trials registered with ClinicalTrials.gov have aggregate results

GlaxoSmithKline, “Controlled-release Paroxetine in Major Depressive Disorder (Double-blind, Placebo-controlledStudy)”, ClinicalTrials.gov NCT00866294, updated October 14, 2010.


ADDIS: Aggregate Data Drug Information System

Assisted evidence synthesis and benefit-risk assessment

Based on a database of clinical trials

Focussed on aggregated dataRelated manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.


Development of ADDIS: concurrent engineering

Software Development

Methodology Research

Open problems Knowledge, methods Case studies

Feedback, use cases

Open problems





Open problems

Knowledge, methods Case studies

Feedback, use cases

Open problems





Open problems Knowledge, methods

Case studies

Feedback, use cases

Open problems






Feedback, use cases

Open problems






Feedback, use cases

Open problems






Feedback, use cases

Open problems


Development of ADDIS: agile

ADDIS requirements highly uncertain

Only vague goals can be set

Much is expected to be discovered ‘on the way’

Agile software development

No full up-front specification of requirements

But: short-term plans and periodic re-evaluation

Supported by 2-3 part-time programmers (since Oct 2009)Related manuscripts:4) G. van Valkenhoef, T. Tervonen, B. de Brock, D. Postmus, Product and Release planning practices for ExtremeProgramming. Proceedings of the 11th International Conference on Agile Software Development (XP2010).5) G. van Valkenhoef, T. Tervonen, B. de Brock, D. Postmus, Quantitative release planning in ExtremeProgramming. Manuscript under review.



ADDIS requirements highly uncertain

Only vague goals can be set

Much is expected to be discovered ‘on the way’

Agile software development

No full up-front specification of requirements

But: short-term plans and periodic re-evaluation

Supported by 2-3 part-time programmers (since Oct 2009)Related manuscripts:4) G. van Valkenhoef, T. Tervonen, B. de Brock, D. Postmus, Product and Release planning practices for ExtremeProgramming. Proceedings of the 11th International Conference on Agile Software Development (XP2010).5) G. van Valkenhoef, T. Tervonen, B. de Brock, D. Postmus, Quantitative release planning in ExtremeProgramming. Manuscript under review.




Development of ADDIS: open

Open development (http://drugis.org/)

Nightly builds (daily), development builds (bi-weekly)

Release: ca. every 3 months

Mailing list

Subscribe if you’re interested!

Public issue tracker

Anyone can report bugs and track progressRoadmap: short-term plans

http://drugis.org/






Development of ADDIS: open source

Open source

Aiming for scientific impact

Ensures others will be able to continue the project

Anyone worried about bugs can review the source code

Allows us to re-use many existing OSS components


Development of ADDIS: open source


Questions?


Introduction

ADDIS global requirements:

Database of clinical trials

Answer efficacy/safety questions

Streamline benefit-risk decision making

For regulatory authorities

Using aggregated data

Intermediate goal: ‘dynamic Cochrane’ (automated meta-analysis)

Store trials in sufficient detail to do meta-analysis

Discover required data-model


Introduction







Intermediate goal: ‘dynamic Cochrane’ (automated meta-analysis)

Store trials in sufficient detail to do meta-analysis

Discover required data-model


Meta-analysis

Hansen et al. Ann Intern Med 2005;143:415-426


Meta-analysis



Meta-analysis



Meta-analysis



Meta-analysis in ADDIS

Supported since ADDIS v0.4 (December 2009)

Database of trials + characteristics + outcomes

Development of data model

Related manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.


Limits of meta-analysis (1)

Hansen et al. (2005) systematic review:

46 studies comparing n = 10 second-generation AD

On efficacy (HAM-D responders) and adverse events



Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2




Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2




Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2




Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2

Hansen et al. (2005) systematic review:


Only 3 meta-analyses, all against fluoxetine



Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2

How to compare paroxetine, sertraline and venlafaxine?

Can we compare sertraline/venlafaxine?

Only one direct trialIgnoring the 11 trials sertr-fluox-venlaIs this justified?

When comparing fluox/parox or fluox/sertr?

Can we ignore the 3 parox-sertr trials?

Parox as comparator → same conclusions?



Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2









Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2



Only one direct trialIgnoring the 11 trials sertr-fluox-venla

Is this justified?






Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2









Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2









Fluoxetine

Paroxetine

Sertraline

Venlafaxine

6

5

6

3 1

2








Conclusion

Meta-analysis is good if we compare two drugs

It is problematic for more

Selection bias: choice of common comparator?Are results of different comparisons consistent?

We need a way to include all trials/drugs


Questions?


Introduction







Intermediate goal: automated network meta-analysis

Meta-analysis of > 2 drugs

No existing software does this

Immediate value to scientific community


Introduction







Intermediate goal: automated network meta-analysis

Meta-analysis of > 2 drugs

No existing software does this

Immediate value to scientific community


Network meta-analysis


Paroxetine

Bupropion

(1)

Duloxetine

(1)

Mirtazapine

(2)

Venlafaxine

(2)

Sertraline

(3)

(1)

Escitalopram

(2)

Fluoxetine

(8)

(2)

(1)

(1)

(7)

Fluvoxamine

(2)

(6)

Citalopram

(1)

(3) (1) (2)

(1) (2)

Network meta-analysis: include all evidence in one analysis


Network Meta-Analysis models

Network meta-analysis models are difficult to specify

Automated in ADDISRelated manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.6) G. van Valkenhoef, T. Tervonen, B. de Brock, H. Hillege, Algorithmic Parameterization of Mixed TreatmentComparisons. Manuscript under review.7) G. van Valkenhoef, B. de Brock, H. Hillege, Automating network meta-analysis. Initiated (conference paper).




Automated in ADDIS

Related manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.6) G. van Valkenhoef, T. Tervonen, B. de Brock, H. Hillege, Algorithmic Parameterization of Mixed TreatmentComparisons. Manuscript under review.7) G. van Valkenhoef, B. de Brock, H. Hillege, Automating network meta-analysis. Initiated (conference paper).




Automated in ADDISRelated manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.6) G. van Valkenhoef, T. Tervonen, B. de Brock, H. Hillege, Algorithmic Parameterization of Mixed TreatmentComparisons. Manuscript under review.7) G. van Valkenhoef, B. de Brock, H. Hillege, Automating network meta-analysis. Initiated (conference paper).


Example: data

Study Fluox Parox VenlaChouinard et al, 1999 67/101 67/102De Wilde et al, 1993 25/41 24/37Fava et al, 1998 31/54 32/55Fava et al, 2002 57/92 64/96Gagiano, 1993 27/45 30/45Schone and Ludwig, 1993 9/52 20/54Alves et al, 1999 30/47 25/40De Nayer et al, 2002 27/73 37/73Dierick et al, 1996 95/161 107/153Rudolph and Feiger, 1999 52/103 57/100Silverstone and Ravindran, 1999 77/121 84/128Tylee et al, 1997 58/170 67/171Ballus et al, 2000 23/43 25/41McPartlin et al, 1998 128/178 137/183


Example: consistency

Fluox

Parox Venla

df ,vdf ,p

dp,v

pair-wise OR network OR

df ,p 1.24 (0.92, 1.67)

1.22 (0.92, 1.61)

df ,v 1.30 (1.03, 1.65)

1.34 (1.08, 1.67)

dp,v 1.20 (0.80, 1.82)

1.11 (0.82, 1.50)



Fluox

Parox Venla

df ,vdf ,p

dp,v


df ,p 1.24 (0.92, 1.67)

1.22 (0.92, 1.61)

df ,v 1.30 (1.03, 1.65)

1.34 (1.08, 1.67)

dp,v 1.20 (0.80, 1.82)

1.11 (0.82, 1.50)



Fluox

Parox Venla

df ,vdf ,p

dp,v


df ,p 1.24 (0.92, 1.67)

1.22 (0.92, 1.61)

df ,v 1.30 (1.03, 1.65)

1.34 (1.08, 1.67)

dp,v 1.20 (0.80, 1.82)

1.11 (0.82, 1.50)



Fluox

Parox Venla

df ,v = df ,p + dp,vdf ,p

dp,vassume consistency: direct andindirect estimates lead to thesame conclusions.


df ,p 1.24 (0.92, 1.67)

1.22 (0.92, 1.61)

df ,v 1.30 (1.03, 1.65)

1.34 (1.08, 1.67)

dp,v 1.20 (0.80, 1.82)

1.11 (0.82, 1.50)



Fluox

Parox Venla

df ,v = df ,p + dp,vdf ,p

dp,vassume consistency: direct andindirect estimates lead to thesame conclusions.


df ,p 1.24 (0.92, 1.67) 1.22 (0.92, 1.61)df ,v 1.30 (1.03, 1.65) 1.34 (1.08, 1.67)dp,v 1.20 (0.80, 1.82) 1.11 (0.82, 1.50)


Consistency

We assume consistency: direct and indirect estimates lead to thesame conclusions.

Estimate all relative effects simultaneously

Including all studies

Leading to consistent conclusions

Also estimate missing comparisons


Extended example

Fluox

Parox Venla

Sertr

Escit

Complexity → consistency greaterconcern

Pair-wise against Fluox, Escitexcluded

Yet, evidence suggests Escit>Fluox

How do the other drugs compare?

Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example

Fluox

Parox Venla

Sertr

Escit





Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example

Fluox

Parox Venla

Sertr

Escit





Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example

Fluox

Parox Venla

Sertr

Escit





Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example

Fluox

Parox Venla

Sertr

Escit





Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example

Fluox

Parox Venla

Sertr

Escit





Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example

Fluox

Parox Venla

Sertr

Escit





Escit 0.59 (0.37, 0.94) 0.69 (0.41, 1.15) 0.74 (0.44, 1.25) 0.81 (0.53, 1.24)Fluox 1.18 (0.91, 1.52) 1.27 (0.99, 1.63) 1.38 (1.10, 1.72)

Parox 1.08 (0.77, 1.51) 1.17 (0.86, 1.59)Sertr 1.09 (0.80, 1.48)

Venla


Extended example: rank probability

Venla > Fluox, Escit > Fluox significant

Not much can be said about the rest

Which should we consider the best?

Which is the worst?

From a Bayesian perspective,

these questions are perfectly reasonable!

We can estimate the probability of Fluoxetine being

‘best’ (rank 1)‘second best’ (rank 2). . .‘worst’ (rank 5)






Which is the worst?










Which is the worst?










Which is the worst?










Which is the worst?








Obstacles

Before the conclusions under consistency can be accepted:

Possible inconsistency should be evaluated:

First, by assessing the studies for exchangeabilitySecond, by statistical means (inconsistency/node-split model)

Assess convergence & run-length of the MCMC simulation

Reasonable priors have to be specified

All of these are research topics

And have (preliminary) implementations in ADDISRelated manuscripts:6) G. van Valkenhoef, T. Tervonen, B. de Brock, H. Hillege, Algorithmic Parameterization of Mixed TreatmentComparisons. Manuscript under review.7) G. van Valkenhoef, B. de Brock, H. Hillege, Automating network meta-analysis. Initiated (conference paper).


Questions?


Break!


Introduction







Intermediate goal: quantitative benefit-risk model

Based on clinical trials or meta-analysis

Making trade-offs explicit

Making uncertainty explicit


Introduction







Intermediate goal: quantitative benefit-risk model

Based on clinical trials or meta-analysis

Making trade-offs explicit

Making uncertainty explicit


A simple stochastic model

The ‘Lynd & O’Brien’ model:

Based on cost-effectiveness analysis techniques

Compares 2 alternatives

On 2 criteria (benefit vs. risk)

Sample (∆B,∆R) values from a joint distribution

Plot them on a plane

Count how many points are below the threshold µLynd, LD and O’Brien, BJ (2004), “Advances in risk-benefit evaluation using probabilistic simulation methods: anapplication to the prophylaxis of deep vein thrombosis.” Journal of Clinical Epidemiology 57(8):795–803.


Lynd & O’Brien example: set up (ADDIS)






Lynd & O’Brien example: data

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

probability

dens

ity

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

probability

dens

ity

Fluoxetine

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

probability

dens

ity

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

probability

dens

ity

Sertraline

−0.1 0.0 0.1 0.2 0.3

01

23

45

6

−0.2 −0.1 0.0 0.1 0.2

01

23

45

Difference

HAM-D

Dropouts

57/92 70/96

24/92 26/96


Lynd & O’Brien example: results (ADDIS)


Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µB better

A better

p = aa+b

count b

count a


Benefit-risk plane


+R

isk

A+

Ris

kB

µ

B better

A better

p = aa+b

count b

count a


Benefit-risk plane


+R

isk

A+

Ris

kB

µ

B better

A better

p = aa+b

count b

count a


Benefit-risk plane


+R

isk

A+

Ris

kB

Trade-off

Trade-off

µ

B better

A better

p = aa+b

count b

count a


Benefit-risk plane


+R

isk

A+

Ris

kB

µThe acceptability threshold.

We are willing to ‘pay’ µ

units risk to get 1 unit of

benefit.

B better

A better

p = aa+b

count b

count a


Benefit-risk plane


+R

isk

A+

Ris

kB

µThe acceptability threshold.

We are willing to ‘pay’ µ

units risk to get 1 unit of

benefit.

B better

A better

p = aa+b

count b

count a


Benefit-risk plane


+R

isk

A+

Ris

kB

µB better

A better

p = aa+b

count b

count a


Example: acceptability curve (ADDIS)


SMAA BR analysis

The Lynd & O’Brien model is limited to 2x2 problems.

Stochastic Multi-criteria Acceptability Analysis (SMAA)allows m × n problems:

m alternativesevaluated on n criteriaperformance of alternative i on criterion j : Ci,j ∼ f (ci,j )

Related manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.8) T. Tervonen, G. van Valkenhoef, E. Buskens, H. Hillege, D. Postmus, A stochastic multi-criteria model forevidence-based decision making in drug benefit-risk analysis. Manuscript under review.9) G. van Valkenhoef, T. Tervonen, J. Zhao, B. de Brock, H. Hillege, D. Postmus, Multi-criteria benefit-riskassessment using network meta-analysis. Partial manuscript.


SMAA BR analysis

SMAA models for benefit-risk:

Can be based on a single trial (8)

Or (network) meta-analysis (9)

And is implemented in ADDIS (3)Related manuscripts:3) G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, H. Hillege, ADDIS: a decision support system forevidence-based medicine. Manuscript under review.8) T. Tervonen, G. van Valkenhoef, E. Buskens, H. Hillege, D. Postmus, A stochastic multi-criteria model forevidence-based decision making in drug benefit-risk analysis. Manuscript under review.9) G. van Valkenhoef, T. Tervonen, J. Zhao, B. de Brock, H. Hillege, D. Postmus, Multi-criteria benefit-riskassessment using network meta-analysis. Partial manuscript.


SMAA example (ADDIS)

SMAA modelbased on networkmeta-analysis.



Measurements (input distributions).



Model without preference information.



Model without preference information.



Preferences for severe depression.



Severe depression results.



Preferences for mild depression.



Mild depression results.


Relevance: EMA BR methodology project

Approach/method Relevance to regulators UsefulnessProbabilistic simulation Can illuminate the risk/benefit trade-off when uncertainty is a major

feature of a regulatory decision.Medium

Bayesian statistics Can integrate evidence and its uncertainty, both pre- and post-approval, with multiple criteria in decision models.

High

MCDA Multi-criteria decision analysis extends decision theory to accommo-date multiple, conflicting objectives. Provides common units of valuefor both benefits and risks.

High

Table: MTC/SMAA integrates 2 of 3 quantitative approaches rated’High’ on usefulness, and 1 rated ’Medium’.

EMA (2010). Benefit-risk methodology project work package 2 report: Applicability of current tools and processesfor regulatory benefit-risk assessment. EMA/549682/2010.


Questions?


Escher 3.2 progress

ADDIS software:

database of trialsautomated (network) meta-analysisstochastic benefit-risk models

Research:

Survey of exisiting information systemsAutomating network meta-analysisDevelopment of benefit-risk method

Publications:

Presented at a number of conferencesSeveral papers under review (5)Journal and conference paper in preparationCase study being initiated


Escher 3.2 progress

ADDIS software:


Research:


Publications:



Escher 3.2 progress

ADDIS software:


Research:


Publications:



Future plans

Case studies

(Network) meta-regression

Handle covariates (dose, baseline severity, . . . )Refine the data model

Extend benefit-risk model

Hierarchical model/value treeQualitative attributes

More links with data sources, data sharing

A collaborative web portal?


Questions?


Thank you!

Documents

Escher 3.2: towards e ective, transparent and …drugis.org/files/valkenhoef-pres-cluster2010.pdfEscher 3.2: towards e ective, transparent and accountable assessment of bene t-risk