© 2014 SynteractHCR. All rights reserved. SHARED WORK. SHARED VISION. Pitfalls in Companion Diagnostics Don't underestimate the power of conditional probabilities

© 2014 SynteractHCR. All rights reserved. SHARED WORK. SHARED VISION.

Pitfalls in Companion Diagnostics

Don't underestimate the power of conditional probabilities


2

A mystery in numbers and its solution

In this presentation, Dr. Stephan de la Motte, Chief Medical Advisor, defines:

The conditional nature of diagnostics

Companion diagnostics as an entire therapeutic strategy

Sensitivity and specificity with regard to diagnoses

The power of conditional probabilities

How to determine the best hypothesis in study protocols when you have a companion diagnostic


3

Diagnostics is conditional

A biomarker predicts...suspicion– if used in clinically healthy

diagnosis– if disease symptoms are given

prognosis– if diagnosis is given

response– if treatment is given

A treatment produces...response (with a probability) in the right patient

side-effects (with a probability) in any patient

A diagnostic assay leads – in the end – to a therapy outcome.


4

Biomarker X scenario

Biomarker X is target for new drug, but does not influence response to standard of care nor normal course of disease

– E.g., tumor-specific kinase inhibited specifically by new drug

Biomarker X is present in 5% of patients with the disease– Companion diagnostic prevents many patients being treated with

new drug unnecessarily

Assay has 97% sensitivity and 98% specificity– Good quality assay

New drug in patients with marker X 80% respondersStandard of care 10% responders

– Huge advantage of new drug over standard of care

This is an ideal scenario for Companion Diagnostics, right?


5

An ideal scenario, right?

WRONG!Because...

80% is the true effectiveness of the new drug

60% is the observed responder rate in a clinical trial

The clinical trial...

significantly underestimates the true value of the new drug

leads to more than twice as many patients getting the new drug unnecessarily

WHY?


6

A word of caution...

"Companion Diagnostics" is...

not only a diagnostic

not only a drug

it is an entire therapeutic strategy!

The next slide shows this strategy of the Biomarker X scenario as a tree of consecutive, conditional probabilities.


7

Decisional algorithm tree

Biomarker Xreally present?

Assay pos.,new drug?


Response tonew drug?

Effect ofstandard?


Effect ofstandard?

Yes

No0.03

0.97

Yes

No

0.95

0.05

No

Yes

0.20

0.80

Yes

No

0.02

0.98

Responder ~ 4%

Non-responder ~ 2%

Normal course ~ 84%

Effect < 0.1%

Responder ~ 0.2%

Non-responder ~ 1%

Normal course ~ 0.1%

Effect ~ 9%

Patients actuallytreated with new drug

No

Yes

0.90

0.10

No

Yes

0.90

0.10

No

Yes

0.90

0.10

Patients misallocated:Not to be treated with new drug, but are, orto be treated with new drug, but should not be.

Biomarker(invisible)

Apparent interpretationAssay outcome Therapy outcome


8

What we see is not what is real!

We never see the real target

We see only... the result of an assay... the outcome of a treatment

(Percent numbers taken from previous slide #7, rightmost column)


9

Sensitivity and Specificity

Don't overlook a diagnosis!

Don't make a healthy person sick!

Sensitivity and specificity indicate the quality of a diagnostic test and both should be close to 100%

But are these criteria – alone – meaningful?


10

The “100% Assay”

"All of you have cancer!"Everyone who has cancer is diagnosed to have cancer

=> 100% sensitivity– No disease is overlooked, because no one is declared as healthy

"None of you have cancer!"Everyone who is healthy is declared to be healthy

=> 100% specificity– No false diagnosis, because no diagnosis is made at all

Yes, this is nonsense, because the value of a diagnostic assay depends on much more than sensitivity or specificity.


11

Predictive values

Predictive value - If we make an observation, how likely is it that it is correct?

Sensitivity and specificity are always given

Predictive values are often overlooked and their usefulness is underestimated

If we see it, is it real? – Important for treatment decisions

Positive predictive value

If we don't see it, does this mean it isn't there? – Important for screening exams

Negative predictive value

PPV and NPV quantify the reliability of assay results in actual populations.

PPV and NPV depend on prevalence.


12

Biomarker X in 5% of patients

Number of patients

'X' present5

'X' absent95

Assay positive('X' detected)

True positives4.85

False positives1.9

PPV0.72

Assay negative('X' not detected)

False negatives0.15

True negatives93.1

NPV0.998

Sensitivity0.97

Specificity0.98

Purple numbers = GivenAll other colors = Derived

Interpretation:Although NPV is almost perfect, PPV is not satisfactory.PPV 0.72 means that 28% of patients with a positive assay don't actually have the Biomarker X target.Apparent responder rate of patients treated with new drug is 60%.


13

Biomarker X in 50% of patients

Number of patients

'X' present50

'X' absent50

Assay positive('X' detected)

True positives48.5

False positives1

PPV0.98

Assay negative('X' not detected)

False negatives1.5

True negatives49

NPV0.97

Sensitivity0.97

Specificity0.98

Purple numbers = GivenAll other colors = Derived

Interpretation:Properties of assay and of drug unchanged; only prevalence is changed.Now, both NPV and PPV are almost perfect.Apparent responder rate is now ~79%, very close to the theoretically best 80%.


14

How to make it more complicated

Previous scenarios were based on the simplified assumption that Biomarker X predicts only the response to a new drug.

In real life, however:

Biomarkers are not only drug targets, but are associated with a better or worse prognosis, even under standard of care

A new drug is targeting biomarker X, but it may show some efficacy also in patients who do not carry X

The next slide shows which probabilities must be modified...

to simulate biomarker X prognosis without new drug

to simulate drug efficacy in patients without biomarker X


15

Modified decisional algorithm tree

Biomarker-associated poor prognosis.

New drug with off-target efficacy.

Outcome changed, not always detectable.(Compare with previous tree on slide #7.)

Biomarker Xreally present?




Effect ofstandard?


Effect ofstandard?

Yes

No0.03

0.97

Yes

No

0.95

0.05

No

Yes

0.20

0.80

Yes

No

0.02

0.98

Responder ~ 4%

Non-responder ~ 1%

Normal course ~ 84%

Effect < 0.1%

Responder ~ 0.5%

Non-responder ~ 1%

Normal course ~ 0.1%

Effect ~ 9%

No

Yes

0.95

0.05

No

Yes

0.75

0.25

No

Yes

0.90

0.10

Biomarker(invisible)

Apparent interpretationAssay outcome Therapy outcome


16

What to do if you have a companion diagnostic...

Check out the prevalence of the biomarker in your target population!

Calculate predictive values!

Work through the tree of conditional probabilites and see if it makes sense!

Calculate different plausible scenarios (sensitivity analysis – "What if?")

If PPV (positive predictive value) is troublesome, utilize a 2nd diagnostic test...sequentially (2nd only if 1st is positive)

...parallel (outcome is positive only if both are positive)

A 2nd diagnostic should be complementary to the 1st...one with high PPV

...the other with high NPV

Avoid raising false expectations!Write the best hypothesis in your study protocol!


THANK YOU!

Dr. Stephan de la MotteChief Medical Advisor

www.SynteractHCR.com

Global Headquarters5759 Fleet Street, Suite 100Carlsbad, CA 92008

+1 760 268 8200

Documents

© 2014 SynteractHCR. All rights reserved. SHARED WORK. SHARED VISION. Pitfalls in Companion Diagnostics Don't underestimate the power of conditional probabilities