Decision Analysis and Benefit : Risk Determinations · Decision Analysis and Benefit : Risk Determinations Telba Irony, PhD Chief, General and Surgical Devices Branch Division of

Decision Analysis

and

Benefit : Risk Determinations

Telba Irony, PhD

Chief, General and Surgical Devices Branch

Division of Biostatistics

Office of Surveillance and Biometrics

Center for Devices and Radiological Health

1

The Science of Small Clinical Trials

November 27-28, 2012

2

Outline

1. Introduction: What is Decision Analysis?

2. A Decision Problem

3. Reducing Uncertainty

4. Value of Information

5. Pre-market Decisions

6. Factors for Benefit : Risk Determinations

1. Introduction: What is Decision Analysis?

• A set of tools to describe, inform and analyze

decision making in the presence of uncertainty

• A method of communication and insight by making

“values” (objectives, preferences) that drive

decisions explicit (transparency)

• A way of combining values and data, accounting for

differences in values and uncertain information

• A method that allows for the determination of

where resources should be allocated to obtain

information that has the highest value for decision

making important for rare diseases 3

Decision Analysis and Small Clinical Trials

Decision Analysis can be used to weigh the benefit of obtaining more information against the loss incurred to obtain such information

4

Larger sample size, better trial

Gains: reduction in the uncertainty

Losses: delay in approval

patients may die or suffer irreversible

damage waiting for treatment

etc.

5

Why is it hard to make decisions?

Preferences: we may not know what we prefer

Most decisions need to be made in the presence of

uncertainty

It is difficult to “formulate” the decision problem.

If we can’t formulate it, we cannot solve it

6

We could use our guts to make decisions……..

Or our brains……..

7

A patient has a huge headache and goes to the doctor fearing

he has a brain tumor.

Does the patient need surgery?

Decisions, decisions…..

Unknown State of Nature: presence or absence of tumor

D+: Patient has a brain tumor

D- : Patient does not have a brain tumor

2. A Decision Problem

Uncertainty…

Formulation of a Decision Problem

8

How would mathematical decision making work in the

patient’s situation?

6 Steps for the formal decision making process

1. Identify all possible outcomes (States of Nature)

• D+ : Patient has a tumor

• D- : Patient does not have a tumor

2. Identify all possible actions or decisions

• Operate

• Don’t Operate

• (Collect more information: use a diagnostic test)

9

3. Quantify the uncertainty about the outcomes via

probabilities:

• P(D+) : Probability that the patient has a tumor

• P(D-) : Probability that he doesn’t have a tumor

4. List all possible consequences

See the decision tree….

The doctor examines the patient and collects the

patient history. He makes an assessment based on

his knowledge and on the prevalence of that tumor

in such patients :

P(D+) = 40% P(D-) = 60%

10

(more information)

yes

no

Tumor

yes

no

Tumor

Operate

yes

no

Good

Bad

Terrible

Great!

Consequences Decision Tree

11

5. Quantify the relative preferences among the

consequences through numerical values (utilities)

Do we know what we prefer?

Great! Good Bad Terrible

Subjective!

Do we know how much we prefer one consequence as

compared to others?

My preferences

Subjective!

Establish a scale

Best consequence: Max: 10 “utiles”

Worst consequence: Min: 0 “utiles”

12

6. Choose the Decision with the Maximum Expected

Utility

Great! Good Bad Terrible

10 8 2 0

13

6. Calculating the Decision with the Maximum

Expected Utility

8 x 0.4 + 2 x 0.6 = 4.4 utiles

yes

no

Tumor

yes

no

Tumor

Operate

yes

no

Good

Bad

Terrible

Great! 10

8

2

0

U (Operate) = U (Good) P (Tumor) + U (Bad) Pr (No Tumor)

U (Operate) = 8 x Pr (D+) + 2 x Pr (D-) =

14

U (Don’t Operate) = U (Great) P(D-) + U (Terrible) P(D+)

Optimal Decision: Don’t Operate!

yes

no

Tumor

yes

no

Tumor

Operate

yes

no

Good

Bad

Terrible

Great!

U (Operate) = 4.4 U (Don’t Operate) = 6

10

8

0

2

U (Don’t Operate) = 10 x 0.6 + 0 x 0.4 = 6 (“utiles”)

15

Diagnostic test? Sample more?

What is the probability that the patient has a tumor

given his test result?

3. Reducing Uncertainty

The doctor made prior probability assessments:

P(D+) = 40% ; P(D-) = 60%

Question

Is there value in gathering more information?

Sensitivity of the Test: Probability of a true positive

Specificity of the Test: Probability of a true negative

16

Assume the patient tests Positive: T+

What is the probability that the patient has the tumor

given he tested positive?

Bayes Theorem

Bayes Theorem relates:

Sensitivity: probability of testing + given he has tumor

with

Positive predictive value: probability of having a tumor

given tested +

P(T+|D+) with P(D+|T+)

17

Bayes Theorem

Prior probability of the tumor: P(D+) = 0.4

Sensitivity of the Test: 99%

Specificity of the Test: 90%

Patient tests Positive: T+

D-)P(D-)|P(T))P(DD|P(T

))P(DD|P(T )T|P(D

Probability he has a tumor given he tested positive = 87%

Plug into the formula, a.k.a.

Bayesian crank …...

Back to the Decision Problem

18

6 Steps

1. Possible outcomes (states of nature) • D+ : Patient has a tumor

• D- : Patient does not have a tumor

2. Actions or decisions • Operate

• Don’t Operate

• Collect more information

3. Quantify the uncertainty via probabilities

• Probability that he has a tumor given T+: 87%

• Probability that he doesn’t have a tumor given T+: 13%

4. List all possible consequences (decision tree)

5. Quantify the relative preferences through utilities (values)

6. Choose the Decision with the maximum expected utility

19

6. Calculating the Decision with the Maximum Expected Utility

U (Operate |T+) = U (Good) P (D+|T+) + U (Bad) Pr (D-|T+)

U (Operate |T+) = 8 x 87% + 2 x 13% = 7.22 (“utiles”)

yes

no

Tumor

yes

no

Tumor

Operate

given T+

yes

no

Good

Bad

Terrible

Great! 10

0

2

8

20

U(Don’t Operate |T+)=U(Great) P(D-|T+) + U(Terrible) P(D+|T+)

U(Don’t Operate |T+) = 10 x 13% + 0 x 0.77% = 1.3 (“utiles”)

yes

no

Tumor

yes

no

Tumor

Operate

given T+

yes

no

Good

Bad

Terrible

Great! 10

0

2

8

21

U (Operate |T+) = 8 x 87% + 2 x 13% = 7.22

U (Don’t Operate |T+) = 10 x 13% + 0 x 0.77% = 1.3

Optimal Decision: Operate!

Same calculations as before with “less uncertainty”,

i.e. with the probabilities updated by the test result

22

Value of Information (T+) =

U(Operate |tested T+) - U(Don’t Operate |no test)

=

7.22 – 6 = 1.22 (“utiles”)

4. Value of Information

4.1. Value of Information for a positive test

23

4.2. Value of Information for a negative test

If the patients tests negative (T-)

Apply Bayes Theorem

Probability he has the tumor when the test is neg. 0.74%

Probability he does not have a tumor when the test is neg. 99.26%

10

0

2

8 yes

no

Tumor

yes

no

Tumor

Operate

given T-

yes

no

Good

Bad

Terrible

Great!

24

U(Operate |T-) = 8 x P(D+|T-) + 2 x P(D-|T-) = 2.04

U(Don’t Operate |T-)= 0 x P(D+|T-) + 10 x P(D-|T-)= 9.92

Optimal Decision: Don’t Operate!

Value of Information (T-) =

U(Don’t Operate |tested T-) - U(Don’t Operate |no test) =

9.92 – 6 = 3.92 (“utiles”)

Calculate the utility of each decision with updated probabilities

25

Expected information for the test: before we see the

test result

Value of Information (T) =

Value Info (T+) x P(T+) + Value Info (T-) x P(T-)

Value of Information of the Test

In this case:

Value Info(T) = 1.22xP(T+) + 3.92xP(T-) = 2.69 (“utiles”)

26

Test?

Whole Problem:

Sequential Decisions

Test Result yes

no

no

yes

no

yes Good

Bad

Tumor

Operate

yes Terrible

Great!

Tumor

no

yes

yes Good

Bad

Tumor

Operate

no

yes

no

Terrible

Great!

Tumor

no

T-

yes

T+

yes Good

Bad

Tumor

Operate

no

yes

no

Terrible

Great!

Tumor

no

yes

27

5. Pre-market decisions

Possible decisions

Approve a device

Don’t approve a device

Get more information (other trial: more sample)

Elicit utilities (values) from stakeholders (the

Center, Physicians, Patients) to establish the

threshold for approval versus non-approval: some

loss in effectiveness may (or may not) compensate

a certain gain in safety.

To make pre-market decisions we need to:

28 28

Different Utility Assessments

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

0.95

Pr(safe|data)

Pr(

effe

ctiv

e|d

ata

)

Device treatments

1. Life threatening

disease: no

alternative is

available

Idea: Utility Assessments and Thresholds for Approval

29


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

0.95

Pr(safe|data)

Pr(

effe

ctiv

e|d

ata

)

Device treatments

1. Life threatening

disease: no alternative is

available

2. Life threatening

disease: alternative is

available


30


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

0.95

Pr(safe|data)

Pr(

effe

ctiv

e|d

ata

)

Device treatments

1. Life threatening

disease: no alternative is

available

2. Life threatening


available

3. Other devices exist but

nothing works well

(osteoarthritis)


31


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.05

0.15

0.25

0.35

0.45

0.55

0.65

0.75

0.85

0.95

Pr(safe|data)

Pr(

effe

ctiv

e|d

ata

)

Device treatments

1. Life threatening

disease: no

alternative is

available

2. Life threatening


available

3. Other devices exist

but nothing works

well (osteoarthritis)

4. Several good

alternatives exist

(stricter)


32 32

Factors to Consider for Benefit - Risk Determinations

(Medical Device Premarket Approval and De Novo Classifications)

http://www.fda.gov/downloads/MedicalDevices/DeviceRegulat

ionandGuidance/GuidanceDocuments/UCM296379.pdf

Guidance Document

issued on March 28, 2012

33

• Benefits

Type of benefit

Magnitude of benefit

Probability of patient experiencing the benefit

Duration of benefit

• Risks

Extent (severity, types, numbers and rates) of the harmful event

Probability of a harmful event

Duration of the harmful event

For diagnostics: risks of false positives and false negatives

Factors

34

• Uncertainty

Design, conduct, quality of clinical studies

Generalization of results

• Severity and chronicity of the disease

• Patient tolerance for risk and perspective on benefit

• Availability of alternative treatments

• Risk mitigation

• Post-market information (what can be deferred?)

• Novel technology for unmet medical need

Additional Factors

35 35

Thank you

36

Question 1

a. Bayes Theorem is the tool to update current information in the

light of new data obtained from a clinical trial.

b. Additional patients in a trial may reduce uncertainty and make

decision making easier.

c. The value of information obtained from 10 more subjects can be

calculated when one uses Bayesian statistics.

d. A positive diagnostic test dramatically increases the probability

that a patient has the disease being tested for.

e. If a diagnostic test does not reduce uncertainty, it has no value

for decision making

(1) a, b, and d are true

(2) c, d, and e are false

(3) All statements are true

(4) a is true and d is false

(5) a is true and c is false

37

a. A small trial is advised when the loss from the delay of waiting for

a larger sample is greater than the value of information obtained

from more subjects.

b. Decision Analysis is an algorithm that tells what decision should

be made.

c. Public Health decision making involves preferences, values, and

subjectivity.

d. Waiting to make a decision is itself a decision that involves risks

and benefits.

e. Decision analysis is a set of mathematical tools that specify a

process to make decisions.

Question 2

(1) a, b, and d are true

(2) d and e are false

(3) All are true

(4) a is true and d is false

(5) a is true and b is false