Statistics in Drug Regulation:The Next 10 Years

Thomas PermuttDirector, Division of Biometrics II

Center for Drug Evaluation and Research

The views expressed are those of the speaker and not necessarily of FDA.

Statutory Standards• Substantial evidence of efficacy• All tests reasonably applicable for safety• Balance not explicit, but history clear

Risk/Benefit• Formerly:

– Very good evidence about direction of mean treatment effect

• Too good? No.– Adverse events:

• Common: statistical but unimportant• Rare: nonstatistical but important

What’s New?

• Rofecoxib• Rosiglitazone• LABA

Rofecoxib• Heart attacks• Large outcome trial

– which was trial in new indication• Now need outcome studies for COX-2 and

maybe nonselective

Rosiglitazone• Nissin meta-analysis• We do meta-analysis• You do meta-analysis• You do outcome trial, maybe

Meta-analysis• Hard• Nonstatistical• Statistical• Both different in regulatory setting

Meta-analysis: Nonstatistical• Better information, but …• Doesn’t fit usual protocol-driven regulatory

framework, either• Do it anyway, but …• Nobody will believe you (or us), so … ?

– sensitivity analysis important

Meta-analysis: Statistical• Fixed vs. random effects

– doesn’t matter much for global null, but– this doesn’t apply to noninferiority

• Attributable vs. relative risk– relative risk “stable” across settings

• different length of study, at least– but attributable risk is what matters– what about zeroes

• Nissin to Congress: “no information”

What triggers this?• “Signal”

– Class effects– Someone else’s meta-analysis

• For diabetes, everything• For COX-2, probably everything

– other COX?

LABA• Believed to cause death

– not “side effect,” death from asthma• Effect mostly “seen” without steroid• So, with steroid?

With Steroid, Show What?• Noninferior to nothing?

– i.e., combination therapy vs. steroid• Noninferior to realistic alternative?

– e.g., increased dose of steroid– why not superior?

• because of benefit

• Interaction with steroid?– i.e., already “know” without steroid: Is with different?– maybe can’t do without steroid anyway

Noninferiority Margins• Not “1.3”

– COX-2– diabetes– asthma!

• Risk-benefit– for direct measures– for surrogates

Surrogate• Everyone likes “hard” endpoints but …• They mostly don’t measure benefit• They are correlated with benefit

Correlation with Benefit• Does drug produce benefit or modify

correlation? (anti-arrythmics, maybe glitazones)

• Qualitative validation hard enough• Quantify benefit very hard

– estimate strength of relationship– and hope it holds

Patient-Reported Outcomes• Hard endpoints are “nice” but they don’t

measure utility• PRO are squishy but relevant• Psychometrics is not evil (now)

Linking Risk and Benefit• Expected utility

– mean efficacy outcome– incidence of AE– (mean effect) X (goodness) – (AE rate) X

(badness)• Other formulas are incorrect

– provided utility is linear wrt effect

It Isn’t Linear• For surrogates• For PROs

Utility Calculations: Example• 50% symptom-free• 50% intolerable adverse events• Good or bad?

– How bad were symptoms?– How bad were adverse events?

Two Drugs• Women have efficacy• Men have adverse

events

• Women have efficacy• Women have adverse

events• Men have nothing

Two Drugs• Women have efficacy• Men have adverse

events• Useful drug

– provided AEs are reversible

• Women have efficacy• Women have adverse

events• Men have nothing• Useless drug

“Expected utility” does not distinguish!

Why Doesn’t Expectation Work?• Because you don’t really measure benefit

– benefit at timepoint (or average over time) is surrogate for long-term benefit

– don’t get long-term benefit if you drop out– LOCF makes it worse

• “Mixing up” safety and efficacy is …– not illegal– not even stupid– “individualized medicine”

• dropout is good biomarker!