Jsm2013,598,sweitzer,randomization metrics,v2,aug08

© 2013 Medidata Solutions, Inc. 1 © 2013 Medidata Solutions, Inc. 1

Dennis Sweitzer Principal Biostatistician

�

Slides & supplementary material wil be posted at: �www.Dennis-Sweitzer.com �

7 August 2013, JSM

Randomization Metrics: Jointly assessing predictability and efficiency loss in covariate adaptive randomization designs


Outline

§ Objective What is the right questions, anyway?

§ Randomness How to measure & from whose perspective?

§ Balance Why? How to measure to match?

§ Simulation Validity?

§ Results

NB: Slides & supplementary material will be posted at: ��

www.Dennis-Sweitzer.com blog.mdsol.com �


Questions

Q: Will unequal subgroups affect randomization performance? Q: What are the impacts of choosing dynamic allocation over permuted block? Q: Dynamic allocation is more deterministic than permuted block, isn’t it? Q: What about randomization performance at interim analysis?

What is the best method for randomizing THIS study design in THIS population of patients?


Measuring Randomness

§  Text with bullets (20 pt) §  Text with bullets (18 pt)


§  Text with bullets (12 pt)

Randomness

Predictability (by observer)

Entropy (no observer)

Periodicity (patterns)

X ⟶

⟶

Y (as function of probabilities)


Measuring Balance




Balance

Efficiency (Variability)

Confounding (Bias)

Deviation from Target (Convenience)

X (as # or % of subjects) ⟶

⟶

Y (as function of probabilities)


Generic Simulation

Convenient covariates…

2 sexes {M,F} 3 age groups {Mid,Yng,Old} 20 sites {a,b,…,t}

Covariate Levels Ratio 2 sexes {M,F} 50:50 3 age groups {Mid,Yng,Old} 33:33:33

10 sites /variants

{a,b,…,j} 10 x 20 each

However ➣ Real-life trials are rarely so neat

Although ➣ Simulated trials usually this neat

Site a b c d e f g h i j k l m n o p q r s t Share 28% 14% 9.3% 6.9% 5.6% 4.6% 4.0% 3.5% 3.1% 2.8% 2.5% 2.3% 2.1% 2.0% 1.9% 1.7% 1.6% 1.5% 1.5% 1.4% Exp.# 56 28 19 14 11 9.3 7.9 6.9 6.2 5.6 5.1 4.6 4.3 4.0 3.7 3.5 3.3 3.1 2.9 2.8

Sex (1:½) Age Group (1:½:⅓ )

Female Male 67% 33%

Mid. Aged 55% 36% ♀, M 18% ♂, M Young 27% 18% ♀ , Y 9% ♂, Y Older 18% 12% ♀, O 6% ♂, O

Site a b c d e f g h i j Share 34% 17% 11% 9% 7% 6% 5% 4% 4% 3% Exp.# 68.3 34.1 22.8 17.1 13.7 11.4 9.8 8.5 7.6 6.8

Small Cells ⟹ Large Impacts

pk ∝1

k + c( )a2 sexes 67:33 3 age groups 55:27:18 10 sites 34 : 17 : 11 : 9 : ... : 3.4

Use Model: Outcome = Treatment + Sex + Age + Sex*Age + Site+ error

Use… Zipf-Mandelbrot Distribution ➣ Sizes of cities, frequencies of words, species abundance, Website hits…

Q: Will unequal subgroups affect performance?

ANCOVA Model Outcome = Treatment + Sex + Age + Sex*Age + Variant/Site+error


Balance: Confounding

Ad hoc: Score ≣ Total of

#subjects in covariate subgroups with 100% of a single treatment

ANCOVA Model Outcome = Treatment + Sex + Age + Sex*Age + Variant/Site+ error

0 2 4 6 8

CR Stratified ………...

PB(1:1) PB(2:2) PB(4:4)

DAS(0%) DAS(15%)

Marginal ………… DAM(0%)

DAM(15%) Strata + Margins

DAE(0%) DAE(15%)

Site + DAC(0%)

DAC(15%) Site+Strata

DAD(0%) DAD(15%)

Equally Distributed Covariates

Zipf-Mandelbrot Covariates

Confounding score⟶

A: Yes, increased

confounding!


Balance: Loss of Efficiency

Ad hoc definition:

Total of #subjects in

covariate subgroups with

100% of a single treatment

ANCOVA Model Outcome = Treatment + Sex + Age + Sex*Age + Variant/site+ error

0 2 4 6 8 10 12 14 16

CR Stratified ………...

PB(1:1) PB(4:4)

DAS(0%) DAS(15%)

Marginal + …. DAM(0%) DAE(0%)

Site + Margins…… DAC(0%)

DAC(15%) Site+Strata+Margin….

DAD(0%) DAD(15%)

Loss of Efficiency (LOE) ⟶

Equally Distributed Covariates

Zipf-Mandelbrot Covariates

! ! = !! + !!!!

(Atkinson, 2003)

Matrix Form of model, where: z ≣treatment allocation α ≣treatment effect β ≣Covariate effects X ≣ Design Matrix

Columns ó Covariates Rows ó Subjects

Loss of Efficiency:

Var(α̂) = σ 2

ztz− ztX(XtX)−1Xtz

LOE = ztX(XtX)−1XtzA: But not efficiency


Randomness: Predictability




Blackwell-Hodges (1957) guessing rule ☞ Game theory interpretation ☞ Always guesses the next assignment will restore balance

Measures

Potential Selection Bias

F ≣ abs(# Correct – Expected # Correct by chance alone)

Pote

ntia

l Sel

ectio

n Bia

s (S

trat

a)

Q: Impacts of choosing dynamic allocation over permuted block ?






Pote

ntia

l Sel

ectio

n Bia

s (S

trat

a)

A: More efficiency, less predictability

Randomization factors

Pb ≣ Sex*Age

daC ≣ Sex + Age + Variant

daD ≣ Sex + Age + Sex*Age +Variant






Pote

ntia

l Sel

ectio

n Bia

s (S

ite)


Pb ≣ Sex*Age

daC ≣ Sex + Age + Site

daD= Sex + Age + Sex*Age +Site

A: … unless the observer knows too much…






Pot

enti

al S

elec

tion

Bia

s (S

ite)


daD= Sex + Age + Sex*Age + Site

daE ≣ Sex + Age + Sex*Age

da? = Sex + Age + Sex*Age + ½ Site

A: However, can adjust weights


Randomness: Entropy/Syntropy




Observed Entropy ≣ Self Information Content

Where: pj ≣ probability of observed treatment choice for patient j Syntropy* •  Average & Rescale to [0,1] so that:

0 ⟹ Max Randomness 1 ⟹ Max Determinism

I = − log(pj )∑

Synt

ropy

* “Syntropy” ― coined by Buckminster Fuller as the

opposite of entropy

Q: Isn’t DA deterministic?

A: A random element makes it as random as PB


Results: Metrics and Changing Sample Size




Changes in Sample

Pote

ntia

l Sel

ectio

n Bia

s (S

trat

a)

Q: What about randomization performance at interim analysis?

A: PB becomes more predictable & a little more efficient

A: DAC is and becomes both less predictable and more efficient


Next Directions

§  Compare more methods: Urn randomization, Optimal-Designs, Novel methods, etc

§  Randomization Metrics vs statistical properties of analyses

§  Optimizing parameters & tweaking algorithms

§  Refining metrics (e.g., Deviation from Target, Periodicity)

§  Exploring quirks in system behavior.

§  For more information (slides, bibliography, supplemental material, etc.) see:

blog.mdsol.com OR

www.Dennis-Sweitzer.com OR www.slideshare.net/denswei


Additional Slides

§ Bibliography

§ Randomization factors used

§ Comparing Methods Example

§ Periodicity Plot


Bibliography

§  Atkinson, AC. (2003) The distribution of loss in two-treatment biased-coin designs. Biostatistics, 2003, 4, 2, pp. 179–193

§  Blackwell, D. and J.Hodges Jr (1957). Design for the control of selection bias. Ann Math Stat 28, 449-460

§  Wikipedia contributors. "Entropy (information theory)." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 23 Apr. 2013. Web. 14 May. 2013.

§  Lebowitsch, J, et al, (2012). “Generalized multidimensional dynamic allocation method”. Statistics in Medicine,2012;


Covariates vs Randomization Factors Analysis: ANCOVA Model Outcome

= Treatment + Site + Sex + Age + Sex*Age

Stratification factors in Randomization Strata Imbalances– within combinations of Sex & Age Marginal Imbalances – within each Sex, Age, and Site

“S”

PB, daS

“M”

daM

“C” daC

“D” daD


Comparing Methods & Parameters




Predictability vs Loss of Efficiency, (n=50) Not much difference in CI Variations on DA: •  daJS, daJM –

(Kuznetsova, 2012) •  mmS, mmM, baM, baF,

baS - experimental

Pote

ntia

l Sel

ectio

n Bia

s (S

trat

a)


Randomness: Periodicity




A la Discrete Fourier Transform •  Amplitude of a periodic variation in

the max{pi,j} of treatment assignments

Period

icity

Technology

Jsm2013,598,sweitzer,randomization metrics,v2,aug08