View
215
Download
2
Category
Tags:
Preview:
Citation preview
Distant Kin in the EM Family
David A. van Dyk
Department of Statistics
University of California, Irvine.
(Joint work with Xiao Li Meng and Taeyong Park.)
Outline The EM Family Tree The Stochastic Cousins Some Odd Relations
o Nested EM and The Partially Blocked Gibbs Sampler
A Newly Found Kinsmano A Stochastic ECME/AECM Sampler:
The Partially Collapsed Gibbs Sampler
The EM Family Tree
EM Algorithm
StochasticSimulation
Variance Calculations
Gauss-Seidel
Efficient DA
Monte CarloIntegration
ECM
ECME
SEMDA
samplerMCEM
PXEM
Efficient DAEM
NEM
AECM
SECM
Algorithms
Methods
1977 (Dempster, Laird, & Rubin)1987 (Tanner & Wong)1990 (Wie & Tanner)1998 (Liu, Rubin, & Wu)1995 (van Dyk, Meng, & Rubin)1991 (Meng & Rubin)1993 (Meng & Rubin)2000 (van Dyk)1994 (Liu & Rubin)1997 (Meng & van Dyk)
Stochastic Cousins
EM Algorithm
Gauss-Seidel
Efficient DA
Monte CarloIntegration
ECM
ECME
MCEM
PXEM
Efficient DAEM
NEM
AECM
DA Sampler
Gibbs Sampler
Marginal DAPX-DA
Partially BlockedGibbs Sampler
???
The EM and DA Algorithms
p(M|)
p(|M)
p(M|)
p(|M)
EM Algorithm DA Sampler
ExpectationStep
MaximizationStep
RandomDraw
BACK
An NEM Algorithmwith a Monte Carlo E-step
NEM Algorithm The Stochastic Version
E-Step
M-Step
Draw
p(M1|)
p(|M1)
p(M|)
p(, M2|M1)
p(M1|M2,) p(M2|M1,) p(M1|M2,) p(M2|M1,)
p(M2|M1,) p(|M) p(M2|M1,) p(|M)
A Partially-Blocked SamplerNEM Algorithm Partially-Blocked Sampler
E-Step
M-Step
Draw
p(M1|)
p(|M1)
p(M1|, M2)
p(, M2|M1)
p(M1|M2,) p(M2|M1,)
p(M2|M1,) p(|M) p(M2|M1,) p(|M)
BACK
Ordering CM-steps in ECMEECME Algorithm
E-Step
M-Step
Draw
p(M|)
p(2|1)p(1|M,2)
Monotone Convergence
ECME Algorithm
p(M|)
p(1|M, 2)p(2|1)
NO Monotone Convergence
Reducing conditioning
Speed upconvergence!
But BE CAREFUL!
Step orderMatters!
A Stochastic Version of ECMEECME Algorithm
E-Step
M-Step
Draw
p(M|)
p(2|1)p(1|M,2)
p(M|)
p(2|1)p(1|M, 2)
Incompatible Conditional Distributions
What is the stationary distribution
of this chain??
AECM and Partially Collapsed Samplers
E
M
D
p(M|)
p(M,2|1)p(1|M,2)
p(M|)
p(1|M,2) p(2|1)
p(M,2|1)
p(1|M,2)
p(M|)
p(1|M,2) p(2|M1,1)
p(M|)
p(2|M1,1)p(1|M,2)
ECME
PartiallyCollapsed
Blocked Sampler
AECM
PartiallyCollapsed
Incompatibledraws
Stationary distributionmust be verified!
Completely Collapsed Samplers
E
M
D
p(M|)
p(M,2|1)p(M,1|2)
p(M|)
p(1|M,2) p(2|1)
p(2|1)p(1|2)
ECME Collapsed Sampler
CompleteCollapse
Blocking (ECME) is a special case of Partial Collapse (AECM). We expect Collapsed Samplers (CM) to perform better than Partially
Collapsed Samplers (AECM). And we expect Collapsing (CM) to perform better than Blocking (ECME).
Many of these relationships are known, I emphasize the connections between EM-type DA-type algorithms.
Reducing Conditioning in Gibbs: The Simplest ExampleConsider a two-step Gibbs Sampler:
The Markov Chain
has stationary dist’n With target margins butWithout the correlation of the target distributionAND converges quickly!
Iteration t
Iteration t+1/2
We regain the target distribution with a one-step shifted chain.
Heads Up!!
Reducing the conditioning within Gibbs involves new challenges:
The order of the draws may effect the stationary distribution of the chain.
The conditional distributions may no be compatible with any joint distribution.
The steps sometimes can be blocked to form an ordinary Gibbs sampler with fewer steps.
An Example from Astronomy
Parameterized Latent Poisson Process
Underlying Poisson intensity is a mixture of a broad feature and several narrow features.
The “line location” and mixture indicator are highly correlated.
Photon energy
Line location
Emission Line
Spectral Model for Photon Counts
An Example from Astronomy
Standard sampler simulates
Photon energy
Line location
Emission Line
Spectral Model for Photon Counts
which may converge very slowly or not at all.
An Incompatible Gibbs Sampler
Computational Gains
Verifying the Stationary Distribution of Sampler 2
The General Strategy
BACK
Recommended