Distant Kin in the EM Family

David A. van Dyk

Department of Statistics

University of California, Irvine.

(Joint work with Xiao Li Meng and Taeyong Park.)

Outline The EM Family Tree The Stochastic Cousins Some Odd Relations

o Nested EM and The Partially Blocked Gibbs Sampler

A Newly Found Kinsmano A Stochastic ECME/AECM Sampler:

The Partially Collapsed Gibbs Sampler

The EM Family Tree

EM Algorithm

StochasticSimulation

Variance Calculations

Gauss-Seidel

Efficient DA

Monte CarloIntegration

ECM

ECME

SEMDA

samplerMCEM

PXEM

Efficient DAEM

NEM

AECM

SECM

Algorithms

Methods

1977 (Dempster, Laird, & Rubin)1987 (Tanner & Wong)1990 (Wie & Tanner)1998 (Liu, Rubin, & Wu)1995 (van Dyk, Meng, & Rubin)1991 (Meng & Rubin)1993 (Meng & Rubin)2000 (van Dyk)1994 (Liu & Rubin)1997 (Meng & van Dyk)

Stochastic Cousins

EM Algorithm

Gauss-Seidel

Efficient DA

Monte CarloIntegration

ECM

ECME

MCEM

PXEM

Efficient DAEM

NEM

AECM

DA Sampler

Gibbs Sampler

Marginal DAPX-DA

Partially BlockedGibbs Sampler

???

The EM and DA Algorithms

p(M|)

p(|M)

p(M|)

p(|M)

EM Algorithm DA Sampler

ExpectationStep

MaximizationStep

RandomDraw

BACK

An NEM Algorithmwith a Monte Carlo E-step

NEM Algorithm The Stochastic Version

E-Step

M-Step

Draw

p(M1|)

p(|M1)

p(M|)

p(, M2|M1)

p(M1|M2,) p(M2|M1,) p(M1|M2,) p(M2|M1,)

p(M2|M1,) p(|M) p(M2|M1,) p(|M)

A Partially-Blocked SamplerNEM Algorithm Partially-Blocked Sampler

E-Step

M-Step

Draw

p(M1|)

p(|M1)

p(M1|, M2)

p(, M2|M1)

p(M1|M2,) p(M2|M1,)

p(M2|M1,) p(|M) p(M2|M1,) p(|M)

BACK

Ordering CM-steps in ECMEECME Algorithm

E-Step

M-Step

Draw

p(M|)

p(2|1)p(1|M,2)

Monotone Convergence

ECME Algorithm

p(M|)

p(1|M, 2)p(2|1)

NO Monotone Convergence

Reducing conditioning

Speed upconvergence!

But BE CAREFUL!

Step orderMatters!

A Stochastic Version of ECMEECME Algorithm

E-Step

M-Step

Draw

p(M|)

p(2|1)p(1|M,2)

p(M|)

p(2|1)p(1|M, 2)

Incompatible Conditional Distributions

What is the stationary distribution

of this chain??

AECM and Partially Collapsed Samplers

E

M

D

p(M|)

p(M,2|1)p(1|M,2)

p(M|)

p(1|M,2) p(2|1)

p(M,2|1)

p(1|M,2)

p(M|)

p(1|M,2) p(2|M1,1)

p(M|)

p(2|M1,1)p(1|M,2)

ECME

PartiallyCollapsed

Blocked Sampler

AECM

PartiallyCollapsed

Incompatibledraws

Stationary distributionmust be verified!

Completely Collapsed Samplers

E

M

D

p(M|)

p(M,2|1)p(M,1|2)

p(M|)

p(1|M,2) p(2|1)

p(2|1)p(1|2)

ECME Collapsed Sampler

CompleteCollapse

Blocking (ECME) is a special case of Partial Collapse (AECM). We expect Collapsed Samplers (CM) to perform better than Partially

Collapsed Samplers (AECM). And we expect Collapsing (CM) to perform better than Blocking (ECME).

Many of these relationships are known, I emphasize the connections between EM-type DA-type algorithms.

Reducing Conditioning in Gibbs: The Simplest ExampleConsider a two-step Gibbs Sampler:

The Markov Chain

has stationary dist’n With target margins butWithout the correlation of the target distributionAND converges quickly!

Iteration t

Iteration t+1/2

We regain the target distribution with a one-step shifted chain.

Heads Up!!

Reducing the conditioning within Gibbs involves new challenges:

The order of the draws may effect the stationary distribution of the chain.

The conditional distributions may no be compatible with any joint distribution.

The steps sometimes can be blocked to form an ordinary Gibbs sampler with fewer steps.

An Example from Astronomy

Parameterized Latent Poisson Process

Underlying Poisson intensity is a mixture of a broad feature and several narrow features.

The “line location” and mixture indicator are highly correlated.

Photon energy

Line location

Emission Line

Spectral Model for Photon Counts

An Example from Astronomy

Standard sampler simulates

Photon energy

Line location

Emission Line

Spectral Model for Photon Counts

which may converge very slowly or not at all.

An Incompatible Gibbs Sampler

Computational Gains

Verifying the Stationary Distribution of Sampler 2

The General Strategy

BACK