7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 1/24

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 2/24

1. EM Algorithm

2. Mixture Model

3. EM Algorithm for Normal Mixture Model4. Simulation

5. Results

6. Conclusions

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 3/24

EM Algorithm

Expectation-Maximization (EM) algorithm is an iterative method for

maximum likelihood estimates of parameters in statistical models, w

model depends on unobserved latent variables.

Latent variable : not directly observed but are rather inferred from o

variables that are observed.

EM algorithm is useful in incomplete-data problems : Missing data, T

distributions, Censored observations, Random effects, Mixtures, etc

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 4/24

EM Algorithm

    = (1, 2, ⋯ , ) : observed incomplete data

    = (1, 2, ⋯ , ) : missing data (latent variable)

    = (1, 2, ⋯ , ) : unknown parameters.

  ) =  ) : incomplete likelihood of .

  , ) =  , ) : complete likelihood of and .

  , ) : conditional distribution of given .

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 5/24

EM Algorithm

1. Expectation Step (E - Step) : Calculate

  (−1), ) = (|,())[log  ,   ].

2. Maximization Step (M – Step) : Choose

() = argmax

  (−1), ).

3. Return to the E-Step unless a stopping criterion has been met such

()  (−1) <   .

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 6/24

  ,   =   ,

log   = log  ,   log , )↔

  =  , )

, )↔

(|,())   log   = (|,())[log  ,   ]   (|,())[lo↔

log   = (|,())[log  ,   ] (|,())[log ,   ]↔

log   =     (−1), )     (−1), )↔

log   =     (−1), )     (−1), )↔

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 7/24

EM Algorithm

1.   ()   (−1), )   ≥     −1     −1 , )

  (−1), ) is increasing.

2.   ()   (−1), )   ≤   (−1)   (−1), )

  (−1), ) is decreasing.

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 8/24

Mixture Model

Mixture model is a probabilistic model for representing the presence

subpopulations within an overall population, without requiring that

observed data set should identify the sub-population to which an in

observation belongs.

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 9/24

Mixture Model

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 10/24

Mixture Model

Basic definition of finite mixture model

 () =   ∑=1

 ()

,      

  1, 2, ⋯ , : mixing proportions or weights

  1     , 2     , ⋯ , () : component densities

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 11/24

Basic definition of finite normal mixture model

    , ) =   ∑=1

  , 2)

,    

  1, 2, ⋯ , : mixing proportions or weights

   | , 2 : normal component densities

Normal Mixture Model

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 12/24

Normal mixture model

    ,   =   ∑=1

  , 2)

,    

    = (1, 2, ⋯ , ) : observed data

    = (1, ⋯ , , 12, ⋯ ,

2) : unknown parameters

   = (1, ⋯ , ) : unknown mixing proportion

EM Algorithm for Normal Mixture Model

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 13/24

Normal mixture model (hierarchical)

    ,   =   ∑=1

  , 2)

      =   12

⋯

,  =   1, ⋯ ,   ~ (1, 1, ⋯ , )

    = (1 , 2

, ⋯ , ) : missing data (latent variable)

EM Algorithm for Normal Mixture Model

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 14/24

Normal mixture model

  ,   ,   =    , )    ) = ∑=1

  , 2)

Complete likelihood of and

  , , ) = ∏=1   ,   ,   =   ∏=1 ∑=1   , 2)

Complete log-likelihood of and

log  , , ) =   ∑= ∑

=1

 log [   , 2)]

EM Algorithm for Normal Mixture Model

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 15/24

Simulation

Define the measures for the evaluation of the asymptotic properties a

1. Bias of for the th component

    =  1

∑=1

(

() )

2. Mean square error (MSE) of for the th component

    =  1

∑=1

(

)2

,

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 16/24

Simulation

Simulate 2-5 components of normal data with the combination of ,

    = 5, 10, 20, 50,100

  2 = 1, 2, 5, 10, 20, 50,100

   = 0.1, 0.2,   ⋯   , 0.9

Fix the total size   = 25, 50, 100, 200,500, 1000, 2000, 5000, 10000.

Repeat 1000 times.

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 17/24

Simulation

Since there are a large number of combination, combine these into

  =   ∑=1

[       2 +   2 �2   2

]

,  =  11 + ⋯ + ,   �2 = 112 + ⋯ +

2

Classify into small (  = 0~100), medium (  = 100~1000), large ( =

Small : mixture of normal data that overlap largely

Medium : mixture of normal data that have some overlap

Large : mixture of normal data that show only slight overlap

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 18/24

Simulation

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 19/24

Results

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 20/24

Results

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 21/24

Results

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 22/24

Results

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 23/24

Conclusions

EM algorithm gives reasonable solutions in an asymptotic unbiased

EM algorithm estimate seem to provide reasonable estimates of the

values.

7/21/2019 1

http://slidepdf.com/reader/full/-15695d09a1a28ab9b02932024 24/24