STAISTICAL ANALYSIS GMM

8/3/2019 STAISTICAL ANALYSIS GMM

http://slidepdf.com/reader/full/staistical-analysis-gmm 1/8

Image Segmentation using Gaussian Mixture

ModelsRahman Farnoosh, Gholamhossein Yari and Behnam Zarpak

Department of Applied Mathematics, University of Science and Technology, 16844, Narmak,Tehran, Iran

Abstract. Recently stochastic models such as mixture models, graphical models, Markov randomfields and hidden Markov models have key role in probabilistic data analysis. Also image segmen-tation means to divide one picture into different types of classes or regions, for example a pictureof geometric shapes has some classes with different colors such as ’circle’, ’rectangle’, ’triangle’and so on. Therefore we can suppose that each class has normal distribution with specify mean and

variance. Thus in general a picture can be Gaussian mixture model. In this paper, we have learnedGaussian mixture model to the pixel of an image as training data and the parameter of the modelare learned by EM-algorithm. Meanwhile pixel labeling corresponded to each pixel of true image isdone by Bayes rule. This hidden or labeled image is constructed during of running EM-algorithm.In fact, we introduce a new numerically method of finding maximum a posterior estimation by usingof EM-algorithm and Gaussians mixture model which we called EM-MAP algorithm.In this algorithm, we have made a sequence of the priors, posteriors and they then convergent toa posterior probability that is called the reference posterior probability. So Maximum a posteriorestimation can be determined by this reference posterior probability which will make labeled im-age. This labeled image shows our segmented image with reduced noises. This method will show inseveral experiments.

Keywords: Bayesian Rule, Gaussian Mixture Model (GMM),Maximum a Posterior (MAP),Expectation-Maximization (EM) Algorithm, Reference Analysis.

PACS: 29. 85. +c, 87. 58. Mj , 07. 05. Tp

INTRODUCTION

Automatically image processing means image segmentation(i.e. dividing an image intodifferent types of regions or classes), recognizing of objects and detecting of edges,etcby machine. All of these can be done after segmentation of a pictures. So image seg-mentation is the most important image problems. In addition noise removing and noisereduction of pictures always are important in classical image problems. In this paper, wedo both segmentation and noise reduction with a probabilistic approach.There are many mathematical and statistical methods for image problems, but this paperargues about GMM as a general Gaussian distribution, EM-algorithm and Bayesian rule.But Bayesian framework usually has many difficulties because the posterior probabilityhas complex form. So we must use Markov Chain Monte Carlo algorithms or Variationalmethods with high computing to find MAP estimation. These methods have worked wellin the last decades [1,5].In this paper, a new numerically EM-MAP algorithm base on Bayesian rule has con-structed. We have used Bernardo’s theory about reference analysis [2] in practice andin image segmentation. it means that in reference analysis a sequence of priors and



posteriors are made and they convergent to a posterior probability which is called ref-erence posterior probability. In this paper, we have used this idea. So we have modifiedEM-algorithm for our image segmentation. After finding reference posterior probability,MAP estimation and pixel labeling can easily make segmented of image.

This paper organized as follows. It first reviews of GMM and its properties. Then we in-troduce the EM-MAP algorithm for learning parameters for a given image as the trainingdata. Choosing initial values of EM-algorithm have discussed in next section. EM -MAPalgorithm never can not convergent without choosing a suitable starting points. But theseinitial information are made by histogram of image. Finally we show some experimentswith simulated images.

GAUSSIAN MIXTURE MODELS

Image is a matrix which each element is a pixel. The value of the pixel is a number that

shows intensity or color of the image. Let X is a random variable that takes these values.For a probability model determination, we can suppose to have mixture of Gaussiandistribution as the following form

f ́

xµ

k

∑i

1 pi N ́

x µ i

σ 2i µ (1)

Where k is the number of regions and pi 0 are weights such that ∑k i 1 pi 1

N ́

µ i

σ 2i µ

1σ

Ô

2 piexp

́

x µ i µ

2

2σ 2i

(2)

Whereµ i

σ i are mean, standard deviation of class i. For a given image X, the lattice dataare the values of pixels and GMM is our pixel base model. However, the parameters areθ

́

p1

pk

µ 1

µ k

σ 21

σ 2k µ and we can guess the number of regions in GMMby histogram of lattice data. This will show in experiments.

EM-MAP ALGORITHM

There is a popular EM algorithm for GMM in several papers [3,6]. We modify it to thefollowing algorithm with the name of EM-MAP algorithm.

1. Input:Observed Image in a vector x j

j 1 2

n and i¾ 1 2

k labels set

2. Initialize: θ ́ 0µ

́

p ́ 0µ

1

p ́ 0µ

k

µ ́ 0µ

1

µ ́ 0µ

k

σ 2 ́

0µ

1

σ 2 ́

0µ

k µ

3. (E-step)

p ́

r · 1µ

i j

P ́

r · 1µ

́

i x j µ

p ́ r µ i

N ́

x j µ ́ r µ

i

σ 2 ́

r µ

i µ

f ́

x j µ

(3)



4. (M-step)•

ˆ pi ́ r · 1µ

1n

n

∑ j 1

p ́ r µ i j

(4)

•

µ̂ ́ r · 1µ

i

∑n j

1 p ́

r · 1µ

i jx j

n ˆ pi ́ r · 1µ

(5)

•

σ̂ 2 ́ r · 1

µ

i

∑n j 1 p ́ r · 1µ

i j ́

x j µ̂ i

́

r · 1µ

µ

n ˆ pi ́

r · 1µ

(6)

5. Iterate steps 3 and 4 until an specify error i.e. ∑i e2i

ε

6. Compute

pl j

ArgMaxi p

́

finalµ

i j

j

1

2

n (7)7. Construct labeled image corresponding of each true image pixel.

This EM-MAP algorithm is a pixel labeling base method such that the labeled imageshows each segment or object by different type of labels. Note that formula (3) is Bayesrule, pr

i is discrete prior probability in stage r and p ́ r · 1µ

i jis discrete posterior probability

in the next stage.In this algorithm, we make a sequence of prior and then posterior until to get conver-gence. The labeled images chooses with MAP of the final posterior.

CHOOSING A PRIOR WITH MAXIMUM ENTROPY PROPERTY

In EM-MAP algorithm, there are some difficulties. How can we choose?

• the number of classes• the weights• the means• the variances

A practical way that we can guess the prior of parameters is drawing the histogramof the observed image. Not only image histogram gives us four above parameters forusing initial values in the EM algorithm, but also this extracted information usually hasmaximum entropy. This claim has shown in experiments. Besides, the final posteriorprobability will get a stable entropy. We also compute the number of misclassificationsof results. This shows that how much our algorithm is well.

EXPERIMENTS

In first example, we make three boxes in an image and add it white noise. The observedimage and its histogram in the figure 1 and figure 2 are shown. Information extracted



5 10 15 20 25 30 35 40 45 50

5

10

15

20

25

0 50 100 150 200 2500

100

200

300

400

500

600

700

800

900

1000

FIGURE 1. a)The Observed Image of Boxes,b)Histogram of the Observed Image

5 10 15 20 25 30 35 40 45 50

5

10

15

20

25

1 2 3 4 5 6 7 8 9 100.66

0.68

0.7

0.72

0.74

0.76

0.78

FIGURE 2. a)Labeled image with reduced noises,b)Entropy curve in each iteration

from the histogram are k

4

p ́ 0µ

́

0

0320

0

1344

0

0576

0

7760µ

or empirical prob-ability with entropy 0 7411

µ ́

0µ

́ 40 75 210 220µ andσ ́

0µ

́ 100 100 100 100µ .The stopping time occurs when L-2 norm of absolute error has very small value. After

running EM-MAP, we had ten-times iteration in figure 2 and the entropy of each itera-tion which goes to a stable or maximum case. We see in segmented image that Blue

l,cyan

2,Yellow 3 and Red

4. There is 0.0008 percentage misclassification that isonly one red instead of yellow pixel is wrong. In this example, pixel labeling and noiseremoving are well done.

In the second example, some different shapes such as circle, triangle, rectangle,etc have considered. The observed image and its histogram in figure 3 have shown.For EM-MAP finding, we need to get initial values by histogram of this image. We



10 20 30 40 50 60 70 80 90 100

5

10

15

20

25

30

35

40

45

50

−50 0 50 100 150 200 2500

500

1000

1500

2000

2500

3000

FIGURE 3. a)The Observed Image of Circle,b)Histogram of the Observed Image

10 20 30 40 50 60 70 80 90 100

5

10

15

20

25

30

35

40

45

50

0 5 10 15 20 250.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9


choose k

3

p ́

0

08

0

34

0

58µ

as empirical probability or relative frequency,µ

́ 38 63 88µ and σ 12 12 12 with norm of error less than 0.01.

In figure 4, we made 3 classes Blue 1,Green

2 and Red 3. There are only 25

missing of classifications or percentage of 0.005. If in EM-MAP algorithm, we computethe entropy of the posterior probability in Each stage of iteration, this entropy will bedecreasing to reach a stable form.The third example is more complex. The number of components is great. In addition,there is dependent noise in image which noise reduction in this case is more difficult.The true image and observed image have shown in figure 5.Again,information extraction can find by drawing histogram of observed image, itmeans



5 10 15 20 25 30

5

10

15

20

25

30

5 10 15 20 25 30

5

10

15

20

25

30

FIGURE 5. a)The True Image of Partition,b)The Observed Image of Partition

0 1 2 3 4 5 6 7 8 90

20

40

60

80

100

120

140

160

180

200

5 10 15 20 25 30

5

10

15

20

25

30

FIGURE 6. a)Histogram of Image,b)The Segmented Image of Partition

• k

10• p ́ 0 0977 0 1377 0 0625 0 0693 0 0361 0 0361 0 1182 0 1904 0 1572 0 0947µ

• µ ́ 1 2 2 5 3 5 4 5 5 5 6 7 7 5 8 5µ

• σ ́ 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5µ

The results have shown in figure 6 with 20 times iteration. In figure 7 the results areshown in 50 times iteration and entropy has reached in a stable case.



5 10 15 20 25 30

5

10

15

20

25

30

0 10 20 30 40 502.16

2.165

2.17

2.175

2.18

2.185

2.19


CONCLUSIONS

In this paper, we make a new numerical EM-GMM-Map algorithm for image segmen-tation and noise reduction. This paper is used BernardoŠs idea about sequence of priorand posterior in reference analysis. We have used known EM-GMM algorithm and weadded numerically MAP estimation. Also the initial values by histogram of image havesuggested which is caused to convergence of EM-MAP method. After convergence of our algorithm, we had stability in entropy. EM-algorithm is iteration algorithm of firstorder [3], so we had slow convergence. We used acceleration convergence such as Stef-

fensen algorithm to have the second order convergence. But later we note that in EM-MAP method, the number of classes will reduce to real classes of image. Finally, EM-algorithm is linear iteration method, so our method is suitable for simple images. It isimportant to note that "for segmentation of real images, the results depend critically onthe features and feature models used" [4] that is not the focus of this paper.

ACKNOWLEDGMENTS

We have many thanks to prof.Mohammad-Djafari for his excellent ideas.

REFERENCES

1. C.Andrieu, N.D.Freitas, A.Doucet, M.I.Jordan, An Introduction to MCMC for Machine Learning,Journal of Machine Learning, 2003, 50, pp. 5-43.

2. J.M. Bernardo and A.F.M. Smith, Bayesian Theory , John Wiley & Sons, 2000.3. L.Xu, M.I.Jordan, On Convergence Properties of the EM Algorithm for Gaussian Mixture, Neural

Computation, 8, 1996, pp. 129-151.4. M.A.T.Figueiredo, Bayesian Image Segmentation Using Gaussian Field Prior, EMMCVPR 2005,

LNCS 3757, pp. 74-89.



5. M.I.Jordan, Z.Ghahramani, T.S.Jaakkola, L.K.Saul, An Introduction to Variational Methods forGraphical Models, Journal of Machine Learning, 1999, 37, pp. 183-233.

6. R.Farnoosh, B.Zarpak, Image Restoration with Gaussian Mixture Models, Wseas Trans. on Mathe-matics, 2004, 4, 3, pp.773-777.

Documents

STAISTICAL ANALYSIS GMM