GRAPH CUTS BY USING LOCAL TEXTURE FEATURES OF WAVELET COEFFICIENT FOR IMAGE SEGMENTATION

2008 International Conference on Multimedia & Expo

GRAPH CUTS BY USING LOCAL TEXTURE GRAPH CUTS BY USING LOCAL TEXTURE FEATURES OF WAVELET COEFFICIENT FEATURES OF WAVELET COEFFICIENT

FOR IMAGE SEGMENTATIONFOR IMAGE SEGMENTATION

Keita Fukuda, Tetsuya Takiguchi, Yasuo ArikiKeita Fukuda, Tetsuya Takiguchi, Yasuo ArikiGraduate School of Engineering, Kobe University, Graduate School of Engineering, Kobe University,

JapanJapan

2008 International Conference on Multimedia & Expo 2

Table of contentsTable of contents

• Introductionconventional method, problem with graph cuts

• Proposed methoditerated graph cuts using texture and smoothing

and detail of each method• Experiments

• Conclusion and Future Work


IntroductionIntroduction

Extracting the foreground objects in static images is one of the most fundamental tasks (Image Segmentation Problem).

“bkg”“obj”


BackgroundBackground

Recently, the image segmentation problem is formalized as an optimal solution problem.

e.g. Snakes, Level Set Method, Graph CutsY.Boykov, M.P.Jolly,“Interactive graph cuts for optimal boundary & region segmentation of object in N-D images”, IEEE International Conference on Computer Vision and Pattern Recognition

possible to compute global optimal solution

The cost function is general enough to include both region and boundary properties of the segments

Advantage of Graph Cuts

S

T

O

B


Graph CutsGraph Cuts

S

T

O

B

min cut

O

B

Assigned the corresponding label

Searching th

e

minimum co

st cu

tt-link

n-link

Background likelihood

S:”obj”

T:”bkg”

O

B

Neighbor similarity

A 3-by-4 image

Segmentation result

Object likelihood

Constructa graph


Problem (1) with GCProblem (1) with GC

It has been difficult to segment images that include complexnoisy edges.

To solve this problem,using iterated graph cuts based on smoothing.

T. Nagahashi, H.Fujiyoshi, and T.Kanade ，“ Image Segmentation Using Iterated Graph Cuts Based on Multi-scale Smoothing ， ” ACCV2007

Noise exists in strong edge


Problem (2) with GCProblem (2) with GC

It is difficult to segment images with an object whose color is similar to the background.

To solve this problem,employing the texture likelihood as well as color likelihood.

The color of object is similar to that of background

Iterated Graph Cuts based on local texture features as well as low frequency features of wavelet coefficient.

Proposed method


To solve problem (2) with GC, high pass subbands (LHk,HLk,HHk) are used for t-link. Local texture features are defined from them.

The likelihood are derived from local texture features as well as color. And the prior probabilities are defined using the previous segmentation result.After initializing the level k, the input image is decomposed into subbands

using multiresolution wavelet analysis.To solve problem (1) with GC, low pass subband (LLk) is used for n-link.

The obtained neighbor similarity is set to n-link as edge cost.

Proposed MethodProposed Method

LL,LH,HL,HHare obtained

Graph Cuts Segmentation

Prior probability

GMM(color+texture)update

k ← k-1

LL: Smoothing

LH,HL,HH: Local texture features

Multiresolution analysis

(level k)

seed

Input Output

t-link

n-link

Graph Cuts segmentation is carried out, and these processes are repeated until k = 0.


Multiresolution Multiresolution analysisanalysis

downsampledHH1LH1

HL1LH2 HH2

HL2LL2

Level 2

LH1

HL1

HH1

LL1

Level 1

LL: low-pass information

LH: vertical

HL: horizontal

HH: diagonal

orientation

Subbands information

for n-link

for t-link

Input imageLevel 1Level 2

Input image downsampled


n-linkn-link

p q

},{ qpB

),(1

2)(

exp 2

2

},{ qpdistII

B qpqp

B{p,q} is large when p and q are similar, it is close to 0 when p and q are very different.

Global to local segmentation can be performed by iterated Graph Cuts with multiresolution analysis using from coarse to fine level for n-link .

finecoarse

Level 3 Level 2 Level 1


Local texture featuresLocal texture features

LH (level 1) HL (level 1)

Local texture features are defined by averaging the absolute wavelet coefficient in the window (3×3) surrounding pixel p

HH (level 1)

HHkHLkLHkfordTNqp

kp ,,||

91

,

)(

Local texture features

d: wavelet coefficient

k: level

Local texture features Tp are larger in complex region, and smaller in flat region.


t-linkt-link

Distance trans.

Pr(O)

Distance trans.

Pr(B)

S:”obj”

T:”bkg”

p

－ ln Pr(B|Yp)

－ ln Pr(O|Yp)

Color (RGB) :Cp

Local texture:Tp

Pr(Yp|O)

Color (RGB) :Cp

Local texture:Tp

Pr(Yp|B)

Background

Object

GMM

GMM

Object likelihood

Background likelihood


Graph Cuts Graph Cuts SegmentationSegmentation

Edge cost of the graph

)|Pr(ln)"("

)|Pr(ln)"("

pp

pp

YBbkgR

YOobjR

The boundary between the object and the background is found by searching the

minimum cost cut on the graph

S

T

O

B

mincut

Yuri Boykov and Vladimir Kolmogorov,“An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Computer Vision”


Summarize proposed methodSummarize proposed method

S:”obj”

T:”bkg”

O

B

t-link (texture with color)n-link

Segmentation result

S:”obj”

T:”bkg”

O

B

Multiresolution at level kk Multiresolution at level k-1k-1

Multiresolution level is set to k-1k-1

The prior probability


Experimental ConditionExperimental Condition We compared using the same labels:method (1) Interactive Graph Cuts [Boykov 04]

method (2) Iterated Graph Cuts using smoothing [Nagahashi 07]

Proposed Iterated Graph Cuts using smoothing and texture Berkley Image database (50 images) The segmentation error rate is defined as follow:

001size image

background in the pixels detected misssize image

object in the pixels detectedmiss(%)

Err

Mask imageOutput imageError (in red and blue )

=


Experimental ResultExperimental Result

Level kk method (1) method (2) Proposed 1 4.62 3.78 3.172 4.62 2.95 2.623 4.62 2.95 1.98

The proposed method improved the error rate compared to two conventional methods.

In this database, two conventional methods could achieve the low error rate.

⇒ The examples with high error rate are shown from next page.

Err (%) in segmentation result at multiresolution level k


Experimental result (1)Experimental result (1) Effect of smoothing

6.27 % 1.16 % 0.85 %

0.79 %3.69 % 0.83 %

The method (2) and proposed method can achieve the better image segmentation for the images with complex edges than the method (1).

Method (1) Method (2) Proposed


Experimental result (2)Experimental result (2)

10.44 % 8.28 % 2.86 %

11.38 % 16.64 % 2.15 %

Effect of local texture featuresMethod (1) Method (2) Proposed

The proposed method can achieve the better image segmentation for the images with object colors similar to background than method (1) and(2)


DiscussionDiscussion　 To solve problem (1) : method (2), proposed

Prior probability　 The brief shape information can be given from previous segmentation result

Smoothing　 removing local strong edges.

　 To solve problem (2) ： proposed Local texture features

for the image with object colors similarto background.


SummarySummary

Graph Cuts by using local texture features of wavelet coefficient for image segmentation.

New concept:Graph Cuts segmentation based on local texture features as well as smoothing process.

In future work: The weight to texture and color for segmentation Others texture features.


Thank you for your attention !Thank you for your attention !


The cost functionThe cost function

Nqp

qpqpPp

pp ffVfDfE,

, ),()()(

propertyregion the:)(Pp

pp fD

propertyboundary :),(,

,Nqp

qpqp ffV

The soft constraints that we impose of label f f are described by the cost function E(f) E(f) as follow:

The regional term assumes that the individual penalties for

assigning pixel p to labels.

The boundary term assumes that a penalty for a

discontinuity between p and q


Minimum cost cutMinimum cost cut

Ｓ

Ｔ

10

2

2

2

6

65

3

97

Ｓ

Ｔ

10

2

2

2

6

65

3

97

3/34/105/5

0/6

5/6

8/92/7

2/2

2/2

0/2

1. Graph Construct

Edge label = capacity

Edge = pipe

Goal : max flow from S to T

Ｓ

Ｔ

2

2

2

6

65

3

97

2 8

Example of a graphResidual GraphCurrent flow

3. Current flow from residual graph

e.g. 8/9 = (flow) / (capacity)

2. Residual graph

Find augmenting paths until finished.



Ｓ

Ｔ

10

2

2

2

6

65

3

97

1. Graph Construct

Edge label = capacity

Edge = pipe

Goal : max flow from S to T

3. Current flow from residual graph

e.g. 8/9 = (flow) / (capacity)

4. Minimum cost cut

Saturated edges are found from 2.

Each node is assigned for the corresponding labels.

Ｓ

Ｔ

10

2

2

2

6

65

3

97

3/34/105/5

0/6

5/6

8/92/7

2/2

2/2

0/2

mincut

S T T

Yuri Boykov and Vladimir Kolmogorov,“An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Computer Vision”

2. Residual graph

Find augmenting paths until finished.


Experimental result (3)Experimental result (3) for the unclear boundary images.

The proposed method is effective for images which have the unclear boundary between object and background.

Method (1) Method (2) Proposed


Time costTime cost

The size of image ： 650×400

method(1) method(2) Proposed

10.8 sec 56.8 sec 82.5 sec

The proposed method carried out 3 times graph cuts segmentation.

And Gaussian Mixture Model is derived from 6 dimensional features.

It is because the proposed method is slow.

Time cost


Discussion (2)Discussion (2)

　 Remained problemThe proposed method is ineffective for a flat

image such as artificial images due to a few edges.

Future work include optimization of the weight to texture and color for segmentation.


Noisy edge problemNoisy edge problem

If the strong edges exist ,

n-link edge cost is small.

(easy to pass the n-link edge)

p q

S

T

p q

S

T

n-link edge cost is large.

(hardly pass the n-link edge)

Else ,


Influence of seeds Influence of seeds positionposition

• Graph cuts technique is quite stable and normally produces the same results regardless of particular seed positioning within the same image.


Prior probabilityPrior probability

• Distance dd from the border between the object and background is normalized from 0.5 to 1.0 .

• The prior probability is defined by using dojb and dbkg.

Prior probability

otherwised

ddifd

bkg

bkgobjobj

1)Pr(O

)Pr(1)Pr( OB


First time First time segmentationsegmentation

S:”obj”

T:”bkg”

p

－ ln Pr(Yp|B)

－ ln Pr(Yp|O)

Pr(Yp|O)

Pr(Yp|B)

Color (RGB) :Cp

Local texture:TpGMM

Color (RGB) :Cp

Local texture:TpGMM

“obj” seeds

“bkg” seeds


Multiresolution levelMultiresolution level

At level 3

The low pass subband LL3 is 1 / 64 * the original image.

The low pass subband LL4 is 1 / 256 * the original image.

At level 2


ParameterParameter

Graph Cuts parameter :

λ = 0.05

σ = 12.0

The constant KK is larger than sum of all n-link costs not to label the opposite label.

Gaussian Mixture Model:

5 components

)|Pr(ln)"("

)|Pr(ln)"("

max1

),(1

2)(

exp

},{:},{

2

2

},{

pp

pp

NqpqqpPp

qpqp

YBbkgR

YOobjR

BK

qpdistII

B


Local Texture featuresLocal Texture features


Results of proposed Results of proposed methodmethod


Result of proposed Result of proposed method method


QuestionQuestion

• Please repeat the question.• Thank you for your question. Are you asking about … ?• What you see here is …• I believe that …• To be honest , we never looked into possibility.• The question was …• There is not the particular deep meaning.• Not yet• Thank you for your advice.


追加スライド追加スライド• n-link の問題について• ユーザの引き方に問題は？• 数パーセントでも違いがはっきりしている• パラメータの説明• Level4 がない理由• 問題点• 一回目のセグメンテーション



Ｓ

Ｔ

10

2

2

2

6

65

3

97

10+5+6+9 = 30

2+2+5+3 = 12 (mincut)

10+2+5+6+9 = 32


Ｓ

Ｔ

2

2

2

6

65

3

97

2 8

input labeled mask output error

Documents

GRAPH CUTS BY USING LOCAL TEXTURE FEATURES OF WAVELET COEFFICIENT FOR IMAGE SEGMENTATION