Image Segmentation by Branch-and-Mincut

Image Segmentation by Branch-and-Mincut

Victor Lempitsky Andrew Blake

Carsten Rother

Victor Lempitsky Andrew Blake

Carsten Rother

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

An exampleAn example

1

0

Fp=0, Bp=1

Fp=1, Bp=0 Ppq

image from UIUC car dataset

Standard “graph cut” segmentation energy [Boykov, Jolly 01]:

......

F p

Bp

P pq p q

s

t

[Freedman, Zhang 05], [Ali, Farag, El-Baz 07],...

10

[Hammer 64] [Greig, Porteous, Seheult 89]

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

10

A harder exampleA harder exampleimage from UIUC car dataset

Optimal ω

Ppq

Optimal x

minx, ω

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Energy optimizationEnergy optimization

Optimization options:

• Choose reasonable ω, solve for x [Freedman, Zhang 05], [Pawan Kumar, Torr, Zisserman’ObjCut 05], [Ali, Farag, El-Baz 07] ....

• Alternate between x and ω (EM) [Rother, Kolmogorov, Blake’ GrabCut 04], [Bray, Kohli, Torr’PoseCut 06], [Kim, Zabih 03]....

• Optimize continuously [Chan, Vese 01], [Leventon, Grimson, Faugeras 00], [Cremers, Osher, Soatto 06], [Wang, Staib 98]...

• Exhaustive search

constant unary potentials (costs) pairwise potentials (costs) ≥ 0ω – global parameter (ω ∊ Ω0)

....shape priors, color distribution/intensity priors (Chan-Vese, GrabCut)...

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Our approachOur approach

[Gavrila, Philomin 99], [Lampert, Blaschko, Hofman 08], [Cremers, Schmidt, Barthel 08]

along x dimensionalong ω dimension

• Extremely large, structured domain• Specific “graph cut” function

• Low-dimensional (discretized) domain• Function of the general form

Branch-and-boundBranch-and-bound

Mincut

Branch-and-Mincut

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Search treeSearch tree

Ω0

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Bounding the energy: an exampleBounding the energy: an example

Fp=1, Bp=0

Fp=0, Bp=1ω1 ω2 Fp=0, Bp=0

min

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

The lower boundThe lower bound

F p

Bp

P pq p q

s

tComputable with mincut!

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Lower boundLower bound

• Monotonic increase towards leaves• Tightness at leaf nodes:

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Lower bound: exampleLower bound: exampleprecomputed computed at runtime

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Branch-and-BoundBranch-and-Bound

Standard best-first branch-and-bound search:

additional speed-up from “reusing” maxflow computations [Kohli,Torr 05]

Small fraction of nodes is visited

lowest lower bound

A

B

C

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Results: shape priorResults: shape prior

30,000,000 shapes

Exhaustive search: 30,000,000 mincutsBranch-and-Mincut: 12,000 mincuts

Speed-up: 2500 times(30 seconds per 312x272 image)

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Results: shape priorResults: shape priorLeft ventricle epicardium tracking (work in progress)

Our segmentationShape prior from other sequences

5,200,000 templates≈20 seconds per frame

Speed-up 1150

Data courtesy: Dr Harald Becher, Department of Cardiovascular Medicine, University of Oxford

Original sequence No shape prior

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Result: shape priorResult: shape prior

Can add feature-based detector

here

Can add feature-based detector

here

UIUC car dataset

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Results: Discrete Chan-Vese functional

Results: Discrete Chan-Vese functional

Global minima of the discrete Chan-Vese functional:

Chan-Vese functional [Chan, Vese 01]:

∊ [0;255]x[0;255]: quad-tree clustering

Speed-up 28-58 times

c_pc_pcf

cb

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

PerformancePerformance

Sample Chan-Vese problem:

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

Results: GrabCutResults: GrabCut

E = -618 E = -624 (speed-up 481) E = -628

E = -593 E = -584 (speed-up 141) E = -607

• ω corresponds to color mixtures• [Rother, Kolmogorov, Blake’ GrabCut 04] uses EM-like search• Branch-and-Mincut searches over 65,536 starting points

V. Lempitsky, A. Blake, C. Rother“Image Segmentation by Branch-And-Mincut”

ConclusionConclusion

•

good energy to integrate low-level and high-level knowledge in segmentation.

• Branch-and-Mincut framework can find its global optimum efficiently in many cases

• Ongoing work: Branch-and-X algorithms

C++ code at http://research.microsoft.com/~victlem/

Branch-and-bound

Branch-and-boundMincut

Dynamic Programm

ing