Object Segmentation

Stella X. Yu

Department of Computer Science

University of California at Berkeley

Yu: Bay Area Vision Meeting 2003 – p.1/20

Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.2/20

Direct Feature Classification for Object Detection

image

recognition

Yu: Bay Area Vision Meeting 2003 – p.3/20

Why Perceptual Organization

15

9

Mahamud multi-object detector

Yu: Bay Area Vision Meeting 2003 – p.4/20

Traditional Use of Perceptual Organization

image

segmentation

figure-ground

recognitionperceptual organization

sequential processing (Marr, Lowe, Witkin, Tenenbaum, ...)

Yu: Bay Area Vision Meeting 2003 – p.5/20

Our Approach to Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.6/20

Our Approach to Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.6/20

Our Approach to Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.6/20

Our Approach to Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.6/20

Our Approach to Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.6/20

Our Approach to Object Segmentation

Yu: Bay Area Vision Meeting 2003 – p.6/20

Grouping in a Graph-Theoretic Framework

�

�

�

��

�

�

�

�

�

��

Yu: Bay Area Vision Meeting 2003 – p.7/20

Grouping in a Graph-Theoretic Framework

�

�

�

��

�

�

�

�

�

��

Representation: G = {V, E, W} = { nodes, edges, weights }

Yu: Bay Area Vision Meeting 2003 – p.7/20

Grouping in a Graph-Theoretic Framework

�

�

�

��

�

�

�

�

�

��

Representation: G = {V, E, W} = { nodes, edges, weights }

Clustering: ΓKV = {V1, . . . , VK} = K-way node partitioning

(Shi & Malik, Zabih, Boykov, Veksler, Kolmogorov,...)

Yu: Bay Area Vision Meeting 2003 – p.7/20

Links in Graph Cuts

�

�

�

��

�

�

�

�

�

��

Yu: Bay Area Vision Meeting 2003 – p.8/20

Links in Graph Cuts

�

�

�

��

�

�

�

�

�

��

�

�

�

��

�

�

�

�

�

��

P Q

Yu: Bay Area Vision Meeting 2003 – p.8/20

Links in Graph Cuts

�

�

�

��

�

�

�

�

�

��

P Q

links(P, Q) =∑

p∈P, q∈Q

W (p, q)

Yu: Bay Area Vision Meeting 2003 – p.8/20

Degree in Graph Cuts

�

�

�

��

�

�

�

�

�

��

P Q

degree(P) = links(P, V)

Yu: Bay Area Vision Meeting 2003 – p.9/20

Linkratio in Graph Cuts

�

�

�

��

�

�

�

�

�

��

P Q

linkratio(P, Q) =links(P, Q)

degree(P)

Yu: Bay Area Vision Meeting 2003 – p.10/20

Goodness of Grouping in Graph Cuts

�

�

�

��

�

�

�

�

�

��

P

V \ P

Maximize within-group connections: linkratio(P, P)

Minimize between-group connections: linkratio(P, V \ P)

Equivalent: linkratio(P, P) + linkratio(P, V \ P) = 1

Yu: Bay Area Vision Meeting 2003 – p.11/20

K-Way Normalized Cuts

�

�

�

��

�

�

�

�

�

��

V1 V2

V3

max knassoc(ΓKV ) =

1

K

K∑

l=1

linkratio(Vl, Vl)

min kncuts(ΓKV ) =

1

K

K∑

l=1

linkratio(Vl, V \ Vl)

Yu: Bay Area Vision Meeting 2003 – p.12/20

Joint Pixel-Patch Grouping: Criterion

A

B

C

ε̄(ΓKV , ΓK

U ; A, B) =1

K

K∑

l=1

linkratio(Ul, Ul; B) · degree(Ul; B)

degree(Vl; A) + degree(Ul; B)

+1

K

K∑

l=1

linkratio(Vl, Vl; A) · degree(Vl; A)

degree(Vl; A) + degree(Ul; B)

Yu: Bay Area Vision Meeting 2003 – p.13/20

Joint Pixel-Patch Grouping: Consistency

A

B

C

ΓKU = {U1, . . . , UK}, ΓK

V = {V1, . . . , VK}

Bias linking patches with their pixels

Yu: Bay Area Vision Meeting 2003 – p.14/20

Computing Constrained Normalized Cuts

• Constrained eigenvalue problem

• Efficient solution using a projector onto the feasible space

• Generalize Rayleigh-Ritz theorem to projected matrices

Yu: Bay Area Vision Meeting 2003 – p.15/20

How Object Knowledge Helps Segmentation

pixel only pixel w/ ROI pixel-patch

541s 150s 110s

Yu: Bay Area Vision Meeting 2003 – p.16/20

How Segmentation Helps Object Detection

image patch density segmentation

Yu: Bay Area Vision Meeting 2003 – p.17/20

When Does Our Method Fail

image patch density segmentation

Yu: Bay Area Vision Meeting 2003 – p.18/20

Equally Applicable to Multiple Objects

Yu: Bay Area Vision Meeting 2003 – p.19/20

Acknowledgements

• Shyjan MahamudJianbo Shi

• CMU HumanID LabCMU Hebert Lab

• ONR N00014-00-1-0915NSF IRI-9817496

Yu: Bay Area Vision Meeting 2003 – p.20/20