Lec13-Image Segmentation part II - CS Departmentbagci/teaching/computervision17/Lec13.pdf•Each pixel = node •Add two nodes F & B •Labeling: link each pixel to either F or B F

CAP5415-ComputerVisionLecture13-Image/VideoSegmentation

PartII

[email protected]

1

Labeling &Segmentation• Labeling isacommonwayformodelingvariouscomputervisionproblems(e.g.opticalflow,imagesegmentation,stereomatching,etc)

2


• Thesetoflabelscanbediscrete(asinimagesegmentation)

3

L = {l1, . . . , lm} with L = m


• Thesetoflabelscanbediscrete(asinimagesegmentation)

• Orcontinuous(asinopticalflow)

4

L = {l1, . . . , lm} with L = m

L ⇢ Rnfor n � 1

Labelingisafunction• Labelsareassignedtosites (pixellocations)

5

Labelingisafunction• Labelsareassignedtosites (pixellocations)• Foragivenimage,wehave

6

|⌦| = Ncols

.Nrows

Labelingisafunction• Labelsareassignedtosites (pixellocations)• Foragivenimage,wehave• Identifyingalabelingfunction(withsegmentation)is

7

|⌦| = Ncols

.Nrows

f : ⌦ ! L


Weaimatcalculatingalabelingfunctionthatminimizesagiven(total)errororenergy

8

|⌦| = Ncols

.Nrows

f : ⌦ ! L


Weaimatcalculatingalabelingfunctionthatminimizesagiven(total)errororenergy

*Aisanadjacencyrelationbetweenpixellocations9

|⌦| = Ncols

.Nrows

f : ⌦ ! L

E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]

EnergyFunctionTheroleofanenergyfunctioninminimization-basedvisionistwofold:

1. asthequantitativemeasureoftheglobalqualityofthesolutionand

2. asaguidetothesearchforaminimalsolution.

Correctsolutionisembeddedastheminimum.10

SegmentationasanEnergyMinimizationProblem

• Edata assignsnon-negativepenaltiestoapixellocation pwhenassigningalabeltothislocation.

11



• Esmooth assignsnon-negativepenaltiesbycomparingtheassignedlabelsfp andfq atadjacentpositionsp andq.

12




13




Thisoptimizationmodelischaracterizedbylocalinteractionsalongedgesbetweenadjacentpixels,andoftencalledMRF(MarkovRandomField)model.

14

EnergyFunction-Details

15

E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]


• SampleDataTerm:

16

E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]

Edata(p, fp) = (p)

(p = 0) = �logP (p 2 BG)

(p = 1) = �logP (p 2 FG)


• SampleDataTerm:

• SampleSmoothnessTerm:

17

E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]

Edata(p, fp) = (p)

(p = 0) = �logP (p 2 BG)

(p = 1) = �logP (p 2 FG)

(p, q) = Kpq�(p 6= q) where

Kpq =exp(��(Ip � Iq)2/(2�2))

||p, q||


18

E(p) =X

p2⌦

p(p) +X

p2⌦

X

q2A(p)

pq(p, q)

p⇤ = argminp2L

E(p)


• Tosolvethisproblem,transformtheenergyfunctionalintomin-cut/max-flowproblemandsolveit!

19

E(p) =X

p2⌦

p(p) +X

p2⌦

X

q2A(p)

pq(p, q)

p⇤ = argminp2L

E(p)

EnergyFunction

20

UnaryCost(dataterm)

DiscontinuityCost

p q

Recap:ImageasaGraph

21

Node/vertexEdge

pixel

p q

GraphCutsforOptimalBoundaryDetection(BoykovICCV2001)

22

F

B

F

B

F

F F

F B

B

B

• Binarylabel:foregroundvs.background• Userlabelssomepixels• Exploit

– StatisticsofknownFg &Bg– Smoothnessoflabel

• Turnintodiscretegraphoptimization– Graphcut(mincut/maxflow)

Graph-Cut

23

EachpixelisconnectedtoitsneighborsinanundirectedgraphGoal:splitnodesintotwosetsAandBbasedonpixelvalues,andtrytoclassifyNeighborsinthesamewaytoo!

Graph-Cut

24

EachpixelisconnectedtoitsneighborsinanundirectedgraphGoal:splitnodesintotwosetsAandBbasedonpixelvalues,andtrytoclassifyNeighborsinthesamewaytoo!

Graph-Cut

25

• Eachpixel=node• AddtwonodesF&B• Labeling:linkeachpixeltoeitherForB

F

B

F

B

F

F F

F B

B

B

Desiredresult

CostFunction:Dataterm

26

• PutoneedgebetweeneachpixelandbothF&G• Weightofedge=minusdataterm

B

F

CostFunction:Smoothnessterm

27

• Addanedgebetweeneachneighborpair• Weight=smoothnessterm

B

F

Min-Cut

28

• Energyoptimizationequivalenttographmincut• Cut:removeedgestodisconnectFfromB• Minimum:minimizesumofcutedgeweight

B

Fcut

Min-Cut

29

Source

Sink

v1 v2

2

5

9

42

1Graph (V, E, C)

Vertices V = {v1, v2 ... vn}Edges E = {(v1, v2) ....}Costs C = {c(1, 2) ....}

Min-Cut

30

Source

Sink

v1 v2

2

5

9

42

1

Whatisast-cut?

Anst-cut(S,T)dividesthenodesbetweensourceandsink.

Whatisthecostofast-cut?

SumofcostofalledgesgoingfromStoT

5 + 2 + 9 = 16

Min-Cut

31

Whatisast-cut?

Anst-cut(S,T)dividesthenodesbetweensourceandsink.

Whatisthecostofast-cut?

SumofcostofalledgesgoingfromStoT

Source

Sink

v1 v2

2

5

9

42

1

2 + 1 + 4 = 7

Whatisthest-mincut?

st-cutwiththeminimumcost

Howtocomputemin-cut?

32

Source

Sink

v1 v2

2

5

9

42

1

Solve the dual maximum flow problem

In every network, the maximum flow equals the cost of the st-mincut

Min-cut\Max-flow Theorem

Compute the maximum flow between Source and Sink

Constraints

Edges: Flow < Capacity

Nodes: Flow in & Flow out

Max-FlowAlgorithms

33

Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Sink

v1 v2

2

5

9

42

1

Algorithms assume non-negative capacity

Flow=0

Max-FlowAlgorithms

34






Source

Sink

v1 v2

2

5

9

42

1

Flow=0

Max-FlowAlgorithms

35






Source

Sink

v1 v2

2-2

5-2

9

42

1

Flow=0+2

Max-FlowAlgorithms

36






Source

Sink

v1 v2

0

3

9

42

1

Flow=2

Max-FlowAlgorithms

37






Source

Sink

v1 v2

0

3

9

42

1

Flow=2

Max-FlowAlgorithms

38






Source

Sink

v1 v2

0

3

9

42

1

Flow=2

Max-FlowAlgorithms

39






Source

Sink

v1 v2

0

3

5

02

1

Flow=2+4

Max-FlowAlgorithms

40






Source

Sink

v1 v2

0

3

5

02

1

Flow=6

Max-FlowAlgorithms

41






Source

Sink

v1 v2

0

3

5

02

1

Flow=6

Max-FlowAlgorithms

42






Source

Sink

v1 v2

0

2

4

02+1

1-1

Flow=6+1

Max-FlowAlgorithms

43






Source

Sink

v1 v2

0

2

4

03

0

Flow=7

Max-FlowAlgorithms

44






Source

Sink

v1 v2

0

2

4

03

0

Flow=7

AnotherExample-MaxFlow

45

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

Ford & Fulkerson algorithm (1956)

Find the path from source to sink

While (path exists)

flow += maximum capacity in the path

Build the residual graph (“subtract” the flow)

Find the path in the residual graph

End


46

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



While (path exists)




End


47

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



While (path exists)




End

flow = 3


48

source

sink

9

5

6-3

8-3

42+3

2

2

5-3

3-3

5

5

311

6

2

3



While (path exists)




End

flow = 3

+3


49

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 3

3


50

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 3

3


51

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 6

3


52

source

sink

9-3

5

3

5-3

45

2

2

2

5

5

3-311

6

2

3-3



While (path exists)




End

flow = 6

3

+3

+3


53

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 6

3

3

3


54

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 6

3

3

3


55

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 11

3

3

3


56

source

sink

6-5

5-5

3

2

45

2

2

2

5-5

5-5

1+51

6

2+5



While (path exists)




End

flow = 11

3

3

3


57

source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 11

3

33


58

source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 11

3

33


59

source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 13

3

33


60

source

sink

1

3

2-2

45

2-2

2-2

2-2

61

6

7



While (path exists)




End

flow = 13

3

33

+2

+2


61

source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 13

3

33

2

2


62

source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 13

3

33

2

2


63

source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 15

3

33

2

2


64

source

sink

1

3-2

4-25

61

6-2

7



While (path exists)




End

flow = 15

3

33+2

2

2-2

+2


65

source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

35

2

2

2


66

source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

35

2

2

2


67

source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

35

2

2

2


68

source

sink

1

1

5

61

4

7


Why is the solution globally optimal ?

flow = 15

3

35

2

2

2


69

source

sink

1

1

5

61

4

7



1. Let S be the set of reachable nodes in the

residual graph

flow = 15

3

35

2

2

2

S


70



1. Let S be the set of reachable nodes in the

residual graph

2. The flow from S to V - S equals to the sum

of capacities from S to V – S

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S


71flow = 15



1. Let S be the set of reachable nodes in the residual graph

2. The flow from S to V - S equals to the sum of capacities from S to V – S

3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A

4. The solution is globally optimal

source

sink

8/9

5/5

5/6

8/8

2/40/2

2/2

0/2

5/5

3/3

5/5

5/5

3/30/10/1

2/6

0/2

3/3

Individual flows obtained by summing up all paths


72

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

T

cost = 18

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

T


73

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

T

cost = 23


74

C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 8(1-x5) + 5(1-x6)

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1 x2

x3

x4

x5

x6


75

C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 5(1-x5) + 5(1-x6)

+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1 x2

x3

x4

x5

x6


76

C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 5(1-x5) + 5(1-x6)

+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

=

3 + 3x1(1-x3) + 3x3(1-x5)

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1 x2

x3

x4

x5

x6


77

C(x) = 3 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 2x5(1-x4) + 6(1-x4)

+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)

+ 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)

3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)

=

3 + 3x2(1-x6) + 3x6(1-x5)

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

33

x1 x2

x3

x4

x5

x6


78

C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)

+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2

3

3

3

x1 x2

x3

x4

x5

x6


79

C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)

+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)

+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)

source

sink

1

1

5

61

4

7

3

35

2

2

2x1 x2

x3

x4

x5

x6 cost = 0

min cut

S

T

cost = 0

• All coefficients positive

• Must be global minimum

S – set of reachable nodes from s

HistoryofMax-FlowAlgorithms

80

Augmenting Path and Push-Relabel

n:#nodesm:#edgesU:maximumedgeweight

Algorithms assume non-negative edge

weights

Ford-FulkersonAlg.

81

In each iteration of the Ford-Fulkerson method, we find some augmenting path p and increase the flow f on each edge of p by the residual capacity cf(p).

Ifuandvarenotconnectedbyanedgeineitherdirection,weassumeimplicitlythatf[u,v]=0.

Thecapacitiesc(u,v)areassumedtobegivenalongwiththegraph,andc(u,v)=0if(u,v)∈E.

ExampleSegmentationinaVideo

82

Image Flow GlobalOptimum

pqw

n-links st-cuts

t

= 0

= 1

E(x) x*

Recap:DataTerm

83

Withoutspatialrelationship,datatermcanbethesameforregionsthatBelongtodifferentobjects.

Imagepatchesshowtheirsimilaritiesalthoughtheypertaintodifferentobjects

SegmentationExample

84

SegmentationExample

85

SegmentationExample

86

Optimality?

87

Seeds GrabCut(iterativeGCWithboxprior)

VascularSurfaceSegmentationviaGC10/1/15

88

GCRefinement

89

Kohli andTorr

GCSegmentationinVideos

90

SegmentationofhumanlamewalkinavideosequencesKohli andTorr

SlideCreditsandReferences• Fredo DurandofMIT• M.Tappen ofAmazon• R.Szelisky,Univ.ofWashington/Seatle• http://www.csd.uwo.ca/faculty/yuri/Abstracts/eccv06-tutorial.html• J.Malcolm,GraphCutinTensorScale• InteractiveGraphCutsforOptimalBoundary&RegionSegmentationofObjectsin

N-Dimages.YuriBoykov andMarie-PierreJolly.InInternationalConferenceonComputerVision,(ICCV),vol.I,2001.http://www.csd.uwo.ca/~yuri/Abstracts/iccv01-abs.html

• http://www.cse.yorku.ca/~aaw/Wang/MaxFlowStart.htm• http://research.microsoft.com/vision/cambridge/i3l/segmentation/GrabCut.htm• http://www.cc.gatech.edu/cpl/projects/graphcuttextures/• AComparativeStudyofEnergyMinimizationMethodsforMarkovRandomFields.

RickSzeliski,Ramin Zabih,DanielScharstein,OlgaVeksler,VladimirKolmogorov,Aseem Agarwala,MarshallTappen,Carsten Rother.ECCV2006www.cs.cornell.edu/~rdz/Papers/SZSVKATR.pdf

• P.Kumar,OxfordUniversity.

91

Documents

Lec13-Image Segmentation part II - CS Departmentbagci/teaching/computervision17/Lec13.pdf•Each pixel = node •Add two nodes F & B •Labeling: link each pixel to either F or B F