Background vs. foreground segmentation of video sequences = +

Simple approach (2)

threshold

Simple approach noise can spoil everything

Energy functional data term

Degeneracy can be trivially minimized bybull C 0 (everything is foreground)

bull B N (take original image as background)

Energy functional data term

C 1

Energy functional data term

there should be enough of background

original images should be close to the restored background image

in the background areas

Energy functional smoothness

For background image B

For background mask C

Energy functional

Edge-preserving smoothnessRegularization term

Quadratic regularization [Tikhonov Arsenin 1977]

ELE

Known to produce very strong isotropic smoothing

Edge-preserving smoothnessRegularization term

Change regularization

ELE

Edge-preserving smoothnessRegularization term

ELE

Edge-preserving smoothnessRegularization term

ELE

n

Change the coordinate system

across the edgealong the edge

Compare

Edge-preserving smoothnessRegularization term

Weak edge (s 0)

Conditions on

Isotropic smoothings) is quadratic at zero

(s)

s

Edge-preserving smoothnessRegularization term

Strong edge (s )

Conditions on

bull no smoothing across the edge

bull more smoothing along the edge

Anisotropic smoothings) does not grow too fast at infinity

(s)

s

Edge-preserving smoothnessRegularization term

ConclusionUsing regularization term of the form

we can achieve both

isotropic smoothness in uniform regions

and anisotropic smoothness on edges

with one function

0 1 2 3 4 50

05

1

15

2

25

3

35

4

Edge-preserving smoothnessRegularization term

Example of an edge-preserving function

0 005 01 015 020

0005

001

0015

002

Edge-preserving smoothnessSpace of Bounded Variations

Even if we have an edge-preserving functional

if the space of solutions u contains only smooth functions we may not achieve the desired minimum

-1 -05 0 05 1-1

-05

0

05

1

Edge-preserving smoothnessSpace of Bounded Variations

which one is ldquobetterrdquo

Bounded Variation ndash ND caseBounded Variation ndash ND case

sup |)(| dxdivffD

)( x

N

1i i

i xdiv

1 || )( ) ( )(

101

L

NN C

bounded open subset function NR )( 1 Lf

Variation of over f

where

φ

Edge-preserving smoothnessSpace of Bounded Variations

integrable (absolute value) and with bounded variation

Functions are not required to have an integrable derivative hellip

What is the meaning of u in the regularization term

Intuitively norm of gradient |u| is

replaced with variation |Du|

Total variation

Theorem (informally) if u BV() then

Hausdorff measure

area gt 0area = 0

How can we measure zero-measure sets

Hausdorff measure

1) cover with balls of diameter

2) sum up diameters for optimal cover (do not waste balls)

3) refine 0

Hausdorff measureFormally

For A RN k-dimensional Hausdorff measure of A

up to normalization factor covers are countable

bull HN is just the Lebesgue measure

bull curve in image

its length = H1 in R2

Total variation

Theorem (more formally) if u BV() then

u+

u-

u(x)

xx0

u+ u- - approximate upper and lower limits

Su = x u+gtu-the jump set

Energy functional

data term

regularization for background image

regularization for background masks

Total variation example

-1 -05 0 05 1 15 2

-1

0

1

2

0

05

1

-1 -05 0 05 1 15 2

-1

0

1

2

0

05

1

= perimeter = 4

Divide each side into n parts

Edge-preserving smoothnessSpace of Bounded Variations

Small total variation(= sum of perimeters)

Large total variation(= sum of perimeters)

Edge-preserving smoothnessSpace of Bounded Variations

Small total variation Large total variation

Edge-preserving smoothnessSpace of Bounded Variations

BV informally functions with discontinuities on curves

Edges are preserved texture is not preserved

original sequencetemporal median energy minimization

in BV

Energy functional

Time-discretized problem

Find minimum of E subject to

Existence of solution

Under usual assumptions

12 R+ R+ strictly convex nondecreasing

with linear growth at infinity

minimum of E exists in BV(BC1hellipCT)

(non-)Uniqueness

is not convex wrt (BC1hellipCT) Solution may not be unique

Uniqueness

But if c 3range2(Nt t=1hellipT x ) then the functional is strictly convex and solution is unique

Interpretation if we are allowed to say that everything is foreground background image is not well-defined

Finding solution

BV is a difficult space you cannot write Euler-Lagrange equations cannot work numerically with function in BV

Strategy bull construct approximating functionals admitting solution in a more regular spacebull solve minimization problem for these functionalsbull find solution as limit of the approximate solutions