Background Subtraction for Temporally Irregular

Dynamic TexturesGerald Dalley, Joshua Migdal,

and W. Eric L. Grimson

Workshop on Applications of Computer Vision (WACV) 2008

8 Jan. 2008 2

Key Prior Approaches

MoG – learn modes for all pixels & track changes Works as long as dynamic textures are regular

Per-pixel kernels (Mittal & Paragios) Can pay attention to higher-level features like optical flow How many kernels to keep?

Spatial kernels (Sheikh & Shah) Handles temporal irregularity in dynamic textures How many kernels to keep?

8 Jan. 2008 3

Results with Mittal & Paragios Traffic Clip

Key: bboxes from

ground truth True

positive False

positive False

negative

No ground truth available

8 Jan. 2008 4

Our Goals

Compactness Speed Complexity growth

Grow with scene complexity, not with the amount of time to see the complexity If a leaf blows in front of a pixel once a minute, we don’t

want to have to keep 1800 kernels

Usable in MoG frameworks MoG variants are very widely used (original

Stauffer & Grimson paper has 1000+ citations)

8 Jan. 2008 5

Pixels Affected by a GivenMixture ComponentOur Model

Mixture of Gaussians (MoG)

Standard MoG

Ours

p(ci j©) /P

j 2N iwj N (ci ;¹ j ;§ j )

ci theobserved color at pixel i

© themodel fwj ;¹ j ;§ j gj

Ni neighborhood of pixel i

p(ci j©) /P

j 2N iwj N (ci ;¹ j ;§ j )

ci theobserved color at pixel i

© themodel fwj ;¹ j ;§ j gj

Ni neighborhood of pixel i

8 Jan. 2008 6

Foreground/Background Classification Find best matching background Gaussian, j

Use neighborhood Squared Mahalanobis Distance

-

=

input

d2i j

mixture model

©1 ©2 ©3

di j = (ci ¡ ¹ j )T § ¡ 1j (ci ¡ ¹ j )di j = (ci ¡ ¹ j )T § ¡ 1j (ci ¡ ¹ j )

8 Jan. 2008 7

hard Stauffer-Grimson

soft Stauffer-Grimson

best match only

update all matches

Model Update OptionsHard UpdatesSoft Updates

Loca

lR

egio

nal

8 Jan. 2008 8

Results on a Classic Dataset: Wallflower WavingTrees

Ground

Truth

Toyam

a et al.

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 9

ROC (Wallflower)

8 Jan. 2008 10

Traffic #410G

round T

ruthM

ittal &

Paragios

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 11

Traffic #150G

round T

ruthM

ittal &

Paragios

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 12

Pushing the Limits:Ducks

Ground

Truth

B

est M

oGO

urs

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 13

Pushing the Limits:Jug

Ground

Truth

Zhong &

S

claroffB

est M

oGO

urs

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 14

Parameter Sensitivity

0 1000 2000 3000 4000 5000 6000 70000

0.2

0.4

0.6

0.8

1

min number of mistakes

frequ

ency

ws=7ws=5ws=3ws=1

traffic

0 500 1000 1500 20000

0.05

0.1

0.15

min number of mistakes

frequ

ency

ws=7ws=5ws=3ws=1

wallflower

max max

8 Jan. 2008 15

Summary

Model representation Same as MoG

FG/BG Classifier Input Add local neighborhood loop

Update Several options, including standard MoG updates

Results Reduces foreground false positives Reduces need for exotic input features (e.g. optical flow) Cost window size∝

8 Jan. 2008 16

Also in the Paper…

Analysis Timing experiments More ROC analysis

Implementation Details MRF choice Update formulas

8 Jan. 2008 17

Thank You

Questions...

8 Jan. 2008 18

Backup Slides

8 Jan. 2008 19

PureSoft SoftLocal PureHard HardLocal0

10

20

30

40

50

60R

elat

ive

Tim

e C

ost

Model Update Computational Costs

W=12

W=32

W=52

W=72

8 Jan. 2008 208 Jan. 2008 20

PureSoft SoftLocal PureHard HardLocal0

20

40

60

80

100R

elat

ive

Tim

e C

ost

Mahalanobis Map Computational Costs

W=12

W=32

W=52

W=72

8 Jan. 2008 21

12345678

1 2 3 4 1 2 3 4 1 2 3 41 2 3 4

Background Model

Observed Image

9x9 Matchin

g

3x3 Matchin

g

Small Windows are Still Useful

8 Jan. 2008 22

c(t¡ 1)ic(t¡ 1)ic(t¡ 1)i c(t)

ic(t)ic(t)i

Á(t)jÁ(t)j

z(t¡ 1)iz(t¡ 1)i

l(t¡ 1)jl(t¡ 1)jl(t¡ 1)j l(t)jl

(t)jl(t)j

JJ

II

z(t)iz(t)i

Á(t¡ 1)jÁ(t¡ 1)j

Generative Model

Particle locations

Particle appearances

Pixel-to-particle assignments

Pixel appearance

8 Jan. 2008 23

Independent Assignments

l(t¡ 1)1

l(t¡ 1)2

l(t¡ 1)3

z(t)i = 2

l(t)1

l(t)2

l(t)3

l(t¡ 1)1l(t¡ 1)1l(t¡ 1)1

l(t¡ 1)2l(t¡ 1)2l(t¡ 1)2

l(t¡ 1)3l(t¡ 1)3l(t¡ 1)3

z(t)i = 2z(t)i = 2z(t)i = 2

l(t)1l(t)1l(t)1

l(t)2l(t)2l(t)2

l(t)3l(t)3l(t)3