Adaptive Background Mixture Models for Real-Time Tracking

Adaptive Background Mixture Models forReal-Time Tracking†

Eric Thul

March 30, 2007

†By Chris Stauffer and W.E.L Grimson [4, 1]

Outline

Problem Statement

Methodology

Experimentation

Discussion

Feedback

Problem StatementTopic

The authors are working to develop an adaptive mixture ofGaussians that can model a changing scene.

Question

How can such a probabilistic model be used to distinguish betweenbackground and foreground pixels in the scene?

Rationale

To understand how effective their probabilistic model performswhen applied to real-time tracking.

Significance

Improvement to the stability and reliability for video surveillancesystems compared to other approaches.

Problem StatementOther Approaches

The authors compare their technique to:

I Averaging images over time to threshold-out foregroundpixels.

I Not robust to scenes with many (slow) moving objects

I Single threshold for the entire scene.

I Ridder et al. [3] improvement of modeling each pixel with aKalman filter.

I Not robust to backgrounds with repetitive change.

I Takes significant time to re-establish the background.

I Wren et al. [5] improvement of modeling each pixel with asingle Gaussian.

I Good indoor performance.

I Not tested for out-door scenes (for repetitive change).

Hypothesis

If we model each pixel as a mixture of Gaussians to determinewhether or not a pixel is part of the background, then we willarrive at an effective approach to separate the background andforeground, which can be used for real-time tracking.

Idealized ExampleOriginal Scene

Idealized ExampleBackground of the Scene

Idealized ExampleForeground of the Scene

Idealized ExampleOriginal Scene with Tracking

MethodologyOverview

Two main parts:

1. Probabilistic model for separating the background andforeground.

I Adaptive mixture of multi-modal Gaussians per pixel.

I Method for updating the Gaussian parameters.

I Heuristic for determining the background.

2. Technique for tracking objects in the foreground.

I Algorithm to label connected components.

I Method for tracking each connected component.

MethodologyThe Pixel Process

Definition

We define the pixel process of a pixel X as the history of it’s valuefrom frame 1 to frame t. This can be represented as follows.

{X1, . . . ,Xi , . . . ,Xt} where 1 ≤ i ≤ t

But what does this tell us?

MethodologyThe Pixel Process Examples 1 and 2

Figure: Example 1 of gradualchange over time.

Figure: Example 2 two of repetitivechange.

MethodologyThe Pixel Process Example 1

0 20 40 60 80 100 120 140 160 1800

20

40

60

80

100

120

140

160

180

Red Channel

Gre

en C

hann

el

Pixel Process Frames[1,325] of Pixel[60,20]

Figure: Plot of [red,green] values of a pixel for frames 1 to 325.


0 20 40 60 80 100 120 140 160 1800

20

40

60

80

100

120

140

160

180

Red Channel

Gre

en C

hann

el

Pixel Process Frames[365,700] of Pixel[60,20]

Figure: Plot of [red,green] values of the same pixel for frames 365 to 700.


Tells us there is a need for an adaptive model.


100 105 110 115 120 125 130 135 140125

130

135

140

145

150

155

160

Red Channel

Gre

en C

hann

el

Pixel Process 286 Frames of Pixel[55,80]

Figure: Plot of [red,green] values of a pixel during monitor flicker.


Tells us there is a need for a multi-modal model.

MethodologyProbabilistic Model

Part 1: Probabilistic model for separating the background andforeground.

MethodologyProbabilistic Model: Mixture of Gaussians

Let {X1, . . . , Xt} be a pixel process for X . The probability ofobserving a X at frame t is as follows, where K is the number ofGaussians in the mixture.

P(Xt) =K∑

i=1

weighti ,t ∗ GaussianPDF (Xt , Meansi ,t , Covariancei ,t)

Assume the Covariance matrix is Variance ∗ Identity matrix.

MethodologyProbabilistic Model: Update Parameters

On the next frame, we have a new pixel Xt where t = t + 1. Wemust update our probabilistic model as follows.

Look at each Gaussian in the mixture of pixel Xt .

I If Xt ≤ 2.5 standard deviations from the mean then labelmatched and stop.

I Any Gaussian not matched is labeled unmatched.


From the labeling of each Gaussian we have the following cases.

1. If Gaussiani ,t is marked as matched.

I Increase the weight.

I Adjust the mean closer to Xt .

I Decrease the variance.

2. If Gaussiani ,t is marked as unmatched.

I Decrease the weight.

3. If all the Gaussians in the mixture for pixel Xt are unmatched.

I Mark Xt as a foreground pixel.

I Find least probable Gaussian in the mixture and set:

I Mean = Xt

I Variance as a high valueI weight as a low value


Equations for the update:

I ωi ,t = (1− α) ∗ ωi ,t−1 + α ∗Mi ,t

I µt = (1− ρ) ∗ µt−1 + ρ ∗ Xt

I σ2t = (1− ρ) ∗ σ2

t−1 + ρ ∗ (Xt − µt)T ∗ (Xt − µt)

I ρ = α ∗ η(Xt , µt−1, Σt−1)

α is the learning parameter.

MethodologyProbabilistic Model: Background Heuristic

Since we already have our foreground pixels, we need to determinewhich Gaussians from the mixtures represent the backgroundpixels. We want distributions which have a high weight and lowvariance.

MethodologyProbabilistic Model: Background Heuristic

To find such distributions the following heuristic is proposed.

1. Order the Gaussians of each mixture byweight/standard devation.

2. Sum the weights in this ordering until the sum is greater thana pre-set threshold T .

I Larger T means more pixel values in the background.

3. Use the means from each Gaussian in the sum to representthe background.

MethodologyTracking Technique

Part 2: Technique for tracking objects in the foreground.

MethodologyTracking Technique: Connected Components

We now have the following:

I Model representing the background.

I Marked set of pixels representing the foreground.

The next step is to segment these foreground pixels into connectedcomponents. An algorithm for this is described by Horn [2].

MethodologyTracking Technique: Horn’s Algorithm

A

D C

B

Figure: A is not marked so continue

MethodologyTracking Technique: Horn’s Algorithm Cases

Given A marked:

I Case 1: D is labeled so copy label to A

I Case 2: B or C are labeled exclusively so copy label to A

I Case 3: B and C are not labeled so pick new label for A

I Case 4: B and C are both labeled

I Case 4a: B and C are labeled differently so copy label andnote equality

I Case 4b: B and C have the same label so copy label to A


A

D C

B

Figure: Case 3: B and C are not labeled so pick new label for A


A

D C

B

Figure: Case 2: B or C are labeled exclusively so copy label to A


A

D C

B

Figure: Case 3: B and C are not labeled so pick new label for A


A

D C

B

Figure: Case 4b: B and C are labeled differently so copy label and noteequal


A

D C

B

Figure: Case 1: D is labeled so copy label to A


A

D C

B

Figure: A is not labeled so continue

MethodologyTracking Technique: Multiple Hypothesis Tracker

The result of Horn’s algorithm is a set of connected componentswe can track across each frame. The essential tool for this is a setof Kalman filters.

Three main steps at each frame:

1. Find the Kalman filter which fits a connected component.

2. Create a new Kalman filter if no existing ones fit.

3. Remove any Kalman filters which are no longer relevant.

Experimentation

The method described has been applied to the following.

I Five outdoor scenes from stationary cameras.

I Technology Square, Cambridge Massachusetts

I Used for over 16 months

I Sun, Overcast, Snow, Rain

I Tracked people and cars

I Other experiments.

I Birds at a feeder, mice at night, fish in a tank, people enteringa lab.

ResultsOvercast Day†

Figure: Original image during anovercast day.

Figure: Foreground detection ofpeople in the scene.

†http://www.ai.mit.edu/projects/vsam/Tracking/index.html

http://www.ai.mit.edu/projects/vsam/Tracking/index.html

ResultsSnowy Day†

Figure: Original image during asnowy day.

Figure: Foreground detection of carsand snow in the scene.



ResultsSunny Day†

Figure: Original image during ansunny day.

Figure: Foreground detection of carsand people in the scene.



ResultsResults Overall

I The Good.

I Robust against rain, snow, sleet, hail, overcast.

I Handles motion from swaying branches, rippling water, noise.

I Works under day and night cycles.

I The Bad.

I Difficulty in full sun (long shadows).

I Very windy and partly cloudy days were also challenges.

I Difficulty with objects overlapping.

I Fast lighting changes were an issue.

ResultsMy Results

Let’s go to the video tape.

DiscussionDrawbacks/Problems

Problems during implementation:

I As time increases, the variance decreases for stable pixels. Ifthe variance gets too small, then noise from the camera ismarked as foreground pixels.

I The mixtures may adapt too quickly to slow moving objectsthat are uniform in color. This will incorporate a foregroundobject into the background.

I The world may not be Gaussian.

DiscussionFuture Work

Authors’:

I More Gaussians in the mixture.

I Full covariance matrix.

I Adding prediction for each Gaussian may improve trackingduring lighting changes.

Me:

I If more than one Gaussian in a mixture can be matched to thenew pixel, pick the closest.

I Factor in the weight when finding the least probabledistribution to replace.

I Fish! (If you have pet fish, I’m very interested)

Feedback

Comments / Questions / Suggestions

References

[1] W.E.L Grimson, C. Stauffer, R. Romano, and L. Lee. Usingadaptive tracking to classify and monitor activities in a site. InComputer Vision and Pattern Recognition, pages 22–29, 1998.

[2] B.K.P. Horn. Robot Vision. The MIT Press McGraw-Hill BookCompany, 1986.

[3] Christof Ridder, Olaf Munkelt, and Harald Kirchner. Adaptivebackground estimation and foreground detection usingkalman-filtering. In International Conference on recentAdvances in Mechatronics, pages 193–199, 1995.

[4] Chris Stauffer and W.E.L Grimson. Adaptive backgroundmixture models for real-time tracking. In Computer Vision andPattern Recognition, volume 2, pages 252–258, 1999.

[5] Christopher R. Wren, Ali Azarbayejani, Trevor Darrell, andAlex Pentland. Pfinder: Real-time tracking of the human body.In Pattern Analysis and Machine Intelligence, volume 19,pages 780–785, 1997.

Documents

Adaptive Background Mixture Models for Real-Time Tracking