Lecture25 Tracking

8/2/2019 Lecture25 Tracking

1/26

Tracking


2/26

Definition of Tracking

Tracking: Generate some conclusions about the motion of

the scene, objects, or the camera, given a

sequence of images.

Knowing this motion, predict where things are

going to project in the next image, so that wedont have so much work looking for them.


3/26

Why Track?

Detection andrecognition are expensive

If we get an idea ofwhere an object is in

the image because we

have an idea of the

motion from previous

images, we need lesswork detecting or

recognizing the object.

Scene

Image Sequence

Detection+ recognition

Tracking

(less detection,no recognition) New

person

A

B

C

A B

C


4/26

Tracking a Silhouette by

Measuring Edge Positions Observations are positions of edges along normals to tracked contour


5/26

Why not Wait and Process the

Set of Images as a Batch? In a car system, detecting and tracking

pedestrians in real time is important.

Recursive methods require less computing


6/26

Implicit Assumptions of

Tracking

Physical cameras do not move instantlyfrom a viewpoint to another

Object do not teleport between places

around the scene

Relative position between camera and scene

changes incrementally

We can model motion


7/26

Related Fields

Signal Detection and Estimation Radar technology


8/26

The Problem: Signal Estimation

We have a system with parameters Scene structure, camera motion, automatic zoom

System state is unknown (hidden)

We have measurements

Components of stable feature points in the

images.

Observations, projections of the state.

We want to recover the state components fromthe observations


9/26

Necessary Models

System Dynamics

Model

Measurement

Model (projection)

(u, v)

Previous StateNext State

StateMeasurement

We use models to describe a priori knowledge about

the world (including external parameters of camera)

the imaging projection process


10/26

State variable

a

A Simple Example of Estimation

by Least Square Method Goal: Find estimate of state

such that the least squareerror between measurements

and the state is minimum

a

=

=n

i

i axC1

2)(2

1

==

===

n

i

i

n

i

i anxaxaC

11

)(0

==n

i

ixn

a1

1

aMeasurement

x

x

txi

a xi-a

ti


11/26

Recursive Least Square Estimation We dont want to wait until

all data have been collected

to get an estimate of thedepth

We dont want to reprocessold data when we make a

new measurement

Recursive method: data at

step i are obtained from

data at step i-

1

a

State variablea

Measurement

x

x

t

a

xi

1 ia ia


12/26

Recursive Least Square Estimation 2

Recursive method: data at

step i are obtained fromdata at step i-1

x

t

a

xi

)(1

11 += iiii axiaa

i

i

k

k

i

k

ki xi

xi

xi

a 1111

11

==+==

=

=1

1

1

1

1

i

k

ki x

i

a

iii xi

ai

ia

1

1 1 +

=


13/26

Recursive Least Square Estimation 3

)(1 11 += iiii ax

i

aaEstimate at step i

Predicted

measure

Innovation

GainActual

measure

Gain specifies how much

do we pay attention

to the difference

between what we expected

and what we actually get


14/26

Least Square Estimation of the

State Vector of a Static System

1. Batch method

a

H1 Hi

H2

x1

xi

x2

aH = iix a

H

H

H

=

nnx

x

x

......

2

1

2

1

aHx =

measurement equation

XHH)(HaT1T =

Find estimate that minimizesa

a)H(Xa)H(XT =

2

1C

We find


15/26

Least Square Estimation

of the State Vectorof a Static System 2

2. Recursive method

a

H1 Hi

H2

x1

xi

x2

) -1iiii-1ii aH(XKaa +=

with

Calculation is similar to calculation of running average

Now we find:

We had: )(1

11

+=iiii

axi

aa

1)( =

=T

iii

T

iii

HHP

HPK

Gain matrixInnovation

Predicted measure


16/26

Dynamic System

i

i

i

i

AV

X

a

ixwAA

tAVV

tVXX

ii

iii

iii

+=

+=

+=

1

11

11

+

=

wA

VX

tt

A

VX

i

i

i

i

i

i

00

100

1001

1

1

1

State of rocket

Measurement

waa -1ii +=

[ ] V

A

V

X

x

i

i

i

i +

= 001 Vxi

+=i

aH

State equation for rocket

Measurement equation

Noise

Tweak factor


17/26

) -1iiiii-1iii aH(xKaa +=

Recursive Least Square

Estimation for a Dynamic System(Kalman Filter)

-1i-1iii waa +=

iiii naHx+=

),0(~ ii Qw N

)R,0(~n ii N

-1i-1i-1i-1i

-1i

T

i-1iii

-1i

Tiii

Tiii

)P'HK(IP

QPP'

RHP'HHP'K

=

+=

+= )(

State equation


Tweak factor for model

Measurement noise

Prediction forxi

Prediction for ai

Gain

Covariance matrix for

prediction error

Covariance for estimation error


18/26

-1i-1i-1i-1i

-1i

T

i

-1i

i

i

-1

i

T

iii

T

iii

)P'HKIP

QafP

afP'

RHP'HHP'K

=

+

=

+=

(

)(

)))( -1iiii-1ii af(H(xKafa +=

Estimation when System Model

is Nonlinear(Extended Kalman Filter)

-1i-1ii wafa += )( iiii vaHx +=State equation Measurement equation

Jacobian Matrix

Differences compared toregular Kalman filter are

circled in red


19/26

Predict next state as using

previous step and dynamic model

Predict regionsof next measurements using

measurement model anduncertainties

Make new measurements xi in

predicted regions

Compute best estimate of next state

Tracking Steps

(u, v)Measurement

Prediction

region

-1iia

),( i-1iii P'aH N

) -1iiiii-1iii aH(xKaa +=Correction of predicted state


20/26

Recursive Least Square Estimation for

a Dynamic System (Kalman Filter)

x

t

Measurementxi

State vectoraiEstimation ia


21/26

Tracking as a Probabilistic

Inference Problem

Find distributions for state vectorai and formeasurement vectorxi. Then we are able to

compute the expectations and

Simplifying assumptions (same as for HMM)

ix

)|(),,,|( -1ii-1i21i aaaaaa PP =L

)|()|()|()|,,,( ikijiiiji axaxaxaxx PPPP KK =

(Conditional independence ofmeasurements given a state)

(Only immediate past matters)

ia


22/26

Tracking as Inference

Prediction

Correction

Produces same results as least square approach if

distributions are Gaussians: Kalman filter See Forsyth and Ponce, Ch. 19

= -1i-1i1-1i-1ii-1i1i axxaaaxxa dPPP ),,|()|(),,|( LL

=

i-1i1iii

-1i1iiii1i

axxaax

xxaaxxxa

dPP

PPP

),,|()|(

),,|()|(),,|(

L

L

L


23/26

)( 11 i-iii-i afhxKafa +=

Kalman Filter for 1D Signals

11 i-i-i wafa +=

iii vahx+=

)0(~ ,qNwi

)0(~ ,rNvi

111

1

2

12

')1(

)(

i-ii-

ii

-iii

phKp

qpfp'

rp'hhp'K

=

+=

+=

State equation


Tweak factor for model

Measurement noise

Prediction for xi

Prediction for ai (a priori estimate)

Gain

Standard deviation for

prediction errorSt.d. for estimation error


24/26

Applications: Structure from Motion

Measurement vector components: Coordinates of corners, salient

points

State vector components:

Camera motion parameters

Scene structure

Is there enough equations?

N corners, 2N measurements

N unknown state components from structure

(distances from first center of projection to3D points)

6 unknown state components from motion

(translation and rotation)

More measurements than unknowns for every

frame if N>6 (2N > N + 6)

xi

Batch methods Recursive methods

(Kalman filter)


25/26

Problems with Tracking

Initial detection If it is too slow we will never catch up

If it is fast, why not do detection at every frame?

Even if raw detection can be done in real time, tracking

saves processing cycles compared to raw detection.

The CPU has other things to do.

Detection is needed again if you lose tracking

Most vision tracking prototypes use initial

detection done by hand(see Forsyth and Ponce for discussion)


26/26

References Kalman, R.E., A New Approach to Linear Prediction

Problems, Transactions of the ASME--Journal of BasicEngineering, pp. 35-45, March 1960.

Sorenson, H.W., Least Squares Estimation: from Gauss to

Kalman, IEEE Spectrum, vol. 7, pp. 63-68, July 1970. http://www.cs.unc.edu/~welch/kalmanLinks.html

D. Forsyth and J. Ponce. Computer Vision: A Modern

Approach, Chapter 19.http://www.cs.berkeley.edu/~daf/book3chaps.html

O. Faugeras.. Three-Dimensional Computer Vision. MIT

Press. Ch. 8, Tracking Tokens over Time.

Documents

Lecture25 Tracking