CPSC 425: Computer Vision

Lecture 17: Optical Flow (cont.)

CPSC 425: Computer Vision

!1

Menu for Today (March 11, 2020)Topics:

— Optical Flow (cont) — Classification

Redings: — Today’s Lecture: Forsyth & Ponce (2nd ed.) 15.1, 15.2

— Next Lecture: Forsyth & Ponce (2nd ed.) 16.1.3, 16.1.4, 16.1.9

Reminders: — Assignment 4: Local Invariant Features and RANSAC due Tuesday — Midterm graded. Grades will be released soon.

— Naive Bayes Classifier — Bayes' Risk

!3

Today’s “fun” Example: Visual Microphone

Follow-up work to previous lecture’s example of Eulerian video magnification

!3

Today’s “fun” Example: Visual Microphone

Follow-up work to previous lecture’s example of Eulerian video magnification

!4

Lecture 16: Re-cap

Optical flow is the apparent motion of brightness patterns in the image

Applications — image and video stabilization in digital cameras, camcorders — motion-compensated video compression schemes such as MPEG — image registration for medical imaging, remote sensing — action recognition — motion segmentation

!5Figure credit: M. Srinivasan

Lecture 16: Re-cap

Aperture Problem

!6

In which direction is the line moving?

Image Credit: Ioannis (Yannis) Gkioulekas (CMU)

Aperture Problem

!7

In which direction is the line moving?


Aperture Problem

!8 Image Credit: Ioannis (Yannis) Gkioulekas (CMU)

Aperture Problem


Aperture Problem


Aperture Problem


Aperture Problem

— Without distinct features to track, the true visual motion is ambiguous

— Locally, one can compute only the component of the visual motion in the direction perpendicular to the contour

!12

Aperture Problem

— Without distinct features to track, the true visual motion is ambiguous

— Locally, one can compute only the component of the visual motion in the direction perpendicular to the contour

!13

Visual Motion

Visual motion is determined when there are distinct features to track, provided: — the features can be detected and localized accurately; and — the features can be correctly matched over time

!14

Motion as Matching

!15

Consider image intensity also to be a function of time, . We write

Optical Flow Constraint Equation

!16

I(x, y, t)

t


Applying the chain rule for differentiation, we obtain

where subscripts denote partial differentiation


!17

I(x, y, t)

t

dI(x, y, t)

dt

= I

x

dx

dt

+ I

y

dy

dt

+ I

t




Define . and . Then is the 2-D motion and the space of all

such and is the 2-D velocity space


!18

I(x, y, t)

t

dI(x, y, t)

dt

= I

x

dx

dt

+ I

y

dy

dt

+ I

t

v =dy

dtu =

dx

dt

[u, v]

u v






Suppose . Then we obtain the (classic) optical flow constraint equation


!19

I(x, y, t)

t

dI(x, y, t)

dt

= I

x

dx

dt

+ I

y

dy

dt

+ I

t

v =dy

dtu =

dx

dt

dI(x, y, t)

dt

= 0

Ix

u+ Iy

v + It

= 0

[u, v]

u v








!20

I(x, y, t)

t

dI(x, y, t)

dt

= I

x

dx

dt

+ I

y

dy

dt

+ I

t

v =dy

dtu =

dx

dt

dI(x, y, t)

dt

= 0

Ix

u+ Iy

v + It

= 0

[u, v]

u v








!21

dI(x, y, t)

dt

= 0

Ix

u+ Iy

v + It

= 0

What does this mean, and why is it reasonable?








!22

dI(x, y, t)

dt

= 0

Ix

u+ Iy

v + It

= 0


Scene point moving through image sequence









!23

dI(x, y, t)

dt

= 0

Ix

u+ Iy

v + It

= 0


Scene point moving through image sequence









!24

dI(x, y, t)

dt

= 0

Ix

u+ Iy

v + It

= 0


constant

Brightness Constancy Assumption: Brightness of the point remains the same


Aside: Derivation of Optical Flow Constraint

!25

For small space-time step, brightness of a point is the same

Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)

),( yx

),( tvytux δδ ++

ttime tttime δ+

),( yx


!26

For small space-time step, brightness of a point is the same

Insight: If the time step is really small,

we can linearize the intensity function



!27

Multivariable Taylor Series Expansion (First order approximation, two variables)



!28


assuming small motion



!29


assuming small motionfixed point

partial derivative

cancel terms



!30



cancel terms



!31



divide by take limit



!32



divide by take limit

Brightness Constancy Equation


How do we compute …

!33 Slide Credit: Ioannis (Yannis) Gkioulekas (CMU)


!34

spatial derivative



!35

spatial derivative

Forward difference Sobel filter Scharr filter

…



!36

spatial derivative


…

temporal derivative



!37

spatial derivative


…

temporal derivative

Frame differencing


Frame Differencing: Example

!38

1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10

1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10

0 0 0 0 00 0 0 0 00 -9 -9 -9 -90 -9 0 0 00 -9 0 0 00 -9 0 0 0

- =

(example of a forward temporal difference)


!39

1 1 1 1 11 1 1 1 11 10 10 10 101 10 10 10 101 10 10 10 101 10 10 10 10

1 1 1 1 11 1 1 1 11 1 1 1 11 1 10 10 101 1 10 10 101 1 10 10 10

- 0 0 0 -- 0 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -- 9 0 0 -

- - - - -0 0 0 0 00 9 9 9 90 0 0 0 00 0 0 0 0- - - - -

0 0 0 0 00 0 0 0 00 -9 -9 -9 -90 -9 0 0 00 -9 0 0 00 -9 0 0 0

y

x

-1 0 1

-1 0 1

y

x

y

x

y

x

y

x



!40

spatial derivative optical flow


…

temporal derivative

Frame differencingHow do you compute this?



!41



…

temporal derivative

Frame differencingWe need to solve for this! (this is the unknown in the

optical flow problem)



!42



…

temporal derivative

Frame differencingSolution lies on a line

Cannot be found uniquely with a single constraint


!43

Equation determines a straight line in velocity space

many combinations of u and v will satisfy the equality



Lucas-Kanade

Observations: 1. The 2-D motion, , at a given point, , has two degrees-of-freedom 2. The partial derivatives, , provide one constraint 3. The 2-D motion, , cannot be determined locally from alone

!44

[u, v] [x, y]

Ix

, Iy

, It

[u, v] Ix

, Iy

, It

Lucas-Kanade


Lucas–Kanade Idea: Obtain additional local constraint by computing the partial derivatives, , in a window centered at the given

!45

[u, v] [x, y]

Ix

, Iy

, It

[u, v] Ix

, Iy

, It

Ix

, Iy

, It

[x, y]

Lucas-Kanade


Lucas–Kanade Idea: Obtain additional local constraint by computing the partial derivatives, , in a window centered at the given

Constant Flow Assumption: nearby pixels will likely have same optical flow

!46

[u, v] [x, y]

Ix

, Iy

, It

[u, v] Ix

, Iy

, It

Ix

, Iy

, It

[x, y]

Suppose is the (original) center point in the window. Let be any other point in the window. This gives us two equations that we can write

and that can be solved locally for and as

provided that and are the same in both equations and provided that the required matrix inverse exists.

!47

[x1, y1] = [x, y] [x2, y2]

x

I

x1u+ I

y1v = �I

t1

I

x2u+ I

y2v = �I

t2

u v

uv

�= �

Ix1 I

y1

Ix2 I

y2

��1 It1

It2

�

u v

Lucas-Kanade Ix

u+ Iy

v + It

= 0Optical Flow Constraint Equation:

Considering all n points in the window, one obtains

which can be written as the matrix equation

where , and

!48

Ix1u+ I

y1v = �It1

Ix2u+ I

y2v = �It2

...Ixnu+ I

ynv = �Itn

A =

2

6664

Ix1 I

y1

Ix2 I

y2

......

Ixn I

yn

3

7775

b = �

2

6664

It1

It2

...Itn

3

7775

Av = b

v = [u, v]T

Lucas-Kanade

A =

2

6664

Ix1 I

y1

Ix2 I

y2

......

Ixn I

yn

3

7775

b = �

2

6664

It1

It2

...Itn

3

7775

Ix

u+ Iy

v + It

= 0Optical Flow Constraint Equation:

The standard least squares solution, , to is

again provided that and are the same in all equations and provided that the rank of is 2 (so that the required inverse exists)

!49

v̄ = (ATA)�1ATb

ATA

v̄ = (ATA)�1ATb

u v

Lucas-Kanade

The standard least squares solution, , to is

again provided that and are the same in all equations and provided that the rank of is 2 (so that the required inverse exists)

!50

v̄ = (ATA)�1ATb

ATA

v̄ = (ATA)�1ATb

u v

Lucas-Kanade

A =

2

6664

Ix1 I

y1

Ix2 I

y2

......

Ixn I

yn

3

7775

b = �

2

6664

It1

It2

...Itn

3

7775

A=

2 6 6 6 4

I x1

I y1

I x2

I y2

. . .. . .

I xn

I yn

3 7 7 7 5

b=

�

2 6 6 6 4

I t1

I t2 . . . I tn

3 7 7 7 5

Note that we can explicitly write down an expression for as

which is identical to the matrix that we saw in the context of Harris corner detection

!51

ATA =

PI2x

PIx

IyP

Ix

Iy

I2y

�

ATA

C

Lucas-Kanade

Note that we can explicitly write down an expression for as

which is identical to the matrix that we saw in the context of Harris corner detection

!52

ATA =

PI2x

PIx

IyP

Ix

Iy

I2y

�

ATA

C

Lucas-Kanade

What does that mean?

A dense method to compute motion, , at every location in an image

Key Assumptions:

1. Motion is slow enough and smooth enough that differential methods apply (i.e., that the partial derivatives, , are well-defined)

2. The optical flow constraint equation holds (i.e., )

3. A window size is chosen so that motion, , is constant in the window

4. A window size is chosen so that the rank of is 2 for the window

!53

Lucas-Kanade Summary

[u, v]

Ix

, Iy

, It

[u, v]

ATA

dI(x, y, t)

dt

= 0

Aside: Optical Flow Smoothness Constraint

Many methods trade off a ‘departure from the optical flow constraint’ cost with a ‘departure from smoothness’ cost.

The optimization objective to minimize becomes

where is a weighing parameter.

!54

E =

Z Z(I

x

u+ Iy

v + It

)2 + �(||5 u||2 + ||5 v||2)

�

!55

smoothness brightness constancy

weight

Horn-Schunck Optical Flow


Horn-Schunck Optical Flow

!56

Brightness constancy

Smoothness


SummaryMotion, like binocular stereo, can be formulated as a matching problem. That is, given a scene point located at in an image acquired at time , what is its position, , in an image acquired at time ?

Assuming image intensity does not change as a consequence of motion, we obtain the (classic) optical flow constraint equation

where , is the 2-D motion at a given point, , and are the partial derivatives of intensity with respect to , , and

Lucas–Kanade is a dense method to compute the motion, , at every location in an image

!57

(x0, y0) t0(x1, y1) t1

Ix

u+ Iy

v + It

= 0

[u, v] [x, y] Ix

, Iy

, It

x y t

[u, v]

Documents

CPSC 425: Computer Vision