Motion Estimation using Markov Random Fields

Motion Estimation using Markov Random Fields

Hrvoje Bogunović

Image Processing Group

Faculty of Electrical Engineering and Computing

University of Zagreb

Summer School on Image Processing, Graz 2004

Overview

• Introduction

• Optical flow

• Markov Random Fields

• OF+MRF combined

• Energy minimization techniques

• Results

Introduction

• Input:– Sequence of images (Video)

• Problem– Extract information about motion

• Applications– Detection, Segmentation, Tracking, Coding

Spatio-temporal spectrum

φ

f

Motion – aliasing

φ

f

1/x

1/t

Large area flicker

Loss of spatialresolution

Large motions - temporal aliasing

φ

fTemporal aliasing

Great loss of spatial resolution

Temporal anti-aliasing

φ

f

• No more overlaping on the f axis. • filtering (anit-aliasing) is performed after sampling, hence the blurring

Motion – eye tracking

φ

f

Motion estimation

• Images are 2-D projections of the 3-D world.

• Problem is represented as a labeling one.– Assign vector to pixel

• Vector field field of movement• Low level vision

– No interpretation

Example Ideal

Problems

• Problem is inherently ill-posed– Solution is not unique

• Aperture problem– Specific to local methods

Optical flow

• Main assumption: Intensity of the object does not change as it moves– Often violated

• First solved by Horn & Schunk– Gradient approach

• Other approaches include– Frequency based– Using corresponding features

Image differencing

Gradient approach

• Local by nature. Aperture problem is significant.

• Image understanding is not required– Very low level

Horn & Schunk

• Intensity stays the same in the direction of movement. I(x,y,t)

• After derivation

Horn & Schunk

• Spatial gradients Ix,Iy

– e.g. Sobel operator

• Temporal gradient It

– Image subtraction

( , ) ( , ) 0x y t

t

I I u v I

I I v

Regularization

• Tikhonov regularization for ill-posed problems

• Add the smoothness term

• Energy function

Result

Problems of the H-S method

• Assumption: There are no discontinuities in the image– Optical flow is over-smoothed.

• Gradient method. Only the edges which are perpendicular to motion vector contribute

• Image regions which are uniform do not contribute.

• Difficulty with large motions (spatial filtering)

Optical flow enhancement

• Optical flow can be piecewise smooth

• Discontinuities can be incorporated

• Solution: use the spatial context

• Problem is posed as a solution of the Bayes classifier. Solution in optimization sense. Search for optimum

Bayes classifier

• Main equation

• Solution using MAP estimation

( , ) ( )( | )

( )

P hypothesis observation P hypothesisP hypothesis observation

P observation

Markov Random Fields

• Suitable: Problems posed as a visual labeling problemn with contextual constraints

• Useful to encode a priori knowledge– required for bayes classifier (smoothness prior)– equvalence to Gibbs random fields (gibbs

distribution, exponential like)• Neighbourhoods• Cliques

– pairs,triples of neighbourhood points)– build the energy function

MRF

• Define sites: rectangular lattice

• Define labels

• define neighbourhood: 4,8 point

• Field is MRF:– P(f)>0

– P(fi|f{S-i})=P(fi|Ni)

Coupled MRF

• Field F is an optical flow field• Field L is a field of discontinuities

– line process

• Position of the two fields.

Context

• neighbourhoods and cliques

Motion estimation equations

Energy for MAP estimation

Parameters are estimated ad hoc

Energy minimization

• Global minimum– Simulated annealing– Genetic Algorithms

• Local minimum– Iterated Conditional Modes (ICM) (steepest

decent)– Highest Confidence First (HCF)

• specific site visiting

Simulated annealing(1) Find the initial temperature of the system T.

(2) Assign initial values of the field to random

(3) For every pixel:

Assign random value to f(i,j)

Calculate the difference in energy before and after If the change is better (diff>0) keep it.

Else keep it with the probability exp(diff/T)

(4) Repeat (3) N1 times

(5) T = f(T) where f decreases monotono

(6) Repeat (3-5) N2 times

Results (Square)

Horn-Schunk OF OF+MRF

Taxi

Results (Taxi)

Line process result (Taxi)

Cube

Results (cube)

Line process result (cube)

Q & A