Image Alignment byImage Averaging

David Hong

NCSSM, IE364 2008

Example Problem 1

In:

Out: 4 0 0s s sX Y

Example Problem 2

In:

Out: 0 4 0s s sX Y

Example Problem 3

In:

Out: 0 0 6s s sX Y

Problem (Formal Statement)

In: , n nu v R

Out: , , such that

translated by and and

rotated by is

s s s

s s

s

X Y

v X Y

u

Motivation

Many Applications:– Special Effects (Movie)– Video Compression– Pattern Recognition– Image Stabilization (Digital Cameras)– Dead-reckoning (Mobile Robotics)

State of the Art

Optical Flow

Lucas-Kanade (1985)

Optical Flow with Smoothness Constraint

Horn-Schunck (1980)

Phase Correlation

X

Y

u(x,y,t)

U(X,Y)

xy

Lucas-Kanade

Lucas-Kanade

0 0, , ,U X Y u x y t

Then the floor-coordinate is (X0,Y0) and the sensor-coordinate is (x,y) at time t.

Let us consider a point on the plane.

From there, we can see:

Lucas-Kanade

0 0, , ,

0

U X Y u x y t

d dU u

dt dtdx

u u udtdyx y t

dt

Differentiating on time gives us:

Lucas-Kanade

0

0

cos sin

sin coss s s

s s s

X Xx

Y Yy

Expressing (x,y) in terms of (X0,Y0) and the sensor position (Xs,Ys,Θs) gives us:

Lucas-Kanade

s

s

s

dX

dtdYu u u u u

x yx y y x dt t

d

dt

Putting the two together, we get:

This is underdetermined!

Algorithm

1, 1, , 1 , 1 , 1 , 1 1, 1, , ,2 's

i j i j i j i j i j i j i j i j s i j i j

s

X

u u u u u u i u u j Y u u

uu’

Improvement by Iteration

u

u’

Iteration 1:

3

2

0

s

s

s

X

Y

u’’

Iteration 2:

1

1

0

s

s

s

X

Y

Overall Estimate:

4

1

0

s

s

s

X

Y

u’’’

Improvement by Iteration

u’

u

u’’

u’(x’,y’)

u’’(x’’,y’’)

Improvement by Iteration

Coordinate Transformation:

cos sin' ''

sin cos' ''ss s

s s s

Xx x

y y Y

Improvement by Iteration

u’

Places u’ is defined

Place we need to evaluate u’

(x’, y’)

(x’0, y’0) (x’1, y’0)

(x’0, y’1) (x’1, y’1)

Improvement by Iteration

1 10 0

0 1 0 1

1 00 1

0 1 1 0

0 11 0

1 0 0 1

0 0

1

Lagrange Interpolation:

' ' ' ''( ', ') ' ' , '

' ' ' '

' ' ' ' ' ' , '

' ' ' '

' ' ' ' ' ' , '

' ' ' '

' ' ' '

'

x x y yu x y u x y

x x y y

x x y yu x y

x x y y

x x y yu x y

x x y y

x x y y

x x

1 10 1 0

' ' , '' ' '

u x yy y

Improvement by Iteration

u’

u

u’’

u’ not defined!

Improvement by Iteration

u’

u’’

u’’ was not evaluated here

Valid Region

Improvement by Iteration

Places we need to evaluate u’

(i-1,0’, j-1,0’ )

(i0,-1’, j0,-1’ )

(i0,1’, j0,1’ )

(i1,0’, j1,0’ )

Performance of Algorithm

Good Surface: Bad Surface:

Algorithm Fails!

Performance of Algorithm

Surface:

Performance of AlgorithmSurface:

Performance of AlgorithmSurface:

Assumptions Made

• The Error Function is locally quadratic

• The floor is linear

Weaknesses

• Many Iterations– Inherent to Technique

• “Fooled” by symmetry (Aliasing problem)– Inherent to Problem

Strengths

• Accurate

• Improvement by Iteration

• Finds Error Function Root by Newton’s

yu

v(xv,yv)

u(xu,yu)

x v

y v

Phase Correlation

xu

Phase Correlation

We consider the image to be like a 2-D wave.

Then, displacement is simply a “phase shift”

Rotation can similarly be found

So, we “correlate” the “phases”

Phase Correlation

Discrete Fourier Transform:

1

1

( )

ijij

ijij

ij ij

Fn

Gn

FG

Phase CorrelationKey Theorem:

,where is the motion by displacement and

is the desired output.

d d

d

d

F W F

d

W

' d dFz F z W Fz

Phase Correlation

1

2

Key Theorem:

' : such that '

:

( ')( ) ( )( )

m

i vd

Let z z R x d z x

Then

F z v e F z Rv

R R

Phase Correlation

Corollary:

( ) ( )

'( ) ' ( )

:

'

Let u v F z v

Let u v F z v

Then

u v u Rv

Weaknesses

• Inaccurate on first iteration

• Boundary Problem (Repetion Assumption)

• High complexity– FFT is “O(nlogn)”

Strengths

• Elegant

• Makes a big leap

• Works well on images with pattern

• Separates displacement and rotation (DFT)

yu

v(xv,yv)

u(xu,yu)

x v

y v

Image Averaging

xu

Image Averaging

• Find a “Center-of-Mass” of each image

• Track the motion of the center-of-mass

w x xx

w x

Weaknesses

• Boundary Problem (Average Point Moves)

• Average is affected by small discretization issues

Strengths

• Elegant• Makes a big leap• Very fast– Complexity of O(n)

• Yields itself well to Improvement by Iteration– Using same technique as in Lucas-Kanade

Handling the Weaknesses

• Here we decide to take an alternative approach

• Separate displacements from rotation

• Do this using FFT (as in Phase Correlation)

Handling the Weaknesses

• We handle rotation first– Post-FFT, only rotation remains

Handling the Weaknesses

Weaknesses

• We introduce an FFT ( O(nlogn) operation)

• However, only requires 2– Phase correlation requires up to 3 or 4

Future Work

• Make Image Alignment Rigorous– Use complex numbers to notate displacement

• Smoothness Constraint• Pre-processing the image• Condition for Convergence

Thank You!