3 math overview - NYU Computer ScienceLecture 3 Overview Projects ... 3 1 2 [ ] [ ] 1 0 high-pass filter 0 1.0 2.3 Slide credit: Bill Freeman Convolution versus FFT • 1-d FFT: O(NlogN)

2/12/2008

1

Math review

Lecture 3

Overview

Projects• Removing JPEG artifcats: “The Duck Caper”

• Hot or Not

Projects

Projects

• David Hockney’s Joiners

Projects

• 2‐D Panorama creation

2/12/2008

2

Projects

• Image Analogies

A A' B B'

Projects

• Seam carving

Projects

• Spectralresponsecurveestimtationestimtation

Overview of today

• Linear Algebra review

• Least Squares Optimization

• Linear Systems

• Fourier domain

Whiteboard PCA ‐ Faces

http://www.cs.princeton.edu/~cdecoro/eigenfaces/

2/12/2008

3

PCA ‐ Faces


PCA ‐ Faces


Overview of today

• Linear Algebra review

• Least Squares Optimization

• Linear Systems

• Fourier domain

Whiteboard

Convolution

∑ −−=⊗=lk

lkglnkmIgInmf,

],[],[],[

I

Slide credit: Bill Freeman

Linear filtering (warm-up slide)

cien

t

1 0 ?original

0Pixel offset

coef

fic 1.0 ?


2/12/2008

4


cien

t

1 0

original0

Pixel offset

coef

fic 1.0

Filtered(no change)


Linear filtering

ffici

ent

1.0

?0

Pixel offset

coef

f

original

?


shift

ffici

ent

1.0

0Pixel offset

coef

f

original shifted


Linear filtering

ffici

ent

?0

Pixel offset

coef

f

original

0.3 ?


Blurring

ffici

ent

0Pixel offset

coef

f

original

0.3

Blurred (filterapplied in both dimensions).


Blur examples

0Pixel offset

coef

ficie

nt

0.3

original

8

filtered

2.4

impulse

Pixel offsetoriginal filtered


2/12/2008

5

Blur examples

impulse

0Pixel offset

coef

ficie

nt0.3

original

8

filtered

2.4

0Pixel offset

coef

ficie

nt

0.3

original

8

filtered

48

4edge

Pixel offsetoriginal filtered



2.0

?1.0

original

0

?0


Linear filtering (no change)

2.0

1.0

original

0 0

Filtered(no change)


Linear filtering

2.0

0 33 ?original

0 0

0.33 ?


(remember blurring)

ffici

ent

0Pixel offset

coef

f

original

0.3

Blurred (filterapplied in both dimensions).


Sharpening

2.0

0 33

original

0 0

0.33

Sharpened original


2/12/2008

6

Sharpening example

cien

t8

11.21.7

8co

effi

-0.3original Sharpened

(differences areaccentuated; constant

areas are left untouched).

-0.25


Sharpening

before after


Spatial resolution and color

R

G

Boriginal


Blurring the G component

R

G

B

original processed


Blurring the R component

R

original processed

G

B


Blurring the B component

R

original

G

Bprocessed


2/12/2008

7

From W. E. Glenn, in Digital I dImages and Human Vision, MIT Press, edited by Watson, 1993


Lab color components

L

a

A rotation of the color coordinates into directions that are morea

b

are more perceptually meaningful: L: luminance, a: red-green, b: blue-yellow


Blurring the L Lab component

L

aa

boriginal processed


Blurring the a Lab component

L

a

original

a

bprocessed


Blurring the b Lab component

L

a

original

a

bprocessed


Overview of today

• Linear Algebra review• Least Squares Optimization• Linear Systems• Fourier domain

2/12/2008

8

Linear image transformations

• In analyzing images, it’s often useful to make a change of basis.

transformed image

Fourier transform, orWavelet transform, or

Steerable pyramid transform

fUFrr

= Vectorized image

transformed image


Self-inverting transforms

FUfrr

−= 1

Same basis functions are used for the inverse transform

FU

FUfr

+=

=

U transpose and complex conjugate


An example of such a transform: the Fourier transform

discrete domain

∑∑−

=

−

=

⎟⎠⎞

⎜⎝⎛ +−

=1

0

1

0

ln

],[],[M

k

N

l

NMkmi

elkfnmFπ

Forward transform

∑∑−

=

−

=

⎟⎠⎞

⎜⎝⎛ ++

=1

0

1

0

ln

],[1],[M

k

N

l

NMkmi

enmFMN

lkfπ

Inverse transform


To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part ---as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The gfunction is a sinusoid with this frequency along the direction, and constant perpendicular to the direction.

u

v( )vyuxie +−π

( )vyuxie +π


Here u and v are larger than in the previous slide.

u

v( )vyuxie +−π

( )vyuxie +π


And larger still...

u

v( )vyuxie +−π

( )vyuxie +π


2/12/2008

9

Phase and Magnitude

• Fourier transform of a real function is complex– difficult to plot, visualize– instead, we can think of the

phase and magnitude of the transform

• Curious fact– all natural images have

about the same magnitude transform

– hence, phase seems to matter, but magnitude the transform

• Phase is the phase of the complex transform

• Magnitude is the magnitude of the complex transform

, glargely doesn’t

• Demonstration– Take two pictures, swap

the phase transforms, compute the inverse - what does the result look like?

Slide credit: Bill Freeman Slide credit: Bill Freeman

This is the magnitude transform of the cheetah pic


This is the phase transform of the cheetah pic



This is the magnitude transform of the zebra pic


2/12/2008

10

This is the phase transform of the zebra pic


Reconstruction with zebra phase, cheetah magnitude


Reconstruction with cheetah phase, zebra magnitude


Example image synthesis with Fourier basis.

• Following are 16 images showing the reconstruction of an image from a random selection of Fourier basis functions.

• Note, the selection of basis functions to include was not made according to basis magnitude. Doing that would have given us an approximate version of the image much sooner.


2


6


2/12/2008

11

18


50


82


136


282


538


2/12/2008

12

1088


2094


4052.

4052


8056.


15366


28743


2/12/2008

13

49190.


65536.


Fourier transform magnitude


Masking out the fundamental and harmonics from periodic pillars


Forsyth&Ponce

Discrete-time, continuous frequency Fourier transform

Oppenheim, Schafer and Buck,Discrete-time signal processing,Prentice Hall, 1999

2/12/2008

14

Why is the Fourier domain particularly useful?

• It tells us the effect of linear convolutions.• There is a fast algorithm for performing the

DFT, allowing for efficient signal filtering.Th F i d i ff lt ti• The Fourier domain offers an alternative domain for understanding and manipulating the image.


Analysis of our simple filters

original0Pixel offset

coef

ficie

nt

1.0

Filtered(no change)(no change)

1

][][1

0

=

= ∑−

=

⎟⎠⎞

⎜⎝⎛−M

k

Mkmi

ekfmFπ

0

1.0 constant



0Pixel offset

coef

ficie

nt

original

1.0

shifted

δ

Mmi

M

k

Mkmi

e

ekfmF

δπ

π

−

−

=

⎟⎠⎞

⎜⎝⎛−

=

= ∑

][][1

0

0

1.0

Constant magnitude, linearly shifted phase



0Pixel offsetco

effic

ient

original

0.3

blurred

⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎠⎞

⎜⎝⎛+=

= ∑−

=

⎟⎠⎞

⎜⎝⎛−

Mm

ekfmFM

k

Mkmi

π

π

cos2131

][][1

0 Low-pass filter

0

1.0



original0

2.0

0

0.33

sharpened

⎟⎟⎠

⎞⎜⎜⎝

⎛⎟⎠⎞

⎜⎝⎛+−=

= ∑−

=

⎟⎠⎞

⎜⎝⎛−

Mm

ekfmFM

k

Mkmi

π

π

cos21312

][][1

0 high-pass filter

0

1.0

2.3


Convolution versus FFT

• 1-d FFT: O(NlogN) computation time, where N is number of samples.

• 2-d FFT: 2N(NlogN), where N is number of pixels on a sidepixels on a side

• Convolution: K N2, where K is number of samples in kernel

• Say N=210, K=100. 2-d FFT: 20 220, while convolution gives 100 220


2/12/2008

15

Sampling and Reconstruction Sampling and Reconstruction

• Simple example: a sign wave

© 2006 Steve Marschner • 86

Undersampling

• What if we “missed” things between the samples?

• Simple example: undersampling a sine wavei i lt i f ti i l t


– unsurprising result: information is lost

Undersampling


• Simple example: undersampling a sine wavei i lt i f ti i l t


– unsurprising result: information is lost

– surprising result: indistinguishable from lower frequency

Undersampling


• Simple example: undersampling a sine wave– unsurprising result: information is lost


– surprising result: indistinguishable from lower frequency

– also was always indistinguishable from higher frequencies

– aliasing: signals “traveling in disguise” as other frequencies

Aliasing in images

2/12/2008

16

The Fourier transform of a sampled signal

F Sample2D f (x,y)( )( )= F f (x, y) δ(x − i,y − j)i=−∞

∞

∑i=−∞

∞

∑⎛ ⎝ ⎜

⎞ ⎠ ⎟

= F f (x,y)( )**F δ(x − i, y − j)i=−∞

∞

∑i=−∞

∞

∑⎛ ⎝ ⎜

⎞ ⎠ ⎟

= F u − i,v − j( )j=−∞

∞

∑i=−∞

∞

∑


Image half-sizing

This image is too big tofit on the screen. Howcan we reduce it?

How to generate a halfHow to generate a half-sized version?

Image sub-sampling

Throw away every other row and column to create a 1/2 size image

- called image sub-sampling

1/4

1/8

Slide by Steve Seitz

Image sub-sampling

1/4 (2x zoom) 1/8 (4x zoom)

Aliasing! What do we do?

1/2


2/12/2008

17

Gaussian (lowpass) pre-filtering

G 1/4

G 1/8

Gaussian 1/2

Solution: filter the image, then subsample• Filter size should double for each ½ size reduction. Why?


Subsampling with Gaussian pre-filtering

G 1/4 G 1/8Gaussian 1/2


Compare with...

1/4 (2x zoom) 1/8 (4x zoom)1/2


Documents

3 math overview - NYU Computer ScienceLecture 3 Overview Projects ... 3 1 2 [ ] [ ] 1 0 high-pass filter 0 1.0 2.3 Slide credit: Bill Freeman Convolution versus FFT • 1-d FFT: O(NlogN)