1

2

Homeworks

• Problem set 1 returned today. Mean: 85. Std dev: 15• Problem set 1 solutions posted (with password

protection) by Tuesday• Note revisions to problem set 2 matlab assignment

– To clarify the problems and make them work out cleaner for you.

– Modifying an existing solution to the revised version will be very simple.

3

Shape from shading

4

A shape-from-shading algorithm( ) ( ) ( ) ( )( )2

,1,2

,,12

,1,2

,,1, 41

jijijijijijijijiji qqqqpppps −+−+−+−= ++++

local surface smoothness( )( )2, ijijsijij qpREr −= local fidelity to image data

( )∑∑ +=i j

ijij rse λ global function to minimize

( ) ( )( )

( ) ( )( )qRqpREqq

qe

pRqpREpp

pe

sklklsklklkl

kl

sklklsklklkl

kl

∂∂

−−−=∂∂

∂∂

−−−=∂∂

,22

,22

λ

λ

derivative w.r.t. surface parameters

( )( )

( )( )qRqpREqq

pRqpREpp

sklklsklklkl

sklklsklklkl

∂∂

−+=

∂∂

−+=

,

,

λ

λ

set derivatives equal to zero.

Iterative algorithm

( )( )

( )( )qRqpREqq

pRqpREpp

snkl

nklskl

nkl

nkl

snkl

nklskl

nkl

nkl

∂∂

−+=

∂∂

−+=

+

+

,

,

1

1

λ

λ

5

• Show fig. 11-10, 11-11.

Horn, 1986

Results using this approach

6

2-d animation

Copyrighted … yada yada yada

7

3-d animation

8

Image filtering

• Reading: – Chapter 7, F&P

• Recommended Reading: – Chapter 7, 8 Horn

Oct. 3, 2002MIT 6.801/6.866Profs. Freeman and Darrell

9

Take 6.341, discrete-time signal processing

• If you want to process pixels, you need to understand signal processing well, so – Take 6.341

• Fantastic set of teachers:– Al Oppenheim – Greg Wornell– Jae Lim

• Web page: http://web.mit.edu/6.341/www/

10

What is image filtering?

• Modify the pixels in an image based on some function of a local neighborhood of the pixels.

5 141 71

5 310

Local image data

7

Modified image data

Some function

11

Linear functions• Simplest: linear filtering.

– Replace each pixel by a linear combination of its neighbors.

• The prescription for the linear combination is called the “convolution kernel”.

5 141 71

5 310

0.50.5 0010

0 00

Local image data kernel

7

Modified image data

12

Convolution

∑ −−=⊗=lk

lkglnkmIgInmf,

],[],[],[

13

Linear filtering (warm-up slide)

?0

Pixel offset

coef

ficie

nt1.0

original

14

Linear filtering (warm-up slide)

0Pixel offset

coef

ficie

nt1.0

original Filtered(no change)

15

Linear filtering

0Pixel offset

coef

ficie

nt

1.0

?original

16

shift

0Pixel offset

coef

ficie

nt

1.0

original shifted

17

Linear filtering

0Pixel offset

coef

ficie

nt0.3 ?

original

18

Blurring

0Pixel offset

coef

ficie

nt0.3

original Blurred (filterapplied in both dimensions).

19

Blur examples2.4

0Pixel offset

coef

ficie

nt

0.3

original

8

impulse

filtered

20

Blur examples2.4

0Pixel offset

coef

ficie

nt

0.3impulse

original

8

filtered

8

0Pixel offset

coef

ficie

nt

0.3

8

original filtered

4edge 4

21

Linear filtering (warm-up slide)

2.0

original

1.0

?0 0

22

Linear filtering (no change)

2.0

original

1.0

0 0

Filtered(no change)

23

Linear filtering

2.0

original

?0

0.33

0

24

(remember blurring)

0Pixel offset

coef

ficie

nt0.3

original Blurred (filterapplied in both dimensions).

25

Sharpening

2.0

original

0

0.33

0

Sharpened original

26

Sharpening example

coef

ficie

nt

-0.3

1.7

original

811.2

8

-0.25Sharpened

(differences areaccentuated; constant

areas are left untouched).

27

Sharpening

before after

28

Oriented filters

29

Gabor filters at differentscales and spatial frequencies

top row shows anti-symmetric (or odd) filters, bottom row thesymmetric (or even) filters.

30

Linear image transformations

• In analyzing images, it’s often useful to make a change of basis.

Fourier transform, orWavelet transform, or

Steerable pyramid transform

fUFrr

=transformed image

Vectorized image

31

Self-inverting transforms

FU

FUfr

r

+

−

=

=

1r

Same basis functions are used for the inverse transform

U transpose and complex conjugate

32

An example of such a transform: the Fourier transform

discrete domain

∑∑−

=

−

=

+−

=1

0

1

0

ln

],[],[M

k

N

l

NMkmi

elkfnmFπ

Forward transform

∑∑−

=

−

=

++

=1

0

1

0

ln

],[1],[M

k

N

l

NMkmi

enmFMN

lkfπ

Inverse transform

33

To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part ---as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction.

34

Here u and v are larger than in the previous slide.

35

And larger still...

36

Phase and Magnitude• Fourier transform of a real

function is complex– difficult to plot, visualize– instead, we can think of the

phase and magnitude of the transform

• Phase is the phase of the complex transform

• Magnitude is the magnitude of the complex transform

• Curious fact– all natural images have

about the same magnitude transform

– hence, phase seems to matter, but magnitude largely doesn’t

• Demonstration– Take two pictures, swap the

phase transforms, compute the inverse - what does the result look like?

37

38

This is the magnitude transform of the cheetah pic

39

This is the phase transform of the cheetah pic

40

41

This is the magnitude transform of the zebrapic

42

This is the phase transform of the zebrapic

43

Reconstruction with zebra phase, cheetah magnitude

44

Reconstruction with cheetah phase, zebra magnitude

45

Example image synthesis with fourier basis.

• 16 images

46

2

47

6

48

18

49

50

50

82

51

136

52

282

53

538

54

1088

55

2094

56

4052.4052

57

8056.

58

15366

59

28743

60

49190.

61

65536.

62

Fourier transform magnitude

63

Masking out the fundamental and harmonics from periodic pillars

64

Name as many functions as you can that retain that same

functional form in the transform domain

65Forsyth&Ponce

66

Discrete-time, continuous frequency Fourier transform

Oppenheim, Schafer and Buck,Discrete-time signal processing,Prentice Hall, 1999

67

Discrete-time, continuous frequency Fourier transform pairs

Oppenheim, Schafer and Buck,Discrete-time signal processing,Prentice Hall, 1999

68

Bracewell’s pictorial dictionary of Fourier transform pairs

Bracewell, The Fourier Transform and its Applications, McGraw Hill 1978

69

Why is the Fourier domain particularly useful?

• It tells us the effect of linear convolutions.

70

Fourier transform of convolution

Consider a (circular) convolution of g and h

hgf ⊗=

71

hgf ⊗=Fourier transform of convolution

Take DFT of both sides

( )hgDFTnmF ⊗=],[

72

hgf ⊗=( )hgDFTnmF ⊗=],[

Fourier transform of convolution

Write the DFT and convolution explicitly

[ ] ∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,

],[],[,M

u

N

v

Nvn

Mumi

lk

elkhlvkugnmFπ

73

hgf ⊗=( )hgDFTnmF ⊗=],[

Fourier transform of convolution

[ ] ∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[,

M

u

N

v

Nvn

Mumi

lkelkhlvkugnmF

π

Move the exponent in

∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,

],[],[M

u

N

v lk

Nvn

Mumi

lkhelvkugπ

74

hgf ⊗=( )hgDFTnmF ⊗=],[

Fourier transform of convolution

[ ] ∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[,

M

u

N

v

Nvn

Mumi

lkelkhlvkugnmF

π

∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[

M

u

N

v lk

Nvn

Mumi

lkhelvkugπ

Change variables in the sum( ) ( )

∑ ∑ ∑−−

−=

−−

−=

+

++

−=

1 1

,

],[],[kM

k

lN

l lk

Nnl

Mmki

lkhegµ υ

υµπυµ

75

hgf ⊗=( )hgDFTnmF ⊗=],[

Fourier transform of convolution

[ ] ∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[,

M

u

N

v

Nvn

Mumi

lkelkhlvkugnmF

π

∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[

M

u

N

v lk

Nvn

Mumi

lkhelvkugπ

( ) ( )

∑ ∑ ∑−−

−=

−−

−=

+

++

−=

1 1

,],[],[

kM

k

lN

l lk

Nnl

Mmki

lkhegµ υ

υµπυµ

Perform the DFT (circular boundary conditions)

[ ]∑

+−

=lk

NMkmi

lkhenmG,

ln

],[,π

76

hgf ⊗=( )hgDFTnmF ⊗=],[

Fourier transform of convolution

[ ] ∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[,

M

u

N

v

Nvn

Mumi

lkelkhlvkugnmF

π

∑∑∑−

=

−

=

+−

−−=1

0

1

0 ,],[],[

M

u

N

v lk

Nvn

Mumi

lkhelvkugπ

( ) ( )

∑ ∑ ∑−−

−=

−−

−=

+

++

−=

1 1

,],[],[

kM

k

lN

l lk

Nnl

Mmki

lkhegµ υ

υµπυµ

[ ]∑

+−

=lk

NMkmi

lkhenmG,

ln

],[,π

Perform the other DFT (circular boundary conditions)

[ ] [ ]nmHnmG ,,=

77

Analysis of our simple filters

78

Analysis of our simple filters

0Pixel offsetco

effic

ient

1.0

original Filtered(no change)

1

],[],[1

0

1

0

ln

=

= ∑∑−

=

−

=

+−M

k

N

l

NMkmi

elkfnmFπ

0

1.0 constant

79

Analysis of our simple filters

0Pixel offsetco

effic

ient

original

1.0

shifted

Mmi

M

k

N

l

NMkmi

e

elkfnmF

δπ

πδ

−

−

=

−

=

+−

=

−= ∑∑

],[],[1

0

1

0

ln

0

1.0

Constant magnitude, linearly shifted phase

80

Analysis of our simple filters

0Pixel offsetco

effic

ient

original

0.3

blurred

+=

= ∑∑−

=

−

=

+−

Mm

elkfnmFM

k

N

l

NMkmi

π

π

cos2131

],[],[1

0

1

0

ln

Low-pass filter

0

1.0

81

Analysis of our simple filters2.0

original0

0.33

0sharpened

+−=

= ∑∑−

=

−

=

+−

Mm

elkfnmFM

k

N

l

NMkmi

π

π

cos21312

],[],[1

0

1

0

ln

high-pass filter

0

1.0

2.3

82

Sampling and aliasing

83

Sampling in 1D takes a continuous function and replaces it with a vector of values, consisting of the function’s values at a set of sample points. We’ll assume that these sample points are on a regular grid, and can place one at each integer for convenience.

84

Sampling in 2D does the same thing, only in 2D. We’ll assume that these sample points are on a regular grid, and can place one at each integer point for convenience.

85

A continuous model for a sampled function

• We want to be able to approximate integrals sensibly

• Leads to– the delta function– model on right

Sample2D f (x,y)( )= f (x, y)δ (x − i, y − j)i=−∞

∞

∑i=−∞

∞

∑

= f (x,y) δ (x − i, y − j)i=−∞

∞

∑i=−∞

∞

∑

86

The Fourier transform of a sampled signal

F Sample2D f (x,y)( )( )= F f (x, y) δ(x − i,y − j)i=−∞

∞

∑i=−∞

∞

∑

= F f (x,y)( )**F δ(x − i, y − j)i=−∞

∞

∑i=−∞

∞

∑

= F u − i,v − j( )j=−∞

∞

∑i=−∞

∞

∑

87

88

89

Aliasing

• Can’t shrink an image by taking every second pixel

• If we do, characteristic errors appear – In the next few slides– Typically, small phenomena look bigger; fast

phenomena can look slower– Common phenomenon

• Wagon wheels rolling the wrong way in movies• Checkerboards misrepresented in ray tracing• Striped shirts look funny on colour television

90

Resample the checkerboard by taking one sample at each circle. In the case of the top left board, new representation is reasonable. Top right also yields a reasonable representation. Bottom left is all black (dubious) and bottom right has checks that are too big.

91

Constructing a pyramid by taking every second pixel leads to layers that badly misrepresent the top layer

92

Smoothing as low-pass filtering

• A filter whose FT is a box is bad, because the filter kernel has infinite support

• Common solution: use a Gaussian– multiplying FT by

Gaussian is equivalent to convolving image with Gaussian.

• The message of the FT is that high frequencies lead to trouble with sampling.

• Solution: suppress high frequencies before sampling– multiply the FT of the

signal with something that suppresses high frequencies

– or convolve with a low-pass filter

93

Sampling without smoothing. Top row shows the images,sampled at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

94

Sampling with smoothing. Top row shows the images. Weget the next image by smoothing the image with a Gaussian with sigma 1 pixel,then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

95

Sampling with smoothing. Top row shows the images. Weget the next image by smoothing the image with a Gaussian with sigma 1.4 pixels,then sampling at every second pixel to get the next; bottom row shows the magnitude spectrum of these images.

96

Thought problemAnalyze

crossed gratings

97

What is a good representation for image analysis?

• Fourier transform domain tells you “what” (textural properties), but not “where”.

• Pixel domain representation tells you “where” (pixel location), but not “what”.

• Want an image representation that gives you a local description of image events—what is happening where.

98

Image pyramids

99

The Gaussian pyramid

• Smooth with gaussians, because– a gaussian*gaussian=another gaussian

• Synthesis – smooth and sample

• Analysis– take the top image

• Gaussians are low pass filters, so repn is redundant

100

101

The Laplacian Pyramid

• Synthesis– preserve difference between upsampled

Gaussian pyramid level and Gaussian pyramid level

– band pass filter - each level represents spatial frequencies (largely) unrepresented at other levels

• Analysis– reconstruct Gaussian pyramid, take top layer

102

103

104

105

Oriented pyramids

• Laplacian pyramid is orientation independent

• Apply an oriented filter to determine orientations at each layer– by clever filter design, we can simplify

synthesis– this represents image information at a particular

scale and orientation

106

Reprinted from “Shiftable MultiScale Transforms,” by Simoncelli et al., IEEE Transactionson Information Theory, 1992, copyright 1992, IEEE