convex

concave

convex

concave

EigenfacesPhotobook/Eigenfaces (MIT Media Lab)

Database

7562 pictures of 3000 people

Photobook/Eigenfaces (MIT Media Lab)

Query ExamplePhotobook/Eigenfaces (MIT Media Lab)

EigenfeaturesPhotobook/Eigenfaces (MIT Media Lab)

Photobook/Eigenfaces (MIT Media Lab)

Eigenfeatures

Photobook/Eigenfaces (MIT Media Lab)

Eigenfeatures

Receiver Operating Characteristic (ROC) Curve

Photobook/Eigenfaces (MIT Media Lab)

Eigenfeatures

Recognition with PCAAmano, Hiura, Yamaguti, and Inokuchi; Atick and Redlich; Bakry, Abo-Elsoud, and Kamel; Belhumeur, Hespanha, and Kriegman; Bhatnagar, Shaw, and Williams; Black and Jepson; Brennan and Principe; Campbell and Flynn; Casasent, Sipe and Talukder; Chan, Nasrabadi and Torrieri; Chung, Kee and Kim; Cootes, Taylor, Cooper and Graham; Covell; Cui and Weng; Daily and Cottrell; Demir, Akarun, and Alpaydin; Duta, Jain and Dubuisson-Jolly; Hallinan; Han and Tewfik; Jebara and Pentland; Kagesawa, Ueno, Kasushi, and Kashiwagi; King and Xu; Kalocsai, Zhao, and Elagin; Lee, Jung, Kwon and Hong; Liu and Wechsler; Menser and Muller; Moghaddam; Moon and Philips; Murase and Nayar; Nishino, Sato, and Ikeuchi; Novak, and Owirka; Nishino, Sato, and Ikeuchi; Ohta, Kohtaro and Ikeuchi; Ong and Gong; Penev and Atick; Penev and Sirivitch; Lorente and Torres; Pentland, Moghaddam, and Starner; Ramanathan, Sum, and Soon; Reiter and Matas; Romdhani, Gong and Psarrou; Shan, Gao, Chen, and Ma; Shen, Fu, Xu, Hsu, Chang, and Meng; Sirivitch and Kirby; Song, Chang, and Shaowei; Torres, Reutter, and Lorente; Turk and Pentland; Watta, Gandhi, and Lakshmanan; Weng and Chen; Yuela, Dai, and Feng; Yuille, Snow, Epstein, and Belhumeur; Zhao, Chellappa, and Krishnaswamy; Zhao and Yang.

Lambertian Reflectance

• Matt surface• Light source is distant• Light reflected equally

to all directions

n̂

cos ( 90 !)

ˆ ˆ( , )

o

T

I E

I l n l E l n n

or

l̂

Photometric Stereo: Factorization

• M is f x p (#images x #pixels)• L is f x 3 – light sources• S is 3 x p – surface normals (scaled by albedo)• Rank(M)=3 (if no noise present)• SVD:

• Ambiguity

Eliminate by forcing integrability

( )( )TM U V LS

M LS

1M LA AS

Relief Sculptures

Illumination Cone

=0.5* +0.2* +0.3*

Ball Face Phone Parrot

#1 48.2 53.7 67.9 42.8

#2 84.4 75.2 83.2 69.7

#3 94.4 90.2 88.2 76.3

#4 96.5 92.1 92.0 81.5

#5 97.9 93.5 94.1 84.7

#6 98.9 94.5 95.2 87.2

#7 99.1 95.3 96.3 88.5

#8 99.3 95.8 96.8 89.7

#9 99.5 96.3 97.2 90.7

#10 99.6 96.6 97.5 91.7

Intuition

0 1 2 30

0.5

1

0 1 2 30

0.5

1

1.5

2

lighting

reflectance

Spherical Harmonics

• Orthonormal basis for functions on the sphere

• n’th order harmonics have 2n+1 components

• Rotation = phase shift (same n, different m)

• In space coordinates: polynomials of degree n

• Funk-Hecke convolution theorem

( )!(2 1)( , ) (cos )

4 ( )!im

nm nm

n mnY P e

n m

2 / 22(1 )

( ) ( 1)2 !

m n mn

nm n n m

z dP z z

n dz

Spherical Harmonics

Z YX

23 1Z XZ YZ22 YX XY

Harmonic Transform of Kernel

1.023

0.495

-0.111

0.05

-0.029

0.886

-0.5

0

0.5

1

1.5

0 1 2 3 4 5 6 7 8

00

( ) max(cos ,0) n nn

k k Y

nk

n

37.5

87.599.22 99.81 99.93 99.97

0

20

40

60

80

100

0 1 2 3 4 5 6 7 8

Cumulative Energy

N

1

N

nn

E

(percents)

Second Order Approximation

21 1 15( ) cos cos 2

64 2 64k

Other Low-D Approximations

Hemisphere Foreshortened Ball (Exp.) Face Model Face (Exp.)

#1 51 62 48 61 54

#2 69 77 84 82 75

#3 88 92 94 92 90

#4 93 95 97 96 92

#5 95 97 98 97 94

#6 98 98 99 98 95

#7 98 99 99 99 95

#8 99 99 99 99 96

#9 99 99 100 99 96

(Ramamoorthi)

Harmonic Images( ) ( , , )nm nm x y zb p r n n n

zn xn yn

2(3 1)zn 2 2( )x yn n x yn n x zn n y zn n

Reconstruction

Reconstruction

Motion + Illumination

Reconstruction

Reconstruction Laser scan

Advantage of Our Method

0 10 20 30 40 500

1

2

3

4

5

0 10 20 30 40 500

10

20

30

40

Disparity error Disparity error

Residue Std intensity

Accounting forillumination variation

Assuming brightnessconstancy

Mutual Information(Viola and Wells)

CameraRotation