Lambertian Reflectance and Linear Subspaces

Ronen Basri David Jacobs Weizmann NEC

Lighting affects appearance

How Complicated is Lighting?

Lighting => infinite DOFs.Set of possible images infinite dimensional (Belhumeur and Kriegman)

But, in many cases, lighting => 9 DOFs.

Our Results

Convex, Lambertian objects: 9D linear space captures >98% of reflectance.

Explains previous empirical results. (Epstein, Hallinan and Yuille; Hallinan;

Belhumeur and Kriegman)

For lighting, justifies low-dim methods.

Simple, analytic form. => New recognition algorithms.

DomainDomain

Lambertian

No cast shadows

Lights far and isotropic

nl

lmax (cos, 0)

Lighting

Images

...

Reflectance

0 1 2 30

0.5

1

0 1 2 30

0.5

1

1.5

2

r

Lighting to Reflectance: Intuition

(See D’Zmura, ‘91; Ramamoorthi and Hanrahan ‘00)

Spherical Harmonics

Orthonormal basis for functions on the sphere

Funk-Hecke convolution theorem

Rotation = Phase Shift

n’th order harmonic has 2n+1 components.

0.886

0.591

0.222

0.037 0.014 0.0070

0.5

1

1.5

0 1 2 3 4 5 6 7 8

Amplitudes of Kernel

n

nA

Reflectance functions near low-dimensional linear subspace

0

)(n

n

nmnmnmnm hLKlkr

Yields 9D linear subspace.

2

0

)(n

n

nmnmnmnm hLK

How accurate is approximation?

Accuracy depends on lighting.

For point source: 9D space captures 99.2% of energy

For any lighting: 9D space captures >98% of energy.

Forming Harmonic images),,()( ZYXrpb nmnm

YX

)(2 222 YXZ XZ YZ)( 22 YX XY

Accuracy of Approximation of Images

Normals present to varying amounts.

Albedo makes some pixels more important.

Worst case approximation arbitrarily bad.

“Average” case approximation should be good.

Models

Query

Find Pose

Compare

Vector: IMatrix: B

Harmonic Images

Comparison Methods

Linear:

Non-negative light:

(See Georghides, Belhumeur and Kriegman)

Non-negative light, first order approximation:

mina

Ba I

min , 0T

aBH a I a

min ,a

Ba I 2 2 2 20 1 2 34a a a a

Previous Linear Methods

Shashua. With no shadows, i=ln

with B = [X,Y,Z].

First harmonic, no DC

Koenderink & van Doorn heuristically

suggest using too.

||IB||min aa

Amano, Hiura, Yamaguti, and Inokuchi; Atick and Redlich; Bakry, Abo-Elsoud, and Kamel; Belhumeur, Hespanha, and Kriegman; Bhatnagar, Shaw, and Williams; Black and Jepson; Brennan and Principe; Campbell and Flynn; Casasent, Sipe and Talukder; Chan, Nasrabadi and Torrieri; Chung, Kee and Kim; Cootes, Taylor, Cooper and Graham; Covell; Cui and Weng; Daily and Cottrell; Demir, Akarun, and Alpaydin; Duta, Jain and Dubuisson-Jolly; Hallinan; Han and Tewfik; Jebara and Pentland; Kagesawa, Ueno, Kasushi, and Kashiwagi; King and Xu; Kalocsai, Zhao, and Elagin; Lee, Jung, Kwon and Hong; Liu and Wechsler; Menser and Muller; Moghaddam; Moon and Philips; Murase and Nayar; Nishino, Sato, and Ikeuchi; Novak, and Owirka; Nishino, Sato, and Ikeuchi; Ohta, Kohtaro and Ikeuchi; Ong and Gong; Penev and Atick; Penev and Sirivitch; Lorente and Torres; Pentland, Moghaddam, and Starner; Ramanathan, Sum, and Soon; Reiter and Matas; Romdhani, Gong and Psarrou; Shan, Gao, Chen, and Ma; Shen, Fu, Xu, Hsu, Chang, and Meng; Sirivitch and Kirby; Song, Chang, and Shaowei; Torres, Reutter, and Lorente; Turk and Pentland; Watta, Gandhi, and Lakshmanan; Weng and Chen; Yuela, Dai, and Feng; Yuille, Snow, Epstein, and Belhumeur; Zhao, Chellappa, and Krishnaswamy; Zhao and Yang.

PCA on many images

Comparison to PCA

Space built analytically

Size and accuracy known

More efficient

time,When pose unknown, rendering and PCA done at run time.

)( vs.)( 22 prOpsO 9100 rs ,

Experiments

3-D Models of 42 faces acquired with scanner.

30 query images for each of 10 faces (300 images).

Pose automatically computed using manually selected features (Blicher and Roy).

Best lighting found for each model; best fitting model wins.

Results

9D Linear Method: 90% correct.

9D Non-negative light: 88% correct.

Ongoing work: Most errors seem due to pose problems. With better poses, results seem near 100%.

Summary

We characterize images object produces.

Useful for recognition with 3D model.

Also tells us how to generalize from images.