81
1 Manifold models for video-based face recognition Pavan Turaga Rama Chellappa

Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Embed Size (px)

Citation preview

Page 1: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

1

Manifold models for video-based face recognition

Pavan Turaga Rama Chellappa

Page 2: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

2

Manifold models of appearance and motion and its application in Video-based FR

Illumination

Shape and Reflectance

Motion

Image Space

Page 3: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

3

Constraints as Manifolds

Examples in high-dimensional

spaces

1. Images 2. Patches

Features And Models

1. Subpaces 2. Linear Dynamic

Models

Manifold Learning Riemannian/Differential Geometry

Page 4: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

How manifolds can help?

Part 1 •  Appearance Models for Detection and Tracking

–  Probabilistic Manifold models of faces –  Shape-illumination manifold

Part 2 •  Joint appearance and motion models

–  Image-set modeling using subspaces –  Video modeling using dynamical models

4

Page 5: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Manifold learning: Formal tools to address questions of intrinsic dimensionality

5

Pose-Illumination manifold Pose-expression manifold

J. B. Tenenbaum, V. de Silva and J. C. Langford, A Global Geometric Framework for Nonlinear Dimensionality Reduction Science 290 (5500): 2319-2323, 22 December 2000 . Sam Roweis & Lawrence Saul, Nonlinear dimensionality reduction by locally linear embedding. Science v.290 no.5500, Dec.22, 2000. pp.2323--2326 He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3): 328-340 (2005)

Page 6: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

6

Riemannian and Differential manifolds

These mappings are called charts, and the set of charts together is called an Atlas

Topological Space: A set X with a collection T of subsets of X with some properties. Haussdorff: Given 2 points, there exist non-overlapping neighborhoods Second Countable: The whole space can be spanned by a system of ‘bases’. e.g. open intervals for the real-line Each neighborhood is homeomorphic with a Euclidean space Topological spaces with these properties are called manifolds

Page 7: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Two strands of research: Similarities and Differences

7

Concept Manifold Learning Differential Geometry

Charts Estimate from a dense sampling of the manifold.

Start with analytically defined charts e.g. stereographic projections of a sphere.

Distance on manifold Shortest path-length on a graph.

Shortest ‘geodesic’ connecting two points.

Point-to-manifold distance

Hard to compute in general.

Hard to compute in general.

Statistics Classical statistics in the charts.

Karcher mean, Karcher variance etc.

Page 8: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Part 1 Example-based manifold models for detection, tracking, and recognition

8

Page 9: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Appearance Manifolds, Murase and Nayar

9

Hiroshi Murase, Shree K. Nayar: Visual learning and recognition of 3-d objects from appearance. International Journal of Computer Vision 14(1): 5-24 (1995) Introduced the idea that high dimensional image data potentially has fewer degrees of freedom, and showed how to exploit it for recognition.

Page 10: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

The Lambertian reflectance model and Illumination Cone (1998)

10 What Is the Set of Images of an Object under All Possible Illumination Conditions? P. N. Belhumeur, and D. J. Kriegman, IJCV 1998.

Page 11: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Manifolds as appearance models for detection/tracking

•  Appearance model: A description of what is a ‘face’. •  Should encode allowable variations: pose, expression,

illumination. •  Simpler appearance models

–  A static image –  A few images with poses, expressions, etc.

•  What do manifolds afford –  A joint representation of pose, illumination, expression –  A formal way of parameterizing observed variations

11

Page 12: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

How to create a manifold appearance model?

12

Estimate person-specific manifold

Manifold of Person A

Manifold of Person B Test video

Page 13: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Tools from manifold learning

13

Image courtesy Wikipedia

Many tools now available LLE, Isomaps, Laplacian Eigenmaps, Diffusion maps etc Each method preserves different Properties e.g. global distances, local structures, etc.

J. B. Tenenbaum, V. de Silva and J. C. Langford, A Global Geometric Framework for Nonlinear Dimensionality Reduction Science 290 (5500): 2319-2323, 22 December 2000 . Sam Roweis & Lawrence Saul, Nonlinear dimensionality reduction by locally linear embedding. Science v.290 no.5500, Dec.22, 2000. pp.2323--2326 He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3): 328-340 (2005)

Page 14: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Tracking as manifold-based search

14

Person-specific manifold

Sub-image of larger image

Kuang-Chih Lee, Jeffrey Ho, Ming-Hsuan Yang, David J. Kriegman: Visual tracking and recognition using probabilistic appearance manifolds. Computer Vision and Image Understanding 99(3): 303-331 (2005)

Page 15: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Problems that arise in face detection and tracking

15

What is the distance of this new point to each manifold?

•  Out of sample extension: Find parameters of a new point.

•  Reconstruct ambient data point from low dimensional embedding.

These are relatively new research directions. A few related papers: Yoshua Bengio, Jean-François Paiement, Pascal Vincent, Olivier Delalleau, Nicolas Le Roux, Marie Ouimet: Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. NIPS 2003. Chen Haojun, Jorge Silva, David Dunson and Lawrence Carin. 2010. "Hierarchical Bayesian Embeddings for Analysis and Synthesis of Dynamic Data." 2010 AAAI Fall Symposium Series. S. Gerber, T. Tasdizen, and R. Whitaker. Dimensionality reduction and principal surfaces via kernel map ma nifolds. In Computer Vision, 2009 IEEE 12th International Conference on, page 529--536, 2009.

Page 16: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Recognition as ‘closest manifold’ search

•  Do not know the person specific manifold explicitly •  Populate with examples •  Find closest manifold on a recursive fashion. •  A hard optimization problem •  Split into two simpler problems

16

Detection sub-problem

Recognition sub-problem

Page 17: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Core problem: Distance of a point to a manifold

17

How to solve this, when only a finite, coarse sampling of the manifold is available?

Page 18: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Probabilistic interpretation: Frame-by-frame recognition

18

Computing the distance of an image patch to the kth manifold using ‘average’ distance to manifold

Using the distance to manifold to obtain a posterior pdf over identities

Page 19: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Simplification using union-of-subspaces

19

Difficult to evaluate

Easy to evaluate (distance of point to plane)

Evaluate using temporal coherence

Page 20: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Imposing temporal coherence

20

Learn subspace-to-subspace transition matrix from training data for each subject

Page 21: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Recognition and tracking

21

Detection sub-problem Solve using distance-to manifold + particle-filter approach

Recognition sub-problem Solve using collection-of-subspace And temporal coherence model

Page 22: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Sample Results

22

Page 23: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Is there a generic constraint on human faces?

•  Given a specific identity and pose (ensured by the re-illumination algorithm), the log-difference images encode variations due to shape and illumination.

•  Hypothesis: This set of log-difference images live on a smooth manifold, and is ‘generic’ to human faces !

•  Named the ‘Shape-Illumination Manifold’ (SIM). •  Motivated by ‘generic-shape different-albedo’ assumptions from

literature.

23

Ognjen Arandjelovic, Roberto Cipolla: Face Recognition from Video Using the Generic Shape-Illumination Manifold. ECCV (4) 2006: 27-40 Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)

Page 24: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Consider the same identity and pose

24

Image Albedo Shape + Illumination

Only a function of shape+illumination Log-Difference Images

Page 25: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

How to use this hypothesis in practice?

25

Data in the image-space encodes variations from pose, illumination, shape etc. Ognjen Arandjelovic, Roberto Cipolla: Face Recognition from Video Using the Generic Shape-Illumination Manifold. ECCV (4) 2006: 27-40 Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)

Page 26: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Matching points and manifold neighborhoods

26

Motion Manifold 1 (Video 1)

Motion Manifold 2 (Video 2)

Identify correspondences by taking into account nearest manifold matches.

Page 27: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

How to use manifold neighborhoods to regularize correspondences ?

27

Page 28: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Face re-illumination using pose correspondences

•  Given two video sequences of the same subject in different illuminations, can one re-illuminate the first video using the second?

•  Yes, if you can find good correspondences in terms of pose.

28

Original Face

Re-illuminated face Ognjen Arandjelovic, Roberto Cipolla: Face Recognition from Video Using the Generic Shape-Illumination Manifold. ECCV (4) 2006: 27-40 Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)

Page 29: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Details of Cost Functions

•  Edge-based distance maps used to measure pose-error.

•  Estimating geodesic distances using Floyd’s algorithm.

•  Genetic algorithm to optimize correspondences.

29

Page 30: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Sample Re-illumination results

30

(a) Original images from a novel video sequence and (b) the result of re-illumination using the proposed genetic algorithm with nearest neighbour-based reconstruction.

Page 31: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Generic manifold constraint for identity matching

•  Input: Several generic videos of same person in multiple illumination conditions.

•  For each video, re-illuminate with video of potential match. •  This ensures pose-correspondence as well. •  Use re-illuminated and pose-corresponding images to obtain log-

difference images. •  Estimate a manifold from these log-difference images.

31

Page 32: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Video matching with the generic SIM model

•  Video 1 from Gallery, Video 2 from person to be recognized. •  Assuming they are from the same person, apply re-illumination

algorithm. •  Use obtained pose-correspondences, to obtain log-likelihood

images. •  If the two videos are from the same person, this set of images

will ‘lie close’ to the generic shape-illumination manifold. •  Question: How to measure closeness of points to a manifold?

32

Page 33: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

How close are a given set of points to a manifold?

•  Technical problem: Measure point-to-manifold distance. •  No easy solution, very hard as already seen before. •  Possible solutions: Manifold learning + out-of-sample extension. •  Proposed solution: Fit GMM to Shape-illumination manifold,

and evaluate likelihood of the new points under the GMM.

33

Page 34: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Algorithms for training and recognition

34

Page 35: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Illustrative Results

35

Page 36: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Part 2 Differential geometric approach for joint

appearance and motion modeling

36

Page 37: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

37

An analytic model for face appearance variation

•  Facial dynamics modeled as LTI system

),0(~)(),()()1(),0(~)(),()()(QNtvtvtAztzRNtwtwtCztf

+=+

+=

A,C,Q,R )0(z ),2(),1(),0( fff

Gaurav Aggarwal, Amit K. Roy Chowdhury, Rama Chellappa: A System Identification Approach for Video- based Face Recognition. ICPR (4) 2004: 175-178. Pavan K. Turaga, Ashok Veeraraghavan, Anuj Srivastava, Rama Chellappa: Statistical Computations on Grassmann and Stiefel Manifolds for Image and Video-Based Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33(11): 2273-2286 (2011)

Page 38: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

38

What do these matrices represent

dxdd Atz R,R)( ∈∈

nxdn Ctf R,R)( ∈∈

Page 39: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

39

Model Fitting

•  Learn the (A,C,Q,R) given a sequence of outputs •  Asymptotically Optimal Solution: EM •  Other approaches include N4SID •  Both are computationally expensive for large feature spaces •  Sub-optimal approach : PCA-ID

Page 40: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

40  

Distance measures on models

•  Comparing  two  dynamical  models  can  be  done  by  comparing  the  column  space  of  the  extended  observability  matrix.  

),0(~)(),()()1(),0(~)(),()()(QNtvtvtAztzRNtwtwtCztf

+=+

+=

)0(

.

...)0()0()0(

.

.)]2([)]1([)]0([

.

.)2()2()1()1()0()0(

.

.)2()1()0(

22 zCACAC

zCACAzCz

zCEzCEzCE

wCzwCzwCz

Efff

E

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

=

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

=

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

=

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

+

+

+

=

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

Page 41: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

41  

Recognition using dynamic models: Compare column spans of observability matrix

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

.

.

211

11

1

ACACC

),0(~)(),()()1(),0(~)(),()()(

11

11

QNtvtvtzAtzRNtwtwtzCtf

+=+

+=),0(~)(),()()1(),0(~)(),()()(

22

22

QNtvtvtzAtzRNtwtwtzCtf

+=+

+=

⎥⎥⎥⎥⎥⎥

⎢⎢⎢⎢⎢⎢

.

.

222

22

2

ACACC

Compare  column  span  

Key concept is distance between subspaces

Page 42: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Subspace models of face recognition is a special case

42

Page 43: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

43  

The Grassmann Manifold

•  Grassmann  Manifold:  The  space  of  m  dimensional  subspaces  in  RD  

Image courtesy Jihun Hamm

Page 44: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

44  

Distance using Subspace angles

P subspace

p basis lorthonorma

angle principal|)(|cos),( 211

21 == − ppPPd T

T

k

T VUYY⎥⎥⎥⎥

⎢⎢⎢⎢

=

σ

σ

σ

2

1

21

Cosines  of  principal  angles      

Page 45: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

45

Formal definitions

•  Topological Space: A set X with a collection T of subsets of X with some properties.

•  Haussdorff: Given 2 points, there exist non-overlapping neighborhoods •  Second Countable: The whole space can be spanned by a system of ‘bases’. e.g. open

intervals for the real-line •  Each neighborhood is homeomorphic with the euclidean space •  Topological spaces with these properties are called manifolds

Page 46: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

46

Differential manifolds: Smooth Mappings

Page 47: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

47

Simplest case

•  Lets consider Rn.

•  Take two open intervals U1 and U2 and consider the chart-changing

•  In fact all vector-spaces are manifolds.

nnxx RR →= :,)( φφ

xxFF nn

=

→= −

)(:2

11 RRφφ

Page 48: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

48

The case of the sphere

ζξ

φφ

/1)(:2

11

=

→= −

FF CC

Page 49: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

49

Tangent Spaces

f

γ

)]0(')[(

))((: 0

γ

γξ

xfD

tfdtdf tx

=

= =

Page 50: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

50

Submanifolds

•  Consider a subset P of a points of a manifold M, •  P = {X in M| f(X) = 0}. •  If the function f() satisfies certain conditions, then the set P is a

submanifold of M. •  Conditions: f() is ‘full-rank’. i.e. its Jacobian is full rank. •  Then, dim(P) = dim(M) – rank(J).

•  Example, consider F:Rn,k → Sym(k), given by F(X) = XTX – I. •  Then, Stiefel = F-1(0) •  Can prove that F is full rank i.e. there exists a Y, for every Z

such that DF(X)[Y] = Z. •  Thus, Stiefel is a submanifold of Rn,k

Page 51: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

51

Quotient Spaces

•  Define an equivalence relation on a manifold M i..e ‘~’. •  We say, X~Y if X is equivalent to Y. •  Then, P = M/~ is called a Quotient. •  P is a manifold if ~ satisfies some properties. •  To each ~ we define a projection. •  Example, Stiefel(n,k) = O(n)/O(n-k). •  Example, Grassmann(n,k) = Stiefel(n,k)/O(k)

Page 52: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

52

What can we do once we are on a manifold

•  Optimization •  Classification and Regression •  Interpolation •  Pretty much anything you can do on vector spaces.

•  But…. need some tools.

Page 53: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

53

Retractions

Page 54: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

54

Example

VPVPVRP +

+=)(

Page 55: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

55

Exponential and Logarithmic Maps

•  Special form of retraction that maps to geodesics )sin()cos()( V

VVVPVExpP +=

Page 56: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

56

Intrinsic Statistics

•  Intrinsic mean

•  Sample based Karcher mean – maximum likelihood estimate for the intrinsic mean

for weighted samples

2argmin ( , ) ( )dM M

d X p X Xψ

µ ψ∈

= ∫

Page 57: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

57  

Geometry of Grassmann manifold

•  Best  studied  in  terms  of  SO(n)  i.e.  Special  Orthogonal  Group  

•  The  exp  map  for  SO(n)  is  the  standard  matrix  exponential.  

matrices symmetric-skew are at )( totangents0tangenttangent

0)0()0(

0at t Evaluate0)()()()(

)()(I Q(0) with Q(t)by given SO(n)in curve aConsider

InSO

QQ

tQtQtQtQItQtQ

T

T

TT

T

=+=>

=+⇒

=

=+

=

=

Page 58: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

58

Grassmann and Stiefel as quotient of SO(n)

•  SO(n): nxn orthonormal matrix

•  Stiefel: Quotient space of SO(n)

Q =

Y =[Q]k =

First k

Page 59: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

59

Grassmann as Quotient of SO(n)

•  Grassmann is a quotient of Stiefel, and thereby also of SO(n).

Y =[Q]k =

First k

G =[Y]= span(Y)

[ ]

[ ]

Stiefel(k) k)-O(n)/O(n

equiv. are)(

*|,

.|)(

,

,,,,

,,

=

⎥⎦

⎤⎢⎣

=

−−

knYYSuppose

YYn

kkn

knkkkknnkn

knnkn

OZZI

O

Page 60: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

60

Tangents of Quotient Spaces

•  Any direction which takes you to a different equivalence class is a valid tangent to the quotient space.

How to compute exp and log maps for Grassmann manifold ? K. Gallivan, A. Srivastava, X. Liu, and P. VanDooren, “Efficient Algorithms for Inferences on Grassmann Manifolds,” In 12th IEEE Workshop Statistical Signal Processing, St. Louis, USA, October 2003.

t!Oexp(tA)J

Page 61: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

61  

Statistics on manifolds: Karcher Mean

Imposing  pdfs  on  manifolds  • Choose  a  ‘Pole’  • Unwrap  the  points  on  manifold  about  the  pole.  • Estimate  pdf  on  the  tangent  plane.  • Wrap  back  the  tangent  plane  pdf  onto  the  manifold.  

Page 62: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

62

Procedure for estimating pdfs

•  Choose a ‘Pole’ •  Unwrap the points on

manifold about the pole. •  Estimate pdf on the

tangent plane. •  Wrap back the tangent

plane pdf onto the manifold.

•  Issues: Which pole to choose ?

•  The ‘Karcher’ mean is the most popular choice.

n = 2, d = 1

Page 63: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

63

Extrinsic Metric: Procrustes Distance

)()( min),( 212121 pppptrPPd T ααα

−−=

1 and where,P subspace

=ℜ∈

=

pp

α

α

P subspace

p basis

Page 64: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

64

Procrustes Metric

)()(min),( 2121212 RXXRXXtrXXd T

R−−=

. where),(),( 21212 XXAAAItrXXd TT

k =−=

•  The Procrustes distance on Stiefel manifold can be derived from the Procrustes representation.

•  When R varies over all kxk real matrices, the distance is given by

Page 65: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

65

Extrinsic estimate of pdf

•  Given several points per class, kernel methods on Grassmann

manifold can be used to estimate class conditional densities with extrinsic divergence measures.

∑=

−− −=n

ii

TTik MXXXXIMKMC

nMXf

1

2/12/1 ])([)(1);(

Page 66: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

66

Video-based FR: MBGC experiment

Page 67: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Pose-specific linear subspaces

Pose specific linear subspace

Page 68: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

68  

Illumination invariant face recognition

Page 69: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Manifold constraints have been exploited in many more biometrics applications

•  Face Recognition •  Yui Man Lui, J. Ross Beveridge: Grassmann Registration Manifolds for Face

Recognition. ECCV (2) 2008: 44-57 •  Mehrtash Tafazzoli Harandi, Conrad Sanderson, Sareh Abolahrari Shirazi, Brian C.

Lovell: Graph embedding discriminant analysis on Grassmannian manifolds for improved image set matching. CVPR 2011: 2705-2712

•  Jihun Ham, Daniel D. Lee: Grassmann discriminant analysis: a unifying view on subspace-based learning. ICML 2008: 376-383

•  We present a few more examples illustrating the benefits in other biometric applications next.

Page 70: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

70

Landmark based Facial Geometric Modeling for Age Estimation and Verification

Age estimation as regression on Grassmann shape manifold

Verification as two-class classification of tangent-vectors of shape manifold

Page 71: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

71

Landmarks and their representation

⎥⎥⎥⎥

⎢⎢⎢⎢

nn yx

yxyx

..22

11 ⎥⎦

⎤⎢⎣

⎥⎥⎥⎥

⎢⎢⎢⎢

2221

121122

11

.. aaaa

yx

yxyx

nn

⎥⎦

⎤⎢⎣

⎥⎥⎥⎥

⎢⎢⎢⎢

2221

121122

11

.. aaaa

yx

yxyx

nn

Same Column Space

Pose and Camera Variations approximated

By Affine transforms New

Landmarks

Single point on Grassmann

manifold ⎥⎦

⎤⎢⎣

⎥⎥⎥⎥

⎢⎢⎢⎢

2221

121122

11

.. aaaa

yx

yxyx

nn

Landmarks Base Face

Page 72: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

72

Results

Cumulative Error on age estimation on FG-Net data

Verification results on FG-Net dataset.

T. Wu, P. Turaga, R. Chellappa, Age Estimation and Face Verification Across Aging using Landmarks, accepted at IEEE Transactions on Information Forensics and Security. 2012.

Page 73: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Blur-invariant Face Recognition

R. Gopalan, S. Taheri, P. Turaga, R. Chellappa, “A Blur-robust Descriptor with Applications to Face Recognition”, PAMI, vol. 34, no. 6, 2012.

Blurred image

Clean image Unknown blur kernel

Noise 2-D Pixel location

Image formation model under blur

Face images under no blur, medium blur, and extreme blur

),(),)(*(),(' 212121 nnnnkynny η+=

Page 74: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Space of blurred images ? Y=[(y*φ1)v (y*φ2)v . . . (y*φN)v]

Y’=[(y’*φ1)v (y’*φ2)v . . . (y’*φN)v]

span(Y) = span(Y’)

Skyy *'=

φ̂̂YY = φ̂ˆˆ' SKYY =

Y’=[(y*kS*φ1)v (y*kS*φ2)v . . . (y*kS*φN)v]

Page 75: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Results on FERET dataset

Page 76: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

The Image Deblurring Problem

76

y Hx γ= +

22argmin || || ( )optimal

xx y Hx E xλ= − +

Data Prior

Page 77: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Manifold prior for deblurring

77

Patches available in abundance ! Size of ‘patch-space’ is far smaller than image-space. Far easier to populate.

Page 78: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Regularization with Manifold Model

•  Let be a patch at location q. •  Let be the projection of patch onto the

patch manifold M. •  Then the optimization problem is recast as

78

( )qp x( ) Pr ( ( ))M qc q oj p x=

2

* * 2 2

, [0,1]

( , ) argmin || || || ( ) ( ) ||qx c

x c y Hx p x c q dqλ= − + −∫

))(()( 1 cAveryHIHHx TT λλ ++= −

Reconstruction of image by patch averaging.

Peyre 2009.

Jie Ni, Pavan K. Turaga, Vishal M. Patel, Rama Chellappa: Example-Driven Manifold Priors for Image Deconvolution. IEEE Transactions on Image Processing 20(11): 3086-3096 (2011)

Page 79: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Results: UMD MURI data

79

A Gaussian blur kernel with standard deviation of 2 is used.

Page 80: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

Conclusion

•  The ‘manifold’ benefits in detection, tracking, recognition, regularizing inverse problems etc.

•  Technical challenges are many, but potential pay-offs are high. •  Current solutions have only scratched the surface with simple

solutions to difficult problems. •  Huge scope for generalization and application to other problems. •  Closer collaboration between biometrics, machine learning,

applied mathematics will prove beneficial.

80

Page 81: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces

References

•  Kuang-Chih Lee, Jeffrey Ho, Ming-Hsuan Yang, David J. Kriegman: Visual tracking and recognition using probabilistic appearance manifolds. Computer Vision and Image Understanding 99(3): 303-331 (2005)

•  Arandjelovic, O. and Cipolla, R.. Face recognition from video using the generic shape-illumination manifold. In: Proc. 9th European Conference on Computer Vision, Graz (Austria) volume LNCS 3954, pages 27-40, Springer, 2006.

•  H. Murase and S.K. Nayar, "Visual Learning and Recognition of 3D Objects from Appearance,",International Journal on Computer Vision,Vol.14, No.1, pp.5-24, Jan, 1995.

•  Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)

•  Gaurav Aggarwal, Amit K. Roy Chowdhury, Rama Chellappa: A System Identification Approach for Video-based Face Recognition. ICPR (4) 2004: 175-178.

•  Pavan K. Turaga, Ashok Veeraraghavan, Anuj Srivastava, Rama Chellappa: Statistical Computations on Grassmann and Stiefel Manifolds for Image and Video-Based Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33(11): 2273-2286 (2011)

Related Applications •  Yui Man Lui, J. Ross Beveridge: Grassmann Registration Manifolds for Face Recognition. ECCV (2) 2008: 44-57 •  Mehrtash Tafazzoli Harandi, Conrad Sanderson, Sareh Abolahrari Shirazi, Brian C. Lovell: Graph embedding discriminant analysis

on Grassmannian manifolds for improved image set matching. CVPR 2011: 2705-2712

•  Jihun Ham, Daniel D. Lee: Grassmann discriminant analysis: a unifying view on subspace-based learning. ICML 2008: 376-383

81