Graph Laplacian Embeddingsce2.umkc.edu/csee/lizhu/teaching/2016.fall.image... · · 2016-10-31Graph Laplacian Embedding Zhu Li Dept of CSEE ... X He, S Yan, Y Hu, P Niyogi, HJ Zhang,

Image Analysis & Retrieval

CS/EE 5590 Special Topics (Class Ids: 44873, 44874)

Fall 2016, M/W 4-5:15pm@Bloch 0012

Lec 15

Graph Laplacian Embedding

Zhu Li

Dept of CSEE, UMKC

Office: FH560E, Email: [email protected], Ph: x 2346.

http://l.web.umkc.edu/lizhu

p.1Z. Li, Image Analysis & Understanding, 2016

mailto:[email protected]

http://l.web.umkc.edu/lizhu

Outline

Recap:

Eigenface

Fisherface

Graph Embedding

Laplacianface

Graph Fourier Transform

Summary


Subspace Learning for Face Recognition

Project face images to a subspace with basis A

Matlab: x=faces*A(:,1:kd)

eigf1

eigf2

eigf3

= 10.9* + 0.4* + 4.7*


PCA & Fisher’s Linear Discriminant

• Between-class scatter

• Within-class scatter

• Where

– c is the number of classes

– i is the mean of class i

– | i | is number of samples of i..

T

ii

c

i

iBS ))((1

c

i x

T

ikikW

ik

xS1

))((

1

2

12


Eigen vs Fisher Projection

p.5

• PCA (Eigenfaces)

Maximizes projected total scatter

• Fisher’s Linear Discriminant

Maximizes ratio of projected between-class to projected within-class scatter, solved by the generalized Eigen problem:

WSWW T

T

WPCA maxarg

WSW

WSWW

W

T

B

T

Wfld maxarg

12

PCA

Fisher𝑆𝐵𝑊 = 𝜆𝑆𝑊𝑊

Z. Li, Image Analysis & Understanding, 2016

LDA implementation

myLDA.m

p.6

% compute class mean

mx = mean(x);

ids = unique(y); m = length(ids);

Sb = zeros(kd, kd);

for k=1:m

indx = find(y==ids(k)); nk(k) = length(indx);

% class mean

m_cx(k,:) = mean(x(indx, :));

% between class scatter

Sb = Sb + nk(k)*(m_cx(k,:) - mx)'*(m_cx(k,:) - mx);

end

% compute intra-class scatter

Sw = zeros(kd, kd);

for k=1:m

indx = find(y==ids(k)); nk(k) = length(indx);

% remove mean

xk = x(indx, :) - repmat(m_cx(k,:), [nk(k), 1]);

% adding up

Sw = Sw + (xk'*xk);

end


LDA Implementation

Find projection by generalized Eigen problem solution

p.7

% generalized eigen problem

[A, v]=eigs(Sb, Sw);

if dbg

figure(31);

subplot(2,2,1); imagesc(Sb);

colormap('gray'); title('S_b');

subplot(2,2,2); imagesc(Sw);

colormap('gray'); title('S_w');

z = x*A; dist = pdist2(z, z);

subplot(2,2,3); imagesc(dist);

title('dist(j,k)');

subplot(2,2,4); stem(diag(v), '.'); grid on;

hold on; title('eig v');

end

𝑆𝐵𝑊 = 𝜆𝑆𝑊𝑊


Fisherface Basis

It is interesting to compare Fisherface with Eigenfacebasis

p.8

FisherfaceEigenface


Fisherface Performance

Fisher vs Eigenface performance: (fisherface.m)

1200 face images, 144 subjects

Eigen kd=32

p.9

144 subjectsROC


Outline

Recap:

Eigenface

Fisherface

Graph Embedding

Locality Preserving Projection

Laplacian Face


Summary


Locality Preserving Projection

Recall the dimension reduction formulations: find w, s.t y=wx:

PCA:

LDA:

p.11

max𝑤

wTSw

S = SB + SW


LPP Formulation –Affinity

To preserve local affinity relationship

Affinity map

Selection of heat kernel size and threshold are important

Hint: affinity matrix should be sparse

p.12

affinity histogram


LPP Formulation –Affinity Supervised

How to utilize the label info ?

Heat map mapping is good for intra-class affinity modelling, but how about intra-class affinity ?

One direct solution is to set affinity to zero for intra class pairs

p.13

% LPP - compute affinity

f_dist1 = pdist2(x1, x1);

% heat kernel size

mdist = mean(f_dist1(:)); h = -

log(0.15)/mdist;

S1 = exp(-h*f_dist1);

id_dist = pdist2(ids, ids);

subplot(2,2,3); imagesc(id_dist);

title('label distance');

S2=S1; S2(find(id_dist~=0)) = 0;

subplot(2,2,4); imagesc(S1);

colormap('gray'); title('affinity-

supervised');


LPP- Affinity Preserving Projection

Find a projection that best preserves the affinity matrix

p.14

min𝑊

𝑖

𝑗

𝑆𝑖,𝑗 𝑦𝑖 − 𝑦𝑗2, 𝑠. 𝑡. 𝑦 = 𝑊𝑥

min𝑊

𝑖

𝑗

𝑆𝑖,𝑗 𝑊𝑥𝑖 −𝑊𝑥𝑗2

𝑠. 𝑡, 𝑆𝑖,𝑗 = −exp 𝑥𝑗 − 𝑥𝑖

2

ℎ, 𝑖𝑓 𝑥𝑗 − 𝑥𝑖 ≤ 𝜃

0, 𝑒𝑙𝑠𝑒


LPP Formulation

L = D-S:

nxn matrix, called graph Laplacian

Normalizing factor: nxn D

Diagonal matrix, entry Dii = sum of col/row affinity

The larger the value, the more important data point is

Constraint on D:


Generalized Eigen Problem

Now the formulation is,

Lagranian

by KKT (Karush-Khun-Tucker) Condition, it is solved by a generalized Eigen problem

p.16

𝐿 𝑤, 𝜆 = 𝑤𝑇𝑋𝐿𝑋𝑇𝑤 − 𝜆(𝑤𝑇𝑋𝐷𝑋𝑤 − 1)


X He, S Yan, Y Hu, P Niyogi, HJ Zhang, “Face Recognition Using Laplacianface”, IEEE Trans PAMI, vol. 27 (3), 328-340, 2005.

Matlab Implementation

laplacianface.m

p.17

%LPP

n_face = 1200; n_subj = length(unique(ids(1:n_face)));

% eigenface

kd=32; x1 = faces(1:n_face,:)*A1(:,1:kd); ids=ids(1:n_face);

% LPP - compute affinity

f_dist1 = pdist2(x1, x1);

% heat kernel size

mdist = mean(f_dist1(:)); h = -log(0.15)/mdist;

S1 = exp(-h*f_dist1);

figure(32); subplot(2,2,1); imagesc(f_dist1); colormap('gray'); title('d(x_i, d_j)');

subplot(2,2,2); imagesc(S1); colormap('gray'); title('affinity');

%subplot(2,2,3); grid on; hold on; [h_aff, v_aff]=hist(S(:), 40); plot(v_aff, h_aff,

'.-');

% utilize supervised info

id_dist = pdist2(ids, ids);

subplot(2,2,3); imagesc(id_dist); title('label distance');

S2=S1; S2(find(id_dist~=0)) = 0;

subplot(2,2,4); imagesc(S1); colormap('gray'); title('affinity-supervised');

% laplacian face

lpp_opt.PCARatio = 1;

[A2, eigv2]=LPP(S2, lpp_opt, x1);


Laplacian Face

Now, we can model face as a LPP projection:

p.18

Eigenface Laplacian Face


Laplacian vs Eigenface

1200 faces, 144 subjects


LPP and PCA

Graph Embedding is an unifying theory on dimension reduction

PCA becomes special case of LPP, if we do not enforce local affinity


LPP and LDA

How about LDA ?

Recall within class scatter:

p.21

This is i-th classData covariance

Li has diagonal entry of 1/ni,Equal affinity among data points


LPP and LDA

Now consider the between class scatter

C is the data covariance, regardless of label

L is graph Laplacian computed from the affinity rule that,

p.22

𝑆 𝑖, 𝑗 =

1

𝑛𝑘, 𝑖𝑓 𝑥𝑖 , 𝑥𝑗𝑎𝑟𝑒 𝑓𝑟𝑜𝑚 𝑐𝑙𝑎𝑠𝑠 𝑘

0, 𝑒𝑙𝑠𝑒


LDA as a special case of LPP

The same generalized Eigen problem


Graph Embedding Interpretation of PCA/LDA/LPP

Affinity graph S, determines the embedding subspace W, via

PCA and LDA are special cases of Graph Embedding

PCA:

LDA

LPP

p.24

𝑆𝑖,𝑗 = −exp 𝑥𝑗 − 𝑥𝑖

2

ℎ, 𝑖𝑓 𝑥𝑗 − 𝑥𝑖 ≤ 𝜃

0, 𝑒𝑙𝑠𝑒

𝑆𝑖,𝑗 =

1

𝑛𝑘, 𝑖𝑓 𝑥𝑖 , 𝑥𝑗 ∈ 𝐶𝑘

0, 𝑒𝑙𝑠𝑒

𝑆𝑖,𝑗 = 1/𝑛

Z. Li, Adv. Multimedia Communciation, 2016

Fall

Applications: facial expression embedding

Facial expressions embedded in a 2-d space via LPP

p.25

frown

sad

happy

neutral


Application: Compression of SIFT

Compression of SIFT, preserve matching relationship, rather than reconstruction:

Z. Li, Image Analysis & Understanding, 2016 p.26

𝐴 = argmin𝑊

𝑘

𝑗

𝑠𝑗,𝑘 𝑊𝑥𝑗 −𝑊𝑥𝑘2

Homework-3: Subspace Methods

Objective:

Understand the graph embedding connections among popular subspace methods like PCA, LDA and LPP

Practical experiences with serious size data set

Data Set:

https://umkc.app.box.com/s/0qu7tc3jb88at2h53l1dpcuqkt9pn7ww

417 subjects, 6650 image face data set, pre-processed to 20x20 pel images, intensity normalized to [0, 1]

Add your own face images, 10~15, frontal

Tasks:

Compute Eigenface, Fisherface and Laplacianface models

ROC plot on verification performance

mAP for retrieval/identification performance


HW-3 test run

Laplacian face is powerful.



David I. Shuman, Sunil K. Narang, Pascal Frossard, Antonio Ortega, Pierre Vandergheynst:The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains. IEEE Signal Process. Mag. 30(3): 83-98 (2013)


http://dblp.uni-trier.de/pers/hd/n/Narang:Sunil_K=

http://dblp.uni-trier.de/pers/hd/f/Frossard:Pascal

http://dblp.uni-trier.de/pers/hd/o/Ortega:Antonio

http://dblp.uni-trier.de/pers/hd/v/Vandergheynst:Pierre

http://dblp.uni-trier.de/db/journals/spm/spm30.html#ShumanNFOV13

Signal on Graph

non-uniformly sampled

p.30Z. Li, Adv. Multimedia Communciation, 2016

Fall


GFT is different from Laplacian Embedding:


Fall

GFT Example

Graph Laplacian


Fall

Normalized Graph Laplacian

Normalize by edge pair degree


Fall

Graph Frourier Transform

Analogous to FT


Fall

Graph Spectrum


Fall

Graph Signal Smoothness

Quadratic form on L:


Fall

Summary

Graph Laplacian Embedding is an unifying theory for feature space dimension reduction PCA is a special case of graph embedding

o Fully connected affinity map, equal importance

LDA is a special case of graph embeddingo Fully connected intra class

o Zero affinity inter class

LPP: preserves pair wise affinity.

GFT: eigen vectors of graph Laplacian, has Fourier Transform like characteristics.

Many applications in Face recognition

Pose estimation

Facial expression modeling

Compression of Graph signals.


Documents

Graph Laplacian Embeddingsce2.umkc.edu/csee/lizhu/teaching/2016.fall.image... · · 2016-10-31Graph Laplacian Embedding Zhu Li Dept of CSEE ... X He, S Yan, Y Hu, P Niyogi, HJ Zhang,