Multilinear Principal Component Analysis of Tensor Objectst8zhu/talks/2oral.pdf · 2017-07-11 · Multilinear Principal Component Analysis of Tensor Objects Introduction Brief Introduction

Multilinear Principal Component Analysis of Tensor Objects

Multilinear Principal Component Analysis ofTensor Objects

Tingyi Zhu

Department of MathematicsTexas A&M University

July 12, 2012


Outline

1 Introduction

2 Basics of Multi-linear AlgebraTensorsMultilinear Projection

3 Multilinear Principal Component AnalysisMultilinear Subspace Learning ProblemMPCA Algorithm

4 Experiment ResultsSynthetic DataFacial Image DatasetMPCA-Based Gait Recognition

5 Summary

6 References


Introduction

Brief introduction

MPCA framework was proposed in 2008 by Lu, Plataniotis andVentsanopoulos in the paper [1].

In this presentation, I will

Review concepts in multilinear algebra

Summarize the MPCA algorithm

Present some experiment results of MPCA


Introduction

Brief Introduction

Principal Component Analysis (PCA)

Transform the original data set to a new set of variables

Reduce the dimensionality

Retain as much as possible the variation present in the originaldata set

A well-known un-supervised linear technique for dimensionalityreduction.


Introduction

Brief Introduction







Introduction

Brief Introduction







Introduction

Brief Introduction







Introduction

Brief Introduction







Introduction

Brief Introduction

To apply PCA on tensor objects, it requires their reshaping intovectors, which results in high processing cost in terms of increasedcomputational and memory demands.

For example, vectorizing a typical gait sequence of size (120×80×20)results in a vector with dimensionality (192000×1).

Besides, reshaping breaks the natural structure and correlation inthe original data.

Therefore, a dimensionality reduction algorithm operating directlyon a tensor object rather than its vectorized version is desirable.


Basics of Multi-linear Algebra

Tensors

What’s Tensor

Tensors are a further extension of ideas we use to define vectorsand matrix.

The elements of a tensor are to be addressed by N indices,where N defines the order of the tensor object and each indexdefines one mode. Thus, vectors are first-order tensors (withN = 1) and matrices are second-order tensors (with N = 2).Tensors with N > 2 can be viewed as a generalization of vectorsand matrices to higher order.



Tensors

An N th-order tensor is denoted as A ∈ RI1×I2×···×IN , which isaddressed by N indices in, n = 1, . . . , N , with each in addressesthe n-mode of A.

The Scalar product of two tensors A,B ∈ RI1×I2×···×IN :

< A,B >=∑i1

∑i2

· · ·∑iN

A(i1, . . . , iN ) · B(i1, . . . , iN ) (1)

.The Frobenius norm of A:

||A||F =√

< A,A >. (2)



Tensors

The ”n-mode vector” of A are defined as the In-dimensional vectorsobtained from A by varying its index in while keeping all the otherindices fixed. A rank-one tensor A equals to the outer product ofN vectors:

A = u(1) ◦ u(2) ◦ · · · ◦ u(N), (3)

i.e.,A(i1, . . . , iN ) = u(1)(i1) · u(2)(i2) · · · · · u(N)(iN ) (4)

for all values of indices.



Tensors

Unfolding A along the n-mode is denoted as:

A(n) ∈ RIn×(I1×···×In−1×In+1×···×IN ), (5)

where the column vectors of A(n) are the n-mode vectors of A.

The figure below illustrates the 1-mode (column mode) unfolding ofa third-order tensor.



Multilinear Projection

Multilinear projection

The n-mode product of a tensor A by a matrix U ∈ RJn×In , denoteas A×n U, is a tensor with entries:

(A×n U)(i1, . . . , in−1, jn, in+1, . . . , iN ) =∑in

A(i1, . . . , iN ) ·U(jn, in). (6)

Visual illustration:




Multilinear projection

Based on the definitions above, a tensor can be projected to anothertensor by N projection matrices U(1),U(2), . . . ,U(N) as:

Y = X ×1 U(1)T ×2 U

(2)T · · · ×N U(N)T . (7)





Visual illustration of tensor-to-tensor projection:




Following standard multilinear algebra, any tensor A can be ex-pressed as the product

A = S ×1 U(1) ×2 U

(2) × · · · ×N U(N) (8)

where S = A×1 U(1)T ×2 U

(2T ) × · · · ×N U(N)T .

A matrix representation of this decomposition can be obtained byunfolding A and S as

A(n) = U(n)·S(n)·(U(n+1)⊗U(n+2)⊗· · ·⊗U(N)⊗U(1)⊗U(2)⊗· · ·⊗U(n+1))T (9)

where ⊗ denotes the Kronecker product.


Multilinear Principal Component Analysis

Multilinear Subspace Learning Problem


The problem of multilinear subspace learning based on the tensor-to-tensor projection can be mathematically defined as follows:

A set of M N th-order tensor samples {X1,X2, . . . ,XM} is availablefor training, where each sample Xm is an I1 × I2 × · · · × IN tensorin a tensor space RI1×I2×···×IN .





The objective is to find a tensor-to-tensor projection {U(n) ∈ RIn×Pn ,n = 1, . . . , N}mapping from the original tensor space RI1⊗RI2 · · ·⊗RIN into a tensor subspace RP1 ⊗RP2 · · ·⊗RPN (with Pn < In, forn = 1, . . . , N):

Ym = Xm ×1 U(1)T ×2 U

(2)T · · · ×N U(N)T , (10)

such that the projected samples (the extracted features) satisfy anoptimality criterion, where the dimensionality of the projected spaceis much lower than the original tensor space.





The MPCA algorithm maximizes the following tensor-based scattermeasure:

ΨY =

M∑m=1

||Ym − Y||2F , (11)

named as the total tensor scatter, where

Y =1

M

M∑m=1

Ym (12)

is the mean sample.



MPCA Algorithm

MPCA Algorithm

There is no known optimal solution which allows for the simultane-ous optimization of the N projection matrices. Since the projectionto an N th-order tensor subspace consists of N projections to N vec-tor subspaces, N optimization subproblems can be solved by findingthe U(n) that maximizes the scatter in the n-mode vector subspace.This is discussed in the following theorem.



MPCA Algorithm

MPCA Algorithm

Theorem

Let {U(n), n = 1, ..., N} be the solution of the optimizationproblem. Then, for given all the other projection matricesU(1), ..., U(n−1), U(n + 1), ..., U(N), the matrix U(n) consistsof the Pn eigenvectors corresponding to the largest Pn eigenvaluesof the matrix

Φ(n) =

M∑m=1

(Xm(n)− X(n)) · UΦ(n) · UTΦ(n) · (Xm(n)− X(n))

T (13)

where

UΦ(n) = U(n+1)⊗U (n+2)⊗· · ·⊗U(N)⊗U(1)⊗· · ·⊗U(n−1). (14)



MPCA Algorithm

MPCA Algorithm(Pseudocode)



MPCA Algorithm

The computations of the projection matrices are interdependent,which implies there is no closed-form solution to the optimizationproblem. Therefore, an iterative procedure is utilized. The inputtensors are centered first. With initializations through full projectiontruncation (FPT), the projection matrices are computed one by onewith all the others fixed (local optimization). The local optimizationprocedure can be repeated until the result converges or a maximumnumber K of iterations is reached.



MPCA Algorithm

MPCA Algorithm(Algorithm flow)


Experiment Results

Synthetic Data

Synthetic Data Generation

100 third-order tensors Am ∈ RI1×I2×I3 are generated per set ac-cording to Am = Bm ×1 C

(1) ×2 C(2) ×3 C

(3) +Dm where

All entries in Bm are drawn from a zero-mean unit-varianceGaussian distribution and are multiplied by [(I1 · I2 · I3)/(i1 ·i2 · i3)]f .

The matrices C(n) are orthogonal matrices obtained by applyingSVD on random matrices with entries drawn from zero-mean,unit-variance Gaussian distribution.

All entries of Dm are drawn from a zero-mean Gaussian distri-bution with variance 0.01.

Three synthetic data sets db1, db2, and db3 with f = 1/2, 1/4, 1/16respectively, are created.


Experiment Results

Synthetic Data

0 5 10 15 20 25 3010

5

106

107

Eigenvalue index

Eig

envalu

e m

agnitude

MPCA Eigenvalue plot for db1

Mode 1

Mode 2

Mode 3

0 5 10 15 20 25 3010

4

105

106

Eigenvalue indexE

igenvalu

e m

agnitude


Mode 1

Mode 2

Mode 3

0 5 10 15 20 25 3010

4

105

106

Eigenvalue index

Eig

envalu

e m

agnitude


Mode 1

Mode 2

Mode 3

0 5 10 15 20 25 300.2

0.4

0.6

0.8

1

Eigenvalue index

Eig

en

va

lue

cu

mu

lative

dis

trib

utio

n

MPCA eigenvalue cumulative distribution for db1

Mode 1

Mode 2

Mode 3

0 5 10 15 20 25 300

0.2

0.4

0.6

0.8

1

Eigenvalue index

Eig

envalu

e c

um

ula

tive d

istr

ibution


Mode 1

Mode 2

Mode 3

0 5 10 15 20 25 300

0.2

0.4

0.6

0.8

1

Eigenvalue index

Eig

envalu

e c

um

ula

tive d

istr

ibution


Mode 1

Mode 2

Mode 3


Experiment Results

Synthetic Data

1 2 3 41.085

1.095

1.105

1.115

1.125x 10

7

Number of Iterations

ψY

Convergence plot for db1 with Q=0.75

FPT

Pseudo Identity

Random (average of 5)

(a)

1 2 3 41.27

1.3

1.33

1.36x 10

6


ψY


FPT

Pseudo Identity


(b)

0 10 20 305000

5500

6000

6500

7000

7500

8000Convergence plot for db3 with Q=0.15


ψY

FPT

Pseudo Identity


(c)

1 3 5 73.85

3.9

3.95

4

4.05

4.1x 10

5


ψY


FPT

Pseudo Identity


(d)

Figure: Convergence plot for different initializations. (a) Convergence plot for db1with Q=0.75. (b) Convergence plot for db2 with Q=0.75. (c) Convergence plot fordb3 with Q=0.15. (d) Convergence plot for db3 with Q=0.75.


Experiment Results

Facial Image Dataset


320 samples of face images

Size of each image: 32×32

Set Q=0.97

Then, P1 = 23 and P2 = 20

91% of original variation captured


Experiment Results



(a) (b)

Figure: Face image data of face recognition technology (FERET) database.(a) The first eight samples out of the total 320 samples. (b) The mean offace image samples.


Experiment Results



0 10 20 30 400

1

2

3

4

5x 10

8

MPCA index

Eig

envalu

e m

agnitude

MPCA Eigenvalue Plot

Mode 1

Mode 2

(a)

100

101

102

0.2

0.4

0.6

0.8

1

Eigenvalue index

Eig

envla

ue c

um

ula

tive d

istr

ibution

MPCA Eigenvalue Cumulative Distribution

Mode 1

Mode 2

(b)

Figure: Eigenvalue magnitudes and their cumulative distributions for thefacial datase.


Experiment Results

MPCA-Based Gait Recognition


Gait gallery data: 731 samples

Size of each sample: I1 × I2 × I3 = 32× 22× 10

I1 and I2 are frame size and I3 is spatial size

Dimensionality of the projected tensors: P1 × P2 × P3 = 26×15× 10.


Experiment Results


(a)

(b)

Figure: (a) 1-mode unfolding of a gait silhouette sample. (b) 1-modeunfolding of the mean of the gait silhouette samples.


Experiment Results


(a)

(b)

(c)

(d)

Figure: Representative ETGs: The first, second, third, 4th most discrimi-native ETGs.


Experiment Results


0 10 20 30 40

102

103

104

105

Eigenvalue index

Eig

en

va

lue

Ma

gn

itu

de

MPCA Eigenvalue Plot

Mode 1

Mode 2

Mode 3

(a)

100

101

102

0.2

0.4

0.6

0.8

1MPCA Eigenvalue Cumulative Distribution

Eigenvalue index

Eig

en

va

lue

cu

mu

lative

dis

trib

utio

n

Mode 1

Mode 2

Mode 3

(b)

Figure: Eigenvalue magnitudes and their cumulative distributions for thegallery set.


Summary

Summary


References

References

Haiping Lu and K. N. Plataniotis and A. N. VenetsanopoulosA Survey of Multilinear Subspace Learning for Tensor Data.Pattern Recognition, 44(7):15401551, 201.

H. Lu and K. N. Plataniotis and A. N. Venetsanopoulosi.Multilinear Principal Component Analysis of Tensor Objects.IEEE Trans. on Neural Networks, 19(1):18–39 2008.

L. D. Lathauwer, B. D. Moor, and J. VandewalleA multilinear singular value decomposition.SIAM J. Matrix Anal. Appl, vol. 21, no. 4, pp. 1253-1278, 2000.

Data Sources:http://www.dsp.utoronto.ca/haiping/index.php?page=code.

Tensor Toolbox:http://csmr.ca.sandia.gov/tgkolda/TensorToolbox/.

Documents

Multilinear Principal Component Analysis of Tensor Objectst8zhu/talks/2oral.pdf · 2017-07-11 · Multilinear Principal Component Analysis of Tensor Objects Introduction Brief Introduction