Proper Orthogonal Decomposition (POD) - Rice Ucaam699-2/slides_POD_699fall2008.pdf · IntroPODResultsPOD Problemextras Proper Orthogonal Decomposition (POD) Saifon Chaturantabut Advisor:

Intro POD Results POD Problem extras

Proper Orthogonal Decomposition (POD)

Saifon Chaturantabut

Advisor: Dr. SorensenCAAM699

Department of Computational and Applied MathematicsRice University

September 5, 2008

Saifon Chaturantabut POD

Intro POD Results POD Problem extras

Outline

1 Introduction/MotivationLow Rank Approximation and PODModel Reduction for ODEs and PDEs

2 Proper Orthogonal Decomposition(POD)Introduction to PODPOD in Euclidean SpacePOD in General Hilbert Space

3 Numerical ResultsSolutions from Full and Reduced Systems of PDE: 1DBurgers’Equation

4 Problem: POD for PDEs with nonlinearitiesNonlinear ApproximationError and Computational Time


Intro POD Results POD Problem extras Low Rank Approx and POD Model Reduction for ODEs and PDEs

Problem in finite dimensional spaceGiven y1, . . . , yn ∈ Rm. Let Y = spany1, y2, . . . , yn ⊂ Rm;r = rank(Y )

y1, . . . , yn ∈ Rm possible (almost) linearly dependent⇒ NOT agood basis for Y

Goal: Find orthonormal basis vectors φ1, . . . , φk that bestapproximates Y , for given k < rSolution: Use Low Rank ApproximationForm a matrix of known data:

Y =

| |y1 . . . yn| |

∈ Rm×n.



Low Rank ApproximationSingular Value Decomposition (SVD)

Let Y ∈ Rm×n, r = rank(Y ), and k < r .

Problem: Low Rank Approximation

minY‖Y − Y‖2

F : rank(Y ) = k

Solution

Y ∗ = Uk Σk V Tk with min error ‖Y − Y ∗‖2

F =∑r

i=k+1 σ2i

where Y = UΣV T is SVD of Y ; Σ = diag(σ1, . . . , σr ) ∈ Rr×r withσ1 ≥ σ2 ≥ . . . ≥ σr > 0

Optimal orthonormal basis of rank k =: POD basis

Y ∗ =∑k

i=1 σiuivTi ⇒ φk

i=1 = uiki=1



EX: Image Compression via SVD

1 % Low Rank Approximation

source: http://demonstrations.wolfram.com/demonstrations.wolfram.com



EX: Image Compression via SVD

50 % Low Rank Approximation

source: http://demonstrations.wolfram.com/demonstrations.wolfram.com



Model Reduction for ODEs

Full-order system (dim = N)

ddt

y(t) = Ky(t) + g(t) + N(y(t))⇒ y(t)

Reduced-order system (dim = k < N)

Let y = Uk y, for Uk ∈ RN×k , with orthonormal columns:UT

k Uk = I ∈ Rk×k .

Ukddt y(t) = KUk y(t) + g(t) + N(Uk y(t))⇒ d

dt y(t) = UTk KUk︸︷︷︸ y(t) + UT

k g(t)︸︷︷︸+UTk N(Uk y(t))

ddt

y(t) = Ky(t) + g(t) + UTk N(Uk y(t))⇒ y(t)

How to construct Uk? .....Use POD!



Model Reduction for PDEsEx. Unsteady 1D Burgers’Equation

∂

∂ty(x , t)− ν ∂

2

∂x2 y(x , t) +∂

∂x

(y(x , t)2

2

)= 0 x ∈ [0,1], t ≥ 0

y(0, t) = y(1, t) = 0, t ≥ 0, y(x ,0) = y0(x), x ∈ [0,1],

Discretized system (Galerkin)

Full-order system (dim = N): FE basis ϕiNi=1

Mhddt

y(t) + νKhy(t)−Nh(y(t)) = 0⇒ yh(x , t) =∑N

i=1 ϕi (x)yi (t)

Reduced-order system (dim = k < N): orthonormal basis φiki=1

Mddt

y(t) + νKy(t)− N(y(t)) = 0⇒ yh(x , t) =∑k

i=1 φi (x)yi (t)

How to construct φiki=1? .....Use POD!Saifon Chaturantabut POD

Intro POD Results POD Problem extras Intro POD in Euclidean Space POD in General Hilbert Space

Proper Orthogonal Decomposition(POD)

POD is a method for finding alow-dimensional approximaterepresentation of:

large-scale dynamicalsystems, e.g. signalanalysis, turbulent fluid flowlarge data set, e.g. imageprocessing

≡ SVD in Euclidean space.Extracts basis functionscontaining characteristicsfrom the system of interestGenerally gives a goodapproximation withsubstantially lower dimension

0 0.5 1−2

−1

0 POD basis # 1

0 0.5 1−4

−2

0

2 POD basis # 2

0 0.5 1−4

−2

0

2 POD basis # 3

0 0.5 1−4

−2

0

2 POD basis # 4

0 0.2 0.4 0.6 0.8 1 0

0.5

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

t

x

Sol of Full System (FE):dim = 100

y

Figure: Plots of the first 4 POD basisand the space of solutions



Definition of POD

Let X be Hilbert Space(I.e. Complete Inner Product Space) withinner product 〈·, ·〉 and norm ‖ · ‖ =

√〈·, ·〉

Given y1, . . . , yn ∈ X . Define yi ≡ snapshot i , ∀i . LetY ≡ spany1, y2, . . . , yn ⊂ X .Given k ≤ n, POD generates a set of orthonormal basis ofdimension k , which minimizes the error from approximating thesnapshots:

POD basis ≡ Optimal solution of:

minφk

i=1

n∑j=1

‖yj − yj‖2, s.t. 〈φi , φj〉 = δij

where yj (x) =∑k

i=1〈yj , φi〉φi (x), an approximation of yj usingφk

i=1.Solve by SVD



Derivation for POD in Euclidean Space: E.g. X = Rm

minφki=1

∑nj=1 ‖yj −

∑ki=1(yT

j φi )φi‖22 ≡ E(φ1, ..., φk )

s.t .

φTi φj = δij =

1 if i = j0 if i 6= j i , j = 1, ..., k

Lagrange Function:

L(φ1, . . . , φk , λ11, . . . , λij , . . . , λkk ) = E(φ1, ..., φk ) +k∑

i,j=1

λij (φTi φj − δij )

KKT Necessary Conditions:

∂∂φi

L = 0⇔ ∑n

j=1 yj (yTj φi ) = λiiφi

λij = 0, ifi 6= j

.

φTi φj = δij



Optimal Condition: Symmetric m-by-m eigenvalue problem

YY Tφi = λiφi

where λi = λii ,Y =

| |y1 . . . yn| |

for i = 1, . . . , k

Error for POD basis:n∑

j=1

‖yi −k∑

i=1

(yTj φi )φi‖2

2 =r∑

i=k+1

λi

λiφi = YY Tφi =∑n

j=1 yj (yTj φi )⇒ λi =

∑nj=1(yT

j φi )2

Since φTi φj = δij and span(Y ) = spanφ1, . . . , φr (from SVD),

then yj =∑r

i=1(yTj φi )φi , forj = 1, . . . ,n, and

n∑j=1

‖yj −k∑

i=1

(yTj φi )φi‖2

2 =n∑

j=1

r∑i=k+1

(yTj φi )

2 =r∑

i=k+1

λi



SOLUTION for POD basis in Rm

Recall Optimality Condition and the POD Error:YY Tφi = λiφi , i = 1, . . . , k∑n

j=1 ‖yj −∑k

i=1(yTj φi )φi‖2

2 =∑r

i=k+1 λi

Optimal solution:

POD basis: φ∗i ki=1 = uik

i=1

Lagrange multiplier: λ∗i = σ2i , i = 1, . . . , k ,

can be obtained by the SVD of Y ∈ Rm×n or EVD ofYY T ∈ Rm×m, i.e.,

Y = UΣV T ⇒ YY T ui = σ2i ui , i = 1, . . . , r ,

where Σ = diag(σ1, . . . , σr ) ∈ Rr×r with σ1 ≥ σ2 ≥ . . . ≥ σr > 0;U = [u1, . . . ,ur ] ∈ Rm×r and V = [v1, . . . , vr ] ∈ Rn×r haveorthonormal columns.This is equivalent to SVD solution for Low Rank Approximation



SOLUTION for POD basis in a Hilbert Space XTwo approaches:

1 Define linear symmetric operator F (w) =∑n

j=1〈w , yj〉yj , w ∈ X .Find eigenfunction ui ∈ X :

EVD: F (ui ) = σ2i ui∑n

j=1〈ui , yj〉yj = σ2i ui ∼ (cf. YY T ui = σ2

i ui )

Sol: φ∗i = ui , λ∗i = σ2

i , i = 1, . . . , k

2 Define linear symmetric operator L = [〈yi , yj〉] ∈ Rn×n. Findeigenvector vi ∈ Rn:

EVD: L vi = σ2i vi

[〈yi , yj〉]vi = σ2i vi ∼ (cf. Y T Yvi = σ2

i vi )

Sol: φ∗i = 1σi

∑nj=1(vi )jyj , λ

∗i = σ2

i ∼ Uk = YVk Σ−1k

NOTE: vi ∈ Rn,but ui , yj ∈ X



Remark: Practical way for computing the POD basis for discretized PDEs

Let ϕiNi=1 ⊂ X ≡ H1(Ω) be finite element(FE) basis.

FE snapshots(solutions): yjni=j ∈ X at time tjn

j=1.

yj = y(x , tj ) =N∑

i=1

Yij (tj )ϕi (x).

L = [〈yi , yj〉] = [m∑

k,`=1

Yik Yj`〈ϕi , ϕj〉] = Y T MY ∈ Rn×n,

where M = [〈ϕi , ϕj〉] ∈ RN×N .POD basis:

φ∗i =1σi

n∑j=1

(vi )jyj ,

whereL vi = σ2

i vi ,

with v1, v2, . . . , vk corresponding to σ1 ≥ σ2 ≥ . . . ≥ σk > 0.


Intro POD Results POD Problem extras Solutions from Full and Reduced Systems of PDE: 1D Burgers’Equation

Numerical Results for PDEs

Recall the 1D Burgers’Equation x ∈ [0, 1], t ≥ 0:∂∂t y(x, t)− ν ∂2

∂x2 y(x, t) + ∂∂x

(y(x,t)2

2

)= 0,

y(0, t) = y(1, t) = 0, t ≥ 0,

y(x, 0) = y0(x), x ∈ [0, 1].

0

0.5

1

0

0.5

1

1.5

20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

x

Sol of Reduced System (Direct POD):dim = 6

t

y

0

0.5

1

0

0.5

1

1.5

20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

x

Sol of Full System (Finite Element):dim = 100

t

y

0 10 20 30 40 50 60 7010−15

10−10

10−5

100 Singular values of the Snapshots

0 0.2 0.4 0.6 0.8 10

0.5

1

x

y(x,

t)

Sol of Full System (Finite Element) :dim = 100

t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x,

t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x,

t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x,

t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x,

t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x,

t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

Figure: Direct PODSaifon Chaturantabut POD

Intro POD Results POD Problem extras Nonlinear Approximation Error and CPU Time

Problem: POD for PDEs with nonlinearities

If we apply the POD basis directly to construct a discretized system,the original system of order N:

Mhddt y(t) + νKhy(t)− Nh(y(t)) = 0

become a system of order k N:

M ddt y(t) + νKy(t)− N(y(t)) = 0,

where the nonlinear term :

N(y(t)) = UTk︸︷︷︸

k×N

Nh(Uk y(t))︸︷︷︸N×1

⇒ Computational Complexity still depends on N!!



Nonlinear Approximation

Recall the problem from bf Direct POD :N(y(t)) = UT︸︷︷︸

k×N

Nh(Uy(t))︸︷︷︸N×1

WANT:

N(y(t))← C︸︷︷︸k×nm

N(y(t))︸︷︷︸nm×1

99K k ,nm N 99K Independent of N

2 Approaches:Precomputing Technique: Simple nonlinearEmpirical Interpolation Method (EIM): General nonlinear



ACCURACY vs. COMPLEXITY

0 5 10 1510−7

10−6

10−5

10−4

10−3

10−2

10−1

100

101

Number of POD basis used

Avera

ge R

ela

tive E

rror

Relative Error:= sumt||yFE(t)−y(t)||/||yFE(t)|| of Reconstruct Snapshots using POD

Direct PODPrecompute PODEIM−POD

0 5 10 150

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Number of POD basis usedcp

u t

ime

(se

c)

CPU TIME for solving reduced−order ODE system

Direct PODPrecompute PODEIM−POD

Figure: LEFT: Error Eavg = 1nt

∑nti=1‖yh(·,ti )−yh(·,ti )‖X‖yh(·,ti )‖X

.RIGHT: CPU time (sec)



QUESTION ?


Intro POD Results POD Problem extras Empirical Interpolation Method (EIM) Solutions from EIM Conclusions and Future Works

Empirical Interpolation Method (EIM) [Patera; 2004]

Approximates nonlinear parametrized functionsLet s(x ;µ) be a parametrized function with spatial variable x ∈ Ωand parameter µ ∈ D .Function approximation from EIM:

s(x ;µ) =nm∑

m=1

qm(x)βm(µ),

where spanqmnmm=1 'M s ≡ s(·;µ) : µ ∈ D and βm(µ) is

specified from the interpolation points zmnmm=1, in Ω:

s(zi ;µ) =nm∑

m=1

qm(zi )βm(µ),

for i = 1, . . . ,nm.



EIM: Numerical Example

s(x ;µ) = (1− x)cos(3πµ(x + 1))e−(1+x)µ,

where x ∈ [−1,1] and µ ∈ [1, π].

−1 −0.5 0 0.5 1−1.5

−1

−0.5

0

0.5

1

1.5

2

x

s(x;

µ)

Plot of Approximate Functions (dim = 10) with Exact Functions (in black solid line)

µ= 1 µ= 1.1713 µ= 1.3855 µ= 3.1416

Figure: Approximate Function from EIM with 10 POD basisSaifon Chaturantabut POD


Plots of Numerical Solutions from 3Approaches

Direct POD

Precomputed POD

EIM-POD

0 10 20 30 40 50 60 7010−15

10−10

10−5

100 Singular values of the Snapshots

Figure: SVD

0 0.2 0.4 0.6 0.8 10

0.5

1

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

xy(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

Figure: Direct POD



0 0.2 0.4 0.6 0.8 10

0.5

1

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

xy(x

,t)

Sol of Reduced System (Precompute POD):dim = 1

t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

Figure: Precomputed POD

0 0.2 0.4 0.6 0.8 10

0.5

1

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)

Sol of Reduced System (EIM−POD):dim = 1

t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

xy(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

0 0.2 0.4 0.6 0.8 10

0.20.40.60.8

x

y(x

,t)


t= 0t= 0.2029t= 0.98551t= 1.2174t= 2

Figure: EIM-POD



Conclusions

Unique Contribution: Cleardescription of the EIM⇒ SuccessfulImplementation of EIM with POD

EIM is comparable to widely acceptedmethods, such as precomputingtechnique

The results suggest that EIM withPOD basis is a promising modelreduction technique for more generalnonlinear PDEs.

Future WorkExtend to higher dimensions

Extend to PDEs with more generalnonlinearities

Apply to practical problem, e.g.optimal control problem

0 10 20 30 40 50 60 7010−8

10−6

10−4

10−2

100

102


Aver

age

Rel

ativ

e Er

ror

Relative Error:=(1/nt)sumi=1n

t (||yFE(t)−y(t)||/||yFE(t)||)2 of Reconstructed Snapshots using different dim of basis from EIM−POD

EIM−POD dim=1EIM−POD dim=2EIM−POD dim=4EIM−POD dim=6EIM−POD dim=10EIM−POD dim=20Direct PODPrecompute POD

0 10 20 30 40 50 60 700

0.2

0.4

0.6

0.8

1

1.2

1.4


CP

U ti

me

(sec

)

CPU TIME for solving reduced−order ODE system

EIM−POD dim=1EIM−POD dim=2EIM−POD dim=4EIM−POD dim=6EIM−POD dim=10EIM−POD dim=20Direct PODPrecompute POD

Figure: Plots of the first 4 POD basisand the space of solutions



Overview

Nonlinear PDE

⇓ L99 FE-Galerkin

FULL Discretized System (ODE)dim = N

⇓ L99 Solve ODE

SNAPSHOTS: y(x, t`)ns`=1

⇓ L99 POD

POD basis: φiki=1

⇓ L99 POD-Galerkin

REDUCED Discretized System (ODE)Linear Term: dim = k N

Non-Linear Term: dim ∼ N −→ −→(i) Memory Saving

(ii) Accuracy

⇓ L99

Nonlinear Approx-EIM

-Precomputing

REDUCED Discretized System (ODE)Linear Term: dim = k N

Non-Linear Term: dim = nm N −→ −→(iii) Efficiency(Time Saving)


Documents

Proper Orthogonal Decomposition (POD) - Rice Ucaam699-2/slides_POD_699fall2008.pdf · IntroPODResultsPOD Problemextras Proper Orthogonal Decomposition (POD) Saifon Chaturantabut Advisor: