87
A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data Pawan Goyal Joint work with Peter Benner (MPI, Magdeburg) Benjamin Peherstorfer (CIMS, NY University, NY) Workshop on Mathematics of Reduced Order Models, ICERM, Brown University, Providence, USA February 17-21, 2020

A Rank Minimization Approach to Learning Dynamical Systems

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: A Rank Minimization Approach to Learning Dynamical Systems

A Rank Minimization Approach to LearningDynamical Systems from Frequency Response Data

Pawan GoyalJoint work with

Peter Benner (MPI, Magdeburg)Benjamin Peherstorfer (CIMS, NY University, NY)

Workshop on Mathematics of Reduced Order Models,

ICERM, Brown University, Providence, USA

February 17-21, 2020

Page 2: A Rank Minimization Approach to Learning Dynamical Systems

Table of Contents

1. Introduction

2. Data-driven Identification

3. Rank Minimization Problems

4. Numerical Examples

5. Measurement Noise

6. Conclusions

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 2/29

Page 3: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Dynamical systems

Ex(t) = f(t,x(t),u(t))), x(0) = 0,

y(t) = Cx(t),where

(generalized) states x(t) ∈ Rn (invertible E ∈ Rn×n),

inputs (controls) u(t) ∈ Rm,

outputs (measurements, quantity of interest) y(t) ∈ Rq.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 3/29

Page 4: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Dynamical systems

Ex(t) = f(t,x(t),u(t))), x(0) = 0,

y(t) = Cx(t),where

(generalized) states x(t) ∈ Rn (invertible E ∈ Rn×n),

inputs (controls) u(t) ∈ Rm,

outputs (measurements, quantity of interest) y(t) ∈ Rq.

System Classes

Classical linear systems: f(x) := Ax(t) + Bu(t),

Delay systems: f(x) := Ax(t) + Aτx(t− τ) + Bu(t),

Second-order system f(x) := Ax(t) + A1

∫ t

0

x(τ)dτ +

∫ t

0

Bu(τ)τ , . . . ,

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 3/29

Page 5: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Dynamical systems

Ex(t) = f(t,x(t),u(t))), x(0) = 0,

y(t) = Cx(t),where

(generalized) states x(t) ∈ Rn (invertible E ∈ Rn×n),

inputs (controls) u(t) ∈ Rm,

outputs (measurements, quantity of interest) y(t) ∈ Rq.

Frequency domain representation

Using Laplace transform, we can convert time-domain representation into the frequency domain.

This yields Y(s) = H(s)U(s).

Hence, H(s), called as transfer function is known, we can write the output of a system for anygiven input.

Moreover, H(s), the transfer function of a system, completely characterize the dynamics.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 3/29

Page 6: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Dynamical systems

Ex(t) = f(t,x(t),u(t))), x(0) = 0,

y(t) = Cx(t),where

(generalized) states x(t) ∈ Rn (invertible E ∈ Rn×n),

inputs (controls) u(t) ∈ Rm,

outputs (measurements, quantity of interest) y(t) ∈ Rq.

Frequency domain representation

Using Laplace transform, we can convert time-domain representation into the frequency domain.

This yields Y(s) = H(s)U(s).

Hence, H(s), called as transfer function is known, we can write the output of a system for anygiven input.

Moreover, H(s), the transfer function of a system, completely characterize the dynamics.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 3/29

Page 7: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Dynamical systems

Ex(t) = f(t,x(t),u(t))), x(0) = 0,

y(t) = Cx(t),where

(generalized) states x(t) ∈ Rn (invertible E ∈ Rn×n),

inputs (controls) u(t) ∈ Rm,

outputs (measurements, quantity of interest) y(t) ∈ Rq.

Frequency domain representation

Using Laplace transform, we can convert time-domain representation into the frequency domain.

This yields Y(s) = H(s)U(s).

Hence, H(s), called as transfer function is known, we can write the output of a system for anygiven input.

Moreover, H(s), the transfer function of a system, completely characterize the dynamics.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 3/29

Page 8: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Dynamical systems

Ex(t) = f(t,x(t),u(t))), x(0) = 0,

y(t) = Cx(t),where

(generalized) states x(t) ∈ Rn (invertible E ∈ Rn×n),

inputs (controls) u(t) ∈ Rm,

outputs (measurements, quantity of interest) y(t) ∈ Rq.

Frequency domain representation

Using Laplace transform, we can convert time-domain representation into the frequency domain.

This yields Y(s) = H(s)U(s).

Hence, H(s), called as transfer function is known, we can write the output of a system for anygiven input.

Moreover, H(s), the transfer function of a system, completely characterize the dynamics.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 3/29

Page 9: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Ex(t) = Ax(t) +Bu(t),

y(t) = Cx(t).

Linear system (standard)

Mx(t) +Dx(t) +Kx(t) = Bu(t),

y(t) = Cx(t).

Second-order system

Ex(t) = Ax(t) +Aτx(t−τ) +Bu(t),

y(t) = Cx(t).

Delay system

Ex(t) = Ax(t) +Aτ

∫ t

0x(τ)dτ +Bu(t),

y(t) = Cx(t).

Integro system

G(s) = C(sE−A)−1B.

Linear system (standard)Time 7→ Frequency

G(s) = C(s2M+ sD+K)−1B.

Second-order systemTime 7→ Frequency

G(s) = C(sE−A−Aτ e

−sτ )−1B.

Delay systemTime 7→ Frequency

G(s) = C

(sE−A−

1

sAτ

)−1

B.

Integro systemTime 7→ Frequency

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 4/29

Page 10: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Ex(t) = Ax(t) +Bu(t),

y(t) = Cx(t).

Linear system (standard)

Mx(t) +Dx(t) +Kx(t) = Bu(t),

y(t) = Cx(t).

Second-order system

Ex(t) = Ax(t) +Aτx(t−τ) +Bu(t),

y(t) = Cx(t).

Delay system

Ex(t) = Ax(t) +Aτ

∫ t

0x(τ)dτ +Bu(t),

y(t) = Cx(t).

Integro system

G(s) = C(sE−A)−1B.

Linear system (standard)Time 7→ Frequency

G(s) = C(s2M+ sD+K)−1B.

Second-order systemTime 7→ Frequency

G(s) = C(sE−A−Aτ e

−sτ )−1B.

Delay systemTime 7→ Frequency

G(s) = C

(sE−A−

1

sAτ

)−1

B.

Integro systemTime 7→ Frequency

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 4/29

Page 11: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Ex(t) = Ax(t) +Bu(t),

y(t) = Cx(t).

Linear system (standard)

Mx(t) +Dx(t) +Kx(t) = Bu(t),

y(t) = Cx(t).

Second-order system

Ex(t) = Ax(t) +Aτx(t−τ) +Bu(t),

y(t) = Cx(t).

Delay system

Ex(t) = Ax(t) +Aτ

∫ t

0x(τ)dτ +Bu(t),

y(t) = Cx(t).

Integro system

G(s) = C(sE−A)−1B.

Linear system (standard)Time 7→ Frequency

G(s) = C(s2M+ sD+K)−1B.

Second-order systemTime 7→ Frequency

G(s) = C(sE−A−Aτ e

−sτ )−1B.

Delay systemTime 7→ Frequency

G(s) = C

(sE−A−

1

sAτ

)−1

B.

Integro systemTime 7→ Frequency

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 4/29

Page 12: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionLinear Systems

Ex(t) = Ax(t) +Bu(t),

y(t) = Cx(t).

Linear system (standard)

Mx(t) +Dx(t) +Kx(t) = Bu(t),

y(t) = Cx(t).

Second-order system

Ex(t) = Ax(t) +Aτx(t−τ) +Bu(t),

y(t) = Cx(t).

Delay system

Ex(t) = Ax(t) +Aτ

∫ t

0x(τ)dτ +Bu(t),

y(t) = Cx(t).

Integro system

G(s) = C(sE−A)−1B.

Linear system (standard)Time 7→ Frequency

G(s) = C(s2M+ sD+K)−1B.

Second-order systemTime 7→ Frequency

G(s) = C(sE−A−Aτ e

−sτ )−1B.

Delay systemTime 7→ Frequency

G(s) = C

(sE−A−

1

sAτ

)−1

B.

Integro systemTime 7→ Frequency

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 4/29

Page 13: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionHow to measure the transfer function

Excite the system

Very useful when system parameters arenot known.

Modeling is done using a proprietarysoftware

not so easy to get system matrices

However, we can obtain transfer functionevaluation much easier

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 5/29

Page 14: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionHow to measure the transfer function

Excite the system

Very useful when system parameters arenot known.

Modeling is done using a proprietarysoftware

not so easy to get system matrices

However, we can obtain transfer functionevaluation much easier

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 5/29

Page 15: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionGoal

Goal of the talk

Build a linear model M such that

(a) it interpolates given transfer function measurements, i.e., HM(jωi) = vi,

(b) the model has the structure, given by engineering experts, e.g. second-order, delay, fractional, etc.

−4 −2 0 2 4

0

1

2

3

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 6/29

Page 16: A Rank Minimization Approach to Learning Dynamical Systems

IntroductionGoal

Goal of the talk

Build a linear model M such that

(a) it interpolates given transfer function measurements, i.e., HM(jωi) = vi,

(b) the model has the structure, given by engineering experts, e.g. second-order, delay, fractional, etc.

−4 −2 0 2 4

0

1

2

3

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 6/29

Page 17: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationAlready talks along these lines

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 7/29

Page 18: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationAlready talks along these lines

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 7/29

Page 19: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationAlready talks along these lines

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 7/29

Page 20: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationAlready talks along these lines

Rational interpolation problem

Given interpolation points σ1, . . . ,σ2l ⊂ C and sample values γ1, . . . , γ2l ⊂ C, construct a rational

function H(s) = C (sE−A)−1

B, satisfying

H(σj) = γj , j = 1, . . . , 2l

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 8/29

Page 21: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Let us recall the Loewner framework in the single-input single-output case.

Data set as

interpolation points σk ∈ C,

sample values γk ∈ C,for k = 1, . . . , 2l.

Partition the data into the left & right sets:

(σk, γk) = (µi,vi) ∪ (λi,wi), k = 1, . . . , 2l, i = 1, . . . , l.

Objective

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Let us organize the data as follows:

Interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . , vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 9/29

Page 22: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Let us recall the Loewner framework in the single-input single-output case.

Data set as

interpolation points σk ∈ C,

sample values γk ∈ C,for k = 1, . . . , 2l.

Partition the data into the left & right sets:

(σk, γk) = (µi,vi) ∪ (λi,wi), k = 1, . . . , 2l, i = 1, . . . , l.

Objective

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Let us organize the data as follows:

Interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . , vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 9/29

Page 23: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Let us recall the Loewner framework in the single-input single-output case.

Data set as

interpolation points σk ∈ C,

sample values γk ∈ C,for k = 1, . . . , 2l.

Partition the data into the left & right sets:

(σk, γk) = (µi,vi) ∪ (λi,wi), k = 1, . . . , 2l, i = 1, . . . , l.

Objective

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Let us organize the data as follows:

Interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . , vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 9/29

Page 24: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Let us recall the Loewner framework in the single-input single-output case.

Data set as

interpolation points σk ∈ C,

sample values γk ∈ C,for k = 1, . . . , 2l.

Partition the data into the left & right sets:

(σk, γk) = (µi,vi) ∪ (λi,wi), k = 1, . . . , 2l, i = 1, . . . , l.

Objective

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Let us organize the data as follows:

Interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . , vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 9/29

Page 25: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Let us recall the Loewner framework in the single-input single-output case.

Data set as

interpolation points σk ∈ C,

sample values γk ∈ C,for k = 1, . . . , 2l.

Partition the data into the left & right sets:

(σk, γk) = (µi,vi) ∪ (λi,wi), k = 1, . . . , 2l, i = 1, . . . , l.

Objective

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Let us organize the data as follows:

Interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . , vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 9/29

Page 26: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Loewner Approach (Matrix form)

Let L and Lσ satisfy:

−LΛ + Lσ = V1T ,

−LTΩ + LσT = W1T ,

The rational function H(s) = C(sE−A)−1B interpolates the data, where

E = −L, A = −Lσ, B = V, and C = W,

and the pencil (L, Lσ) is regular.

Remarks

No need to solve Sylvester equations ⇒ matrices L and Lσ have analytic expressions.

rank (L) = order of minimal realization = r.

Hence, a compression step using SVD of L and Lσ can be performed to obtain a minimal or approximate.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 10/29

Page 27: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Loewner Approach (Matrix form)

Let L and Lσ satisfy:

−LΛ + Lσ = V1T ,

−LTΩ + LσT = W1T ,

The rational function H(s) = C(sE−A)−1B interpolates the data, where

E = −L, A = −Lσ, B = V, and C = W,

and the pencil (L, Lσ) is regular.

Remarks

No need to solve Sylvester equations ⇒ matrices L and Lσ have analytic expressions.

rank (L) = order of minimal realization = r.

Hence, a compression step using SVD of L and Lσ can be performed to obtain a minimal or approximate.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 10/29

Page 28: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Loewner Approach (Matrix form)

Let L and Lσ satisfy:

−LΛ + Lσ = V1T ,

−LTΩ + LσT = W1T ,

The rational function H(s) = C(sE−A)−1B interpolates the data, where

E = −L, A = −Lσ, B = V, and C = W,

and the pencil (L, Lσ) is regular.

Remarks

No need to solve Sylvester equations ⇒ matrices L and Lσ have analytic expressions.

rank (L) = order of minimal realization = r.

Hence, a compression step using SVD of L and Lσ can be performed to obtain a minimal or approximate.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 10/29

Page 29: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Loewner Approach (Matrix form)

Let L and Lσ satisfy:

−LΛ + Lσ = V1T ,

−LTΩ + LσT = W1T ,

The rational function H(s) = C(sE−A)−1B interpolates the data, where

E = −L, A = −Lσ, B = V, and C = W,

and the pencil (L, Lσ) is regular.

Remarks

No need to solve Sylvester equations ⇒ matrices L and Lσ have analytic expressions.

rank (L) = order of minimal realization = r.

Hence, a compression step using SVD of L and Lσ can be performed to obtain a minimal or approximate.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 10/29

Page 30: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationLoewner framework

Loewner Approach (Matrix form)

Let L and Lσ satisfy:

−LΛ + Lσ = V1T ,

−LTΩ + LσT = W1T ,

The rational function H(s) = C(sE−A)−1B interpolates the data, where

E = −L, A = −Lσ, B = V, and C = W,

and the pencil (L, Lσ) is regular.

Remarks

No need to solve Sylvester equations ⇒ matrices L and Lσ have analytic expressions.

rank (L) = order of minimal realization = r.

Hence, a compression step using SVD of L and Lσ can be performed to obtain a minimal or approximate.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 10/29

Page 31: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

Objective: rational functions

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Objective: structured (non-)rational functions

Find a (non-)rational function Hnr(s) = C (f1(s)A1 + f2(s)A2 + f3(s)A3)−1

B such that

Hnr(µj) = vj , Hnr(λi) = wi.

Fit into second-order systems, delay systems, fractional systems, etc.Once again, we organize the data as follows:

interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . ,vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 11/29

Page 32: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

Objective: rational functions

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Objective: structured (non-)rational functions

Find a (non-)rational function Hnr(s) = C (f1(s)A1 + f2(s)A2 + f3(s)A3)−1

B such that

Hnr(µj) = vj , Hnr(λi) = wi.

Fit into second-order systems, delay systems, fractional systems, etc.

Once again, we organize the data as follows:

interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . ,vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 11/29

Page 33: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

Objective: rational functions

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Objective: structured (non-)rational functions

Find a (non-)rational function Hnr(s) = C (f1(s)A1 + f2(s)A2 + f3(s)A3)−1

B such that

Hnr(µj) = vj , Hnr(λi) = wi.

Fit into second-order systems, delay systems, fractional systems, etc.For instance, second-order systems: f1(s) = s2, f2(s) = s, and f3(s) = 1.

Once again, we organizethe data as follows:

interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . ,vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 11/29

Page 34: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

Objective: rational functions

Find a rational function H(s) = C (sE−A)−1

B such that

H(µj) = vj , H(λi) = wi.

Objective: structured (non-)rational functions

Find a (non-)rational function Hnr(s) = C (f1(s)A1 + f2(s)A2 + f3(s)A3)−1

B such that

Hnr(µj) = vj , Hnr(λi) = wi.

Fit into second-order systems, delay systems, fractional systems, etc.Once again, we organize the data as follows:

interpolation points : Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Sample values : V =[v1, . . . ,vl

]T, W =

[w1, . . . ,wl

]T.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 11/29

Page 35: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

Identification of structured systems [Unger/Schulze/Beattie/Gugercin ’16]

Let us say the matrices A1, A2, and A3 satisfy:

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

where FΛi = diag (fi(λ1), . . . , fi(λl)) , FΩ

i = diag (fi(µ1), . . . , fi(µl)) , and V and W are vectors,containing measurements.

The function Hnr(s) = WT (f1(s)A1 + f2(s)A2 + f3(s)A3)−1V interpolates the data, i.e.,

Hnr(λi) = wi, Hnr(µi) = vi,

assuming [A1,A2,A3] is of row full-rank. If it is not full-rank, a compression step can be performed.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 12/29

Page 36: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

Identification of structured systems [Unger/Schulze/Beattie/Gugercin ’16]

Let us say the matrices A1, A2, and A3 satisfy:

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

where FΛi = diag (fi(λ1), . . . , fi(λl)) , FΩ

i = diag (fi(µ1), . . . , fi(µl)) , and V and W are vectors,containing measurements.

The function Hnr(s) = WT (f1(s)A1 + f2(s)A2 + f3(s)A3)−1V interpolates the data, i.e.,

Hnr(λi) = wi, Hnr(µi) = vi,

assuming [A1,A2,A3] is of row full-rank. If it is not full-rank, a compression step can be performed.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 12/29

Page 37: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 38: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 39: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 40: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

Remark: [Unger ’16] have tried to enforce additional con-straints/conditions to utilize extra variables.

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 41: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 42: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 43: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationStructured Linear Systems

RealizingH = C(sE−A)−1B

Number of equations: 2l2

Number of variables: 2l2

Unique solution

Analytical expression (divide anddifference)

RealizingH = C(f1(s)A1 + f2(s)A2 + f3(s)A3)−1B

Number of equations: 2l2

Number of variables: 3l2

Infinitely many solutions

No such thing

In dynamical systems, simplicity can be defined as “minimal order systems, describing the dynamics, orinterpolating the data ”.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 13/29

Page 44: A Rank Minimization Approach to Learning Dynamical Systems

Data-driven IdentificationHow to put together

Identification of structured systems [Unger/Schulze/Beattie/Gugercin ’16]

Let us say the matrices A1, A2, and A3 satisfy:

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

where FΛi = diag (fi(λ1), . . . , fi(λl)) , FΩ

i = diag (fi(µ1), . . . , fi(µl)) ,

A result [Benner/G./Pontes ’19]

rank([A1,A2,A3

])= minimum order of a realization that interpolates the data.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 14/29

Page 45: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsOptimization Formulation

A rank-minimization Problem Formulation

minA1,A2,A3

rank([A1,A2,A3

])subject to

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

where FΛi = diag (fi(λ1), . . . , fi(λl)) , FΩ

i = diag (fi(µ1), . . . , fi(µl)) , and V and W are vectors,containing measurements.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 15/29

Page 46: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsIllustration

A delay example

Consider a delay system whose transfer function is: H(s) = (s+ 1− 0.25e−s)−1.

Take four distinct measurements: H(σ1),H(σ2),H(µ1),H(µ2)

Necessary Conditions for interpolation [Unger et. al ’16]

E [ σ1σ2

] + A [ 11 ] + Aτ

[e−σ1

e−σ2

]=[H(µ1)H(µ2)

]1T =: B1T ,

ET [ µ1µ2 ] + AT [ 1

1 ] + ATτ

[e−µ1

e−µ2

]=[H(σ1)H(σ2)

]1T =: CT 1T ,

Every triplet (E,A,Aτ ), satisfying the above equations, interpolates the data.

Infinite possibility since 3 · 4 variables and 2 · 4 equations.

For Aτ = 0, it yields a rational function, obtained by the Loewner approach, of order r = 2.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 16/29

Page 47: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsIllustration

A delay example

Consider a delay system whose transfer function: H(s) = (s+ 1− 0.25e−s)−1.

Take four distinct measurements: H(σ1),H(σ2),H(µ1),H(µ2)

Identification (inverse) problem

Given measurements, identify a delay model that interpolates the measurement.In other words, construct a state-space model:

Ex(t) = Ax(t) + Aτx(t− 1) + Bu(t), y(t) = Cx(t)

such that HIden(σi) = H(σi) and HIden(µi) = H(µi), i ∈ 1, 2.

Necessary Conditions for interpolation [Unger et. al ’16]

E [ σ1σ2

] + A [ 11 ] + Aτ

[e−σ1

e−σ2

]=[H(µ1)H(µ2)

]1T =: B1T ,

ET [ µ1µ2 ] + AT [ 1

1 ] + ATτ

[e−µ1

e−µ2

]=[H(σ1)H(σ2)

]1T =: CT 1T ,

Every triplet (E,A,Aτ ), satisfying the above equations, interpolates the data.

Infinite possibility since 3 · 4 variables and 2 · 4 equations.

For Aτ = 0, it yields a rational function, obtained by the Loewner approach, of order r = 2.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 16/29

Page 48: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsIllustration

Identification (inverse) problem

Given measurements, identify a delay model that interpolates the measurement.In other words, construct a state-space model:

Ex(t) = Ax(t) + Aτx(t− 1) + Bu(t), y(t) = Cx(t)

such that HIden(σi) = H(σi) and HIden(µi) = H(µi), i ∈ 1, 2.

Necessary Conditions for interpolation [Unger et. al ’16]

E [ σ1σ2

] + A [ 11 ] + Aτ

[e−σ1

e−σ2

]=[H(µ1)H(µ2)

]1T =: B1T ,

ET [ µ1µ2 ] + AT [ 1

1 ] + ATτ

[e−µ1

e−µ2

]=[H(σ1)H(σ2)

]1T =: CT 1T ,

Every triplet (E,A,Aτ ), satisfying the above equations, interpolates the data.

Infinite possibility since 3 · 4 variables and 2 · 4 equations.

For Aτ = 0, it yields a rational function, obtained by the Loewner approach, of order r = 2.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 16/29

Page 49: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsIllustration

Identification (inverse) problem

Given measurements, identify a delay model that interpolates the measurement.In other words, construct a state-space model:

Ex(t) = Ax(t) + Aτx(t− 1) + Bu(t), y(t) = Cx(t)

such that HIden(σi) = H(σi) and HIden(µi) = H(µi), i ∈ 1, 2.

Necessary Conditions for interpolation [Unger et. al ’16]

E [ σ1σ2

] + A [ 11 ] + Aτ

[e−σ1

e−σ2

]=[H(µ1)H(µ2)

]1T =: B1T ,

ET [ µ1µ2 ] + AT [ 1

1 ] + ATτ

[e−µ1

e−µ2

]=[H(σ1)H(σ2)

]1T =: CT 1T ,

Every triplet (E,A,Aτ ), satisfying the above equations, interpolates the data.

Infinite possibility since 3 · 4 variables and 2 · 4 equations.

For Aτ = 0, it yields a rational function, obtained by the Loewner approach, of order r = 2.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 16/29

Page 50: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsUsing a rank-minimization approach

We seek among infinite triplets that minimize as follows:

rank([E,A,Aτ

]),

satisfying

E [ σ1σ2

] + A [ 11 ] + Aτ

[e−σ1

e−σ2

]=[H(µ1)H(µ2)

]1T =: B1T ,

ET [ µ1µ2 ] + AT [ 1

1 ] + ATτ

[e−µ1

e−µ2

]=[H(σ1)H(σ2)

]1T =: CT 1T .

If we solve, then we get

E =[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

A = −[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

Aτ = 0.25[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

].

Note that the rank of[E,A,Aτ

]= 1.

Hence, using a compression step, we can obtain the same transfer function.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 17/29

Page 51: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsUsing a rank-minimization approach

We seek among infinite triplets that minimize as follows:

rank([E,A,Aτ

]),

If we solve, then we get

E =[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

A = −[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

Aτ = 0.25[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

].

Note that the rank of[E,A,Aτ

]= 1.

Hence, using a compression step, we can obtain the same transfer function.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 17/29

Page 52: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsUsing a rank-minimization approach

We seek among infinite triplets that minimize as follows:

rank([E,A,Aτ

]),

If we solve, then we get

E =[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

A = −[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

Aτ = 0.25[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

].

Note that the rank of[E,A,Aτ

]= 1.

Hence, using a compression step, we can obtain the same transfer function.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 17/29

Page 53: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsUsing a rank-minimization approach

We seek among infinite triplets that minimize as follows:

rank([E,A,Aτ

]),

If we solve, then we get

E =[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

A = −[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

],

Aτ = 0.25[H(µ1)

H(µ2)

] [1 11 1

] [H(σ1)

H(σ2)

].

Note that the rank of[E,A,Aτ

]= 1.

Hence, using a compression step, we can obtain the same transfer function.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 17/29

Page 54: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsChallenges

We have set-up a possible ideal problem to identify low-order models.

But rank-minimization problems are NP hard ones, and there is little hope of finding the globalminimum efficiently in all instances.

A general rank-minimization formulation

minX

rank (X) subject to A vec (X) = b

A rank-minimization Problem Formulation

minA1,A2,A3

rank([A1,A2,A3

])subject to

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 18/29

Page 55: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsChallenges

We have set-up a possible ideal problem to identify low-order models.But rank-minimization problems are NP hard ones, and there is little hope of finding the globalminimum efficiently in all instances.

A general rank-minimization formulation

minX

rank (X) subject to A vec (X) = b

A rank-minimization Problem Formulation

minA1,A2,A3

rank([A1,A2,A3

])subject to

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 18/29

Page 56: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsChallenges

We have set-up a possible ideal problem to identify low-order models.But rank-minimization problems are NP hard ones, and there is little hope of finding the globalminimum efficiently in all instances.

A general rank-minimization formulation

minX

rank (X) subject to A vec (X) = b

A rank-minimization Problem Formulation

minA1,A2,A3

rank([A1,A2,A3

])subject to

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 18/29

Page 57: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsChallenges

We have set-up a possible ideal problem to identify low-order models.But rank-minimization problems are NP hard ones, and there is little hope of finding the globalminimum efficiently in all instances.

A general rank-minimization formulation

minX

rank (X) subject to A vec (X) = b

A rank-minimization Problem Formulation

minA1,A2,A3

rank([A1,A2,A3

])subject to

A1FΛ1 + A2F

Λ2 + A3F

Λ3 = V1T ,

A1TFΩ

1 + A2TFΩ

2 + A3TFΩ

3 = W1T ,

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 18/29

Page 58: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsRelaxation

minX

rank (X) subject to A vec (X) = b (2)

Relaxed: minX

∑i g(σi) subject to A vec (X) = b

Observe: rank (X) = ‖σ(X)‖l0 , where σ(X) =[σ1, . . . ,σn

].

What we are going to look at, instead, efficient heuristics.

‖σ(X)‖l0 → ‖σ(X)‖l1 := ‖X‖∗ (nuclear norm of X).– the best convex relaxation. [Fazel ’02]

‖σ(X)‖l0 →∑i(σi)

p

– concave function but better approximation of cardinality.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 19/29

Page 59: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsRelaxation

minX

rank (X) subject to A vec (X) = b (2)

Relaxed: minX

∑i g(σi) subject to A vec (X) = b

Observe: rank (X) = ‖σ(X)‖l0 , where σ(X) =[σ1, . . . ,σn

].

What we are going to look at, instead, efficient heuristics.

‖σ(X)‖l0 → ‖σ(X)‖l1 := ‖X‖∗ (nuclear norm of X).– the best convex relaxation. [Fazel ’02]

‖σ(X)‖l0 →∑i(σi)

p

– concave function but better approximation of cardinality.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 19/29

Page 60: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsRelaxation

minX

rank (X) subject to A vec (X) = b (2)

Relaxed: minX

∑i g(σi) subject to A vec (X) = b

Observe: rank (X) = ‖σ(X)‖l0 , where σ(X) =[σ1, . . . ,σn

].

What we are going to look at, instead, efficient heuristics.

‖σ(X)‖l0 → ‖σ(X)‖l1 := ‖X‖∗ (nuclear norm of X).– the best convex relaxation. [Fazel ’02]

‖σ(X)‖l0 →∑i(σi)

p

– concave function but better approximation of cardinality.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 19/29

Page 61: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsRelaxation

minX

rank (X) subject to A vec (X) = b (2)

Relaxed: minX

∑i g(σi) subject to A vec (X) = b

Observe: rank (X) = ‖σ(X)‖l0 , where σ(X) =[σ1, . . . ,σn

].

What we are going to look at, instead, efficient heuristics.

‖σ(X)‖l0 → ‖σ(X)‖l1 := ‖X‖∗ (nuclear norm of X).– the best convex relaxation. [Fazel ’02]

‖σ(X)‖l0 →∑i(σi)

p

– concave function but better approximation of cardinality.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 19/29

Page 62: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsRelaxation

minX

rank (X) subject to A vec (X) = b (2)

Relaxed: minX

∑i g(σi) subject to A vec (X) = b

Observe: rank (X) = ‖σ(X)‖l0 , where σ(X) =[σ1, . . . ,σn

].

What we are going to look at, instead, efficient heuristics.

‖σ(X)‖l0 → ‖σ(X)‖l1 := ‖X‖∗ (nuclear norm of X).– the best convex relaxation. [Fazel ’02]

‖σ(X)‖l0 →∑i(σi)

p

– concave function but better approximation of cardinality.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 19/29

Page 63: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsSingular value thresholding

An ideal problem

minX rank (X) subject to A vec (X) = b

Relaxation

minX

∑i g(σi) subject to A vec (X) = b

If g(σi) = σi:

minX ‖X‖∗ subject to A vec (X) = b

Note that similar iterations are possible for a concave heuristics of the rank function, i.e.,g(σi) = σ0.5

i .

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 20/29

Page 64: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsSingular value thresholding

An ideal problem

minX rank (X) subject to A vec (X) = b

Relaxation

minX

∑i g(σi) subject to A vec (X) = b

If g(σi) = σi:

minX ‖X‖∗ subject to A vec (X) = b

Note that similar iterations are possible for a concave heuristics of the rank function, i.e.,g(σi) = σ0.5

i .

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 20/29

Page 65: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsSingular value thresholding

An ideal problem

minX rank (X) subject to A vec (X) = b

Relaxation

minX

∑i g(σi) subject to A vec (X) = b

If g(σi) = σi:

minX ‖X‖∗ subject to A vec (X) = b

Shrinkage Operator

Let M be a matrix and its SVD be M = UΣV∗ with Σ = diag (σ1, . . . ,σn). The shrinkage operatorDτ is defined as

Dτ (M) = UDτ (Σ)V∗, Dτ (Σ) = diag ((σ1 − τ)+, . . . , (σn − τ)+) ,

where t+ = max(t, 0).

Note that similar iterations are possible for a concave heuristics of the rank function, i.e.,g(σi) = σ0.5

i .

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 20/29

Page 66: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsSingular value thresholding

An ideal problem

minX rank (X) subject to A vec (X) = b

Relaxation

minX

∑i g(σi) subject to A vec (X) = b

If g(σi) = σi:

minX ‖X‖∗ subject to A vec (X) = b

Shrinkage Operator

Let M be a matrix and its SVD be M = UΣV∗ with Σ = diag (σ1, . . . ,σn). The shrinkage operatorDτ is defined as

Dτ (M) = UDτ (Σ)V∗, Dτ (Σ) = diag ((σ1 − τ)+, . . . , (σn − τ)+) ,

where t+ = max(t, 0).

Note that similar iterations are possible for a concave heuristics of the rank function, i.e.,g(σi) = σ0.5

i .

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 20/29

Page 67: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsSingular value thresholding

An ideal problem

minX rank (X) subject to A vec (X) = b

Relaxation

minX

∑i g(σi) subject to A vec (X) = b

If g(σi) = σi:

minX ‖X‖∗ subject to A vec (X) = b

Uzawa’s iterations [Cai/Candes/Shen ’10]

Optimal solution is given byz = AT (yk−1);Z = reshape(z,n, 3n) n : number of data pointsXk = Dτ (AT (yk−1)))yk = yk−1 + δk

(b−A vec

(XK

))

Note that similar iterations are possible for a concave heuristics of the rank function, i.e.,g(σi) = σ0.5

i .

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 20/29

Page 68: A Rank Minimization Approach to Learning Dynamical Systems

Rank Minimization ProblemsSingular value thresholding

An ideal problem

minX rank (X) subject to A vec (X) = b

Relaxation

minX

∑i g(σi) subject to A vec (X) = b

If g(σi) = σi:

minX ‖X‖∗ subject to A vec (X) = b

Uzawa’s iterations [Cai/Candes/Shen ’10]

Optimal solution is given byz = AT (yk−1);Z = reshape(z,n, 3n) n : number of data pointsXk = Dτ (AT (yk−1)))yk = yk−1 + δk

(b−A vec

(XK

))Note that similar iterations are possible for a concave heuristics of the rank function, i.e.,g(σi) = σ0.5

i .

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 20/29

Page 69: A Rank Minimization Approach to Learning Dynamical Systems

Numerical ExamplesFishtail example

A second-order example: Fishtail example (n = 779, 232)

Consider 296 points and look for a minimal order second-order system that fits the data.

Mx(t) + Dx(t) + Kx(t) = Bu(t), y(t) = Cx(t).

10−2 10−1 100 101 102 103 104 10510−2

10−1

100

101

freq

‖H(s

)‖

training data

10−2 10−1 100 101 102 103 104 10510−2

10−1

100

101

freq

‖H(s

)‖

training data

Rank-based (r = 20)

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 21/29

Page 70: A Rank Minimization Approach to Learning Dynamical Systems

Numerical ExamplesFishtail example

A second-order example: Fishtail example (n = 779, 232)

Consider 296 points and look for a minimal order second-order system that fits the data.

10−2 10−1 100 101 102 103 104 10510−2

10−1

100

101

freq

‖H(s

)‖

training data

Rank-based (r = 20)

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 21/29

Page 71: A Rank Minimization Approach to Learning Dynamical Systems

Numerical ExamplesA delay example

A delay example

Consider 40 points and look for a minimal order delay system that fits the data.

Ex(t) = Ax(t) + Aτx(t− 1) + Bu(t), y(t) = Cx(t).

10−1 100 101 102 10310−3

10−2

10−1

100

101

freq

‖H(s

)‖

Rank-based (r = 5)

Orig (n = 5)training data

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 22/29

Page 72: A Rank Minimization Approach to Learning Dynamical Systems

Numerical ExamplesA delay example

A delay example

Consider 40 points and look for a minimal order delay system that fits the data.

10−1 100 101 102 10310−3

10−2

10−1

100

101

freq

‖H(s

)‖

Rank-based (r = 5)

Orig (n = 5)training data

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 22/29

Page 73: A Rank Minimization Approach to Learning Dynamical Systems

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 23/29

Page 74: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Data obtained using sensors or in a lab are corrupted with noise.

Hence, data-based algorithm should be robust with respect to noise.

Consider 10 data points

10−1 100 101 102 10310−3

10−2

10−1

100

freq

‖H‖

training data

Let us apply theLoewner method tohave a rational function(E,A,B,C).

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 24/29

Page 75: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Data obtained using sensors or in a lab are corrupted with noise.

Hence, data-based algorithm should be robust with respect to noise.

Consider 10 data points

10−1 100 101 102 10310−3

10−2

10−1

100

freq

‖H‖

training data

Let us apply theLoewner method tohave a rational function(E,A,B,C).

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 24/29

Page 76: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Data obtained using sensors or in a lab are corrupted with noise.

Hence, data-based algorithm should be robust with respect to noise.

Consider 10 data points

10−1 100 101 102 10310−3

10−2

10−1

100

freq

‖H‖

training data

Let us apply theLoewner method tohave a rational function(E,A,B,C).

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 24/29

Page 77: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Observe the decay of singular values of the Loewner pencil.

2 4 6 8 1010−20

10−14

10−8

10−2

We obtain a 3rd order system, interpolating the data.

E =

1 0 00 1 00 0 1

,

A =

−1 100 0−100 −1 00 0 −3

B =

−1.5461−0.7585−1.1096

C =

−0.8456−0.5727−0.5587

.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 25/29

Page 78: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Observe the decay of singular values of the Loewner pencil.

We obtain a 3rd order system, interpolating the data.

10−1 100 101 102 10310−3

10−2

10−1

100

freq

‖H‖

originalLoewnertraining data E =

1 0 00 1 00 0 1

,

A =

−1 100 0−100 −1 00 0 −3

B =

−1.5461−0.7585−1.1096

C =

−0.8456−0.5727−0.5587

.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 25/29

Page 79: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Add 1% noise in the data and construct 3rd-order dynamical system using the Loewner method.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 26/29

Page 80: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseIllustration of Loewner with noise

Add 1% noise in the data and construct 3rd-order dynamical system using the Loewner method.

10−1 100 101 102 10310−4

10−3

10−2

10−1

100

originalLoewnertraining data

2 4 6 8 1010−5

10−4

10−3

10−2

10−1

100

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 26/29

Page 81: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseRank-based problem formulation

A rank-minimization Problem Formulation + Noise (for the simplest linear system)

minE,A

rank([E,A

])(3)

subject to

‖EΛ + A−V1T ‖F ≤ ε,‖ETΩ + AT −W1T ‖F ≤ ε,

where Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Intuition: Looking for a minimal-order systems at approximately interpolate data but not exactly.

A similar formulation was proposed in [Fazel/Hindi/Byod ’04] in the context of system identificationusing time-domain noisy data.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 27/29

Page 82: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseRank-based problem formulation

A rank-minimization Problem Formulation + Noise (for the simplest linear system)

minE,A

rank([E,A

])(3)

subject to

‖EΛ + A−V1T ‖F ≤ ε,‖ETΩ + AT −W1T ‖F ≤ ε,

where Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Intuition: Looking for a minimal-order systems at approximately interpolate data but not exactly.

A similar formulation was proposed in [Fazel/Hindi/Byod ’04] in the context of system identificationusing time-domain noisy data.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 27/29

Page 83: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseRank-based problem formulation

A rank-minimization Problem Formulation + Noise (for the simplest linear system)

minE,A

rank([E,A

])(3)

subject to

‖EΛ + A−V1T ‖F ≤ ε,‖ETΩ + AT −W1T ‖F ≤ ε,

where Λ = diag (λ1, . . . ,λl) , Ω = diag (µ1, . . . ,µl) ,

Intuition: Looking for a minimal-order systems at approximately interpolate data but not exactly.

A similar formulation was proposed in [Fazel/Hindi/Byod ’04] in the context of system identificationusing time-domain noisy data.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 27/29

Page 84: A Rank Minimization Approach to Learning Dynamical Systems

Measurement NoiseRank-based problem formulation

10−1 100 101 102 10310−4

10−3

10−2

10−1

100

originalLoewnerRank-basedtraining data

2 4 6 8 1010−20

10−14

10−8

10−2

LoewnerRank-based

We could recover the original system.

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 28/29

Page 85: A Rank Minimization Approach to Learning Dynamical Systems

Conclusions

Contribution of this talk

Identification of linear systems from frequency response

Can impose structure obtained using prior engineering knowledge

An efficient algorithm to solve the optimization problem

Proof of concepts by mean of numerical examples

Open questions and future work

Working with data obtained in a lab

What if the structure is not known: structure discovery

Need to do analysis for noisy case

Thank you for your attention!!

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 29/29

Page 86: A Rank Minimization Approach to Learning Dynamical Systems

Conclusions

Contribution of this talk

Identification of linear systems from frequency response

Can impose structure obtained using prior engineering knowledge

An efficient algorithm to solve the optimization problem

Proof of concepts by mean of numerical examples

Open questions and future work

Working with data obtained in a lab

What if the structure is not known: structure discovery

Need to do analysis for noisy case

Thank you for your attention!!

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 29/29

Page 87: A Rank Minimization Approach to Learning Dynamical Systems

Conclusions

Contribution of this talk

Identification of linear systems from frequency response

Can impose structure obtained using prior engineering knowledge

An efficient algorithm to solve the optimization problem

Proof of concepts by mean of numerical examples

Open questions and future work

Working with data obtained in a lab

What if the structure is not known: structure discovery

Need to do analysis for noisy case

Thank you for your attention!!

©Pawan Goyal, [email protected] A Rank Minimization Approach to Learning Dynamical Systems from Frequency Response Data 29/29