SYSTEMS Identification

SYSTEMSSYSTEMSIdentificationIdentification

Ali Karimpour

Assistant Professor

Ferdowsi University of Mashhad

2

lecture 9

Ali Karimpour Dec 2009

Lecture 9

Asymptotic Distribution of Parameter Estimators

Topics to be covered include:

Central Limit Theorem

The Prediction-Error approach: Basic theorem

Expression for the Asymptotic Variance

Frequency-Domain Expressions for the Asymptotic Variance

Distribution of Estimation for the correlation Approach

Distribution of Estimation for the Instrumental Variable Methods

3

lecture 9


Overview

If convergence is guaranteed, then * N

But, how fast does the estimate approach the limit?

What is the probability distribution of ? * N

The variance analysis of this chapter will reveal:

a) The estimate converges to at a rate proportional to

b) Distribution converges to a Gaussian distribution: N(0,Q)

c) Covariance matrix Q, depends on - The number of samples/data set size: N,

- The parameter sensitivity of the predictor: - The noise variance

*N

1

4

lecture 9





The Prediction-Error approach: Basic Theorem





5

lecture 9



The mathematical tool needed for asymptotic variance analysis is “Central Limit” theorems.

Example: Consider two independent random variable, X and Y, with the same uniform distribution, shown in Figure below.

Define another random variable Z as the sum of X and Y: Z=X+Y. we can obtain the distribution of Z, as :

dxxzfxfzf YXZ )()()(

6

lecture 9


In general, the PDF of a random variable approaches a Gaussian

distribution, regardless of the PDF of each , as N gets larger.

N

iiX

1


Further, consider W=X+Y+Z. The resultant PDF is getting close to a Gaussian distribution

The resultant PDF is getting close to a Gaussian distribution.

7

lecture 9



Let be a d-dimensional random variable with : ,...1,0, tX t

)( tXEm

]))([( Ttt mXmXEQ

Mean

Cov

Consider the sum of given by:mX t

N

tt mX

N1

N )(1

Y

Then, as N tends to infinity, the distribution of converges to the Gaussian distribution given by PDF:

NY

),0(YN N QNondistributi

8

lecture 9










9

lecture 9


M

NNN

DZV

),(minarg

),(

2

11),( 2

1

tN

ZVN

t

NN

0),( NNN ZV

Then, with prime denoting differentiation with respect to ,

Expanding around gives:*

)(),(),(),(0 ** NN

NN

NN

NN ZVZVZV

),( NNN ZV

is a vector “between” *& N

Applying the Central Limit Theorem, we can obtain the distribution of estimate as N tends to infinity.

Let be an estimate based on the prediction error method N

M

NNN

DZV

),(minarg

),(

2

11),( 2

1

tN

ZVN

t

NN

The Prediction-Error Approach

10

lecture 9



),( NN ZV

),()],([)( *1* NN

NNN ZVZV

),(),(1

),( **

1

* ttN

ZVN

t

NN

Assume that is nonsingular, then:

** )(),(),( *

ty

d

dt

d

dt

Where as usual:

To obtain the distribution of , and must be computed as N tends to infinity.

* N

),( * N

N ZV

),()],([)( *1* NN

NNN ZVZV

),( NN ZV

N

NN

NNN ZV

d

dZV

|),(),(

11

lecture 9


For simplicity, we first assume that the predictor is given by a linear regression:

Ttty )()|(

The actual data is generated by ( is the parameter vector of the true system)

)()()( 0* tetty T

So:

Therefore:

*

)(|)|( * ttyd

d

d

d T

)|()(),( tytyt

)()()()(),( 0*

0** tettett TT

N

t

NN tet

NZV

10

* )()(1

),(


12

lecture 9


Let us treat as a random variable. Its mean is zero, since: tXtet )()( 0

The covariance is

Consider:

Appling the central limit Theorem:


0)]([)]([)]()([ 00 teEtEtetEm

RttEteEXX

stforssetetEmXmXEXX

TTst

TTst

Tst

02

0

00

])()([)]([)cov(

,0])()()()([]))([()cov(

)()(1

)(1

10

1

tetN

mXN

YN

t

N

ttN

),0(),( 0* RNZVNYN N

NN

13

lecture 9


Next, compute :

And:


),( NNN ZV

14

lecture 9


We obtain:

So:

),()],([)( *1* NN

NNN ZVZV

NasQNN N ),0()( *


RttN

ZV TN

tN

NNN

))()((1

lim),(1

),0(),( 0* RNZVNYN N

NN

15

lecture 9


The extended result of estimate distribution is summarized in the following theorem, i.e. Ljun’g Textbook Theorem 9-1.

Theorem 1 Consider the estimate determined by:

Assume that the model structure is linear and uniformly stable and that the data set satisfies the quasi stationary requirements. Assume also that converges with probability 1 to a unique parameter vector involved in :

M

NNN

DZV

),(minarg

),(

2

11),( 2

1

tN

ZVN

t

NN

N

MD

Also we have:

And that:

Converge to with probability 1

0)( NV


*|),()|(1

lim)(1

*

tty

d

d

NV

N

tN

tm

16

lecture 9


Where is the ensemble mean given by:

Then, the distribution of converges to the Gaussian distribution given by

)( NN

})],()][,({[.lim TNN

NN

NZVZVENQ

11 )]([)]([ VQVP

),0()( PNN N


tm

17

lecture 9


As stated formally in Theorem 1, the distribution of converges to a Gaussian distribution for the broad class of system identification problems. This implies that the covariance of asymptotically converges to:

)( NN

PN

Cov N1ˆ

N

This is called the asymptotic covariance matrix and it depends not only on

(a) the number of samples/data set size: N, but also on (b) the parameter sensitivity of the predictor:

(c) Noise variance

)(),(),( ty

d

dt

d

dt

1*1* )]([)]([ VQVP


18

lecture 9










19

lecture 9


Let us compute the covariance once again for the general case:

Unlike the linear regression, the sensitivity is a function of θ,

Quadratic Criterion

)(),(}{ 000 tetandDc Assume that is a white noise with zero mean and variances . We have:0

20

lecture 9


),(),()(),(),(),()( **0

*** ttEtetEttEV TT

),(),(),(),(1

lim

),()()(),(lim

**0

**

10

*

1 10

*2

ttEttEN

stetetEN

NQ

TTN

tN

Ts

N

t

N

sN

Similarly

Hence:1**

0 )],(),([ ttEP T

The asymptotic variance is therefore a) inversely proportional to the number of samples, b) proportional to the noise variance, and c) inversely related to the parameter sensitivity.

The more a parameter affects the prediction, the smaller the variance becomes .

PN

Cov N1ˆ

Quadratic Criterion

21

lecture 9


1

1

]),(),(1

[

N

tN

TNNN tt

NP

),(2

11 2

1N

N

tN t

N

A very important and useful aspect of expressions for the asymptotic covariance matrix is that it can be estimated from data. Having N data points and determined we mat use:N

Since is not known, the asymptotic variance cannot be determined. N

sufficient data samples needed for assuming the model accuracy may be obtained.

Quadratic Criterion

22

lecture 9


Consider the system

)()1()1()(: 00 tetutyatyS

)(tu )(0 te 0

Suppose that the coefficient for is known and the system is identified in the model structure

)1( tu

)1()1()( 0 tutyaty

)1()(ˆ),( tytyd

dt

and are two independent white noise with variances and respectively

We have:

atetutyatyM ).()1()1()(: 00

Or

Example : Covariance of LS Estimates

23

lecture 9


Hence

)1(20

tyEP

002

0 )1(2)0()0( yvy RaRaR

0)0()0()1( 0 yuyy RRaR

To compute the covariance, square the first equation and take the expectation:

Multiplying the first equation by and taking expectation gives:

The last equality follows, since does not affect (due to the time delay )

20

02

1)0()1(

aRtyE y

0

200 1

a

NaCov N

)1( ty

)(ty)(tu

Hence:


24

lecture 9


Cov (a) = 0.0471


Assume a=0.1,

Estimated values for parameter a, for 100 independent experiment using LSE, is shown in the bellow Figure.

,1 10

25

lecture 9


Cov (a) = 0.0014


,10 10 Now, assume a=0.1, ,10 10

26

lecture 9


Cov (a) = 0.1204


Now, assume a=0.1, ,1 100

27

lecture 9


Example : Covariance of an MA(1) Parameter

Consider the system

is white noise with variance . The MA(1) model structure is used:0)}({ 0 te

)1()()( 000 tectety

)1()1()( tycctyccty

)1(),1(),1(),( tyctyctcct

0cc )1(

1

1),( 01

00

te

qcct

Nc

N

c

ctENcCov N

20

02

0 1

),(

1

Given the predictor 4.18:

Differentiation w.r.t c gives

At we have :

If is the PEM estimate of c:

)1()()( tcetety

)|(ˆ)(1)()()(1)()()|(ˆ tytyqCtyqAtuqBty

28

lecture 9


),),,((),),,((),([1

),(1

tttttN

ZVN

t

NN

N

t

NN tt

NZV

1

),),,((1

),(

),,()( t ),,()( t

),),,((),(),),,((),),,((),(

),),,((),(),(),),,((),()(

ttEtttEtttE

tttEttttEVT

For general Norm ),,( t

We have:

Similarly:

We can use the asymptotic normality result in this more general form whenever required. The expression for the asymptotic covariance matrix is rather complicated in general.

Asymptotic Variance for general Norms.

29

lecture 9


)(),,(),),,((1

),(1

twithttN

ZVConsiderN

t

NN

0))(( teE Assume that . Then under assumption after straightforward calculations:

MS

100 )],(),()[( ttEkP T

20

20

))](([

))](([)(

teE

teEk

02

02 )()(,1)()(,

2

1)( tEeksoeandee Clearly for for quadratic

The choice of in the criterion only acts as scaling of the covariance matrix

Asymptotic Variance for general Norms.

30

lecture 9










31

lecture 9


Frequency-Domain Expressions for the Asymptotic Variance.

The asymptotic variance has different expression in the frequency domain, which we will find useful for variance analysis and experiment design.

Let transfer function and noise model be consolidated into a matrix

The gradient of T, that is, the sensitivity of T to θ, is

For a predictor, we have already defined W(q,θ,) and z(t), s.t.

32

lecture 9


Therefore the predictor sensitivity is given by

Where

Substituting in the first equation:


33

lecture 9


At (the true system), note and0 )(),( 00 tet

where

Ttetutx )]()([)( 00

Let be the spectrum matrix of :)(0

x )(0 tx

Using the familiar formula:


34

lecture 9


For the noise spectrum,

Using this in equation below:

We have:

The asymptotic variance in the frequency domain.


35

lecture 9










36

lecture 9


The Correlation Approach

0),(ˆ

NN

DN Zfsol

M

N

tF

NN tt

NZf

1

),(),(1

),(

),()(),()(0 ,N

NNNN

NN

NN ZfZfZf

),()(),( tqLtF )(),()(),(),( tuqKtyqKt uy

By Taylor expansion we have:

),( * t ),( * NN ZV ),( * t

),( NN Zf

This is entirely analogous with the previous one obtained for PE approach, with he difference that in is replaced with in .

We shall confine ourselves to the case study in Theorem 8.6, that is, and linearly generated instruments. We thus have:

)(

37

lecture 9



Theorem : consider by 0),(ˆ

NN

DN Zfsol

M

),( t

});,(),,(),,(),,({ Muyuy DqKd

dqK

d

dqKqK

Assume that is computed for a linear, uniformly stable model structureAnd that:

NaspwN 1..*

)( *f

NasZEfN NN ,0),( *

),0()( * PNAsN N

1*1* )]([)]([ fQfP

),(),(.lim ** NTN

NN

NZfZfENQ

TF tqLtEttEf )],()()[,(),(),()(

is a uniformly stable family of filters.

Assume also that that is nonsingular and that

Then

38

lecture 9



39

lecture 9


Example : Covariance of an MA(1) Parameter

Consider the system

is white noise with variance . Using PLR method:0)}({ 0 te

)1()()( 000 tectety

0cc )1(1

1),( 01

00

te

qcctAt we have :

40

lecture 9










41

lecture 9


Instrumental Variable Methods

Suppose the true system is given as

Where e(t) is white noise with variance independent of {u(t)}. then0

is independent of {u(t)} and hence of if the system operates in open loop. Thus is a solution to:

),( * t

0

We have:

42

lecture 9



To get an asymptotic distribution, we shall assume it is the only solution to . MD

Introduce also the monic filter

Intersecting into these Eqs.,

43

lecture 9


Consider the system

)()1()1()(: 00 tetutyatyS

)(tu )(0 te 0

Suppose that the coefficient for is known and the system is identified in the model structure.

Let a be estimated by the IV method using:

)1( tu

and are two independent white noise with variances and respectively


By comparing the above system with

We have

44

lecture 9



Documents

SYSTEMS Identification