Correlation Functions in Time and Frequency Domain

8/10/2019 Correlation Functions in Time and Frequency Domain

1/48

2/11/2010

DelftUniversity ofTechnology

System Identification

&

Parameter Estimation

Correlation functions in time & frequency domain

Alfred C. Schouten, Dept. of Biomechanical Engineering (BMechE), Fac. 3mE

Wb2301: SIPE lecture 2


2/48

2SIPE, lecture 2 | xx

Contents

Correlation functions in time domain

Autocorrelations (Lecture 1)

Cross-correlations

General properties of estimators

Bias / variance / consistency

Estimators for correlation functions

Fourier transform

Correlation functions in frequency domain

Auto-spectral density

Cross-spectral density

Estimators for spectral densities


3/48


Lecture 1: (auto)correlation

functions

Autocorrelation function

Autocovariance

function

Autocorrelation coefficient

( ) ( ) 2 = = ( ) ( ) ( ) ( )xx x x xx xC E x t x t

[ ] = ( ) ( ) ( )xx E x t x t

2

2 0

= = =

( ) ( ) ( ) ( )( ) ( )

x x xx x xxxx

x x x xx

x t x t Cr E C


4/48


Lecture 1: autocovariance

function

Autocovariance

function:

If = 0, then autocovariance

and autocorrelation functions are identical

At zero lag:

( )( )

[ ] [ ] [ ] 2

2 2 2

2

=

= +

= +

=

( ) ( ) ( )

( ) ( ) ( ) ( )

( )

( )

xx x x

x x x

xx x x x

xx x

C E x t x t

E x t x t E x t E x t E

( )2 20( ) ( )xx x xC E x t

= =


5/48


Example autocovariances

Matlab

demo: Lec2_example_Cuu.m

0 1 2 3 4 5

-5

0

5

white

signal

-2 -1 0 1 2

-2

0

2Cuu

0 1 2 3 4 5-1

0

1

low-pass

-2 -1 0 1 2-0.05

0

0.05

0 1 2 3 4 5-1

0

1

sinusoid

-2 -1 0 1 2-0.5

0

0.5

0 1 2 3 4 5-5

0

5

sin.

+noise

time [s]

-2 -1 0 1 2-2

0

2

tau [s]


6/48


2D Probability density function

{ }dyytyydxxtxxdxdyyxf yx ++


7/48


2D Probability density function

Probability for certain values of y(t) given certain values of x(t)

Co-variance of y(t) with x(t): y(t) is related with x(t)

No co-variance between y(t) and x(t): 2D probability density functionis circular

Covariance between y(t) and x(t): 2D probability density function is

ellipsoidal

No co-variance between y(t) and x(t):

y(t) and x(t) are independent

no relation exist

transfer function is zero !


8/48


Cross-correlation functions

Measure for common structure in two signals:

Cross-correlation

Cross-covariance

Cross-correlation coefficient

[ ]( ) ( ) ( )xy E x t y t =

( )( ) = = ( ) ( ) ( ) ( )xy x y xy x yC E x t y t

0 0

= =

( ) ( )( )( )

( ) ( )

y xyxxy

x y xx yy

y t Cx tr E

C C


9/48


Example: Estimation of a Time Delay

Consider the system:

Additive noise v(t) is stochastic and uncorrelated to x(t),

Cross-covariance of the measured input and output in case:

input x(t) is white noise (stochastic)

0( ) ( ) ( )y t x t v t = +


10/48


Example: Estimation of a Time Delay

Matlab

demo: Lec2_example_Cuy_delay.m

0 5 10 15 20 25 30 35 40 45 50

-5

0

5

u(t)

time [s]

u

0 5 10 15 20 25 30 35 40 45 50-5

0

5

y(t)

time [s]

u

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-1

0

1

Cuy

()

[s]


11/48


Effects of Noise on Autocorrelations

Often, correlation functions must be estimated from measurements.

Consider x (t ) and y (t ), corrupted with noise n (t ) and v (t ) resp.:

Noises have zero mean and are independent of the signals and of eachother.

Autocorrelation:

Additive noise will bias autocorrelation functions!

( ) ( ) ( )

( ) ( ) ( )

w t x t n t

z t y t v t

= +

= +

( ) ( )( ) ( ) ( ) ( ) ( )zz E y t v t y t v t = + +

( ) ( ) ( ) ( )yy yv vy vv = + + +

( ) ( )yy vv = +


12/48


Effects of Noise on Cross-correlations

Cross-correlation:

Additive noise will not bias cross-correlation functions!

( )( )

= + +

= + + +

=

( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )

wz

xy xv ny nv

xy

E x t n t y t v t


13/48


Noise Reduction

Longer recordings

More information, same amount of noise

Repeat the experiment Noise cancels out by averaging from exactly the same inputs

Improve Signal-to-Noise Ratio

Concentrate power at specified frequencies, assuming the noisepower remains the same


14/48


Properties of Estimators

Formula given (autocovariances

/ spectral densities) areestimators for the true

relations.

What are the properties of these estimators? bias / variance / consistency

Bias: structural error

Variance: random error

Consistent: A consistent estimator is an estimator that converges,in probability, to the quantity being estimated as the sample size

grows.


15/48


Example: estimator for mean value

and variance of a signal

Signal xk

, with

k=1N

Estimator for the signal mean:

Estimator for the signal variance:

What are the expected values of both estimators?

Expectation operator: E{.}

( )

=

=

=

=

N

k

xkx

N

k

kx

x

N

xN

1

22

1

1

1


16/48


Estimator for the mean value

{ } { }

{ } { }

{ } xN

k

xx

xk

N

k

k

N

k

kx

N

k

kx

NE

xExE

xENxNEE

xN

==

==

=

=

=

=

==

=

1

11

1

1

11

1


17/48


Estimator for the variance

Estimator is biased and consistent

( )

{ } { } { } { }

{ }

222

1 1 1

2

21 1 1 1

2 2

21 1 1 1

2 2 2 2 2 2 2

21

1 1 1

1 2 1

1 2 1

1 2 1 2

k l k l

N N N

x k x k l

k k l

N N N N

k k l l k k l l k

N N N N

x k k l l kk l l k

N

x x x x x x x x x x

k

x x xN N N

x x x x xN N N

E E x E x x E x xN N N

E K KN N N

= = =

= = = =

= = = =

=

= =

= +

= +

= + + +

{ }

1 1 1

2 2 2 2

22 1 1 1 1 1 1

N N N

k l l

x x x xNE N

N N N N

= = =

= + = =


18/48


Estimator for the variance

biased

estimator:

unbiased

estimator:

(default!)

( )

{ } 2221

22

111

1

xxx

N

k

xkx

N

N

NE

xN

=

=

= =

( )

{ } 221

22

1

1

xx

N

k

xkx

E

xN

=

= =


19/48


Estimates of Correlation Functions

Cross-correlation:

Let x(t) and y(t) are finite time realizations of ergodic

processes. Then:

Practically, signals are finite time and sampled every t.Signal x(t) is sampled at t=0, t, , (N-1)tgiving x(i) with i=1, 2, , N.

Then:

with i

=1, 2, , N

1

2 ( ) lim ( ) ( )

T

xyT

T

x t y t dtT

=

1

=

=

( ) ( ) ( )N

xy

i

x i y iN

[ ]( ) ( ) ( )xy E x t y t =


20/48


Estimates of Correlation Functions

Unbiased estimator:

Variance of the estimator increases with lag!To avoid this, divide by N (biased estimator):

Use large Nto minimize bias! I.e.

Similar estimators can be derived for the covariance andcorrelation coefficient functions

1

=

=

( ) ( ) ( )N

xyi

x i y iN

1 ( ) ( ) ( )N

xyi x i y iN =

=

1

N

N


21/48


Additional demos

Calculation of cross-covariance: Lec2_calculation_Cuy.m

1 ( ) ( ) ( )N

xy

i

x i y i

N

=

=


22/48


Cross-covariance of some basic systems

Example systems (Matlab

Demo: Lec2_example_systems.m):

Time delay

1st order

2nd order

Input signals:

White noise (WN)

Low pass filtered noise (LP: 1/(0.5s+1) )

ti d l 1st d t 2nd d t


23/48


-5 0 5-0.5

0

0.5

1time delay

Cuy

(WN)

-5 0 5-0.1

0

0.1

Cuy

(LP)

-5 0 50

0.5

1

impulse

time [s]

-5 0 5-0.01

0

0.01

0.021storder system

-5 0 5-0.02

0

0.02

0.04

-5 0 50

1

2

time [s]

-5 0 5-0.05

0

0.052ndorder system

-5 0 5-0.2

0

0.2

-5 0 5-5

0

5

10

time [s]


24/48


Intermezzo: Fourier Transform

Fourier Transform:

Mapping between time-domain and frequency-domain

One-to-one mapping

Unique: inverse Fourier Transform exists

Linear technique

Inverse Fourier Transform:

( )( ) ( ) 2( ) j ftX f x t x t e dt

= =

( )( ) ( )1 2( ) j tfx t X f X f e df

= =


25/48


Fourier transformation

y(t) is an arbitrary signal

Euler formula:

symmetric part: cos(z)

anti-symmetric part: sin(z)

2( ) ( ) j ftY f y t e dt =

2 2 2( ) ( ) cos(2 ) *sin(2 )j ft j ft j fte re e im e ft j ft = + =

( ) ( )cos sinjze z j z = +


26/48


Example Fourier Transformation

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5

-1

-0.5

0

0.5

1

1.5

( ) ( )HzHzty 5sin5.0sin)( +=

Y(0.5Hz): 5 Hz signal will be averaged out

2( ) ( ) j ftY f y t e dt =


27/48


Example Fourier Transformation

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5

-1

-0.5

0

0.5

1

1.5

-0.5 0 0.5 1 1.50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

y(t) = sin(0.5t) Y() = (-0.5)


28/48


Frequency Domain Expressions

Discrete Fourier Transform:

where f

takes values 0, 1, , N-1

multiples of

Inverse Fourier Transform:

( )2

1

( ) ( ) ( )ftN jN

t

U f u t u t e

=

= =

( )2

1

1

1( ) ( ) ( )

ftN jN

f

u t U f U f eN

=

= =

1f

N t =


29/48


Fourier transform of signals

Discrete Fourier Transform (DFT)

DFT maps N real values in time domain to N complex values infrequency domain

Double information? DFT is symmetric U(-r)=U(r)* Information for N/2 complex values

( ) [ ]

( ){ } ( )

[ ]1,,1,0);(

)(

1,,1,0;

1

0

/2

==

=

NrrU

ekukuDFTrU

Nkku

N

k

Nrkj

2,,1,0

Nr

P t


30/48


Power spectrum or

auto-spectral density

auto-spectral density Sxx

is Fourier Transform of autocorrelation

properties:

Real values (no imaginary part)

Symmetry: Sxx

(f)=Sxx

(-f) => only positive frequencies are analyzed

Cxx

(0) =

Sxx

(f) df

= x2

Area under Sxx

(f) is equal to variance of signal (Parsevals

theorem)

( )

( ) 2

( ) ( ) ( )

( ) ( ) ( )

j

xx xx xx

jf

xx xx xx

S e d

S f e d

= =

= =

C t


31/48


Cross-spectrum orCross-spectral density

cross-spectral density Sxy

:

properties:

Complex values Sxy

*(f)=Sxy

(-f) => only positive frequencies are analyzed

Sxy

(f) describes the interdependency of signals x(t) and y(t) in

frequency domain (gain and phase)

if xy

() = 0, then Sxy

(f) = 0 for all frequencies

( )

2( ) ( ) ( ) j fxy xy xy

S f e d

= =


32/48


Estimators for the power spectrum

Fourier transform of the autocorrelation function, the power spectrum:

using the time average estimate:

and multiplication with

gives:

* : complex conjugate:

1 2

0

( ) ( )fN jN

uu uuS f e

=

=

1 2 2

0

1 ( )

( ) ( ) ( )i f if N Nj j

N Nuu

i

S f u i e u i eN

= =

=

1 ( ) ( ) ( )N

uui

u i u iN

=

=

20 1

= =( )i i f

jNe e

1 *( ) ( )U n f U n f N

= *( ) ( )U n f U n f =

Indirect approach

and direct approachgive equal result!


33/48


Estimator for the cross-spectrum

Fourier transform of the cross-correlation function is called the cross-spectrum:

1 * ( ) ( ) ( )uy

S f U n f Y n f N

=

1 2

0

==

( ) ( )

fN jN

uy uyS f e


34/48


Time-domain vs. Frequency-domain

x(t), y(t)

xy

()

Cxy

()

rxy

()

X(f), Y(f)

Sxy

(f)

xy

(f)

input, output

cross-correlation

function

cross-covariance

function

correlation

coefficient

input, output

cross-spectral

density

coherence

FourierTransformationTime

Domain Frequency Domain


35/48


Discrete and Continuous Signals

Reconstruction

DA conversion

Distortion: Hold Circuits

Sampling

AD conversion

Distortion: "Dirac Comb"

Continuous system: Plant

Digital Controller: Plant performance enhancement


36/48


Sampling, Diracs Comb


37/48


Models of Linear Systems

System:

Linear systems obey both scaling and superposition property!

( ) ( ( ))y t N u t=

1 1 1 1

2 2 2 2

1 1 2 2 1 1 2 2

=

=

+ = +

( ) ( ( ))

( ) ( ( ))

( ) ( ) ( ( ) ( ))

k y t N k u t

k y t N k u t

k y t k y t N k u t k u t

N

u(t) y(t)


38/48


Time domain models

Assume input is a pulse with:

Response of linear system N to

pulse:

Assume u(t) is weighted sum of

pulses:

( )1

for 0, 2

0 otherwise

tt

d t t t


39/48


Time domain models

The output is:

Limit case: unit-area pulse d(t,t) is an impulse:

Then h(t) is the impulse response function (IRF) and the output

the convolution integral

( ) ( )0

lim ,t

d t t t

=

( )( )

( )

=

=

=

=

=

( ) ( ( ))

,

,

kk

k

k

y t N u t

u N d t k t t

u h t k t t

( ) ( )

= ( )y t h u t d


40/48


Time domain models

Causal system: h(t)=0 for tT

Discrete

( ) ( )

1

0

== ( )

T

y t h u t

( ) ( ) = ( )T

o

y t h u t d


41/48


Frequency domain models

Time-domain: convolution integral

Fourier transform:( ) ( ) ( )

( ) ( )

( ) ( )

( )

2

2

2

= =

=

=

=

=

( ) ( )

( )

( ) ( ) ( )

j ft

ft

f

Y f y t h u t d

h u t d e dt

h u t e dtd

U f h e d

Y f H f U f

( ) ( )

= ( )y t h u t d

Convolution in time domain is multiplication infrequency domain

(and vice-versa)


42/48


Estimators in time domain

Cross-covariance

When analyzing the dynamics of a dynamical system interested in

variations of the signals around it mean (and not average value).

Assuming signals with zero-mean

( ) ( ){ }

( ) ( )( ) ( ') ( ') '

( ) ( ') ( ) ( ') 'uy

y t h u t h t u t t dt

C E u t y t E h t u t u t t dt

= =

= =

Basic identification with cross-


43/48


Basic identification with cross-covariance

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( )

'

multiply with ( ): ' ( )

' '

white noise: 0 for 0; 0 1

uy un uu

uu uu

y t n t h t u t - t' dt'

u t u t y t u t n t h t u t u t - t' dt'

C C h t' C t dt

C C

= +

= +

= +

= =

( ) ( ) ( )

Other 'tricks' needed when u(t) is not white

uy unC C h = +

?y(t)

u(t)

n(t)

Basic identification with spectral


44/48


Basic identification with spectral

densities

( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( )

'

multiply with ( ): ' ( )

' '

Fourier transform:

if 0:

uy un uu

uy un uu

un

y t n t h t u t - t' dt'

u t u t y t u t n t h t u t u t - t' dt'

C C h t' C t dt

S f S f H f S f

SS f H f

= +

= +

= +

= +

= =

( )

( )uy

uu

f

S f

?y(t)

u(t)

n(t)

Identification:


45/48


Identification:

time-domain vs. frequency-domain

Time domain

Unknown system: impulse response function (IRF) of h(t) over

limited time Mostly: direct model parameterization (fit of physics model)

Frequency domain

Unknown system: frequency response function (FRF) H(f) for number

of frequencies

?y(t)

u(t)

n(t)

Example IRF & FRF of typical


46/48


Example IRF & FRF of typical

systems

Time delay

First order system

Second order system

Unstable system

(try Matlab!)


47/48


Book: Westwick

& Kearney

Lecture 1: signals

Chapter 1, all

Chapter 2, sec. 2.1

2.3.1

Lecture 2: Correlation functions in time & frequency domain

Chapter 2, sec. 2.3.2

2.3.4

Chapter 3, sec. 3.1

3.2

Lecture 3: Estimators for impulse & frequency response functions

Chapter 5, sec. 5.1

5.3


48/48


Next week: lecture 3

Lecture 3:

Estimation of IRF and FRF

Estimator for coherence

Open loop and closed loop system identification

Documents

Correlation Functions in Time and Frequency Domain