Lesson 6: Estimation of the Autocovariance Function of a ... · Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Umberto Triacca Dipartimento di Ingegneria

Lesson 6: Estimation of the AutocovarianceFunction of a Stationary Process

Umberto Triacca

Dipartimento di Ingegneria e Scienze dell’Informazione e MatematicaUniversita dell’Aquila,

[email protected]

Umberto Triacca Lesson 6: Estimation of the Autocovariance Function of a Stationary Process

Introduction

The autocovariance function plays a key role in the construction ofthe DGP’s model of our time series.

Thus an important question is:

Is it possible to obtain a ’good’ estimate of theautocovariance function?


Introduction

This question is not trivial since we observe only one realization ofthe process.

We cannot, for example, observe a second realization of Italianannual GDP for the period from 2001 to 2011.

In general, with a single realization we are unable to estimate amoment function of a stochastic process.


The Mean Function of a stochastic process

First, we consider the mean function.Definition. Let {xt ; t ∈ Z} be a stochastic process such thatVar(xt) <∞ ∀t ∈ Z. The function

µx : Z→ R

defined byµx(t) = E (xt)

is called Mean Function of the stochastic process {xt ; t ∈ Z}.


The Mean Function of a weakly stationary process

Let {xt ; t ∈ Z} be a weakly stationary stochastic process. Themean function is given by

µx(t) = µ ∀t ∈ Z

Thus, for a weakly stationary process, the mean function is theconstant function which takes always the same value µ.


Estimation of Mean Function

How do we estimate µ?



If were possible to obtain m independent realizations{x

(j)1 , ..., x

(j)T

}j = 1, ...,m

we could estimate the mean of {xt ; t ∈ Z} by averaging over therealizations

µ(δ)m =

1

m

m∑j=1

x(j)δ , δ ∈ {1, 2, ...,T}



Since

E

[(µ

(δ)m − µ

)2]

=γx(0)

m

we have

limm→∞E

[(µ

(δ)m − µ

)2]

= 0

The estimator µ(δ)m converges in quadratic mean to µ as the

number of realizations increase.

This implies that µ(δ)m is consistent for µ.



Unfortunately, as we have already seen, it is impossible to obtainmultiple realizations. We have a single finite realization

x1, x2, . . . , xT


Estimation of Mean and Autocovariance Function

Then, the only way to perform the average is along the time axis.

⇓

The mean and covariance functions can be estimated by theircorresponding time averages:

xT =1

T

T∑t=1

xt

and

γx(k) =1

T

T∑t=k+1

(xt − xT )(xt−k − xT ) for k = 0, 1, . . . ,T − 1



Now the question is: are these estimators ‘good’?



The answer is yes, if the stationary process is ergodic

Ergodic comes from Greek.From ergon ‘energy, work’and hodos‘way, path’.


Ergodic processes

What is an ergodic process?


Ergodic processes

Definition. A stationary stochastic process is said ergodic, withrespect to a given population moment, if the sample (or time)moment for a single finite realization of length T converges inquadratic mean to the population moment as T increases to ∞.

Remark. Note that we cannot simply refer to a process as ergodic.Ergodicity must be related directly to the particular populationmoment, e.g. mean value, covariance, etc.


Ergodic processes

In particular, a stationary process xt with mean µ is mean-ergodic if

limT→∞E[(xT − µ)2

]= 0

and it is covariance-ergodic if

limT→∞E

( 1

T

T∑t=k+1

(xt − xT )(xt−k − xT )− γx(k)

)2 = 0


ergodic processes

It is important to underline that covariance-ergodicity impliesmean-ergodicity, but not the reverse.


ergodic processes

A simple example of mean and covariance-ergodic process is theprocess

ut ∼ i .i .d .N(0, 1)

Problem: Show that the process ut ∼ i .i .d .N(0, 1) is mean-ergodic

We observe that the process ut has no memory, in the sense thatthe value of the process at time t is uncorrelated with all pastvalues up to time t − 1.


ergodic processes

It is important to note that not all stationary processes areergodic.


Ergodic processes

Consider the stationary process {xt ; t ∈ Z}, with xt = A ∀t ∈ Z,where A is a random variable with mean 3 and variance 7.

This process is not mean-ergodic since the sample mean xT doesnot converge to µ as T →∞


Ergodic processes

In fact we have

limT→∞E[(xT − 3)2

]= 7 6= 0


The meaning of the ergodicity

The ergodicity is a matter of information contained in a singlerealization of a long duration of the process.



If the process is not too persistent (ergodicity), so that eachelement of the realization x1, x2, . . . , xT will contain someinformation not available from the other elements, then a singlerealization of a long duration will be sufficient to obtain a goodestimate of its moments.



We have seen that:

1 the process ut ∼ i .i .d .N(0, 1) is mean-ergodic;

2 the process xt = A, where A is a random variable with mean 3and variance 7 is not mean-ergodic.



Consider the process ut ∼ i .i .d .N(0, 1) and let

u1, u2, . . . , uT

be a time series from this process. Each element of the time seriesu1, u2, . . . , uT will contain some information not available from theother elements. There are new information contained in thenew observations.



Now, we consider the process xt = A, where A is a randomvariable with mean 3 and variance 7. We note that a time seriesfrom this process will has the form

a, a, ..., a

where with a we have denoted a realization of the random variableA.Thus in the time series

x1 = a, x2 = a, ..., xT = a

there is the same infomation contained in the first observation x1.There aren’t new information contained in the newobservations.



The process ut has no memory, in the sense that the value of theprocess at time t is uncorrelated with all past values up to timet − 1. The process xt = A, instead, is too persistent.


Ergodic Theorems

Now, we present some theoretical results, called ergodictheorems. These theorems give some necessary and/or sufficientcondition for mean-ergodicity of a stationary process.


Ergodic Theorems

Theorem. (Slutsky’s Theorem) A stationary process xt with meanµ and autocovariance function γx(k) is mean-ergodic iff

limT→∞1

T

T−1∑k=0

γx(k) = 0


Ergodic Theorems

We note that

1

T

T−1∑k=0

γ(k) = Cov(xT , xT ).

Thus

limT→∞1

T

T−1∑k=0

γ(k) = 0 ⇐⇒ limT→∞Cov(xT , xT ) = 0

We have that a stationary process xt is mean-ergodic if and only ifas the sample size T is increased there is less and less correlation(covariance) between the sample mean xT and the last observationxT .


Ergodic Theorems

Corollary 1 (Sufficient condition for mean-ergodicity). Let xt be astationary process with mean µ and autocovariance function γx(k).If

limk→∞γx(k) = 0,

then xt is mean-ergodic.

Intuitively, this result tell us that if any two random variablespositioned far apart in the sequence are almost uncorrelated, thensome new information can be continually added so that the samplemean will approach the popolation mean.


Ergodic Theorems

Corollary 2. (Sufficient condition for mean-ergodicity) Let xt be astationary process with mean µ and and autocovariance functionγx(k). If

∞∑k=0

|γx(k)| <∞,

then xt is mean-ergodic.


Ergodic Theorems

If∞∑k=0

|γx(k)| <∞,

we say that the autocovariance function is absolutely sommable


Ergodicity under Gaussianity

Let {xt ; t ∈ Z} be a stationary process with mean µ andautocovariance function γx(k). If the process is Gaussian, then1. the condition of absolute summability of covariance function

∞∑k=0

|γx(k)| <∞

is sufficient to ensure ergodicity for all moments.2. the condition

limk→∞γx(k) = 0

is necessary and sufficient.


Ergodicity under Gaussianity

Heuristically, a Gaussian stationary process is ergodic if and only ifany two random variables positioned far apart in the sequence arealmost independently distributed. That is, for sufficiently large k ,xt and xt−k are nearly independent.


Ergodic processes

Conclusion

For stationary ergodic processes, we do not need to observeseparate independent realizations of the process in order to obtaina consistent estimate of its mean value or other moments.

A good estimate of the moments of the process can be obtainedconsidering only one sufficiently long realization of the process.


Documents

Lesson 6: Estimation of the Autocovariance Function of a ... · Lesson 6: Estimation of the Autocovariance Function of a Stationary Process Umberto Triacca Dipartimento di Ingegneria