1Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Marcel DettlingInstitute for Data Analysis and Process Design

Zurich University of Applied Sciences

marcel.dettling@zhaw.ch

http://stat.ethz.ch/~dettling

ETH Zürich, May 9, 2011

2Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Prewhitening the Gas Furnace DataWhat to do:

- AR(p)-models are fitted to the differenced series

- The residual time series are Ut and Vt, white noise

- Check the sample cross correlation (see next slide)

- Model the output as a linear combination of pastinput values: transfer function model.

3Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Prewhitening the Gas Furnace Data

0 5 10 15 20

-0.5

0.0

0.5

1.0

Lag

AC

F

PrW.Out

0 5 10 15 20

-0.5

0.0

0.5

1.0

Lag

PrW.Out & PrW.Inp

-20 -15 -10 -5 0

-0.5

0.0

0.5

1.0

Lag

AC

F

PrW.Inp & PrW.Out

0 5 10 15 20

-0.5

0.0

0.5

1.0

Lag

PrW.Inp

4Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Transfer Function ModelsProperties:

- Transfer function models are an option to describe the dependency between two time series.

- The first (input) series influences the second (output) one, but there is no feedback from output to input.

- The influence from input to output only goes „forward“.

The model is:

2, 2 1, 10

( )t j t j tj

X X E

5Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Transfer Function ModelsThe model is:

- E[Et]=0.

- Et and X1,s are uncorrelated for all t and s.

- Et and Es are usually correlated.

- For simplicity of notation, we here assume that the series have been mean centered.

2, 2 1, 10

( )t j t j tj

X X E

6Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Cross CovarianceWhen plugging-in, we obtain for the cross covariance:

- If only finitely many coefficients are different from zero, we could generate a linear equation system, plug-inand to obtain the estimates .

This is not a statistically efficient estimation method.

21 2, 1, 1, 1, 110 0

( ) ( , ) , ( )t k t j t k j t jj j

k Cov X X Cov X X k j

1̂21̂ ˆ j

7Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Special Case: X1,t UncorrelatedIf X1,t was an uncorrelated series, we would obtain thecoefficients of the transfer function model quite easily:

However, this is usually not the case. We can then:

- transform all series in a clever way- the transfer function model has identical coefficients- the new, transformed input series is uncorrelated

see blackboard for the transformation…

21

11

( )(0)kk

8Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Gas Furnace Transformed

0 5 10 15 20

-0.5

0.0

0.5

1.0

Lag

AC

F

pr.Out

0 5 10 15 20

-0.5

0.0

0.5

1.0

Lag

pr.Out & pr.Inp

-20 -15 -10 -5 0

-0.5

0.0

0.5

1.0

Lag

AC

F

pr.Inp & pr.Out

0 5 10 15 20

-0.5

0.0

0.5

1.0

Lag

pr.Inp

9Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Gas Furnace: Final Remarks• In the previous slide, we see the empirical cross correlations

of the two series and .

• The coefficients from the transfer function model will beproportional to the empirical cross correlations. We can al-ready now conjecture that the output is delayed by 3-7 times, i.e. 27-63 seconds.

• The formula for the transfer function model coefficients is:

21ˆˆ ˆ ( )ˆ

Zk

D

k

21ˆ ( )k tD tZ

ˆk

10Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

State Space ModelsBasic idea: There is a stochastic process/time series

which we cannot directly observe, but only under the addition of some white noise.

Thus: We observe the time series

where , i.i.d.

Example: = # of fish in a lake= # estimated number of fish from a sample

tZ

t t tY Z U 2~ (0, )tU N

tZtY

11Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Notation and Terminology for an AR(1)We assume that the true underlying process is an AR(1), i.e.

,

where

are i.i.d. innovations, „process noise“.

In practice, we only observe , as realizations of the process

, with , i.i.d.

and additionally, the are independent of , for all s,t, thus they are independent „observation white noise“.

1 1t t tZ a Z E

t t tY Z U 2~ (0, )tU N

ty

2~ (0, )t EE N

tU sEsZ

12Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

More TerminologyWe call

the „state equation“, and

the „observation equation“.

On top of that, we remember once again that the „process noise“ is an innovation that affects all future values and thus also , whereas only influences the currentobservation , but no future ones.

1 1t t tZ a Z E

t t tY Z U

tYtU

tE t kZ

t kY

13Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

AR(1)-Example with α=0.7

Time

yt1

0 20 40 60 80 100

-2-1

01

2

Zustands-Prozess X_tBeobachtungs-Prozess Y_t

AR(1)-Example

14Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

ACF/PACF of Zt

Time

serie

s

0 20 40 60 80 100

-1.0

-0.5

0.0

0.5

1.0

-0.2

0.4

1.0

Lag k

Aut

o-K

orr.

0 5 10 15 20

-0.2

0.2

0.6

Lag k

part.

Aut

okor

r

1 5 10 15 20

15Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

ACF/PACF of Yt

Time

serie

s

0 20 40 60 80 100

-2-1

01

2-0

.20.

41.

0

Lag k

Aut

o-K

orr.

0 5 10 15 20

-0.2

0.1

Lag k

part.

Aut

okor

r

1 5 10 15 20

16Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

AR(1)-Example with α=1

Time

yt1

0 20 40 60 80 100

-20

24

6

Zustands-Prozess X_tBeobachtungs-Prozess Y_t

AR(1)-Example

17Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

ACF/PACF of Zt

Time

serie

s

0 20 40 60 80 100

-10

12

34

5-0

.20.

41.

0

Lag k

Aut

o-K

orr.

0 5 10 15 20

-0.2

0.4

1.0

Lag k

part.

Aut

okor

r

1 5 10 15 20

18Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

What is the goal?The goal of State Space Modeling/Kalman Filtering is:

To uncover the „de-noised“ process Zt from theobserved process Yt.

• The algorithm of Kalman Filtering works with non-stationary time series, too.

• The algorithm is based on a maximum-likelihood-principle where one assume normal distortions.

• There are extensions to multi-dimensional state spacemodels. This is partly discussed in the exercises.

19Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

Summary of Kalman FilteringSummary:

1) The Kalman Filter is a recursive algorithm

2) It relies on an update idea, i.e. we update theforecast with the difference .

3) The weight of the update is determined by therelation between the process variance and theobservation white noise .

4) This relies on the knowledge of . In practicewe have procedures for simultaneous estimation.

1,ˆ

t tZ 1 1,ˆ( )t t ty Y

2E

22 2, , ,Eg h

20

Applied Time Series AnalysisFS 2011 – Week 11Additional Remarks1) For the recursive approach of Kalman filtering, initial

values are necessary. Their choice is not crucial, theirinfluence cancels out rapidly.

2) The procedures yield forecast and filter intervals: and

3) State space models are a very rich class. Every ARIMA(p,d,q) can be written in state space form, andthe Kalman filter can be used for estimating thecoefficients.

4) We can also use Kalman filtering for smoothing, i.e.providing with T>t.

1, 1,ˆ 1.96t t t tZ R 1, 1 1, 1

ˆ 1.96t t t tZ R

1| TtZ Y

21Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

AR(1)-Example with α=0.7

Time

yt1

0 20 40 60 80 100

-2-1

01

2

Zustands-Prozess X_tBeobachtungs-Prozess Y_tFilter-Werte mit 95%-Filter-Intervall

AR(1)-Example with alpha=0.7

22Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

AR(1)-Example with α=1.0

Time

yt2

0 20 40 60 80 100

-3-2

-10

12

3

Zustands-Prozess X_tBeobachtungs-Prozess Y_tFilter-Werte mit 95%-Filter-Intervall

AR(1)-Example with alpha=1.0

23Marcel Dettling, Zurich University of Applied Sciences

Applied Time Series AnalysisFS 2011 – Week 11

AR(1)-Smoothing with α=1.0

Time

yt2

0 20 40 60 80 100

-3-2

-10

12

3

Zustands-Prozess X_tBeobachtungs-Prozess Y_tFilter-Werte mit 95%-Filter-Intervall

Smoothing instead of filtering