Analisis Bayesiano de Series Temporales

8/10/2019 Analisis Bayesiano de Series Temporales

1/17

Analisis Bayesiano de Series Temporales


Raquel Prado

Universidad de California, Santa Cruz

Julio, 2013


Definitions

BASIC DEFINITIONS AND INFERENCE


Definitions

Applications and Objectives

Univariate time series analysisModeling and inference: Describing the latent structure of a single series

time

0 1000 2000 3000

-4

00

-300

-200

-100

0

100

200


Definitions


Multivariate time series analysis

What if we have multiple time series or a time series vector,yt= (y1,t, . . . , yk,t)

,at each time t?For instance, the electroencephalogram (EEG) time series just

displayed is one of several EEGs recorded at different locations

over the scalp of a patient. We are interested in discovering

common features accross the multiple EEG signals.


2/17


Definitions


Univariate and multivariate time series analysisForecasting. Example: Annual per capita gross domestic product (GDP).

1950 1960 1970 1980

0.0

4

0.

08

Austria

1950 1960 1970 1980

4

5

6

7

8

Canada

1950 1960 1970 1980

10

20

30

France

1950 1960 1970 1980

6

10

14

18

Germany

1950 1960 1970 1980

20

40

60

80

Greece

1950 1960 1970 1980

1.0

2.

0

Italy

1950 1960 1970 1980

20

30

Sweden

1950 1960 1970 1980

1.

0

1.4

1.8

UK

1950 1960 1970 1980

5

6

7

8

USA


Definitions


Online monitoring and control

Monitoring a time series to detect possible changes in real time.

Example: TAR(1)

yt= (1)yt1 +

(1)t , +ytd>0 (M1)

(2)

yt1 +

(2)

t , +ytd 0 (M2),where

(1)t N(0, v1)and (2)t N(0, v2).


Definitions


y1:T and1:T

0 500 1000 1500

4

0

2

4

6

time(a)

0 500 1000 1500

1.0

1.4

1.8

time(b)

Here(1) =0.9, (2) = 0.3,d=1, = 1.5,and v1= v2= 1.Also,t=1 ifyt M1 andt=2 ifyt M2.


Definitions


Goals of time series analysis in the exampleOnline monitoring and control

If transitions between modes occur in response to acontrollers action1:Tis known and so, we can:

infer the parameters of the stochastic models that controlthe settings, i.e., infer(i) andvi,and

learn about transition rule, i.e., infer and d;

If transitions do not occur in response to a controllers

action we need to make inferences on1:Tas well.


3/17


Definitions


Other goals

Clustering

Time series models as components of models with

additional structure: e.g., regression models,

spatio-temporal models, etc.

Tracking and online learning.


Definitions

Stationarity

Stationarity

Definition{yt, t T } iscompletely or strongly stationaryif, for anysequence of timest1, . . . , tnand any lag h,the distribution of(yt1 , . . . , ytn)

is identical to the distribution of (yt1+h, . . . , ytn+h).

Definition{yt, t T } isweakly or second order stationaryif for anysequencet1, . . . , tn,and anyh,the first and second jointmoments of(yt1 , . . . , ytn)

and those of(yt1+h, . . . , ytn+h) exist

and are identical.

Complete stationarity implies second order stationarity but the

converse is not necessarily true.


Definitions

Stationarity

StationaritySecond order stationarity

If {yt} is weakly stationary E(yt) = ,V(yt) =v,andCov(yt, ys) =(s t).

Gaussian time series processes: strong and weak

stationarity are equivalent.


Definitions

Stationarity

Stationarity

time

0 500 1000 1500 2000

-300

-200

-100

0

100

200

time

0 50 100 150 200

-300

-200

-100

0

100

200

time

0 50 100 150 200

-300

-200

-100

0

100

200


4/17


Definitions

The ACF

The autocorrelation function (ACF)

DefinitionThe autocovarianceof {yt} is defined as

(s, t) = Cov(yt, ys) =E{(yt t)(ys s)}.

If {yt} is stationary we can write (h) =Cov(yt, yth).Definition

Theautocorrelation function(ACF) is given by

(s, t) = (s, t)(s, s)(t, t)

.

For stationary processes we can write (h) =(h)/(0).


Definitions

The ACF

The sample autocorrelation function

DefinitionThesample autocovarianceis given by

(h) = 1

T

Tht=1

(yt y)(yt+h y),

wherey= Tt=1 yt/T is the sample mean.Definition

Thesample autocorrelationis given by (h) = (h)(0) .


Definitions

The ACF

The ACF: Examples

White Noise. Letyt N(0, v)for allt,withCov(yt, ys) =0 ift=s.Then,(0) =v,(h) =0 for allh

=0, and so,(0) =1

and(h) =0 for allh =0.

AR(1). Letyt=yt1+ t, t N(0, v).Then,

(0) = v

(1 2) ,

(h) = |h|(0).


Definitions

The ACF

ACF of AR(1)

0 10 20 30 40 50

1.0

0.

5

0.0

0

.5

1.

0

h

= 0.9

= 0.9

= 0.3


5/17


Definitions

The ACF

Sample ACF of AR(1)

0 5 10 15 20

0.0

0.2

0.4

0.6

0.8

1.0

(a)Lag

ACF

= 0.9

0 5 10 15 201.0

0.5

0.0

0.5

1.0

(b)Lag

ACF

= 0.9

0 5 10 15 20

0.0

0.2

0.4

0.6

0.8

1.0

(c)Lag

ACF

= 0.3


ML and Bayesian Inference

Bayes theorem: Independent Observations

p(|y1:T) =

likelihood p(y1:T|)

priorp()

p(y1:T) predictive

,

with

p(y1:T) = p(y1:T|)p()d.Alternatively, we can write:

p(|y1:T) p(yT|y1:(T1),) p(|y1:(T1)) p(yT|)

likelihood

p(|y1:(T1)) predictive

.



Bayes theorem: Dependence on(t 1)

p(|y1:T) p()prior

p(y1|)T

t=2

p(yt|yt1,) likelihood

.



Bayes theorem: Dependence on(t 1)Example

AR(1): yt=yt1+ t, t N(0, v),and so = (, v). Conditional likelihood: p(yt

|yt1,) =N(yt

|yt1, v);

p(y1|) =N(0, v/(1 2));Then,

p(|y1:T) p()

(1 2)(2v)T/2

exp

Q

()

2v

,

with

Q() = y21 (1 2) +T

t=2

(yt yt1)2 Q()


6/17



Bayes theorem: Dependence on(t 1)

Example

AR(1)(cont.): We can also use theconditional likelihoodp(y2:T|, y1)as an approximation to the full likelihood andobtain the posterior

p(|y1:T) p()v(T1)/2expQ()2v



Estimation

Maximum likelihood estimation (MLE):Find = MLE thatmaximizes p(y1:T|).

Maximum a posteriori estimation (MAP):Find= MAPthat maximizesp(|y1:T).

Least squares estimation (LSE):Write the model as

y= F+ , N(0, vI).

with dim(y) = nand dim() =pso that

p(y|F,, v) = (2v)n/2exp(Q(y,)/2v) ,

and find that minimizesQ(y,).



Bayesian Estimation

Consider again the modely = F+ ,with N(0, vI).Theposterior density is given by

p(, v|y) p(, v) p(y|, v) p(, v) (2v)n/2exp(Q(y,)/2v)

where

Q(, y) = (y F)(y F) = ( )(FF)( ) +R,

with = (FF)1Fyand R= (y F)(y F). The MLE of is ; The MLE ofv isR/n,however,s2 =R/(n p)is used

instead.



Bayesian Estimation

Reference prior:p(, v) 1/v

p(|y, F)is Student-t withn pdegrees of freedom, modeand density

p(|y, F) |FF|1/2

1 + ( )FF( )/(n p)s2n/2

Whenn p(|y, F) N(|, s2(FF)1). p(v|y) =IG

(np)2 ,

(np)s2

2 .


7/17



Bayesian EstimationConjugate Prior:

p(, v) = p(|v)p(v) = N(|m0, vC0) IG(v|n0/2, d0/2)

p(, v|F, y) v{(p+n+n0)/2+1}

e(m0)C

1

0 (m0)+(yF

)(yF

)+d02v

(y|F, v) N(Fm0, v(FC0F + In)); (|F, v) N(m, vC),with

m = m0+ C0F[FC0F + In]

1(y Fm0)C = C0 C0F[FC0F + In]1FC0,



Bayesian Estimation (Conjugate prior)

(v|F, y) IG(n/2, d/2)withn =n+n0 andd =eQ1e +d0,with

e= (y Fm0), and Q= (FC0F + I).

(|y1:n, F) Tn [m, dC/n].



Estimation

Example

ML, MAP, and LS estimators for the AR(1) model.

yt=yt1+ t, witht

N(0, 1). In this case = .

The conditional MLE is found by maximizing

exp{ Q()/2} (or by minimizingQ()). Therefore,= ML=

nt=2 ytyt1/

nt=2 y

2t1.

MLE of unconditional likelihood is obtained by maximizing

p(y1:n|)or by minimizing

0.5[log(1 2) Q()].

We need methods such as Newton-Raphson or scoring to

findML.



AR(1)conditional and unconditional likelihoods; simulateddata with = 0.9;MLEs = 0.9069and = 0.8979.

0.6 0.7 0.8 0.9 1.0

140

120

100

80

60


8/17



AR(1)conditional and unconditional posteriors with priors

N(0, c),c= 1 and c=0.01

0.6 0.7 0.8 0.9 1.0

140

120

100

80

60

40

0.6 0.7 0.8 0.9 1.0

140

120

100

80

60

40



Bayesian Estimation (Conjugate Analysis)

Reference analysis in the AR(1) model. For the conditional likelihoodML=

nt=2 yt1yt/

n1t=1 y

2t .

Under the reference priorMAP= ML.

Also,

R=n

t=2

y2t(n

t=2 ytyt1)2

n1t=1 y

2t

,

and sos2

=R/(n 2)estimatesv. Marginal posterior for : Student-t withn 2 degrees of

freedom, centered atMLwith scales2(FF)1.

Marginal posterior forv :Inv 2(v|n 2, s2)or,equivalently,(n 2)s2/2IG(v|(n 2)/2, (n 2)s2/2).



AR(1)reference analysis; 500 simulated observations with= 0.9and v=100.

Frequency

0.84 0.8 8 0.92 0 .96

0

200

400

600

800

1000

(a)v

Frequency

90 100 120 140

0

200

400

600

800

1000

1200

1400

(b)



Bayesian Estimation: Non-Conjugate Analysis

AR(1)with full likelihood: The prior p(, v) 1/vdoes notlead to a closed form posterior distribution when the full

likelihood is used. We obtain

p(, v|y1:n) v(n/2+1)(1 2)1/2expQ()

2v

.

How can we summarize posterior inference in this case?

Via simulation-based methods such as Markov chain

Monte Carlo...


9/17



MCMC: The Metropolis Hastings Algorithm

Creates a sequence of random draws,

(1)

,

(2)

, . . . ,whosedistributions converge to the target distribution, p(|y1:n).1. Draw(0) withp((0)|y1:n)> 0 fromp0().2. Form= 1, 2, . . . ,(until convergence):

(a) Sample J(|(m1))(b) Compute the importance ratio

r= p(|y1:n)/J(|(m1))p((m

1)|y1:n)/J((m1)|).

(c) Set

(m) =

with probability= min(r, 1)

(m1) otherwise.



MCMC: AR(1)case

MCMC for AR(1)with full likelihood.

Samplev(m) from(v|, y1:n) IG(n/2, Q()/2)(Gibbsstep, every draw will be accepted)

Sample

N(m1), c .



MCMC: AR(1)example

0 200 400 600 800 1000

0.0

0.2

0.4

0.6

0.8

1.0

iteration

(a)

0 200 400 600 800 1000

0.0

0.5

1

.0

1.5

2.0

iteration

v

(b)

Frequency

0.86 0.90 0.94

0

20

40

60

80

v

Frequency

1.00 1.05 1.10 1.15

0

20

40

60

80


Time Domain Models

Autoregressions

AR(p)Models

An autoregression of order p,or AR(p),has the form

yt=

p

j=1

jytj+ t,

wheretis a sequence of uncorrelated error terms typicallyassumed Gaussian, i.e.,t N(0, v).Under Gaussianity, ify = (yT, yT1, . . . , yp+1)

,we have

p(y

|y1:p) =

T

t=p+1 p(yt|y(tp):(t1)) =T

t=p+1 N(yt|ft, v) = N(y

|F, vIn)

with = (1, . . . , p),ft= (yt1, . . . , ytp)

,F = [fT, . . . , fp+1].


10/17


Time Domain Models

Autoregressions

AR Models: Causality and Stationarity

DefinitionAn AR(p)processyt iscausalif it can be written as

yt= (B)t=

j=0

jtj,

withBthe backshift operator Bt=t1, 0= 1 and

j=0 |j| < .DefinitionTheAR characteristic polynomialis defined as:

(u) =1 p

j=1

juj.


Time Domain Models

Autoregressions

AR Models: Causality and Stationarity

ytis causal only when(u)has all its roots outside the unitcircle (or the reciprocal roots inside the unit circle). In other

words,yt is causal if(u) =0 only when |u| >1. Causality

Stationarity.


Time Domain Models

Autoregressions

AR Models: State-space representation

yt AR(p)can be written as

yt = Fxt

xt = Gxt1+ t,

withxt= (yt, yt1, . . . , ytp+1), t= (t, 0, . . . , 0)

,F= (1, 0, . . . , 0) and

G=

1 2 3 p1 p1 0 0 0 00 1 0 0 0... . . . 0

...

0 0 1 0

.


Time Domain Models

Autoregressions

AR Models: State-space representation

The eigenvalues of the matrix G, 1, . . . , pare thereciprocal roots of the AR characteristic polynomial theAR characteristic polynomial can written as:

pj=1

(1 ju).

The expected behavior of the process in the future is given

by

ft(h) =E(yt+h|y1:t) = FGhxt=p

j=1ct,j

hj,

withct,j=djet,j,and dj,et,jelements ofd = EF,and

et=E1xt,whereE is an eigenmatrix of G.


11/17


Time Domain Models

Autoregressions

AR Models: Forecast function

Ifyt is such that |j|


12/17


Time Domain Models

Autoregressions

AR Models: PACF

Let(h, h)be thepartial autocorrelation coefficient at lag h,given by

(h, h) =

(y1, y0) =(1) h= 1

(yh yh1h , y0 yh10 ) h> 1,

withyh1h the minimum mean square linear predictor ofyhgiven

yh1, . . . , y1,and yh10 the minimum mean square linearpredictor ofy0 giveny1, . . . , yh1.

Result:Ifyt AR(p), (h, h) =0 for allh> p.


Time Domain Models

Autoregressions

AR Models: Computing the PACF

n

n=

n,with

nann

nmatrix with elements

{(hj)}nj=1, n= ((1), . . . , (n)),andn= ((n, 1), . . . , (n, n))

.

Durbin-Levinson recursion. Forn= 0 (0, 0) = 0 and forn 1

(n, n) =(n) n1h=1 (n 1, h)(n h)

1

n1h=1 (n

1, h)(h)

,

with

(n, h) =(n 1, h) (n, n)(n 1, n h),

forn 2 andh= 1 : (n 1).Sample PACF can also be computed using these algorithms.


Time Domain Models

Autoregressions

AR Models: Yule-Walker Estimation

p= p, v= (0) p1p p.

It can be shown that

T( ) N(0, v1p ),

and thatv is close tov whenT is large.


Time Domain Models

Autoregressions

AR Models: MLE and Bayesian estimation

MLE.Find that maximizes

p(y|, v, y1:p) =T

t=p+1p(yt|, v, y(tp):(t1))

=T

t=p+1

N(yt|ft, v) = N(y|F, vIn).

Bayesian. Combinep(y|, v, y1:p)with priorp(, v). Reference priorp(, v) 1/v.

Conjugate priorp(|v) =N(|m0, vC0)andp(v) =IG(n0/2, d0/2). Non-conjugate.


13/17


Time Domain Models

Autoregressions

AR Models: EEG data analysis

time

voltage(mcv)

0 100 200 300 400

-300

-200

-100

0

100

200

Posterior mean from AR(8) reference

analysis (n= 392):

= (0.27, 0.07, 0.13, 0.15,0.11, 0.15, 0.23, 0.14)

ands= 61.52.These estimates leadto the following estimates of thereciprocal characteristic roots:

(0.97, 12.73); (0.81, 5.10);(0.72, 2.99); (0.66, 2.23).


Time Domain Models

Autoregressions

AR Models: EEG data analysisForecast function

time

voltage(mcv)

0 100 200 300 400 500 600

-300

-200

-100

0

100

200

Future sample

time

voltage(mcv)

0 100 200 300 400 500 600

-300

-200

-100

0

100

200


Time Domain Models

Autoregressions

AR Models: Model Order Assessment

Choose a valuep and for allp p compute Akaikes Information Criterion (AIC):

2p+nlog(s2

p).

Bayesian Information Criterion (BIC):

log(n)p+nlog(s2p).

Marginal:

p(y(p

+1):T|y1:p , p) = p(y(p+1):T|p, v, y1:p)p(p, v)dpdv.Heren= T p.


Time Domain Models

Autoregressions

Order Assessment in EEG Example: Takep =25 andn= 400 p.

mm

m

m

m

m

m m m m mm

mm

m

m mm

mm m

mm

mm

a a

a

a

a

a

aa a

a a a a a a a a a a a a a a

a a

b

b

b

b

b

b

b bb b

bb

b b

bb

b

bb

bb

bb

bb

AR order p

log-

likelihood

5 10 15 20 25

-60

-50

-40

-30

-20

-10

0


14/17


Time Domain Models

Autoregressions

AR Models: Initial ObservationsFull likelihood:

p(y1:T|, v) = p(y(p+1):T|, v, y1:p)p(y1:p|, v)= p(y|, v, xp)p(xp|, v).

What aboutp(xp|, v)? N(xp|0, A)withA known. N(xp|0, vA())withA()depending on through the

autocorrelation function and

p(y1:T|, v) vT/2|A()|1/2exp(Q(y1:T,)/2v),where

Q(y1:T,) =T

t=p+1

(yt ft)2 + xpA()1xp.


Time Domain Models

Autoregressions

AR Models: Initial ObservationsIt can be shown (e.g., see Box, Jenkins, and Reinsel, 2008) that

Q(y1:T,) =a 2b + C,witha, b,and C obtained from

D=

a bb C

,

andD a (p+1)

(p+1)withDij= T+1jir=0 yi+ryj+r. If |A()|1/2 is ignored when computing p(y1:T|, v),the

likelihood function is that of a standard linear model form

and so, ifp(, v) 1/vwe have a normal/inverse-gammaposterior with

=C1b.

Jeffreys prior is approximatelyp(, v) |A()|1/2v1/2.Analisis Bayesiano de Series Temporales

Time Domain Models

Autoregressions

AR Models: Structured non-conjugate priors

Ifyt AR(p),ytis causal and stationary if all the AR reciprocalroots have moduli less than one.

Huerta and West (1999) proposed priors on the reciprocalcharacteristic roots as follows.

LetCbe the maximum number of pairs of complex roots

andRthe maximum number of real roots with p= 2C+ R.

Denote the complex roots as (rj, j),forj=1 : Cand thereal roots asrj,forj= (C+1) : (R+C).

Then...


Time Domain Models

Autoregressions

AR Models: Structured non-conjugate priors Prior on the real reciprocal roots.

rj r,1I(1)(rj) + c,0I0(rj) + r,1I1(rj) ++(1

r,0

r,1

r,1)gr(rj),

withgr()a continuous distribution on(1, 1), e.g.,gr() =U(| 1, 1).

Prior on the complex reciprocal roots.

rj c,0I0(rj) + c,1I1(rj) + (1 c,1 c,0)gc(rj),j h(j),

withgc(rj)and h(j)continuous distributions on 0< rj


15/17


Time Domain Models

ARMA Models

ARMA Models

ytfollows an autoregressive moving average model,ARMA(p, q),if

yt =

pi=1

iyti+

qj=1

jtj+ t,

We can also write

(1 1B . . . pBp) (B)

yt = (1 + 1B+ . . . + qBq) (B)

t

We typically assumet N(0, v).If q= 0 yt AR(p)and ifp= 0 yt MA(q).


Time Domain Models

ARMA Models

ARMA Models

DefinitionA MA(q)process is identifiable or invertibleif the roots of theMA characteristic polynomial(u)lie outside the unit circle. Inthis case is possible to write the process as an infinite order AR.

Example

Letyt MA(1)with MA coefficient . The process is stationaryfor all and

(h) =

1 h= 0

(1+2) h= 1

0 otherwise.

Note that a MA process with coefficient 1/has the same ACF the identifiability condition is 1/ >1.


Time Domain Models

ARMA Models

An ARMA(p, q)process iscausalif the roots of (u)lie outsidethe unit circle. In this case:

yt= 1(B)(B)t= (B)t=

j=0

jtj,

with(B)(B) = (B).The js can be found by solving thehomogeneous difference equations

jp

h=1

hjh= 0, j max(p, q+1),

with initial conditions

jj

h=1

hjh=j, 0 j


16/17


17/17


Time Domain Models

ARMA Models

MA coefficient index

coefficient

0 2 4 6 8

-0.5

0.0

0.5

1.0

1.5

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

-

Note:the optimal ARMA(p, q)model for these data, among allthe models withp, q 8,is an ARMA(2, 2).The MLEs for theMA coefficients are 1= 1.37 and 2= 0.51.


Time Domain Models

ARMA Models

ARMA Models: Inference

MLE and least squares estimation. See, e.g., Shumway

and Stoffer, 2006.

Inference via state-space representation. E.g., Kohn and

Ansley (1985), Harvey (1981, 1991).

Bayesian estimation: Monahan (1983); Marriott & Smith

(1992); Chib and Greenberg (1994); Box, Jenkins, and

Reinsel (2008); Zellner (1996); Marriott, Ravishanker,Gelfand, and Pai (1996); Barnett, Kohn and Sheather

(1997) among others...


Time Domain Models

ARMA Models

Other Related Models

ARIMAyt ARIMA(p, d, q)

(1 1B . . . pBp

)(1 B)d

yt= (1 + 1B+ . . . + qBq

)t.

SARMA

(1 1Bs . . . pBps)yt= (1 + 1Bs + . . . + qBqs)t. Multiplicative Seasonal ARMA

p(B)P

(Bs)(1

B)dyt= q(B)Q

(Bs)t.

Documents

Analisis Bayesiano de Series Temporales