Lecture 4, Non-linear Time SeriesMagnus Wiktorsson

“It’s not a bug, it’s a feature!”

▶ Why are we using linear models?▶ Properties▶ Limitations

▶ Properties of non-linear systems.▶ Limit cycles▶ Jumps▶ Non-symmetric distributions▶ Bifurcations▶ Chaos▶ Non-linear dependence

“It’s not a bug, it’s a feature!”

▶ Why are we using linear models?▶ Properties▶ Limitations

▶ Properties of non-linear systems.▶ Limit cycles▶ Jumps▶ Non-symmetric distributions▶ Bifurcations▶ Chaos▶ Non-linear dependence

General properties

▶ Assume causal system

f(Yn,Yn−1, . . . ,Y1) = εn

▶ Invertable system

Yn = f⋆(εn, . . . , ε1)

▶ Volterra series.

Suppose that f∗ is sufficientlywell-behaved, then there exists a sequence of boundedfunctions∞∑

k=0|ψk| <∞,

∞∑k=0

∞∑l=0

|ψkl| <∞,∞∑

k=0

∞∑l=0

∞∑m=0

|ψklm| <∞, ...

General properties

▶ Assume causal system

f(Yn,Yn−1, . . . ,Y1) = εn

▶ Invertable system

Yn = f⋆(εn, . . . , ε1)

▶ Volterra series. Suppose that f∗ is sufficientlywell-behaved, then there exists a sequence of boundedfunctions∞∑

k=0|ψk| <∞,

∞∑k=0

∞∑l=0

|ψkl| <∞,

∞∑k=0

∞∑l=0

∞∑m=0

|ψklm| <∞, ...

Volterra serieswhere

µ = f∗(0), ψk = (∂f∗∂ϵt−k

), ψkl = (∂2f∗

∂ϵt−k∂ϵt−l), ... (1)

Approximate the general model by

Yt = µ+∞∑

k=0ψkϵt−k +

∞∑k=0

∞∑l=0

ψklϵt−kϵt−l

+∞∑

k=0

∞∑l=0

∞∑m=0

ψklmϵt−kϵt−lϵt−m + . . . (2)

This results in generalized transfer functions. NOTE thatsuperposition is lost!

These transfer functions do not care if {ϵ} is deterministicof stochastic!

Volterra serieswhere

µ = f∗(0), ψk = (∂f∗∂ϵt−k

), ψkl = (∂2f∗

∂ϵt−k∂ϵt−l), ... (1)

Approximate the general model by

Yt = µ+∞∑

k=0ψkϵt−k +

∞∑k=0

∞∑l=0

ψklϵt−kϵt−l

+∞∑

k=0

∞∑l=0

∞∑m=0

ψklmϵt−kϵt−lϵt−m + . . . (2)

This results in generalized transfer functions.

NOTE thatsuperposition is lost!

These transfer functions do not care if {ϵ} is deterministicof stochastic!

Volterra serieswhere

µ = f∗(0), ψk = (∂f∗∂ϵt−k

), ψkl = (∂2f∗

∂ϵt−k∂ϵt−l), ... (1)

Approximate the general model by

Yt = µ+∞∑

k=0ψkϵt−k +

∞∑k=0

∞∑l=0

ψklϵt−kϵt−l

+∞∑

k=0

∞∑l=0

∞∑m=0

ψklmϵt−kϵt−lϵt−m + . . . (2)

This results in generalized transfer functions. NOTE thatsuperposition is lost!

These transfer functions do not care if {ϵ} is deterministicof stochastic!

Frequency doubling

Now assume that we introduce a spectral representation ofthe noise.▶ Let’s start with a single frequency, ϵk = A exp(iω ∗ k)

▶ This results in frequency doubling▶ Proof by inserting the signal in Eq (2).▶ Question: What happens with a non-linear system if

the noise ϵk is white noise?▶ Conclusion: Black box non-linear system identification

is far more complicated that linear systemidentification.

Frequency doubling

Now assume that we introduce a spectral representation ofthe noise.▶ Let’s start with a single frequency, ϵk = A exp(iω ∗ k)▶ This results in frequency doubling▶ Proof by inserting the signal in Eq (2).

▶ Question: What happens with a non-linear system ifthe noise ϵk is white noise?

▶ Conclusion: Black box non-linear system identificationis far more complicated that linear systemidentification.

Frequency doubling

Now assume that we introduce a spectral representation ofthe noise.▶ Let’s start with a single frequency, ϵk = A exp(iω ∗ k)▶ This results in frequency doubling▶ Proof by inserting the signal in Eq (2).▶ Question: What happens with a non-linear system if

the noise ϵk is white noise?

▶ Conclusion: Black box non-linear system identificationis far more complicated that linear systemidentification.

Frequency doubling

Now assume that we introduce a spectral representation ofthe noise.▶ Let’s start with a single frequency, ϵk = A exp(iω ∗ k)▶ This results in frequency doubling▶ Proof by inserting the signal in Eq (2).▶ Question: What happens with a non-linear system if

the noise ϵk is white noise?▶ Conclusion: Black box non-linear system identification

is far more complicated that linear systemidentification.

Regime models

The model is generated from a set of simple models▶ SETAR▶ STAR▶ HMM

SETAR - Self-Exciting Threshold AR

The SETAR(l; d; k1, k2, . . . , kl) model is given by :

Yt = a(Jt)0 +

kJt∑i=1

a(Jt)i Yt−i + ϵ

(Jt)t (3)

where the index (Jt) is described by

Jt =

1 for Yt−d ∈ R12 for Yt−d ∈ R2... ...l for Yt−d ∈ Rl.

(4)

NOTE that it is difficult to estimate the boundaries for theregimes

SETAR - Self-Exciting Threshold AR

The SETAR(l; d; k1, k2, . . . , kl) model is given by :

Yt = a(Jt)0 +

kJt∑i=1

a(Jt)i Yt−i + ϵ

(Jt)t (3)

where the index (Jt) is described by

Jt =

1 for Yt−d ∈ R12 for Yt−d ∈ R2... ...l for Yt−d ∈ Rl.

(4)

NOTE that it is difficult to estimate the boundaries for theregimes

SETARMA

▶ Similar ideas can be included in ARMA models,leading to SETARMA models.

▶ Often easy to add ’asymmetric’ terms in the AR orMA polynomials, e.g.

yn + a1yn−1 = en +(c1 + c′11{en−1≤0}

)en−1

STAR - Smooth Threshold ARThe STAR(k) model:

Yt = a0+k∑

j=1ajYt−j+

b0 +k∑

j=1bjYt−j

G(Yt−d)+ϵt (5)

where G(Yt−d) now is the transition function lying betweenzero and one, as for instance the standard Gaussiandistribution.In the literature two specifications for G(·) are commonlyconsidered, namely the logistic and exponential functions:

G(y) = (1 + exp(−γL(y − cL)))−1; γL > 0 (6)

G(y) = 1 − exp(−γE(y − cE)2); γE > 0 (7)

where γL and γE are transition parameters, cL and cE arethreshold parameters (location parameters).

PJM electricity market

Prices at the PJM market

Simple model of the power market

▶ Demand

D(Q) = a + bQ + c cos(2πt/50) + ε (8)

▶ Supply

S(Q) = α0+β0Q+G(Q,Qbreak)(α1+β1(Q−Qbreak)+)(9)

where G is a transition function.▶ Solve numerically for t = 1, . . . to get the quantity Q

and price P.

Supply and Demand

50 60 70 80 90 100 110 1200

100

200

300

400

500

600

700

800

900

1000

Supply

MaxDemand

MinDemand

Figure: Supply and demand curves (varies across the season) forour artificial market

Prices

0 50 100 150 200 250

250

300

350

400

450

500

550

600

650

700

750

Figure: Note the seasonality as well as the non-Gaussiandistribution.

Distribution of prices

300 350 400 450 500 550 600 650 700

Data

0.001

0.003

0.010.02

0.05

0.10

0.25

0.50

0.75

0.90

0.95

0.980.99

0.997

0.999

Pro

ba

bili

ty

Normal Probability Plot

Figure: Same property

HMM - Hidden Markov Models

Another alternative is to let the regime shift stochastically,as in the Hidden Markov Model. Let

Yt = a(Jt)0 +

kJt∑i=1

a(Jt)i Yt−i + ϵ

(Jt)t (10)

where the state variable Jt follows a latent Markov chain.

NOTE that parameter estimation is slightly morecomplicated than before.

HMM - Hidden Markov Models

Another alternative is to let the regime shift stochastically,as in the Hidden Markov Model. Let

Yt = a(Jt)0 +

kJt∑i=1

a(Jt)i Yt−i + ϵ

(Jt)t (10)

where the state variable Jt follows a latent Markov chain.

NOTE that parameter estimation is slightly morecomplicated than before.

Case: Electricity spot price, (Regland &Lindström, 2012)

The electricity spot price is very non-Gaussian

Feb05 Feb07 Feb09

0

50

100

150

200

250

EEX spot

Feb05 Feb07 Feb09−4

−3

−2

−1

0

1

EEX log(spot)−log(forward)

Figure: The electricity spot price (left) and spread, defined as thedifference between the logarithm of the spot and the logarithmof the forward (right). Data from the German EEX market.

▶ The spread accounts for virtually all seasonality, butthere are still bursts of volatility.

▶ The logarithm of the spot, yt, was modeled using aHMM regime switching model with three states, anormal state with mean-reverting dynamics, a spike(upward jumps) state and a drop (downward jumps)state.

This is mathematically given by :

∆y(B)t+1 = α

(µt − y(B)

t

)+ σϵt

y(S)t+1 = ZS,t + µt, ZS ∼ F (µS, σS)

y(D)t+1 = −ZD,t + µt, ZD ∼ F (µD, σD)

where µt is approximately the logarithm of the monthahead forward price.The regimes are switching according to a Markov chainRt = {B,S,D} governed by the transition matrix

Π =

1 − πBU − πBD πBS πBDπSB 1 − πSB 0πDB 0 1 − πDB

.

Feb05 Feb06 Feb07 Feb08 Feb09 Feb10−4

−3

−2

−1

0

1

Spr

ead

Feb05 Feb06 Feb07 Feb08 Feb09 Feb10−1

0

1

Reg

ime

prob

Figure: Fit of the independent spike model applied to EEX data

Extension used for stability evaluation of the power systemin (Lindström, Norén & Madsen, 2015) by making thetransition matrix time-inhomogeneous.

Case: What happens with large scaleintroduction of electric cars/battery?

Jan02 Jan04 Jan06 Jan08 Jan10 Jan12

0.4

0.5

0.6

0.7

0.8

0.9

1

Mod

ified

Nor

mal

ized

Con

sum

ptio

n

0 %10 %

Battery capacity (%) 0 5 10 15Base prob. 0.8794 0.8827 0.9066 0.9461Spike prob. 0.0304 0.0292 0.0196 0.0081Drop prob. 0.0902 0.0881 0.0738 0.0458

Table: Unconditional regime probabilities when having a perfectbattery with 0, 5, 10, and 15 % system capacity.

HMMs for portfolio optimizationRecall the stylized facts for stock indices.

We can also useHMMs for portfolio optimization.▶ Model given by

Xt = µSt + εSt (11)

with µ1, µ2, σ1, σ2, π1 stat. prob. for the first stateand λ = γ11 + γ22 − 1 is the second largest eigenvalueto the transition matrix, Γ.

First, consider the autocorrelation (for k > 0):

r(k) = π1(1 − π1)(µ1 − µ2)2

σ21π1 + σ2

2(1 − π1) + π1(1 − π1)(µ1 − µ2)2λk

(12)See Nystrup et al (2016) for details.

HMMs for portfolio optimizationRecall the stylized facts for stock indices. We can also useHMMs for portfolio optimization.▶ Model given by

Xt = µSt + εSt (11)

with µ1, µ2, σ1, σ2, π1 stat. prob. for the first stateand λ = γ11 + γ22 − 1 is the second largest eigenvalueto the transition matrix, Γ.

First, consider the autocorrelation (for k > 0):

r(k) = π1(1 − π1)(µ1 − µ2)2

σ21π1 + σ2

2(1 − π1) + π1(1 − π1)(µ1 − µ2)2λk

(12)See Nystrup et al (2016) for details.

Simulation example

Here the model is given by

Γ =

[0.98 0.020.1 0.9

].

with µ = [0.01 − 0.02] and σ = [0.04 0.20].Interpretation of parameters: Staying on average1/(1 − 0.98) = 50 days in the good state vs 10 days in thebad state.

Realizations

100 200 300 400 500 600 700 800 900 1000

0

2

4

6

0 100 200 300 400 500 600 700 800 900 1000

1

1.5

2

0 100 200 300 400 500 600 700 800 900 1000

-1

0

1

Figure: Cumulative returns (top), Markov states (middle) andreturns (bottom).

Autocorrelation

0 2 4 6 8 10 12 14 16 18 20

Lag

-0.2

0

0.2

0.4

0.6

0.8

1

Sam

ple

Auto

corr

ela

tion

Sample Autocorrelation Function

0 2 4 6 8 10 12 14 16 18 20

Lag

-0.2

0

0.2

0.4

0.6

0.8

1

Sam

ple

Auto

corr

ela

tion

Sample Autocorrelation Function

Figure: Autocorrelation for returns (left) and abs returns (right)

Trading strategy

Figure: Trading strategy in US stocks and bonds

General State space models

A Grey box approach is to include as much prior knowledgeas possible.

Consider the General State Space model:

xn+1 = f(n, xn, un) + g(n, xn, un)en+1

yn+1 = h(n + 1, xn+1, un+1) + wn+1

where {xn}n≥0 is a latent process and {yn}n≥0 is thesequence of observations.▶ Interpretations?▶ Practical considerations

General State space models

A Grey box approach is to include as much prior knowledgeas possible.

Consider the General State Space model:

xn+1 = f(n, xn, un) + g(n, xn, un)en+1

yn+1 = h(n + 1, xn+1, un+1) + wn+1

where {xn}n≥0 is a latent process and {yn}n≥0 is thesequence of observations.▶ Interpretations?▶ Practical considerations

Example: The Black & Scholes model

The Black & Scholes (1973) model is often used for optionvaluation.

x :dS = µStdt + σStdWt,

y :

[SMarket

ncMarket

K (Sn, ·)

]=

[SModel

ncModel

K (Sn, ·)

]+ wn.︸︷︷︸

Ask-Bix spread

This structure allows us to separate actual price variationfrom market micro structure.

Some references

▶ Lindström, E., & Regland, F. (2012). Modelingextreme dependence between European electricitymarkets. Energy economics, 34(4), 899-904. http://dx.doi.org/10.1016/j.eneco.2012.04.006

▶ Lindström, E., Norén, V., & Madsen, H. (2015).Consumption management in the Nord Pool region: Astability analysis. Applied Energy, 146, 239-246.http://dx.doi.org/10.1016/j.apenergy.2015.01.113

▶ Nystrup, P., Hansen, B. W., Madsen, H., &Lindström, E. (2015). Regime-based versus staticasset allocation: Letting the data speak. The Journalof Portfolio Management, 42(1), 103-109.http://dx.doi.org/10.3905/jpm.2015.42.1.103

Cont.▶ Nystrup, P., Madsen, H., & Lindström, E. (2016). Long

memory of financial time series and hidden Markov modelswith time-varying parameters. Journal of Forecastinghttp://dx.doi.org/10.1002/for.2447

▶ Lindström, E., Ströjby, J., Brodén, M., Wiktorsson, M., &Holst, J. (2008). Sequential calibration of options.Computational Statistics & Data Analysis, 52(6),2877-2891.

▶ Lindström, E., & Gou, J. (2013). Simultaneous calibrationand quadratic hedging of options. Quantitative andQualitative Analysis in Social Sciences

▶ Lindström, E., & Åkerlindh, C. (2018). Optimal AdaptiveSequential Calibration of Option Models. In Handbook ofRecent Advances in Commodity and Financial Modeling(pp. 165-181). Springer,https://doi.org/10.1007/978-3-319-61320-8_8