strategies Optimal trading strategies designed using Bayesian inferenceas.utia.cz/files/prezentace.pdf · 2008. 5. 19. · Optimal control Dynamic programming Bayesian inference Model

Optimal tradingstrategies

designed usingBayesian inference

Jan Sindelar

Introduction

ProblemformulationModel discussion

Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function

Further approximations

Corrections toapproximations

ResultsExperiment specification

Future plans

Optimal trading strategies designedusing Bayesian inference

Jan SindelarDepartment of Adaptive Systems, UTIA CAS

Department of Probability and Mathematical Statistics, Charles University

e-mail: [email protected]



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Our goal is to devise a smart trading strategy:I Minimizing given loss function. The loss function

should be made as objective as possible.I Gaining experience from historical data. We do not

follow the martingale model of market price evolutiontraditionally used in Stochastic Finance.

I Being as exactly optimal as possible, with theoptimization still numerically feasible (we need acompromise)

I Profiting in the market. We therefore believe themarket is not efficient in the sense that we are able topredict future market price evolution. This is a verystrong assumption and has not yet been proven(even empirically).



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Although we do not use the techniques used in Stochastic Finance by applyingthe martingale and Markov properties of a stochastic process, there are ideasof Stochastic Finance we use in the Optimal Control problem we are trying tosolve. We therefore sketch the traditional approach of SF.

Stock price evolution model:

ln(S(T )) = ln(S(t)) +

∫ T

t

[α(s)−

12σ2(s)

]ds +

∫ T

tσ(s)dW (s) (1)

This is a generalized geometric Brownian motion. Usually Stochastic Financefits this model to the data, switches to a risk-neutral measure if such a measureexists and values derivative securities - forwards, futures and options, in a riskfree manner in case of a complete market, otherwise applies a utility functionand values the derivatives according to risk and return. Because the harderstparameter to fit to market data is the parameter α(s), Stochastic Financeusually doesn’t try to predict future price evolution to profit. Under therisk-neutral measure, no profit can be made at all, because all securities returnthe risk-free interest rate and the discounted process is a martingale.

Sound principles:Lognormal distribution of prices preventing them from being negative andscaling variance(rise from 1 to 1.1 is of the same probability as rise from 100 to110), compensation of the bias of conditional expected value by the term− 1

2σ2(s)ds



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

As far as the choice of the model is concerned we go much further than SF inthe most general case. In the continuous time case our model could be writtenas

ln(S(T )) =

∫ T

t

[ N∑n=0

αn(s)fn(Xn)−12σ2(s)

]ds +

∫ T

tσ(s)dW (s) (2)

where Xn are any Fs-measurable quantities and fn are general functions, butwe will see that the choice of quantities Xn and the choice of functions f istrimmed to a very narrow class in order to prevent numerical feasibility of themodel. Our model, in contrast to that of SF, does not retain the property ofbeing Markovian, nor it is a martingale after an easy change of measure, wework under the real probability measure all the time. The difficulty of the modelshows up, when we want to make a multistep ahead prediction, where we needto predict all the quantities Xn needed for such a prediction as well. Usually ouroptimization problem is solved in a discrete time and we therefore switch todiscrete case from now on. In the discrete case our model can be written as

ln(S(t + 1)) =N∑

n=0

βn(t)fn(Xn)−12σ2(t) + σ(t)et (3)

where now Xn are Ft -measurable quantities, f are functions, β are weights andet = ∆W (1) is a random variable with N(0, 1) distribution.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

In the case we already have a model chosen for a stock price evolution, wehave to devise a trading strategy being the best according to a "most objective"criterion for a trader - the loss function. As soon as we have a loss function(generally dependent on any quantity known at given time), we suppose, weobtain the best trading by solving an Optimal Control problem:

Theorem (Optimal control)Let ∆∗ be a set of positions allowed to be held in an asset in the market at atime and letR∗T = {(∆(0),∆(1), . . . ,∆(T )) : ∆(t) ∈ ∆∗,∆(t) is Ft −measurable} be aset of possible strategies (e.g.set of sequences allowed positions). An optimalcausal strategy RT ∈ R∗T is a strategy that satisfies

RT = argmin

{minR∗T

E [u (Z [PT ])]

}where Z (PT ) is a loss function chosen by the decision maker, PT is a set ofFT -measurable quantities and u() is an utility function1 (concave, increasing),penalizing risk.

1We assume, preferences of the decision maker fulfil the necessary axiomsfor their numerical representation to be allowed in Von Neumann-Morgensternrepresentation U(µ) =

∫u(x)µ(dx)



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

The problem of optimal control is often solved in the area of StochasticFinance. In the case of SF price evolution model we could again applymartingale techniques to come from a minimization problem to a system ofdifferential equations. There we usually minimize the utility of wealth or utility ofconsumption. In our case we fit the model to the data using Bayesian methodsand we are therefore dealing with uncertainty in estimation of the probabilitydensity of price evolution. The price in our model is not a martingale and wetherefore try to solve the minimization directly. In order to simplify modelimplementation (should be improved in the future), we replace the utilityfunction by direct penalization of risk in the loss function. Our utility function isu(Pt ) ≡ I(Pt ) and we can therefore use the method of Dynamic Programming:

Theorem (Dynamic Programming)As long as we optimize over a finite horizon T an optimal strategy in theminimization problem

RT = argmin

{minR∗T

E [Z (PT )]

}can be constructed in a value-wise way against the course of time. For every toptimal position ∆(t) is chosen so that

ν (Pt ) = min∆(t)∈∆∗

E [ν (Pt+1) |Ft ]

where ν (Pt ) is a so called Bellman function (sometimes called value function).



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Theorem (Dynamic Programming - continued)The recursion starts with

ν (Pt+1) = Z (Pt+1)

and the reached minimum has the value

ν (P0) = minR∗

E [Z (PT )]

And in case of an additive loss function, the recursion simplifies further

DefinitionWe can split the Ft -measurable quantity Pt into three parts Pt = (∆t , dt , Pt−1), where ∆t is theposition held in an asset from time t to t + 1 (optimal control parameter) and dt is the data received attime t .

Theorem (Dynamic programming for additive loss)In case of an additive loss function, the optimum stated in previous theorems can be found as follows

ν (Pt ) = min∆(t)∈∆∗

E[z(

∆t , dt+1)

+ ν(Pt+1

)|Ft]

starting from

ν(PT +1

)= 0



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

To be able to compute any of the expected values of Optimal Control, we needto know the distribution functions f (PT |Ft ), where T − t is the horizon we usefor optimization. A method we use for estimating this PDF and also otherintermediary density functions is the method of Bayesian learning.

DefinitionWe call the predictive PDF f (dt+1|Ft ) the outer model of the system. For the purpose of Bayesianlearning we split this model into two parts f (dt+1|Pt ) =

∫f (dt+1|Pt , θt )f (θt |Pt )dθt , where

f (dt+1|Pt , θt ) is the parametrized model chosen by the decision maker (to be able to solve the dynamicprogramming, this model is chosen from a friendly class of models in agreement with the previouslydiscussed class of models) and f (θt |Pt ) is the parameter PDF, the actual PDF used for Bayesian learning

Theorem (Generalized Bayesian filtering)The evolution of the PDF f (θt |Pt ) is described by the following two-step recursion starting from a socalled prior PDF f (θ0)

Time updating:

f (θt+1|Pt+1) =

∫f (θt+1|Pt+1, θt )f (θt |Pt+1)dθt

reflecting the physical evolution of θt → θt+1. To be able to perform this step, we need to know thephysical model of parameter evolution f (θt+1|Pt+1, θt ).

Data updating:

f (θt |Pt+1) =f (dt+1|Pt , θt )f (θt |Pt )

f (dt+1|Pt )

If we don’t know the time evolution model of the parameters, we can simplify the problem by supposing,the parameters are constant. We can than set f (θt+1|Pt+1, θt ) = δ(θt − θ) and therefore skip the timeupdating step. This aproximation can be very rough and we present some techniques to soften theinfluence of it later on.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Corresponding to the earlier mentioned allowed form of our model, we canwrite the parameterized model PDF in the normal regression form

f (dt+1|Pt , θt ) = f (dt+1|Pt ,Wt , βt ) =

1√(2π)m |Wt |

exp{−

12

[dt+1 − βt · z(t)]T Wt [dt+1 − βt · z(t)]

}where z(t) is a regressor vector of dimension n and its entries are past data(d1, . . . , dt ) and the data can be transformed by any, possibly nonlinearfunction fn mentioned previously, dt+1 is a vector of new data of dimension mthat again can be transformed by a function fn, βt is a matrix of parameters andWt is a covariance matrix. For our model to remain tractable, we allow thefunctions fn to be either fn = ln() or fn = I() making this a normal or log-normalconditional PDF. We can rewrite this model in a form that will later allow us tomake learning from possibly difficult multiplication of PDFs into an easyalgebraic operations with matrices

f (dt+1|Pt ,Wt , βt ) =1√

(2π)m |Wt |exp

{−

12

tr(

Wt[βt −I

]h(t)hT (t)

[βt−I

])}where hT (t) = (zT (t), dT

t+1) is a vector of new data and past data. We canfurther simplify the computations by factorizing this PDF for multidimensionalregressand using the chain rule for probabilities

f (dt+1|Pt ,Wt , βt ) = f (d1t+1|Pt ,Wt , βt )f (d2t+1|d1t+1,Pt ,Wt , βt ) . . .

f (dnt+1|d1t+1, . . . , dn−1t+1,Pt ,Wt , βt )

We can then estimate each model with a single regressand separately.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

We next have to choose the prior PDF f (θ0). This choice comes from outsidethe model and we therefore have to use expert information only, which can bea problem in certain applications. In our case, we have a lot of data entries ineach time series and we therefore believe the prior PDF not to be important. Anatural choice is then a so called conjugate prior PDF to the normal regressionmodel, which can be written in a Gauss-Wishart form

f (P0,W0, β0) =1

J(V , ν)

√|Wt |(ν+n−m−1)

exp{−

12

tr(

W0[β0 −I

]V[β0−I

])}

J(V , ν) being a normalization factor

J(V , ν) = (2ν+nπm+n−1)m2

m∏j=1

Γ

(ν + 1− j

2|Vz |

ν−m2 |V |−

ν2

)where Vz is a top-left n× n submatrix of matrix V and Γ is the gamma function.Because we don’t consider the initial PDF to be very important, we can choosearbritrary V and W0. In case of such a choice of prior PDF, the posteriorparameter PDF stays in the Gauss-Wishart form and we only have to updatethe sufficient statistics V and ν by incorporating new data

f (βt ,Wt |Ft ) = fG−W (βt ,Wt |V (t), ν(t))

where

V (t) = V (t − 1) + h(t)hT (t)

ν(t) = ν(t − 1) + 1



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

In case of the use of normal regression form model, the predictive PDF can bewritten as

f (dt+1|Pt ) =J(V + h(t)hT (t), ν + 1)

√2πJ(V , ν)

and it is a Student distribution.

Now, when we came this far, we still don’t know the right structure of themodel, we are trying to estimate. There are still the functions, possiblymodifying the data entries in the model and there is an unknown number ofregressors in the model which help us predict the value of regressands. Boththese attributes of the model can be obtained from expert information, but wecan also employ hypothesis testing with Bayesian estimation to find out whichof the model fits the data best. As it could be done similarly with themodification functions, we will assume we know them and we will focus onselecting the right amount and type of regressors. We will use part of thedataset to learn the structure of the model. We generally have a maximalstructure of regressors to choose from in the beginning. As we will perform thehypothesis testing for each possible combination of regressors taken from themaximal structure, we need the structure not to be to large as if there are nregressors in the maximal structure there are 2n possible combinations to test.In our models it basically means, that we will use maximally autoregression ofsecond order. From experience and to keep the problem simple, we suppose,that the structure of the model is constant in time. If we mark the combinationsof regressors in competing models by k , it means that

k(t) = k(0) = k



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

We again suppose, the parameters don’t evolve over time while trying to findout what regressors are important for the final model structure. We additionallycondition the predictive PDFs on the model structure k

f (dt+1|Pt , k) =J(V + h(t)hT (t), ν + 1, k)

√2πJ(V , ν, k)

An accurate solution to Bayesian structure estimation is again too difficult tosolve as we have to overcome the "curse of dimensionality". We therefore usean approximate algorithm , described in ...



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

The loss function is always supplied from outside of the model and is always subjective. Its subjectivitycomes from people’s different behavior when considering predicted gains and risk. In our models, wehave originally chosen a very simple loss function, concerning only cumulative gains:

z(T ) =T∑

t=0

∆(t) [S(t)− S(t + 1)] + C |a(t)|

where a(t) = ∆(t)− ∆(t − 1) and C are the transaction costs of slippage and commission.The lossfunction depended only on the prediction of the stock price. In the presence of such a loss function, thetrading system was too confident of itself and it traded too much (will be shown later) and kept loosingbecause of high transaction costs. Because the function is linear in predicted price, it can be shown that itis always optimal to be in the market. Later we tried to incorporate risk aversion into the loss function bysetting

z(T ) =T∑

t=0

∆(t) [S(t)− S(t + 1)] + C |a(t)| + BI∆(t) 6=0var[St+1|Pt

]where B is a positive constant. Such a model stays out of the market according to the height of constantB, but usually doesn’t hold the trades for high enough period of time to profit and is again beaten by theheight of transaction costs. The best working loss function so far is one also penalizing the actions with ahigher penalty

z(T ) =T∑

t=0

∆(t) [S(t)− S(t + 1)] + AC |a(t)| + BI∆(t) 6=0var[St+1|Pt

]where A is a positive constant.

Possible correct solution:Introduce an adequate utility function, possibly a utility of wealth - computationally difficult, but probablyfeasible.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Receding horizon:When optimization is performed on a time series, it should be performed up toa time, when our control ends in a natural way (expiration day of an option orfuture contract) or possibly to infinity (in case of trading a stock). Because suchan optimization is often not feasible, we have to shorten the period, weoptimize over. The horizon chosen for such optimization is called Recedinghorizon, we continue by the following algorithm

Algorithm (Receding horizon strategy)Repeat the following process for t ∈ (0,T )

1. Find the strategy Rt minimizing

E

[ t+h∑τ=t

z(Pτ )|Ft

]

2. Use the the first optimal ∆t

3. Move the window to t + 1

Super-cautious strategy:We use an approximate equality up to the receding horizon

f (θ|Pτ ) ≈ f (θ|Pt ) τ = t , . . . , t + h

so that we don’t change the posterior parameter PDF up to the recedinghorizon.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Exponential forgetting:We supposed constant parameters of the model in theprevious discussion, because of our poor knowledge of theirtime evolution model. Such an assumption is too strong.Because of that, we simplify the parameter evolution byemploying an exponential forgetting, giving lower weights toolder data entries. In the case of using Gauss-Wishart priordistribution and normal regression form of the model, suchforgetting becomes very simple

V (t) = λ[V (t − 1) + h(t)hT (t)

]ν(t) = λ [ν(t − 1) + 1]

where λ ∈ 〈0,1〉 is a forgetting factor. The lower the forgettingfactor the faster we forget older data entries.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Iterations spread in time:A receding horizon guaranteeing good approximation of thecorrect optimal strategy can be too large for us to perform theoptimization. We can therefore select a shorter horizon asstated in the approximation part and use a so called iterationspread in time. In case of an additive loss function, we canselect an optimal decision ∆(t) at time t , if we know the futureBellman function up to the final horizon T

ν(Pt ) = min∆(t)∈∆∗

E [z(dt+1,Pt ) + ν(Pt+1)|Pt ]

Because the dynamic programming is an iterative search forthe Bellman function ν(Pt+1), we use ν(Pt ) as the estimate ofν(Pt+1) at time t . For such an iteration, we have to choose aninitial guess of ν(0), perform the above optimization on ashorter receding horizon (here mentioned for a horizon 1) andadd the estimated Bellman function, when we move thewindow to time t + 1.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

Data: The time series, we perform our experiments on are mostly U.S.commodity futures data, provided by Colosseum a.s. The time series have afew specifics, we have to mention

I The model supposed for price evolution was supposed for the case of astock market. Futures data can be biased because of the interest rate ortransfer and storage expenses. The model should probably be modifiedto account for such influences.

I Futures data have certain delivery dates and given underlying is tradedin number of contracts with differing delivery months. The time series weuse are artificially joined time series of active months throughout thetrading period, where an active month is the one with highest volume inthe current day. The artificial joining can roughly damage the estimationin the model. In the future, we want to switch to learning for eachdelivery month separately.

I We have about 30 channels of data to learn on. These channels containdaily open, high, low, close prices, volume, commitment of traders dataand expert channels supplied by Colosseum a.s. We choose thechannels influencing price evolution by the earlier methods of Bayesianstructure estimation.

I The estimation is performed on about 20 years of data, artificially joined,containing about 5000 entries.



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans



Jan Sindelar

Introduction


Optimal control

Dynamic programming

Bayesian inference

Model structure

Loss function




Future plans

I Solving of an exact Optimal Control problem with a specified utilityfunction.

I Incorporating logarithmic transformation of some data into themultidimensional model.

I Incorporating results of present Bachelor’s work in the following fields1. Improved forgetting: Now all the channels are forgotten with a

same weight during Bayesian estimation. Correctly, we shouldhave a chance of different forgetting for different regressors.

2. Transformations of data beyond ln() : As mentioned, the futuresdata can be biased by interest rates and other expenses andtherefore they might not follow a lognormal distribution.

I Search for extremes - smoothing of trading over the receding horizonI Improving mathematical exactness

Documents

strategies Optimal trading strategies designed using Bayesian inferenceas.utia.cz/files/prezentace.pdf · 2008. 5. 19. · Optimal control Dynamic programming Bayesian inference Model