Fin285a:Computer Simulations andRisk Assessment

Section 9Backtesting and Stress Testing

Danı́elson, 8.1-8.3.1, 8.5, 8.6

Overview

Fall 2017: Fin285: 9 2 / 25

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing

What is backtesting?

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing

Fall 2017: Fin285: 9 3 / 25

Backtest basics

Fall 2017: Fin285: 9 4 / 25

•Test VaR and ES estimates

•Use past data

•Forecast into the future (windows)

• Important for regulation (Basel II and III)

•Out of sample forecasting

⇒Clean future forecasts

⇒VaR estimates at time t use only data before time t

Regulatory issues

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing

Fall 2017: Fin285: 9 5 / 25

Market risk capital (reserves)

Fall 2017: Fin285: 9 6 / 25

•Capital (liquid) held in reserves by banks and other financials

⇒Required by regulation

⇒ Important during crisis

•Previously: Set by fraction of portfolio in different asset classes

⇒Sovereign debt

⇒Corporate debt

⇒Home mortgages

⇒Equity

•Now (> 2005): Also, involves some aspect of internal risk (VaR)measures (Internal models)

•Needs to be carefully monitored (backtesting)

Example

Fall 2017: Fin285: 9 7 / 25

Capitalt = Ct = Atmax(VaRt(1%),VaRt(1%)) + Constant (9.1)

VaRt(1%) = Average over previous 60 days (9.2)

At =

3 if vt ≤ 4

3 + 0.2(vt − 4) if 5 ≤ vt ≤ 9

4 if 10 ≤ vt

(9.3)

vt = VaR(1%) violations over past 250 trading days

Think about the dynamics of this:

•Volatility goes up

•vt rises

•Need to shift portfolio to cash

Backtesting details

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing

Fall 2017: Fin285: 9 8 / 25

Estimation windows (length = 500)

Fall 2017: Fin285: 9 9 / 25

[1, 500],VaR501

[2, 501],VaR502

Full Sample

[3, 502],VaR503

Estimation and Testing

Fall 2017: Fin285: 9 10 / 25

Estimation

Start End Test

1 500 5012 501 502. . . . . . . . .

Recursive windows

Fall 2017: Fin285: 9 11 / 25

Windows can extend all the way back to the start of thedata:

Estimation

Start End Test

1 500 5011 501 5021 502 5031 503 504. . . . . . . . .

VaR violations (exceptions)

Fall 2017: Fin285: 9 12 / 25

ηt =

{

1 if yt ≤ −VaRt

0 if yt > −VaRt

(9.4)

v1 =∑

ηt (9.5)

V R =v1

pWT

=Observed violations

Expected violations=

v1

WT

p(9.6)

•WT testing length

• If working, V R ≈ 1

Backtest examples

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing

Fall 2017: Fin285: 9 13 / 25

Null hypothesis

Fall 2017: Fin285: 9 14 / 25

•VaR(p)

•vt is binomial (like coin)

•Two features to test:

•VaR probability:

⇒Pr(ηt = 0) = 1− p

⇒Pr(ηt = 1) = p

⇒Variability of p estimates in windows (see our old examples)

• Independence:

⇒ηt independent over time

⇒ηt are unpredictable

Backtest code

Fall 2017: Fin285: 9 15 / 25

•Python: backTest.py

•Rolling historical VaR

•Estimate VaR using past 500 days

•Report violations and violation ratio

•Simpler than code in textbook

•Statistical test: like coin, and election polls

Independence

Fall 2017: Fin285: 9 16 / 25

•VaR violations (exceptions) should be independent

•Find conditional probability that violation (ηt) follows violation(ηt−1)

•For independence this is equal to the VaR probability level

•Can also look at correlations of violations

•ηt should be unpredictable using any past data

Testing independence

Fall 2017: Fin285: 9 17 / 25

Pr(ηt = 1|ηt−1 = 1) =Pr((ηt = 1)and(ηt−1 = 1))

Pr(ηt = 1)(9.7)

•Python: backTest.m

•Find conditional probability (above)

•For independence this is equal to the VaR probability level

•A violation today contains no information on tomorrow

Testing expected shortfall

Fall 2017: Fin285: 9 18 / 25

•This can be done

•A little trickier than VaR

•Requires more data

•See textbook

Testing expected shortfall

Fall 2017: Fin285: 9 19 / 25

•First, assume rolling expected tail return:

ESt(p) = R̃t (9.8)

•Now, key ratio:E(Rt|Rt ≤ −VaR(p))

ESt(p)= 1 (9.9)

•For correct ES estimate this ratio should be 1

•Python: backTestES.py

•Difficult to do analytic statistical tests

•Bootstrap confidence bands

•Generally requires more data than testing VaR

Problems with backtesting

Fall 2017: Fin285: 9 20 / 25

•Window/time selection

⇒Enough extreme events?

⇒Regime changes

•True “out of sample” (data snooping)

⇒Need clean tests

⇒This rarely happens

⇒Multiple model problems

•Critical for policy and reserves

Stress testing

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing

Fall 2017: Fin285: 9 21 / 25

Scenario analysis

Fall 2017: Fin285: 9 22 / 25

•Test important scenarios

•Find portfolio sensitivity to key risk factors

Vt =K∑

j=1

βjft,j (9.10)

•Extreme moves in key risk factors

•Types of tests

⇒Replay actual events (crashes of 1987/2008)

⇒Generate new extreme events (Greece leaves Euro, Brexit)

⇒Monte-carlo with extra weight on extremes

⇒Structural breaks in the times series (changing correlations)

•Scenarios often dictated by regulation

Smooth incorporation into risk models

Fall 2017: Fin285: 9 23 / 25

(f, fstress) simulations for normal and stressed situations.

fnew =

{

f with probability (1− ǫ)

fstress with probability ǫ(9.11)

Stress test problems

Fall 2017: Fin285: 9 24 / 25

•Finding scenarios

•Probabilities for worst case scenario??

⇒Preparing for a 10,000 year flood

•What if institutions “plan around” given scenarios, but not readyfor other events?This is like our “manipulating VaR” examples.

•Endogenous feedback:

⇒What happens when everyone is hit with this shock

⇒Fire sales across markets

Overview

Fall 2017: Fin285: 9 25 / 25

What is backtesting?

Regulatory issues

Backtesting details

Backtest examples

Stress testing