Scenario Analysis to Validate Models

Scenario Analysis to Validate Models

Martin GoldbergSenior DirectorHead of Model Quality ReviewStandard and Poor’[email protected]

2. Scenario Analysis to Validate Models March 4, 2010

The Usual Caveats

This presentation expresses my own personal

opinions and may not represent the views of Standard and Poor’s Rating Services.

This is not intended to be a fully inclusive handbook. Risk management is not a simple task, and you will have to put thought and effort into it beyond attending a conference, if managing model risk is your goal.

My experience is primarily with valuation, VaR, Op Risk, and Credit Risk models, so this may be less informative about validation of stat arb and portfolio construction models.


Outline of My Talk

1. Definitions of Terms2. Using the appropriate scenarios for

validating various types of financial models3. Validation scenarios as distinct from model

scenarios or Monte Carlo modeling4. Benchmarking using alternative models5. How do you correctly conduct scenario

analysis?6. What are its limitations?7. Conclusions


Definition of Scenario

• I define “scenario” as a (possibly hypothetical) time series of values and/or changes for all market factors, prices, rates, and other inputs to the relevant model, with defined start and end dates and defined intermediate time intervals.– The start date is usually “now”– Whether the scenario unfolds in microseconds, days, or

years depends on the intended use of the model.– An input might have a time series of “unchanged” if you

aren’t interested in that particular factor, which is equivalent to an explicit assumption that the effects of that factor on the model results are negligible or not of concern.

– A “lesser” input could also be assumed to be some deterministic function of the “important” risk factors.


Definition of Scenario Analysis

• I define “scenario analysis” as – choosing one or more scenarios, – running the relevant model in each scenario, and – studying (analyzing) the results of each scenario to draw

conclusions. – The conclusion could be that you need more or different

scenarios.


Definition of Model

• I assume that this is limited to automated quantitative financial models, which I define as

• A quantitative model is a controlled view of certain real world dynamics that is used to infer the likely consequences of some pre-specified assumptions under various circumstances. – May not capture all the nuances of the real world. – A quantitative calculation based on one or more assumptions. – Not black boxes of revealed truth but merely numerical

expressions of some view of how the world would be likely to behave.

– Relies on assumptions about the behavior of people, organizations, acts of the natural world, and the use of other models by market participants.

– A mixture of behavioral psychology, statistics, numerical methods, and subjective opinions.


Definition of Model Validation

• I define the validation of a financial model as a test of how suitable it is for its intended use, which involves a simultaneous test of assumptions, inputs, calibration, implementation, and usage.– The physical sciences have laws of nature called

"theories," that observation or experiments can verify or disprove.

– In finance, however, there are merely significant tendencies and patterns

– Quantitative financial models are necessarily generalizations that events in the real world will sometimes contradict..

– Different assumptions and different intended uses will in general lead to different models.

– Models intended for one use may not be suitable for other uses.


Appropriate scenarios – a menu• There are several ways of choosing validation

scenarios, and often it is helpful to use more than one.– Historical scenarios – observed past intervals that might be a

useful test for the model. This is also called backtesting.– Sensitivity tests – Here the start date and the end date are

both now, and all but one (or perhaps a few) of the inputs are unchanged. (Instantaneous jumps)

– Ladder tests – a set of sensitivity tests with various widely spaced values of the same input.

– Monte Carlo – Generate lots of random scenarios, examine each individually, and see which ones cause the model to behave oddly or unexpectedly. Look for patterns. This is not the same as statistical analysis of the result distribution.

– Hand-picked – What particular few scenarios are most instructive abut the model’s behavior and limitations?

– Stress scenarios – any of the above where the changes are quite large


Appropriate scenarios for validating any model

• The point of running scenarios for validation is to see how the model behaves, separately from any details about the positions that are being modeled. The size of portfolios being run through the model should, in my opinion, all be quite small, and at least as diverse as anything you plausibly might use the model for in the foreseeable future. • Scenarios are tests of the model’s response

to market inputs, not to position inputs. In a long enough scenario, you may wish to include the feedback effects of portfolio or model changes that would be caused with certainty by embedded contract triggers or company policies.


Scenarios for validating Valuation Models

• Try to include scenarios that just barely trigger, or just barely miss triggering, any knock-outs, contingencies, etc. in the instrument being modeled. • Look for scenarios that cause the model to

“break” and try to determine where the breakpoints are.• Compare the model Greeks to the valuation

outputs for small changes, at a variety of starting points for the scenarios.• For path-dependent valuations, the scenarios

should explore a variety of different-shaped paths.


Scenarios for validating regression models

• For regression-type models, such as are used in, for example, portfolio optimization, operational risk capital calculations, some forms of stat arb, and credit scorecards, scenario analysis other than historical backtesting may not be the best tool. • A regression on historical data would not

give the same results if history had been different, and counterfactual hypothetical scenarios may not give the validation effort any useful information.


Scenarios for validating Portfolio Optimization and VaR model Inputs

• The classic variance-covariance Markowitz framework used by some of these models can lend itself to scenario testing on the input pre-processing.

• In this simulated example, the Gaussian drivers of two processes are 61% correlated. Consider scenarios where we test robustness to skewness in the distribution of one or both observed processes. A rank correlation remains stable, but the Pearson correlation is an underestimate of concordance. Skewness of equity indices: Australia is -2.8, US -1.2


Scenarios for validating predictive models

• For some forms of predictive model, such as technical analysis, scenarios can be used to see how quickly the model perceives new trends or reversals, and how credible that perception is.• For a structured credit waterfall model,

informative scenarios would extend to the structure’s final maturity, and try to “break” the model with various shapes of stressful paths of the relevant inputs.


Stress Test Scenario Example• Here is an example of a ladder test where the market moves

outside the range of appropriateness• Calculate the forward implied volatility from given Black-

Scholes implied vols from quoted options at 1 and 2 year tenors. Suppose the volatility of the 1 year option is 5%. The forward vol in a simple model (no jumps, no autocorrelation) would be

• If the vol curve drops too quickly, as with some commodities, you need a more complicated model since imaginary forward vols are not meaningful

shortshortlonglong TT 22

Error in forward vol model

-2%0%2%4%6%8%

10%12%14%

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%

Implied vol at longer tenor

forw

ard

vol

Imaginary forward vols


Validation scenarios vs Scenario-based models

Scenario Validation• Stresses larger than model can handle appropriately• Designed to test model• Scenarios to test facets of model• Examine each scenario result• Choose a few representative positions

• Only report results that are informative or unexpected• Calibrate if needed, and not necessarily to any particular date• Scenarios usually ad-hoc, need not be plausible• Scenarios might include regime changes

Scenario-based model• Stresses large but plausible OR Monte-Carlo with no plausibility

test• Designed for informative outputs• Meaningful scenarios• Possibly only use ensemble statistics• Run entire portfolio through each scenario• Report all results

• Periodic up-to-date recalibration

• Scenarios subject to senior management scrutiny• Has market changed regimes enough to make model irrelevant or

obsolete?


Benchmarking Using Alternative Models

• The best validation technique, which takes the longest but gives the most assurance that a model does what it should, is to run the model in question, and a benchmark model, on many scenarios with various portfolios, and study where they get different results. • If the benchmark model is off-the-shelf

software, the differences in results are likely due to different assumptions.• If the benchmark is a custom built replica of

the model being validated, independently coded using the same specifications, then differences are either coding bugs or inconsistency with the specifications.


Benchmarking with Partial Alternatives

• One way to test the model’s assumptions and features is to use a custom benchmark model where one or more features can be switched off or new ones added.• Run the model and this partial benchmark

model, on many scenarios with various portfolios, and see how much each assumption leads to different results, and look for synergistic effects.


Correct conduct of scenario analysis – First few steps

1. This should be one of the later steps in the validation process, after you are comfortable that the code isn’t too buggy

2. Determine which inputs you will be varying in the scenarios, and if there are any “special” values – strikes, barriers, trigger points, etc.

3. Design some of the scenarios before you run any of them• This will help get buy-in from model users and builders

if something unfortunate turns up in your testing• Ameliorate the governance issue of “Quants don’t run

this bank”• More on this sort of thing later from Victor Dvortsov’s

talk4. Decide if it’s worth the effort to have a

benchmark, and if so, what kind? (Off the shelf, exact replica, adjustable replica, custom-built alternate methodology, other)


Correct conduct of scenario analysis – CPU time

5. Do univariate ladder tests on all the inputs separately. Look for discontinuities, unexpected non-linearities, and especially for the Blue Screen of Death. Run scenarios out to both way above and way below any plausible values.

6. (If there are many inputs) Run Monte Carlo, with independent (0% correlation) inputs and extremely high volatility. The purpose is to explore as much of input/parameter space as you can, to look for pockets or edges of unexpected results. • Remember the ensemble statistics are not what you’re

after. You need to be able to identify and analyze individual scenarios that stressed the model. None of these scenarios are intended to be realistic.


Correct conduct of scenario analysis – more CPU time

7. Design and run scenarios that might stress the model – various path shapes, just missing or just barely crossing any trigger levels, etc.– This should be informed by the MC in the previous

step.8. Test if model outputs are self-consistent.

For example, does a delta or vega match what happens with a small shift?


Correct maintenance of scenario analysis – Historical Backtesting

• Model validation is more than just backtesting, but historical backtests are an important step.• Since historical backtesting is easy to

automate, and can be run without extra user inputs, best practice would be to frequently run historical backtests and have the computer flag or notify you if and when something odd shows up.


Limitations of Scenario Testing

• All models have limitations. These are just models, not reality.• Since these tests only change inputs, and not

“baked-in” assumptions, parameters, nor methodologies, a model could behave entirely self-consistently with different inputs yet still have flaws or issues.• Historical backtests where the model is

confronted with reality in many scenarios, might not cover historical periods stressful enough to expose model issues. No historical scenario using a currency with a pegged FX rate can test how the model handles FX rate shifts.


More Limitations of Scenario Testing

• Hypothetical scenarios may not discover some regions where the model has issues, especially when the extreme values are okay but the questionable area is in the middle somewhere with seemingly innocuous paths.– Although some models can be shown to have issues, you

cannot in general prove that none exist; only that you didn’t find any.

• Scenario testing may uncover potential issues but has nothing to say on what to do about them. You can– Avert your eyes and hope it doesn’t happen in reality,

(POOR) or– Pass on the information and go away, (NOT GOOD) or

better yet– See if there is a fix for the issue (PREFERRED)


Conclusions

• Both hypothetical and historical scenarios are important• Validation scenarios need not be similar to

“production” scenarios.• Stress the model until it falls off the cliff,

even if the cliff seems too remote for the market to ever reach.• Agree on how your scenario test results will

be used before you start any tests – buy-in is important.• Scenario testing is a useful tool to have in

the validator’s toolbox• Scenario tests raise questions, but might not

answer them


Questions?

Documents

Scenario Analysis to Validate Models