Embed Size (px)
Citation preview
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting Model
for Stress Testing
A Three-Stage Approach
James Lu
The 17th Annual OpRisk North America 2015, New York
March 2015
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Agenda
Operational Risk Loss Forecasting for Stress Testing
– Regulatory Expectation and Challenges
– Industry Practice and Methodology
Rebuilding the Operational Risk Loss Forecasting Model
– Background and Challenges
– A Three-Stage Approach
– ARIMAX Model to Project External Losses
– OLS Model to Project Internal Losses
– Simulation to Estimate Tail Risk Using Scenario Analysis
– The Model Results
– Conclusions and Limitations
2
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for Stress TestingRegulatory Background and Expectation
3
The Fed’s CCAR requires banks to have internal capital planning processes
– Operational risk loss is a component of PPNR, which is the first building block of the CCAR process
Operational risk loss estimation methodologies vary among banks and so do the Fed’s expectations
The Fed has given guidance regarding estimation methodologies
– Stronger practices vs. lagging practices
Banks should use internal op risk loss data as a starting point to provide historical perspective
– They can then incorporate forward-looking elements, idiosyncratic risks, and tail events to estimate
losses for the CCAR projection window
Operational Risk Loss Forecasting Challenges
Limited internal loss history
Link loss history with macroeconomic drivers, including bank-specific risk drivers
Quantify extreme severe but plausible loss events
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
4
Operational Risk Loss Forecasting for CCARIndustry Practice and Challenges
Operational Risk Loss Estimation Flowchart
Banks’ approaches vary in each of the following steps due to data richness and business
sophistication. A few banks conduct benchmark analysis.
Internal loss historical database, supplemented by
external historical lossMacroeconomic variables
Scenario analysis add-on to capture tail risk which internal loss history may have not experienced
Analytical models or judgment-based estimation to predict future ops risk losses for the baseline
scenario
Analytical models or judgment-based estimation to predict future ops risk losses for stressed
scenarios and bank idiosyncratic risk
Management overlay for legal/litigation/compliance risk and other risks
Benchmarks to either support or challenge the loss projections
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCARIndustry Practice and Challenges
Common operational risk loss estimation methodologies
Modeling Approaches
– Regression model using macroeconomic variables
– The ARIMA/ARIMAX model to capture the auto regressive nature of loss history
– Modified loss distribution approach
– Scenario analysis to capture tail risk
Criteria for Selecting the Final Model
The selected modeling approach should have the following characteristics:
– Segmentation needs to be granular enough given the data availability
– Model design and outcomes should be empirically robust and statistically sound
– Model performance needs to exhibit reasonable levels of stability and accuracy
– Scenario analysis add-on and management overlay should be fully reviewed, challenged and well
documented
5
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Agenda
Operational Risk Loss Forecasting for Stress Testing
– Regulatory Expectation and Challenges
– Industry Practice and Methodology
Rebuilding the Operational Risk Loss Forecasting Model
– Background and Challenges
– A Three-Stage Approach
– ARIMAX Model to Project External Losses
– OLS Model to Project Internal Losses
– Simulation to Estimate Tail Risk Using Scenario Analysis
– The Model Results
– Conclusions and Limitations
6
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Modeling Challenges
Rebuilding the Ops Risk Loss Forecasting ModelBackground and Challenges
7
The bank has less than four years (i.e.15 quarters) of operational risk data in its database,
which reliably and consistently captures operational risk loss dates, loss events, lines of
business, recoveries, etc.
ORX ops-risk database has quarterly loss data since 2002. How do we link external loss
history with internal loss history?
What is the granularity of modeling, given both the richness of external loss data and short
history (and data sparseness) of internal loss data?
Literature and industry experience indicate a lack of strong predictors for ops-risk loss
model, so developing statistically sound and empirically robust models is a challenge
Quantitatively-driven scenario analysis add-on as a tail risk is still evolving and therefore a
challenge for most banks to factor it into their models
New Modeling Approach
The newly developed three-stage model and quantitatively-driven scenario analysis
respond to the challenges faced by both the bank and the industry
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Three-Stage Modeling Methodology
8
The three-stage model utilizes both external and limited internal ops-risk loss data to
forecast the future nine quarters’ (9Q) ops-risk loss for the bank in both the baseline and
stress scenarios
Management overlay is added to capture legal/litigation/compliance risk
The implemented model meets the regulatory guidelines for stronger practices regarding
ops-risk loss forecasting methodologies
Ops-Risk Loss Forecasting Model
Stage 1 Stage 3Stage 2
The ARIMAX model to
forecast future
9Q external
ops-risk loss by
geography and LoB using
macroeconomic variables
The OLS model to forecast
future
9Q internal ops-risk loss
by geography and LoB
using projected external
quarterly loss from Stage
1 and
macroeconomic/bank
idiosyncratic risk drivers
Simulate scenario analysis
results to capture tail risk
that was not experienced
in internal loss history
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCAR
9
Three-Stage Modeling Process Flowchart
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
horizon by geography
and LoB
ORX
Loss
History
Macro
Variables
Projected external
losses by
geography and LoB
Mapped
Macro
Variables
Mapped
Bank
Specific
Variables
Bank
Internal
Loss
History
Projected 9Q of
internal losses by
geography and LoB
The regression model
to predict 9Q ops-risk
losses for CCAR
horizon by geography
and LoB
Scenario
Analysis by
Event Type 1
Scenario
Analysis by
Event Type 2
Scenario
Analysis by
Event Type 3
Projected 9Q internal
losses with tail risk
Projected 9Q internal
losses with tail risk
and management
overlay
Stage 1
Stage 2Stage 3
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCARStage 1: ARIMAX Model Using External ORX Data (1/3)
10
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
horizon by Geography
and LoB
ORX
Loss
History
Macro
Variables
Projected external losses
by geography and LoB
Map the bank’s portfolio to ORX data structure by geography and LoB. There
are six geography and LoB combinations. This mapping is based on the
assumption that the bank and other banks in ORX have similar risk drivers by
geography and LoB combination
Data integrity and cleansing include data series normalization, missing data and outlier treatment,
specifically quarterly external loss is normalized by the number of banks in the ORX.
Variable transformations are tested, and the selected dependent variable is the logarithm of quarter-over-
quarter change of ops-risk loss per bank
The ARIMAX model offers the following benefits:
– Captures the relationship between observations across time
– Estimates relationships and produces forecasts that utilize both the information in past values of the series (quarterly
loss) and the information contained in independent variables (macroeconomic variables)
– Methodology is mature and easy to diagnose and estimate
– ORX data series has 12 years of data and covers economic cycles
The ARIMAX model has one limitation:
– In its data history, ORX data does not capture the bank’s portfolio size, which could be a driver
GEO 1 – Retail
GEO 1 – Commercial
GEO 2 – Retail
GEO 2 – Commercial
GEO 3 – Retail
GEO 1 – Corporate & Others
Bank
(Geography & LoB)
GEO 1
GEO 2
GEO 3
GEO 4
GEO 5
ORX Data
Geography LoB
Retail
Commercial
Corporate & Others
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
𝜆𝑡,𝑝𝑂𝑅𝑋 = 𝑙𝑜𝑔
𝐿𝑡,𝑝𝑂𝑅𝑋
𝐿𝑡−1,𝑝𝑂𝑅𝑋 = 𝑙𝑜𝑔𝐿𝑡,𝑝
𝑂𝑅𝑋 − 𝑙𝑜𝑔𝐿𝑡−1,𝑝𝑂𝑅𝑋 = 𝑓 𝑥𝑝, 𝜙𝑝, 𝜃𝑝
Operational Risk Loss Forecasting for CCARStage 1: ARIMAX Model Using External ORX Data (2/3)
11
The ARIMAX model specification
– AR: Autoregressive – Time series is a function of its own past
– MA: Moving Average – Time series is a function of past “shocks”(deviations, innovations,
errors, etc.)
– I: Integrated – Differencing provides stochastic trend and seasonal components, so
forecasting requires integration or undifferencing
– X: Exogenous – Time series is influenced by external factors
The selected dependent variable is a logarithm of QoQ ops-risk loss ratios (𝜆𝑡,𝑝𝑂𝑅𝑋) for a proxy of
geography and LoB combination, and it measures the relative change of ops-risk loss. It can be
modeled as a function of macroeconomic variables 𝑥𝑝 along with lagged values of ORX log-
returns (ϕ) and the lagged forecast errors (θ) which are specific to that proxy (i.e.𝑓 𝑥𝑝, 𝜙𝑝, 𝜃𝑝
The ARIMAX model combines time-series analysis and regression analysis. Hence, in many cases, it can produce better
forecasts than using either technique alone. The ARIMAX model includes a structural (economic) explanation for the part
of the variance of the dependent variable that can be explained structurally while explaining the part of the variance that
cannot be explained structurally with a time-series model.
We choose the best model based on
– Low AIC
– Low MAPE
– Statistical significance of macroeconomic variables, MA, and AR terms
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
horizon by geography
and LoB
ORX
Loss
History
Macro
Variables
Projected external losses
by geography and LoB
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCARStage 1: The ARIMAX Model Using External ORX Data (3/3)
12
The model outcomes indicate that all macroeconomic and structural variables are significant
and economically explainable
The same model is used to project 9Q external losses for baseline and stress scenarios
The ARIMAX model to
predict 9Q ops-risk
losses for CCAR
horizon by geography
and LoB
ORX
Loss
History
Macro
Variables
Projected external losses
by geography and LoB
GEO 1 – Retail Portfolio
Conditional Least Squares Estimation
Parameter EstimateStandard
Error t ValueApproxPr > |t| Lag Variable
MA1,1 -0.47548 0.15491 -2.88 0.0066 1 log_loss_ratio_Rtl
VAR1 6.84435 3.19104 2.68 0.0110 0 CPI_SA_YY_Q8
VAR2 1
GEO 1 – Commercial Portfolio
3.02520 4.12118 3.65 0.0008 0 Unemp_YY_Abs_Q4
VAR4 0.25252 0.05616 3.78 0.0005 0 GDP_QQ_Rel_Q8
Conditional Least Squares Estimation
Parameter EstimateStandard
Error t ValueApproxPr > |t| Lag Variable
MA1,1 -1.15427 0.43507 -2.75 0.0100 1 log_loss_ratio_Cml
MA1,2 -1.28583 0.30382 -3.57 0.0012 2 log_loss_ratio_Cml
MA1,3 -1.51444 0.32225 -4.13 0.0003 3 log_loss_ratio_Cml
AR1,1 -1.12522 0.37971 -2.81 0.0084 1 log_loss_ratio_Cml
AR1,2 -0.78412 0.23962 -3.26 0.0027 2 log_loss_ratio_Cml
AR1,3 -1.021041 0.24620 -4.07 0.0003 3 log_loss_ratio_Cml
VAR1 27.27729 12.40279 2.04 0.0502 0 Unemp_QQ_Abs_Q6
VAR2 -0.0008847 0.0004363 -2.27 0.0300 0 GDP_QQ_Abs_Q2
VAR3 0.0552E-7 9.5156E-8 -2.18 0.0369 0 Pers_Bk_Filings_YY_Abs_$_Q4
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
13
Projected
external
losses by
geography
and LoB
Mapped
Macro
Variables
Mapped
Bank
Specific
Variables
Bank
Internal
Loss
History
Projected 9Q
internal losses
by geography
and LoB
Regression model to
predict 9Q ops-risk
losses for CCAR
horizon by geography
and LoB
Operational Risk Loss Forecasting for CCARStage 2: Regression Model for the Bank’s Internal Loss Forecasting (1/2)
The Stage 2 model uses the forecasted external losses from the ARIMAX model in Stage 1, along with
macroeconomic and bank-specific risk factors to forecast the bank’s internal losses. The regression
model was selected over the ARIMAX model due to the short history of internal loss data. Internal loss
forecast model is specified as
𝐿𝑡,𝑝𝐼𝑛𝑡 = 𝐿 + 𝑥 + 𝜆𝑡,𝑝
𝑂𝑅𝑋
That is, the bank’s internal loss is a function of the average quarterly historical loss 𝐿, macro and bank-
specific variables 𝑥 (including balance sheet size), and the forecasted values of the ORX losses, 𝜆𝑡,𝑝𝑂𝑅𝑋
It is worthwhile to note that
– We include forecasted external loss as an internal loss driver. This is based on the assumption that
the bank and other banks in ORX share similar operational risk loss trends
– We include the bank’s internal loss average in the regression model on the assumption that the
historical average is the starting point of future loss. In other words, future loss will be around the
historical average, adjusted by forecasted external loss, other risk factors, and balance sheet size
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
14
Projected
external
losses by
geography
and LoB
Mapped
Macro
Variables
Mapped
Bank
Specific
Variables
Bank
Internal
Loss
History
Projected 9Q
internal losses
by geography
and LoB
Regression model to
predict 9Q ops--risk
losses for CCAR horizon
by geography and LoB
Operational Risk Loss Forecasting for CCARStage 2: Regression Model for the Bank’s External Loss Forecasting (2/2)
The model outcomes indicate that the identified risk drivers are significant and economically explainable
The same model is used to project the baseline scenario and all stress scenarios for 9Q of the bank’s internal losses
GEO 1 Commercial Internal OLS Model (Adjusted R 2 = 0. 52)
Parameter Estimates
Variable Label DFParameter
EstimateStandard
Error t Value Pr > |t| Tolerance V.I
CPI_SA_YY_Q4 1 -176.0824 46.56550 -4.04 0.0014 0.99499 1.0
log_lossratio_hat Forecast for log_dloss_sm4 1 14.6227 7.06965 2.29 0.0397 0.99499 1.0
Analysis Variable : MAPE
Minimum Mean Median Maximum
0.0205061 1.7752406 0.8452006 9.2587490
GEO 1Retail Internal OLS Model (Adjusted R 2= 0.852)
Parameter Estimates
Variable DFParameter
EstimateStandard
Error t Value Pr > |t| ToleranceVarianceInflation
Intercept 1 15.20999 2.94609 5.51 0.0003 . 0
Auto_Sales_NSA 1 2.42056 0.46628 4.60 0.0010 0.18560 5.38782
HPI 1 -0.10252 0.02193 -4.38 0.0014 0.61482 1.62650
Loglossratio_hat_Q3 1 -0.84000 0.31150 -2.66 0.0239 0.53596 1.86580
Analysis Variable : MAPE
Minimum Mean Median Maximum
0.0258840 0.2257054 0.1640946 0.8851183
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCARPerformance Testing
15
Model performance testing was conducted to compare the new model with the previous
years’ models on both model stability (Adjusted R2) and predicting accuracy (Mean
Absolute Percent Error, or MAPE*)
The new model is significantly better than the previous model on both stability and
accuracy. In addition, the new model more appropriately reflects the risk characteristics of
ops-risk and is better aligned with the supervisory guidance of operational risk loss
forecasting methodologies for CCAR
The Fed examiner’s comment was positive: “Model better designed and performance
improved significantly.”
* Mean Absolute Percent Error (MAPE): Absolute (Forecast – Actual) / Actual** Not based on the same model specification, therefore not a direct comparison
Model Performance Comparison – Stability**
Model Performance Comparison – Accuracy
Geography & LoBPrior Model New Model
R2 Adj. R2 R2 Adj. R2
GEO 1 Retail 0.18 0.13 0.88 0.85
GEO 1 - Commercial 0.21 0.14 0.55 0.52
Geography & LoBPrior Model New Model
Min Mean Max Min Mean Max
GEO 1 Retail 12% 136% 892% 2.30% 23% 89%
GEO 1 - Commercial 2% 359% 6629% 2.00% 178% 926%
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
16
Projected 9Q
internal
losses by
geography
and LoB
Scenario
Analysis by
Event Type 1
Scenario
Analysis by
Event Type 2
Scenario
Analysis by
Event Type 3
Projected
9Q internal
losses with
tail risk
Projected 9Q
internal losses with
tail risk and
management
overlay
The Stage 3 of the model involves simulating scenario analysis results to capture ops-risk tail risk, which
are the frequency and severity that current internal loss history has not experienced.
Scenario analysis is based on the seven Basel II AMA ops-risk loss event categories*. A group of SMEs
discussed and selected certain categories that are specifically applicable to the bank’s exposure and
loss experience
For the selected categories below, SMEs determined the “Most Likely” and “Worst Plausible” loss
frequency and severity
We use Poisson distribution to simulate loss distribution to select certain percentiles as tail risk losses
under extreme scenarios. We select the median as the baseline and 55 percentile for adverse and 60
percentile as the severely adverse and BHC stress scenarios (e.g. a scalar of 118% is applied to the
adverse scenario over the baseline scenario for each of the 9Q in the projection window)
Operational Risk Loss Forecasting for CCARStage 3: Simulation of Scenario Analysis for Ops Risk Tail Risk
* The seven loss event categories are internal fraud; external fraud; employment practices and workplace
safety; clients, products and business practices; damage to physical assets; business disruption and system
failure; and execution, delivery and process management
LoB Event Category M.L Freq. M.L. Sev. M.L. Loss W.P. Freq. W.P. Sev. W.P. Loss W.P./M.L. Ratio
GEO 1 Commercial CPBP - Securitization Breach 10 0.25 2.5 0.0067 5000 33.5 13.4
GEO 1 Retail Internal Fraud 1500 0.0001 0.15 0.02 500 10 66.67
Corporate Treasury EDPM 1 2.5 2.5 0.02 200 4 1.60
Corporate IT BDSF - Outage 60 0.01 0.6 0.002 50000 100 166.67
LoB Event Category 45%ile Median 55%ile 60%ile 65%ile
GEO 1 -
Commercial
CPBP - Securitization
Breach84% 100% 121% 145% 175%
GEO 1 - Retail Internal Fraud 93% 100% 106% 117% 142%
Corporate
TreasuryEDPM 75% 100% 132% 168% 215%
Corporate IT BDSF - Outage 90% 100% 113% 129% 151%
Average 86% 100% 118% 140% 171%
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCAROps-Risk Loss Forecasting Results (Bank Total Ops Risk Loss)
17
The bank’s ops-risk loss forecasting for the baseline scenario and three stress scenarios
Tail risk scalars of 118% and 140% are applied to adverse and BHC stress/severe adverse scenarios,
respectively, over the baseline loss projections
Finally, management overlay of $150mn (scaled) is added to the three stress scenarios to cover
legal/litigation/compliance risk for the projection window (not included in the graphs below)
0
20
40
60
80
100
120
The Bank Ops Risk Loss Forecasting
Baseline Adverse BHC Stress Sev. Adverse
0
20
40
60
80
100
120
140
160
180
The Bank Ops Risk Loss Forecasting with Tail Risk
Baseline Adverse BHC Stress Sev. Adverse
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
Operational Risk Loss Forecasting for CCARConclusion and Limitations
18
The link between Stage 1 and Stage 2 of the model is based on the assumption that the
bank’s ops-risk losses and other banks’ losses have similar loss drivers within the same
geography and line of business
Historical average as the starting point for internal loss projection should be on the
conservative side and also in alignment with regulatory expectations (historical
perspective view)
Scenario creation is not based on quantitative analysis, but rather based on forward-
looking of stress scenarios pertinent to the bank’s portfolio and also based on known loss
experience
Tail risk scalars are judgment-based
This ops-risk loss forecasting does not incorporate internal control (e.g. Risk Control Self
Assessment or RCSA) and other risk indicators into the estimation methodology.
However, a few banks have incorporated these into their estimation methodologies
Benchmark and sensitivity analysis have not been conducted, but they are Fed
expectations in the long run
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
www.CRISIL.com/gra
© 2
014 C
RIS
IL L
td.
All
rights
reserv
ed.
