Model Risk Management Framework and Quantification: Perspectives on Potential Approaches

Model Risk ManagementFramework and Quantification:Perspectives on Potential Approaches

Michael Jacobs, Ph.D., CFAPrincipal DirectorAccenture Consulting Finance and Risk ServicesRisk Models, Methodologies and Analytics

April 6, 2016Hofstra UniversityGARP Chapter Meeting

2

The views expressed in the following material are the

author’s and do not necessarily represent the views of

the Global Association of Risk Professionals (GARP),

its Membership or its Management.

3

Agenda

Copyright © 2016 Accenture All rights reserved.

Introduction

Supervisory developments in model risk management

Model development and validation functions

The audit function

Model risk governance framework

01

02

03 04

05

Measurement and aggregation of model risk

0706Conceptual issues: defining a model

4

Introduction – Model Risk Management


• Banks and financial institutions are continuously examining their target state Model Risk Management (MRM) capabilities to support emerging regulatory and business agendas across multiple dimensions

• The field of MRM continues to evolve with organizations developing a robust MRM framework and capabilities.

• To date, industry MRM efforts have focused primarily on model risk management for individual models.• Now, more institutions are shifting focus toward aggregating firm-wide model risk.• Regulatory guidance specifically focuses on measuring risk in individual and in aggregate:

• Next, we will provide background on issues in MRM, including an overview of supervisory guidance.• Then we will discuss various approaches to measuring and aggregating model risk across an institution.

“Banks should consider risk from individual models and in the aggregate.” -Supervisory Guidance on Model Risk Management SR 11-7

“…Board and senior management should establish a strong model risk management framework… grounded in an understanding of model risk—not just for individual models but also in the aggregate.” -Supervisory Guidance on Model Risk Management SR 11-7

5

Introduction – Model Risk Management (continued)


• Modern risk modeling (e.g., Merton, 1974) increasingly relies on advanced mathematical, statistical and numerical techniques to help measure and manage risk in credit portfolios.

• This gives rise to model risk (OCC 2011-12, FED-BOG SR 11-7), defined as the potential that a model used to assess financial risks does not accurately capture those risks, and the possibility of understating inherent dangers stemming from very rare, yet plausible occurrences perhaps not in reference data-sets or historical patterns of data.

• In recent years there has been a trend in financial institutions towards greater use of models in decision-making. This is driven to some extent by the evolution of prudential regulation, but also it also manifest itself in all aspects of management.

• In this regard, a high proportion of bank decisions are automated through decision models, which can be either statistical in nature, or a methodology that constitutes a rule set (MIT, 2005). There are several areas in which we see this trend manifest itself.

• First, as of late there has been an increased use of electronic trading platforms known as algorithmic trading that execute trade commands which have been pre-programmed (e.g., through timing, price or volume), and can be initiated with no manual intervention.

• Second, partly encouraged by the Basel regulations (Basel Committee on Banking Supervision (BCBS) 1999, 2005, 2006, 2009a, 2009b, 2010), banks are increasingly using decision models in their origination, monitoring and credit recovery processes. Therefore, the viability of a loan commitment is determined by its probability of default (PD) of the client, or the loss given default(LGD) of the loan.

6



• Similarly, banks may monitor customer accounts and anticipate credit deterioration using automated alert models, which pre-classify customers and determine their credit limits.

• In the commercial arena, customers are able to select a product’s characteristics (e.g., loan amount, term, purpose, etc.) and the system in turn makes a real-time decision on the viability and price of the transaction.

• The use of valuation models for products and financial instruments has become widespread in financial institutions, in both the capital markets as well as the asset-liability management (ALM) businesses.

• Furthermore, customer onboarding, engagement and marketing campaign models have become increasingly prevalent. Such models are used to automatically establish customer loyalty and engagement actions, both in the first stage of the relationship with the institution, as well as at any time in the customer life cycle.

• Other examples include the calculation of economic capital (EC) charges for various types of exposures, subject to different risk types (e.g., credit, market, operational, ALM, etc.) through their individual components.

• This includes the quantification of the bank’s current liquidity position, projected under alternative scenarios, as well as the projection of balance sheet and income statements for the use in stress testing (ST).

• Furthermore, this includes the modeling of many key components in business planning and development, such as optimal bundling, customer or non-customer income or churn.

7



• The use of models brings many palpable benefits, including:• Automated decision-making, which in turn improves efficiency through reducing analysis and manual

decision-related costs.• Objective decision-making which facilitates the consistency of outputs across a set of similar inputs • Ability to synthesize complex issues such as a bank’s aggregate risk.

• However, model use also gives rise to risks and costs, including:• Direct resource costs (e.g., human capital and economic opportunity cost), as well as time expended in

model development and deployment.• The risk of relying upon output of an incorrect or misused model, for which there exists many recent

examples of specific instances of large losses.

• Model error encompasses phenomena such as simplifications of or approximations to reality, incorrect or missing assumptions, incorrect design processes, measurement or estimation error.

• On the other hand, model misuse includes applying models outside the use for which they were designed and so suited.

• Model risk so defined is potentially very significant and has captured the attention of regulators and institutions, whose approaches range from mitigation via model validation, to the establishment of comprehensive frameworks for active model risk management.

8



• In the case of advanced institutions, such active management has been formulated into a model risk management (MRM) framework that sets out the guidelines for the entire design, development, implementation, validation, inventory and use processes.

• Such developments have been fortified by the fact that prudential supervision and regulation are now requiring such frameworks, as stated in guidance issued by entities such as the Federal Reserve System (FED) and the Office of the Comptroller of the Currency (OCC) (OCC 2011-12, FED-BOG SR 11-7) in the U.S., which are serving as starting points for the industry.

• However, the prudential supervisory guidance fails to provide detail in regard to model risk quantification. An exception to this are rather specialized situations, such as the valuation of certain instruments, in which cases there may even be a requirement for model valuation adjustments (AVA), that may result in in a larger capital requirement or in the possible use of a capital buffer for model risk as a mitigating factor in a broader sense, without its calculation being specified.

• Having set this context, our thought piece “Emerging Trends in Model Risk Management” aims to provide an holistic view of model risk quantification, including a survey of approaches to model risk aggregation:

• Address the conceptual issue of defining a model • Review the supervisory framework with respect to model risk management • Address the model risk quantification question • Present some perspectives on the aggregation of model risk • Provide an empirical example of model risk quantification in the context of stress testing bank credit risk portfolios

9

Supervisory Developments in Model Risk Management


• At this point in time, the supervisory guidance on model risk management is limited, and the regulations tend to be principle-based – that is, there is a lack of specificity in how to define and treat model risk.

• In addition to the guidance issued by the OCC and Fed, there exist some other regulatory references to model risk, that we may categorize into three main types:

• Valuation Adjustments: These are strictures governing a requirement to conservatively adjust the values of certain instruments to account for any potential model risk. These regulations are particularly applicable to financial derivatives.

• Internal Capital Adequacy Assessment Process (ICAAP) Capital Buffer: The 2nd Pillar of Basel II, as well as some local supervisors’ version of this, govern the need to hold capital for all risks deemed material to the institution, as part of the ICAAP. This also holds for a component of ICAAP, the annual stress testing exercise in the US, as described below.

• Comprehensive Capital Adequacy Review (CCAR): In this setting, supervisors may require an institution to hold additional capital for model risk, in spite of the fact that many entities already do so.

• Other References: These include other lower profile mentions of model risk, which are often considered to be subordinate or a component of other supervisory aspects. An important case is the BCBS’s imposition of the leverage ratio as a mitigant to model risk.

• However, the principal regulatory occurrence was the publication of the Supervisory Guidance on Model Risk Management (OCC 2011-12, FED-BOG SR 11-7) by the OCC and Fed in the US. This is the first such guidance in which model risk is clearly defined, and in which entities are urged to have an established framework and processes in place to manage such risk.

10

Supervisory Developments in Model Risk Management (continued)


• In order to meet this objective, the guidance lays out a set of guidelines or principles summarized as follows:

• Model risk is to be managed in the same manner as any other risks faced by an institution. That is, model risk managers should identify the sources, assess the likelihood of occurrence and the severity given the occurrence of a model failure.

• As with other risk types, model risk can never be completely eradicated, rather only controlled by effective management. While comprehensive model development combined with rigorous model validation can mitigate model risk, it will never eliminate it.

• Given that model risk management is to be treated likewise with respect to other risk types, institutions should establish a model risk management (MRM) framework, that is approved and overseen by senior management and the Board of Directors.

• While a well-documented, prudent and structured approach to model development and validation aspects (i.e., inputs, outputs and design) can be efficacious in managing model risk, this is not a substitute for the continual process of improving models.

• A prudent use of models is likely to involve elements such as conservative adjustments, stress testing or possibly capital buffers. However, an over-reliance on these conservative elements may lead to the misuse of models.

• Model risk emanates from many sources, consequently institutions should consider the interaction of these factors, and try to quantify aggregated model risk resulting from their combination.

11

Supervisory Developments in Model Risk Management (continued)


• While the organizational structure of the MRM framework is clearly at the discretion of individual institutions, there are certain strictures of good governance that should be observed, namely the separation of the following elements:

• Model Ownership: Identification and measurement of the model risk to which the institution is exposed

• Model Control: Limit-setting, follow-up and independent validation

• Model Compliance: The set of processes that facilitates the performance of ownership and control rules in accordance with established policies

• Ultimate responsibility, oversight and approval of the MRM framework should reside with the Board of Directors. Furthermore, the Board should be notified of any significant emerging model risk to which the firm is exposed.

• Effective challenge is the principal form of model risk mitigation. This is defined as critical analysis by objective parties who have the following capabilities:

• Experienced professionals in the lines of business in which the model is used

• Have the ability to identify model limitations and assess the validity of assumptions

• Are capable of suggesting model risk mitigants and model improvements

• We gather from this that supervisors are expecting institutions to architect MRM frameworks that feature model development and validation criteria which are formalized, promote careful model use, set criteria to assess model performance, define policy governance and applicable standards of documentation.

• Such a holistic approach to model risk is a relatively novel occurrence in the industry. We expect this to become increasingly prevalent as a wide swath of institutions eventually adopt this approach, as have the more dynamic institutions by this point in time.

12

Model Risk and Governance Framework


The organization structure to address model risk should establish clear lines for development and monitoring, andprovide the correct incentive structures, as well as cross-functional involvement in model risk discussions.

Model Administrator(s)

Board Risk Committee

Board

Model Risk CommitteeTreasury

Inte

rnal

Aud

it

Finance

Risk

Line of Business

• Independence between the model development and validation work streamshelps align incentives for effective model governance

• Internal Audit Review of all processes

Model Validation and

Review

Model Owner(s)ModelUser

Model Implementation

Line of Business

Enterprise Analytics

Methodology Model Development

Model Documentation

Change Management

OngoingMonitoring

13

Roles and Responsibilities


The key roles involved in the model governance process are discussed below. In practice some overlap may occur among the first three roles, with validation, the Model Risk Committee and Internal Audit being independent functions.

Model Owner

• A person assigned responsibility for developing and/or maintaining a model.• Each model should have only one model owner. The model owner is responsible for developing a

complete model documentation that complies with the Model Validation Department’s model document template.

• Should also lead the model implementation process, with the model user.

Model User • A person assigned responsibility for the use of a model for business purposes. Each model could have multiple model users. The model user may belong to a Line of Business (LoB) team.

Model Administrator

• A person who is usually the senior manager of a model owner and is often responsible for assisting the model owner with certain model governance responsibilities. The model administrator is responsible for guiding the team’s model developers in developing a complete model documentation that complies with the Model Validation Department’s model document template. Each model should have only one model administrator.

Model Validation and Monitoring

• An independent organizational unit assigned responsibility to implement and oversee the Model Governance Program on a daily basis. This unit is also typically responsible for model monitoring.

Model Risk Committee

• The senior management committee with responsibility for overseeing the Model Governance Program. The committee reports risk issues and trends to the Risk Committee of the Board or take appropriate action.

Internal Audit • Internal Audit Function for independent review of all model governance activities.

14

Modeling Function


The favored approach for setting up an efficient modeling function is to have a modular team structure with specialized Pods or Center of Excellence (CoE) addressing different aspects of the lifecycle. An example is provided below.

Methodology Development Documentation Validation

Enterprise Analytics

Implementation

Model Risk Committee

Responsible for model design research, develops list of approved methodologies and steps.

Executes the development steps for the model(s) concerned, follows an established process.

Documents models via standardized approved templates, works with all other functions.

Cross functional (LoB, IT, Analytics) team specializing in model implementation and User Acceptance Testing process.

Independentvalidation and reviewfunction.

15

Modeling Validation Function


A robust approach to model validation should cover a quantitative and qualitative review of modelsacross 4 broad areas: data, methodology, processes and governance.

• Is the development sample appropriately chosen?

• Is the data base sufficient for the defined rating requirements?

• Is the quality of the data sufficient enough to develop a model?

• Is the data being transferred correctly between the systems?

• How is data integrity assured?• Is application security effective?

• Are the processes efficient enough to secure an effective execution of the ratings calculation?

• Are the rating models well documented with sufficient developmental evidence?

• Are the rating outputs effectively used to run and manage the business?

• Is there a feedback process in place in order to update the tool in case of identified inadequacies or issues?

• Does validation result support discriminatory power and calibration of the rating model, even in times of economic cycles?

• Is the rating system stable over time?• Is the bank’s rating system accurate?• Is the rating approach state-of-the-art?• Are there appropriate and relevant

theories and assumptions regarding the rating models in place? Has this been verified by appropriate validation?

• Are the organizational governance policy frameworks for the development, ongoing monitoring and use of the rating models robust enough?

• Is there a set mechanism for annual review of rating models?

• Is the rating model’s performance being monitored, and included in audit processes?

Methodology

Governance

Data

Processes

Quantitative Validation

Qualitative Validation

Validation Areas

16

Model Validation Common Challenges


There are challenges around the availability of data and limitations to the verificationmethodologies used during model validation. Accenture has identified the following challengesand tools/techniques to address these.

• Combine other bank or market data.• Combine internal portfolio segments with similar risk characteristics.• Combine different rating categories.• Relax default definition to create pseudo defaults.

Low Default Incidence

• Methodology validation by portfolio. Example, mathematical form of the model.• Industry benchmark for factor selection by line of business and industry.

Model Complexity

• Data access, completeness and portfolio representation.• Data quality tools.

Data Quality

• Industry standards for traffic lights by portfolio segments.• Strong documentation for regulatory and internal model validation team.

Inconsistent Modeling Standards

• Developmental evidence of vendor products including components and design.• Ongoing monitoring and outcome analysis of vendor model performance using

model output.Off-the-Shelf Vendor Models

Challenges Tools/Techniques

17

Model Monitoring: Risk Rating and Inventory


A risk rating system for model issues is essential for systematic escalation and making tracking and issue resolution a function of model importance. A comprehensive model inventory is the basis of all monitoring.

A model inventory should be an organized enterprise level repository of all models within the organization or at a bare minimum of all Tier 1 and 2 models. The Fed notes in its CCAR guidelines that banks with lagging practices were unable to identify all models in their capital computation process.

Model Inventory Contents of a Good Model InventoryModel name and descriptionVersion number (if applicable)Model ownerBusiness unitModel statusRisk typeModel documentation available or not and if available where it is locatedModel tierStatus (in development, being validated, in production, etc.)Production system/platform(s) and production system/platform owner(s)Date of the most recent independent validation or the expected completion dateValidation status

The level of review applied to each model should be commensurate with the level of risk posed bythe model; more frequent and more extensive review steps will be applied to the most complex and highest impact models.

Model Risk Rating Level of Concern

Low Med High

Mod

el R

isk Tier 1

High Materiality, External

Disclosure/ Regulatory

Impact

Qualification Finding Finding

Tier 2 High Materiality Comment Qualification Finding

Tier 3 Other Comment Comment Qualification

18

Model Monitoring: Monitoring Process and Tools


Model monitoring should be regular and comprehensive, covering all aspects of model performance (and not just discrimination performance). An intuitive “traffic light” system for model diagnostics is a preferred practice in this area.

KS Statistic

Gini Coefficient

% Change in Gini

Discrimination

Portfolio Information

Discrimination Scoring System Stability

CharacterStability Calibration Overall

Score

Automated Recommen

dation

Overall

Herfindahl Index

Overrides

Migration

Scoring

Overall

PSI

Event PSI

Non-Event PSI

System Stability

Overall

Character Stability Index

Information Value Variance

Character Stability

Overall

Hosmer Lemeshow Test

Calibration Curve Shape Test

Calibration

Overall

Combined Score

Modelers Assessment

Overall

Automated Recommendation

Legend: Event PSI: Event Population Stability IndexKS Statistic: Kolmogorov-Smirnov Statistic

Non-Event PSI: Non-Event Population Stability IndexPSI: Population Stability Index

19

Model Change Management


Model change management should incorporate model risk rating and the complexity of the change as inputs. It should also include standardized processes for supporting change recommendation as well as execution.

Discrimination

Scoring

System Stability

Character Stability

Calibration

Overall

Monitoring Change ApprovalEscalation Process

Change Execution

Re-development

Re-calibration

Dropping/Addition of Variables

Change Validation

Independent Validation of changes executed by the Model validation team, with report to the MRC.

Change Implementation

Implementation of recommended changes by the Implementation team.

20

Audit Function’s Role in Model Risk Management


Regulators have been focused on model risk management since the credit crisis of 2007, an event which put much pressure on the Audit functions. Accenture can assist clients by providing support to Audit Functions with reviews of quantitative models across banking and trading books

Regulatory Expectations

Banks under scrutiny from the Fed and OCC to improve model risk and model review processes. Increased Regulatory expectations on the Audit function as 3rd line of defense for improved model

risk management. Regulators expect the Audit function has established processes with in-house experience and skills

to deliver in-depth review of quantitative models and model risk frameworks.

AuditFunctions

Serve as a 3rd line of defense delivering specialized and quantitative reviews of bank models and frameworks.

Established processes, policies and procedures in place that provide a replicable basis to effectively challenge independent model validators and model developers.

21

Audit Function’s Role in the Model Risk Management Framework


Finalize review of the scope (Matters Requiring Attention, Matters Requiring Immediate Attention, Internal Reviews)

Finalize relevant documents, policies, procedures and standards

Gather relevant information

Compile relevant internal policies, procedures, standards

Collect relevant model documents, validation reports, testing files

Assessment of completeness of submission

Assess against regulatory guidance such as Basel, OCC, notice of proposed rule-making, etc.

Review of submitted documents against internal submission standards and procedure guidelines

Assess submitted documents with industry preferred practices and emerging market trends

Identify gaps in model development process

Identify gaps in model validation process

Develop discussion documents and conduct reviews with MRM and LoB

Review submitted documents against regulatory expectations, internal policies and industry preferred practices

Develop methodology gaps

Develop audit assessment report

Review Scoping Document Review Regulatory and Industry Assessment

Gap Assessment and Documentation

Review Issue and Materiality Assessment

Testing Planning (steps, timing resources)

22

Emerging Trends: Quantification of Model Risk


Identification Quantification Mitigation

- Data error- Missing variables- Insufficient time period- Insufficient sample

- Sampling error- Parameter uncertainty- Computational complexity- Invalid assumptions

- Model not re-calibrated in a timely manner- Model used for incorrect purpose- Model execution error

- Measure decay in predictive power between re-estimations- Measure impact of not using the model

- Measures of statistical estimation error- Benchmark models- Market benchmarks

- Sensitivity of output to exclusion of variables or data point /time periods

- Data quality assurance processes

- Quantitative capital buffer for MR- Conservatism in inputs, estimates and outputs- Model backtesting and stress testing

- Strict model control environment and governance - Ongoing monitoring (limits, alerts, etc.)

Dat

aE

stim

atio

nU

sage

MR Source

23

Conceptual Issues: Defining a Model


• In the analysis of model risk, the primary question is how to define a model. According to the supervisory guidance, a model could constitute any quantitative method, methodology or rule set.

• What any of these have in common is that these approaches apply statistical, economic, financial, or mathematical theories, techniques or assumptions to help transform input data into quantitative estimates. Broadly speaking, this construct has three components:

• Inputs: These constitute either data (“hard” or an expert opinion based), hypotheses, assumptions or other model output.

• Processing Apparatus: This is a method, technique, system or algorithm for transforming model inputs into model outputs. This may be statistical, mathematical or judgmental.

• Reporting Component: This is the system for converting model outputs into a form that is useful for making business decisions.

• According to the supervisory guidance, it is always the case that the definition of a model also encompasses quantitative approaches in which the inputs are partially, or completely, qualitative or expert-based.

• The only proviso that distinguishes a non-hard data input system such as a model is that the output is somehow quantitative in nature.

• Therefore, the concept of a model, as understood herein, is far broader than the concept of a mathematical algorithm which can be seen in this context to be an incomplete interpretation.

• In particular, this interpretation admits expert judgmental constructs as well.

24

Conceptual Issues: Defining a Model (continued)


• Nevertheless, it can be easily seen that such a definition is still subject to interpretation, as per the previous point and what follows should be considered as models:

• An algorithm for the calculation of value-at-risk (VaR) or credit VaR (C-VaR), for credit or market risk respectively, using either a structural model, Monte Carlo or historical simulation;

• a scoring model that can be calibrated to estimate a PD for a loan portfolio, using techniques such as logistic regression; or,

• valuation constructs with respect to instruments (e.g., derivatives, credit exposures, insurance contracts, etc.) or portfolios of such.

• The converse under this definition would call into question whether the following are models:• Data aggregation algorithms such as statistical summary statistics with respect to a data series (e.g., sums, averages,

standard deviations, etc.), financial ratios or calculators that combine model output (e.g., an expected loss calculation for Allowance for Loan and Lease Losses (ALLL);

• a growth projection, or a discounted cash flow calculation, based on a single or limited number of years of historical rate of change(s), or market discount rates; or,

• a rule set for decision-making that is based upon a single variable and either a judgmentally or data-based threshold (e.g., an acceptance vs. rejection decision for a real estate loan based upon a Loan-to-Value ratio).

• However, institutions have the latitude to decide what is in scope as a model, and would be subject to model risk, and therefore affected by model risk policies and supervisory guidance.

• Such scope will often not be precisely defined, hence the application of expert subjective judgment should necessarily be a component of the process. However, once such scope is crystalized, a companion decision should be at hand regarding which models to analyze (e.g., risk, commercial, financial, etc.).

25

Model Interdependencies and Challenges


• One challenge in aggregating total model risk is assessing the model interdependencies, which impact the total model risk for a firm.

• Regulators expect firms to consider such interdependencies as part of the process: “Aggregate model risk is affected by interaction and dependencies among models; reliance on common assumptions, data, or methodologies; and any other factors that could adversely affect several models and their outputs at the same time.” – OCC 2011-12: Supervisory Guidance on Model Risk Management.

• A simple approach would look at making “conservative” assumptions in adding model risk outputs from different models, and where appropriate, model risk across models as if the risks were perfectly correlated.

• A more advanced approach may involve expanding model inventory to create a detailed taxonomy of identify model and input dependencies: • Downstream Models – determine impact on downstream models originating from the base

model’s outputs• Common Inputs – assess the impact of input issues on multiple models sharing those

inputs

26

Measurement and Aggregation of Model Risk


• Measuring and aggregation of model risk is a new and evolving field • Multiple methodologies are being considered but none are established as a

“standard” • What follows are three general approaches for discussion

Approach 1: Model risk scorecardA qualitative scorecard approach that considers inherent model risk factors and model risk mitigation activities to present an aggregate view of model risk across an institution.

Approach 3: Model uncertaintyA bottom-up, quantitative approach to measuring model risk based on model risk sensitivity and model risk control activities such as benchmarking, backtesting, consideration of alternative modeling techniques, and corresponding impact.

Approach 2: Advanced Operational riskA quantitative approach to model risk aggregation based in part on the event and scenario modeling techniques of the Advanced Measurement Approach (AMA) to operational risk.

27

Approach 1: Model Risk Scorecard


Borrowing from the traditional Enterprise Risk Management (ERM) frameworks, the Model Risk Scorecard Approach considers inherent model risk factors and model risk mitigation activities to present an aggregate view of model risk across an institution. To measure model risk using the Scorecard Approach, each model in the institution’s model inventory is assessed based on the following factors.

Model risk scorecard approach

Model uncertainty approach

Advanced operational risk approach

Inherent Model Risk

Model Risk Mitigation

Overall Model Risk

28

Approach 1: Model Risk Scorecard (continued)





Inherent Model Risk

The inherent model risk represents the starting point of the model risk management process for an individual model, and can be a function of a number of factors (similar to the approach for tiering models), including:• Model complexity• Higher uncertainty about inputs and assumptions• Broader use• Larger potential impact

Model Risk Mitigation

The inherent model risk is mitigated (or hedged) by a combination of model risk control activities within the business units and by dedicated MRM control functions:• On-going model risk management by model owners (First Line of

Defense)• Model performance monitoring• On-going model benchmarking • On-going model backtesting

• Activities of the MRM Control functions (Second Line of Defense)• Model validation (initial and periodic) and resulting findings• Other model control activities (e.g., controls around model use, model

calibration, model limit monitoring, etc.)

Overall Model Risk

• The overall model risk is a function of inherent model risk and model risk mitigation activities.

• Can be used to measure and monitor model risk for individual models and in aggregate, and for periodic reporting to senior management and the board

29

Approach 1: Model Risk Scorecard (continued)


Overall Model Risk depends on specific combination of inherent model risk factors and model risk mitigation. Below are several illustrative examples of potential combinations:

Scorecard Model A Model B Model CInherent Model Risk

Model Complexity Score Low High LowModel Impact Score Low High Low

Model Risk MitigationModel Risk Management Activities Score

High Low Low

Model Validation Activities Score High Low HighOverall Model Risk

Overall Model Risk Low High Medium




30

Approach 1: Model Risk Scorecard – Risk Aggregation


• .

Model A

Model B

Model C

Model D

Risk Weight

Risk Weight

Risk Weight

Risk Weight




Total Model Risk

• To measure aggregate model risk for the entire model inventory, an aggregation methodology can be applied.

• Aggregation can utilize weights to arrive at a more representative overall score.

• The resulting total is a subjective score; however, it can be utilized as a relative measure for period to period monitoring.

31

Approach 1: Model Risk Scorecard (concluded)


Advantages

• Effectively aggregates model risk across institution.

• Can leverage existing model inventory and risk classifications.

• Clear, easy to understand reporting tools for senior management.

• Could be rolled out more quickly than some alternate approaches.

Disadvantages

• Aggregates but does not attempt to quantify model risk.

• Scores, weights, and overall results are subjective and based on judgment.

• Limited ability to capture model inter-dependence.




32

Approach 2: Advanced Operational Risk – An Overview


• Approach based in part on the event and scenario modeling techniques of the Advanced Measurement Approach (AMA) to operational risk.

• Model risk viewed as a function of the frequency and severity of potential model risk events.

• Based on a combination of historical experience and scenario analysis.

Loss Event Information

Loss DistributionsOperational Risk Model

Total Loss Distribution

Event Frequency

Severity of Loss

Loss Event Data

Management Judgment

Simulation Engine

Loss

am

ount

($)

1 yr

Capitalnumber

Monte Carlo Simulator




33

Approach 2: Advanced Operational Risk – An Overview (continued)


• A database of historical Model Risk events used to estimate the historical component of model risk.

• Model-risk specific consideration: common loss measure (P&L, capital impact, etc.)

• Data gathering considerations: • External data sparse – industry loss event databases may not include model risk events

or may not identify them as such; • Internal data may not be tagged as “model risk” related;

• Consideration can be given to using backtesting/benchmarking data as proxy.

Model Risk Event Data

• Similarly to AMA scenario process, workshops with model owners and users can be used to determine model risk event types, frequency, and severity

• Workshops focus on capturing the universe of model-specific risk events: • What could go wrong? • What could be the range of potential impacts, and their frequency?

• Highly subjective and judgment-based process common to the rest of AMA framework

• Attempts to capture potential model risk events that have not yet occurred.

Model Risk Scenario Analysis




34

Approach 2: Advanced Operational Risk (concluded)


Advantages

• It is empirically grounded, therefore subject to model validation, and likely to be favored by regulators.

• Aggregates and quantifies model risk in a coherent manner.

• Institutions already using AMA can leverage existing infrastructure and processes.

• Can be used as input in estimating economic capital for model risk.

Disadvantages

• Historical model risk event data tends to be sparse.

• Scenario-based frequency and severity estimates highly subjective and based on judgment.1

• Distribution assumptions are subjective and can affect outcomes.1

• More complex to implement, especially without existing AMA infrastructure.




1 Common to all AMA models

35

Approach 3: Model Uncertainty


• A bottom-up, quantitative approach.

• Based on model risk sensitivity.

• Model uncertainty measurements derived from benchmarking, backtesting, consideration of alternative modeling techniques, and corresponding impact.

Model Output

Benchmarking Output(including backtesting and alternative modeling techniques)

Model Uncertainty




36

Approach 3: Model Uncertainty (continued)


Additional Considerations

This approach utilizes information from various sources and translates model risk into metrics such as P&L or capital impact.

• Model output sensitivity to key modeling assumptions

• Model benchmarks

- Parallel run results

- Alternative models

- Alternative approach/methodology

• Output benchmarks – internal or external data

- Industry data

- Counterparty data

• Backtesting results

• Model validation results and observations




37

Approach 3: Model Uncertainty (continued)


Illustrative Examples

The examples that follow demonstrate how model output variability can be traced to downstream metrics like P&L and capital impact.

Model benchmarks

• Sensitivity analysis to compare model outputs and an alternative benchmark model.

• Example: Historical Value-at-Risk (VaR) model can be compared to a simpler alternative methodology such as Variance-Covariance method.

Input benchmarks

• Sensitivity analysis to compare model outputs based on data from external sources (e.g. alternative source, aggregated/consensus data).

• Example: Alternative prepayment speeds can be obtained from an external source and used to assess the impact.

Model validation findings

• Recalculations during the validation process indicated a range of variability of model results based on key modeling assumptions.

• Example: Impact on model outputs of interpolation settings such as using linear vs. spline.




38

Approach 3: Model Uncertainty (concluded)


Advantages

• Can perform bottom-up quantification of model risk for individual models.

• Allows for a more rigorous and less subjective measurement of model risk.

• Can be used to capture model inter-dependence.1

• Can be used as input in estimating economic capital for model risk.

Disadvantages

• Measurement and aggregation assumptions are highly subjective and based on judgment.

• Development of individual, model-specific approaches can be more time and resource intensive.

• Maintenance relies on the on-going performance of model control activities (benchmarking, backtesting, external data comparisons, etc.).




39

Model Risk Factors


• While there are many ways to measure and aggregate model risk, it is useful to define how such measurement system would respond to changes in key “Model Risk Factors.”

• Holding everything else constant, the following factors can be expected to increase or decrease aggregate model risk.

Model risk factor Aggregate risk

Introduction of a new model

Change to an existing model

Increase in model exposure

Model use change

Model error identified

Backtesting exceptions outside normal range

Backtesting not performed during period

Higher divergence from benchmarks

Model not validated for some time

A model validation has recently been performed

Periodic backtesting process initiated

Additional model controls put in place

Model retired and is no longer in use

Model has been in use for some time

40

Conclusion


• Models can be tremendously helpful to financial services firms.

• As the use of models increases, so does model risk.

• This is an area in which the potential impact of flawed modeling can be great, and it is also an area receiving more regulatory attention and scrutiny.

• Firms can benefit from a structured approach to model risk that incorporates both a framework for developing and testing of models as well as effective governance of matters such as model validation, inventory and aggregation.

• When properly designed and implemented, models should be a valuable resource for financial institutions, but firm should deploy well-conceived programs to improve models’ utility while identifying, quantifying and mitigating their potential risk.

41

References


• A classic example in the field of credit risk is the structural modeling framework, which underlies any of the leading models used in the industry (e.g., JP Morgan Chase & Co.’s CreditMetrics), the classic reference being Merton, R., 1974, “On the pricing of corporate debt: The risk structure of interest rates,” Journal of Finance 29(2): 449–470. Access at: http://dspace.mit.edu/bitstream/handle/1721.1/1874/SWP-0684-14514372.pdf?sequence=1.

• Refer to the following regulatory guidance; (The Basel Committee on Banking Supervision, 2009, “Principles for Sound Stress Testing Practices and Supervision - Consultative Paper,” The Basel Committee on Banking Supervision, May 2009 (No. 155). Access at: http://www.bis.org/publ/bcbs155.htm. Academic, “A coherent framework for stress testing,” Jeremy Berkowitz, Journal of Risk, Winter 1999/2000. Access at: https://www.risk.net/data/basel/pdf/basel_jor_v2n2a1.pdf.

• “Automated decision-making Comes of Age,” Thomas H. Davenport and Jeanne G. Harris, MIT Sloan School Management Review, July 2005. Access at: http://sloanreview.mit.edu/article/automated-decision-making-comes-of-age/.

• A recent example of this is an automated trade command that occurred on May 6, 2010 and that resulted in a $4.1 Billion “flash crash” of the New York Stock Exchange (NYSE). The NYSE fell more than 1,000 points, subsequently recovering to its previous value in only 15 minutes time. “Findings regarding the market events of May 6, 2010, Report of the staff of the CFTC and SEC to the Joint Advisory Committee on Emerging Regulatory Issues.” Securities & Exchange Commission and Commodity Futures Trading Commission, September 30, 2010. Access at: https://www.sec.gov/news/studies/2010/marketevents-report.pdf.

• “Supervisory Guidance on Model Risk Management,” The Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency, SR 11-7, April 4, 2011. Access at: http://www.federalreserve.gov/bankinforeg/srletters/sr1107a1.pdf.

• “Validation of economic capital models: State of the practice, supervisory expectations and results from a bank study,” Michael Jacobs Jr., Journal of Risk Management in Financial Institutions, January 2010.

• “Range of practices and issues in economic capital frameworks – final paper,” Basel Committee on Banking Supervision, March 2009. Access at: http://www.bis.org/publ/bcbs152.htm.

• Refer to the following regulatory guidance; (The Basel Committee on Banking Supervision, 2009, “Principles for Sound Stress Testing Practices and Supervision - Consultative Paper,” The Basel Committee on Banking Supervision, May 2009 (No. 155). Access at: http://www.bis.org/publ/bcbs155.htm. Academic, “A coherent framework for stress testing,” Jeremy Berkowitz, Journal of Risk, Winter 1999/2000. Access at: https://www.risk.net/data/basel/pdf/basel_jor_v2n2a1.pdf. “Principles for sound stress testing practices and supervision – final paper,” Basel Committee on Banking Supervision, May 2009. Access at: http://www.bis.org/publ/bcbs155.htm.

Principal Director in the Risk Analytics practice of Accenture. He specializes in risk model development and validation across a range of risk and product types.Mike has 25 years experience in financial risk modeling and analytics, having worked 5 years for Deloitte and PwC as a Director in the risk modeling and analytics practice, with a focus on regulatory solutions; 7 years as a Senior Economist and Lead Modeling Expert at the OCC, focusing on ERM and Model Risk; and 8 years in banking as a Senior Vice-President at JPMC and SMBC, developing wholesale credit risk and economic capital models.Skills include model development & validation for CCAR, PPNR, credit / market / operational risk; Basel and ICAAPP; model risk management; financial regulation; advanced statistical and optimization methodologies.

Global Financial Institution: Counterparty Credit Risk / CVA & Basel IRB Market Risk Advisory • Advisory as a quantitative modeling expert in assessment and development of the

client’s CCR modeling framework for the Internal Models Methodology• Assisted in redeveloping underlying models and methodologies for PD / LGD / EAD,

EPE, credit correlation across varied exposure types and asset classes• Developed documentation of model development, validation and process flow / control

across the entire CCR engine, addressing regulatory MRAsLarge Commercial Bank: Capital Planning & CCAR Model Development• Advisory as a quantitative modeling expert in an assessment of the bank’s framework

for capital planning and stress testing. • Model development and validation for operational risk, CRE / C&I and retail credit, as

well as loss forecasting frameworks. • Advisory on ICAAP as a quantitative modeling expert in a comprehensive gap

assessment of the bank’s framework for capital planning and stress testing. Super-Regional Bank: Model Risk Management Advisory• Development of a Model Risk Management framework for all material models • Drafted policies & procedures related to model risk governance, model validation and

development, model risk appetite, model inventory & designed model risk scorecards. • Helped the client build out an enterprise risk management (ERM) function through

working closely with lines of business and IT to implement model risk process workflow control procedures, remediate validation findings and establish operating procedures for the interface between model owners and ERM.

• Contributed to model governance and validation sections of the Client‘s inaugural capital plan submitted in preparation for participating in the supervisory CCAR process.

Large Non-Bank Financial Institution: Model Risk Management Advisory• Advisory to the corporate MRM group as a quantitative modeling expert. • Assisted in responding to supervisory MRAs related to the model validation processes

covering all material models (PD/LGD, rating scorecards, ALLL/Reserving, Stress Testing / CCAR, Economic Capital).

Select Experience

Experience

Michael Jacobs Jr, PhD, CFAPrincipal DirectorFinance & Risk ServicesModels, Methodologies & Analytics

1345 Avenue of the AmericasNew York, NY 10105T: (917) [email protected]

Functional/Technical Knowledge:• Statistical modeling & optimization• Model development & validation• Economic capital modeling• Model risk management• Operational risk modeling• Stress testing

Industry Experience:• Banking & capital markets• Prudential regulation• Credit derivatives &

counterparty credit• Wholesale credit• Basel / ICAAP/ CCAR

Background

Michael Jacobs Jr, PhD, CFA

PhD Mathematical Finance CUNY Graduate Center / Zicklin-Baruch School of Business; MA Mathematical Economics, SUNY Stony Brook School of the Arts; MS Applied Mathematics & Statistics, BS Engineering Mathematics SUNY Stony Brook / School of Engineering and the Applied Sciences; Chartered Financial Analyst

Education• Extensive publication track record in many prestigious refereed

practitioner & academic journals (JFM, JRMFI, JFI, JCR, JPM, JRMV) • Has given talks at several high profile industry venues (PRMIA, GARP)

Thought Leadership

Model Risk Management Framework and QuantificationPerspectives on Potential Approach

43Copyright © 2016 Accenture All rights reserved.

Disclaimer:

This presentation is intended for general informational purposes only and does not take into account the reader’s specific circumstances, and may not reflect the most current developments. Accenture disclaims, to the fullest extent permitted by applicable law, any and all liability for the accuracy and completeness of the information in this presentation and for any acts or omissions made based on such information. Accenture does not provide legal, regulatory, audit, or tax advice. Readers are responsible for obtaining such advice from their own legal counsel or other licensed professionals.

About Accenture

Accenture is a leading global professional services company, providing a broad range of services and solutions in strategy, consulting, digital, technology and operations. Combining unmatched experience and specialized skills across more than 40 industries and all business functions—underpinned by the world’s largest delivery network—Accenture works at the intersection of business and technology to help clients improve their performance and create sustainable value for their stakeholders. With more than 373,000 people serving clients in more than 120 countries, Accenture drives innovation to improve the way the world works and lives. Visit us at www.accenture.com

Accenture, its logo, and High Performance Delivered are trademarks of Accenture.

C r e a t i n g a c u l t u r e o f r i s k a w a r e n e s s ®

Global Association ofRisk Professionals

111 Town Square Place14th FloorJersey City, New Jersey 07310U.S.A.+ 1 201.719.7210

2nd FloorBengal Wing9A Devonshire SquareLondon, EC2M 4YNU.K.+ 44 (0) 20 7397 9630

www.garp.org

About GARP | The Global Association of Risk Professionals (GARP) is a not-for-profit global membership organization dedicated to preparing professionals and organizations to make better informed risk decisions. Membership represents over 150,000 risk management practitioners and researchers from banks, investment management firms, government agencies, academic institutions, and corporations from more than 195 countries and territories. GARP administers the Financial Risk Manager (FRM®) and the Energy Risk Professional (ERP®) Exams; certifications recognized by risk professionals worldwide. GARP also helps advance the role of risk management via comprehensive professional education and training for professionals of all levels. www.garp.org.

44 | © 2014 Global Association of Risk Professionals. All rights reserved.

Economy & Finance

Model Risk Management Framework and Quantification: Perspectives on Potential Approaches