Model Validation

© 2015 Protiviti Inc.

CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party.

TREPP BANKING SYMPOSIUM MODEL RISK MANAGEMENT

OCTOBER 9, 2015

TABLE OF CONTENT

• MODEL DEFINED

• EFFECTIVE CHALLENGE

• TOP ISSUES

• THREE LINES OF DEFENSE

• VALIDATION TASK & TECHNIQUES

• ROLE OF INTERNAL AUDIT

• QUESTIONS



A Model Defined A model is a computer-based mathematical construct that processes data inputs and produces

outputs used in business decision making

3

A model is not a decision-maker

A model is only as good as the assumptions and inputs

that go into it

Even a great model will “fail” if it is used inappropriately

Even a great model that has been used appropriately will

need to be updated for environmental changes



Financial Models Differ From Other User Developed Tools

4

• Provided by vendors or developed in house

using a wide range of platforms.

• An approximate representation of the real

world that can be used to calculate prices,

risks, reserves, and strategies for financial

products.

• An estimate based on a set of assumptions

about the behavior of the people,

organizations, risks, and other models

participating in the marketplace

• A model may reside within a user developed

tool.

A Quantitative Financial Model is:

• Also known as an End User Tool,

Spreadsheet or just tool.

• End user tools tabulate known or agreed

values.

• If a calculation is objectively true with no

possibility of an incorrect assumption, it is not

a model.

A User Developed Tool is:

Model Validation and Audit should provide effective challenge around controlling risks associated with

all business tools.



Effective Challenge In Model Risk Management Effective challenge for model risk involves three guiding principles: independence from model

developers, competent resources, and an influence within the organization that values and

supports the function’s role.

5

Senior Management & Board Oversight

Risk Identification & Assessment Risk Measurement Risk Reporting,

Monitoring & Management

Effective Challenge

Re

so

urc

es Organizational Structure

Risk Culture

Policies & Practices

Independence/Incentives

• The effective challenge function (e.g.

Model Validation, Governance

Committees, Internal Audit) or model

risk should be distinctly separate

from the model development process

• Challenge should also be

appropriately supported with well-

designed compensation practices

Competence

• Individuals in the challenge function

require technical expertise and

modeling skills to effectively validate

model assumptions, limitations,

usage, and outcomes

• Understanding of business

processes and financial knowledge

enhance the challenge function’s

value in risk management

Influence

• Effective challenge has explicit

authority and responsibilities

supported by policies and procedures

as well as Senior Management’s

commitment

• The challenge function has stature

and visibility within the organization

and is a respected partner to the

business



Guidance For Effective Challenge of Model Risk Management The OCC/FED supervisory guidance on model risk management outlines key principles and

control mechanisms for model risk, including effective challenge.

6

• Developers should consider models’ impact on business performance from user feedback and should factor in

suggestions and criticism from sources independent of the business

• Existence of effective challenge is required to identify model limitations and assumptions

Model Development, Implementation, and Use

• Validation of models should be performed by a function independent of the developers

• Each model should be reviewed at least annually to assess material changes and whether validation activities

are sufficient.

• Models should be validated for conceptual soundness including the validity of assumptions, limitations, and

outcomes

• Models from 3rd party vendors should be evaluated for appropriateness to the institution’s situation and require

ongoing monitoring by the institution and the vendor

Model Validation

• Board and Senior Management should support effective challenge through establishment of a robust model risk

framework that includes policies and procedures outlining the roles and responsibilities for model validation and

quality assurance functions

• Risk Management functions are responsible for managing model validation and review processes to ensure

effective challenge occurs

• Internal Audit evaluates the design and effectiveness of controls for model risk management and communicates

audit findings to the business for remediation

• Use of external resources for model validation, compliance, or audit requires clear lines of communication to

internal parties for identified issues

Model Governance, Policies, and Controls



Top 10 Issues In Model Validation

7

Data constraints 1

Lack of explanation for business judgment 2

Inadequate model documentation 3

Invalid or unrealistic assumptions 4

Too many performance testing exceptions 5

Difficulty in choosing the appropriate level of conservatism 6

Lack of alternative model analysis 7

Inadequate validation of vendor models 8

Inadequate understanding of modeling software functionalities 9

Model risk has inadequate stature in the organization 10



Model Governance & Validation

2nd Line of Defense

Line of Business

1st Line of Defense

Industry Leading Framework for Model Risk Management

8

Model Governance and

Model Validation

Model Governance Committee(s)

Board of Directors

Model Owners and Model Users

Internal Audit

3rd Line of Defense

Internal

Audit



Industry Leading Practices

9

First Line of Defense Model Developer

Second Line of Defense Risk Management /

Model Validation Group

Third Line of Defense Internal Audit

The Issuance of OCC bulletin 2011-12 / FRB SR 7-11 has prompted a shift in all banks to

place heightened emphasis on Model Risk Management, including model validation

The largest banks have begun to appoint a Chief Model Risk Officer

Medium-sized banks have adopted model risk policies designed to drive a structured

model risk governance process with a focus on effective challenge

Some banks with assets over $10 billion now have a dedicated model validation function

Model Risk Management Infrastructures focuses on three levels of control

Executive Management and The Board of Directors are expected to review and approve

a Bank’s model risk governance process



Roles in Model Risk Management Model validation and internal audit ensure controls are designed adequately and implemented

effectively to reduce the likelihood of erroneous model output or incorrect interpretation of model

results.

10

Activity

Players Development Implementation Production Reporting Validation

Model

Developers

• Code

• Document

assumptions,

theory, and

testing

• Conduct user

acceptance

tests (UAT)

• Monitor

performance

• Support user

interpretation

of model

results

• Provide code

and

documentation

Independent

Validators

• None (maintain

independence)

• Review

implementation

• Periodic

reviews

• Ensure model

reports are

used

appropriately

• Evaluate

• conceptual

soundness,

outcomes

analysis,

• ongoing

monitoring

Internal

Audit

• Evaluate

adequacy of

development

guidelines

• Test

completeness

of development

documentation

• Test systems

development

life cycle

(SDLC)

• Evaluate

model risk

monitoring

controls

• Evaluate

completeness

of reports

• Evaluate design

adequacy of

validation

guidelines

• Test

effectiveness of

validations



Validation Tasks

11

• Interview model developers and other key stakeholders to obtain an understanding of the models

• Review and assess whether the documentation is sufficient to allow a third party to reconstruct the

model if necessary

Interviews & Model

Documentation

Review

• Review developmental evidence for the model

• Review assumptions

• Conduct impact analysis for underlying assumptions

• Assess theoretical limitations

• Assess whether theoretical and conceptual framework aligns with the purpose

Model framework,

Theory & Design

• Review calculations within ranges of industry practices and regulatory requirements

• Independently replicate calculation components

• Independently replicate the model calculation process and benchmark outputs

Process, Code

& Math

• Review the model control environment by testing a sample of model inputs if applicable

• Review the reasonableness of the calculation of input variables

• Review and evaluate missing data handling process

• Review and evaluate the performance of input model if applicable

• Assess the reasonableness of key parameter inputs and assumptions

• Assess the alignment of qualitative and quantitative assumptions, supporting business rationale,

and statistical analysis, if applicable

Model Input Quality

• In-sample, out-of-sample and out-of-time sample validation tests

• Multiple testing approaches may be applied for different types of models

Performance

Testing



Common Validation Techniques There are a variety of techniques that can be applied in the performance of model validations.

Some of the various techniques are described below. Many of these techniques are applied in

combination with other testing approaches.

12

Techniques Description

Back Testing Compare actual outcomes with model predicted outcomes using statistical measures.

Sensitivity Analysis Evaluate the sensitivity of changes in model output in relation to changes in input (macro

variables, interest rates, home prices, etc).

Benchmarking Compare internal model results to other like models (e.g. industry models).

Model Replication Replicate design, attributes, and mathematical form of models. Compare outputs between

parallel replica and the actual model.

Cash Flow

Replication

Estimate cash flows outside of the model validating that costs, expenses, instrument

attributes, and assumptions have been properly considered.

Calibration Testing

Where model inputs are not available. Calibrating the score or log-odds to observed default

rates over various time periods. Calibrations techniques such as determining calibration of

FICO scores to default rates.

Assumption &

Theory Review Justify the reasonableness of assumptions and modeling simplifications.



Industry Model Validation Approaches Below is an illustration of where certain techniques maybe useful given model type or subject

matter. The appropriateness of specific procedures must be determined by qualified personnel

and in the context of the model’s design and intended use.

13

Model Types Back

Testing

Sensitivity

Analysis

Bench-

marking

Model (or

Modular)

Replication

Cash Flow

Replication

Calibration

Testing

Assumption

Theory &

Review

Scorecards a a a

Interest Rate a a a a a Forecasting

Models a a a a

Valuations a a a a

Marketing a a a

Operational Risk

Models a a a a

Economic Capital

Models a a a

Roll Rate Models a a a

Pricing Models a a a a a a

Liquidity Models a a a a

AML Thresholds a a a



Model Audit Approach—3rd Line Review

14

Process Test of Design Test of

Effectiveness Sample Size (#)

Organization structure/roles & responsibilities N/A N/A

Risk reports submitted to the Board and/or

Senior Management

Development standards & documentation

Identification

Risk rating

Usage

Performance metrics & thresholds

Issues closure / extension

Change management

Adjustments / overrides

Annual review

Validation


CONFIDENTIAL: An Equal Opportunity Employer M/F/D/V. This document is for your company's internal use only and may not be copied nor distributed to another third party. 15

Questions?

16