Embed Size (px)
Citation preview
White Paper Model Risk Management in Financial Services
1
Model Risk Management in Financial Services
WHITE PAPER
White Paper Model Risk Management in Financial Services
2
WHITE PAPER
Model Risk Management in Financial Services
Introduction
Financial markets have evolved dramatically in the last decade.
Technological innovations, such as cheaper and more powerful
storage and computing capabilities, and more powerful and
lower cost network connectivity, enable the markets to move
much faster and become more connected globally. Competitive
market forces push financial services organizations to adopt
increasingly sophisticated quantitative models to cope with the
faster and more globally connected markets, and to gain an
edge over competition. Today, an overwhelming majority of key
financial decisions in financial services organizations are made
with the assistance of these quantitative models. Models are
used in a broad range of activities, including estimating customer
response; reviewing and pricing loan applications; forecasting
economic events (such as defaults, claims, etc.); assessing the
value of collateral; determining the value of instruments and/
or positions; measuring capital and reserve adequacy; directing
investigations regarding fraud and financial crimes.
The rapid proliferation of these often highly complex models
and the heavy reliance financial institutions have on them,
plus the uncertainty surrounding global economic conditions,
posed significant risks to safety and soundness of the entire
US economic system during the last economic downturn. Not
surprisingly, Model Risk Management (MRM) has been an area
of intense focus by regulators and economic planners.
The failures of financial services that many deemed “too big
to fail” highlighted the substantial inadequacies in model
risk management, not just in these failed institutions, but in
the entire financial services industry. The realization of these
inadequacies in the aftermath of these corporate failures brought
significant regulatory changes.
The capital adequacy requirements of Comprehensive Capital
Analysis and Review (CCAR), Dodd-Frank Act stress testing
(DFAST), and Basel, to name a few, exist to reduce systemic
risks to the overall financial system. These programs require
regulated organizations to have adequate risk management,
including model risk management processes.
OCC bulletin 2000-16 and more recently OCC Bulletin 2011-12
specifically address model risk management. These regulations
guide regulated organizations to adopt a robust Model Risk
Management framework.
ChallengeFinancial services organizations are facing increasing regulatory scrutiny with regard to how they measure and manage model risk.
At StakeNon-compliance could have severe regulatory consequences, including damage to company’s reputation and potentially higher capital requirements
Solution Leveraging the experience of experts well versed in Model Risk Management and applying best practices will mitigate these risks
White Paper Model Risk Management in Financial Services
3
Robust Model Risk Management Framework
A robust model risk management framework should be comprehensive and incorporate the entire process from development and
deployment to the use of quantitative models. The model risk experts at CenturyLink have found the following areas to be most
important in managing model risk.
Clear Definitions of Model and Model RiskOCC Bulletin 2011-12 defines a model as “a quantitative method, system, or approach that applies statistical, economic, financial, or
mathematical theories, techniques, and assumptions to process input data into quantitative estimates”. The published definition requires
the institution to establish explicit criteria for what is or is not a “model”, with the understanding that if the chosen definition is too narrow,
regulators will demand the definition be expanded and may issue a Matters Requiring Attention (MRA )requiring the firm to do so.
From there, the next challenge is to catalog the analytic objects within the firm and apply this definition.
Consider two examples:
a) A multi-tab spreadsheet that forecasts customer service call
volume week by week as a function of the number of new
and existing customers, product upgrades and seasonal
factors is clearly a model. The forecast is an estimate with
uncertainty based on a set of inputs, and there are many
theoretical ways to link these inputs with the outputs.
b) The process to determine whether a deposit at the bank
branch requires a Currency Transaction Report (CTR) to be filed
is most likely not a model. The requirements for a CTR have
been explicitly defined by the government, and any calculations
done are aggregations of observed data. There is no inherent
uncertainty here; if the criteria are met, the report is filed.
Unfortunately, there are countless examples that do not cleanly
fit into one bucket or the other, and those cases will thoroughly
test the robustness of the firm’s definition of a model.
In parallel to developing a definition and building an inventory, firms
must also declare the kinds of risks they intend to identify, manage
and/or mitigate as “model risk”. Academic research on “model risk”
tends to focus more on the pure quantitative uncertainty associated
with the choice of the model (model specification error) or the
estimation of a model from data (model estimation error). Practically
speaking, model risk as defined by the regulatory guidance is far
more sweeping. One helpful factor to consider is the materiality
of the model. Being able to qualify the impact of a model (and a
potential model failure) based on how broadly the model is used
should drive how deeply to investigate each of the possible types of
risk under which the model is subject.
• Clear Definitions of Model and Model Risk
• Well Defined Accountabilities
• Formal Model Governance Policies and Procedures
• Thorough Narrative Documentation
Components of a Robust Model Risk Management Framework:• Principle of Effective Challenge
• Regular & Ongoing Review
• Stress Testing
White Paper Model Risk Management in Financial Services
4
Well Defined AccountabilitiesModel Risk Management requires clear accountabilities on two
fronts. First, each model must have an “owner”, an accountable
party that is responsible for all aspects of the risk associated with
the use of the model in the business. The model owner must have
sufficient business accountability and authority to manage these risks
accordingly. The second area of accountability is the management of
the Model Risk Management policies and procedures. Typically this
is done through the establishment of a Model Risk Management
function within the broader Risk Management organization. This
organization must have sufficient authority to publish and enforce
requirements and escalate issues to the appropriate executive or
Board of Directors committee as needed.
Formal Model Governance Policies and ProceduresAll models must be governed by formalized policies and
procedures that specify the scope of, define terms relevant
to, and clearly identify the roles and responsibilities for,
model risk management. These policies and procedures must
be consistent with strong risk management principles and
supervisory expectations, and must be documented and validated.
Organizational idiosyncrasies creating unique risk exposures
should be incorporated in these policies and procedures.
Thorough Narrative DocumentationWhile documentation requirements are just one of the specific areas addressed by Model Risk Management policies and procedures,
we believe they warrant special attention. The success of the Model Risk Management process is predicated on accurate capture and
representation of not only the steps taken to develop the model, but the thought process behind those steps. Simply knowing the
ingredients and the instructions to create the model fails to address why the model was designed and constructed in the manner chosen,
what alternatives if any were considered, and what concerns the model developers were aware of when they developed the model.
Experience suggests the success of a Model Risk Management program depends highly on the willingness and ability of model
developers to put on paper the “why” and not just the “what”, for example:
• The data chosen to estimate the model (portfolio, time
period, dependent and independent variables, etc.)
• The method(s) ultimately used to construct the model
as well as alternatives considered and any theoretical or
practical considerations which led to the method(s) used.
• The criteria under which candidate models are compared
• Key assumptions associated with the development and/or
use of the model
The documentation must tell the story behind the decisions made, and the model developer plays a pivotal role in that storytelling.
CenturyLink has successfully helped multiple clients improve their documentation by partnering with model developers to ask and
answer these questions and to put that information into a readable narrative (not a PowerPoint deck).
White Paper Model Risk Management in Financial Services
5
Principle of Effective ChallengeEffective challenge requires that all models are thoroughly evaluated
by competent subject matter experts that are able to provide an
independent assessment of the key assumptions, strengths, risk
and limitations of the model. Effective challenge requires that the
reviewer be incented for the transparency and thoroughness of the
review. Typically that would dictate that the reviewer not be within
the “chain of command” of the Model Owner. There are multiple
ways this independence can be ensured; in some cases, hiring of
external consultants to perform the effective challenge is beneficial
to ensure this principle is upheld.
Regular/Ongoing ReviewModel risk management activities should follow a regular and published schedule. This includes model monitoring and regular reviews
of models post deployment. Many model related issues are only discovered through such activities.
Stress testing
OCC 2012-14 provided guidance on key principles of a stress testing framework:
• Capturing a sufficiently broad scope of enterprise exposures
• Incorporating those exposures in a flexible and forward-
looking manner
• Leveraging multiple conceptually sound approaches,
• Deriving actionable results that can inform decision making
• Applying strong governance and effective control.
• Executing stress testing at an enterprise/portfolio level.
These more comprehensive regulatory requirements combine the features mentioned above with strict implementation timelines, testing
the resource capacities of the firms subject to these regulations. Firms often have a backlog of models to review and validate prior to Stress
Testing submission deadlines; these models cover a wide cross-section of business lines. In addition, the execution of stress testing on the
various portfolios, according to internal and regulatory requirements, is now an integral part of Model Risk Management. Completing this
work prior to the submission deadlines may require additional resources well versed in Model Risk Management.
White Paper Model Risk Management in Financial Services
6
We have conducted independent reviews/validations of hundreds of models for a wide array of major financial organizations, including
six Systemically Important Financial Institutions (SIFI) firms. In conducting a validation, our data scientists follow our proven methodology
of analysis, testing and reporting based on the specific individual needs of the client. Key components of our methodology include:
1. Discovery Interview: Interview select model developers, data
analysts, model risk officers, business owners, and others
experts to gain an understanding of business objectives,
model development processes, how models are used, and
any monitoring that has already occurred, along with known
gaps and issues. Our team will also request and review any
model documentation and other relevant artifacts.
2. Model Conceptual Design Review: Assess conceptual
soundness of model design and methodology in meeting
business objectives; assess consistency with industry practice;
assess consistency with internal model development policies
and procedures; explore alternative approaches to modeling;
document model assumptions and limitations.
3. Data Quality and Integrity Testing: Evaluate data lineage and
its appropriateness; evaluate data processing including data
extraction and data merging; assess data quality through
data checking for completeness, accuracy, and invalid or
missing data; execute data process code to reconcile results
against data used for modeling.
4. Data Processing Testing: Review processing of data inputs
in modeling process, including explanatory variables
transformations; review/replicate target variable definition
and creation; assess key choices in modeling process such
as variable reduction, model selection, initialization and
iteration techniques, and termination conditions; review model
development documentations for accuracy and completeness.
5. Model Development Review & Testing: Execute model
development code to confirm development outputs;
reconcile estimations against documentations; assess
model coefficient estimations against expectations;
verify model consistency with theoretical requirements;
benchmark against internal and external models; benchmark
modeling choices against alternatives and assess impact.
6. Outcome Analysis/Back testing: Review current
performance tracking effort; review monitoring process and
reports, including measures of stability for both the model
inputs and outputs, as well as the validity of the predictions
in explaining the metric the model seeks to predict;
outcomes analysis against in-sample data and out-sample
data; evaluate model fitting to measure actual vs. predicted
performance; benchmark against internal and external
model performance; evaluate the impact of the business
environment, model usage, and policy changes through an
assortment of tests; examine rank ordering.
7. Model Implementation Testing: Verify implementation
fidelity to model development specifications; assess
implementation environment and controls; develop ongoing
testing procedures and control process; review exception
handling in model implementation; review performance
of out-of-time sample data, including outcome analysis
for out-of-time sample data; refresh model if necessary;
for forecasting, evaluate mathematical calculations and
expert managerial judgment process including blending and
adjustments that factor in contemporaneous and forward
looking views to arrive at final forecasts.
8. Stress testing: Review stress testing results on varying
plausible internal and regulatory economic scenarios if it
is required; provide suggestions on additional stress test
scenarios and approaches; if desired, suggest reverse stress
testing strategies to further stress test the model.
9. Validation Report: Deliver a written report describing the
steps taken in the validation, where effective challenge
was applied, and any issues identified. Code and output of
hands-on work is also turned over to client to be kept in its
model repository.
CenturyLink has successfully applied this methodology with hundreds of clients across the financial industry, and has validated
models used in CCAR, DFAST and Basel, including non-traditional models built by third parties and those using judgmental inputs.
CenturyLink offers subject matter experts with strong hands-on industry experience and the flexibility of a blended onshore or offshore staffing model. Our Global IT Services & Solutions practice is structured to
comprehensively meet client needs by delivering:
• High-level model risk management process reviews
• Detailed hands-on programming
• Implementation services
• Validation expertise
CenturyLink Expertise in Model Risk Management
White Paper Model Risk Management in Financial Services
7
Benefits of Model Risk Management
For many firms, the perceived benefit of a successful Model Risk Management program goes well beyond satisfying regulatory
requirements. Our perspective is that the success of Model Risk Management requires a value-added perspective. When partnering
with clients on Model Risk engagements, we strive to make every engagement more than an exercise in compliance with policies
and procedures. Below are some examples that our data scientists have witnessed with some of our client engagements.
• In an overwhelming majority of independent review
exercises, we have found the documented intent did not
match the code that was executed. Whether the discrepancy
was substantial or not, the exercise of clarifying the intent
has led to several reconsiderations and re-designs of models.
• Exercises like benchmarking along with the principle of
effective challenge promote a culture of accountability
of thought (know your assumptions) and continual
improvement. Model developers can start a new model
with a set of recommendations on paper regarding potential
improvements to the predecessor model.
• Spending time as an independent reviewer of a colleague’s
work often leads to more thoughtful model development
choices in one’s own model development work.
• Evaluation of quantitative methods previously not
considered models formalized the decision process
associated with establishing what is now considered a
model. Statistical tools like sampling and formal hypothesis
testing were then applied and helped shape how these
newly identified models were to be updated.
• Regular ongoing and formal monitoring of metrics identified
data shifts that may not have been noticed or addressed for
a significant period of time.
In Conclusion
The largest U.S. banking institutions are addressing the risk of
quantitative model usage through the creation of a Model Risk
Management program, and some firms are much further along
in the journey compared with like organizations. While there may
be some unique challenges a firm may face, our experience has
shown that the key issues are known and able to be overcome
with the right level of effort and expertise. CenturyLink’s team
of data scientists comprise demonstrated understanding and
experience of Model Risk Management to address the issues
your firm may be facing today.
In addition to Model Risk Management, CenturyLink’s Data
Analytics practice provides solutions to clients in the areas of Big
Data technology solutions, Data Management, Visualization and
Advanced Predictive Analytics across a wide array of industries.
For more information on our Big Data and Advanced Analytics
capabilities request a free consultation with our experts today.
References“Supervisory Guidance on Model Risk Management”, OCC
Bulletin 2011-12. http://www.occ.gov/news-issuances/
bulletins/2011/bulletin-2011-12.html
“Interagency Stress Testing Guidance”, OCC Bulletin 2012-24.
http://www.occ.treas.gov/news-issuances/bulletins/2012/
bulletin-2012-14.html
©2016 CenturyLink. All Rights Reserved. The CenturyLink mark, pathways logo and certain CenturyLink product names are the property of CenturyLink. All other marks are the property of their respective owners. Services not available everywhere. Business customers only. CenturyLink may change or cancel services or substitute similar services at its sole discretion without notice.782022216 - model-risk-management-financial-services-whitepaper-WP160010
Global Headquarters Monroe, LA (800) 784-2105
EMEA Headquarters United Kingdom +44 (0)118 322 6000
Asia Pacific Headquarters Singapore +65 6768 8098
Canada Headquarters Toronto, ON 1-877-387-3764
About CenturyLink Business
CenturyLink, Inc. is the third largest telecommunications
company in the United States. Headquartered in Monroe, LA,
CenturyLink is an S&P 500 company and is included among the
Fortune 500 list of America’s largest corporations. CenturyLink
Business delivers innovative private and public networking and
managed services for global businesses on virtual, dedicated
and colocation platforms. It is a global leader in data and voice
networks, cloud infrastructure and hosted IT solutions for
enterprise business customers.
For more information visit www.centurylink.com/enterprise.
About the AuthorsKeith Schleicher is Managing Director and Head of Decision
Sciences at CenturyLink and leads the Banking & Financial
Services Analytics Practice within CenturyLink. Keith brings
20+ years of predictive analytics and credit risk management
experience, and 10+ years of experience in Model Risk
Management. Keith earned his M.S. in Statistics from Ohio
State. He is an active member of the American Statistical
Association, and has presented at multiple conferences on the
use of statistical tools in credit risk monitoring. Keith can be
reached at [email protected].
Qing Sun is Senior Manager, Decision Science and has spent
18+ years in various roles in structured finance and predictive
analytics. Prior to joining CenturyLink, he spent 11 years at
Fannie Mae as a mortgage credit trader and a portfolio analyst.
In addition, Qing has served as an AVP for First Union Securities
in its Structured Transactions and Analytic Research Group and
Senior Analyst for Ocwen Financial Corp. Qing earned his MBA
in Finance from University of Illinois at Urbana-Champaign and
has been a CFA Charter Holder since 2001. He can be reached at
