Embed Size (px)
DESCRIPTION
Citation preview
© 2014 Experian Information Solutions, Inc. All rights reserved. Experian and the marks used herein are service marks or registered trademarks of Experian Information Solutions, Inc.
Other product and company names mentioned herein are the trademarks of their respective owners. No part of this copyrighted work may be reproduced, modified, or distributed in
any form or manner without the prior written permission of Experian. Experian Public.
Regulatory requirements for model risk governance continue to evolve Is your organization prepared to meet these requirements?
Linda Haran Experian
Robert Stone Experian
#vision2014
2 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Regulatory environment and Model Risk Governance ServicesSM
Model stress testing
Model risk governance best practices
Agenda
3 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
The regulatory
environment
and Model Risk
Governance ServicesSM
Robert Stone
4 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Bank compliance: Model Risk Governance U.S. standards – OCC Bulletin 2011-12
Bulletin 2000-16 2000
Bulletin 2011-12 2011
The Supervisory Guidance on Model Risk Management OCC Bulletin 2011–12 extends the scope beyond standard model validation to policies, practices, standards for:
Benchmarking
Model development
Model use and implementation
Model governance and controls
All national banks and federal savings associations are directly impacted
Other financial services institutions are indirectly impacted and should consider as industry best practice
5 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model Risk Governance Regulatory requirements
Policies
All financial institutions that use models in their decisioning processes must ensure that their internal policies and procedures are consistent with the guidance
What is expected of institutions
for compliance?
Assessment of model risk
Banks need to mitigate the potential risks arising from the reliance on models that may be improperly validated or tested
Managing model risk
Models risk can be
considerably
reduced through
rigorous model
validation
procedures
Validation requirements
The OCC requires that financial institutions perform ongoing model validations at least once a year, include an independent review, and produce proper model documentation
6 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
The way forward
Financial regulators expect greater use of models by financial entities:
► “The experience and judgment of developers, as much as their technical knowledge, greatly influence the appropriate selection of inputs and processing components. The training and experience of developers exercising such judgment affects the extent of model risk.”
Therefore, regulatory examination and supervision regimes will increasing their oversight and review of the institution’s management of its models in three stages:
1. Through examination of model development, implementation, and use consistent with the 2011 guidance
2. Through examination of a sound model validation process
3. Through examination of a governance framework with defined roles and responsibilities for clear communication of the model’s limitations and assumptions as well as the authority to restrict the model’s usage by the institution
Model Risk Governance The regulatory horizon
7 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model stress
testing analysis
Linda Haran
8 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model risk governance
Model validation Back-testing
Benchmarking
Sensitivity analysis
Stress-testing
Analytical approach
applies to both
generic and custom built credit risk models
Stress-testing
9 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Unconventional times for model governance
OCC Bulletin 2011–12 defines stress testing of a model to:
► “Evaluate model performance over a wide range of parameter input values, including extreme values, that are correlated with macro economic factors”
The model stress testing analysis will determine how the model performs in extreme economic environments
Stress testing Requirements and model risk assessment
10 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Capturing the
impact of a
recession to a risk model
through its natural cycle
Economic cycling Stressing the model through a downturn
Stressing model performance on the overall
population and by risk segments of prime and non-prime
Increased utilization, delinquency behaviors
Late stage recession
Inquires and new trades increasing
Early stage recession
11 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Does a model tend to lose its ability to rank order when stressed under economically difficult conditions?
What could cause the most risk and instability of your models in a highly stressed scenario?
Stress testing Analysis hypothesis
12 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Create a series of economic attributes in the context of time
► Unemployment
► Consumer Price Index
► Gross Domestic Product
Determine what types of consumer credit characteristics correlate to changing economic conditions over time
Identify those highly correlated attributes that are included in the algorithm of an Experian bureau risk model
Determine stress values for each attributes identified
Assess model performance under stressed conditions
Stress testing Defining the input parameters
13 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
0
1,000
2,000
3,000
4,000
5,000
90
0-9
00
86
0-8
79
82
0-8
39
78
0-7
99
74
0-7
59
70
0-7
19
66
0-6
79
62
0-6
39
58
0-5
99
54
0-5
59
50
0-5
19
46
0-4
79
42
0-4
39
38
1-3
99
30
0-3
70
Normalized population distribution
Stressed inquiry attribute distribution
0
500
1,000
1,500
2,000
90
0-9
00
86
0-8
79
82
0-8
39
78
0-7
99
74
0-7
59
70
0-7
19
66
0-6
79
62
0-6
39
58
0-5
99
54
0-5
59
50
0-5
19
46
0-4
79
42
0-4
39
38
1-3
99
30
0-3
70
Normalized population distribution
Stressed inquiry attribute distribution
0
1,000
2,000
3,000
4,000
5,000
90
0-9
00
88
0-8
99
86
0-8
79
84
0-8
59
82
0-8
39
80
0-8
19
78
0-7
99
76
0-7
79
74
0-7
59
72
0-7
39
70
0-7
19
68
0-6
99
66
0-6
79
64
0-6
59
62
0-6
39
60
0-6
19
58
0-5
99
56
0-5
79
54
0-5
59
52
0-5
39
50
0-5
19
48
0-4
99
46
0-4
79
44
0-4
59
42
0-4
39
40
0-4
19
38
1-3
99
36
9-3
76
30
0-3
00
Normalized population distribution Stressed inquiry attribute distribution
Stressing the model Increased inquiry behaviors
Prime Subprime
14 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
0
1,000
2,000
3,000
4,000
5,000
90
0-9
00
86
0-8
79
82
0-8
39
78
0-7
99
74
0-7
59
70
0-7
19
66
0-6
79
62
0-6
39
58
0-5
99
54
0-5
59
50
0-5
19
46
0-4
79
42
0-4
39
38
1-3
99
30
0-3
70
Normalized population distribution
Stressed utilization attribute distribution
0
500
1,000
1,500
2,000
90
0-9
00
86
0-8
79
82
0-8
39
78
0-7
99
74
0-7
59
70
0-7
19
66
0-6
79
62
0-6
39
58
0-5
99
54
0-5
59
50
0-5
19
46
0-4
79
42
0-4
39
38
1-3
99
30
0-3
70
Normalized population distribution
Stressed utilization attribute distribution
0
1,000
2,000
3,000
4,000
5,000
90
0-9
00
88
0-8
99
86
0-8
79
84
0-8
59
82
0-8
39
80
0-8
19
78
0-7
99
76
0-7
79
74
0-7
59
72
0-7
39
70
0-7
19
68
0-6
99
66
0-6
79
64
0-6
59
62
0-6
39
60
0-6
19
58
0-5
99
56
0-5
79
54
0-5
59
52
0-5
39
50
0-5
19
48
0-4
99
46
0-4
79
44
0-4
59
42
0-4
39
40
0-4
19
38
1-3
99
36
9-3
76
30
0-3
00
Normalized population distribution Stressed utilization attribute distribution
Stressing the model Increased utilization behaviors
Prime Subprime
15 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
0
1,000
2,000
3,000
4,000
5,000
6,000
90
0-9
00
86
0-8
79
82
0-8
39
78
0-7
99
74
0-7
59
70
0-7
19
66
0-6
79
62
0-6
39
58
0-5
99
54
0-5
59
50
0-5
19
46
0-4
79
42
0-4
39
38
1-3
99
30
0-3
70
Normalized population distribution
Stressed delinquency attribute distribution
0
500
1,000
1,500
2,000
900-9
00
860-8
79
820-8
39
780-7
99
740-7
59
700-7
19
660-6
79
620-6
39
580-5
99
540-5
59
500-5
19
460-4
79
420-4
39
381-3
99
300-3
70
Normalized population distribution
Stressed delinquency attribute distribution
0
1,000
2,000
3,000
4,000
5,000
6,000
90
0-9
00
88
0-8
99
86
0-8
79
84
0-8
59
82
0-8
39
80
0-8
19
78
0-7
99
76
0-7
79
74
0-7
59
72
0-7
39
70
0-7
19
68
0-6
99
66
0-6
79
64
0-6
59
62
0-6
39
60
0-6
19
58
0-5
99
56
0-5
79
54
0-5
59
52
0-5
39
50
0-5
19
48
0-4
99
46
0-4
79
44
0-4
59
42
0-4
39
40
0-4
19
38
1-3
99
36
9-3
76
30
0-3
00
Normalized population distribution Stressed delinquency attribute distribution
Stressing the model Increased delinquency behaviors
Prime Subprime
16 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
When an attribute in the model with a high coefficient is taken to its maximum valid value, the score becomes highly unstable
KS drops by 31 points
Model performance: Under what scenario would a model dramatically under perform?
0
1,000
2,000
3,000
4,000
5,000
6,000
90
0-9
00
88
0-8
99
86
0-8
79
84
0-8
59
82
0-8
39
80
0-8
19
78
0-7
99
76
0-7
79
74
0-7
59
72
0-7
39
70
0-7
19
68
0-6
99
66
0-6
79
64
0-6
59
62
0-6
39
60
0-6
19
58
0-5
99
56
0-5
79
54
0-5
59
52
0-5
39
50
0-5
19
48
0-4
99
46
0-4
79
44
0-4
59
42
0-4
39
40
0-4
19
38
1-3
99
36
9-3
76
30
0-3
07
Normalized population distribution Stressed inquiry attribute Max inquiry attribute
17 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model risk governance
best practices
Robert Stone
18 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Subjective judgment is employed in all model development
Design, theory and logic should support a clear statement of purpose
► Comparison should be made among alternatives
Measure, where possible, and understand model uncertainty
Model cycle Best practices
Development, use
and implementation
19 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model cycle Best practices
Validation
In addition to model power or accuracy, validation should identify potential limitations and assumptions and assess their impact
Vendor models should be incorporated into a bank’s broader risk management framework
Model validators should be given authority to challenge and appropriate incentives
Development, use
and implementation
20 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Ensures effective challenge, and take prompt remedial action where necessary
Broadly divided into ownership, controls and compliance
Model cycle Best practices
Development, use
and implementation Validation Governance
21 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model Risk Governance business review Case study – a Top-5 retail bank
Time Short Long
Va
lid
ati
on
vo
lum
e/s
tan
da
rdiz
ati
on
High
Low
Data
flow
Model
type
Validation
requirement
Validation
elements
Assessment
results
1 2 3
Score files exchange
Risk models
Asset models
Balance models
Line models
Pricing models
Point estimate
business use
Rank ordering
business use
Actual vs. expected characteristics
Actual vs. expected score dist.
Actual vs. expected outcome dist.
Override analysis
Scorecard performance
Concordance/shifting
Future model performance
Single factor analysis
Benchmarking analysis
Calibration/redesign issues
Control limit determination
Business use guidance
Population stability analysis
K-S analysis
22 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model Risk Governance business review Case study – a community bank in the Midwest
Business challenge
Client was seeking a model validation of its consumer credit scoring model in accordance with the OCC Guidance 2011-12 model risk management.
Solution Client leveraged Experian’s Global Consulting Practice and analytics team’s expert knowledge of best practice, data and modeling to ensure that the best model validation approach.
Quantitative model validation Standard tests of the models were conducted to validate performance and accuracy, along with benchmarking to other essential models.
Qualitative model validation Experian reviewed and gained a thorough understanding of the client’s modeling efforts, policies and procedures. Information on the client’s existing practices were compared to industry best practice in order to create the final deliverables, which included a gap analysis, strategic roadmap and actionable recommendation.
Benefit to client The client was able to leverage the strategic roadmap and recommendations to proactively move toward best practices and fulfill regulatory demands prior to further scrutiny and review.
The client leveraged Experian’s model validation team to augment existing staffing in a cost effective approach.
23 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model Risk Governance is not a new concept, but a way of thinking that needs to be an integrated part of an organization
Many organizations have invested in methods, resources, processes and technology to assess monitor, manage and model their credit risk
Changes in economic environment or misapplication of models exposes an organization
Regulators expect greater use of risk models and as a result oversight has increased to review the management of these models
Summary
24 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Model Risk Governance is not a new concept, but a way of thinking that needs to be an integrated part of an organization
Increased oversight leads to uncertainty of expectations and resource challenges
It’s not a one-time exercise, but a valuable ongoing re-calibration to better manage risk and return on investment
The investment leads to better decisions and enhanced business performance
Summary
Discover Experian’s® best-in-class
Model Risk Governance services
25 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
For additional information, please contact:
Hear the latest from Vision 2014
in the Daily Roundup:
www.experian.com/vision/blog
@ExperianVision | #vision2014
Follow us on Twitter
26 © 2014 Experian Information Solutions, Inc. All rights reserved. Experian Public.
Visit the Experian Expert Bar to learn more about
the topics and products covered in this presentation.
