28
Supported by: Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value Executive Summary Internal Model Industry Forum

IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Embed Size (px)

Citation preview

Page 1: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Supported by:

Internal Model Industry Forum:

The validation cycle: developing sustainable confidence and valueExecutive Summary

Internal ModelIndustry Forum

Page 2: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Foreword 1

Cycle overview 2

Stage a Potential triggers 6

Stage b Trigger impact assessment 10

Stage c IMV scope, planning and prioritisation 12

Stage d Execution 14

Stage e Assessment, conclusion and reporting 16

Stage f Lessons learned and improvement 18

Stage g Communication and action 20

Stages h, i and j Governance, operations and people 22

Project team and supporters 24

IMIF and IRM 25

Contents

©2015 The Institute of Risk Management.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the express permission of the copyright owner. Permission will generally be granted for use of the material from this document on condition that the source is clearly credited as being the Institute of Risk Management.

IRM does not necessarily endorse the views expressed or products described by individual authors within this document.

Page 3: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value 1

Under the Solvency II regulatory framework, the firm’s internal capital model is at the heart of risk and capital evaluation. It will therefore be a key input to a wide range of business and strategic decisions. The internal model validation

(IMV) process is critical for an internal model to be considered credible by business decision makers. While the technical aspects of the model are relatively well developed, industry players, regulators and model users agree that there is some way to go in order to create real value with internal models. In particular the key areas of challenge are:

• How to ensure the right level of engagement and understanding from the board and executives on the internal model and its limitations

• How to ensure a cost effective and value-adding internal model validation process beyond complying with regulation

• What capabilities and flexibility firms will need to support uses for internal models that go beyond solvency calculations

• How to address areas of vulnerability in models, such as the modelling of diversification benefits and operational risks

This is the first document to be published by the Internal Model Industry Forum. It provides a summary of our guidance, developed by expert practitioners, about the internal model validation cycle and what IMV looks like post Solvency II implementation.

This booklet will form part of a series offering guidance on different aspects of internal risk models, and will be accompanied by further resources and case studies available online.

I would like to thank the members of the workstream, led by Rob Merry from HSBC with support from PwC, for their work researching and developing the approach in this booklet. The members of our IMIF Steering Committee also provided valuable input and quality control. We are also grateful to representatives from the Bank of England (PRA) who have enabled us to maintain a continuous and positive dialogue between industry and the regulator on the work of the IMIF.

I would also like to thank our sponsors PwC, Milliman, Deloitte and KPMG. Also, thanks are due to the Institute and Faculty of Actuaries and to ORIC International for their input to this project. As a not-for-profit organisation, IRM relies on enlightened industry support to help us publish documents like this. It is this kind of support that helps help us maximise our investment in the development and delivery of world class risk management education and professional development.

Jose Morago, IRM Chairman and Founder of the Internal Model Industry Forum (IMIF)

Foreword

Page 4: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value2

Introduction Insurance has become increasingly reliant on sophisticated models against a backdrop of a challenging and changing financial environment. Making sure models achieve and maintain the confidence of the board and regulators has never been more important or demanding. Validation is the process which provides much of this confidence and assurance. It must be a resilient and sustainable process which can be relied on.

Moreover, Solvency II is a game changer; defining new standards in demonstrating validation and that capital models remain fit for purpose. These new standards are not exclusively for Internal Model firms as they are equally applicable to Standard Formula firms. We anticipate that boards and regulators will expect the same principles to be applied proportionately to all models which provide critical insight into the business or are used to make business decisions.

Validation is in addition to what the model owners and developers do to satisfy themselves that the model is fit for purpose. It has a vital role to play in independently answering three key questions: is it the right model for the job? is there evidence that the model is not right? and does it do what the users want or need it to do? The Internal Model Industry Forum at IRM has pulled together a wide range of senior industry professionals to assist firms as they transition into this new stage of model management.

A good plan needs to deliver both the top down questions that the board needs answered, along with the bottom up validation (i.e. does it work?). Achieving this goal is not solely down to IMV planning or process: top down challenge needs to be provided by different stakeholders of the model that the board can rely upon.

Validation CycleThis first booklet in a series sets out a practical definition, stage by stage, of the validation cycle, along with “best practice” principles. This will help firms to decide the right activity to keep their models fit-for-purpose. In particular, it provides Solvency II IMAP firms with a structure that can be used to check all the necessary elements are present in the on-going management of their Solvency II Internal Model under the new regulations. Additional booklets in the series will deal with specific aspects of the cycle in more depth.

Cycle Overview

Page 5: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value 3

Stages of the CycleThe validation cycle is the process that ensures a firm’s model remains ‘fit for purpose’. The cycle is separate and independent from the model and its developers/owners. This requirement is particularly important for Solvency II Internal Model firms but is equally applicable to all the firm’s material models.

Starting at the far left hand side, we see the internal model as fundamental to the cycle and, importantly from a validation perspective, the usages of the model. Elements of the validation cycle can vary significantly depending on what the model is used for. Each stage is explored further in subsequent sections.

VALIDATION CYC

LEV

ALID

ATION CYCLEVALIDATION

CYC

LEV

ALI

DATIO

N CYCLE

c Scope, planning and prioritisation

b Trigger impact assessment against model risk appetite

f Lessons learned & reporting

i Operations j People

h Governanced Execution

e Assessment, conclusion & reporting

a Potential triggers for Internal Model Validation

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

g Communication of, and actions on, validation findings

Page 6: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value4

The cycle has a number of elements that inter connect. By appropriately following these elements, key stakeholders will be able to make risk based decisions, using the model’s output, with confidence. This booklet is designed to assist in the engagement of key stakeholders, such as board members and executive management, in this emerging area.

Many of the elements of the validation cycle are familiar to firms. However, best practice now requires firms to demonstrate, with evidence, that the cycle and associated processes are being actively and effectively carried out.

This articulation of the validation cycle has been ‘road-tested’ with a wide range of firms from both the life and general insurance sectors to ensure that it is useful. In addition, we have surveyed a range of firms (both in terms of size and sector) to get their view of how prepared they feel they are for each stage of the cycle in connection with their Solvency II model. The results reflect their own view and these are likely to alter as firms adjust to the requirements of the Solvency II regime as it develops over the coming years. For simplicity, the results have been summarised into a simple RAG status ranging from Low (less than 35% mature) to High (more than 75% mature).

The survey results allow firms to gauge where they are in developing their validation cycle relative to their peers.

The new challengeSolvency II, and the resulting best practice that is emerging for model validation, demand that key stakeholders understand model limitations and the suitability of the expert judgements applied. In addition, they need to put processes in place to ensure the model continues to be suitable in the future. The big challenge relates to dealing with triggers; those events that signal either the internal or external environment has changed and hence model validation may be required. Having an appropriate response is vital for ensuring the right validation outcome. Without it, even the best validation work is likely to become stale as the environment the model operates in develops and changes.

An IMIF survey conducted last year indicated that less than a third of firms thought the validation cycle for internal Solvency II models would achieve its potential. This booklet indicates how this position can be rectified.

Page 7: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value 5

TriggersTriggers can be as simple as a material development of the model or as complex as the emergence of Ebola. Both raise the question – is the model still fit-for-purpose? In isolation, the trigger does not answer this question. However, when combined with the model risk impact assessment, these two elements of the cycle can indicate what validation work is required and keeps key stakeholders updated on the potential risks of using the model until the validation work is completed.

Not all triggers will necessarily lead to validation of the model. But firms will need to understand where the boundaries are for having confidence in their model, so that they can see if the triggers, through the model risk assessment process, take them outside this space. These boundaries will depend on the firm’s own risk appetite and model usage.

Homing in on the right triggers and deciding when validation is and, more importantly, is not required is arguably the most challenging part of the validation cycle. This will be explored in the next section with further examples.

Page 8: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value6

What is it?The cycle’s first stage is focussed on determining when validation should occur and informing what exactly should be validated – creating an intelligent process which recognises what is changing in the model’s environment.

Specifically, this could cover changes:

• in the model itself;

• in the company;

• in the overall company risk assessment and risk profile;

• in the industry; and

• in the wider world generally.

Examples of potential triggers are listed on the following page.

Without this, any formulaic and periodic validation will quickly become irrelevant. Worse still, it will remain an expensive and needless cost providing false comfort.

*Industry self-assessment – see page 4.

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Low

Industry view on maturity level*

Stage a: Potential triggers for Internal Model Validation

Page 9: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value 7

Triggers for validation: key principlesOptimal: Triggers should give firms effective coverage of both internal and external factors impacting the model.

Timely: The triggers should be timely so validation can take place within the business timeframe for model use.

Tailored: The triggers should be forward looking and tailored to model usage.

Accurate: Monitoring data should be robust and reliable.

Sensitive: Triggers should be sensitive to model limitations.

Manageable: The numbers of triggers should be manageable to achieve the principles above.

“The cycle starts by looking at triggers for validation. This involves assessing exactly what is changing – and considering what impact the changes have on validation requirements.”

Page 10: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

8 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Stage a: Potential triggers for Internal Model Validation (cont)

Example potential Model Validation Triggers Trigger Category

Understand what changes have been made to the model Internal drivers

Follow-up on actions from last year’s report Findings from previous cycle

Initial or subsequent regular reviews as required by model validation policy

Minimum IMV requirements

Changes in business strategy, including delays to implementing strategies

Internal drivers

Material changes in own funds, MCR, SCR, tech provisions and/or other balance sheet items

Risk profile change

Market crashes External drivers

Regulator requests additional ad hoc reporting / numbers Minimum IMV requirements

Breaches of other risk appetites that may indicate insufficient understanding of risk

Internal drivers

Change to internal risk profile e.g. new product launches or changes to investment strategies

Risk profile change

Change in governance arrangements (IMV) Indirect impact

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Low

Industry view on maturity level

Page 11: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

9Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Triggers need to cover a broad spectrum, so firms remain confident in their model. The identification of triggers must be timely and reflect the firm’s general appetite for risk.

Of course, triggers could be as simple as model changes. The assessment should therefore incorporate the model change process. However triggers need to go further than just understanding or validating model change. For instance, a change in business mix may not require model change but can significantly move risk profile and so therefore what validation work is performed.

Benefits • Model stakeholders preserve their understanding and confidence in the model.

• A continuous feedback loop between model users (e.g. proposition development, risk) and model developers allows users to understand the impact of decisions made.

• A better and more current understanding of the risk profile and environment in which the model is operating.

“Principles for the triggers are quite straightforward – they need to capture all changes and developments that could impact the application and use of the model.”

Page 12: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

10 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

What is it?The “trigger impact assessment” against model risk appetite stage is the “pre planning” and scope identification of the model validation process.

Model risk appetite focuses on filtering and assessing potential trigger events and deciding if they should translate into validation activities. Triggers are filtered against the purposes and “error appetite” in the model to ensure that IMV is only carried out when required.

Model risk appetite can be defined differently depending on the model use and acceptable limitations. The assessment can be both quantitative or qualitative. The outcome from this assessment is a conclusion of what validation, or validation work, is necessary.

Some triggers are planned - i.e. a rotating policy of when different areas will be validated. However, as internal model validation becomes less cyclical over time, we increasingly expect most validation to be dynamic rather than cyclic. This will improve both the efficiency and effectiveness of the process.

To bring this alive, there is a step-by-step worked example of a trigger and associated risk assessment on the IRM website at www.theirm.org/knowledge-and-resources/thought-leadership/ creating-value-through-internal-models/validation-resources/

Stage b: Trigger impact assessment

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Low

Industry view on maturity level

Page 13: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

11Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Trigger impact assessment: key principlesRelevant: Firms should have a formal mechanism for assessing potential triggers for validation against their internal model risk.

Appropriate: Model risk impact assessment should vary for different uses of the model ensuring models remain fit for purpose. Not all potential triggers result in additional validation.

Proportionate: Assessment should consider proportionality – not all potential triggers result in additional validation.

Continuous: should be regularly assessed as some changes develop slowly over time.

Planning: The impact of triggers on validation scope should be understood and reflected in the validation plan.

Benefits • Validation is carried out only when needed and at a cost effective level.

• Improved understanding on what the model can and cannot do (its limitations).

• Trigger assessments enable a firm to gain an initial understanding of the magnitude of the impact and whether it is material or not in a time efficient manner.

• Stops firms using the model for purposes for which it was not designed – thus reducing risk of poor decision making.

Page 14: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

12 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

What is it?The assessment of triggers needs to feed into a clear plan of actual validation activities. The plan should encompass planned depth, tools, timescales as well as the process by which the validation team manages the validation activity.

Having considered the required level of model confidence, firms need a framework for BAU validation and contingency plans in the event of unexpected triggers.

A good plan needs to deliver both the top down questions the Board need answered along with the bottom up validation (i.e. does it work?).

Plan prioritisationDefining the validation scope and plan enables execution in the most efficient and timely manner, especially for urgent unexpected validation activity. This way validation is also proportionate and appropriate and so maintains confidence in the model in a range of circumstances.

Understanding the business uses of the model will help prioritise activity so things are done in the right order and at the right time. Key issues are the model change process, unpredictability and level of challenge of the status quo. Prioritisation ensures a healthy ongoing exercise.

Stage c: Scope, planning and prioritisation

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Medium

Industry view on maturity level

Page 15: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

13Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Scope, planning and prioritisation: key principlesPlan: Firms should develop and maintain an internal model validation scope and plan.

Appropriate: Plans need to reflect materiality, proportionality, business need and the outputs from the trigger assessment. It should include both “top down” and “bottom up” validation.

Timely: Plans need to ensure actions are carried out at the right time.

Contingency: Validation should include planned variability and unpredictability.

Usage: Validation scope, plan and prioritisation should be consistent with the model risk impact assessment.

Challenge: The scope and plan should articulate how objective challenge of the model will be achieved.

Plan developmentNot all validation activity is equal and not all validation activity is required every time.

The right skills and level of effort needed to be identified for different validation activities. The plan should consider: 1) the purpose of the validation, 2) the use of the model, 3) frequency of the review, and 4) how quickly results are required. Timing of key uses (e.g. 1/1 renewals, M&A window, ALM rebalance, etc.) is important.

Benefits• A nimble and efficient internal model validation process that does what is required,

as quickly and cheaply as possible.

• It also ensures that validation supports model uses rather than hinders it.

• The results of the validation along with the plan demonstrate to external stakeholders that robust internal governance and control processes exist.

Page 16: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

14 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

What is it?Validation execution is where validation is actually carried out. It is the process of proving or challenging that the model is fit for purpose by looking at all aspects of the model – from evidencing the suitability of the data, assumptions and model results as well as ensuring model documentation and governance is appropriate.

Market feedback suggests this part of the cycle is relatively well developed. However, this is unsurprising given that every firm would have needed to execute validation even if there is no clear cycle in place. There is a need to continually improve the process over time to ensure it remains efficient while meeting the likely increasing demands of the business.

This is also an aspect of the cycle that is going to be further covered by other IMIF workstreams in a later publication.

Regulators and external validators have seen a wide range of validation in terms of quality. Therefore, there is a need to continually improve the process over time to ensure it remains efficient whilst meeting the likely increasing demands of the business and regulator.

Stage d: ExecutionU

ses

of t

he in

tern

al m

odel

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

High

Industry view on maturity level

Page 17: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

15Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Execution: key principlesIndependence: Validation should be carried out with an independence of mind and encourage free thinking.

Correct skills: Individuals with the right skills, experience and knowledge are crucial.

Improvement: Firms should expect the validation process to become more efficient over time.

Evidence: Validation testing should be documented and evidenced to a re-performance standard.

Balance: The split between 1st line and 2nd line validation will vary from company to company.

Benefits• Validation provides the board with comfort that the model results are appropriate

for use and consistent with established market practice.

• Validation should become more cost effective over time as the process improves and the business gets better at targeting validation activity.

Page 18: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

16 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Low

Industry view on maturity level

What is it?Model validation is complex and so the output needs careful and focused interpretation.

Overall, conclusions need to factor in the materiality of the validation tests and findings to ensure the right conclusions are reached. This is complex as validation is far from black or white.

Validation for the various uses of the model will demand different standards of validation (e.g. validation for SCR vs. reinsurance strategy). Gold-plating needs to be avoided to make validation cost effective.

The limitations and inherent uncertainties of the model, by usage, should clearly emerge when good validation is carried out. This could involve a specific assessment of model risk as part of the firm’s ORSA or risk framework.

These limitations need to be well communicated so that the model is only used where it is appropriate to do so, and by stakeholders who fully understand the implications. Feedback from NEDs surveyed by IMIF has indicated that firms have more work to do in this area.

By focusing on the stakeholders’ needs, the reporting answers the right questions that the business and regulators need answering at the right time.

Top down questions address the requirements of the board and the regulators’ expectations to understand the key drivers and the limitations of the model, while clear, detailed and technical questions help the model owners and developers.

Stage e: Assessment, conclusion and reporting

Page 19: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

17Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Assessment, conclusion and reporting: key principlesClear Reports: Firms need reports that clearly communicate conclusions to stakeholders.

Clear Conclusions: Reports need clear conclusions and an overall fit for purpose rating.

Recommendations: Reports should include recommendations to reduce model weaknesses.

Risk Framework: Validation findings should explicitly link to the firm’s risk management framework (including the ORSA) so the risks of using the model are acknowledged.

Independence: Report findings should be written independently of model owners influence.

Limitations: Reports should clearly state the limitations of both the validation and the model.

Benefits• Provides independent quality assurance over the model and comfort that the model is

fit for purpose.

• Feeds into model development and better, more informed business decisions.

• Assures external stakeholders that robust internal governance and control processes exist.

Page 20: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

18 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Medium

Industry view on maturity level

Stage f: Lessons learned and improvement

What is it?The purpose of the lessons learned and improvement stage is to identify improvements to the validation cycle, processes and execution to ensure that they are continuously evolving.

This is distinct from model improvements and findings arising from validation.

However, the requirements of independent model validation and the importance of expert judgment also mean an element of bespoke / periodic independent validation is also always going to be vital.

This process is also about ensuring that validation improves; for instance by deploying alternative validation methods.

Ultimately, a firm needs to know if it is getting appropriate return on its significant validation investment.

The critical considerations are:

• that the cycle is efficient and effective;

• that the cycle is embedded and is a core component of the on-going running, development and improvement of the model; and

• that the validation activity is fit for purpose and continually meets stakeholders’ needs.

Page 21: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

19Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Lessons learned: key principlesImprovement: Firms need a regular process where lessons learned are captured and improvements implemented so validation remains proportionate and cost effective and improves over time.

Efficiency: Validation must become quicker and more embedded so that it becomes an integral part of ensuring the model runs well rather than being a one off activity.

Embedding: Sufficient on-going validation in the running and development of the model will provide firms with cost effective assurance through the business as usual activities.

Fit for purpose: Firms should continually assess the validation cycle to ensure that it is fit for their business.

Benefits• A changing and developing validation approach is a powerful evidence of a healthy

model rather than a constantly changing model.

• Models need to keep abreast of environmental changes and validation ensures these changes are appropriate.

• Embedding validation in the business as usual cycle gives stakeholders timely assurance when it is needed.

• Facilitates wider business understanding of the model.

Page 22: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

20 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

Medium

Industry view on maturity level

Stage g: Communication of, and actions on, validation findings

What is it?Communication to different stakeholders must be appropriate, in terms of content and level of detail, for their specific needs.

Successful communication of validation findings should mean that they are eagerly received, absorbed and considered by model owners, management and the board. It should also lead to a better understanding of the risks faced and to what extent they are modelled. This provides an opportunity to further educate key stakeholders.

Findings should not fall to just one person or department but should involve appropriate people across the organisation at different levels.

It is essential that findings should be communicated in the context of the scope and purpose of the validation. It is usually helpful for this to include the assessment of model risk and uncertainty (particularly around expert judgement), to underline the importance of the validation effort. Developing a communications strategy should be part of the validation scoping and planning stage. Being clear as to what is communicated to whom, how, and in what detail at each iteration of the cycle is crucial.

Page 23: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

21Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Communication and action: key principlesClarity: Firms should have a clear communication strategy and plan for the communication of their validation results (how, when and to whom).

Demand: There should be a regular cycle of reporting of validation results (push) as well as an on-going firm appetite to request the results of validation (pull).

Buy-in: Appropriate senior input should be provided at various stages of the validation results communication.

Management actions: Firms should have a clear process for responding to and remediating findings raised by validation activity.

Benefits• Clear and effective communication of validation findings enables continued efficient

improvement in the model and further education of key stakeholders: expanding the firms understanding and confidence of the model and its use; and evidences the validation process for regulatory purposes.

“Successful communication should result in a useful debate around the appropriate actions and how best to achieve them based on the findings.”

Page 24: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

22 Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Use

s of

the

inte

rnal

mod

el

INTERNALMODEL

a Potential triggers for Internal Model Validation

g Communication of, and actions on, validation findings

c Scope,

planning and prioritisation

b Trigger impact

assessment against model risk appetite

f Lessons

learned & reporting

i Operations

j People

h Governance

d Execution

e Assessment,

conclusion & reporting

High

Industry view on maturity level

Stages h: Governance, i: Operations and j: People

What is it?This part of the cycle is about how the firm operates the validation cycle in practice. It includes the systems, processes and controls that the firm puts in place to enable the efficient, timely and effective completion of the cycle.

This is really about individual firms adopting the same governance, control environment and rigour over the IMV cycle as it would for any other part of its business.

‘Operationalising’ the validation cycleIt would be expected that the following would be common:

• Robust planning – a plan is in place that ensures that the IMV process is clearly mapped and that timescales are achievable .

• Risk management – the risks to the delivery of the validation cycle are understood and appropriate mitigation is in place.

• Control environment – an appropriate control environment is in place, based on the firm’s own standards and practices. This will include appropriate controls and sign-off of the validation activity.

• Documentation – complete documentation is in place.

• Line-of-sight – A clear line-of-sight exists between the signed-off validation approach and the final output which can be used to prove that all activities have been adequately performed.

Page 25: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

23Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value

Governance, operations and people: key principlesNo undue influence: IMV governance allows the validators to remain independent and free from undue influence.

Skills: they have access to the required skills, knowledge and experience for the IMV cycle.

Succession: there is an appropriate talent management process for IMV.

Governance: there is an appropriate level of governance throughout the IMV cycle.

Quality: an appropriate operational system is developed to support the IMV cycle, allowing validation activity to be delivered in a cost effective and efficient manner.

Benefits• Ensures on-going challenge, allowing understanding and knowledge to be shared.

• Broader understanding of an internal model and a firm’s risk management system.

• Minimises the opportunity for a material failure or oversight in the performance of the validation cycle.

Page 26: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value24

Our project teamWe would like to thank the following people and their employers for their work on this document:

Workstream leader:

Rob Merry, Global Head of Independent Model Review, Insurance and Pensions, HSBC

Consultancy support:

Bill Gasson, Director, Insurance Consulting, PwC

Barney Wanstall, Director, Insurance Assurance, PwC

Workstream members:

Parth Patel, Chief Actuary and Head of Enterprise Risk Management, Ascot Underwriting

Andrew Daniels, Head of Reporting and Analysis, Standard Life

Kerry- Anne Saggers, Senior Manager, Insurance & Pensions, Independent Model Review, HSBC

Nirav Morjaria, Head of Economic Capital and Model Risk Governance, HSBC

Sabine Pommer, Senior Associate, Insurance Consulting, PwC

Our steering committeeThe IMIF steering committee comprises:José Morago, Aviva (Chair)

Tamsin Abbey, Deloitte

Raphael Borrel, Vitality

Simon Cartlidge, L&G

Alastair Clarkson, Standard Life

Robert Collinson, Towers Watson

Caroline Coombe, ORIC International

Karun Deep, RSA

Sebastien Delfaud, Bank of England (PRA)

Vishal Desai, Bank of England (PRA)

Steven Graham, Institute and Faculty of Actuaries

David Innes, RSA

Roger Jackson, KPMG

Tony Jeffery, Bank of England (PRA)

Rob Merry, HSBC

David Paul, E&Y

Niraj Shah, Ageas

Michael Sicsic, Aviva and ORIC International

Justin Skinner, QBE

Barney Wanstall, PwC

Russell Ward, Milliman

Carolyn Williams, IRM

Page 27: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

Internal Model Industry Forum: The validation cycle: developing sustainable confidence and value 25

The Internal Model Industry Forum This document has been produced by the Internal Model Industry Forum (IMIF). The Institute of Risk Management (IRM) set up the IMIF in 2015 to address the key questions and challenges that insurers face in the use, understanding and validation of internal risk models. It is designed to work in a collaborative way to develop and share good practice to ensure that these models add value to the organisation and support regulatory compliance. IMIF now has over 300 members and we have run a series of Forum meetings to explore key issues. A number of workstreams are also undertaking research and we aim to publish the results along with other useful resources and guidance.

As the leading organisation promoting education and professional development in all aspects of risk management, IRM is pleased to be able to support this industry initiative to share good practice.

More information about the IMIF and its work can be found on the IRM website www.theirm.org

Who are the IRM?This work has been supported by members of IRM, which has provided leadership and guidance to the emerging risk management profession for nearly 30 years. Through its training, qualifications and thought leadership work, which includes seminars, special interest and regional groups, IRM combines sound academic work with the practical experience of its members working across diverse organisations worldwide. IRM would like to thank everyone involved in the IMIF project.

Page 28: IRM IMIF Validation Cycle v6€¦ · regulators and model users agree that there is some way to go in order to create real value with internal ... about the internal model validation

IRM T: +44(0) 20 7709 9808 E: [email protected] www.theirm.org

Institute of Risk Management 2nd Floor, Sackville House 143-149 Fenchurch Street London EC3M 6BN United Kingdom

Russell Ward [email protected]/

Roger Jackson [email protected]/

Tamsin [email protected]/uk/en.html

Barney Wanstall [email protected]

Bill Gasson [email protected] www.pwc.co.uk/

Our supportersAs a not-for-profit organisation, IRM is reliant on industry support to publish guidance like this. We would like particularly to thank the following organisations who have made this publication possible:

PricewaterhouseCoopers LLP