Embed Size (px)
Citation preview
‹#›
Discussion on ModelValidationAugust 20, 2015
‹#›
Model Risk Management and Validation
Robin SawyerAlan Hodgson
‹#›
Contact:Robin,[email protected]
404.575.8905
191 Peachtree Street, NE
Suite 2700
Atlanta, GA 30303
EXPERIENCERobin has been with DHG since 2003 serving financial services companies in advisory and externalaudit capacities. Prior to joining DHG, Robin served as a manager with a Big Four accounting firm,serving primarily financial services companies in the areas of external audit and advisory servicesand spent part of his career in industry where he served as the U.S. controller for a large,international investment management company headquartered in Sydney, Australia. With more than30 years of total business experience in both industry and public accounting roles, Robin brings aunique perspective and depth of experience in serving his clients. He works primarily with publiccompanies in the areas of Model Risk Management and Model Validation services, Mergers andAcquisitions accounting consulting, and external audit. Robin strives to provide his clients with ahighly practical approach to problem solving that not only helps meet the requirements of externalauditors and regulators but is also operationally expedient.
Robin Sawyer| CPAPartner, DHG Financial Services
Meet Your Instructor …
‹#›
Contact:[email protected] 1 West 4th StreetSuite 700Winston-Salem, NC 27101
INDUSTRY EXPERIENCEAlan has eight years of experience in model validation and valuation services. Alan focuses onengagements in the financial services industry and specializes in model validation for bothinternally and externally developed models. Alan is a firm resource for financial modeling and hasexperience validating statistical, quantitative and vendor models with specific experience in liquidityand capital stress testing.
Experience Includes:• Provided guidance on documentation standards and regulatory expectations as a part of the
annual stress testing process including effective challenge of model theory, model design,accuracy and appropriateness of model calculations and process documentation
• Assisted financial institutions in the review and enhancement of model documentation and modelrisk management policies and procedures
• Performed gap analysis to assess compliance with regulatory requirements and industry bestpractices
• Utilized SAS Enterprise Guide to perform statistical analysis on equations through regressionprocedures to test for heteroskedasticity, normality, autocorrelation, multicollinearity, stationarity,etc.
• Benchmarked internally developed and external (vendor) models to comparable industrybenchmarks
Alan Hodgson| CFASenior Manager, DHG Valuation Services
Meet Your Instructor …
‹#›
Agenda
TopicIntroductionOverview of Model Risk ManagementModel vs. End User ToolModeling ExpectationsModel ValidationInternal Audit ViewQuestions
‹#›
Introduction
‹#›
Overview of Model Risk Management
‹#›
Mod
el R
isk
Man
agem
ent i
n Fo
cus
“The
mos
t im
port
ant q
uest
ion
abou
t any
fina
ncia
l mod
el is
how
wro
ng it
islik
ely
to b
e, a
nd h
ow u
sefu
l it i
s des
pite
the
assu
mpt
ions
. You
mus
t sta
rt w
ithm
odel
s and
then
ove
rlay
them
with
com
mon
sens
e an
d ex
peri
ence
.” - P
aul W
ilmot
t, Fi
nanc
ial M
odel
ers’
Man
ifest
o (J
anua
ry 2
009)
“Mod
els,
by th
eir n
atur
e, a
re v
ast s
impl
ifica
tions
of a
com
plex
eco
nom
icre
ality
”… “
The
Fede
ral R
eser
ve B
oard
’s h
ighl
y so
phis
ticat
ed fo
reca
stin
g sy
stem
did
not f
ores
ee a
rece
ssio
n un
til th
e cr
isis
hit.
” - A
lan
Gre
ensp
an, T
he M
ap a
nd T
he T
erri
tory
2.0
(201
3)
‹#›
Com
mon
Mod
el R
isks
Con
cept
ual R
iskTh
e ri
sk th
at th
e m
odel
ing
conc
epts
are
not
suita
ble
for t
he p
urpo
se o
f the
app
licat
ion.
Impl
emen
tatio
nR
isk
Ther
e ar
e tw
o ty
pes o
f im
plem
enta
tion
risk:
•
The
risk
that
the
wro
ng a
lgor
ithm
s wer
e ch
osen
to im
plem
ent t
he sp
ecifi
ed m
odel
ing
conc
epts
•Th
e ris
k th
at a
ppro
pria
te a
lgor
ithm
s wer
e ch
osen
, but
they
con
tain
cod
ing
erro
rs a
ndbu
gs.
Inpu
t Risk
The
risk
that
the
inpu
t par
amet
ers a
re in
appr
opria
te, i
ncom
plet
e, o
r ina
ccur
ate
Out
put R
iskTh
e ris
k th
at th
e ke
y fig
ures
and
stat
istic
s tha
t can
be
prod
uced
by
the
mod
el d
o no
t sup
port
the
busi
ness
pur
pose
or a
re to
o se
nsiti
ve w
ith re
spec
t to
the
prov
ided
inpu
t par
amet
ers.
Rep
ortin
g R
iskTh
e ris
k th
at th
e re
pres
enta
tion
of th
e ou
tput
for t
he b
usin
ess u
sers
is in
com
plet
e or
mis
lead
ing.
‹#›
Mod
el R
isk
Man
agem
ent
•B
e co
gniz
ant o
f unf
avor
able
con
sequ
ence
s of
deci
sion
s bas
ed o
n m
odel
s tha
t are
inco
rrec
t or
mis
used
•Pr
ovid
ing
“effe
ctiv
e ch
alle
nge”
by
staf
f with
appr
opria
te in
cent
ives
, com
pete
nce,
and
influ
ence
•C
reat
ing
a so
und
mod
el fr
amew
ork
is c
ritic
al to
asu
cces
sful
risk
man
agem
ent s
trate
gy
‹#›
Gui
danc
e on
Mod
el R
isk
Man
agem
ent
•So
und
prac
tices
for m
odel
risk
man
agem
ent
‹#›
Gui
danc
e on
Mod
el R
isk
Man
agem
ent
Supe
rviso
ry G
uida
nce
on M
odel
Risk
Man
agem
ent –
SR
Lette
r 11-
7 an
d M
odel
Valid
atio
n Pr
inci
ples
App
lied
to R
isk a
nd C
apita
l Mod
els i
n th
e In
sura
nce
Indu
stry
•Pr
ovid
ed in
201
1 an
d 20
12 b
ut h
as re
ceiv
ed in
crea
sed
focu
s with
in th
e pa
st fe
wye
ars
•In
clud
es g
uida
nce
for a
ll as
pect
s of m
odel
risk
man
agem
ent
–M
odel
dev
elop
men
t, im
plem
enta
tion
and
use
–M
odel
val
idat
ion
–G
over
nanc
e–
Mod
el ri
sk m
anag
emen
t pol
icy
–M
odel
inve
ntor
y•
Cus
tom
izat
ion
is e
xpec
ted
‹#›
Mod
el F
ram
ewor
k
‹#›
Mod
el R
isk
Man
agem
ent P
olic
y
Mod
el ri
sk m
anag
emen
t act
iviti
es sh
ould
be
form
aliz
ed th
roug
h po
licie
s and
proc
edur
es to
ens
ure
good
gov
erna
nce
prac
tices
. Th
e m
odel
risk
man
agem
ent p
olic
y se
ts th
e pr
otoc
ol fo
r mod
el o
wne
rs, u
sers
and
valid
ator
s to
ensu
re a
lignm
ent w
ith su
perv
isor
y gu
idan
ce a
nd e
xpec
tatio
n.
‹#›
Mod
el In
vent
ory
Gen
eral
Gui
delin
es
•Fi
rm-w
ide
inve
ntor
y•
Mod
el st
atus
•M
odel
var
iatio
n•
Des
crip
tion
shou
ld in
clud
e–
Mod
el n
ame
and
owne
r–
Purp
ose
–Pr
oduc
ts–
Use
restr
ictio
ns–
Type
and
sour
ce o
f inp
uts
–O
utpu
ts an
d in
tend
ed u
se–
Last
upda
te a
nd c
urre
nt st
atus
–Ex
cept
ions
to p
olic
y–
Valid
atio
n sta
tus a
nd d
ates
–Ti
mef
ram
e fo
r rev
iew
and
ong
oing
mon
itorin
g
‹#›
Mod
el v
s. E
nd U
ser
Tool
‹#›
Wha
t is
a m
odel
?
“The
term
mod
el re
fers
to a
qua
ntita
tive
met
hod,
syst
em, o
r app
roac
h th
at a
pplie
sst
atis
tical
, eco
nom
ic, fi
nanc
ial,
or m
athe
mat
ical
theo
ries
, tec
hniq
ues,
and
assu
mpt
ions
to p
roce
ss in
put d
ata
into
qua
ntita
tive
estim
ates
” - S
R 1
1-7
Supe
rvis
ory
Gui
danc
e
Mod
el C
hara
cter
istic
s
•M
odel
s are
sim
plifi
ed re
pres
enta
tions
of r
eal w
orld
rela
tions
hips
with
diffe
rent
cha
ract
eris
tics,
valu
es a
nd e
vent
s•
Lim
itatio
ns i
n m
odel
s cre
ate
the
need
for q
ualit
ativ
e ap
proa
ches
inad
ditio
n to
qua
litat
ive
mod
elin
g•
The
use
of m
odel
s has
led
to m
odel
risk
, whi
ch is
the
resu
lt of
adv
erse
cond
ition
s due
to in
corr
ect d
ecis
ions
mad
e ba
sed
on m
odel
ed o
utpu
t
‹#›
End
Use
r To
ols
(EU
T)
End
Use
r To
ol C
hara
cter
istic
s
•To
ol ti
ed to
a d
eskt
op, t
ool,
or p
rodu
ct, m
ade
up o
f log
ic/c
ompo
nent
s•
Dev
elop
ed a
nd m
anag
ed b
y th
e en
d us
er o
r a th
ird p
arty
•Sp
read
shee
t/dat
abas
e fo
rm•
Util
ize
calc
ulat
ions
, mac
ros,
scrip
ts o
r cod
ing
•C
ompl
ex sp
read
shee
t with
exc
el a
dd-o
ns•
Stan
dard
EU
T po
licy
for t
ool u
se a
nd p
roto
cols
‹#›
Mod
el v
s. E
UT
Dec
isio
n P
roce
ss
Dev
elop
gui
delin
es fo
rdi
ffere
ntia
tion
of to
ol v
s.m
odel
for
orga
niza
tiona
lpu
rpos
es
Fore
cast
ing
wor
kstr
eam
prov
ides
rel
evan
tin
form
atio
n to
Risk
Man
agem
ent a
nd k
eyst
akeh
olde
rs
Col
labo
ratio
n be
twee
nR
isk M
anag
emen
t and
key
stak
ehol
ders
tode
term
ine
whe
ther
tool
isa
mod
el o
r EU
T
Dev
elop
EU
T/M
odel
Polic
y an
d so
cial
ize
with
stak
ehol
ders
‹#›
EU
T P
olic
ies
EUT’
s diff
er in
pol
icy
requ
irem
ents
in c
ompa
rison
to m
odel
s. Th
ere
is cu
rrent
ly n
o cl
ear
guid
ance
rela
ted
to st
anda
rds f
or E
UT
polic
y de
velo
pmen
t.
Exec
utiv
e Su
mm
ary
EUT
Gov
erna
nce
EUT
Cont
rols
EUT
Met
hodo
logy
Inpu
t and
Out
put D
ata
Cont
rols
Doc
umen
tatio
n Re
quire
men
ts
App
endi
x A
– E
UT
Inve
ntor
y
App
endi
x B
– EU
T Se
lf A
sses
smen
t
Illus
trat
ive
EUT
Polic
y Ta
ble
of C
onte
nts
K
ey C
onsid
erat
ions
•
Polic
ies s
houl
d be
revi
ewed
and
upd
ated
on
anan
nual
bas
is
•Po
licy
shou
ld c
onta
in th
e fo
llow
ing:
‒Ro
les a
nd re
spon
sibili
ties
‒A
ccou
ntab
ilitie
s‒
Cont
rols
‒Pr
otoc
ols
•
Evid
ence
that
staf
f and
Sen
ior M
anag
emen
t are
adhe
ring
to th
e po
licy
(Exa
mpl
e: E
UT
self
asse
ssm
ent)
•
Form
al p
roce
dure
for p
olic
y ex
cept
ions
‹#›
Mod
elin
g E
xpec
tati
ons
‹#›
Gen
eral
Mod
elin
g C
onsi
dera
tions
Foun
datio
nal
Elem
ents
•R
epea
tabl
e an
d tra
nspa
rent
mod
elin
g pr
oces
s•
Supp
orte
d by
em
piric
al e
vide
nce
•A
ble
to p
rodu
ce c
redi
ble
estim
ates
alig
ned
to sc
enar
io•
Mat
eria
lity
of a
giv
en p
ortfo
lio o
r act
ivity
•Se
gmen
t mod
elin
g ba
sed
on ri
sk c
hara
cter
istic
s (no
t nec
essa
rily
bylin
e of
bus
ines
s or p
rodu
ct ty
pe)
•Q
ualit
ativ
e an
d qu
antit
ativ
e pr
ojec
tions
are
exp
ecte
d
Dat
a•
Inte
rnal
dat
a w
here
ava
ilabl
e, e
xter
nal d
ata
as a
ppro
pria
te•
Gra
nula
r dat
a to
mod
el ri
sk c
hara
cter
istic
s
‹#›
Con
serv
atis
m a
nd C
redi
bilit
y
Mod
elin
g ac
tiviti
es a
nd th
e as
sum
ptio
ns th
at d
rive
resu
lts n
eed
to b
e ap
prop
riate
lyco
nser
vativ
e du
e to
the
inhe
rent
unc
erta
inty
of p
rosp
ectiv
e m
odel
s.
‹#›
Mod
el D
ocum
enta
tion
Mod
el U
ser’s
Con
tribu
tion
to M
odel
Risk
Man
agem
ent•
Cont
ribut
e to
mod
el v
alid
atio
n ac
tiviti
es b
y pr
ovid
ing
com
preh
ensiv
ein
form
atio
n on
the
mod
el•
Allo
w fo
r the
iden
tifica
tion
of m
odel
risk
•Co
ntrib
ute
to b
usin
ess c
ontin
uity
thro
ugh
mem
oria
lizat
ion
of m
odel
attri
bute
s•
Prov
ides
stak
ehol
ders
info
rmat
ion
of th
e lim
itatio
ns a
nd w
eakn
esse
sof
mod
els
•M
akes
com
plia
nce
with
pol
icy
trans
pare
nt
Key
Doc
umen
tatio
nEl
emen
ts
•D
emog
raph
ic in
form
atio
n•
Exec
utiv
e su
mm
ary
•M
odel
ing
data
•M
odel
ing
appr
oach
•M
odel
ing
assu
mpt
ions
•M
odel
lim
itatio
ns•
Mod
el e
stim
atio
n / d
evel
opm
ent
•Im
plem
enta
tion
testi
ng•
Tech
nica
l spe
cific
atio
ns•
Use
r gui
de•
Ope
ratio
nal c
ontro
ls•
Mod
el ri
sk m
onito
ring
•Ch
ange
man
agem
ent
‹#›
Mod
el V
alid
atio
n
‹#›
Del
iver
s ass
umpt
ions
and
dat
a to
the
mod
el
Mai
n M
odel
Com
pone
nts
Inpu
t
Proc
essi
ng Rep
ortin
g
Tran
sfor
ms i
nput
s int
o es
timat
es
Tran
slat
es th
e es
timat
es fr
om th
e pr
oces
sing
com
pone
nt in
to u
sefu
l inf
orm
atio
n
‹#›
Mod
el V
alid
atio
n
‹#›
Com
preh
ensi
ve M
odel
Val
idat
ion
An
effe
ctiv
e va
lidat
ion
fram
ewor
k sh
ould
incl
ude
thre
e co
re e
lem
ents
‹#›
Mod
el V
alid
atio
n: C
once
ptua
l Sou
ndne
ss
“The
val
idat
ion
proc
ess s
houl
d en
sure
that
qua
litat
ive,
judg
men
tal a
sses
smen
ts a
reco
nduc
ted
in a
n ap
prop
riat
e an
d sy
stem
atic
man
ner,
are
wel
l sup
port
ed, a
nd a
redo
cum
ente
d” - S
R 1
1-7
Supe
rvis
ory
Gui
danc
e
Mod
el V
alid
atio
n A
ctiv
ities
•
Qua
lity
of d
esig
n an
d co
nstru
ctio
n•
Alte
rnat
ive
theo
ries
•Se
nsiti
vity
ana
lysi
s•
Doc
umen
tatio
n•
Evid
ence
supp
ortin
g m
etho
ds u
sed
and
varia
bles
sele
cted
•R
evie
w o
f mod
el e
nhan
cem
ents
or c
hang
es si
nce
last
val
idat
ion
‹#›
Mod
el V
alid
atio
n: O
ngoi
ng M
onito
ring
“Mon
itori
ng c
onfir
ms t
hat t
he m
odel
is a
ppro
pria
tely
impl
emen
ted
and
is b
eing
use
dan
d is
per
form
ing
as in
tend
ed”
- SR
11-
7 Su
perv
isor
y G
uida
nce
Mod
el V
alid
atio
n A
ctiv
ities
•
Mon
itorin
g sh
ould
con
tinue
ove
r tim
e w
ith a
freq
uenc
y ap
prop
riate
toth
e na
ture
of t
he m
odel
•A
ppro
pria
tely
impl
emen
ted
and
verifi
catio
n ch
ecks
to e
nsur
e al
lco
mpo
nent
s are
func
tioni
ng a
s des
igne
d•
Cha
nges
may
requ
ire a
djus
tmen
t, re
deve
lopm
ent o
r rep
lace
men
t•
Ben
chm
arki
ng a
nd q
ualit
y co
ntro
l thr
esho
lds
‹#›
Mod
el V
alid
atio
n: O
utco
mes
Ana
lysi
s
“Com
pari
sons
hel
p to
eva
luat
e m
odel
per
form
ance
by
esta
blis
hing
exp
ecte
d ra
nges
for
thos
e ac
tual
out
com
es in
rela
tion
to th
e in
tend
ed o
bjec
tives
and
ass
essi
ng th
e re
ason
sfo
r obs
erve
d va
riat
ion
betw
een
the
two”
- SR
11-
7 Su
perv
isor
y G
uida
nce
Mod
el V
alid
atio
n A
ctiv
ities
•
Com
parin
g ou
tput
s to
actu
al o
utco
mes
with
qua
lity
cont
rol t
hres
hold
s in
plac
e
•Q
uant
itativ
e an
d qu
alita
tive
test
ing
com
men
sura
te w
ith:
oM
odel
des
ign
oC
ompl
exity
oAv
aila
ble
data
oM
agni
tude
of p
oten
tial r
isk
•Pa
ralle
l out
com
es a
naly
sis
•B
ack
test
ing
‹#›
The
Mod
el V
alid
atio
n P
roce
ss
Mod
el R
isk Id
entifi
catio
nD
esig
n Va
lidat
ion
Test
PlanP
erfo
rm V
alid
atio
n A
ctiv
ities
Doc
umen
t and
Rem
edia
teIs
sues
Issu
e Re
port
and
Gui
danc
e
•M
odel
ow
ner w
alk
thro
ugh
of m
odel
deve
lopm
ent,
impl
emen
tatio
n an
d us
e•
Revi
ew m
odel
docu
men
tatio
n•
Iden
tify
and
docu
men
tm
odel
risk
s for
eval
uatio
n-
Conc
eptu
al ri
sk-
Impl
emen
tatio
n ris
k-
Inpu
t risk
-O
utpu
t risk
-Re
porti
ng ri
sk
•Co
nstru
ct te
sting
plan
for m
odel
bas
edon
iden
tified
mod
elris
ks•
Doc
umen
t the
test
proc
edur
es fo
r eac
hm
odel
risk
•A
ssig
n tim
elin
es a
ndow
ners
hip
for t
estin
gac
tiviti
es
•Pe
rform
val
idat
ion
proc
edur
es o
utlin
edin
the
testi
ng p
lan
•A
djus
t tes
t pla
n an
dad
d ad
ditio
nal
proc
edur
es a
s nee
ded
base
d on
test
resu
lts
•O
ngoi
ng d
ialo
gue
betw
een
valid
atio
nte
am a
nd m
odel
owne
r to
disc
uss
obse
rvat
ions
•Re
spon
ses t
oob
serv
atio
ns fr
omm
odel
ow
ner
•D
ocum
ent i
ssue
sid
entifi
ed a
nd d
evise
rem
edia
tion
actio
ns•
Ass
ign
acco
unta
bilit
ies
•D
raft
valid
atio
nre
port
and
obta
insig
n-of
f fro
m m
odel
owne
r and
risk
man
agem
ent
•Pr
ovid
e gu
idan
ce a
ndre
com
men
datio
ns to
mod
el o
wne
r for
mod
el e
nhan
cem
ent
Ong
oing
Com
mun
icat
ion
‹#›
Inte
rnal
Aud
it V
iew
‹#›
Inte
rnal
Aud
it’s
Rol
e in
Mod
elin
g / M
odel
Ris
k
Inte
rnal
Aud
it’s
Rol
e is
NO
T to
dupl
icat
e m
odel
risk
man
agem
ent
activ
ities
Inte
rnal
Aud
it’s
Rol
e is
toA
sses
s the
ove
rall
effe
ctiv
enes
s of t
he m
odel
risk
man
agem
ent f
ram
ewor
k
Eval
uate
whe
ther
mod
el ri
sk m
anag
emen
t is c
ompr
ehen
sive
,rig
orou
s and
effe
ctiv
e
Verif
y th
at p
olic
ies a
nd p
roce
dure
s are
in p
lace
, com
ply
with
guid
ance
, and
are
bei
ng fo
llow
ed A
sses
s com
plet
enes
s and
acc
urac
y of
the
mod
el in
vent
ory
and
the
proc
ess f
or d
evel
opin
g an
d m
aint
aini
ng th
e in
vent
ory
Rev
iew
the
proc
ess u
sed
to v
alid
ate
mod
els,
repo
rt an
dre
med
iate
find
ings
, and
the
proc
ess f
or k
eepi
ngdo
cum
enta
tion
curr
ent
Und
erst
and
mod
elin
g ex
pect
atio
ns to
iden
tify
whe
ther
effe
ctiv
e ch
alle
nge
of a
ssum
ptio
ns a
nd re
sults
is ta
king
pla
ce
‹#›
For
Mor
e In
form
atio
n …
Ala
n H
odgs
onW
inst
on-S
alem
, NC
336.
714.
1620
alan
.hod
gson
@dh
gllp
.com
Rob
in S
awye
rA
tlant
a, G
A40
4.57
5.89
05ro
bin.
saw
yer@
dhgl
lp.c
om
‹#›
Questions
