Suspicious Activity
Monitoring; An Innovative and Auditable Approach to
Threshold Tuning
Scot Luther
2
Executive Summary A strong and sufficient suspicious activity monitoring system is a critical component, amongst many, in a
robust Bank Secrecy Act (“BSA”)/Anti-Money Laundering (“AML”) Compliance Program. “Suspicious
activity reporting forms the cornerstone of the BSA reporting system.”1 Suspicious activity monitoring
systems of most financial institutions are typically reviewed and audited as a part of a regularly scheduled
independent audit, which is required as one of the pillars of in a comprehensive BSA/AML Compliance
Program. For most regulated financial institutions, implementing a rigorous suspicious activity monitoring
system is no longer a Herculean feat, so long as proper planning and execution are engaged. In fact, there
are many third-party vendors that can be leveraged to provide software and consulting services to
implement such a system. However, once the implementation concludes and the system is stabilized and
operational, every organization faces the challenge of threshold tuning, and determining how and when
to do so.
To date, it has been a generally accepted practice for financial institutions to execute periodic threshold
tuning to fulfill regulatory and audit requirements. In many cases, the period of review is annually. A
financial institution may spend tens or hundreds of thousands of dollars in hard and soft costs on a
threshold tuning exercise to ensure obligations are met, only to find that the thresholds in place are still strong and sufficient. In contrast, a financial institution may execute an annual threshold tuning exercise
that confirms the existing thresholds are indeed weak and prone to risk. A subsequent lookback may also
indicate that thresholds should have been reviewed and tuned six or nine months prior and a significant number of reportable activity may have been missed. Each scenario mentioned fails to demonstrate an
effective, auditable threshold tuning methodology and both scenarios face ancillary costs and risks as a
result.
This whitepaper provides a high-level overview of a suspicious activity monitoring system and the
necessity of threshold tuning as a critical component to the typology validation. Most importantly, the
whitepaper will introduce a conceptual implementation of a software solution that provides financial institutions and suspicious activity monitoring system managers automated, data-driven predictive
indicators and alerts that a threshold should be reviewed and possibly tuned. The solution would be
integrated to the suspicious activity monitoring system in place. It would also be running in real-time to
provide alerts that could justify the obsolescence of an annual periodic threshold tuning exercise. The
predictive indicators would also include data snapshot captures that financially rationalize threshold
tuning exercises from a business perspective, all while providing an auditor concrete justification for a
threshold tuning exercise.
1 Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014, pp. 399
(R-5).
3
Compliance Culture and BSA/AML Compliance Program Management Effective management of an organization’s BSA/AML Compliance Program is a monumental task that requires championship from organizational leadership, as well as support from all employees. In
addition to managing and maintaining the health of the business, corporate leadership is also
responsible for ensuring that compliance obligations are fulfilled and that appropriate support and
resources are allocated to fulfilling such requirements. As a baseline, it is essential for organizational leadership to create and support a culture of compliance from top to bottom in the company. “For a
BSA/AML compliance program to be effective, it should have the demonstrable support of the
leadership (as appropriate based on the financial institution’s size and structure).”2 A significant component of said support and culture is managing the business congruent to a strong and sufficient BSA/AML Risk Assessment. For the purpose of this document, I will not conduct an in-depth analysis and
outline of the BSA/AML Risk Assessment. However, it is critical to note and understand that the BSA/AML
Risk Assessment is the fundamental foundation of a robust BSA/AML Compliance Program and that all roads from risk mitigation and management must lead back to the BSA/AML Risk Assessment. Most important to this document is that the suspicious activity monitoring system, as well as the proposed
tuning solution, should be derived from the BSA/AML Risk Assessment.
Suspicious Activity Monitoring System (“SAMS”) A fundamental and necessary internal control of the BSA/AML Compliance Program is an effective and
comprehensive transaction monitoring system that will identify potentially suspicious activity. “Suspicious
activity monitoring and reporting are critical internal controls. Proper monitoring and reporting processes
are essential to ensuring that the bank has an adequate and effective BSA compliance program.”3 As was
previously discussed, such potentially suspicious activity is derived from the most recent and current BSA/AML Risk Assessment. A SAMS could be as simple as manual reviews of a few defined red flags that is managed by one or two people handling alerts. In contrast, there are systems in place that are
extremely complex, inclusive of hundreds of identified red flags and vulnerabilities with multiple layers of thresholds that are managed by hundreds of alert analysts. No matter which end of the spectrum that you are on, the system is critical nonetheless and the complexity of your system should be derived from
your BSA/AML Risk Assessment. In accordance with the Federal Financial Institutions Examination Council (“FFIEC”), the “sophistication of monitoring systems should be
dictated by the bank’s risk profile”. 4
For the purpose of this document, I have diagrammed a very simplified and very high-level overview of a
basic SAMS. It must be understood that SAMS solutions in large scale implementations are extremely
complex in design and configuration. The diagram provided below is meant to simply be used to depict the location and positioning of an innovative and alternative Tuning Engine solution that could be
employed to replace traditional red flag tuning schedules. See Diagram A below.
2 FinCEN, “FIN-2014-A007”, August 2014, pp. 2. 3,4 Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014, pp. 66.
4
Diagram A
No matter how simple or complex the SAMS is, the system remains a critical component in a robust BSA/AML Compliance Program. SAMS continuously supports the global efforts to prevent money
laundering and combat terrorist financing. While fulfilling regulatory obligations, these systems can also
be leveraged to save an organization thousands of dollars in fraud losses. “Actively managing AML
monitoring data can reduce costs, save time, avoid regulatory remediation, and improve assurance.”5
When utilized to the fullest extent of its capabilities, these systems will pay for themselves in true
monetary savings, all while fulfilling regulatory requirements.
5 PricewaterhouseCoopers, “From source to surveillance: the hidden risk in AML monitoring system
optimization”, September 2010, pp. 1.
5
Below are just a few of tangible benefits of a properly implemented and managed SAMS.
Identify Suspicious Activity to be Reviewed and/or Reported – The main purpose of a suspicious
activity monitoring system is to identify suspicious activity that needs to be reviewed and
potentially reported to the appropriate financial intelligence unit (FIU). Properly implemented
systems are designed to achieve exactly this, which supports a financial institution’s efforts in
fulfilling regulatory obligations.
Fulfill Regulatory Obligations Efficiently – A well planned and executed implementation for a
SAMS will provide effective tools to an organization that will enable it to efficiently fulfill the
portion of regulatory obligations for internal controls, suspicious activity monitoring, and
suspicious activity reporting. The SAMS will automate, analyze, and alert analysts based on
millions of records of data that would be an insurmountable task for human beings to perform
manually.
Identify & Prevent Fraud Losses – Real-time and near real-time monitoring systems are able to
identify and prevent potential fraud as it is happening. Once flagged, the system can
automatically invoke suspension of the account and/or card from being able to be used until contact is made with the customer. Depending on the type of transaction processed, this
recognition and intervention can yield thousands of dollars in savings from fraud losses. In many
cases the fraud that is identified is also suspicious activity that needs to be reported to an FIU.
Supports a Process-Driven Enterprise – A SAMS systematic approach to management supports
the concept of process definition, management, and improvement. Automated systems
challenge organizations to clearly define how a process truly works. The product of such an
exercise is one or many lucid, organized process maps that outline exactly how a process flows. Once the processes are agreed upon and well understood by key stakeholders, the organization
is able to begin implementing rules and thresholds that support the management of the process
in alignment with the BSA/AML Risk Assessment.
Data Driven Visibility – The data in a SAMS is invaluable. Because the data is organized in a
normalized structure, reporting solutions can be configured to generate output reports for multiple functional department uses. In addition to the regulatory and compliance benefits, reporting can also be tailored to corporate leadership that identifies risks, costs, as well as
fluidity and changes in the business.
Although a SAMS is remarkably useful and beneficial to an organization, these systems do not come
without challenges. There are many ancillary costs, both hard and soft, that are associated to managing
these systems in large institutions.
System Management Resources – A SAMS is typically a fairly complicated system to operate
and maintain. The initial cost of the software itself is merely one component. Once the
software is implemented and in place, skilled personnel are required to manage and maintain
the systems. These costs must be accounted for in the organizational annual budget when
implementing such a system. In addition, applying personnel that are not qualified for the job
may be worse than having no personnel at all.
Data Integrity – The infamous saying “garbage in garbage out” holds true with these systems. It is paramount that properly qualified resources are engaged to manage the data that goes through
the transaction normalization process and ultimately is infused into the SAMS
6
Transaction Data Mart. Data that is not properly managed creates unexpected risks for the
business that could yield catastrophic consequences.
Obsolete or Invalid Thresholds – Resources in the business world today are stretched incredibly
thin. One of the many poor ways to utilize time and effort of people is to work on tasks that are
invalid or obsolete. If an organization lacks effective management processes around red flag
definitions, the analysts that handle alerts will likely be spending time unnecessarily on alerts that have no value. Threshold tuning is a necessary function in managing a SAMS to ensure the
“engine” is running optimally. Without effective processes and controls around the maintenance
of the system, an organization may be missing critical, reportable activity, or the alert system is
being inundated with alerts that serve no value. In either case, the lack of attention to these areas
proves to be extremely expensive.
The Problem with Traditional Threshold Tuning of a Suspicious Activity Monitoring System There have
been tremendous advancements made with suspicious activity monitoring solutions, identification, and
reporting over the past twenty to thirty years. The solutions in place today are much more robust than
five years ago, and even more so than ten to twenty years ago. However, amidst all of the improvements
made with SAMS solutions, there is still an industry-wide problem in place at most organizations involving
threshold tuning. The red flags defined in a SAMS are only as good as the static thresholds that are
currently applied to the ever-dynamic business they are serving. Unfortunately, some of the static
thresholds could be out of date for months and the institution doesn’t even know it.
“Additionally, it is imperative to execute a threshold-tuning exercise on a periodic basis that consists of generating an investigating alerts just below the threshold values. This exercise gives insight into the
existence (or lack) of suspicious activity just below the set thresholds. Existence of such activity will require the thresholds to be lowered. If there is no suspicious activity just below the threshold values, then a separate exercise consisting of lifting the threshold values above the current values can be
performed. If this exercise yields the same alerts, then there may be a case for lifting the threshold
values in production.”6
As previously mentioned, to date many organizations have tried to mitigate the risk of stale thresholds
by implementing a regularly scheduled threshold tuning exercise. In many institutions this is an annual review of all red flags and thresholds. Therein lies a critical defect; an annual review to confirm the
validity of thresholds of a near real-time or a real-time monitoring solution.
Depending on the number of red flags and thresholds involved, this could be an extremely expensive
and time-consuming exercise. Unfortunately, the only way to justify to an examiner that the SAMS
thresholds in place are effective is to complete the exercise.
Among many others, below are at least three fundamental issues with a periodic threshold tuning
exercise.
Cost – The exercise is needed for justification so it must be completed. The hard and soft costs
incurred are an assumed “cost of doing business” to fulfill the regulatory obligation.
6 Protiviti, “Implementing AML Transaction Monitoring Systems: Critical Considerations”, September 2013, pp. 5.
7
Confirmation Thresholds Are Effective – The double-edge sword result of a threshold tuning
exercise is that the business confirms that the thresholds in the SAMS are effective. This means
the organization blindly wasted money, time, effort, and realized opportunity-cost losses as a result of performing the exercise, only to reaffirm that the thresholds were effective. If the business had known prior to performing the exercise that the thresholds were effective
there would have been no business justification to complete it.
Confirmation Thresholds Are Ineffective – If a threshold tuning exercise confirms that the
thresholds are ineffective, it could be misconstrued that the exercise was justified and successful. The business would make necessary corrections to the thresholds, re-run an exercise
to confirm effectiveness, and proceed with business as usual for the next 12 months. What may
or may not be answered by the business is how long have the thresholds been ineffective? If the thresholds have been ineffective for 10 months and the last exercise occurred 12 months
ago, it is likely the organization has either missed reportable SAR activity and/or alert analysts
have been wasting their time addressing false positive alerts. In either case, the organization
was at risk for some undetermined amount of time.
Threshold tuning is a necessary process in effective model management of a SAMS. The thresholds
defined for red flags in the SAMS are static by design; in fact they must be for auditability purposes. These static thresholds allow for concrete management and measurement of activity and alerts for suspicious activity. As institutions change, grow and/or shrink, the validity of thresholds in the SAMS
will likely become out-of-date and ineffective.
In some cases an organization may not even have a regimen defined that outlines how and when a tuning
exercise is to occur. The lack of this process leaves the organization vulnerable to missing SAR worthy
activity, or generating so many false positives that the team that manages alerts becomes ineffective due
to false positive overload. “The absence of periodic tuning of scenarios often results in numerous false
positives, which in turn delay alert investigation and ultimately lead to missed reporting deadlines.”7 In
either case, this is a costly mistake that organizations are able to overcome.
An Innovative and Auditable Approach for Threshold Tuning of a Suspicious Activity Monitoring System With the obvious gaps in traditional threshold tuning schedules and methodologies today, there is a
significant opportunity to improve processes and reduce risk within your organization by leveraging
automation, software, and processing power. Simple and complex computing can be used to alert SAMS
managers that transaction history trends are changing and static thresholds in place may be obsolete. The
value in this type of solution is that the subjective guesswork of when to tune is removed and you let your data and predictive indicators make the decision for you. From a business perspective this
justification balances the questions of cost. From an audit perspective the systematic approach is more
robust because it shortens the window of missing reportable activity, it reduces the time that analysts
work on false positive noise, and the data from the solution justifies any threshold modifications that occur as a result of the exercise.
7 Protiviti, “Views on AML Transaction Monitoring Systems”, 2013, pp. 14.
8
The following information outlines a very high-level system that leverages the data you already use today, in conjunction with your existing control tables of your SAMS, to output alerts and predictive indicators
of when to review and possibly tune your thresholds. In addition to the high-level overview, a live red flag
and data example is used to framework how this is implemented today. Lastly, we cover a summarization
of benefits of the solution, including the viability of an auditor using the solution to audit an organization’s
SAMS using static datasets at certain times.
To revisit the previous diagram outlining a high-level SAMS configuration, this solution will focus on the
components highlighted in blue in Diagram B below. Note that this diagram is a zoomed in scope of a
subset of Diagram A above. The only true addition is the concept of Elasticity Data.
Diagram B
What is Elasticity in this context and why does it matter? By simple definition, elasticity is referred to as “stretchiness”. In business you might hear executives or leadership refer to “stretch thinking” or thinking beyond your traditional myopic view. In the context of this solution, we are accomplishing exactly that. The goal here is to stretch the static thresholds in place
to a finite position. By doing so, the scope of data brought into the evaluation increases and decreases, depending on the stretch. With that increase or decrease in data, you will then be able to determine if there is any additional information that can be ascertained by the stretch as it relates to the red flag in
question.
9
Tuning Engine The Tuning Engine is the software that handles the processing of elasticity data. At a very high level, this
software solution will iterate through your red flags, as well as each flag’s associated elasticity data, to
provide you variable alert volumes at each elastic interval. The alert volume at each interval will equip
you with the predictive indicators necessary to justify, or not, completing a tuning exercise on a particular red flag.
Tuning Engine Database The Tuning Engine Database (TED) is very simple in structure. The TED consists of a normalized data table
to hold your SAMS threshold data. The SAMS threshold data is used as the baseline for the Tuning Engine. A data table for elasticity data is also required. This data table will hold attributes of how much to flex a
threshold, and how elastic each step will be.
Use Transaction Data That You Already Have It is likely that your organization has already spent thousands of dollars to implement a strong and
sufficient SAMS. There is tremendous value in the data the SAMS holds, as well as the engine of the
solution that generates your alerts. Apart from what the SAMS is originally designed for, this same
conceptual processing can be applied, leveraging the data that is already used. With the addition of a
new structure of data holding elasticity information, the Tuning Engine system is able to achieve near real-time visibility into when a tuning exercise may be needed. This is powerful when justifying why a
tuning exercise is being proposed, and prevailing in an audit when you have tuned thresholds outside of a regularly scheduled threshold tuning exercise.
Use SAMS Threshold Data as Your Baseline The system requires a baseline. The best place to start is by using the thresholds of the SAMS. These
thresholds, if implemented correctly, should be the most relevant thresholds that align to each
corresponding red flag, which ties back to the organizational risk assessment. The Tuning Engine will use
the baseline thresholds as a place to start. Using that data, in addition to the elasticity values set in the
Tuning Engine database, the Tuning Engine can efficiently step through vast amounts of data to provide
you insights into alert volumes at varying threshold fluctuations.
Live Example in Operation We will now look in depth at a live working example of a simple red flag in practice titled “8 or More ATM
Withdrawals in 7 days. Note that with this red flag there are two static thresholds defined in the SAMS
database. These two static thresholds are a count of 8 ATM withdrawals within a timeframe of 7 days. The purpose of the existing thresholds set is such that the flag should catch cardholders that are
excessively using their card to withdraw cash from an ATM, at least one time per day for a period of at least one week, one of those days withdrawing twice.
Each day the SAMS conducts an evaluation of live data using the threshold data of this red flag. At the
conclusion of the evaluation each day, most of the days the evaluation yields no results. However on
some occasions there are two or three customers that generate an alert to be reviewed.
Note the very simplistic and high-level Diagram C below containing the TED data tables.
1111
Now let’s evaluate the results with elasticity from the Tuning Engine. Below is an outline of the scope of the example.
The date range of data used for this example is July 1, 2016 through July 31, 2016
“Current Date” simulation is set for July 20, 2016. This example will allow an auditor to review
the subset of data as if the day were July 20, 2016.
Given the elastic thresholds set above, there are 35 permutations evaluated, with time
threshold elastic spanning between the dates of July 9 through July 20 (5 days back through 11
days back), as well as count threshold elasticity from 6 transactions up to 10 transactions.
Diagram D Below is an image of the first records of transaction data used in the example. There are 8676 total records in the population for this example. To review all of the actual data used, see the attached Excel file titled RedFlagExampleData.xlsx >> Transaction Data worksheet.
No Elasticity Implemented As expected and as is most common, when no elasticity is implemented into the model, no results are
returned.
1212
Full Elasticity Implemented Using the example of the elastic thresholds outlined in Diagram C above, let’s evaluate the results with
full elasticity implemented. To review the results of full flexion being implemented, see the attached
Excel file titled RedFlagExampleData.xlsx >> Results-Full Elasticity worksheet.
Customer ID
Min ATM
Transaction
Count
Min Count
Elastic
Threshold
Min Time
Elastic
Threshold
Time Unit
Max ATM
Transaction
Count
Max Count
Elastic
Threshold
Max Time
Elastic
Threshold
Time Unit
94246847 6 6 -8 days back 6 6 -11 days back
96183347 6 6 -10 days back 6 6 -11 days back
106050219 6 6 -9 days back 8 8 -11 days back
106098970 6 6 -8 days back 8 8 -11 days back
106102796 7 6 -9 days back 9 9 -11 days back
111379131 6 6 -10 days back 6 6 -11 days back
113498258 6 6 -5 days back 7 7 -11 days back
114645430 6 6 -10 days back 6 6 -11 days back
118065927 6 6 -11 days back 6 6 -11 days back
124297195 6 6 -7 days back 7 7 -11 days back
126648296 6 6 -7 days back 6 6 -11 days back
140897528 6 6 -8 days back 6 6 -11 days back
141433932 6 6 -9 days back 6 6 -11 days back
145213407 6 6 -5 days back 6 6 -11 days back
So What Does This Mean? Keep in mind that the example used for the purpose of this exercise is rather simple in scope. In large
institutions the complexity of the red flag may include multiple dimensions for elasticity. In this case
there are only two dimensions, which are the Transaction Count and Time. However, there is some
interesting information that can be extrapolated from this evaluation and data.
Are the thresholds of this red flag strong and sufficient? The data suggests that they might be. An analyst will only be able to conclude this assumption definitively by reviewing specific
examples in the data.
On the upper end of the threshold elasticity for the transaction count, you will note that there is
only one example (Customer ID 106102796) in which the Maximum ATM Transaction Count exceeded the existing threshold set in the SAMS. You will note that the threshold was only
exceeded after 3 additional days back of data were evaluated.
On the lower end of the threshold elasticity for the transaction count, you will note that there are
two examples (Customer IDs 113498258, 145213407) in which the customer has used their card
to withdraw funds from an ATM 6 times in 5 days. Under the general concept and purpose of the
red flag, it would be valuable to look at additional history for these two individuals to determine
if this type of activity is normal, expected, and in alignment with their customer profiles.
1313
Substantiation for a Threshold Tune, or not By executing this simple exercise, a system analyst is able to complete a quick evaluation of whether or
not the thresholds of the red flag may need to be tuned. Let’s assume now that the data from this exercise
did indicate that a threshold tuning exercise was necessary. If the organization has an annual periodic
review schedule for threshold tuning and this exercise was run three months since the last review, the
analyst would be making a major mistake by waiting nine months for the next review. It would be hugely
beneficial to the organization to execute a review on this particular red flag and tune the thresholds
accordingly. However, from an audit perspective, when the decision is made internally to tune thresholds
outside of regular periodic review schedule, an examiner/auditor would likely ask why that occurred and
what data does the company have on record to substantiate the change? This solution provides a significant value add to the organization for substantiation. Complete with the
snapshot of data and the consolidated analysis, the system analyst and the organization will be
equipped with the necessary data driven documentation required for an examiner/auditor.
Scaling the Solution for a Paradigm Shift in Threshold Tuning Schedules As previously mentioned, a threshold tuning exercise is nothing short of a significant undertaking for an organization. It is costly in time required, resources allocated, and cash spent. Considering the simple
example above, if this were the only red flag for the business, the cost to validate the thresholds in place
and tune if necessary would be relatively small and palatable. However, if you multiply the complexity of the red flag, then add many more red flags in the portfolio; the cost of this exercise grows
exponentially. Again, at the point in time when the annual periodic review for threshold tuning comes, resources must be dedicated to the project, and ultimately the result could be such that the thresholds
in place are strong and sufficient, or they needed attention months ago. Neither outcome is positive.
The implementation of a full-scale system described above provides an organization an auditable, scalable
solution that is schedule agnostic. When a solution such as this is applied, an organization can throw out their “periodic review for tuning”. Similar to the automatic alerts from a SAMS, as automation is
deployed, you can be confident that the solution will alert system analysts when additional due diligence
review should be completed on the thresholds of a red flag to determine if a threshold tuning exercise is
required. In addition to that targeted visibility, the organization will only be conducting reviews on the
thresholds of red flags that may need attention versus an end-to-end review of all red flag thresholds. This
affords the organization the opportunity to allocate the appropriate amount of resources based on the
predictive indicators triggered, and spend valuable time and money based on a data driven output.
Benefit of a Tuning Engine being in the Auditors Toolbox As an auditor, we’ve all spent an exorbitant amount of time evaluating and confirming the validity of threshold tuning. Generally speaking, the audit review is based on the results of an annual periodic review
of red flag thresholds. Under normal circumstances we review troves of data, organizational analysis, conclusions made, and then we must perform our own analysis of the work of the organization to
determine reasonable veracity of the actions, or not, taken.
It stands to be said that there are many significant benefits to an organization by leveraging this type of a
solution to proactively monitor red flag thresholds and evaluate tuning requirements based on predictive
indicators. As an auditor, we stand to benefit greatly by using it as well. For example, assume you are
working for a client that has this solution implemented. When you begin to audit the work of the persons managing the solution, you would just need to confirm the execution of their solution using
the same static thresholds, elastic thresholds, and associated steps. After inputting the information
1414
from their system into your system, your results should match their results. Once you have confirmed
the two configurations yield congruent results, you can now spend time on the truly valuable analysis; what has the company done with the resulting information and why? This is certainly a very simplistic
approach, but in reality the time spent pulling and evaluating sets of data is reduced dramatically for an
auditor. In another example, consider a client that only completes a periodic review of red flag thresholds
annually and does not have a solution like this implemented. As you begin auditing their work, you could
easily input the static thresholds of their SAMS in your solution. You would also input the thresholds they
used in their last review exercise into the elastic thresholds. Lastly, you would use the same data that they used in their review. Again, your results should match their results. However, you could easily make
adjustments to the elastic thresholds to expand/contract the scope of data and provide valuable insight back to the customer regarding potential vulnerabilities, or further support that their static thresholds are strong and sufficient. In both cases, an auditor can improve efficiency, generate
value added insight to clients, and improve the quality of the audit output by using a solution similar to
that proposed above.
Model Risk Validation & OCC 2011-12 Due to the complexity and nature of OCC 2011-12, it is indeed conceivable to construct a publication
solely on Model Risk Validation. In an effort to remain concise on this subject, I intend to outline only
the highlights of model risk validation as it relates to a Tuning Engine solution as proposed above. “Model risk management encompasses governance and control mechanisms such as board and senior management oversight, policies and procedures, controls and compliance….”8 As such, the solution
proposed above is indeed subject to model risk management and model validation pursuant to the
guidance outlined in OCC 2011-12. “Model validation is the set of processes and activities intended to
verify that models are performing as expected, in line with their design objectives and business uses.”9
The implementation of appropriate model validation can be completed in tandem with the model validation of the SAMS that the Tuning Engine serves. As is the case with traditional model risk
management, the Tuning Engine model “should include disciplined and knowledgeable development and implementation processes that are consistent with the situation and goals of the model user”10, as
well as organizational policy. Whether you are an FI, a program manager, or a service provider, the
validation of a Tuning Engine solution must be completed by competent individuals and they must adhere to organized validation processes. The good news is that the majority of the work is already
defined, well understood, and implemented for organizations that apply principles set forth by OCC 2011-12 for their SAMS. The Tuning Engine in operation is merely an extension of the SAMS in place, ergo
the model validation exercise could easily be extended and scaled to encompass the Tuning Engine as
well.
8,9,10 Office of the Comptroller of the Currency (OCC), “OCC 2011-12 Supervisory Guidance on Model Risk
management”, April 2011, pp. 28, pp. 99, pp. 510.
1515
Lastly, the Tuning Engine in operation could indeed be leveraged as a tool for SAMS model validation. For example, when a Tuning Engine delivers alerts on the predictive indicators, action must be taken to
determine if tuning is required. It is expected that this would occur periodically, but the volume and
velocity of such alerts on a single red flag should be rare. In the event that the Tuning Engine repeatedly
alerts system analysts for review on a single red flag, there may be a larger underlying problem with the
SAMS model, potentially pointing system analysts to an indication that the SAMS model itself may be
weak. The Tuning Engine has the ancillary benefit of providing data driven indication that model validation
may be required on a single red flag, and perhaps on a SAMS system as a whole.
Conclusion As an auditor, we have a fiduciary responsibility to assess and confirm the validity, or not, of processes
in place within an institution’s BSA/AML Compliance Program. Included as a critical component of that program is suspicious activity monitoring. The size and complexity of the systems that provide a medium
of identifying suspicious activity directly correlates to the size and complexity of the organization that the system serves. “Suspicious activity monitoring and reporting are critical internal controls. Proper monitoring and reporting processes are essential to ensuring that the bank has an adequate and effective
BSA compliance program. Appropriate policies, procedures, and processes should be in place to monitor and identify unusual activity. The sophistication of monitoring systems should be dictated by the bank’s risk profile, with particular emphasis on the composition of higher-risk products, services, customers, entities, and geographies. The bank should ensure adequate staff is assigned to the identification, research, and reporting of suspicious activities, taking into account the bank’s overall risk profile and the
volume of transactions.”11
Institutions with robust corporate governance embrace the responsibility of ensuring that suspicious
activity monitoring solutions are implemented, maintained, and relevant to the organizational risk
assessment. Threshold tuning is an exercise that is paramount in the maintenance and relevancy of any
suspicious activity monitoring solution. Improperly tuned solutions leave the institution vulnerable to
regulatory violations, as well as a potential for significant revenue loss. It is common for institutions to
engage in an annual periodic review of thresholds in an effort to fulfill regulatory obligations, as well as
to glean insight into the status quo of the validity of thresholds of current red flags. Regardless of scope, the costs of conducting such a review are significant to institutions and it is paramount that the efforts
put forth to the task be focused and calculated.
The aforementioned Tuning Engine software solution is designed to accomplish exactly that. The Tuning
Engine, when scoped and scaled to the needs of the SAMS in place, allows an organization to focus its
personnel and monetary resources only to those thresholds that need review, versus applying attention
to 100 percent of the thresholds of 100 percent of red flags. In addition, the solution would effectively
deprecate a periodic review schedule, no matter what the timeframe is. Periodic reviews of thresholds
would no longer be required with the solution in place as the system would be monitoring for fluidity in
the data and alerting analysts of thresholds that should be reviewed 24/7/365. In short, the Tuning Engine
software solution sets forth the capability of identifying and tuning the thresholds of red flags responsibly, and with substantiation, for a fraction of the resources and money that is spent by institutions today.
11 Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014, pp. 66.
1616
Works Cited 1. Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014. 2. FinCEN, “FIN-2014-A007”, August 2014. 3. PricewaterhouseCoopers, “From source to surveillance: the hidden risk in AML monitoring system
optimization”, September 2010. 4. Protiviti, “Implementing AML Transaction Monitoring Systems: Critical Considerations”, September
2013. 5. Protiviti, “Views on AML Transaction Monitoring Systems”, 2013. 6. Office of the Comptroller of the Currency (OCC), “OCC 2011-12 Supervisory Guidance on Model Risk
management”, April 2011.