Suspicious Activity

Monitoring; An Innovative and Auditable Approach to

Threshold Tuning

Scot Luther

2

Executive Summary A strong and sufficient suspicious activity monitoring system is a critical component, amongst many, in a

robust Bank Secrecy Act (“BSA”)/Anti-Money Laundering (“AML”) Compliance Program. “Suspicious

activity reporting forms the cornerstone of the BSA reporting system.”1 Suspicious activity monitoring

systems of most financial institutions are typically reviewed and audited as a part of a regularly scheduled

independent audit, which is required as one of the pillars of in a comprehensive BSA/AML Compliance

Program. For most regulated financial institutions, implementing a rigorous suspicious activity monitoring

system is no longer a Herculean feat, so long as proper planning and execution are engaged. In fact, there

are many third-party vendors that can be leveraged to provide software and consulting services to

implement such a system. However, once the implementation concludes and the system is stabilized and

operational, every organization faces the challenge of threshold tuning, and determining how and when

to do so.

To date, it has been a generally accepted practice for financial institutions to execute periodic threshold

tuning to fulfill regulatory and audit requirements. In many cases, the period of review is annually. A

financial institution may spend tens or hundreds of thousands of dollars in hard and soft costs on a

threshold tuning exercise to ensure obligations are met, only to find that the thresholds in place are still strong and sufficient. In contrast, a financial institution may execute an annual threshold tuning exercise

that confirms the existing thresholds are indeed weak and prone to risk. A subsequent lookback may also

indicate that thresholds should have been reviewed and tuned six or nine months prior and a significant number of reportable activity may have been missed. Each scenario mentioned fails to demonstrate an

effective, auditable threshold tuning methodology and both scenarios face ancillary costs and risks as a

result.

This whitepaper provides a high-level overview of a suspicious activity monitoring system and the

necessity of threshold tuning as a critical component to the typology validation. Most importantly, the

whitepaper will introduce a conceptual implementation of a software solution that provides financial institutions and suspicious activity monitoring system managers automated, data-driven predictive

indicators and alerts that a threshold should be reviewed and possibly tuned. The solution would be

integrated to the suspicious activity monitoring system in place. It would also be running in real-time to

provide alerts that could justify the obsolescence of an annual periodic threshold tuning exercise. The

predictive indicators would also include data snapshot captures that financially rationalize threshold

tuning exercises from a business perspective, all while providing an auditor concrete justification for a

threshold tuning exercise.

1 Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014, pp. 399

(R-5).

3

Compliance Culture and BSA/AML Compliance Program Management Effective management of an organization’s BSA/AML Compliance Program is a monumental task that requires championship from organizational leadership, as well as support from all employees. In

addition to managing and maintaining the health of the business, corporate leadership is also

responsible for ensuring that compliance obligations are fulfilled and that appropriate support and

resources are allocated to fulfilling such requirements. As a baseline, it is essential for organizational leadership to create and support a culture of compliance from top to bottom in the company. “For a

BSA/AML compliance program to be effective, it should have the demonstrable support of the

leadership (as appropriate based on the financial institution’s size and structure).”2 A significant component of said support and culture is managing the business congruent to a strong and sufficient BSA/AML Risk Assessment. For the purpose of this document, I will not conduct an in-depth analysis and

outline of the BSA/AML Risk Assessment. However, it is critical to note and understand that the BSA/AML

Risk Assessment is the fundamental foundation of a robust BSA/AML Compliance Program and that all roads from risk mitigation and management must lead back to the BSA/AML Risk Assessment. Most important to this document is that the suspicious activity monitoring system, as well as the proposed

tuning solution, should be derived from the BSA/AML Risk Assessment.

Suspicious Activity Monitoring System (“SAMS”) A fundamental and necessary internal control of the BSA/AML Compliance Program is an effective and

comprehensive transaction monitoring system that will identify potentially suspicious activity. “Suspicious

activity monitoring and reporting are critical internal controls. Proper monitoring and reporting processes

are essential to ensuring that the bank has an adequate and effective BSA compliance program.”3 As was

previously discussed, such potentially suspicious activity is derived from the most recent and current BSA/AML Risk Assessment. A SAMS could be as simple as manual reviews of a few defined red flags that is managed by one or two people handling alerts. In contrast, there are systems in place that are

extremely complex, inclusive of hundreds of identified red flags and vulnerabilities with multiple layers of thresholds that are managed by hundreds of alert analysts. No matter which end of the spectrum that you are on, the system is critical nonetheless and the complexity of your system should be derived from

your BSA/AML Risk Assessment. In accordance with the Federal Financial Institutions Examination Council (“FFIEC”), the “sophistication of monitoring systems should be

dictated by the bank’s risk profile”. 4

For the purpose of this document, I have diagrammed a very simplified and very high-level overview of a

basic SAMS. It must be understood that SAMS solutions in large scale implementations are extremely

complex in design and configuration. The diagram provided below is meant to simply be used to depict the location and positioning of an innovative and alternative Tuning Engine solution that could be

employed to replace traditional red flag tuning schedules. See Diagram A below.

2 FinCEN, “FIN-2014-A007”, August 2014, pp. 2. 3,4 Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014, pp. 66.

4

Diagram A

No matter how simple or complex the SAMS is, the system remains a critical component in a robust BSA/AML Compliance Program. SAMS continuously supports the global efforts to prevent money

laundering and combat terrorist financing. While fulfilling regulatory obligations, these systems can also

be leveraged to save an organization thousands of dollars in fraud losses. “Actively managing AML

monitoring data can reduce costs, save time, avoid regulatory remediation, and improve assurance.”5

When utilized to the fullest extent of its capabilities, these systems will pay for themselves in true

monetary savings, all while fulfilling regulatory requirements.

5 PricewaterhouseCoopers, “From source to surveillance: the hidden risk in AML monitoring system

optimization”, September 2010, pp. 1.

5

Below are just a few of tangible benefits of a properly implemented and managed SAMS.

Identify Suspicious Activity to be Reviewed and/or Reported – The main purpose of a suspicious

activity monitoring system is to identify suspicious activity that needs to be reviewed and

potentially reported to the appropriate financial intelligence unit (FIU). Properly implemented

systems are designed to achieve exactly this, which supports a financial institution’s efforts in

fulfilling regulatory obligations.

Fulfill Regulatory Obligations Efficiently – A well planned and executed implementation for a

SAMS will provide effective tools to an organization that will enable it to efficiently fulfill the

portion of regulatory obligations for internal controls, suspicious activity monitoring, and

suspicious activity reporting. The SAMS will automate, analyze, and alert analysts based on

millions of records of data that would be an insurmountable task for human beings to perform

manually.

Identify & Prevent Fraud Losses – Real-time and near real-time monitoring systems are able to

identify and prevent potential fraud as it is happening. Once flagged, the system can

automatically invoke suspension of the account and/or card from being able to be used until contact is made with the customer. Depending on the type of transaction processed, this

recognition and intervention can yield thousands of dollars in savings from fraud losses. In many

cases the fraud that is identified is also suspicious activity that needs to be reported to an FIU.

Supports a Process-Driven Enterprise – A SAMS systematic approach to management supports

the concept of process definition, management, and improvement. Automated systems

challenge organizations to clearly define how a process truly works. The product of such an

exercise is one or many lucid, organized process maps that outline exactly how a process flows. Once the processes are agreed upon and well understood by key stakeholders, the organization

is able to begin implementing rules and thresholds that support the management of the process

in alignment with the BSA/AML Risk Assessment.

Data Driven Visibility – The data in a SAMS is invaluable. Because the data is organized in a

normalized structure, reporting solutions can be configured to generate output reports for multiple functional department uses. In addition to the regulatory and compliance benefits, reporting can also be tailored to corporate leadership that identifies risks, costs, as well as

fluidity and changes in the business.

Although a SAMS is remarkably useful and beneficial to an organization, these systems do not come

without challenges. There are many ancillary costs, both hard and soft, that are associated to managing

these systems in large institutions.

System Management Resources – A SAMS is typically a fairly complicated system to operate

and maintain. The initial cost of the software itself is merely one component. Once the

software is implemented and in place, skilled personnel are required to manage and maintain

the systems. These costs must be accounted for in the organizational annual budget when

implementing such a system. In addition, applying personnel that are not qualified for the job

may be worse than having no personnel at all.

Data Integrity – The infamous saying “garbage in garbage out” holds true with these systems. It is paramount that properly qualified resources are engaged to manage the data that goes through

the transaction normalization process and ultimately is infused into the SAMS

6

Transaction Data Mart. Data that is not properly managed creates unexpected risks for the

business that could yield catastrophic consequences.

Obsolete or Invalid Thresholds – Resources in the business world today are stretched incredibly

thin. One of the many poor ways to utilize time and effort of people is to work on tasks that are

invalid or obsolete. If an organization lacks effective management processes around red flag

definitions, the analysts that handle alerts will likely be spending time unnecessarily on alerts that have no value. Threshold tuning is a necessary function in managing a SAMS to ensure the

“engine” is running optimally. Without effective processes and controls around the maintenance

of the system, an organization may be missing critical, reportable activity, or the alert system is

being inundated with alerts that serve no value. In either case, the lack of attention to these areas

proves to be extremely expensive.

The Problem with Traditional Threshold Tuning of a Suspicious Activity Monitoring System There have

been tremendous advancements made with suspicious activity monitoring solutions, identification, and

reporting over the past twenty to thirty years. The solutions in place today are much more robust than

five years ago, and even more so than ten to twenty years ago. However, amidst all of the improvements

made with SAMS solutions, there is still an industry-wide problem in place at most organizations involving

threshold tuning. The red flags defined in a SAMS are only as good as the static thresholds that are

currently applied to the ever-dynamic business they are serving. Unfortunately, some of the static

thresholds could be out of date for months and the institution doesn’t even know it.

“Additionally, it is imperative to execute a threshold-tuning exercise on a periodic basis that consists of generating an investigating alerts just below the threshold values. This exercise gives insight into the

existence (or lack) of suspicious activity just below the set thresholds. Existence of such activity will require the thresholds to be lowered. If there is no suspicious activity just below the threshold values, then a separate exercise consisting of lifting the threshold values above the current values can be

performed. If this exercise yields the same alerts, then there may be a case for lifting the threshold

values in production.”6

As previously mentioned, to date many organizations have tried to mitigate the risk of stale thresholds

by implementing a regularly scheduled threshold tuning exercise. In many institutions this is an annual review of all red flags and thresholds. Therein lies a critical defect; an annual review to confirm the

validity of thresholds of a near real-time or a real-time monitoring solution.

Depending on the number of red flags and thresholds involved, this could be an extremely expensive

and time-consuming exercise. Unfortunately, the only way to justify to an examiner that the SAMS

thresholds in place are effective is to complete the exercise.

Among many others, below are at least three fundamental issues with a periodic threshold tuning

exercise.

Cost – The exercise is needed for justification so it must be completed. The hard and soft costs

incurred are an assumed “cost of doing business” to fulfill the regulatory obligation.

6 Protiviti, “Implementing AML Transaction Monitoring Systems: Critical Considerations”, September 2013, pp. 5.

7

Confirmation Thresholds Are Effective – The double-edge sword result of a threshold tuning

exercise is that the business confirms that the thresholds in the SAMS are effective. This means

the organization blindly wasted money, time, effort, and realized opportunity-cost losses as a result of performing the exercise, only to reaffirm that the thresholds were effective. If the business had known prior to performing the exercise that the thresholds were effective

there would have been no business justification to complete it.

Confirmation Thresholds Are Ineffective – If a threshold tuning exercise confirms that the

thresholds are ineffective, it could be misconstrued that the exercise was justified and successful. The business would make necessary corrections to the thresholds, re-run an exercise

to confirm effectiveness, and proceed with business as usual for the next 12 months. What may

or may not be answered by the business is how long have the thresholds been ineffective? If the thresholds have been ineffective for 10 months and the last exercise occurred 12 months

ago, it is likely the organization has either missed reportable SAR activity and/or alert analysts

have been wasting their time addressing false positive alerts. In either case, the organization

was at risk for some undetermined amount of time.

Threshold tuning is a necessary process in effective model management of a SAMS. The thresholds

defined for red flags in the SAMS are static by design; in fact they must be for auditability purposes. These static thresholds allow for concrete management and measurement of activity and alerts for suspicious activity. As institutions change, grow and/or shrink, the validity of thresholds in the SAMS

will likely become out-of-date and ineffective.

In some cases an organization may not even have a regimen defined that outlines how and when a tuning

exercise is to occur. The lack of this process leaves the organization vulnerable to missing SAR worthy

activity, or generating so many false positives that the team that manages alerts becomes ineffective due

to false positive overload. “The absence of periodic tuning of scenarios often results in numerous false

positives, which in turn delay alert investigation and ultimately lead to missed reporting deadlines.”7 In

either case, this is a costly mistake that organizations are able to overcome.

An Innovative and Auditable Approach for Threshold Tuning of a Suspicious Activity Monitoring System With the obvious gaps in traditional threshold tuning schedules and methodologies today, there is a

significant opportunity to improve processes and reduce risk within your organization by leveraging

automation, software, and processing power. Simple and complex computing can be used to alert SAMS

managers that transaction history trends are changing and static thresholds in place may be obsolete. The

value in this type of solution is that the subjective guesswork of when to tune is removed and you let your data and predictive indicators make the decision for you. From a business perspective this

justification balances the questions of cost. From an audit perspective the systematic approach is more

robust because it shortens the window of missing reportable activity, it reduces the time that analysts

work on false positive noise, and the data from the solution justifies any threshold modifications that occur as a result of the exercise.

7 Protiviti, “Views on AML Transaction Monitoring Systems”, 2013, pp. 14.

8

The following information outlines a very high-level system that leverages the data you already use today, in conjunction with your existing control tables of your SAMS, to output alerts and predictive indicators

of when to review and possibly tune your thresholds. In addition to the high-level overview, a live red flag

and data example is used to framework how this is implemented today. Lastly, we cover a summarization

of benefits of the solution, including the viability of an auditor using the solution to audit an organization’s

SAMS using static datasets at certain times.

To revisit the previous diagram outlining a high-level SAMS configuration, this solution will focus on the

components highlighted in blue in Diagram B below. Note that this diagram is a zoomed in scope of a

subset of Diagram A above. The only true addition is the concept of Elasticity Data.

Diagram B

What is Elasticity in this context and why does it matter? By simple definition, elasticity is referred to as “stretchiness”. In business you might hear executives or leadership refer to “stretch thinking” or thinking beyond your traditional myopic view. In the context of this solution, we are accomplishing exactly that. The goal here is to stretch the static thresholds in place

to a finite position. By doing so, the scope of data brought into the evaluation increases and decreases, depending on the stretch. With that increase or decrease in data, you will then be able to determine if there is any additional information that can be ascertained by the stretch as it relates to the red flag in

question.

9

Tuning Engine The Tuning Engine is the software that handles the processing of elasticity data. At a very high level, this

software solution will iterate through your red flags, as well as each flag’s associated elasticity data, to

provide you variable alert volumes at each elastic interval. The alert volume at each interval will equip

you with the predictive indicators necessary to justify, or not, completing a tuning exercise on a particular red flag.

Tuning Engine Database The Tuning Engine Database (TED) is very simple in structure. The TED consists of a normalized data table

to hold your SAMS threshold data. The SAMS threshold data is used as the baseline for the Tuning Engine. A data table for elasticity data is also required. This data table will hold attributes of how much to flex a

threshold, and how elastic each step will be.

Use Transaction Data That You Already Have It is likely that your organization has already spent thousands of dollars to implement a strong and

sufficient SAMS. There is tremendous value in the data the SAMS holds, as well as the engine of the

solution that generates your alerts. Apart from what the SAMS is originally designed for, this same

conceptual processing can be applied, leveraging the data that is already used. With the addition of a

new structure of data holding elasticity information, the Tuning Engine system is able to achieve near real-time visibility into when a tuning exercise may be needed. This is powerful when justifying why a

tuning exercise is being proposed, and prevailing in an audit when you have tuned thresholds outside of a regularly scheduled threshold tuning exercise.

Use SAMS Threshold Data as Your Baseline The system requires a baseline. The best place to start is by using the thresholds of the SAMS. These

thresholds, if implemented correctly, should be the most relevant thresholds that align to each

corresponding red flag, which ties back to the organizational risk assessment. The Tuning Engine will use

the baseline thresholds as a place to start. Using that data, in addition to the elasticity values set in the

Tuning Engine database, the Tuning Engine can efficiently step through vast amounts of data to provide

you insights into alert volumes at varying threshold fluctuations.

Live Example in Operation We will now look in depth at a live working example of a simple red flag in practice titled “8 or More ATM

Withdrawals in 7 days. Note that with this red flag there are two static thresholds defined in the SAMS

database. These two static thresholds are a count of 8 ATM withdrawals within a timeframe of 7 days. The purpose of the existing thresholds set is such that the flag should catch cardholders that are

excessively using their card to withdraw cash from an ATM, at least one time per day for a period of at least one week, one of those days withdrawing twice.

Each day the SAMS conducts an evaluation of live data using the threshold data of this red flag. At the

conclusion of the evaluation each day, most of the days the evaluation yields no results. However on

some occasions there are two or three customers that generate an alert to be reviewed.

Note the very simplistic and high-level Diagram C below containing the TED data tables.

1010

Diagram C Example evaluating a red flag titled “8 or More ATM Withdrawals in 7 Days”

1111

Now let’s evaluate the results with elasticity from the Tuning Engine. Below is an outline of the scope of the example.

The date range of data used for this example is July 1, 2016 through July 31, 2016

“Current Date” simulation is set for July 20, 2016. This example will allow an auditor to review

the subset of data as if the day were July 20, 2016.

Given the elastic thresholds set above, there are 35 permutations evaluated, with time

threshold elastic spanning between the dates of July 9 through July 20 (5 days back through 11

days back), as well as count threshold elasticity from 6 transactions up to 10 transactions.

Diagram D Below is an image of the first records of transaction data used in the example. There are 8676 total records in the population for this example. To review all of the actual data used, see the attached Excel file titled RedFlagExampleData.xlsx >> Transaction Data worksheet.

No Elasticity Implemented As expected and as is most common, when no elasticity is implemented into the model, no results are

returned.

1212

Full Elasticity Implemented Using the example of the elastic thresholds outlined in Diagram C above, let’s evaluate the results with

full elasticity implemented. To review the results of full flexion being implemented, see the attached

Excel file titled RedFlagExampleData.xlsx >> Results-Full Elasticity worksheet.

Customer ID

Min ATM

Transaction

Count

Min Count

Elastic

Threshold

Min Time

Elastic

Threshold

Time Unit

Max ATM

Transaction

Count

Max Count

Elastic

Threshold

Max Time

Elastic

Threshold

Time Unit

94246847 6 6 -8 days back 6 6 -11 days back

96183347 6 6 -10 days back 6 6 -11 days back

106050219 6 6 -9 days back 8 8 -11 days back

106098970 6 6 -8 days back 8 8 -11 days back

106102796 7 6 -9 days back 9 9 -11 days back

111379131 6 6 -10 days back 6 6 -11 days back

113498258 6 6 -5 days back 7 7 -11 days back

114645430 6 6 -10 days back 6 6 -11 days back

118065927 6 6 -11 days back 6 6 -11 days back

124297195 6 6 -7 days back 7 7 -11 days back

126648296 6 6 -7 days back 6 6 -11 days back

140897528 6 6 -8 days back 6 6 -11 days back

141433932 6 6 -9 days back 6 6 -11 days back

145213407 6 6 -5 days back 6 6 -11 days back

So What Does This Mean? Keep in mind that the example used for the purpose of this exercise is rather simple in scope. In large

institutions the complexity of the red flag may include multiple dimensions for elasticity. In this case

there are only two dimensions, which are the Transaction Count and Time. However, there is some

interesting information that can be extrapolated from this evaluation and data.

Are the thresholds of this red flag strong and sufficient? The data suggests that they might be. An analyst will only be able to conclude this assumption definitively by reviewing specific

examples in the data.

On the upper end of the threshold elasticity for the transaction count, you will note that there is

only one example (Customer ID 106102796) in which the Maximum ATM Transaction Count exceeded the existing threshold set in the SAMS. You will note that the threshold was only

exceeded after 3 additional days back of data were evaluated.

On the lower end of the threshold elasticity for the transaction count, you will note that there are

two examples (Customer IDs 113498258, 145213407) in which the customer has used their card

to withdraw funds from an ATM 6 times in 5 days. Under the general concept and purpose of the

red flag, it would be valuable to look at additional history for these two individuals to determine

if this type of activity is normal, expected, and in alignment with their customer profiles.

1313

Substantiation for a Threshold Tune, or not By executing this simple exercise, a system analyst is able to complete a quick evaluation of whether or

not the thresholds of the red flag may need to be tuned. Let’s assume now that the data from this exercise

did indicate that a threshold tuning exercise was necessary. If the organization has an annual periodic

review schedule for threshold tuning and this exercise was run three months since the last review, the

analyst would be making a major mistake by waiting nine months for the next review. It would be hugely

beneficial to the organization to execute a review on this particular red flag and tune the thresholds

accordingly. However, from an audit perspective, when the decision is made internally to tune thresholds

outside of regular periodic review schedule, an examiner/auditor would likely ask why that occurred and

what data does the company have on record to substantiate the change? This solution provides a significant value add to the organization for substantiation. Complete with the

snapshot of data and the consolidated analysis, the system analyst and the organization will be

equipped with the necessary data driven documentation required for an examiner/auditor.

Scaling the Solution for a Paradigm Shift in Threshold Tuning Schedules As previously mentioned, a threshold tuning exercise is nothing short of a significant undertaking for an organization. It is costly in time required, resources allocated, and cash spent. Considering the simple

example above, if this were the only red flag for the business, the cost to validate the thresholds in place

and tune if necessary would be relatively small and palatable. However, if you multiply the complexity of the red flag, then add many more red flags in the portfolio; the cost of this exercise grows

exponentially. Again, at the point in time when the annual periodic review for threshold tuning comes, resources must be dedicated to the project, and ultimately the result could be such that the thresholds

in place are strong and sufficient, or they needed attention months ago. Neither outcome is positive.

The implementation of a full-scale system described above provides an organization an auditable, scalable

solution that is schedule agnostic. When a solution such as this is applied, an organization can throw out their “periodic review for tuning”. Similar to the automatic alerts from a SAMS, as automation is

deployed, you can be confident that the solution will alert system analysts when additional due diligence

review should be completed on the thresholds of a red flag to determine if a threshold tuning exercise is

required. In addition to that targeted visibility, the organization will only be conducting reviews on the

thresholds of red flags that may need attention versus an end-to-end review of all red flag thresholds. This

affords the organization the opportunity to allocate the appropriate amount of resources based on the

predictive indicators triggered, and spend valuable time and money based on a data driven output.

Benefit of a Tuning Engine being in the Auditors Toolbox As an auditor, we’ve all spent an exorbitant amount of time evaluating and confirming the validity of threshold tuning. Generally speaking, the audit review is based on the results of an annual periodic review

of red flag thresholds. Under normal circumstances we review troves of data, organizational analysis, conclusions made, and then we must perform our own analysis of the work of the organization to

determine reasonable veracity of the actions, or not, taken.

It stands to be said that there are many significant benefits to an organization by leveraging this type of a

solution to proactively monitor red flag thresholds and evaluate tuning requirements based on predictive

indicators. As an auditor, we stand to benefit greatly by using it as well. For example, assume you are

working for a client that has this solution implemented. When you begin to audit the work of the persons managing the solution, you would just need to confirm the execution of their solution using

the same static thresholds, elastic thresholds, and associated steps. After inputting the information

1414

from their system into your system, your results should match their results. Once you have confirmed

the two configurations yield congruent results, you can now spend time on the truly valuable analysis; what has the company done with the resulting information and why? This is certainly a very simplistic

approach, but in reality the time spent pulling and evaluating sets of data is reduced dramatically for an

auditor. In another example, consider a client that only completes a periodic review of red flag thresholds

annually and does not have a solution like this implemented. As you begin auditing their work, you could

easily input the static thresholds of their SAMS in your solution. You would also input the thresholds they

used in their last review exercise into the elastic thresholds. Lastly, you would use the same data that they used in their review. Again, your results should match their results. However, you could easily make

adjustments to the elastic thresholds to expand/contract the scope of data and provide valuable insight back to the customer regarding potential vulnerabilities, or further support that their static thresholds are strong and sufficient. In both cases, an auditor can improve efficiency, generate

value added insight to clients, and improve the quality of the audit output by using a solution similar to

that proposed above.

Model Risk Validation & OCC 2011-12 Due to the complexity and nature of OCC 2011-12, it is indeed conceivable to construct a publication

solely on Model Risk Validation. In an effort to remain concise on this subject, I intend to outline only

the highlights of model risk validation as it relates to a Tuning Engine solution as proposed above. “Model risk management encompasses governance and control mechanisms such as board and senior management oversight, policies and procedures, controls and compliance….”8 As such, the solution

proposed above is indeed subject to model risk management and model validation pursuant to the

guidance outlined in OCC 2011-12. “Model validation is the set of processes and activities intended to

verify that models are performing as expected, in line with their design objectives and business uses.”9

The implementation of appropriate model validation can be completed in tandem with the model validation of the SAMS that the Tuning Engine serves. As is the case with traditional model risk

management, the Tuning Engine model “should include disciplined and knowledgeable development and implementation processes that are consistent with the situation and goals of the model user”10, as

well as organizational policy. Whether you are an FI, a program manager, or a service provider, the

validation of a Tuning Engine solution must be completed by competent individuals and they must adhere to organized validation processes. The good news is that the majority of the work is already

defined, well understood, and implemented for organizations that apply principles set forth by OCC 2011-12 for their SAMS. The Tuning Engine in operation is merely an extension of the SAMS in place, ergo

the model validation exercise could easily be extended and scaled to encompass the Tuning Engine as

well.

8,9,10 Office of the Comptroller of the Currency (OCC), “OCC 2011-12 Supervisory Guidance on Model Risk

management”, April 2011, pp. 28, pp. 99, pp. 510.

1515

Lastly, the Tuning Engine in operation could indeed be leveraged as a tool for SAMS model validation. For example, when a Tuning Engine delivers alerts on the predictive indicators, action must be taken to

determine if tuning is required. It is expected that this would occur periodically, but the volume and

velocity of such alerts on a single red flag should be rare. In the event that the Tuning Engine repeatedly

alerts system analysts for review on a single red flag, there may be a larger underlying problem with the

SAMS model, potentially pointing system analysts to an indication that the SAMS model itself may be

weak. The Tuning Engine has the ancillary benefit of providing data driven indication that model validation

may be required on a single red flag, and perhaps on a SAMS system as a whole.

Conclusion As an auditor, we have a fiduciary responsibility to assess and confirm the validity, or not, of processes

in place within an institution’s BSA/AML Compliance Program. Included as a critical component of that program is suspicious activity monitoring. The size and complexity of the systems that provide a medium

of identifying suspicious activity directly correlates to the size and complexity of the organization that the system serves. “Suspicious activity monitoring and reporting are critical internal controls. Proper monitoring and reporting processes are essential to ensuring that the bank has an adequate and effective

BSA compliance program. Appropriate policies, procedures, and processes should be in place to monitor and identify unusual activity. The sophistication of monitoring systems should be dictated by the bank’s risk profile, with particular emphasis on the composition of higher-risk products, services, customers, entities, and geographies. The bank should ensure adequate staff is assigned to the identification, research, and reporting of suspicious activities, taking into account the bank’s overall risk profile and the

volume of transactions.”11

Institutions with robust corporate governance embrace the responsibility of ensuring that suspicious

activity monitoring solutions are implemented, maintained, and relevant to the organizational risk

assessment. Threshold tuning is an exercise that is paramount in the maintenance and relevancy of any

suspicious activity monitoring solution. Improperly tuned solutions leave the institution vulnerable to

regulatory violations, as well as a potential for significant revenue loss. It is common for institutions to

engage in an annual periodic review of thresholds in an effort to fulfill regulatory obligations, as well as

to glean insight into the status quo of the validity of thresholds of current red flags. Regardless of scope, the costs of conducting such a review are significant to institutions and it is paramount that the efforts

put forth to the task be focused and calculated.

The aforementioned Tuning Engine software solution is designed to accomplish exactly that. The Tuning

Engine, when scoped and scaled to the needs of the SAMS in place, allows an organization to focus its

personnel and monetary resources only to those thresholds that need review, versus applying attention

to 100 percent of the thresholds of 100 percent of red flags. In addition, the solution would effectively

deprecate a periodic review schedule, no matter what the timeframe is. Periodic reviews of thresholds

would no longer be required with the solution in place as the system would be monitoring for fluidity in

the data and alerting analysts of thresholds that should be reviewed 24/7/365. In short, the Tuning Engine

software solution sets forth the capability of identifying and tuning the thresholds of red flags responsibly, and with substantiation, for a fraction of the resources and money that is spent by institutions today.

11 Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014, pp. 66.

1616

Works Cited 1. Federal Financial Institutions Examination Council (FFIEC), “FFIEC Examination Manual”, 2014. 2. FinCEN, “FIN-2014-A007”, August 2014. 3. PricewaterhouseCoopers, “From source to surveillance: the hidden risk in AML monitoring system

optimization”, September 2010. 4. Protiviti, “Implementing AML Transaction Monitoring Systems: Critical Considerations”, September

2013. 5. Protiviti, “Views on AML Transaction Monitoring Systems”, 2013. 6. Office of the Comptroller of the Currency (OCC), “OCC 2011-12 Supervisory Guidance on Model Risk

management”, April 2011.