1 Insight. Oversight. Foresight. SMMichigan Texas Florida North Carolina

Data Analytics:Implementation

Presented by:Robin D. Hoag, CPA, CGMA, CMCShareholder, Financial Institutions Group

Region 3 MeetingSeptember 18 – 20, 2019Lansing, Michigan

2

Financial Institutions GroupToday’s Objectives

• Introduction to data analytics• Why it’s important to Internal Auditors• Overview of the key elements, attributes, challenges• Steps in the data analytics process• Data analytic tools• Roles and responsibilities• Applications for Internal Audit• Resources

3

Financial Institutions GroupWhat is Data Analytics? Definition

• The process of inspecting, cleansing, transforming, and modeling data with the objective of highlighting meaningful information, suggesting conclusions, and supporting decision-making.

4

Financial Institutions GroupData Analytics: More!

• Problem-solving process• Extracts insights• Historical, real-time, or predictive• Data analytics (DA) can be:

• Risk-focused (i.e., controls effectiveness, fraud, waste, policy/regulatory non-compliance)

• Performance-focused (i.e., increased sales, decreased costs, improved profitability.)

5

Financial Institutions GroupFOCUS on Relationships

• Identify and interpret relationships among variables to facilitate decision-making using the Five W’s:• Who• What• Why• Where• When

6

Financial Institutions Group

Why Is Data Analytics Important To Internal Audit?

Strategic Area EnhancementCredit Union Expectations Audit coverage, quality, business

impact, on a finite audit budgetRegulatory Expectations Stronger assurance and quantifiable

resultsCompetitive Landscape Strengthen capabilities

Seek new talentIncreased Value Deeper discussion on issues

Develop/strengthen relationships

Talent Development Strengthen business skills Appeal to other staff members Boost recruiting

7

Financial Institutions Group

Why Is Data Analytics Important To Internal Audit?

• Internal audit departments leverage data analytics in order to:• Identify additional risk• Increase scope and coverage - assurance• Better understand existing risk through data• Identify holes in control systems• Augment limited resources• Provide insights to management

8

Financial Institutions Group

Why Is Data Analytics Important To Internal Audit?• Some areas that benefit from data analytics:

• Loan operations• Accounting and general ledger transactions• Accounts payable• Payroll• Deposit transactions• Compliance• Legal and regulatory compliance

9 Insight. Oversight. Foresight. SM

Challenges To Using Data Analytics

10

Financial Institutions GroupPeople

• Limited resources (financial and human) to execute on a sustained basis

• Appetite for investment in time and training needed to develop an effective DA / AI process

• Someone needs to create, run, and maintain queries• Proficiency using analytic software• Proficiency in performing analysis

The top barrier for implementation of big data analytics is “inadequate staffing or skills for big data analytics.”

(Source: The Data Warehousing Institute (TDWI))

11

Financial Institutions GroupPeople: Senior Analysts

12 Insight. Oversight. Foresight. SM

Past, Present, and Not So Distant Future

13

Financial Institutions Group

“Computers will disrupt work habits and replace old jobs with ones that are radically different.”

Isaac Asimov predicts 2019 (in 1983)

14

Financial Institutions Group

We are in the Midst of a Data-Enabled Shift that is Transforming Our Stakeholders and How We Serve Them

We have transformed from a “Digital Driven Economy” to where the “Economy is Digital”

15

Financial Institutions GroupImpact of Analytics

16

Financial Institutions Group

Artificial Intelligence: Introducing the Technology

“I think the simplest definition of AI is that it’s a machine or a computer exhibiting the characteristics that we would normally associate with human intelligence”

Robin Grosset, CTO MindBridge AI

17

Financial Institutions GroupLet’s Get a Bit Technical

Artificial Intelligence

1956+

Early artificial

intelligence stirs

excitement

Machine Learning1980+

Machine learning begins

to flourish

Deep Learning2010+

Deep learning breakthroughs drive AI

boom (Google Cat)

18

Financial Institutions Group

Comparing Ensemble AI with Traditional Rules

Rules alone: Ranked risk score

Machine learning: Ranked risk score

Recent real-world example:In 1.6M transactions, flagging those of interest • Machine learning identified the fraud as 34th riskiest transaction• Traditional rules ranked the fraud ~31,000th in priority to examine

900X improvement

19

Financial Institutions Group

What Can Machines Do Better Than Humans?

20

Financial Institutions GroupTrust Within AI-Enabled Analytics

21

Financial Institutions GroupTraditional CAATs

Traditional CAAT tools are rules-based, with limited coverage:

22

Financial Institutions GroupUsing AI

23

Financial Institutions GroupThe Art of Evaluating AI

Its potential for good and progress will only fully emerge when algorithms become explainable in simple language

24

Financial Institutions GroupTake-Aways

• The pace of technological disruption is accelerating and will continue to impact us

• Big data and improved processing power, drive the development of AI applications

• Big data is underutilized in the internal audit space and AI can help harvest those opportunities

• AI can help with narrow tasks (data processing, risk scoring, search) with the human focusing on cognitive tasks (communicating and advising)

• Adopting AI is a journey and not a silver bullet. Educate yourself and start small.

25 Insight. Oversight. Foresight. SM

Data Analytics Tools

26

Financial Institutions GroupData Analytics Tools

• The right data analytic software will:• Handle large data sets efficiently• Integrate well with big-data• Include wide-array of analytical and statistical functions and procedures• Be relatively easy to program• Log procedures performed on data• Allow users to easily re-run analysis with minor changes• Be scalable with regards to the platform• Ensure the vendor’s vision is inline with the organization’s vision• Include training and support

27

Financial Institutions GroupData Analytics Tools

• Basic• Microsoft Excel• Microsoft Access

• Integrated query tools• PeopleSoft• SAP• Oracle• JDE

• Specialized DA visualization software• Tableau• Qlikview/Qlik Sense

• Data analytics• Mindbridge• Treasure data• NICE• Periscope• Zoho• DXC• Sisense Data analytics• Inflow• AWS

• Specialized auditing software• ACE, IDEA, Arbytus, SAS

28 Insight. Oversight. Foresight. SM

Data Analytic Team’sRoles and Responsibilities

29

Financial Institutions Group

Key Roles and Responsibilities: Internal Audit

Splitting the analytics roles - essential ingredients…

1. Audit management & staff Provides comprehensive understanding of the audit objectives Identifies opportunities to introduce data analytics into the audit

process Drives demand through personal insights and relationships Keeps focus on solving audit related issues

30

Financial Institutions Group

Key Roles and Responsibilities: Internal Audit2. Data analytics SME

Proficient in use of DA tools and is able to design queries and manipulate data easily Experienced auditor with a knack for analysis May have knowledge of advanced statistical topics and modeling Excellent problem solving skills

31

Financial Institutions Group

Key Roles and Responsibilities: Internal Audit3. Data specialist

Strong programming and coding proficiency Has been a database administrator or systems analyst Has spent time as developer and has built applications Expertise in core IT related functions in querying, data extraction,

cleansing, and manipulation

32 Insight. Oversight. Foresight. SM

Data Analytic Applications for Internal Audit

33 Insight. Oversight. Foresight. SM

Practical Examples

34

Financial Institutions Group

Data Analytics Applied to Accounts Payable

• AP tests can be designed to address risks, cost savings and/or recoveries

• Data analytic tests can be designed to identify any of the following:• Improper disbursements• Duplicate payments• Unapproved purchases• Payments for items not received• Payments in excess of approval levels• Missed discounts or credits

35

Financial Institutions GroupAP Analysis: A Few Ideas

• Improper payments or questionable disbursements• Detect duplicate payments using dates, payees, vendor invoice numbers and

amounts*• Identify invoices or payments to vendors without a valid purchase order*• Look for invoices from vendors not in approved vendor file• Find invoices for more than one purchase order authorization*• Identify multiple invoices with the same item description*• Extract vendors with duplicate invoice numbers*• Look for multiple invoices for the same amount on the same date*• Find invoice payments issued on non-business days (Saturdays and Sundays)• Identify multiple invoices at or just under approval cut-off levels• Identify credits issued by or outstanding with vendors• Identify goods invoiced and paid, but not shown as being received• Look for payments to vendors not on contract.

* signifies potential for recoveries

36

Financial Institutions GroupAP Analysis: A Few More Ideas

• Look for multiple payments to the same vendor on the same date or for the same amount (excluding recurring charges, such as rent)*

• Stratify vendor balances, check amounts, invoice amounts, PO amounts, etc., for unusual trends or exceptions*

• Calculate and validate annualized unit price changes in PO/payments for the same product over time*

• Review sequence of check numbers for gaps• Identify payments where no discount was taken*• Review changes to the vendor master file

* signifies potential for recoveries

37

Financial Institutions GroupAP Analysis: A Few More Ideas

• Look for multiple payments to the same vendor on the same date or for the same amount (excluding recurring charges, such as rent)*

• Stratify vendor balances, check amounts, invoice amounts, PO amounts, etc., for unusual trends or exceptions*

• Calculate and validate annualized unit price changes in PO/payments for the same product over time*

• Review sequence of check numbers for gaps• Identify payments where no discount was taken*• Review changes to the vendor master file

* signifies potential for recoveries

38

Financial Institutions GroupAccounts Payable Schemes

• Phantom vendors• Match names, addresses, phone numbers, bank accounts

and taxpayer identification numbers between vendor source documents.

• Verify existence of vendors using a PO Box for an address

39

Financial Institutions GroupAccounts Payable Schemes

• Kickback or conflict-of-interest• Vendor prices greater than standard• Price increases greater than acceptable percentages• Continued purchases in spite of high rates of returns, rejects,

or credits• High volume purchases from one vendor• Frequent change orders• Identify payments to vendors with same names, addresses,

phone numbers, etc., as employees

40

Financial Institutions GroupAccounts Payable Schemes

• Bidding and contracting• Patterns of rotation among vendors• Bids that are exceptionally lower than those of other vendors• Low winning bids followed by numerous change orders• Excessive use of one contractor in a competitive field• Patterns in awards to vendors• Identical bids• Multilateral drops in bid prices (accompanied by the entry of new

competitor)• Competitors with the same addresses, principals, sales agents, phone

numbers, etc.• Vendors with same names, addresses, phone numbers, etc., as

employees.

41

Financial Institutions GroupOther Applications for Data Analytics

• Accounts Receivable• Valid sales orders• Accurate product pricing• Authorized shipments• Proper invoicing• Valid cash receipts• Timely collections & write-offs• Sales contract compliance• Other adjustments

• Payroll• Accurate & authorized payments• Timely & accurate hires &

terminations• Reasonable OT & commissions• Proper timekeeping & attendance• Search for non-existent

employees and other payroll schemes

• Comparison of periods for unusual trends

42

Financial Institutions GroupOther Applications for Data Analytics

• General ledger• Journal entries• Closing activities• Adjustments

• Master files• Members• Loans

• Travel and entertainment• Purchasing cards

• Data quality• Reasonable• Within expected range• Validity• Complete

• Compliance• BSA/AML• Money service business

43

Financial Institutions Group

Rules Based View of Audit Procedures (Legacy CAATs)

A legacy but still common practice encourages the use of audit testing tools where each test is done one-by-one. Each test is performed in isolation and then examined by the auditor to look for issues.

This focuses the auditor on specific issues and helps to verify the presence of controls and good accounting practices.

These techniques increase the odds of finding anomalies. But this can be improved

44

Financial Institutions GroupRules Based Tests

Typical audit test pattern: Tests performed one by one and human auditor inspects results

45

Financial Institutions GroupHuman Hypothesis Generation

What can I say about these transactions? What else is interesting?

A Human Centric View of

Analysis

46

Financial Institutions GroupRules Based Testing: Combinations

By considering all tests together and viewing a combined result score, you can see higher risks floating to the top which fail more tests.

More Risk Factors Fewer Risk Factors

Ranked Interest Score

Conclusion: Combining the test outcomes and understanding the intersect produces better results when trying to understand transaction risk.

47

Financial Institutions GroupA Real World Scenario: Rules Combined

Rules-based testing can create wide (large) buckets of transactions

48

Financial Institutions GroupA Real World Scenario: Machine Learning

Machine learning techniques create much better differentiation of transaction risk factors.

Ranked Risk Factor Scores Histogram of Risk Factor Scores

Good distribution and separation enables appropriate focus and use of time

Smooth range of risk scores, no large buckets

49

Financial Institutions Group

Quick Comparison: Machine Learning vs. Rules

Conclusion: Machine learning is proving a better tool to understand risk factors and it produces better outcomes than rules based approaches alone.

In an example scenario “Unusual Cash Disbursements”: • Rules: flagged a transaction as normal putting it in the 30th percentile of risk • Machine Learning: flagged the same transaction in the 3rd percentile of risk

50

Financial Institutions GroupMachine Learning: Rare Flows

Characteristics of potentially inappropriate journal entries• Made to unrelated, unusual (e.g., unusual combinations of

debits and credits), or seldom-used accounts

Audit Rationale

51

Financial Institutions GroupMachine Learning: Rare Flows

Algorithm Finds: Unusual Journal Entries based on frequency of account interactions

For each and every transaction (100%), measure how often they occur and flag rare interactions

Algorithm Class: Unsupervised Machine Learning

52

Questions?

53

Financial Institutions GroupReferences

ACFE (2018). Report to the Nations. Retrieved 4/30/2019: https://www.acfe.com/report-to-the-nations/2018/

AICPA (2018). Beyond robotics: How AI can help improve audit process. Retrieved: https://blog.aicpa.org/2018/08/beyond-robotics-how-ai-can-help-improve-the-audit-process.html#sthash.3Lzmii1i.uAOqhSbF.dpbs

AICPA (2019). A CPA’s Introduction to AI: From Algorithms to Deep Learning, What you Need to Know. Retrieved:

ARRIA (2018). Why the Finance Industry is Ripe for AI Disruption. Retrieved: http://blog.arria.com/why-the-finance-industry-is-ripe-for-ai-disruption

https://www.aicpa.org/content/dam/aicpa/interestareas/frc/assuranceadvisoryservices/downloadabledocuments/cpas-introduction-to-ai-from-algorithms.pdf

BBC (2018). Carillon: The audit industry’s existential question. Retrieved: https://www.bbc.com/news/business-44201251

Change.org(2019). Change Oxford English Dictionary’s Archaic Definition Of The Word ‘Accountant'. Retrieved: https://www.change.org/p/oxford-english-dictionary-change-oxford-english-dictionary-s-archaic-definition-of-an-accountant?utm_source=twitter&utm_medium=social&utm_campaign=change_oed

CNBC (2019). 40% of A.I. start-ups in Europe have almost nothing to do with A.I., research finds. Retrieved: https://www.cnbc.com/2019/03/06/40-percent-of-ai-start-ups-in-europe-not-related-to-ai-mmc-report.html

54

Financial Institutions GroupReferences

EY (2018). How artificial intelligence will transform the audit. Retrieved: https://www.ey.com/en_gl/assurance/how-artificial-intelligence-will-transform-the-audit

Financial Times (2018). PwC’s failure to spot Colonial fraud spells trouble for auditors. Retrieved: https://www.ft.com/content/c2cc45d6-f1f6-11e7-b220-857e26d1aca4

Forbes (2016). How Artificial Intelligence Will Transform The Delivery Of Legal Services. Retrieved: https://www.forbes.com/sites/markcohen1/2016/09/06/artificial-intelligence-and-legal-delivery/#20c428cf22cd

Forbes (2018). How Much Data Do We Create Every Day? The Mind – Blowing Stats Everyone Should Read. Retrieved: https://www.forbes.com/sites/insights-kpmg/2018/10/19/artificial-intelligence-real-breakthroughs-the-practice-and-promise-of-ai-in-auditing/#1aee86126c10https://www.forbes.com/sites/bernardmarr/2018/05/21/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read/#7ef6f8cf60ba

Forbes (2018b). Artificial Intelligence, Real Breakthroughs: The Practice And Promise Of AI In Auditing. Retrieved:

Huffpost (2015). Disrupting Today’s Healthcare System. Retrieved: https://www.huffpost.com/entry/disrupting-todays-healthc_b_8512200

55

Financial Institutions GroupReferences

• Journal of Accountancy (2011). Special Focus Report: Going Paperless. Retrieved: https://www.journalofaccountancy.com/news/2011/jul/july2011goingpaperless.html

• Markets and Markets (2019). Artificial Intelligence Market by Offering (Hardware, Software, Services), Technology (Machine Learning, Natural Language Processing, Context-Aware Computing, Computer Vison), End-User Industry, and Geography –Global Forecast to 2025. Retrieved: https://www.marketsandmarkets.com/Market-Reports/artificial-intelligence-market-74851580.html

• Riedl,Mark (2017). Human-Centered Artificial Intelligence. Retrieved: https://medium.com/@mark_riedl/human-centered-artificial-intelligence-70b019f956d1

• Spangler, William et al. (pending publication). Educating Business Students for the Age of Intelligent Machines: A Framework for On-Line AI-enabled Learning.

• TechTerms (no date). Data. Retrieved 4/30/2019: https://techterms.com/definition/data• The CPA Journal (2017). Big Data in Business Analytics: Implications for the Audit Profession. Retrieved:

https://www.cpajournal.com/2017/06/26/big-data-business-analytics-implications-audit-profession/• The Globe and Mail (2017). Supreme Court says Livent auditors liable but sets conditions. Retrieved:

https://www.theglobeandmail.com/report-on-business/supreme-court-says-livent-auditors-liable-but-sets-conditions/article37393018/

• The Guardian (2017). BT loses almost £8bn in value as Italy accounting scandal deepens. Retrieved: https://www.theguardian.com/business/2017/jan/24/bt-loses-7bn-in-value-as-italian-accounting-scandal-deepens

• The Guardian (2018). KPMG to fine staff £100 for late time sheet. Retrieved: https://www.theguardian.com/business/2018/dec/20/kpmg-to-fine-staff-for-late-time-sheets

• Wikipedia (no date). Optical character recognition: Retrieved: https://en.wikipedia.org/wiki/Optical_character_recognition• Wikipdedia (no date). Cloud computing. Retrieved: https://en.wikipedia.org/wiki/Cloud_computing• Wikipedia (no date). Blockchain. Retrieved: https://en.wikipedia.org/wiki/Blockchain• Wikipedia (no date). Big data. Retrieved: https://en.wikipedia.org/wiki/Big_data• Wikipedia (no date). Machine Learning. Retrieved: https://en.wikipedia.org/wiki/Machine_learning• World Economic Forum (2018). The Future of Jobs Report 2018. Retrieved: https://www.weforum.org/reports/the-future-of-jobs-

report-2018

56 Insight. Oversight. Foresight. SM

Thank You!

Michigan Texas Florida North Carolina

Robin D. Hoag, CPA, CGMA, CMCShareholder, Financial Institutions GroupOffice: (248) 244-3242Cell: (248) 709-1270hoag@doeren.com