35
1 Harnessing the Power of Innovation Marc Sabino Chief Auditor Head of IA Innovation, Citigroup Robert Mullen Director IA Consumer Analytics and Reporting, Citigroup October 28 th , 2019

Harnessing the Power of Innovation

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Harnessing the Power of Innovation

1

Harnessing the Power of Innovation

Marc Sabino

Chief Auditor – Head of IA Innovation, Citigroup

Robert Mullen

Director – IA Consumer Analytics and Reporting, Citigroup

October 28th, 2019

Page 2: Harnessing the Power of Innovation

2

Participating in the Polling SessionDownload the IIA Events App

How to Vote:

• Download the app on your device:

– “IIA EVENTS” in App Store /

Google Play

– Login with the email you

registered with

– Password is diiasc19 (case

sensitive)

• Open “VOTING” from the menu,

select this session

• Click “VOTE NOW”

Page 3: Harnessing the Power of Innovation

3

Citi-Sponsored Social Media EngagementDownload the Snapchat App

SHARE

SNAP

SAVE

Remember to enable your location

settings to see our fun Citi IA filter

Hit the down arrow and save your snap

Email your photos to [email protected]

and tag us using the hashtag #LifeAtCiti for

a chance to be featured on @CitiCareers

#LifeatCiti@CitiCareers

Page 4: Harnessing the Power of Innovation

4

IntroductionMarc Sabino & Robert Mullen

Robert MullenDirector, IA Consumer Analytics and Reporting, Citigroup

Marc SabinoChief Auditor, Innovation, Citigroup

Page 5: Harnessing the Power of Innovation

5

Polling Question

What is the size of your audit organization?

Some sizes you could submit are:

• 0 – 50 employees

• 51 – 100 employees

• 101 – 500 employees

• 500+ employees

Join the conference poll in order to submit your answer!

Page 6: Harnessing the Power of Innovation

6

Polling Question

My organization receives support from senior

management to expand data analytics capabilities.

Please select from the below:

• Strongly Agree

• Agree

• Slightly Disagree

• Disagree

Join the conference poll in order to submit your answer!

Page 7: Harnessing the Power of Innovation

7

Polling Question

What technology do you use the most in

implementing your testing?

Please select from below:

• Excel, VBA, and Kutools

• SQL, SAS, and other DB Management Languages

• Off the shelf analytics tool (e.g. ACL, Alteryx, etc.)

• Robotic Process Automation, via Automation Anywhere or other languages

• Python for HTML interactions

• Machine Learning Via Python, Tensorflow, or other languages

• Artificial Intelligence Via Python, Prolog, Java, or other languages

Join the conference poll in order to submit your answer!

Page 8: Harnessing the Power of Innovation

8

How is Citi responding to new

methods and structures with

strategic insights on risk and

the control environment?

Page 9: Harnessing the Power of Innovation

9

May 7, 2019

Introduction to Audit InnovationOur Mission

Smarter Auditing – significantly improve the control environment and assurance through

large population testing, anomaly detection, and new techniques

Advanced

Analytics

Integrated

Solutions

Alerting

& Insights

Natural

Language

Processing

Machine

Learning

Augmented

Intelligence

Audit of the

FutureInsightful Risk Assessment

Smarter Testing

❑Deeper testing through automation and full population

testing

❑Use of multiple solutions to help identify thematic

patterns across countries and entities

Improving the Auditor Journey

❑Using technology and innovation to support the auditor

of the future

❑Sharing subject matter expertise, yielding insights

and popularizing a data driven mindset

❑Identifying emerging risks and issues to drive insights

and connect the dots

❑Using data and integrated platforms to uncover hidden

patterns and develop new hypotheses

Page 10: Harnessing the Power of Innovation

10

How do the degrees of

innovation deliver greater

assurance, build a premier

function, and help auditors

identify hidden patterns and

emerging risks?

Page 11: Harnessing the Power of Innovation

11

Impact of Innovation

When auditors spend their time on highly repetitive and mundane tasks, it reduces the amount of time that they can

dedicate to stakeholders or high value-add activities.

❑Reduction of error rates and added assurance by

testing full population

❑Auditors are more empowered from the utilization and

exploration of data, augmenting their decision-making

and creating the auditor of the future

❑Auditors can identify emerging risks and issues to

connect the dots and drive strategic conversations

with stakeholders

❑Real time escalation of thematic issues

❑Widespread use of data helps gain an understanding

and strengthening of entire control environment,

providing thematic insight

❑Timely identification of issue themes and root causes

enhances business controls

Benefits for the Auditor Benefits for the Stakeholder

Page 12: Harnessing the Power of Innovation

12

Impact of Innovation

❑To demonstrate that Internal Audit is a positive change agent for their entire organization in regards to Innovation

❑That the new tools and techniques leveraged to enhance their core businesses are also being leveraged within Internal Audit to further enhance the control environment

❑Assurance that Internal Audit has the subject matter expertise to audit new and emerging analytical tools and technology

❑Reduced risk of non-compliance to regulatory

requirements

❑Increased depth of coverage using full populations

❑Increased frequency of coverage from continuous

auditing

Benefits for the Board Regulatory Agencies

Page 13: Harnessing the Power of Innovation

13

What does the average

auditor’s journey look like for

adopting and utilizing

analytics in audit work?

Page 14: Harnessing the Power of Innovation

14

Our Journey from Audit Analytics to Advanced Innovation

Ch

ara

cte

risti

cs

Ma

turi

ty

Innovative Approach

Advanced

Transformation

Development and Global

use of Repeatable Analytics

Leading

Self-Service / Web-enabled

Auditing Solutions

Established

Develop Audit Innovation

Vision and Strategy

Smarter Assurance Greater InsightsAdded Capacity

All Inclusive

Encompassing

Time

The

Innovation Journey

Overview

IA Innovation embraces

transformative change – moving

beyond our original focus on testing

to encompassing and transforming

many aspects of the full audit cycle.

Our Innovation journey

demonstrates our continuous

improvement process.

Full Audit Cycle

Page 15: Harnessing the Power of Innovation

15

What leading tools and

techniques can be

utilized for analytics?

Page 16: Harnessing the Power of Innovation

16

May 7, 2019

Degrees of Innovation

Levels of Innovation Sophistication

CAATTS and

Audit Analytics

a b c

a b c

a b c

a b 9w e

z6

Scripting of scripts

Linear taskOrchestrated

activities

Read

e-mailCopy-paste Enter data

Send

confirmation

Non-

standard

Machine Learning & Natural

Language Processing

Contextual

Inference

Process Dynamic

a

b ¿?

c

d

¿?

Understand

different process

Convert

images to text

Understand

and learns

Systems Self Aware

Cognitive / AI

PredictiveSelf

Learning

Self

Healing

Predicts future

behaviors

Virtual

Assistants

Auto

Feedback

❑ Process of automating repetitive tasks

❑ Implementing rules and logic

❑ Productionizing the scripts for repetition

❑ Analysis of past and present data sources

to make algorithmic predictions

❑ Using thematic analysis to identify emerging

risks and regulatory policy matching

❑Machine programmed to think, work,

evolve and react like humans

❑ Helping humans become faster and

smarter at the tasks they're performing

Innovation covers a spectrum of technologies, including VBA, SAS, RPA, NLP, and Artificial Intelligence

Page 17: Harnessing the Power of Innovation

17

Analytical Tools used in Audits and Issue ValidtionIA Consumer – Highlights

IA Consumer leverages multiple tools in audits and issues validation to test the completeness and accuracy of a

control or data set back to source systems

Powerful and Flexible Tools

❑SAS Enterprise Guide, Microsoft SQL Server, Oracle SQL Developer are powerful tools that enable auditors to

perform both one time and repeatable validations of nearly any control or issue remediation that uses data

❑The tools are flexible, providing auditors the capability to analyze/test in both direct an indirect ways

❑Any value add analysis could become a continuous audit and placed on a scheduler

Impact

❑Entirely independent validation from source to final product

❑Provides the power and flexibility auditors need to effectively test the complex and data extensive

businesses we oversee

❑Increases the effectiveness of the assurance provided by Internal Audit

Traditional Approach

❑Manual spreadsheet review of data

populations provided by the business

(limited independence)

❑Non-statistical sample review to host

systems (e.g. document review)

❑You don’t know what you can’t see

Innovative Approach

❑Size doesn’t matter, the size of the data has no impact on the

scope of the review

❑Provides the capability to review the entire process/data flow

for gaps

❑Enhanced completeness and accuracy testing using

templates, scripts, and programs

Page 18: Harnessing the Power of Innovation

18

Analytical Tools used in Audits and Issue ValidtionIA Consumer – Regulatory Issue Validation

IA Consumer works with many regulators across the globe. In the US, they include, among others, the Federal Reserve Bank (FRB),

Office of the Comptroller of Currency (OCC), Consumer Finance Protection Bureau (CFPB), and the Department of Justice (DOJ).

The Challenge

❑How to validate large customer remediation's where complex analytics and waterfalls we’re employed to satisfy

the requirements of an Enforcement Action/Consent Order or Matter Requiring Attention (MRA)

❑How to test deep enough to provide the company sufficient assurance that work performed by the business was

complete and accurate, thus minimizing the chances of a regulatory reopen

Impact

❑Initially, much of the effort resulted in retargets and IA reopens

❑Eventually, the efforts dramatically reduced the regulatory reopen rate, new business procedures, and an

improved control environment

❑Today, the use of analytics for customer remediation's is an expectation for all organizations involved

The Solution

❑Implement a validation model within Internal Audit that leverages analytics from source systems to final

remediation population to increase confidence levels

❑Provide the business new guidelines for documentation analytics

❑Employ an agile engagement model between Internal Audit and the business to increase the likelihood of

success

Page 19: Harnessing the Power of Innovation

19

Analytical Tools used in Audits and Issue ValidtionIA Consumer – Regulatory Issue Validation (cont’d)

Internal Audit’s expectations of the business in regards to documenting analytics

1. Supporting documentation includes the full scope of the analysis, from original data source(s) to finished product. Including:

a. Succinct beginning-to-end documentation for practical review (layman’s terms). Target audience is executive management,

b. Internal Audit and regulators.

c. Comprehensive technical documentation used to perform the analysis.

2. Supporting artifacts (e.g. code/logic) and data repositories should be supplied to provide the capability for a third party to re- perform

the complete analysis.

3. Data sources are documented, including supporting evidence that they are classified as the most appropriate source, or equivalent, for

the intended purpose (e.g. gold source, master files, etc.). Data flow diagrams are preferred for moderate to complex environments.

4. A data dictionary is provided for tables and relevant data fields; especially those used to include or exclude records (e.g. active or

inactive accounts, product codes, etc.).

5. Data profiling or quality review is performed on all data sources. The review should be comprehensive and include supporting

documentation to evidence completeness (e.g. graphs displaying counts of records distributed by relevant time periods).

6. Any data limitations and known gaps are disclosed, including any management decisions to remediate, compensate, or exclude with

supporting rationale.

7. For funnel or waterfall analysis, clearly define each group or bucket categorization. Including:

a. Supporting documentation for calculations.

b. Evidence that all records have been accounted for in aggregate.

8. A qualified checker(s) or reviewer(s) provides credible challenge for each phase of the analysis. Their efforts should be

documented, including any observations and subsequent revisions or versions that were performed.

Page 20: Harnessing the Power of Innovation

20

Robotics in AuditFINRA License Testing Bot

Advanced robotic techniques automate manually intensive tests which are repeatable to increase efficiency

while enhancing audit assurance

Traditional Approach

❑15 minutes to manually check one broker

would take 1 year to test 5500 brokers

❑ 5% error rate and sample size of 25

Robotics

❑Programmable device that can perform tasks and interact with its environment, without the aid

of human interaction

❑Implementation without altering existing tech infrastructure

❑Utilizes automation, machine learning, and cognitive / artificial intelligence

Impact

❑Enhanced assurance through increased sample size

❑100% of broker populations tested in 3 business areas

❑Time Savings through the automation of repetitive manual tasks

Innovative Approach

❑1 minute to check one broker

❑0 Hours used by auditor for testing, allowing

auditor to focus on exceptions

❑100% population testing

Page 21: Harnessing the Power of Innovation

21

Machine Learning and Natural Language Processing in AuditCustomer Complaints

Monitoring of complaints and social media data to identify trends and emerging themes to improve risk

assessment and predictive risk capabilities

Traditional Approach

❑Limited to sampling of internal complaints

❑Reactive vs predictive – utilizing social media

as a leading indicator

❑Requires time consuming manual reading

and categorization of applicable complaints

Machine Learning & Natural Language Processing

❑Sourcing Complaints from social media to enable early risk identification and intervention

❑Differentiate between comments and complaints

❑Categorize complaints into appropriate category

❑Visualize data to identify outlier activity and emerging risks

Impact

❑Leveraging solution to identify areas of increased risk based on customer experience

❑New data points to identify emerging risks and hidden patterns

❑Lending insight that can be used for audit scoping and business monitoring

Innovative Approach

❑Millions of messages analyzed in < 3 hours

❑Consistency in categorization without individual bias

❑Potential leading indicator of formal complaints and other

risk factors

Page 22: Harnessing the Power of Innovation

22

Machine Learning and Natural Language Processing in AuditCustomer Complaints

1 2

1• Look at trends in the number of social media

complaints over time, compared to other channels

such as internal or CFPB

2• Identify complaint categories and easily detect

patterns and anomalies

3• Filter for and select the complaints of interest

• Customized filters possible for specific teams

3 4

4• Perform a thematic analysis of complaints within a

specific business or process

• Customized thematic views possible for specific

teams

565

• Easy, visual filtering capability enabled via word

cloud

6• View and export filtered complaints to Excel for

additional slicing and dicing

Page 23: Harnessing the Power of Innovation

23

Polling Question

When using audit analytics and innovative techniques, where do

you think the strongest impact is to an organization?

Some Areas to consider for your answer are:

• Planning

• Fieldwork

• Testing

• Reporting

Join the conference poll in order to submit your answer!

Page 24: Harnessing the Power of Innovation

24

Robotics, Machine Learning and Natural Language ProcessingEnterprise Platform

Web-based solution execution platform which centralizes Innovation tools for every phase of the audit

life cycle, enabling consistent & globally accessible monitoring, testing, & reporting

Traditional Approach

❑Use of Analytics limited to auditor skillset

❑Analytics performed in siloes

❑Solution execution performed on an ad-hoc

basis

Enterprise Platform

❑Web based solution execution enables global on-demand testing and monitoring

❑One stop shop for all Innovation offerings including Online Solutions, Packaged Solution and

Robotics requests

❑Allows the audit team to leverage analytics and innovation throughout all phases of the audit

life cycle

Impact

❑Cross-functional and utilized by auditors and business leaders

❑Empowers auditors to embrace data driven mindset

❑Reduction of manual documentation and mobile compatibility, facilitating auditor of the future

Innovative Approach

❑Solutions & Bots at the click of a button

❑Customized threshold monitoring & alerting

❑Consistent use of control tests

Page 25: Harnessing the Power of Innovation

25

Enterprise Analytcal ToolsEnd User Computing (EUC) Solution

Enterprise solution that includes prebuilt tests and the capability to run custom reports from data

sourced from the host system

Traditional Approach

❑Individual requests for data from the business

for each audit

❑Single use testing approach

❑Manual review

Enterprise Analytics

❑Highlights overdue reviews, changes to existing EUCs and ratings, and other analysis aligned to

Citi policy and EUC testing plans

❑Common global platform

Impact

❑Uniform testing opportunity across the entire audit universe

❑Time savings using standardized metrics, documentation and reporting sources

Innovative Approach

❑Uniform default tests available to all from a reliable data

source

❑Centralized platform to add/enhance tests

Page 26: Harnessing the Power of Innovation

26

Enterprise Analytical ToolsEnd User Computing (EUC) Solution

Host System Enterprise Analytical Platform

Direct Data Feed

Test Test Name Test Description

EUC01 Required Fields Identify missing (blank/null) fields within EUC portal data.

EUC02 Overdue Reviews This analytic identifies overdue reviews.

EUC03

Upgraded /

Downgraded Risk

Assessments

Identify all risk assessments that have been upgraded or

downgraded the last year.

EUC04 Missing EUCs Identify EUCs removed during the last year.

EUC05 EUC Complexity Alignment of EUC Characteristics with complexity values.

EUC06 Retirement Plan Verify that all High risk EUCs have a retirement plan.

Test Examples

Page 27: Harnessing the Power of Innovation

27

Optical Character Recognition (OCR)Algorithmic Solution

Algorithmic solution that compares affidavits from various sources and detects anomalies across the

various files, leading to efficiencies over traditional sampling approach.

Traditional Approach

❑Reviews are done on a manual basis

❑Comparisons done as sample of 40

❑1 document comparison can take as much as

15 minutes to review

Enterprise Platform

❑Algorithmic solution enables efficient on-demand testing and monitoring of affidavits

❑Individual characters are compared one another and assessed for changes

❑Allows the audit team to leverage analytics to assess larger populations of documents

Impact

❑Cross-functional and ready to expand and scale to other types of documents

❑Enhances auditors ability to only consider true exceptions and leverage continuous monitoring

❑Adds value to audit work, as well as having applicability in regulatory and business uses

Innovative Approach

❑Solutions run quickly and are less prone to error

❑Comparisons can take place across entire document

population

❑Consistent use of control tests

Page 28: Harnessing the Power of Innovation

28

Optical Character Recognition (OCR)Algorithmic Solution

1

21

2

3

3 4

4

56

5

6

Status: If the documents were compared

successfully, the Status will be ‘Completed’.

However, if there was some issue with the

comparison, such as a failure to compare due to

poor quality of the original documents, the status

will be something other than Completed

Difference Detected: If there is one or more

differences found in the documents, the value will

be Yes. Otherwise, if the two documents are

exactly alike, the value will be No.

Content Dissimilarity: Illustrates the magnitude of

the differences, i.e. the more difference, the more

bars

Amount Match: If a change is detected in an

Amount field

Date Match: If a change is detected in a Date

field

View Differences: View the differences in the two

documents side by side (see slide 7 for more

details)

Page 29: Harnessing the Power of Innovation

29

Polling Question

Are there heighted expectations for enhanced IA coverage using analytics

and innovations in your organization?

Please select from the below:

• Strongly Agree

• Agree

• Slightly Disagree

• Disagree

Join the conference poll in order to submit your answer!

Page 30: Harnessing the Power of Innovation

30

What’s Citi approach to

increase the use of

Analytics and Innovation

in Internal Audit?

Page 31: Harnessing the Power of Innovation

31

Innovations Team

Auditors

Auditors

Citi’s Multifaceted Approach to Expand Analytics and Innovation

The multifaceted approach includes both top down and bottom up efforts to leverage the best of both

worlds to increase the development and use of analytics and innovations

Top Down

❑Dedicated Innovation Team

❑Staffed with innovation and analytics skill sets

❑Integrated engagement with audit teams

❑Focused on enterprise or large scale solutions

Impact

❑Leverages the best ideas from all viewpoints and backgrounds, from

technologist/data scientist to auditor

❑Ensures analytical capabilities are not limited to the few, empowering the

organization to increase their capacity at the fastest rate possible

❑Fosters practical and value add solutions that work beyond the test lab

Bottom Up

❑Staffed with audit and analytics skill sets

❑Leverage most appropriate solution or tool depending on audit scope

❑Provide recommendations for future solutions

Page 32: Harnessing the Power of Innovation

32

What are the skills and

considerations for the People,

Processes, and Technology of a

Audit Function of the Future?

Page 33: Harnessing the Power of Innovation

33

Considerations for People, Process, and Techology

Citi is expanding and enhancing all three elements of People, Process, and Technology to harness the

power of innovation, including:

Process

❑Innovation opportunities are identified continuously (culture)

❑Analytics demand for audits and issue validation is identified in advance

❑Ongoing engagement between Innovations and Audit Teams

People

❑Hiring technically skilled innovation staff

❑Providing analytical training to current staff

❑Modifying our new hire profiles to include analytical skills

❑Consider the technology being leveraged by those you audit to streamline future

technical engagements with stakeholders

❑For enterprise solutions, what platform(s) will best serve both current and emerging

technologies

❑Consider the user base of the proposed technology to ensure sufficient long term

support

People Process

Technology

Technology

Page 34: Harnessing the Power of Innovation

34

Questions and Answers

Thank you for joining us today! We will now take questions.

Page 35: Harnessing the Power of Innovation

Thank You!