Continuous audit: today and tomorrow

Continuous audit:today and tomorrow

Miklos A. VasarhelyiKPMG Professor – Rutgers UniversitySenior Consultant- AT&T Laboratories

2

Continuous Audit and Reporting Laboratory

Outline

• An evolving framework• Some Key issues / the state of the art• Some CARLAB experiences• Six Steps in Implementing CA• Organizational Context• Opportunities and Challenges• Conclusions

An evolving audit framework

4


An evolving audit framework

Assurance ofData elements

Data level

Assurance

Assurance ofKey Processes

Process level

Assurance

Assurance ofReports

Report level

Assurance

•XML/ XBRL datum•Generated and modified by different processes•Balkanization of data•Control / Assurance tags

•Process reviews a la Systrust•Internal or outsourced•Third party processes are to become the norm•Intra and Inter process controls an issue

•Compliance reports becoming commonplace•Traditional audit is an instance of RLA•Generated and modified by different processes

5


An evolving continuous auditframework

•Automation

•Sensoring

•ERP

•E-Commerce

Continuous

Audit

ContinuousControl

Monitoring

Continuous

Audit

Data

CA = CCM+ C(D)ACA -> Continuous AuditCCM -> Continuous Control MonitoringC(D)A -> Continuous Data Assurance

6


Some Key Issues

• Two recent surveys (ACL and PWC) show that a large number of key companies are attempting to perform continuous audit like functions

• An industry of software is evolving with ACL, IDEA, APPROVA, and others growing rapidly

• Control Monitoring and Continuous Data Assurance are the main approaches

• The first recorded application was AT&T Bell Laboratories CPAS effort in the 1986-1991 period

• The Rutgers CarLab is working in leading applications

7


Continuous Auditing Value Proposition•

– Improved business performance

• Innovations in information technology & analytical modelling enable:

– More frequent, timely, accurate & relevant business performance information

– Lower compliance risk

– Cost reduction

8


CAR-Lab Experiences• Control monitoring at Siemens

• Transaction monitoring at Unibanco

• Continuous (data) assurance at HCA

• Other

– Conceptual developments

– Simulating Liberty

– EBR work

– KPMG projects

Overview of CaR-Lab examples

10


Siemens' – Project Value Proposition

Operational Audit Why CA at Siemens?

•Improve Governance (Fraud Detection,

SOX Compliance, Monitoring, etc)

•Reduce Compliance Costs

•Improve skill level and quality of work life

for auditing and compliance Associates

•Move closer to real time reporting

capabilities

•ETC….

Operational Audit

Value Proposition“Value = Quality + Cost”

COST:

• Consider a large multinational corporation with 400 auditors (internal & external), each with a fully absorbed (sal./fee, benefits, travel, etc.) $200,000/yr cost for a total annual compliance cost of $80 million dollars. Assume further that the proposed continuous auditing model cost $1 million dollars to develop and implement and only reduced manual compliance effort by 25% in the firm. The annual net estimated savings or cost avoidance of this project for the firm defined above would be:

$19 Million dollars (Or nearly $100 million dollars over 5 years)!

Note: Leverage the model further by increasing the percentage of impact or in support of other assurance or monitoring functions and the value proposition grows.

Expanded Audit Coverage

Significant Cost Savings

Automated Business Process Controls Monitoring Project

11


Siemens' – Project Features

• Formalize & automate internal audit procedures used for business process controls monitoring

• Conduct “man vs. model” assessments

• Calibrate “exception rules” to optimize model performance

• Scale up to all SAP instances

• Increase frequency of model application, where feasible

• Transition to Approva application and extend the model where optimal

12


A 3 pronged approach to audit automation• Automate audit plan using delivered Rule

Sets: Est 25% of a typical manual audit plan

• Automate using external data sets (Static & Variable): Est an additional 25% a typical manual audit plan

• Re-enginer manual controls into automated controls with improved control precision: Est an additional 25% a typical manual audit plan• Total = Automation Opportunity ~75%!!

13


MCP

A.A.S(audit ActionItems)

From SiemensApprovaand otherliterature

Class of

Auditable

Actions

----

of Audit

Processes

AuditEvidence

Receptacle

MasterAudit

Program AuditParameterization

Tool

OtherStatic

Parameters

EvergreenOpinion

InferenceEngine

AuditorManagementOperating Alarm Flows

Operating Alarm Flows

CAControl

Dashboard

Deter-ministic

Stocha-stic

ExternalTable

comparisons

Snapshotcomparisons Other

RemoteAudit

Communic.Tool

DataExtraction

InteractiveMail

ManagementTool

SustainableObject

VerificationTool

Other

14


IT / IA Continuous Auditing Program at Unibanco

Continuous Audit at our Bank Mission

Automatically evaluate risks and controls on a continuous basis in order to identify exceptions and anomalies, trends and risk indicators.

I ssue opinions about controls, risk assessment for top management, audit committee and other interested parties. Contribute to corporate Governance of the Conglomerate.

Scope

All products, processes and services in the conglomerate that allow for the systemic extraction and analysis of data generated by Information Technology.

Approach

Extraction and analysis of information of the products, processes and services that exist in the Conglomerate using IT to improve timeliness and scope of Internal audit work..

Inform events of non-compliance in these being effective on the generation of products needed for the prevention and correction of risks and events.

PRODUTIVITY WITH QUALITY AND EFFICIENCY

15


Unibanco – Some CA Program Features

• Automated monitoring of over 5 million customer accounts on a daily basis using 25 automated procedures to:– Detect errors– Deter inappropriate events & behaviors– Reduce or avoid financial losses– Help assure compliance with existing laws, policies, norms

and procedures

• Examples of “low hanging fruit:”– Customer advances– Excess over credit limit– Returned checks– Federal tax payment cancellations– TED emissions (should this be omissions?)

16


Unibanco – Advances to Clients Monitoring

17


Continuous Data Assurance (CDA) at a Major Health Services Provides (HSP)• HSP is a large national provider of healthcare services,

composed of locally managed facilities that include numerous hospitals and outpatient surgery centers.

• IT internal audit provided access to unfiltered extracts from their transactional databases, comprising all procurement cycle daily transactions from October 1st, 2003 through June 30th, 2004: Over 500,000 data points.

• Dataset mimics what a CDA system has to deal with: highly disaggregate data flowing through CA system in real time.

• Audit procedures have to be developed for this environment.

18


Analytical Procedures in CA• Analytical procedures used in the planning, substantive testing, and

reviewing stages of an audit. We focus on substantive testing.

• In conventional auditing first apply analytical procedures to identify potential problems, Then, focus detailed transaction testing on the identified problem areas.

• In CDA the sequence is reversed:

1. Use automated general transaction tests to all the transactions and filter out identified exceptions for resolution.

2. Apply automated analytical procedures to the filtered transaction stream to identify unforeseen problems.

3. Alarm humans to investigate anomalies.

19


Continuous Data Assurance• Automation of Transaction Testing:

– Formalization of business process rules as transaction integrity and validity constraints.

– Verification of transaction integrity and validity detection of exceptions generation of alarms.

• Automation of Analytical Procedures:

– Selection of critical business process metrics and development of stable business flow (continuity) equations.

– Monitoring of continuity equation residuals detection of anomalies generation of alarms.

20


Enterprise System Landscape

Ordering

Accounts Payable

Materials Management

Sales

Accounts Receivable Human Resources

Business Data Warehouse

Automatic Transaction Verification

Exception Alarms

Automatic Analytical Monitoring: Continuity Equations

Anomaly Alarms

Continuous Data Assurance System

Responsible Enterprise Personnel

21


Establishing Data Integrity: A Procurement Example

• Referential integrity along the business cycle and identification of completed cycles:P.O. Shipment receipt voucher payment.

• Identification of data consistency issues and automatic alarms to resolve exceptions:– Changes in purchase order vendor numbers;– Discrepancies between the totals and the sums of line

items;– Discrepancies between matched voucher amounts.

22


Detection of Exceptions• Referential integrity violations

– PO without matching requisition– Received item without matching PO– Payments without matching received items

• Data integrity violations– PO has zero order quantity– Received item has negative quantity– Invalid payment check numbers (e.g. All 0s)– Gross payment amount is smaller than net payment

amount

23


Continuity Equation Based CDA• Continuity Equations:

– Stable probabilistic models of highly disaggregated business processes, uses as the expectation models for process based analytical procedures.

– Originated in physical sciences (various conservation laws: e.g. mass, momentum, charge).

• Continuity equations are developed using statistical methodologies of:1. Linear regression modeling (LRM);2. Simultaneous equation modeling (SEM);3. Multivariate time series modeling (MTSM) using various

Vector Autoregressive Models (VAR).

24


Basic Procurement Cycle

P.O.(t1)

Receive(t2)

Voucher(t3)

t2-t1

t3-t2

25


Ideal Continuity Equations of Basic Procurement Cycle

Receive(t2)= P.O.(t1)

Voucher(t3)= Receive(t2)

• Aren’t partial deliveries allowed?

• Are all orders delivered after exactly the same time lag?

• Are there any feedback loops?

26


P.O.(t)= 0.24*P.O.(t-4) + 0.25*P.O.(t-14)+ 0.56*Receive(t-15) + εPO

Receive(t)= 0.26*P.O.(t-4) + 0.21*P.O.(t-6)+ 0.60*Voucher(t-10) + εR

Voucher(t)=0.54*Receive(t-1) - 0.17*P.O.(t-9) + 0.22*P.O.(t-17) + 0.24*Receive(t-17) + εV

Estimated Continuity Equations of Procurement Using VAR Model

27


Detection of Anomalies

• Anomalies are detected if:– Observed P.O.(t) < Predicted P.O.(t) - Var

or– Observed P.O.(t) > Predicted P.O.(t) + Var

• Similarly for:– Receive(t)– Voucher(t)

• Var = acceptable threshold of variance.• If there is anomaly generate alarm!

28


Measuring Anomaly Detection• False positive error (false alarm, Type I error): A non-

anomaly mistakenly detected by the model as an anomaly. Decreases efficiency.

• False negative error (Type II error): An anomaly failed to be detected by the model. Decreases effectiveness.

• Detection rate is used for clear presentation purpose: The rate of successful detection of seeded errors.

• A good analytical model is expected to have good anomaly detection capability: low false negative error rate (i.e. high detection rate) and low false positive error rate.

29


Simulated Error Correction• Access to highly disaggregate data in real time makes it

possible for CA system to detect, investigate and correct anomalies also in (nearly) real-time.

• Real-time error correction enables utilizing the corrected rather than the erroneous data in revised continuity equation benchmarks.

• Real-time error correction is likely to benefit future anomaly detection. We investigate the magnitude of this benefit using simulation.

• Error correction raises important issues about auditor independence, and the line between auditing and monitoring of business processes.

30


Benefit of Real-time Error Correction: MTSM

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

10%E 50%E 100%E 200%E 400%E

MTSM_Error_Correction MTSM_No_Error_Correction

32


Takeaways from HSP Study• Various statistical methods can be used to derive expectation

models of acceptable quality.

• But key is access to highly disaggregate data, not which benchmark is used. With such data, most reasonable continuity equation models give usable results.

• Real-time error correction significantly improves error detection.

• More disaggregated models are not always better: weekly data can be more stable than the daily one.

• Alarms have to be managed – trade-off between Type I and Type II errors.

Implementation Issues in CA

34


• Background – While technologies of continuous audit

have been extensively discussed and are progressively emerging the more mundane issues of their implementation in a socio-technical environment have been neglected

– http://www.theiia.org/itaudit/features/in-depth-features-2-10-08/feature-2/

35


2. Rule

5. Follow-up

1. PriorityAreas

6. Action andReaction

4. Parameterization

3. FrequencyAudit Control Panel

Six steps of process implementation

36


– 1. Identification of Priority Areas • Modularize risk areas, rate these risks and

evaluate the cost x benefits• Identify the basic audit objects• Choose critical business processes that will

be the focus of continuous audit (low hanging fruit)

• Identify key data in for the implementation of Continuous Audit in the mapped processes

• Political Considerations

37


• Key Objective of Audit Procedure – Detective– Deterrent– Financial– Compliance

38


• 2. Rules of Monitoring and Auditing – Once an area of CA is chosen the “rules” of

monitoring, alarming, and assurance must be established

– These must take into consideration the legal and environmental issues as well as the objectives of the particular process

– The CA process is established adopting certain rules, frequencies, and parameters.

– e.g. we will monitor bank accounts in overdrafts or in excess limits

39


•3. Frequency – The natural rhythm of the process

• Timing of computer processes• Timing of business processes

– Cost benefit considerations– Nature of procedure objectives

• Deterrence• Prevention

40


– 4. Parameterization•Define parameter to analyze in

accordance with the risk•eg.: Monitoring all accounts in

overdrafts in daily basis , that have a balance of debt 20% larger than its limit of loan and bigger than 1000 USD

41


• 5. Follow-up– Who will receive the alarm?

• Management?• Audit leadership?• Immediate superior of the responsible for the data• The timing of the follow up

– Pass the alarm along immediately– Reconcile the alarm prior to follow up– Wait for 3 sequential days of similar alarms to follow

up• Escalation guidelines

– E.g. after three days send to the immediate superior’s superior or wait for 3 days prior to the re-escalation

42


• 6. Action and Reaction– Guidelines for dealing with

auditees•Lack of bias•Consistency of response•Guidelines for individual factor

considerations•Concern with collusion

Organizational Issues

44


• Organizational Structure for CA– Is CA a part of the audit function or of

management?– Its part of the audit function– Should there be a separate continuous

audit group?– Yes, to facilitate its implementation

progressively in the many areas of continuous audit

45


• Workforce Effects – Progressively labor requirements

for the traditional audits supported by CA will reduce and deeper audit will become possible

– Rebalancing of workforces– High technological competencies

needed

Opportunities and Challenges

47


Opportunities for business and research (1)• Control system measurement

– We are in a pre-paradigmatic stage of control documentation and measurement

– We do not know how to monitor controls in large ERPs– We do not know how to provide a really supportable opinion on

controls – We do not know how to rate combinations of controls

48


Opportunities (2)

• Business Process Monitoring and Alarming– Auditors have to carve a position on the new monitoring and

control environment– Auditors can collect exception “alarms” as trusted parties and

incorporate these into evidentiary matter– Auditors can be “trusted”

49


Opportunities (3)

• Automatic Confirmation Tools– Confirmations will have an increased evidentiary role with

eventual elimination of population and integrity worries– Intelligent confirmatory tags can do much– Database to database hand-shaking will be medium– Business opportunity for auditors

50


Opportunities (4)

• Audit bots (agents)– Many of the basic audit functions can be emulated by software– These must be eventually developed by the profession to work

hand-in-hand with human auditors in the new audit world– These agents will work on all areas including: 1) audit

planning, 2) analytical reviews, 4) confirmations, and )5 evergreen opinions

51


Opportunities (5)

• Collecting forensic trails– Auditor “black” box

• Publishing real-time authenticated reports for different compliance masters

• Publishing FD independent compliance reports

52


Challenges

• Standards are needed for CA– Audit monitoring needs to be defined– Types of evidence are to change and must be reconsidered– Independence needs to be re-defined

• The billing model has to be restructured to bill on function not hours

53


Challenges

• Audit firms must put improved knowledge collection and management processes to feed their audit analytic toolkit

• Audit firms have to engage in auditor automation and pro-actively promote corporate data collection during-the-process

• Value added must be justified in terms of data quality

54


• Conclusions – Attention must be paid to the

organizational processes that implement continuous audit

– There are 6 key steps to progressively implement a CA program module by module

– The CA process is dynamic and CA management will change schedule and parameters of each process

– The organization of the audit process must be evolved progressively

Documents

Continuous audit: today and tomorrow