View
233
Download
12
Category
Preview:
Citation preview
© 2016 High Water AdvisorsConfidential. Do not distribute.
Data Analytics for IT Audit
Data Analytics Overview
ISACA, North Texas Chapter
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 1
About Me
20+ years data analytics experience
‒ Current: Client Solutions Director, High Water Advisors
‒ 2 years developing SAS programs analyzing occupation
information for US Department of Labors’ O*NET system
‒ 8 years with Nortel Networks developing simulations,
using predictive analytics, & managing a DA function
‒ 5 years with federal contractor, RTI International, leading
and performing operations and IT audits
‒ 3 years with ACL Professional Services
» Supported large public & financial sector clients
» SME for SOX-related test automation
Instructor for “Successful Data Analytics” course by the MIS
Training Institute
Local instructor for ISACA CISA exam prep
Jim Tarantino ACDA, CISA, CRISC
jim.tarantino@highwateradvisors.com
+1 855 RISK WATCH
www.linkedin.com/in/jimtarantino
@JimTarantino
www. highwateradvisors.com
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 2
Most IA departments have barely scratched the surface on achieving
the benefits of audit analytics
The challenge is largely a change-management issue, not a
technology issue
‒ Initially, successful analytics are 90% design and 10% technology
» However, technology does increase in importance as the need for
collaboration and sustainability increases
‒ The challenge for most auditors is learning to think differently about audit testing
‒ Poor processes result in the use of analytics stagnating in many organizations
My Beliefs
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 3
What is “data analytics”?
Using technology to enable various stages of data analysis
Current landscape of common technologies used for audit analytics
Business case for data analytics and their application to IT audit
How data analytics can be applied to phases of the IT audit process,
from IT risk assessment, planning, fieldwork, reporting, and follow-up
Key Topics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 4
Simply put:
‒ Breaking relevant problems and questions down into manageable sizes
‒ Using factual evidence to solve problems and deduce answers to questions
An analysis has four key elements:
1. Data / information
2. Logical reasoning / argument
3. Finding / results
4. Lesson / conclusion
Data Analysis
It’s elementary! Auditors
are natural analysts.
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 5
Specially-designed, auditor-friendly, data analysis routines and visualizations, enabling
internal auditors to gather and analyze raw digital evidence from the organization’s
information systems in order to conduct more comprehensive, objective, repeatable
and efficient assessments of risks and controls
Audit Analytics
Digital Evidence + Algorithms + Technology(set of operations)(business data) (understood by computers)
Dear Watson! With this
contraption, auditors can:• Quickly view lots of digital evidence
• Find interesting patterns in the data
• Relate multiple information sources
• Automate their thinking/reasoning
• Make compelling, evidence-based
conclusions
• Route key findings to those who can
do something about it
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 6
Based on… Categories/Types Description
Degree of
standardization &
automation
• Ad hoc
• Repeatable
• Continuous
• Explorative and investigative in nature
• Periodic analysis of processes, often from multiple data sources
• “Always on” — automated, scheduled auditing & monitoring of key processes
Time-orientation of the
question
• Descriptive
• Diagnostic
• Predictive
• Prescriptive
• What happened?
• Why did it happen?
• What will likely happen next?
• What’s the best course of action?
Content focus• Risk (KRIs)
• Performance (KPIs)
• Focused on the likelihood, impact and/or control status of a risk
• Focused on business or operational performance
Data type• Quantitative
• Qualitative
• Analysis of numeric values (e.g., aging, Benford’s Law, stratification)
• Analysis of text values (e.g., keyword matching, sentiment analysis)
Role in the analysis
process
• Data Preparation
• Data Profiling
• Utility
• Outlier
• Exception
• Planned routines to clean and standardize data
• Routines to provide quick visual summaries of data
• Helper routines that are not audit specific (e.g., keyword match)
• Routines that look for extreme cases relative to a group
• Routines that detect violations of a business rule
‘Types’ of Audit Analytics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 7
We deal with big populations (hosts, assets, incidents, ID’s)
Rate of flux in infrastructure, applications, users, regulations, threats
Variety of data sources creates issues
‒ Require special tools and permissions (e.g., Active Directory)
‒ Unstructured data (e.g., event logs, .ini files)
Scanning/testing tools
‒ Create verbose output
‒ Output suited for manual viewing (xml)
‒ Often built for point-in-time (not continuous scans)
Audit Analytics for IT
© 2015 High Water AdvisorsConfidential. Do not distribute without permission. Page: 8
“…it is nearly impossible to
conduct an effective audit
without using technology”
GTAG 16, Page 6
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 9
Enabling an Analysis Methodology with Technology
Locate, Understand
& Obtain Source Data
Validate, Transform
& Prepare Data
Interpret
& Validate Results
Report Findings
& Conclusions
Run Modeling
Procedures & Routines
Define Problem/Question
& Plan Approach
What do you want to know? How can you answer the question?
Enabled by a central database of prior results and insights
What data is needed and in what format?
Enabled by systematic access to mapped data sources
Are source data complete, accurate & ready to analyze?
Enabled by standard validation & data prep routines
What routines, models & visualizations best fit our objective?
Enabled by standard commands, routines, and datasets
Are the results valid and what do they indicate?
Enabled by standard export to validate & verify
Who needs to know the results & by when?
Enabled by standardized, repeatable report production
FR
AM
ES
OLV
EA
CT
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 10
Example: Active Directory
Locate, Understand
& Obtain Source Data
Validate, Transform
& Prepare Data
Interpret
& Validate Results
Report Findings
& Conclusions
Run Modeling
Procedures & Routines
Define Problem/Question
& Plan Approach
Do we have contractors with account expirations later than contract close out? Any history of this issue?
Use PowerShell, ADFind, vbScript to retrieve AD user listing.
Pull contracts/subcontracts from Oracle Financials.
AD: Isolate “user” object class. Convert long dates to YYYYMMDD. Filter contracts database to active contracts. Fuzzy match AD name and contractor name on contract.
Filter for contractors where ‘accountExpires’ > policy date
Stratify by date
Profile by contract, by business unit
Verify results with SME.
Prepare exception spreadsheet and email to IT.
FR
AM
ES
OLV
EA
CT
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 11
Example: Help Desk
Locate, Understand
& Obtain Source Data
Validate, Transform
& Prepare Data
Interpret
& Validate Results
Report Findings
& Conclusions
Run Modeling
Procedures & Routines
Define Problem/Question
& Plan Approach
Do we have an unacceptable rate of aging and reissued high-severity tickets?
Extract ticket header/detail from service now.
Join ticket head/detail and format dates to YYYYMMDD.
Isolate high severity tickets.
Age tickets and flag those older than 5 days.
Profile by user, by help desk personnel, by location
Verify results with SME.
Prepare exception spreadsheet and email to IT.
FR
AM
ES
OLV
EA
CT
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 12
Key Activities on the ‘Data Value Map’
Prepare
Produce
Task / Event / Activity
Measure / Observe
Record
Clone /
Replicate
Backup /
ArchiveWarehouse /
Repository
Post to
another app
Discover
Locate
Understand
Map
Persist (Store)
Routines
ResultsSource
Expose
Alerts / Notifications
Trends
Integrated Analysis
Content for BI / GRC
What people, process, and technology enablers are in place to (1) avoid data value
loss and (2) ensure sustainable value creation using data analytics?
Acquire
Indirect via Extract
Direct via Connector
Integrate
Blend
Templates
Analyze
Model Problem
Analyze / Visualize
Define Problem
Interpret / Validate
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 13
TRADITIONAL BI
Audit Analytic Technology Landscape
VISUALIZATION
TeamMate Analytics
ANALYTICS
MS OFFICE EXCEL ADD-INS CLIENT/SERVER CAATS ADVANCED DA WORKFLOW
DATA MGT / ETL MONITORING UTILITY SCRIPTING
TIGHT ERP
INTEGRATION
LOOSE ERP
INTEGRATION
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 14
Haven't We Seen This All Before?
Decision Support Systems
Expert Systems
Data Warehouses / Marts
Business Intelligence
Business Analytics
Data Mining
Data Science
Digital & networked processes
IoT and (“hyper-”) quantified self
Self-service and social mindset
“Open” software, data, & knowledge
Commodity hardware & virtualization
DA evolved for desktop & cloud use
‒ Can handle 4 V’s: Volume, Variety, Velocity, Veracity
‒ Automation, scheduling, collaboration, visualization
Prior Practices Current Tech Trends
We’re processing a greater amount of integrated, timely, and relevant digital evidence,
leading to richer insights, better decisions, and more effective actions
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 15
How Risk Lives in Your Systems
Configurable system controls may be:• Disabled (past, present, future)• Misconfigured or left to default settings• Outdated/obsolete
Internal system processes may override expected controls, for example:• Sundry invoices bypass 3-way match• Auto-PO or auto-goods receipt generation• Credit and replace practices override pricing
Failed or faulty processing routines may impact data integrity and availability• Gaps in time series• Mishandled NULL values• Changes in scale/units
• Incomplete or erroneous legacy data• Sample or test data left in the system• Faulty join and aggregation logic• Hardware/software constraints
Realistic transaction
Typos, workarounds, duplicates, unstandardized text, truncated/censored entries
Unauthorized C/R/U/D as elevated user
Managerial override
‘Dirty data’ from downstream system
Buffer overflows, transmission problems
Exception report ignored,
misinterpreted, or
unreliable
System warnings are soft, ignored, or misinterpreted
Enterprise Application
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 16
Standard Reports Often Go Only So Far
Example: Monitoring customer credit changes
Standard reports often provide information needed to effectively monitor processes and
risk, but they may not provide it in a way that easily highlight abnormal conditions
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 17
Challenges w/ Traditional Audit Process
Sampling
Rotational/cyclical auditing
Audit process and data silos
Auditor turnover
Keeping abreast of process/organizational changes
New approaches must be adopted to evaluate
risks and controls in a sustainable way
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 18
Add Value
Better insight into areas of management concern (data profiling / trending)
Increased ability to quantify issues
Streamlined data correlation (spanning data and process silos)
Increased responsiveness to management requests
Increase Efficiency
Increased cyclical/rotational audit efficiency (often after year 1)
Decrease cycle time to get through audit universe
Increased breadth of audit coverage (more auditable entities)
Reduced travel (data accessed from anywhere)Standardized, repeatable procedures (less reliance on the individual)Decreased reliance on IT for data acquisitionIncreased confidence & efficiency in verifying data accuracy/completeness
Reduce Risk
•Increased responsiveness to key risks (current and emerging)•Identify problems closer to the first occurrence (more real-time detection)•100% transaction review (providing greater depth of coverage & assurance)•Stratifying the population, thereby honing risk assessment•Keep abreast of system, process and organizational changes
Benefits of Audit Analytics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 19
DA Recognized as a Standard Practice
IIA Quality Assessment
‒ Conducted by an external independent team of qualified audit professionals well-versed in the Standards, assessment methodology, and successful internal audit practices
‒ Report audit activity’s conformance or nonconformance with the Standards and any recommendations for improvement opportunities
‒ More commonly flagging the lack of a robust data analytics program as a reportable concern
Data analysis within audit has finally moved from being
a leading practice to a standard practice
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 20
The Typical Audit Process
Risk Assessment
Audit Planning
Fieldwork / Testing
ReportingFollow-up & Monitoring
Audit Planning
Fieldwork / Testing
ReportingFollow-up & Monitoring
Audit Planning
Fieldwork / Testing
ReportingFollow-up & Monitoring
Audit Planning
Fieldwork / Testing
ReportingFollow-up & Monitoring
** darker color = greater DA usage
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 21
Data Analysis and Audit Process Stages
12 3 45
Typical implementation order of audit analytics over time
• Increasing involvement of other lines of defense
• Where is your organization in this process?
• Could IA benefit from doing more at a specific stage?
Risk Assessment
Audit Planning
Fieldwork & Testing
ReportingFollow-up
and Monitoring
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 22
The “Data-Enabled” Audit Process
Risk Assessment
Audit Planning
Fieldwork & Testing
ReportingFollow-up
and Monitoring
Identify data-driven
risk indicators
(current and
emerging)
Continuous Risk
Assessment
Aggregated across
all auditable
entities
Audit entity-
specific
Analysis of survey
and other data-
gathering activities
for trends or
correlation
Focused
discussions with
management about
anomalous audit
entities
Review previous
findings / results
Team brainstorm &
prioritize potential
data analytics
Data profiling of
relevant data and
meta-data
Assess available
data / plan additional
data acquisition
Data trending and
ratio analysis
Re-validate risk
assessment priorities
and refine audit plan
Develop data-driven
audit program and
test routines
Record future
analytic ideas
Independently
access audit-
relevant data
Data exploration
100% population
testing for
exceptions /
attributes
Automated, risk-
based sampling
Ad-hoc data
analysis
Queries / routines
for batch analysis
Store results and
analytics centrally
for other auditors
and future re-use
Visual dashboards,
scorecards, reports
& storyboards with
drill-down capabilities
Generate alerts &
notifications based
on business rules
Data-driven risk
quantification,
including where risk
initially identified
through manual
processes
Evaluate need for
continuous audit
procedures
Assess potential
future analytic
improvements
Provide management
with analytic
prototypes
Automated re-
testing of
resolution
Quality
Timing
Visual issue
trending
“Entity
Correlation”
Employees /
Vendors
Locations /
Departments
How are data analytics being applied across the IT audit process?
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 23
Audit Analytics mixes digital evidence, algorithms, & technology
Use technology and algorithms to enable the stages of data analysis
Vast market of technologies that can be used for audit analytics
Traditional audit challenges make it difficult to address risky systems
Analytics can help IT auditors Data analytics across various phases of
the IT audit process
Concluding Thoughts
© 2016 High Water AdvisorsConfidential. Do not distribute.
Data Analytics for IT Audit
Potential & Common Tests for IT Audit
ISACA, North Texas Chapter
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 25
Logical and physical access
Configuration and change management
Data integrity and master data management
SIEM and cybersecurity
IT project management and SDLC
Key Topics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 26
Question: Are functionality/permissions/rights provision to
appropriate users in a timely, efficient manner?
Challenge: Usually need additional multiple data pieces to clarify if
legitimate employees (Employee Master), with access (User Listing),
have the correct access (Roles/Entitlements)
Approach:
Logical Access – Account Provisioning
Application Access
HR/Employee Masterfile
Unmatched Right/Secondary:
(Primary key is blank) Employee, no
application access ID, possible
phantom employee
Matched: (Primary key = Secondary key) Application Access ID, and Employee record are both present,
combine with the Profiles file to see if access is appropriate for the department the employee works within
Unmatched Left/Primary:
(Secondary key is blank) Application
Access ID, no Employee record,
possible phantom ID
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 27
Question: Are functionality/permissions/rights terminated in a timely,
efficient manner?
Challenge: Obtaining an HR dataset with chronological hire,
transfer, promotion, leave, and termination information
Approach:
Logical Access – Account Deactivation/Termination
Application Access
HR/Employee Terminations
Matched: (Primary key = Secondary key) Application Access ID, and Employee record are both present,
combine with other information like date, manager, job title/code to assess risk
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 28
Question: Do we have duty conflicts? Have they been exploited?
Challenge: SoD projects tend to languish trying to figure where to
start remediation.
Approach:
‒ Recurse through roles table to see which roles coexist.
‒ Correlate duty conflict table to select those duties where conflicts exist
Logical Access – Segregation of Duties
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 29
Question: Do all the employees with the same job title have the
same permissions?
Challenge: This type of analysis typically requires a script that can
iterate of user and possible permissions within an application.
Approach:
‒ Obtain each users permissions and job function
‒ Cluster users by function
‒ Identify users with permissions different than their peers
Logical Access – Reasonable Permissions
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 30
Questions: Are passwords exposed in clear text? Are passwords
reset timely? Are users timed out?
Challenge: Obtaining an HR dataset with chronological hire,
transfer, promotion, leave, and termination information
Approach:
‒ Directory Command – password.txt files
‒ Active Directory
» Policy Compliance: (filters or computed fields)
o cutoffdate - pwdLastSet > policy pwd life
o cutoffdate – lastLogon (dormant accounts)
» Couldn’t happen:
o accountCreated < date of hire (above x days)
o lastLogoff < lastLogon (or they are still logged in)
Logical Access – Passwords & Account Management
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 31
Question: Are facilities being access by unauthorized persons?
Challenge: Many doors, many people, many access transactions
Approach(es):
‒ Ingress/Egress logs correlated with valid badge holder
‒ Sequence analysis: Interior door access missing exterior door access
» Profile incidents by building, person
Physical Access – Physical Security
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 32
Question: Are configurations applied correctly and consistently
across information assets?
Challenge: Direct access to configuration data. Sometimes in
unstructured files.
Approach(es):
‒ Webserver XML Configuration, parese for key values and compare to standard
‒ Extract SAP/Oracle config tables and correlate settings to external standard
‒ Validate that the settting cross affiliates and company codes
Configuration Baselining
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 33
Questions: Are change to systems authorized, tested, implemented,
and communicated by the appropriate person in a timely manner?
Challenge: Detecting changes and correlating them with something
else that should have changed.
Approach:
‒ Obtain directory listings (before & after)
‒ Correlate to something else that should have changed
» System Development or Project Management records
» Change management records
» Testing (QA) records
» Incident/problem management records
» Help Desk tickets (all of the above could/should be here)
‒ Who made the change and when?
Change Management
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 34
Detecting file changes on Windows
Windows PowerShell:‒ Change to the directory containing the scope of the files:
cd c:\dir\with\in-scope\files
‒ Files last written to (changed) after a certain date:
get-childitem –recurse | where-object {$_.lastwritetime -gt “5/1/2012”} | out-
file “changedfileslist.txt” –Encoding ASCII –width 240
‒ To find all the files in the current and sub directories written to
(changed) in the last 15 days:$DateToCompare = (Get-date).AddDays(-15)
Get-Childitem –recurse | where-object {$_.lastwritetime –gt
$DateToCompare} | out-file “changedfileslist.txt” –Encoding ASCII
–width 240
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 35
Detecting file changes on Linux
Linux files changed within 30 days:find /directory –mtime -30 –ls
(modified time) (detailed list)
add: > changedfiles.txt to redirect to a file
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 36
Detecting Changes – General Model
Most Recent Listing
Prior Listing
Additions
On both listings, further analysis
needed of date modified - Changes
Deletions
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 37
Question: Have important items change on a host? Has a critical
event happened (or not) that may leave the host vulnerable?
Challenge: Obtaining log files. Parsing event information.
Approach(es):
‒ New events since last run
‒ Missing events
‒ Classify events
‒ Keyword search
‒ Sample events, then:
» Correlate with help desk tickets
» Correlate with change control documentation
System Event Logs
Current Log
Summed Prior Log History
New Event
Recurring
event
Missed Event
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 38
Question: have critical events occurred and not been reported?
Challenge: Mining verbose logs
Approach(es):
‒ Import event logs/Query event DB
‒ Parse event strings
‒ Classify by event, asset, date, text tag
System Event Logs
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 39
Question: Am I exposing configuration and network settings of hosts
that can be found externally? Which vulnerabilities are not getting
addressed in a timely manner?
Challenge: Budget friendly tools don’t scale well over time or across
multiple hosts.
Approach:
‒ Create 2 scans per host (prior / current)
‒ For every host, join prior and current to identify new vulnerabilities
InfoSec - Configuration & Vulnerability Scans
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 40
Nessus
The Host Properties table has data stacked , 7 rows (at most) per host
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 41
The machine, date header values can be captured
The consistent 3 values in an item, “issue” . “score”, “result” can be
captured
The “detail” section of an issue is variable, class challenge – your
homework assignment, figure this out
Microsoft Baseline Security Scanner
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 42
Question: Have important items change on a host? Has a critical
event happened (or not) that may leave the host vulnerable?
Challenge: Obtaining log files. Parsing event information.
Approach(es):
‒ Kismet (free), will poll access points whether or not they are broadcasting their
SSID
‒ Generates a summary report of what was found and packet level detail, the
summary tags are “wireless-network”
» Correlate valid AP listing
» Identify rogue Aps
» Identify missing APs
Wireless network scans
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 43
Audit and monitor timely maintenance of critical systems (Network,
OS, App, or Data)
Determine whether planned maintenance schedules aligned with
historical incident/downtime trends
Deploy advanced models to predict optimal maintenance schedules
(predictive maintenance)
System Maintenance
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 44
DA Completeness - Reconcile source data in DB with independent
record(s)
Data Quality – data elements critical in line with expected data
conditions (standardized formats)
Referential Integrity - Verify that relationships between core data
(transactions) accurately and map completely to referential data
Database
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 45
SLA metrics recalculation
Help desk data quality
System re-performance
Data conversion verification
Process-to-Application Analytics
© 2016 High Water AdvisorsConfidential. Do not distribute.
Data Analytics for IT Audit
Making analytics sustainable
ISACA, North Texas Chapter
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 47
Data analytic approaches for IT audit planning and risk assessment
Automation and moving towards continuous IT risk assessment and
auditing
Tips for making IT data analytics a sustainable practice
Best practices for data analytics program implementation within IT
audit
Key Topics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 48
IT Audit Planning
Consolidate & summarize daily
“high” report
Consolidate IT help desk tickets
‒ Profile high-risk events by
function (e.g., ops, DB,
network)
Query event database items
‒ Profile high risk events by
asset, user, time, location
Obtain “Project Priorities” list
Review versions of key plans
‒ Priority differences (add/drop)
‒ Resource drift
‒ Scope creep
‒ Project slippage
Compare current & prior lists
‒ New, dropped, changed
projects
ApplicationInfrastructure
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 49
Data Profiling
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 50
Visualizations != Analysis
‒ Visualizations are information that has to be interpreted and given context
Examples: Graphs, Scorecards, Dashboards, Storyboards
Risk Assessment: Visualization/Storyboards
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 51
Uptime: 92%
Performance/response time: 80%
Data loss: 56%
Number of open issues: 52%
Average time to fix: 51%
Security breaches: 49%
Mean time between failures: 38%
Types of Indicators
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 52
Visual Display of Risk Information
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 53
Tech: ACL GRC / Results Manager
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 54
Tech: IDEA – CaseWare Monitor
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 55
Tech: SAP GRC / Process Control
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 56
Tech: SAS Enterprise GRC
Sustainable Analytics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 58
Audit Analytics are Sophisticated AND Sustainable
Sophisticated
• Analysis is linked to business objective?
• Leverage larger data volumes and variety?
• Leverage state-of-the-art technologies?
• The analysis uses a novel approach?
• Calculations, routines, and/or number of
variables considered are beyond traditional
analytical approaches?
Sustainable
• Am I re-performing the analysis?
• Can I easily repeat the analytic steps?
• Can I perform the analysis independent of IT?
• Does the analytic minimize ‘Time-to-Support’?
• Does the analytic minimize ‘Time-to-Use’?
• Can I scale the analytic?
• Can I share the analytic?
• Can the analytic be self-service?Low
High
Low High
Sustainability
S
op
his
tica
tio
n You are here
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 59
Document Standards & Guidelines
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 60
Object Examples
Collections
Folders
FY2015 SOX Testing
AP – Account Payable
Analytics AP01_detectDuplicateVendor
ACL Projects SOX_AccountsPayableTests
Script
Subscripts
AP01_duplicateNameCheck
AP01_duplicateNameCheck2
Tables
Views
Indexes
T_temp P_prepared S_source AP01_duplicateName_D
Default_view Essential_Fields Approvals Key_Dates
i_BSAK
Parameters
Variables
Filters
Computed fields
p_cUserName
v_dStateDate
f_isValueGT1000
c_nInvoiceAmount
Output files ctl_controlReport.xlsx AP01_duplicateName.xlsx
Naming Conventions
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 61
Start projects from a template
Include reusable scripts in the
template
Complexity can be added or
removed depending on the nature of
the project
Standard Project
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 62
Standard High Value Analytics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 63
Shared data files (source, results)
Shared scripts / Analytic apps
Shared resource files (e.g., keyword lists)
Analytic documentation
Job aids & Checklists
Analytic ideas (new ideas, enhancements)
Application / DB inventory (w/ points of contact)
Repositories
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 64
QA
Developer vs. Independent QA
Non-technical QA
Technical QA
Capturing test results and updating analytics
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 65
Goal to ensure that analytic:
‒ Works as intended
‒ Reports intended results
‒ Does not miss any desired results
Key decisions:
‒ Who will be testing?
‒ What data will be used?
‒ Positive testing? Negative testing? Both?
‒ Stress testing included?
Testing
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 66
Largely driven by your industry and internal policies
Data to consider
‒ Source data
‒ Temporary files created during analytic
‒ Results
May also need to consider:
‒ Data in transit
‒ Data in memory
Data Security
DA Program Implementation
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 68
IA See Benefits, but Lacks Planning
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 69
Data Analytics Program
Formal, sponsored organizational effort to use data, models,
and fact-based management to evaluate the state of the
business and drive decisions and actions
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 70
Data Analytics Maturity Model
• Increased assurance & value-added services at higher levels
• Increased maturity and sophisticated DA typically requires
increased time and budget for:
Initial
Repeatable
Defined
Managed
Optimizing
► No formal data analytics
approach, procedures, or
methodology
► Performed occasionally, ad-
hoc, and often unplanned
► Dependent on skills of
limited number of staff
► Purpose-built tools are not
readily available
► Often focused on quick
development over ease of
repeatability
► Recognized as value-add
to the audit
► Partially integrated into
target audit process
► Often covers both audit
planning and fieldwork
► Structured approach, but
not yet institutionalized
► Tests reused and added
in subsequent audits
► Tools are available, but
not applied consistently or
correctly
► Well-documented and
consistently followed DA
methodology
► Centralized, structured
knowledge management:
Data
Audit tests
Results
Supporting docs
► Collaboration
► Controlled, secure access
to data and routines
► Established data access
protocols
► Suites of tests available
to audit team
► Concurrent, ongoing
auditing of multiple areas
► Structured issue reporting
and tracking
► Formalized process for
changing / improving
analytic routines
► Management monitoring
of own process
► IA assesses management
monitoring activities
► Continuous management
& reduction of false-
positives
► IA conducts continuous
risk assessment
Level 1 Level 2 Level 3 Level 4 Level 5
o Technology
o Data Management
o System Administration &
Operations
o Queries/Models
o Collaboration
o Governance
o Enterprise Integration
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 71
Understand the key drivers for DA
Create a vision of DA use within the audit function
‒ Define short- and long-term goals within context of vision
Anticipate implementation barriers
Create initial project plan
Develop performance metrics & measurement system
Assign responsibilities
Put plan into action
Measure and track progress
Review and refine approach over time
Program Strategy and Approach
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 72
Increase the value and quality of audit findings? Where?
‒ Finding money for bottom-line impact?
‒ Identifying operational inefficiencies?
Reduce the risk of knowledge loss? Improve audit consistency?
Implement continuous auditing? What scope? How soon?
Assurance: Increased coverage? Sleep better at night?
Increased efficiencies within audit? By when?
‒ To increase the number of audits within a year?
‒ To do more comprehensive testing within the same population of audits?
‒ To reduce audit costs?
Offload manual testing and re-focus on more strategic tasks and analysis?
‒ Most time-consuming manual processes?
‒ Testing disliked by auditors?
Key Value Drivers
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 73
Direct and Inspire Through Vision
Ensures everyone working towards same end and helps reduce
unnecessary diversions
Drives consistency of decision-making
‒ Short- and longer-term priorities
‒ Investments in resources and technology
“Let’s take a
vacation!”
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 74
Create Confidence Before Accelerating
Set realistic short-term goals
Use all available resources—don’t reinvent the wheel
‒ Public domain
‒ Peer organizations
‒ User groups
‒ Trusted partners
Conduct pilot project(s)
Accelerate only after building experience and internal support
necessary for sustainable growth
‒ Controlled acceleration = proactive, not reactive
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 75
Anticipate Implementation Challenges
Adequate Budget
Technology
Training
Professional services
Effective Support & Direction
Getting management buy-in and understanding of
effort
Poorly-defined analytic scope
Supportive Policy
Competing DA initiatives
Incongruent processes / policies
International issues
Capable and Available Resources
Understanding source data and related business
processes
Strategies for locating, accessing and integrating
disparate systems and formats
Availability of internal expertise to evaluate
results/anomalies
Outsourced IT
Dealing with false positives
Data Access & Quality
Data preparation time to resolve issues with data
quality
Manually-maintained data and manual controls
External and cloud-based data
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 76
DA Capability Levels – Management Tasks
Level 1: Initial
Level 2: Repeatable
Level 3: Defined
Level 4: Managed
Level 5: Optimizing
Early-Maturity
• Set clear vision & direction
• Consider IA role in continuous
monitoring
• Encourage experimentation/creativity
• Address ‘prior budget’ challenge
• Create incentives, or dis-incentives,
depending on your style
• Prioritize data access & preparation first
• Lynchpin for analytics and harder to
outsource efficiently
Mid-Maturity
• Champion the vision
• Communicate KPIs (current & desired)
• Eliminate data access as a hurdle
• Negotiate access with the CIO, not every
DBA
• Stress process over individual preference
• Establish effective QA process
• Set foundation for working smarter, not
harder
• Knowledge management and
collaboration are critical at this stage
Late Maturity
• Champion and lead the change management
required for Continuous Risk Assessment
• Recognize and support the additional
investment required to move from ad-hoc or
repetitive to continuous
• Scheduled data feeds
• Script logic adjustments
• Managing (and reducing) false-positives
• Facilitate breakdown of organizational silos
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 77
Practice Effective Change Management
There are multiple actors involved‒ Audit team, Internal Stakeholders, and External Stakeholders
‒ Identify and make plans to proactively address influencers / detractors
Change will occur in many areas of the department‒ New roles and responsibilities will required new skills, knowledge and behaviors
‒ Generating new types of deliverables and integrating new value to the business
‒ Introduction of new technologies and dependencies on upstream and downstream IT activity
‒ Revisions to existing audit processes, procedures, & schedules
» Add time to audit planning for audits where data analytics have not been attempted before
» Team brainstorming during planning instead of relying on the AIC
» Adding analytics steps to audit planning and audit QA checklists
» Documenting when DA was used/not used and why
» Capturing DA ideas in repository
» Data profiling and using reports/results in management interviews
Consider incentives to help get through the pain of doing something new
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 78
Track Relevant KPIs / Metrics
Value
# or % of audit findings resulting from
analytics (both low-level & audit-committee-
level)
Hours saved (audit hours, IT hours)
Coverage
# or % of transactions reviewed as part of the
audit (vs. what was covered by a traditional
sample)
# or % of key systems where direct data
access has been established
Number of key business processes with at
least X analytics
Efficiency
Time to complete specific audit steps
Cost of Audit (incl. travel…useful when
analytics may reduce travel time)
Collaboration (within audit team)
# Analytic requests / # Analytic submissions /
# Analytic self-serve downloads
# of Auditors trained and/or certified on data
analytic technology
# of auditors certified as data analysts
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 79
Technology Planning & Maintenance
Establish a solid data foundation
Plan the appropriate mix of Excel, CAATs, Visualization technology
licenses across the team
Ensure technologies support activities across the various stage of
data analysis
The need for technology increases as you move towards continuous
operations (the required skills change, too!)
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 80
Expertise to Embed Within the Function
Data access & modeling
‒ Access, prepare, and make data
available
Data analysis
‒ Perform advanced analysis and create
analytics and results for sharing
Data literacy / consumption
‒ Review and interpret results
‒ Perform simple analysis independently
DA Management
‒ Oversight of analytic-enabled audit
activities
» Guide analytics planning
» Monitoring progress and setting
boundaries
» Compliance with DA guidelines &
standards
» Ensuring valid results
» “Future proofing” the work
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 81
Training and Development
Identify gaps in skills, knowledge and understanding
Personal and team learning goals and metrics
Progress review and coaching/mentoring
Relevant assignments and challenges
Resource availability
‒ Reference material
‒ Usage examples
‒ Guidance and support
Suitable learning strategies
‒ Just-in-Time and blended learning approaches
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 82
Staffing and Retention
Map required skills to existing competencies
Develop staffing, recruiting, and interviewing plans
Identify critical long-term skills and competencies
Key strategies to consider
‒ Job pathing through increasingly technical roles
‒ Employee engagement / retention
‒ Staff coverage & backups
‒ Succession planning
‒ Documentation
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 83
Sustainable success is driven from a sound vision
Initial success requires momentum, but sustainability requires
effective planning and design
Everyone in the department plays a role
Audit management must be willing to adapt and change the audit
process to support effective analytic usage
Don’t just expect to evolve over time, plan for it
Concluding Thoughts
© 2015 High Water AdvisorsConfidential. Do not distribute without permission. Page: 84
Questions / Thoughts?
jim.tarantino@highwateradvisors.com
1-855-RISK-WATCH
Wrap-up
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 86
Workbook for a Successful Audit Analytics Program
‒ Just launched this week at IIA GAM
‒ 50% off introductory rate through the end of March, 2016
Set of working documents to guide you
through decisions
Checklists for CAE and management
Get your copy at:
‒ www.highwateradvisors.com/workbook-for-a-successful-audit-analytics-program
Just released: Workbook for a successful program
© 2016 High Water AdvisorsConfidential. Do not distribute without permission. Page: 87
High Water Advisors Can Help
Analytic development, QA and/or
optimization
Strategy development & coaching
those responsible for
managing/implementing analytic
programs
Training for those developing data
analytic routines
‒ Strategic concepts
(i.e. dealing with false-positives)
‒ Sustainable Analytic Design
‒ Technology-specific programming
Configuration baselining
‒ Recommended settings
‒ Reasonableness
‒ Comparative to similar entities
Transactional analysis & profiling
Control circumvention assessment
(suspicious activity)
Security assessment & back-door
activity analysis
Cybersecurity
SAP-Centric Data AnalyticsGeneral Data Analytics
Recommended