Performing An Effective Project Audit by Muema Lombe, 2011

8/6/2019 Performing An Effective Project Audit by Muema Lombe, 2011

1/56

Session #H8

, ,

11:30am

Muema Lombe, CRISC, CSSLP, CGEIT, CISA

IT Audit Manager

Endurance Services Ltd.

2011 Muema Lombe


2/56

Key Points

The many reasons projects fail

Project audit roles & responsibilities

Understanding project risk factors

Case studies of project audits & samplefindin s

The key components of a project auditprogram

2011 Muema Lombe


3/56


4/56

Project Management Methodologies

PMI PMBOK A Guide to the Project Management Body of Knowledge

(PMBOK Guide) is a book which presents a set of standardterminology and guidelines for project management.

It was first published by the Project Management Institute in

1987. PMI lobal standards rovide uidelines, rules and

characteristics for project management.

http://www.pmi.org/

2011 Muema Lombe


5/56


Prince2 PRojects IN Controlled Environments (PRINCE) is a project

management method. It covers the management, control andorganisation of a project. "PRINCE2" refers to the secondmajor version of this method and is a registered trademark ofthe Office of Government Commerce (OGC), an independentoffice of HM Treasury of the United Kingdom.

http://www.prince-officialsite.com/home/home.asp

2011 Muema Lombe


6/56


Waterfall The waterfall model is a sequential design process, often

used in software development processes, in which progressis seen as flowing steadily downwards (like a waterfall)through the phases of Conception, Initiation, Analysis,Design, Construction, Testing and Maintenance.

2011 Muema Lombe


7/56


The "waterfall model". Progress flows from the top to the bottom, like a waterfall.

2011 Muema Lombe


8/56

Key Points

The many reasons projects fail Project audit roles & responsibilities




2011 Muema Lombe


9/56

The Many Reasons Project Fail

Failed project example: Denver Airport Baggage System

Early Warning Signs of Project Failure

2011 Muema Lombe


10/56


Denver Airport Baggage System Case Study

2011 Muema Lombe


11/56



Originally billed as the most advanced system in the world, thebaggage handling system at the new Denver InternationalAirport was to become one of the most notorious examples ofproject failure. Originally planned to automate the handling ofbaggage through the entire airport, the system proved to be farmore complex than some had original believed. The problemsbuilding the system resulted in the newly complete airport

sitting idle for 16 months while engineers worked on getting the.

The delay added approximately $560M USD to the cost of theairport

2011 Muema Lombe


12/56



The Denver debacle is a tem late for failure that man other ro ects have followed.As with so many other failures, Denver suffered from:

1. The underestimation of complexity

2. A lack of lannin resultin in subse uent chan es in strate

3. Excessive schedule pressure

4. Lack of due diligence

5. Making firm commitments in the face of massive risks and uncertainty.

7. Communications breakdowns

8. People working in silos

9. Poor design

. a ure o per orm r s managemen

11. Failure to understand the implication change requests might have

12. Lack of management oversight

2011 Muema Lombe

Source: http://calleam.com/WTPF/wp-content/uploads/articles/DIABaggage.pdf


13/56


2011 Muema Lombe

Source: EARLY WARNING SIGNS OF IT PROJECT FAILURE: THE

DOMINANT DOZEN by Leon A. Kappelman, Robert McKeeman, andLixuan Zhang


14/56

Key Points





2011 Muema Lombe


15/56

Project Audit Roles & Responsibilities

IT Audit Role on Pre/Post ImplementationReviews

IT Audit Level of Engagement

2011 Muema Lombe


16/56


PMO Project Role to monitor and report project status, andto monitor and report project costs.

IT Internal Audit Project Role opine on the overall controlenvironment of the new project, system or application byevaluating, testing, and commenting on the effectiveness ofrisk management, control and governance processesfocusing on technology risks.

2011 Muema Lombe


17/56


Reviews To ensure new systems, or applications include all

consultative and proactive manner by evaluating,testing, and commenting on the effectiveness of risk

management, control, and governance processes.

2011 Muema Lombe


18/56


The level and depth of IT audit involvement in preand post implementation reviews is based on the

ro ect risk assessment,

project team's project management experience,

level of management involvement,

size and complexity of the initiative, and mpac on e organ za on e n a ve s e aye or

unsuccessfully implemented.

2011 Muema Lombe


19/56


Internal auditors' involvement in an organization's system conversion initiatives can range from minimal involvement (level1) to extensive audit efforts throughout every phase of the project (level 10).

Level 1: Audit risk assessment during the project initiation phase.

Level 23: Audit review of documentation and project deliverables.

Level 45: Attend project meetings, conduct some interviews, and produce verbal audit reports. Level 67: Increased audit efforts, conduct more interviews, and produce formal audit reports.

Level 89: Review all milestones, perform extensive audit tests, and produce formal and comprehensive audit reports.

.

Internal auditors should determine their level of involvement and approach during the project's initiation phase. The auditinvolvement decision should be based on the project risk assessment, as well as factors such as the project team's

project management experience, level of management involvement, size and complexity of the initiative, and impact.

needs to be defined during the audit project planning phase. Following step 1 of the generic eight-step audit processwill complete the definition of audit's level of involvement. Further adjustment of audit involvement may be requireddepending on the results of the projects efforts and auditors' assessment throughout the project's life cycle. Anotherimportant consideration to discuss with management and the project team is the internal auditors' roles andres onsibilities in attendin ro ect team meetin s throu hout the conversion audit.

Source: IIA.org

2011 Muema Lombe


20/56


Reporting Once of the level of engagement has been decided, agree on the format,

.

DISTRIBUTION: Will reports be distributed to all project participants? Projectsponsor? Project steering committee?;

FREQUENCY: Will reports be prepared quarterly, monthly, or at the end of thepre pos mp emen a on rev ew

FORMAT: Agree on format - will the end result be a format audit report? If areport, will it be a detailed report? Executive summary? Will the end result be a

memo with observations rather than a report with issues?

2011 Muema Lombe


21/56

Key Points





2011 Muema Lombe


22/56

Understanding Project Risk Factors

2011 Muema Lombe

Source: Project risk factors checklistwww.techrepublic.com


23/56

Understanding Project Risk Factors

2011 Muema Lombe

Source: Project risk factors checklistwww.techrepublic.com


24/56

Key Points





2011 Muema Lombe


25/56

Case Studies of Project Audits & Sample

Case #2: Regulatory Reporting Platform

2011 Muema Lombe


26/56


customers to view, filter and report on data. Audit Involvement: IT audit participated in

wee y pro ec eam mee ngs as an o server,and prepared a quarterly memo summarizing

project status and observations. Findings:

Resource Management

Software Development Process

Vendor Management

Scope & Requirements Management

Business Participation

2011 Muema Lombe


27/56


Findings:

Resource Management

Business analysts shortage of team members with the right skill sets to write

requirements. Project managers team members did not have sufficient experience running

large, complex projects.

Technical architect no one with end-to-end responsibility for overall technicaldesign

Priorities ongoing resource conflicts due to competing priorities and lack ofdedicated project resources.

Offshoring occurred during critical development time period and disruptedmanagements project focus.

Roles & responsibilities given the above issues, the role of team memberswas unclear and led to an over-reliance on mana ement b consensus durin

the requirements process.

2011 Muema Lombe


28/56


29/56


Findings:

Vendor Management

Vendor Selection insufficient screening of 3rd party software packages.

Vendor Oversight inadequate management of consultants assigned criticaldevelopment work.

2011 Muema Lombe


30/56


Findings:

Scope & Requirements Management

Project Scope the project scope was not realistic for the timeframe requested.

Requirements expertise analyst team members lacked adequate training inrequirements development and were not dedicated to the task.

2011 Muema Lombe


31/56


Findings:

Business Participation

Steering Committee the steering committee served more as a working group

than a senior management oversight committee. As a result, the key businessstakeholders were not included, making it difficult to insure they received earlynotice of emerging problems.

2011 Muema Lombe


32/56


regulatory reporting in 3 European countries. Audit Involvement: IT audit participated in the

pro ec y mee ng w e eam ea mon yand reviewing available project

documentation. IT audit prepared a monthly

observations.

Findings:

Project Management Design Documentation

Resources

2011 Muema Lombe

ppage


33/56


Findings:

Project Management

Lack of a Project Management Resource - Six months into an 18 month project,

a project manager has not been assigned to the project, as a result, projectmonitoring and tracking is not being performed.

2011 Muema Lombe


34/56


Findings:

Design Documentation

Inadequate Design Documentation - Detailed design documentation was not

prepared. A flowchart served as the sole design document. The flowchartcontained limited details with vague explanation of how to receive data fromsource systems and further process the data.

2011 Muema Lombe


35/56


Findings:

Resources

Inadequate Project Resources In addition to the lack of a dedicated project

manager, 2 resources are expected to document regulatory reportingrequirements for 3 countries, in addition to their daily responsibilities.

2011 Muema Lombe


36/56


Findings:

Slippage

Project slippage IT audit noted 2 weeks of slippage based on the original

project plan. The plan does not include any timing for contingency, as such thiscould impact the timely delivery of the project. Additionally, this slippage wasnot communicated to the Project Sponsor such that appropriate remedial actioncou e a en.

2011 Muema Lombe


37/56

Key Points



Case studies of project audits & sample

findin s


2011 Muema Lombe


38/56

The Key Components of a Project Audit Program

Project Sponsorship

Project Organization

Quality Assurance Management

Procurement Management

Project Milestones

Scope Measurement

Project Approach Management

Schedule Management

3rd Party Management

Communication Management

Interdependency Management

Change Management

Cost Management

Personnel Resource Management

Risk Management

Transition Management

Project Conclusion Management

2011 Muema Lombe


39/56


40/56


Project Organization (Roles and Responsibilities)

This element refers to how the team is structured, reporting relationships, liaison roles. It isg y es ra e a user epar men s a e nc u e w n e pro ec eam n or er o

ensure that the system fully reflects user needs and to obtain greater user commitment to the

system. Some aspects of the project may be delegated to outside consultants andcontractors. It is essential in each case to specify the tasks and deliverables to be providedb each art , and the dates b which the must be com leted.

Audit should verify the roles and responsibilities have been defined and appropriate tasksand deliverables have been defined.

2011 Muema Lombe


41/56


42/56


Scope Measurement

The element refers to ensuring the scope and objectives must be defined properly. Thesecan e groupe un er wo ma n ea ngs:

The project charters, which defines project goals and contains an outline proposal for the

project. This outlines describes the proposed system and the information it will use. Costs,benefits, preliminary schedule, and the impact the system will have on the organization and

.

The enterprise description is an important component, especially on larger projects withmany people involved. It lays out the users need and environment and relates them to thoseof the organization and external factors. Without an enterprise description there is a greater

chance that users real needs and the overall requirements of the organization will not besatisfied.

2011 Muema Lombe


43/56


Project Approach Management

It is highly desirable for the project to be defined formally in a document that describes itsscope an e ro es o ose nvo ve . xper ence as s own a users w o are prepare oproceed on an informal basis have little understanding of what is involved in implementing

the system.

2011 Muema Lombe


44/56


Schedule management

This element refers to the process of allocating activities (identified in the breakdown andgiven estimated times) to members of a project team against a calendar. Considerations forskill level ro ect team structure and size and schedulin and bud etin resources shouldbe included in the development methodology.

Audit should ensure the necessary resources are allocated to the project and.

2011 Muema Lombe


45/56


Cost Management

This element relates to ensuring the initial pre-determined costs remain at or below projectedcos s. orma y on a arge sca e pro ec , a res o w e es a s e , a w e nc u ein the project charter, which will state if a project is going to exceed projected costs by a

certain dollar amount or percentage, then the project sponsor should be notified immediately.In some instances, senior management and approval are required.

2011 Muema Lombe


46/56


47/56


Risk Management

This element relates to risks that could adversely effect the completion of the project withine g ven ours an or m ng. s s suc as a r par y ven or go ng ou o us ness,

project staff turnover should be evaluated.

2011 Muema Lombe


48/56


Quality Assurance Management

This element relates to the process of ensuring that all deliverables are of required qualityan a a wor as een carr e ou accura e y an o an appropr a e s an ar . songoing effort throughout the development cycle and system use.

2011 Muema Lombe


49/56


Procurement Management

This element relates to ensuring that all the necessary resources, equipment, hardware,so ware can e o a ne or e pro ec o mee pro ec ea nes.

2011 Muema Lombe


50/56


3rd Party Management

Consideration should be given to look at contract agreements, maintenance agreements,m ng o expec e wor comp e on, resource a oca on, p ys ca oca on o par y.

Evaluation of due diligence work is imperative at the earliest point possible.

If the project team is implementing 3rd party software, you should consider where the sourcecode for the software is stored and if in the contract, we are allowed to obtain the source

rd

2011 Muema Lombe

C


51/56


Communication Management

This element relates to formal organizational arrangements which should be adopted toac a e success u exp o a on o n orma on ec no ogy. ese nc u e e mpor ance o

having a project sponsor, the need for informal and formal user involvement, the role of

corporate information and project steering committees, and the advantages of standardizedcommunication between these various parties. Historically, many projects have failedbecause of inade uate communication between MIS and the user de artments

2011 Muema Lombe

Th K C f P j A di P


52/56


Interdependency Management

This element relates to other projects that depend on the initial project to succeed or otherpro ec s a mus comp e e pr or o s pro ec so s pro ec w succee . e pro ecteam should be aware of all of these projects and just be figured into the timing and

milestone documents.

2011 Muema Lombe

Th K C t f P j t A dit P


53/56


Change Management

This element relates to maintaining version control over project documents relatingspec ca y o e n a pro ec . s wou nvo ve any ocumen s a wou e use osupport the System Development Life Cycle of this project and track the project until

completion.

2011 Muema Lombe


54/56



55/56


Project Conclusion and Turnover Management

If the project was planned correctly and user involvement has been adequate, the installationo e sys em an e rans er o suppor respons y s ou procee smoo y. s aspecific responsibility of the project director and project manager to satisfy themselves,

before allowing a system to be installed, that: All stages of testing, including acceptance testing, have been completed properly.

Adequate user procedures have been prepared and the users are trained to the required level and want to workwith the system positively.

The MIS department is prepared to run the system.

All required controls and security procedures are in place.

2011 Muema Lombe

Summary


56/56

Summary

METHODOLOGY: Understand the project managementmethodology deployed at your organization

-the project (e.g. pre-implementation? Post-implementation? Silentparticipant in project team? Other?)

: gree on orma , requency an s r u on oreporting.

RISK: Understand project risk factors. SCOPE: Define the scope of your audit.

EXECUTE: Execute and report on the audit.

2011 Muema Lombe

Documents

Performing An Effective Project Audit by Muema Lombe, 2011