Effective audit involvement in systems development

Computer Audit Update November 1991

In short, users need to recognize that while the IT department plays a vital organizational and technical role it is the user who is responsible for ensur ing that the IT resources are used effectively. It is their use of the technology to achieve their objectives that determines whether the organization is receiving value for money from its investment.

Flexibility is essential when evaluating a package as an alternative to an in-house development. The organization seeking a package which will exactly fulfil its needs is likely to be disappointed. But software which satisfies, say 80% of the users' needs, can often be found and made available within an acceptable timescale. It may also be sensible to invite users to question whether it would be more cost effective for them to alter existing work practices to fit around the package than to expect the package to fit their long standing day to day procedures.

Performance measurement

How can senior managers keep their finger on the pulse of the IT department and how can they minimize the risk of being unpleasantly surprised by reports that something is not going according to plan? The answer to these questions must lie in applying indicators of the performance of the technology: hardware, software and network. While the IT industry provides a range of tools for monitoring the eff iciency of hardware and software, the complexity of modern day computing means that specialist skills are invariably called for. The challenge facing the IT managers is how much dependence to place upon the suppliers' advice on optimum performance.

Conclusion

Most organizations in the current economic climate face a number of exacting management challenges and a period of considerable change. An efficient, effective and economic IT service will be an essential prerequisite if such changes are to be managed successfully. Those who recognize that IT is a strategic lever, and who are actively considering and managing the areas, are more likely to be the success stories in the

90s. The challenge is to grasp the opportunity and ensure that IT provides value for money. Audit has a real opportunity to make a positive contribution to helping organizations achieve such success.

Chris Hurford, IPFA, joined the UK's District Audit Service in 1985 and is now associate director of the UK Audit Commission, with overall responsibility for advising the Commission's auditors on computer audit ing in local government, and for the provision of computing facilities throughout all the Commission's offices.

EFFECTIVE AUDIT INVOLVEMENT IN SYSTEMS DEVELOPMENT

Stan Dormer

The style wheel

There are many approaches taken to the audit of application systems and these form a progression from passive to active styles. They are illustrated in the form of a wheel (Figure 1). As you go clockwise around the wheel so the approach becomes more active and the time spent on the audit increases. Inevitably a point is reached where the level of involvement raises the question of whether audit independence and objectivity is being maintained. I leave this matter to the discretion of the department involved!

Time critical

It has long been recognized that the earlier the auditor gets involved in the systems development life cycle (SDLC) the greater the likelihood of an active dialogue developing and thus the greater the opportunity to influence the ultimate outcome. Different audit approaches demand different stages of involvement and different overall elapsed time spans. Figure 2 illustrates this theme.

©1991 Elsevier Science Publishers Ltd 7


Consultancy

Business Control Objectives/Systems Based Audit/VFM (BCO/SBA/3E)

Hindsight/Black Box (HBB)

t Intuitive Audit Approach (IAA)

Business Control Objectives Systems Based Audit (BCO/SBA)

Figure 1: The Audit Style Wheel ~ From Passive to Active

Value added through audit

The concept of value added is used to mean some increased benefit to the organization derived by the audit of an application. The value added through the audit of an application is some function of:

• Work input by the audit department (negative value factor) - - Factor 1.

Additional work input contributed by the systems design team (negative value factor) Factor2.

Improved contribution to the organization through better control and system effectiveness (positive value factor) - - Factor 3

Different audit approaches involve different audit costs in the form of the amount of work units

SDLC/PHASE

Post.Imp Review Implementation Test Programming Detail Design Outline Design Feasibility

Audit Approach IAA BCO/SBA/

HBB BCO/SBA 3E

I CorLsultancy

Figure 2: The SDL C and Audit Involvement

8 ©1991 Elsevier Science Publishers Ud


Work Units

12

10

8

6

4

2

0

HBB IAA BCO/SBA BCO/SBA/3E CONSULTANCY

FACTOR1 ~ Internal Audit Work Units

FACTOR2 [ i ' ] Additional Work Input by Design Team in Work Units

Figure 3: Combined Work Effort of Design Team and Internal Audit

involved. Figure 3 illustrates some averaged out values for evaluating the same application using different approaches. These figures will vary from organization to organization but the shape of the curve will broadly speaking be the same. If we make the assumption that the greater the interaction caused by the audit team to events during the SDLC then the greater the impact there will be on the design team's labours Factor2. Factors 1 & 2are combined in Figure 3 to give the combined work unit curve for the internal audit and the design team.

We must next make the assumption that the more methodical the approach and the greater

the work input by audit then the more, likely it will be that there will be improved control and greater system effectiveness. Reviews of various experiences of audit departments bear out this assumption and lead to a contribution chart, Figure 4, this represents Factor 3.

Figure 5 shows the effect of combining the results for Factors 1, 2 & 3. This implies that the greatest value is added by a proactive audit approach that includes value-for-money issues as well as a methodical structured approach to control issues.

Benefits Gained Consultancy Overall value 10 Curve 8 BCO/SBA/3E

6

-2 -4

. B B J~ \ pc°/sB~ k

~ C o n s u l t a n e y -6 IAA

Figure 4:FACTOR3 - Increased value through Int. Audit Contribution

Figure 5: Combination of increased value and work input

O1991 Elsevier Science Publishers Ltd 9

Computer Audit Update No vember 1991

However there is a snag. What has been revealed so far is that the better the job of work that is done within an application area, the better the pay-off of that particular system. But, it is known that the proactive audit approach involving VFM issues takes more time than other types of audit. What about the other systems that could have been looked at, but were omitted because there was insufficient time as a result of focusing attention on a few high quality audits in depth? In other words, value added cannot be maximized unless there is also the correct coverage.

A question of coverage

It is known from Factor 1 that the different audit approaches require relative work input (WI) of roughly the following:

HBB WI=I

IAA WI=5

BCO/SBA WI=3

BCO/SBA/3E Wl=4

This means that in the time that all applications could be covered using HBB approaches (wait till they go live then do a hindsight black box review), only 20% of them could be covered on an IAA basis, 33% of them on a BCO/SBA basis or 25% of them on a BCO/SBA/3E basis. So whilst the value added could be maximized for one system, a better opportunity might be missed elsewhere. What is needed is a mechanism for selecting the best candidates for review such that the value added can be maximized by adopting the correct audit style within the bounds of the real resources. A risk directed approach is required with a framework that predicts which systems are most likely to damage the organization if they fail to pedorm to expectations or are inadequately controlled.

Triage system for resource allocation

Assume a simple split of applications into three categories, and that the distribution of applications is roughly equal across the categories (this is borne out reasonably well in practice in most organizations), the categories would be: high risk, medium risk, and low risk. Assume that a stable equilibrium will be reached where there are as many sys tems in development as there are live (this is regrettable, but, true to life). Assume that an audit approach will be picked that is most beneficial to the category of application that is a candidate for review, thus:

• HBB for all live systems;

• BCO/SBA/3E for high risk applications in development;

• BCO/SDA for medium risk applications in development;

• and that low risk applications will be ignored until they are implemented.

Given the assumptions made this results in a workload of high:medium:low in a ratio of 5:4:1 (that is 90% of audit effort is concentrated away from low risk applications - - but they are not totally excluded from the reckoning as they become HBB candidates when live). In addition this gives a ratio of work input between systems in development:live systems of 7:3 (that is, 70% of audit effort is directed to affecting the course of events for systems which have not yet been implemented - - this again reflects the reality of trying to make changes before the event).

There now remains the requirement to be able to put applications into their appropriate risk categories. To do this, the following simple system is suggested. The system has six risk factors which are applied to each application, each factor is spl i t three ways into sub-categories (high, medium and low), each sub-category is scored (high=9, medium=6, low=3). In order for the risk factors to reflect their relative importance, weighting is applied to the

10 @1991 Elsevier Science Publishers Ltd

Computer Audit Update No vember 1991

Sub-cat Score Weight Value (S) (W) (SxW)

1. Potential impact on assets/liab's/cash flow H 9 32 M 6 32 L 3 32

2. Value of information held H 9 16 M 6 16

L 3 16

3. Value of Continuity H 9 8 M 6 8 L 3 8

4. Potential for consequential damage H 9 4 M 6 4 L 3 ~1

5. Impact if legal requirements not met H 9 2 M 6 2 L 3 2

6. Value of development in jeopardy H 9 1 M 6 1 L 3 1

7. Sum of values Total

Figure 6: Application Risk Factoring Table

scores. Finally the sum of the risk scores gives the total score for the application. The final score is compared with a range of scores to identify the application's category. The category will then determine the audit approach adopted for a system development review.

In this scheme (see Figure 6), applications scoring in the range 189-315 are rated 'low risk', applications scoring in the range 316-441 are rated 'medium risk', applications scoring in the range 442 -567 are rated 'high r isk'. A comparison of sample scores for some known applications can be used to adjust the width of the scoring ranges. Similarly factors and weights can be tuned to specific needs. A simple s p r e a d s h e e t / d a t a b a s e on a PC can be implemented, to capture data, in moments.

What has been achieved is to add maximum value by using the approach with the highest yield on the most appropriate subjects. So, having chosen the applications to review and the method of approach, the audi t can be considered, adding in some high pay-off tweaks.

BackUght the audit environment

It helps considerably to audit in a control c o n s c i o u s env i ronment . Aud i to rs in organizations which have a financial outlook have s ign i f i can t l y less p rob lems in communicating the need for adequate control and effectiveness than auditors in, for example, entrepreneurial retail environments. However, we can weight the dice more in our favour, for example by:



Allocating some time for educating key players in design roles in matters of control and audit interest.

Prepar ing a con t ro ls sect ion in the organization's systems development standards guide.

Actively promoting corporate issues like the need for a computer security policy, or the need for an overall IT strategy, or the need for contingency plans.

Ensuring that an annual state-of-the-art report on corporate security and control matters is issued from the audit department, showing whether matters are improving or otherwise: Indicate in this the key issues for the coming year.

Auditors need not be passive recipients of documentation. They can actively promote methods of working which will assist the rapidity with which audits can be conducted, for example by:

Encouraging auditors to obtain system over- views through briefing sessions with design teams rather than by wading through specifications.

Encouraging changes to the systems development standards so that all matters concerning control issues fall into a separate chapter within specifications using a standard model template.

Preparing standard questionnaires that the design team complete, concerning control issues, at the end of each phase of the SDLC.

Ensuring that key issues raised within one project, that may be symptomatic of the design culture, are rapidly communicated to other project teams that might be in a similar current position.

Ensuring that deficiencies in control or effectiveness gleaned from earlier audits are communicated as a matter of course, to any new projects that might replace or supplant ap-

plications containing these deficiencies.

Applications and systems audits are not one way processes whereby the designer is required to change his ways whilst the auditor moves on to his next project. Rapport and credibility will be considerably enhanced if the auditor adopts the BCO driven approach so the designer can see:

That auditors do have as clear a perspective of their mission as they expect the designer to have of his.

• That controls are based on what the business needs not what the auditor feels he needs.

That controls will be developed with a pur- pose as a designed element of the system, by design professionals and not arbitrarily im- posed.

That perceived weaknesses will be communicated at a stage early enough to be able to accommodate change.

That controls address issues which are as fundamental to the success of the system as any other functional element of the design.

The audit process is varied according to the style adopted, however, it is worth considering the following for various audit styles:

HBB Audits

• CAAT based 100% sampling.

• Combining Internal Audit interrogation needs with those of external auditors.

BCO/SBA Audits

• BCOs created by auditor/line management dialogue with signed off agreement.

Integration of computer installation and system software review results with SDLC applications work.

° Immediately it becomes apparent that BCO

12 O1991 Elsevier Science Publishers Ltd


unlikely to be met via current approach, notification of potential weakness.

Immediately BCO secured by control process truncate further work against that BCO.

• Identify CAATs required early in the SDLC, so that they can be ready for the live system.

• Decide whether to issue interim report before live date if BCOs likely to be impacted.

• Ensure audit tests based on BCO control process analysis.

Ensure audit documentation in fit state to be used for HBB approach in future (when live cycle review comes up).

BCO/SBN3E A u d i t s -

Ensure that quantification and performance targets, for economy, efficiency and effectiveness, are set early in SDLC for this system and that they will be measurable in the live system environment.

Was value added?

Internal Audit is a service department that assists management. Take a leaf from the service programmes being run by the better customer oriented organizations:

• Ensure that SDLC systems auditors are client orientated.

• Understand fully the SDLC approach being used within the organization.

Understand fully the behavioural implications of a proactive approach and yet the need to maintain independence and objectivity.

• Work to clear objectives within an appropriate audit style.

• Learn the subtlety of persuasion and negotia- tion skills.

• Keep working data on systems in order to be able to update audit timing and risk factors.

Finally, find out if the work done has actually added value, pose the following questions: what did the audit find of a serious nature and therefore what has been prevented? How long did the audit take and what did it cost? Could the audit have been more successful, quicker, more clear cut? Could procedures be improved?

Additionally, in order to get direct feedback from the customer as to the value of the audit, it is possible to prepare a standard audit/customer questionnaire that is issued to the SDLC team and the line manager responsible for live use of the system. Such a document can ask, for example, the following questions: How did internal audit perform in your eyes? Did we help or hinder the development? What was our most valuable contribution? What was our least valuable contribution?

Stan Dormer is a consultant with System Security Ltd, and has spent more than fourteen years working in computer audit. This article appeared in a slightly different format as a paper at the Compacs '91 Conference.

AN 'INTELLIGENT' APPROACH TO AUDIT TRAIL ANALYSIS

Frank Hickman

As organizations increase their dependency on information technology so the pressure to maintain the securi ty of that technology increases accordingly. Traditional approaches to computer security rely on the maintenance of a transaction or usage log, with perhaps daily review of the logs undertaken manually. This is an increasingly onerous and expensive task and one which, because of the sheer volume of


Documents

Effective audit involvement in systems development