T E C H N O L O G Y

the cpa & the c o m p u t e r

Evaluating Software Risk as Part of aRnancial AuditBy Yigal Rechtntan

The attention standards setters have given to the comput-erized environment has varied over the years. Currently,referenees to information technologies (ÍT) are sprinkledaround the margins of audit standards. Yet in most com-

plex organizations (defined here as a work environment largeenough to enable a degree of segregation of duties), themajority of an entity's internal controls are handled by IT.According to a GAO estimate, internal controls are approxi-mately 80% performed in some way by a computerized fea-ture {Standardsß>r lulernal Control in the Federal Government.November 1999. www.gao.gov/special.pubs/aiu0u21p.pdf).

6 8

Reliance on IT controls has become more formally recognizedunder the Public Company Accounting Oversight Board's(PCAOB) Auditing Standards 2 and 5. AS5 requires thefollowing:

127.] As part of evaluating the period-end financial reportingpnxess. the auditor should :Lsses.s—• Inputs, procedures f)ertbmied. and outputs of the processesthe company u.ses to prtxiuce its annual and quarterly finan-cial statements;• The extent of information technology ... involvement inthe period-end financial reporting process.

JUNE 2009 / THE CPA JOURNAL

Similarly. Statement on AuditingStandards 109. Understanding the Entityand Its Environment and Assessing theRisks of Material Misstatement, requiresatiditors to understand the risks in the inter-nal controls environment and specificallyassess the effect of IT on these risks.

Generally Accepted Auditing Standards(GAAS) require tliat "|t|he auditor should alsounderstand how information technologyaffects relevant control activities" (AU314.92). Accordingly. "[t]he use of infíMina-tion technology affects the way controlacti\ ities axt implemented." Other portions ofGAAS eliüx)rate on tJiis requirement, makingsure that aiiditi>rs ait cognizant of the breadthof IT controls as part of intemal controls.;ind o\' tlie specific nature of these controlsiuid the special types of risks they present.

Accordingly, auditors should be informedabout tlie risks and i-equirements with respectto fT-itiatcd audit risks. Tlic auditing of automated. pn:-prognuiimed. and otiierwise tech-nologically implemented contn)ls should beconsidereil in the same way thai nonauto-mated contrtils aie. with more attention givenlo the risks presented by computeri/x^ con-trols. Auditors should not assume that sub-st;uitive icsting can replace a detection mech-;mi.sm. Tliis thinking misses a crucial part inunderstanding how a computerized infor-mation system is vulnerable: Electronicrecords and pmgrams can change instanta-neously and without leaving a pafx̂ r tniil. Inaddition, computer applications can piixiucea kirge volume of ti"ansactions and bal;ïncesso large that traditional audit testing wouldbe a few orders (if magnittide below tlie pop-ulation si/e. It is thus imperative that audi-tors properly and fully undcTstand. assess, andrespond to the transactions, balances, and dis-closures prxxkiceti by these systems.

Tbe best appR)aL:h for an auditor is acombination of several types of auditresponses: I ) testing of the results of theIT processing; 2) re-performing selecttransactions; and 3) evaluating the qualityof the software u.sed to generate transac-tions and the amuunts that make theirway to the financial statements' balancesand disclosures. The first two methodsare very much like the old-fa.shioned prx>cedures that enabled auditors to auditaround the computer.

Tlic earlier paradigm assumed that com-puter processing was linear and simple.rhis view coasidered IT an "input-process-

output" activity, meaning that the validityof the inputs and outputs would ensurethe prtxiess is valid too. Eor reasons dis-cussed below, this assumption no longerholds true. Auditors who are interested inassessing and evaluating the quality of ITprcx:essing should consider software qual-ity as a primary methixl of evaluation andassessing control and detection risks with-in the audit risk assessment process. Thisarticle aims to describe and analyze the var-ious methods of assessing IT risks, espe-cially as related to the evaluation of soft-ware quality. These methods includebenchmarking, regression analysis, review-ing program code, and user acceptance.

BenchmarkingThe common thread among early com-

puter applications written before the popu-ladziition of the personal computer (PC) agewas this: Applications could be described aslinea]\ albeit complex, functions. A linearpRX'essing paradigm is one where there isonly one single process at any given time.Such a process is simple because it startswith iin input that is processed and resultsin an output; the output is often the input ofthe next process. Linear prtK'esses have oneand only one acti\e pnx-essing unit at anygiven time. (By contrast, parallel pntess-ing entails multiple processes with varyingdegrees of interdependence.) The complex-ities of a business solution were pared downto smaller and simpler progj-ams. Early com-puter programs were constructed such thatthe output of one program became an inter-mediary report as well as the input of thenext program. Data structures were alsorelatively simple: relational databases anddata silos were not yet common.

To financial auditors, these intermediatesteps, which were often in the form of rec-onciling reports, gave rise to the conceptof auditing artiund the computer. Becauseeach program processed data and pnxlucedsome type of reptirt. the auditors could fol-low the trail of prtx^essing from the initialinput until the final—and financially mate-rial—output. When il came to substantiat-ing the assertions that were produced bythese programs, auditors often used bench-marking to track only the changes that hadoccuned since the last audited peritïd.

For example, suppose that a general ledgerapplication combines monthly joumal entriesfirom various areas, such as accounts receiv-

able, accounts payable, and piiyroll. A lega-cy software would be written such that eachof these mixlules creates an interface file tothe genenil ledger program (step 1 ); a utili-ty prognim sciuis tbe interface file iuid vali-dates its input (step 2): tlie output file of tlieutility program becomes an input lile for thegeneral ledger program for general joumalentries (step 3). In this simplified ex;imple.the auditors could simply test the initial, inter-mediate, and final steps without underst:ind-ing exactly bow the softwiire worthed. If thetewere a change in sofhviire (e.g.. by adding adepartment ctxle to the chart ol' accounts"structure), auditors could benchmark tbe soft-watis by assuming that, except for tlie onenoted change, idl liinctionalities remained thesame and. as such, the only significant test-ing needed would be to ascertain tliat the œr-rect department was nxorded in tlie month-ly joumal prxxcssing.

As discu.ssed earlier, auditing aniund thecomputer is no longer viable, primarilybecau.se œmptiter programs are now bun-dled into software suites and the internalinterchanges between modtiles is not read-ily tractable. Eor the same reasons thatsimple computing models are no longerused, the concept of benchmarking cannotbe effectively applied in many cases.Benchmarking can effectively assist theauditors when linear software is used andwhen the interdependcncies between vari-ous parts of the software can be very clear-ly defined. AS5 pennits the use of bench-marking as a way to rely on software.Nevertheless, financial and technical audi-tors should be cognizant of the substiuitialweaknes.ses of this audit nietliodology whena complex input/output environment exists.Not only do the ptt)gram parameters andlogics need to be benclimarked but so dosignificant underlying operating systemlibriu-ies, auxiliary system functions, andother programs that operate as part andparcel of the software application.

Auditors should be aware of the preva-lence of "libraries'" of common functions fara particular softw^-e application. The use oflibraries almost always negates clear dis-tinctions between parts of the stïftware, evenif they can be otherwise logically separat-al. Libraries are a compilation of commonlyused functions that are shared amongmany modules of the same application.Eor example, a function tliat Itwks up anaccount in the chart of accounts may be

JUME 2009/THE CPA JOURNAL 69

placed in a libmry, which will allow all mcxl-ules of the applications to access it. Whenthe accounts receivable module needs tosearch for an account number in the chartof accounts, the function in the library isbeing used. The same function is used whena payroll nnxJule requin:s a kxikup in thechart of accounts. Becau.se libniries inter-connect several modules with their com-monly usetl functions, the logical separationof these nuxJules is obscured by the com-mon soltw;irc. When this (X'curs, the m(xl-ules become pail of one large applicationthat c;inm)t be readily evaluated fur changes,making it inappropriate to use benchmark-ing pitx'cdures for as.sessing risk.

Regression AnalysisEvaluating relationships between observed

instiuices and related results is well docu-mented in statistics. The strength of a rela-titiaship hetween an independen! variable anda dependent variable is often measured byR-square. R-square—known as the cocft'i-cient of detennination—measures how mucha set of data varies, and in particular the vari-ation 0Ï the dependent variable based onmovement of the independent variable. R-square is meiusure-d on a scale from -I to 1where a -1 represents a perfect inverserelationship and 1 represents a perfectdirect relationship; 0 represents no relatit)n-ship. For example, if the relationship betweengn)ss sales and sales commissions paid is0.85, it means that 85 cents of every dollarof sales commissions paid is related to themagnitude oí' gross sales.

When it comes to evaluating software,especially a calculation nnxlule that is crit-ical and material to the Imancial statement,the use of regression analy.sis and R-squ;irecan be a helpful tool. Two conditions oftenneed to be satisfied in order for a regres-sion iinalysis to lake place: First, the m(xl-ule must relate materially to a fmancialstatement assertion, and, second, inputs andoutputs must be readily available and costeffective. To explain the second condi-tion, note that the effectiveness of regres-sion analysis relies on a high number ofobservations. A software module that isbeing tested needs to be "fed" manyinstances of data and the output needs tobe evaluated for all of them. Accordingly.if manual entry is required, the audit pro-cedure is going to be costly. The purposeof the regression testing is to identify

anomalies. This is an effective way to iden-tify outputs that do not conform lo thespecification, especially in a complexsoftware application. When a change ismade to the logic of the software, a regres-.sion analysis process will re-create real-lifescenarios and values and evaluate the sys-tem's response. Tlie real-life scenarios mustbe somewhat random and somewhat con-sistent, because the goal is to test therevised softw;ire for its response, seekingoutlying circumstances.

Example. A telephone concern hasmany billing plans and a .software modulecalculates coinmissions for the sales forcebased on any number of variables on a per-contract basis. The software can be modi-fied in such a way that instead of manual-ly entering each contract, it can electroni-cally receive a large quantity of contractinfonnation and output the results in an

To assess the risk related to

softwaiB, a code review generBlly

includes at least an evaluation of

the change management process.

electronically efficient format. The pro-cessing is nonlinear: Multiple values andpriKesses funnel into a single result, andthe interdependencies make it a complexsoftware application. Several passes ofthe same data may be required, renderingthe software application nonlinear. Aregression analysis is then applied to theelectronic input and output. The purposeof the regression analysis is to kxate pos-sible anomalies in relationships betweenvariables. Suppose that .sales commissionsare higher for cellular phone contracts withtwo phones or more. A regression analy-sis should result in an R-square thatapproaches 1 for the reiationship betweenthe type of contract and number ofphones used (independent variables) andthe commission paid on these contracts.

In a sen.se. this is a highly complex ana-lyiical pr(x:edure that dties not evaluate theplausibility uf the dat;i, but instead the oper-ation of the calculation mtxlule. An outli-er for this type of a regression analysiswould occur where the output does nolagree with the specifics designed by thebusiness unit.

The caveat of using a regression analy-sis as a tool for evaluating software risksis that access to the source code is gener-ally required, as well as a level of ctK)per-ation that would allow high-quantity sam-ples of input and output in order to facili-tate the priKedure. The procedure is bestsuited ibr in-house programs and applica-tions, where there is at least one criticalmodule and reliable programming person-nel who can facilitate ihe audit assess-ment procedure.

Cede ReviewCode reviews are a very effective

method for establishing the lisk of inctir-rect operation by software that has an effecton a material as.seition in the financial state-ments. Similar to regression analyses, codereviews can be done at several levels andunder certain conditions. The key letiuircment is that the software c<Kle is availableand technical experts arc on hand to eval-uate it. OxJe reviews are sometimes akinto benchmarking because some varietiesevaluate not only the changes made buialso the priKess under which such changeswere made. To that effect, auditors shoukibe familiar with best practices in the waythat software pmgrams ;uid applications aremixiified. In brief, some of the.se best prac-tices include documentation of howchanges are made, tests performed innoncritical enviamments. and user accep-tance of the final prixJuct.

To as.sess the risk related to software,a code review generally includes at leasian evaluation of the change manage-ment process . More often, a codereview involves some degree of scrutinyot the actual programming language usedto develop an application. To accomplisha code review, a human or machine scansthe application .source programming CIKICto evaluate its susceptibility to knownprogramming errt)rs. One such error caninclude hardwired values instead olvariable, configurable values. For exam-ple, a payroll program should not write

70 JUNE 2009 / THE CPA JOURNAL

the FICA percentage into the source code; if the FICA per-centage needs to be changed, finding all the instances wherethai hardwired value of 7.659^ is used will be costly and mayrender the software unreliable.

If the regular development of the software is undertaken inaccoidaiice with software development best practices, auditors canmoi-e readily rely on the software's d(x:umentation. These ofteninclude budgets, initiation forms, technical specifications, andIhe results of user testing that, in the aggregate, will indicate thatyood programming practices were undertaken in developing thesoftwiire. Reviewing ihe source ctxJe with a human eye shouldbe done by ¡in expert programmer. There are also automated solu-tions tliat can be configured to search source code for certain typesof breaches in programming.

User AcceptanceAs part of practices for developing software code, auditors

should be aware that initial, intemal, and final approval by endusers is crucial to the quality of the software. Users are often thesubject-matter experts who can most prudently specify ;ind eval-uate the uses that the software should have. Even software thatis not programmable per se might have changes that are drivenby specific values. For example, software may have a fixedlogic but contain kxjkup tables; a mistaken value in the kwkuptable can cause the software to operate incorrectly, even if theprogramming logic is correct. One example familiar to most CPAsis payroll software that requires annual updates to the witliliold-ing tables and FICA limits.

In situations where the software program code changes, endusers' spécifications, involvement, and approval are essentialto the quality of the resulting software product. When the devel-opment process is adequately documented, it provides audi-lors with a cradle-to-grave audit trail. In most environments,even those that only use configurable software (such as the pay-roll application example above), user acceptance procedureshave a direct effect on the auditor's assessment and the audi-tor's reliance on the results of the balances and disclosuresthat the software produces.

Exatnple. A medical billing practice uses industry-specificsoftware to help process claims; the practice also uses thesoftware to generate accounting entries for its own internal use.Assume that there is a change in the formula by which the med-ical billing practice charges the doctors who are its customers.The software vendor receives an initial request Ibr the changeand follows up with a diKumented liowchart, along with exam-ples of what the new formula would be; the practice reviewsihe complete kit and formally approves il. The software devel-oper then provides a preliminary program that runs in a testenvironment. The users are given sufficient time andresources to evaluate the new program in the te.st environmenttefore foniially accepting and approving the change. The soft-ware vendor reviews all the test results to prove that the pro-gram is operated as intended under the new formula. The soft-ware vendor then selects an appropriate date and, in coordina-tion with the users, migrates the new application from thetesting environment to the real production environment. Oncethe new .software is installed, users conduct further testing before

approving the software in a final formal communication. Thistype of change management allows all parlies involved—fromthe software developer to the users to regulatory and account-ing stakeholders—to be satisfied that all necessary qualitycontrol steps have been taken to ensure a successful revisionthat will serve the user's needs.

Mitigating RisksComputer applications and information technologies present

certain audit risks to even the best-prepared auditor. Many audi-tors consider IT specialists to be unnecessjuy because substan-tive auditing is considered suftlcient lo compensate for a lack ofcontrols in the computerized environment. Tliis author believessuch auditors may be in error. IT systenLs have become t«) Iargeand complex—and busines,ses rely upon them so pervasively—that auditors cannol ignore the audit risks ihat they represent.Understanding, assessing, and responding to audit risk.s as theyarise from the IT environment is not just a practical míttter. It isa necessary part of adhering to auditing standards. Ü

Yigal Rechtinan, CPA, CFE, CISM, CITP, is a director for infor-mation teclinoloiiw technology iissurance. and forensic servicesat Buchbinder Tunick & Co.. LLP. New York. N.Y.

Join us this summer to earn your CPB!last Chance to Sign Up!

Check out what FAE has in store for you in J U N E !

Personal Financial PlanningConferenceTuesday, June 9. 2009Bernstein Global WealthManagement1345 Avenue of ttie AmericasNew York. NY 10105Course Code: 25275011

Anti-Fraud/Anfi-MoneyLaundering ConterenceWednesday, June 10,2009New York Marriott Marquis atTimes SquareNew York, NY 10036Course Code: 25175011

CFOs, Controllers, andFinancial ExecutivesConierenceThursday, June 11, 2009New York Helmsley HotelNew York, NY 10017Course Code: 25269011

Accounting and Auditing inthe Non-Public (Non-Issuer)Environment ConferenceTuesday, June 16, 2009New York Marriott Marquis atTimes SquareNew York. NY 10036Course Code: 25137011

1RS Practice and ProceduresConferenceThursday. June 25, 2009New York Marriott Marquis atTimes SquareNew York, NY 10036Course Code: 25609011

FAE

Register today with your POP Pass. Visit www.nysscpa org/faeorg/popintrohtm

For more information, please visit v™w,nysscpa.org, or call 800-537-3635.

JUNE 2009 / THE CPA JOURNAL 71